You are here

Monitoring with Munin

I use Munin (http://munin-monitoring.org/) to keep an eye on the servers performance and spot problems before they become critical. Munin consist of a front-end and a back-end both can be installed on the same server, so it's possible to run Munin on the Aegir server. I do however run the front-end on another machine, which is dedicated to monitor the other servers.

If the server does not already run a web-server (Apache or Nginx) you need to install it first to run the Munin front-end.

~$ apt-get install apache2
~$ apt-get install munin-node munin-plugins-extra libwww-perl munin

Apache configuration (front-end)

Will start be configuring Apache on the front-end server and redirect all communication for the domain munin. to https to enhance the security on the server. Edit the configuration file and add the virtual host settings listed below.

~$ nano -w /etc/apache2/sites-available/munin

The newest version of Munin comes with an Apache configuration file in /etc/munin/apache.conf, which is used by Apache as default.

<VirtualHost :80>
    ServerAdmin webmaster@<servername>
   ServerName munin.<servername>

   RewriteEngine On
   RewriteCond %{HTTPS} !=on
  RewriteRule ^(.
) https://%{SERVER_NAME}$1 [R,L]

  DocumentRoot /var/cache/munin/www

ErrorLog ${APACHE_LOG_DIR}/error.log

  LogLevel warn
  CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>

<IfModule mod_ssl.c>
<VirtualHost default:443>
ServerAdmin webmaster@<servername>
   ServerName munin.<servername>

   DocumentRoot /var/cache/munin/www
  <Directory />
        Options FollowSymLinks
     AllowOverride All

                # Setup basic password authentication
                AuthUserFile /etc/apache2/munin.pwd
                AuthType Basic
                AuthName "Munin login"
                Require valid-user
    </Directory>
<Directory /var/cache/munin/www/>
        Options FollowSymLinks MultiViews
      AllowOverride All
      Order allow,deny
       allow from all
</Directory>

    ErrorLog ${APACHE_LOG_DIR}/error.log

  # Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
    LogLevel warn
  CustomLog ${APACHE_LOG_DIR}/ssl_access.log combined

   SSLEngine on
   SSLCertificateFile    /etc/ssl/certs/ssl-cert-snakeoil.pem
SSLCertificateKeyFile /etc/ssl/private/ssl-cert-snakeoil.key

  <FilesMatch ".(cgi|shtml|phtml|php)$">
        SSLOptions +StdEnvVars
</FilesMatch>
    <Directory /usr/lib/cgi-bin>
     SSLOptions +StdEnvVars
</Directory>

    BrowserMatch "MSIE [2-6]" \
      nokeepalive ssl-unclean-shutdown \
     downgrade-1.0 force-response-1.0
   # MSIE 7 and newer should be able to use keepalive
BrowserMatch "MSIE [17-9]" ssl-unclean-shutdown
</VirtualHost>
</IfModule>

Enable the rewrite and SSL Apache extensions and enable the Munin virtual host created above.

~$ a2enmod ssl rewrite
~$ a2ensite munin
~$ a2dissite default
~$ /etc/init.d/apache2 restart

The virtual host configuration above sets up basic password protection, so we need to create the password file with the command below. You can add more users to the file, just omit the -c on the command line.

~$ htpasswd -c /etc/apache2/munin.pwd <username> 

Munin

Next step is to configure Munin front-end and back-end on the monitoring server before looking into connecting to a remote server and pulling stats.

Front-end

Edit the Munin configuration file and rename the localhost server name.

~$ nano -w /etc/munin/munin.conf

The name should be the fully qualified domain name (FQDN) for the server because Munin will build a server tree based on the FQDN's.

[<servername>]
    address 127.0.0.1
    port 4949
    use_node_name yes

Back-end

The back-end consists of plug-ins, which is used to probe the server in different ways to get statistical information. All Munin's plug-ins are location in "/usr/share/munin/plugins/" and to activate a given plug-in you need to create a symbolic link in "/etc/munin/plugins/" as below. You can deactivate a plug-in be removing the sym-link.

~$ ln -s /usr/share/munin/plugins/cpuspeed /etc/munin/plugins/

After changing the symbolic links you have to restart the node for the changes to take effect.

~$ /etc/init.d/munin-node restart

Remote Munin node(s)

We now have the monitoring server running and we are now going to look at how to set up a remote node on a Aegir server. We are interested in monitoring Apache, APC, Memcached, MySQL and general server load.

Start by installing the node with the command below as root.

~$ apt-get install munin-node munin-plugins-extra libwww-perl

Aegir Apache configuration

When installing Aegir on a server it basically takes over the Apache configuration and we need to be able to pull the Apache status page etc. It's possible to add custom Apache configuration to the Aegir stack in the folder "~/config/server_master/apache/pre.d/". This part of the guide assumes that you have install Aegir with the home directory "/home/www/aegir".

~$ nano -w /home/www/aegir/config/server_master/apache/pre.d/munin

Add the following code to the file.

<VirtualHost *:80>
  ServerName localhost

  DocumentRoot /var/www
  <Directory />
    Options FollowSymLinks
    AllowOverride All
  </Directory>
  <Directory /var/www/>
    Options FollowSymLinks MultiViews
    AllowOverride All
    Order deny,allow
    Deny from all
    Allow from localhost ip6-localhost
    Allow from 127.0.0.1
  </Directory>

  ErrorLog ${APACHE_LOG_DIR}/error.log

  LogLevel warn
  CustomLog ${APACHE_LOG_DIR}/access.log combined

  # Set server status handler
  <Location /server-status>
    SetHandler server-status
    Order deny,allow
    Deny from all
    Allow from localhost ip6-localhost
    Allow from 127.0.0.1
  </Location>
</VirtualHost>

Munin plug-ins

As on the monitoring server plug-ins are enabled by creating symbolic links and the configuration is located in the file "/etc/munin/plugin-conf.d/munin-node".

Apache

To monitor Apache check that the following sym-links exists in "/etc/munin/plugins".

  • apache_accesses -> /usr/share/munin/plugins/apache_accesses
  • apache_processes -> /usr/share/munin/plugins/apache_processes
  • apache_volume -> /usr/share/munin/plugins/apache_volume

Edit the node configuration file and add the code below to set up the Apache plug-in.

~$ nano -w /etc/munin/plugin-conf.d/munin-node
[apache_processes]
env.url http://localhost:%d/server-status?auto
env.port 80 

[apache_accesses]
env.url http://localhost:%d/server-status?auto
env.port 80

[apache_volume]
env.url http://localhost:%d/server-status?auto
env.port 80

APC

Start by downloading the plug-in from here: http://code.google.com/p/munin-php-apc/downloads/detail?name=munin_plugi... and create sym-links to install the plug-in.

~$ wget http://munin-php-apc.googlecode.com/files/munin_plugin_php_apc-0.1.zip
~$ unzip munin_plugin_php_apc-0.1.zip
~$ cp php_apc/apc_info.php /var/www/
~$ cp php_apc/php_apc_ /usr/share/munin/plugins/
~$ ln -s /usr/share/munin/plugins/php_apc_ /etc/munin/plugins/php_apc_files
~$ ln -s /usr/share/munin/plugins/php_apc_ /etc/munin/plugins/php_apc_fragmentation
~$ ln -s /usr/share/munin/plugins/php_apc_ /etc/munin/plugins/php_apc_hit_miss
~$ ln -s /usr/share/munin/plugins/php_apc_ /etc/munin/plugins/php_apc_purge
~$ ln -s /usr/share/munin/plugins/php_apc_ /etc/munin/plugins/php_apc_rates
~$ ln -s /usr/share/munin/plugins/php_apc_ /etc/munin/plugins/php_apc_usage

Edit the node configuration file and set up APC.

~$ nano -w /etc/munin/plugin-conf.d/munin-node
[php_apc_*]
user root
env.url http://localhost/apc_info.php?auto
~$ /etc/init.d/munin-node restart

Memcache

Download memcache plugin memcached.tar.gz the plug-in was found here: http://munin-monitoring.org/browser/trunk/plugins/node.d/memcached_.in.

The plug-in is written in Perl and needs the Memcached Perl library installed to communicate with memcache.

~$ apt-get install libcache-memcached-perl
~$ tar -zxvf memcached.tar.gz
~$ cp memcached_ /usr/share/munin/plugins
~$ ln -s /usr/share/munin/plugins/memcached_ /etc/munin/plugins/memcached_rates
~$ ln -s /usr/share/munin/plugins/memcached_ /etc/munin/plugins/memcached_bytes
~$ ln -s /usr/share/munin/plugins/memcached_ /etc/munin/plugins/memcached_counters
~$ /etc/init.d/munin-node restart

Secure connection to node(s)

The create a secure connection between the monitoring server and the node(s) I create SSH tunnels and uses auto ssh to ensure that the connection is recreate automatically on failure. You have to use a new port for each node. The connection is create from the monitoring server to the node and have to be created with a user that has SSH access to the node.

~$ apt-get install autossh
~$ autossh -fN <servername/node> -L <local port>:localhost:4949

Add the node to the Munin front-end by edit munin.conf and add the node the the server tree as below.

~$ nano -w /etc/munin/munin.conf
[<servername>]
    address 127.0.0.1
    port <local port>
    use_node_name yes

To create the SSH tunnel after a reboot you can add the following command to "/etc/rc.local". It do require that you have a user that are allowed to ssh into the remote server without password (public/private keys).

/usr/bin/autossh -fN <user>@<servername> -L <local port>:localhost:4949 -i /home/<user>/.ssh/id_rsa

Add new comment

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.