Munin surveys all your servers and remembers what it saw. It presents all the information in graphs through a web interface. Its emphasis is on plug and play capabilities. While it may not be the most scalable solution, it's flexible, yet easy to understand when monitoring a dozen or so servers.

Using Munin you can easily monitor the performance of your computers, networks, SANs, applications, weather measurements and whatever comes to mind. It makes it easy to determine "what's different today" when a performance problem crops up. It makes it easy to see how you're doing capacity-wise on any resources.

Munin uses the excellent ​RRDTool (written by Tobi Oetiker) and the framework is written in Perl, while plugins may be written in any language.

This how-to guide explains how to install and configure Munin and Munin Node on CentOS 5,6,7 or Fedora.

Note: This guide assumes you are familiar with SSH and basic command line navigation. These instructions apply primarily to customers who have Virtual Private Servers or Dedicated servers. If you do not have root-level access you will not be able to make these changes.

1. Installation

1.1 Install EPEL Repository

All files required for installation of Munin are contained in the EPEL repository. The EPEL repository provides easy to install packages for commonly used software.

# yum install epel-release

If this command does not work, perhaps because the CentOS Extras repository is disabled, the section below provides manual installation instructions based on your distribution version.

CentOS and Red Hat Enterprise Linux 5.x

# wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-5.noarch.rpm sudo rpm -Uvh epel-release-5*.rpm

CentOS and Red Hat Enterprise Linux 6.x

# wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm sudo rpm -Uvh epel-release-6*.rpm

CentOS and Red Hat Enterprise Linux 7.x

# wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm sudo rpm -Uvh epel-release-7*.rpm

If you receive a File Not Found Error message when trying to download the package, the version number might have changed. You can access the latest version of the RPM installer from the Fedora EPEL wiki page. The wiki page also includes additional instructions for Red Hat Network subscribers who are installing the EPEL repository.

Note: There is no need for Fedora users to install the EPEL repository. Munin is already included in Fedora and can be installed using yum (see step 1.2).

1.2 Installation Munin and Munin Node

• Munin is the daemon service aggregating all data, which is then pulled to the daemon. It then uses RRDtool to log and generate updated graphs and it renders all webpages. It runs only on your host server.
• Munin Node collects data on your servers. It needs to be installed and configured on each server you wish to monitor.

Before we begin, first ensure that your server is up-to-date:

# yum update

You are now ready to issue the command to install both Munin and Munin Node on your server. This command also installs all dependencies, such as RRDTool that are required by Munin.

# yum install munin munin-node

Munin node will have to be installed on each server you wish to monitor. Follow the above steps, but only install the munin-node service on each server you wish to monitor:

# yum install munin-node

Now create the system startup links for Munin and startup the Munin daemon for the first time:

CentOS 5,6 and Fedora

# service munin-node start
# chkconfig --level 35 munin-node on

CentOS 7

# systemctl enable munin-node.service
# systemctl start munin-node.service

Note: Munin requires a working web server on the host server, such as Apache or Nginx to render and display its statistics pages.

1.3 Remove Munin and Munin Node

# yum remove munin-node

2. Configuring Munin and Munin Node

Note: Munin requires firewall port 4949 to be opened. The Security of your Munin installation is outside the scope of this tutorial. For a guide detailing the ins and outs of the popular CSF firewall please follow this guide

2.1 Configuring Munin

The configuration file for Munin can be found at /etc/munin/munin.conf

# dbdir   /var/lib/munin
# htmldir /var/www/html/munin
# logdir  /var/log/munin
# rundir  /var/run/munin

You can simply uncomment the values above. The defaults are fine in most cases. You can also adjust the htmldir accordingly depending on where you would like the files to be generated.

In case you'd like Munin to be tied in to a DirectAdmin account, you'd enter the following:

dbdir   /var/lib/munin
htmldir /home/username/domains/domain.com/)
logdir  /var/log/munin
rundir  /var/run/munin

The host tree should be adjusted as well. You can set a custom name for your server. All remote servers you wish to monitor need to be listed here:

# a simple host tree
[localhost.or.enter.your.custom.server.name.here]
address 127.0.0.1
use_node_name yes

[remote.server.or.enter.your.custom.server.name.here]
address xxx.xxx.xxx.xxx
use_node_name yes

Once the munin.conf file has been updated, you’ll need to adjust the permissions of the html directory.

In the example below I set my html directory to /var/www/html/munin . To allow munin to write to it we’ll have to change ownership to the correct user:

# chown munin:munin /var/www/html/munin

2.2 Password protecting Munin (optional)

Edit /etc/httpd/conf.d/munin.conf and uncomment if required the htpasswd section as below:

# AuthUserFile /etc/munin/munin-htpasswd
# AuthName "Munin"
# AuthType Basic
# require valid-user

to:

AuthUserFile /etc/munin/munin-htpasswd
AuthName "Munin"
AuthType Basic
require valid-user

If required be sure to rename the AuthName to a username you prefer.

Then password protect Munin with a username and password using Apache basic auth module as shown below. The username "Munin" can be replaced with anything you desire:

# htpasswd /etc/munin/munin-htpasswd Munin

2.3 Limit access to Munin by IP address (optional)

Open the following file /etc/httpd/conf.d/munin.conf and add the following lines to the directory directive

Order Deny,Allow
Deny from all
Allow from 127.0.0.1 192.168.1.0/24

Where xxx.xxx.xxx.xxx/24 represents the external IP address you wish to view the statistics with. If you wish to allow all clients access to Munin graphs, change the line Allow from 127.0.0.1 xxx.xxx.xxx.xxx/24 to Allow from All.

Now is a good time to restart Munin in order to make the changes take effect:

CentOS 5,6 and Fedora

# service munin-node start

CentOS 7

# sysctl restart munin-node.service

2.4 Configuring Munin Node

The configuration file for Munin Node can be found at /etc/munin/munin-node.conf

For security reason, I'd like Munin to run under a less risky user.

Change:

user root
group root

to:

user munin
group munin

Alter the listening address to bind to localhost only:

# Which address to bind to;
host *
host 127.0.0.1

Add the IP Address of the Munin Daemon: Do not forget to leave localhost (127.0.0.1) active when configuring external munin nodes in munin-node.conf So, you'll need a total of 2IP's minimum.

allow ^127\.0\.0\.1$
allow ^12\.12\.12\.12$   ^IP\.ADD\.RE\.SS$ of munin master

3. E-Mail Notifications / Alerting

3.1 Configure Contacts

Munin has basic support for sending mails when specified warning and critical thresholds are exceeded. If you aren’t already running a notification system, such as Nagios, Munin’s notification are quick to setup and work reliably. A few changes to munin.conf are all that’s needed.

First, you need to configure contacts, which enables the notification system.

contacts me # may be unnecessary and cause warnings
contact.me.command mail -s "Munin notification ${var:host}" user@host.tld
contact.me.always_send warning critical
contact.alert.command mail -s "Munin notification ${var:host}" user@host.tld
contact.alert.always_send critical

Above, I define a contact entry named me. Pick any name you would like. Next, the specific command action is defined. It’s a simple string to be executed by a shell which will receive the body of the alert message as STDIN. For sending email an appropriate command such as mail is used. Finally, mails will always be sent on any warning or critical state.

if required you can add a specific user that you send the alert to in the host tree of /etc/munin/munin.conf:

[domain;serverA]
address aaa.aaa.aaa.aaa
use_node_name yes
contacts me alert

3.2 Set Tresholds

Tresholds can be set by editing the host tree in /etc/munin/munin.conf.

[domain;serverA]
address aaa.aaa.aaa.aaa
load.warning 3
load.critical 5
diskstats_latency.sda.avgrdwait.warning 0:3
diskstats_latency.sda.avgrdwait.critical 0:5
diskstats_latency.sda.avgwrwait.warning 0:3
diskstats_latency.sda.avgwrwait.critical 0:5
diskstats_latency.vg_host_lv_swap.avgrdwait.warning 0:10
diskstats_latency.vg_host_lv_swap.avgwrwait.warning -5:5
use_node_name yes
contacts me alert

The easiest way to figure out the correct graph name is by hovering over the graph.

4. Plugins

4.1 MySQL

If you’re running a server with CentOS 5/6 with Munin installed to graph the system stats, you’ll mostly likely notice that the generated graphs do not include MySQL.

This is expected behaviour since cPanel, DirectAdmin or Virtualmin ask you to define a root password for your MySQL server (so it isn’t left unsecured out in the open). Unfortunately, by default, Munin tries to read MySQL data using the root account and without password and thus fails.

4.1.1 MySQL User Setup

DirectAdmin

Enter the following to reveal the MySQL login credentials on your DirectAdmin based server

# cat /usr/local/directadmin/conf/mysql.conf

Virtualmin

Login to Virtualmin and navigate to Webmin > Servers > MySQL Database Server. Create a new user munin and define a password, set it to only log in from localhost and add the Superuser permission.

Now navigate to Database Permissions and add the previously created munin user on database MySQL, again only for localhost and only grant the Select table data permission.

4.1.2 MySQL Dependencies

For MySQL monitoring to work we need a few dependencies.

For Mysql 5.1x:

yum install perl-Cache perl-Cache-Cache
yum install perl-IPC-ShareLite perl-DBD-MySQL

For Mysql 5.5x and higher:

sudo perl -MCPAN -e 'install +YAML'
shell> perl -MCPAN -e shell
cpan> install DBI
cpan> install DBD::mysql

or by using CPAN:

cpan> fforce install DBD::mysqlc

if perl-Cache-Cache is missing, first install and enable the EPEL repository (see 1.1) and then continue by installing the module.

yum install perl-Cache-Cache

4.1.3 MySQL User Credentials

DirectAdmin

Open the /usr/share/munin/plugins/mysql_ file and edit:

You should see something like this with the credentials left blank:

my %config = (
    'dsn'        => $ENV{'mysqlconnection'} || 'DBI:mysql:mysql',
    'user'       => $ENV{'mysqluser'}       || 'da_admin',
    'password'   => $ENV{'mysqlpassword'}   || 'removed',
);

Virtualmin

Open the /etc/munin/plugin-conf.d/munin-node file and add (or edit if already exists):

[mysql*]
env.mysqladmin /usr/bin/mysqladmin
env.mysqluser munin
env.mysqlpassword your-previously-defined-password
env.mysqlopts -umunin -pyour-previously-defined-password

If you’re using innodb tables, you will need to create the /etc/munin/plugin-conf.d/mysql_innodb file to ignore InnoDB related errors (which Munin will most likely generate).

[mysql_innodb]
env.warning 0
env.critical 0

4.1.4 MySQL Select services to monitor

Enter the following to reveal what MySQL plugins are available:

# /usr/share/munin/plugins/mysql_ suggest
mysql_bin_relay_log
mysql_commands
mysql_connections
mysql_files_tables
mysql_innodb_bpool
mysql_innodb_bpool_act
mysql_innodb_insert_buf
mysql_innodb_io
mysql_innodb_io_pend
mysql_innodb_log
mysql_innodb_rows
mysql_innodb_semaphores
mysql_innodb_tnx
mysql_myisam_indexes
mysql_network_traffic
mysql_qcache
mysql_qcache_mem
mysql_replication
mysql_select_types
mysql_slow
mysql_sorts
mysql_table_locks
mysql_tmp_tables

To enable all of them simply run the following:

cd /etc/munin/plugins
ln -sf /usr/share/munin/plugins/mysql_ mysql_
for i in `./mysql_ suggest`; \
do ln -sf /usr/share/munin/plugins/mysql_ $i; done

If you only need a few of them you can apply them this way:

cd /etc/munin/plugins
ln -sf cmysql_ mysql_bin_relay_log
ln -sf /usr/share/munin/plugins/mysql_ mysql_commands
ln -sf /usr/share/munin/plugins/mysql_ mysql_connections
ln -sf /usr/share/munin/plugins/mysql_ mysql_files_tables
ln -sf /usr/share/munin/plugins/mysql_ mysql_innodb_bpool

4.1.5 MySQL Test plugin

To test if the plugin works, run in the console:

munin-node-configure --suggest | grep mysql

The output should be something like below:

mysql_                     | yes  | yes (+bin_relay_log +commands       and a lot of other blabber

Reload Munin Node

Be sure to reload munin-node:

/etc/init.d/munin-node restart

or for CentOS7

systemctl restart munin-node

4.2 Soft RAID/SMART Monitoring

Monitor your drive and RAID health and performance. Begin by ensuring that you have smartmontools installed:

yum install smartmontools

Symlink the plugins:

cd /etc/munin/plugins
ln -s /usr/share/munin/plugins/smart_ smart_sda
ln -s /usr/share/munin/plugins/smart_ smart_sdb

Edit the following file:

nano /etc/munin/plugin-conf.d/munin-node

Add the following:

[smart_*]
user root
group disk

[smart_sda]
env.smartargs -H -l error -d ata

[smart_sdb]
env.smartargs -H -l error -d ata

4.3 Ping

It's always good to have a clue about your connectivity to the rest of the world. You can setup the ping plugin using a simple correctly formatted symlink:

# ln -s /usr/share/munin/plugins/ping_  /etc/munin/plugins/ping_google.com
# ln -s /usr/share/munin/plugins/ping_  /etc/munin/plugins/ping_ovh.fr
# ln -s /usr/share/munin/plugins/ping_  /etc/munin/plugins/ping_example.com

4.4 FTP

To monitor FTP activity for an account lftp

yum install lftp

Once lftp is installed, it's time to create our monitoring script: /usr/share/munin/plugins/backup_space

#!/bin/bash
MAXLABEL=20

if [ "$1" == "autoconf" ]; then
        echo yes
  exit 0
fi

if [ "$1" == "config" ]; then
  echo 'graph_title Backup-Space'
  echo 'graph_args --base 1000 -l 0'
  echo 'graph_vlabel Space in GB'
  echo 'graph_category Backup'
  echo 'graph_order max space'
  echo 'space.label Current'
  echo 'space.draw LINE'
  echo 'max.label Max'
  echo 'max.draw AREA'
  exit 0
fi

echo -en "space.value " && echo du . | lftp -u ${user},${pass} ${ip} | awk -v LIMIT="${max}" '$1~/[0-9]+/ {QUOTA=LIMIT*1024*1024; print ($1)/1024/1024 }' | tail -1
echo -en "max.value " && echo ${max}

save this file and create the following symlink:

ln -s /usr/share/munin/plugins/backup_space.sh /etc/munin/plugins/backup_space

Edit the /etc/munin/plugin-conf.d/munin-node file

[backup_space]
env.user username
env.pass passwort
env.ip ftpserver
env.max 100

If you test the script and get the error Certificate verification: Not trusted:

# /etc/munin/plugins# munin-run backup_space

Edit /etc/lftp.conf

And set:

ssl:verify-certificate no

5. Debugging Munin

5.1 Compatibility check

To see which plug-ins are installed on a munin node, log into it and run the munin-node-configure command:

# munin-node-configure
Plugin                     | Used | Extra information                      
------                     | ---- | -----------------                      
acpi                       | no   |                                        
apache_accesses            | no   |                                        
apache_processes           | no   |                                        
apache_volume              | no   |                                                                              
courier_mta_mailqueue      | no   |                                        
courier_mta_mailstats      | no   |                                        
courier_mta_mailvolume     | no   |                                        
cps_                       | no   |                                        
cpu                        | yes  |                                        
cupsys_pages               | no   |                                        
df                         | yes  |                                        
df_abs                     | no   |                                        
df_inode                   | yes  |                                        
entropy                    | yes  |                                        
exim_mailqueue             | yes  |                                        
exim_mailstats             | yes  |                                        
forks                      | yes  |                                        
fw_conntrack               | no   |                                        
...

The output is longer than that, but that's generally what you'll see — the plug-ins with “yes” next to them are enabled, the ones with “no” next to them are installed but disabled.

Deciding what to enable

To help you decide whether you want to enable any other munin plug-ins, run munin-node-configure with the ”–suggest“ option: munin-node-configure --suggest

# munin-node-configure --suggest
Plugin                     | Used | Suggestions                            
------                     | ---- | -----------                            
acpi                       | no   | no [cannot read /proc/acpi/thermal_zone/*/temperature]
amavis                     | no   | no                                     
apache_accesses            | no   | no [ExtendedStatus option for apache mod_status is missing on port 80]
apache_processes           | no   | no [ExtendedStatus option for apache mod_status is missing on port 80]
apache_volume              | no   | no [ExtendedStatus option for apache mod_status is missing on port 80]
apc_envunit_               | no   | no [no units to monitor]               
bonding_err_               | no   | no [No /proc/net/bonding]              
courier_mta_mailqueue      | no   | no [spooldir not found]                
courier_mta_mailstats      | no   | no [could not find executable]         
courier_mta_mailvolume     | no   | no [could not find executable]         
cps_                       | no   | no                                     
cpu                        | yes  | yes                                    
cpuspeed                   | no   | no [missing /sys/devices/system/cpu/cpu0/cpufreq/stats/time_in_state]
cupsys_pages               | no   | no [could not find logdir]             
df                         | yes  | yes                                    
df_inode                   | yes  | yes                                    
diskstats                  | yes  | yes                                    
entropy                    | yes  | yes                                    
exim_mailqueue             | no   | no [no exiqgrep]                       
exim_mailstats             | no   | no ['/usr/sbin/exim -bP log_file_path' returned an error]
fail2ban                   | no   | no [/usr/bin/fail2ban-client not found]
...

If your output includes any error messages at the end or complaints about “junk output”, don't worry. That just means a plug-in was queried that didn't give a proper response, it doesn't affect munin's operation.

Note that the output with ”–suggest“ is usually shorter than the complete list of installed plug-ins. When you ask munin-node-configure for its suggestions, it lists plug-ins that are written to respond to that query and also omits many plug-ins that are already installed.

Each plug-in name is followed by a “yes” or “no” indicating whether or not it's currently enabled. The last column might contain a suggestion regarding a plug-in's status. A “yes” suggestion indicates that a plug-in is currently supported by your system and can be enabled without making changes (like the postfix plug-ins in the example above). Other entries in the “Suggestions” column can offer details about issues that need to be corrected before enabling a given plug-in.

5.2 Test Plugins

On a host where munin-node is installed, you can check to see whether the plugin runs by using the munin-run command. You can test a plugin with and without the optional config and debug variables.

Regular run:

# munin-run df
_dev_hda1.value 83

Config run:

# munin-run df config
graph_title Filesystem usage (in %)
graph_args --upper-limit 100 -l 0
graph_vlabel %
graph_category disk
graph_info This graph shows disk usage on the machine.
_dev_hda1.label /
_dev_hda1.info / (ext3) -> /dev/hda1
_dev_hda1.warning 92
_dev_hda1.critical 98

5.3 Telnet to run munin-node plugin

Does the plugin run through munin-node, with and without config? Hint: Telnet to port 4949.

Regular run:

# telnet foo.example.com 4949
Trying foo.example.com...
Connected to foo.example.com.
Escape character is '^]'.
# munin node at foo.example.com
fetch df
_dev_hda1.value 83
[...]
.

With config:

# telnet foo.example.com 4949
Trying foo.example.com...
Connected to foo.example.com.
Escape character is '^]'.
# munin node at foo.example.com
config df
graph_title Filesystem usage (in %)
graph_args --upper-limit 100 -l 0
graph_vlabel %
graph_category disk
graph_info This graph shows disk usage on the machine.
_dev_hda1.label /boot
_dev_hda1.info /boot (ext3) -> /dev/hda1
_dev_hda1.warning 92
_dev_hda1.critical 98
[...]
.

If the plugin does run with munin-run but not through telnet, you probably have a PATH problem. Tip: Set env.PATH for the plugin in the plugin's environment file.

5.4 Verify Munin Services

you can verify that the Munin service is running with ps and netstat:

#ps aux | grep munin
apache    17835  0.2  0.8 141140  8480 ?     Ss   07:51   0:00 /usr/sbin/munin-node
root      18186  0.0  0.0 103220   844 pts/0 S+   07:52   0:00 grep munin

# netstat -anp | grep munin-node
tcp        0      0 0.0.0.0:4949       0.0.0.0:*      LISTEN      17835/munin-node

In the example above, you can see that the daemon process is running and that it is listening on tcp port 4949.