Home Server Munin

Index.

12-Feb-2012: Initial release.
25-Dec-2012: adjusted for Slackware.
08-Dec-2013: added OpenVPN configuration.

 

 

Introduction.

 

munin graph

munin graph

Munin is a monitoring tool that makes nice graphs of your system, but can also detect problems. Before you go ahead, you should read the Arch wiki about munin. For Slackware, you need to build munin packages yourself.

Munin uses a client-server model. On the Home Server a client (munin-node) and the server (munin) will be installed. On the virtual remote server only the clients (munin-node).

 

 

 

Installation.

Build the munin and munin-node packages. In the source directory that I use, both build scripts are in the same munin directory. Build the munin package first, and then the munin-node package. Install both on the home server, on all other server only munin-node must be installed.

 

Configure munin-node.

Next, configure the munin-node plugins. There are a lot and a some of them will not work until you create a small configuration file. To see the problems run the following command and look at the suggestions:

root@homsrv:~# /usr/sbin/munin-node-configure --suggest
Plugin                     | Used | Suggestions                            
------                     | ---- | -----------                            
acpi                       | no   | no [cannot read /proc/acpi/thermal_zone/*/te
mperature]
amavis                     | no   | no                                     
apache_accesses            | no   | no [LWP::UserAgent not found]          
apache_processes           | no   | no                                     
apache_volume              | no   | no [LWP::UserAgent not found]          
apc_envunit_               | no   | no [no units to monitor]               
bonding_err_               | no   | no [No /proc/net/bonding]              
courier_mta_mailqueue      | no   | no [spooldir not found]                
courier_mta_mailstats      | no   | no [could not find executable]         
courier_mta_mailvolume     | no   | no [could not find executable]         
cps_                       | no   | no                                     
cpu                        | no   | yes                                    
cpuspeed                   | no   | no [missing /sys/devices/system/cpu/cpu0/cpu
freq/stats/time_in_state]
cupsys_pages               | no   | yes                                    
df                         | no   | yes                                    
df_inode                   | no   | yes                                    
diskstats                  | no   | yes                                    
entropy                    | no   | yes                                    
exim_mailqueue             | no   | no [no exiqgrep]                       
exim_mailstats             | no   | no ['/usr/sbin/exim -bP log_file_path' retur
ned an error]
fail2ban                   | no   | no [/usr/bin/fail2ban-client not found]
forks                      | no   | yes                                    
fw_conntrack               | no   | no                                     
fw_forwarded_local         | no   | no [/proc/net/ip_conntrack missing or not re
adable]
fw_packets                 | no   | yes                                    
hddtemp_smartctl           | no   | no [smartctl not found]                

   removed a long list

root@homsrv:~#

 

A lot of plugins will never work because you don’t have use these programs. Others need to be fixed. If you look at fw_conntrack you see that it will not be available. This is because munin runs as user munin, even if you started this command as root. You have to write a small file /etc/munin/plugin-conf.d/fw_conntrack:

[fw_conntrack]
user root

[fw_forwarded_local]
user root

 

Now the plugins fw_conntrack and fw_forwarded_local will be run as root.Run /usr/sbin/munin-node-configure –suggest again and see that it now works:

fw_conntrack               | no   | yes                                    
fw_forwarded_local         | no   | yes                                    
fw_packets                 | no   | yes

 

In the example files you find all my changed configuration files. Now activate the suggested plugins, goto the plugins directory and see the symlinks of the active plugins:

root@homsrv: plugin-conf.d# /usr/sbin/munin-node-configure --shell | sh
# The following plugins caused errors:
# apache_processes:
#       Junk printed to stderr
# http_loadtime:
#       Junk printed to stderr
# mysql_:
#       Non-zero exit during autoconf (2)
# tomcat_:
#       Non-zero exit during autoconf (126)
root@homsrv: plugin-conf.d# cd ../plugins
root@homsrv: plugins# ls
cpu@                 if_err_eth1@  memory@               open_files@
cupsys_pages@        if_err_tap0@  munin_stats@          open_inodes@
df@                  if_eth0@      netstat@              postfix_mailqueue@
df_inode@            if_eth1@      nfs4_client@          postfix_mailvolume@
diskstats@           if_tap0@      nfs_client@           processes@
entropy@             interrupts@   nfsd@                 proc_pri@
forks@               iostat@       nfsd4@                swap@
fw_conntrack@        iostat_ios@   ntp_kernel_err@       threads@
fw_forwarded_local@  irqstats@     ntp_kernel_pll_freq@  uptime@
fw_packets@          load@         ntp_kernel_pll_off@   users@
if_err_eth0@         lpstat@       ntp_offset@           vmstat@
root@homsrv: plugins#

 

You should now remove plugins that are detected but are not needed like nfsd, nfs_client and nfs4_client because we only have the nfs4_server. Then, the bridged network interface br0 shows here as eth0 and tap0. Rename eth0 into br0 and remove tap0.

root@homsrv: plugins# mv if_eth0 if_br0
root@homsrv: plugins# rm if_tap0 nfs4_client nfs_client nfsd cupsys_pages lpstat
root@homsrv: plugins#

 

Configure /etc/munin/munin.conf:

htmldir /srv/http/munin
includedir /etc/munin/munin-conf.d
# a simple host tree
[homsrv.wpl.ym]
    address 127.0.0.1
    use_node_name yes

 

Munin on the home server is now configured. Start it and add munin-node to /etc/rc.conf.

root@homsrv: ~# mkdir /srv/http/munin
root@homsrv: ~# chown munin:munin /srv/http/munin
root@homsrv: ~# crontab /etc/munin/munin-cron-entry -u munin
root@homsrv: ~# cd /etc/postfix-in/
root@homsrv: postfix-in# vi aliases
  Add "munin: root" line
root@homsrv: postfix-in# make
newaliases -C `pwd`
root@homsrv: postfix-in# cd
root@homsrv: ~# /etc/rc.d/rc.munin-node start
Starting munin-node done.
root@homsrv: ~#

 

 

Configure web server.

Add a munin.wpl.ym entry to the bind nameserver. Then add the following section to /etc/nginx/conf/nginx.conf:

    # munin.wpl.ym
    server {
        listen          10.126.160.253:80;
        listen          [2001:1af8:fecf:7ea0::202]:80;
        server_name     munin.wpl.ym;
        error_log       logs/munin_error_log;
        access_log      logs/munin_access_log main;
        root            /srv/http/munin;
        index           index.html;
    }

 

After you have restarted the nginx web server, browse to http://munin.wpl.ym to see the munin results.

 

Configure gateway server.

Install munin-node, possibly using the already build package if the architecture is the same as the home server. Use the same procedure as on the Home Server to configure the plugins. Now you must also configure /etc/munin/munin-node.conf so that the Home Server is allowed to connect to the node, and that the node software only listens on the network card that is connected to the Home Server, eth0. Here are the relevant lines that must be changed:

allow ^10\.126\.161\.2$
host 10.126.161.1

 

Now you may start the node:

[root@gateway ~]# /etc/rc.d/rc.munin-node start
Starting munin-node done.
[root@gateway ~]#

 

When you go back to the Home Server, use the telnet command to test if you can connect to the remote node:

root@homsrv: ~# telnet 10.126.161.1 4949
Trying 10.126.161.1...
Connected to 10.126.161.1.
Escape character is '^]'.
# munin node at gateway.wpl.ym
QUIT
Connection closed by foreign host.
root@homsrv: ~#

 

Add the remote to the Home Server. The relevant section in /etc/munin/munin.conf looks like:

[homsrv.wpl.ym]
    address 127.0.0.1
    use_node_name yes

[gateway.wpl.ym]
    address 10.126.161.1

[nginx.wpl.ym]
    address 10.126.162.4

[wpl.ym;]
    node_order homsrv.wpl.ym gateway.wpl.ym nginx.wpl.ym

 

After you added the new node, you must restart munin-node on the Home Server.

 

 

OpenVPN on the home server.

On the home server you might have configured OpenVPN for tap (vridged) and tun (routed) clients. The default munin plugin doesn’t handle multiple configurations of OpenVPN. On this github page you can find scripts that does handle this situation. Below is the /etc/munin/plugin-conf.d/openvpn_usercount that I use:

[openvpn_usercount_tap]
user root
env.statusfile /etc/openvpn/server-tap/openvpn-status.log

[openvpn_usercount_tun]
user root
env.statusfile /etc/openvpn/server-tun/openvpn-status.log

 

 

Download.

See the download page for the script and configuration files.