Enabling munin node plug-ins on CentOS

Munin uses plug-ins to determine what data is gathered and reported. It includes several plug-ins for the types of data most people would be interested in, but not all of those plug-ins are enabled on a fresh installation.


What are plug-ins?

When a munin node gathers data about a slice so it can be graphed, the node reads instructions from files called "plug-ins" to determine what information to collect. Several plug-ins are installed with munin but not all of them are used by default. Fortunately a munin node installation includes a command that lets you see what plug-ins are active and can help you decide which others to enable.

The munin-node-configure command

The "munin-node-configure" command can list the status of installed munin plug-ins, suggest which to enable, and provide a shell script that will automate enabling the appropriate plug-ins. Using this command isn't the only way to perform these tasks, but it's convenient and easy to use. Easy is good.

Listing installed plug-ins

To see which plug-ins are installed on a munin node, log into it and run the "munin-node-configure" command by itself:

$ sudo /usr/sbin/munin-node-configure
Plugin                     | Used | Extra information                      
------                     | ---- | -----------------                      
acpi                       | no   |                                        
apache_accesses            | no   |                                        
apache_processes           | no   |                                        
apache_volume              | no   |                                                                              
courier_mta_mailqueue      | no   |                                        
courier_mta_mailstats      | no   |                                        
courier_mta_mailvolume     | no   |                                        
cps_                       | no   |                                        
cpu                        | yes  |                                        
cupsys_pages               | no   |                                        
df                         | yes  |                                        
df_abs                     | no   |                                        
df_inode                   | yes  |                                        
entropy                    | yes  |                                        
exim_mailqueue             | yes  |                                        
exim_mailstats             | yes  |                                        
forks                      | yes  |                                        
fw_conntrack               | no   |                                        
...

The output is longer than that, but that's generally what you'll see — the plug-ins with "yes" next to them are enabled, the ones with "no" next to them are installed but disabled.

Deciding what to enable

To help you decide whether you want to enable any other munin plug-ins, run munin-node-configure with the "--suggest" option:

$ sudo /usr/sbin/munin-node-configure --suggest
Plugin                     | Used | Suggestions                            
------                     | ---- | -----------                            
acpi                       | no   | no [cannot read /proc/acpi/thermal_zone/*/temperature]
amavis                     | no   | no                                     
apache_accesses            | no   | no [ExtendedStatus option for apache mod_status is missing on port 80]
apache_processes           | no   | no [ExtendedStatus option for apache mod_status is missing on port 80]
apache_volume              | no   | no [ExtendedStatus option for apache mod_status is missing on port 80]
apc_envunit_               | no   | no [no units to monitor]               
bonding_err_               | no   | no [No /proc/net/bonding]              
courier_mta_mailqueue      | no   | no [spooldir not found]                
courier_mta_mailstats      | no   | no [could not find executable]         
courier_mta_mailvolume     | no   | no [could not find executable]         
cps_                       | no   | no                                     
cpu                        | yes  | yes                                    
cpuspeed                   | no   | no [missing /sys/devices/system/cpu/cpu0/cpufreq/stats/time_in_state]
cupsys_pages               | no   | no [could not find logdir]             
df                         | yes  | yes                                    
df_inode                   | yes  | yes                                    
diskstats                  | yes  | yes                                    
entropy                    | yes  | yes                                    
exim_mailqueue             | no   | no [no exiqgrep]                       
exim_mailstats             | no   | no ['/usr/sbin/exim -bP log_file_path' returned an error]
fail2ban                   | no   | no [/usr/bin/fail2ban-client not found]
...

If your output includes any error messages at the end or complaints about "junk output", don't worry. That just means a plug-in was queried that didn't give a proper response, it doesn't affect munin's operation.

Note that the output with "--suggest" is usually shorter than the complete list of installed plug-ins. When you ask munin-node-configure for its suggestions, it lists plug-ins that are written to respond to that query and also omits many plug-ins that are already installed.

Each plug-in name is followed by a "yes" or "no" indicating whether or not it's currently enabled. The last column might contain a suggestion regarding a plug-in's status. A "yes" suggestion indicates that a plug-in is currently supported by your system and can be enabled without making changes (like the postfix plug-ins in the example above). Other entries in the "Suggestions" column can offer details about issues that need to be corrected before enabling a given plug-in.

Example in-depth: Apache plug-ins

To illustrate an approach to resolving suggestions for a plug-in, let's look at the apache family of plug-ins. In the example above, the suggestion for the apache monitor includes:

[ExtendedStatus option for apache mod_status is missing on port 80]

This error message might require a bit of research to interpret, but to save you time: The problem is that munin gets information about apache's activity by querying apache's status page, and the information it wants is only on that page if the "ExtendedStatus" setting is "on" in apache's configuration.

If you want to enable munin's apache monitoring, don't fret. You probably just need to add "ExtendedStatus on" to your apache server configuration and restart apache. If that doesn't work, or if you want more details, you'll need to read up on apache's "mod_status" module. We have an article detailing how to enable mod_status and how to read its output here.

Once we've ensured that apache's mod_status module is enabled and ExtendedStatus is turned on, we can run munin-node-configure yet again:

$ sudo /usr/sbin/munin-node-configure --suggest
Plugin                     | Used | Suggestions                            
------                     | ---- | -----------                            
...        
apache_accesses            | no   | yes                                    
apache_processes           | no   | yes                                    
apache_volume              | no   | yes 
...

That's more like it.

Enabling suggested plug-ins

So you've gone to a mess of trouble getting to the point where munin has no reservations about running the plug-ins you want. Now to actually enable them.

We could go into file locations and making symlinks and such, but let's opt for some automation instead. Take a look at the output from running munin-node-configure with the "--shell" option:

$ sudo /usr/sbin/munin-node-configure --shell
ln -s /usr/share/munin/plugins/apache_accesses /etc/munin/plugins/apache_accesses
ln -s /usr/share/munin/plugins/apache_processes /etc/munin/plugins/apache_processes
ln -s /usr/share/munin/plugins/apache_volume /etc/munin/plugins/apache_volume
ln -s /usr/share/munin/plugins/postfix_mailqueue /etc/munin/plugins/postfix_mailqueue
ln -s /usr/share/munin/plugins/postfix_mailvolume /etc/munin/plugins/postfix_mailvolume

If that looks like a bunch of shell commands for making symlinks, that's because it is a bunch of shell commands for making symlinks. Specifically, they're symlinks that tell munin to enable the plug-ins they're linking to. In the above example, the plug-ins for monitoring apache and postfix would be enabled.

Before you copy and paste, though, let's work a little shell magic to turn the above into a one-line command. Run this command on your munin node:

sudo /usr/sbin/munin-node-configure --shell | sudo sh

That's just like the command we ran to get a list of shell commands, but with a little bit tacked onto the end. The "|" character is known as a "pipe" in the unix shell. The "sh" after it (also run with sudo) is the command to run a basic Bourne shell. Combine the two, and in plain English it means, "take the output of the command before the pipe, and run it in a shell." Or, a plainer English summary: "Make those symlinks for me."

Indeed, once you run munin-node-configure with the --shell option and pipe it to sh, you can see how the suggestions change:

$ sudo /usr/sbin/munin-node-configure --suggest
Plugin                     | Used | Suggestions                            
------                     | ---- | -----------                            
acpi                       | no   | no [cannot read /proc/acpi/thermal_zone/*/temperature]
amavis                     | no   | no                                     
apache_accesses            | yes  | yes                                    
apache_processes           | yes  | yes                                    
apache_volume              | yes  | yes                                    
apc_envunit_               | no   | no [no units to monitor]               
bonding_err_               | no   | no [No /proc/net/bonding]              
courier_mta_mailqueue      | no   | no [spooldir not found]                
courier_mta_mailstats      | no   | no [could not find executable]         
courier_mta_mailvolume     | no   | no [could not find executable]         
cps_                       | no   | no                                     
cpu                        | yes  | yes                                    
cpuspeed                   | no   | no [missing /sys/devices/system/cpu/cpu0/cpufreq/stats/time_in_state]
cupsys_pages               | no   | no [could not find logdir]             
df                         | yes  | yes                                    
df_inode                   | yes  | yes                                    
diskstats                  | yes  | yes                                    
entropy                    | yes  | yes                                    
exim_mailqueue             | no   | no [no exiqgrep]                       
exim_mailstats             | no   | no ['/usr/sbin/exim -bP log_file_path' returned an error]
fail2ban                   | no   | no [/usr/bin/fail2ban-client not found]
...

Now that the apache has been enabled, that status is reflected in the suggestions list (and for postfix later in the list, not shown above). If you were to wait a few minutes to let the munin master run and query this node, you would see graphs for apache and postfix added to the node's reports.

Plug-in locations

In case you have trouble with the munin-node-configure command, or if you don't trust its automation, or you simply want to look more closely at the plug-ins (some do contain useful information in their comments), you can find the plug-in files here:

/usr/share/munin/plugins

Enabling a plug-in is just a matter of making a symlink from the original plug-in file to the munin node's active plug-in directory, which is here:

/etc/munin/plugins

Similarly, you can disable a plug-in by removing its symlink from the node's active plug-in directory.

Summary

The munin-node-configure command is a pretty handy way to check on your munin plug-ins and figure out what needs to be done in order to enable more. You should now be able to look over the list of plug-ins and decide which ones you'd like to see enabled on your munin reports. Bear in mind, however, that each munin node has to be configured individually. If you have more than one node, you'll have to enable a new plug-in on all nodes you want using it (not just the one running the munin master).

Fortunately once you've been through all the wordy details above, you'll know what the commands are and what to do with their output. They boil down to logging onto each node and running:

sudo /usr/sbin/munin-node-configure --suggest

Then fixing the issues for any plug-ins you want enabled, and then running:

sudo /usr/sbin/munin-node-configure --shell | sudo sh

And waiting for munin's reports to update. Piece of cake.

The next article in this series will discuss some advanced topics for munin, including installing plug-ins from munin's online repository and customizing the organization of the host tree on your report page.

  • -- Jered

Article Comments:

curtis commented Wed May 05 13:38:32 UTC 2010:

Great Articles. I successfully enabled extended status in Apache and can now access server-status.

However, munin --suggest comes back with the following note for Apache plugins: "Port 80 Forbidden"

Also, for MySQL --suggest comes back with"Missing dependency Cache::Cache"

How can I solve this?

How can I solve this?

Jered commented Wed May 05 17:15:37 UTC 2010:

For the response you got for apache, your iptables rules may be blocking port 80. That's the most likely reason I can think of for it to report port 80 as "forbidden". If you haven't already, try using lynx to access the server-status page from the slice, and see if it can access the regular index page as well.

On the MySQL response, it wants a perl library (that's usually what the "::" in the middle indicates, since perl libraries will use that in their designation). You can install that one with "sudo yum install perl-cache-cache". Then run the --suggest again to see if that resolved it, or if there's something else it thinks it's missing.

Nima commented Fri Aug 27 08:55:02 UTC 2010:

In order to enable MySQL plugin you'll need to add following lines to the /etc/munin/plugin-conf.d/munin-node [mysql*] env.mysqlopts -u yourusername -pyourpass

Brian commented Mon Nov 29 02:10:01 UTC 2010:

I had the same MySQL error and resolved it on an Ubuntu 10.04 LTS server with:

aptitude install libcache-cache-perl libipc-sharelite-perl

I also get the same Apache forbidden error, but haven't diagnosed yet.

electronico_nc commented Tue Nov 30 08:26:14 UTC 2010:

@Brian : Thanks for the missing Perl dependencies! For Apache forbidden error, you need to configure /etc/munin/apache.conf on the machine that displays Munin, add (accord to your network configuration): Allow from localhost 192.168.1.253/24 ::1 Hope this helps.

Brian commented Fri Dec 03 16:56:27 UTC 2010:

For me the Apache forbidden error was a bonehead mistake. I had a .htaccess 301 redirect in /var/www that redirected / (i.e., everything) to a remote site so /server-status was being redirected (unbeknownst to me) to another server where I was in fact Forbidden from remotely accessing server-status (if it is even enabled on that remote server, I don't know). This problem was hard to detect because the remote server is running the same version of Ubuntu as the local server, so the Forbidden error looked identical to me.

Jeremy Clarke commented Wed Feb 16 23:44:32 UTC 2011:

Thanks for another great tutorial that applies to me even though I'm on a different host! My impression of Slicehost grows daily.

This post has developed some really serious formatting errors somehow. It seems several headings in the middle are gone and paragraphs run together out of control.

Also while you're at it, now might be a good time to go through these articles and make sure the all link to the next/prev posts in the series, as it's kind of annoying to have to track them down using tags.

Thanks!

Want to comment?


(not made public)

(optional)

(use plain text or Markdown syntax)