Following up on the article about installing a munin master slice, if you want to monitor additional slices you'll need to install a munin node service on each.
What's a munin node?
A munin node is a data collector for munin that runs on a slice. A munin master will query the node periodically for its data, and then the master will parse that data and graph it. This article is intended for installation of just a munin node on a different slice from the munin master. If you do not yet have a munin master set up, please follow the instructions in the previous article in this series.
Installing the node
To install just a munin node on a slice, run the command:
sudo yum install munin-node
Configuring the node
Now that the node is installed you'll need to make sure the munin master will be able to communicate with it. The first things you'll need to know are the backend IP addresses of the node slice and of the munin master slice. You can find IP addresses in the SliceManager, either in the list of your slices or on the manager screen for a specific slice.
The second IP address listed for a slice, the one that starts with "10.", is the backend IP address. If no backend address is listed, contact Slicehost support and they can add one to the slice.
Make note of the backend IP addresses of both the node slice and the munin master slice. You'll use the backend IP address for munin's communication because that address is not accessible from the rest of the Internet, so it's a little more secure. Even better, traffic on the backend IP addresses does not count toward your slices' bandwidth use for the month (free is good).
Open the /etc/munin/munin-node.conf file and look for this entry:
# A list of addresses that are allowed to connect. This must be a # regular expression, due to brain damage in Net::Server, which # doesn't understand CIDR-style network notation. You may repeat # the allow line as many times as you'd like allow ^127\.0\.0\.1$
That last line is the one you'll want to change. The address has to be in the form of a regular expression because of a perl module used by munin. Fortunately you don't need to know what a regular expression is, or how to write one. Just know that the default expression, "^127.0.0.1$", refers to the IP address "127.0.0.1". You want to change that address to reflect the backend IP address of the munin master slice. If your munin master is at "10.33.22.11", for example, you would change that entry to look like:
# A list of addresses that are allowed to connect. This must be a # regular expression, due to brain damage in Net::Server, which # doesn't understand CIDR-style network notation. You may repeat # the allow line as many times as you'd like allow ^10\.33\.22\.11$
Next you'll want to move further down in the munin-node.conf file, looking for this entry:
# Which address to bind to; host *
Replace the "*" with the backend IP address of the node slice, as in:
# Which address to bind to; host 10.2.3.4
Edit the node's iptables
The example iptables rules used in Slicehost setup articles close the port the munin master would use to connect to a munin node, as do the default Fedora iptables rules.
To add the iptables rules you would need to add to allow the munin master to connect to your new node, run the following commands on the node slice:
sudo /sbin/iptables -I INPUT -p tcp --dport 4949 -m state --state NEW,ESTABLISHED -j ACCEPT sudo /sbin/iptables -I OUTPUT -p tcp --dport 4949 -m state --state ESTABLISHED -j ACCEPT
To save those rules so they'll take effect next time the slice is rebooted, run:
sudo /sbin/service iptables save
Start the node
The munin node wasn't started by the installer, so now that we're done configuring it let's start that up:
sudo /etc/init.d/munin-node start
To make sure the munin-node service starts when your slice reboots, you'll also want to make the service active:
sudo /sbin/chkconfig munin-node on
That will complete the setup of the new node slice. Next you'll need to configure the master to gather data from the new node.
Adding the node to the master
Connect to the munin master server and edit the /etc/munin/munin.conf file to add the new node to your munin site. In munin.conf look for the "simple host tree" section where you would have set up the master's local node previously.
# a simple host tree [slicename] address 127.0.0.1 use_node_name yes
The easiest way to add the new node to the reporting is to duplicate the host entry you already have, then edit the name and address to match the new node. For example:
# a simple host tree [slicename] address 127.0.0.1 use_node_name yes [otherslicename] address 10.2.3.4 use_node_name yes
Remember not to use a space in the host entry name. Munin's graphing scripts don't seem to like it.
Edit the master's iptables
You will want to add iptables rules on the munin master to be sure it can get out to the new node by running the following commands:
sudo /sbin/iptables -I OUTPUT -p tcp --dport 4949 -m state --state NEW,ESTABLISHED -j ACCEPT sudo /sbin/iptables -I INPUT -p tcp --dport 4949 -m state --state ESTABLISHED -j ACCEPT
And to save those rules so they'll take effect next time the slice is rebooted, run:
sudo /sbin/service iptables save
See the results
Remember that the munin master runs every five minutes, so it may take that long for the new node to appear on your munin reports. After an appropriate wait visit your main munin page to see if you have a new node listed.
Hopefully you'll see the new node under the entry for your master node, similar to this screenshot:
If you don't see an entry for the new node at all, check the configuration of the munin master (munin.conf on the master slice) to make sure the entry was properly added to the host tree.
If you see an entry for the new node but no services are listed in brackets, it means the master knows about the new node but hasn't been able to connect to it. Make sure the node service is running on the node slice, and if it's running, go back through this guide to double-check the IP addresses in the configurations for the master and node. You might also make sure iptables isn't blocking the master's connection to the node.
If you see the new node and a list of report categories next to it, then you're set. Just remember that it will take a while for munin to collect enough data to make any noticeable graphs for the new node.
Adding a node to a munin master boils down to configuring and starting a munin-node service on the new node, then telling the munin master about it. If you have more than one node to add, go through this guide for each additional node to get them onto the munin reports too. With that done you'll have a single page you can go to that will link to reports for all your slices. Convenient!
The next article in this series will discuss additional service plug-ins for munin, how to tell munin nodes to use the ones you need, and how to install new plug-ins on nodes.
- -- Jered