IP failover - Slice setup and installing Heartbeat

Following from the previous article, we can now look at setting up our Slices and installing Heartbeat.

This is a very easy process that allows for quick access to a High Availability Slice setup.


Slices

First of all, you will need two Slices. You can add more Slices from the Slicemanager - log in and then click 'Add a Slice'. We recommend using the same size Slice with the same distro.

Requesting a failover IP

Do note that you do not need a new IP address allocated to your Slice - not only are they $2 a month but there is no need as your existing IP can be used for these purposes.

All you need to do is open a Support Request from the Slicemanager (under the Help tab).

In your request, please specifically define which Slice IP you want to share and which Slice you want to share it with.

In this, and future, articles, I am going to use the Slice names of 'Master' and 'Slave'.

The process is quickly completed.

Unless specifically mentioned, all commands will need to be run on BOTH Slices.

Let's go ahead and update the Slices.

Update the Slices

I am assuming you are using Ubuntu Hardy for your Slices. Please note that these articles may not work on another distribution. However, the principle is the same and the configuration files should be the same.

Log into the Slice and issue this command:

sudo aptitude update

Once you have done that, have a check to see if anything needs upgrading on the Slice:

sudo aptitude safe-upgrade

Done.

Installing Heartbeat

Note that we will be using version 2.1.3 for this article. Version 1 has very different configuration files.

Now we are ready to install Heartbeat (again, note this is to be done on BOTH Slices):

sudo aptitude install heartbeat

Well, that's it really.

Configuration files.

There are three files that need to be created and configured for this to work.

They are located in the /etc/heartbeat/ folder.

authkeys

The first file is called 'authkeys'. Perhaps discernible from the file name, but this will hold our authorisation keys so only our heartbeat installs will talk to each other.

This file will be exactly the same on BOTH Slices:

sudo nano /etc/heartbeat/authkeys

The contents are as simple as this:

auth 1
1 sha1 YourSecretPassPhrase

The first line defines the auth number (you can have multiple failover IPs that need more than one authkey).

The second simply defines your passphrase and uses sha1 to encrypt it.

Again, the file need to be the same on both Slices.

You must change the permissions of this file - it must not be 'world' readable (Heartbeat will fail if it is).

On both Slices change the permissions:

sudo chmod 600 /etc/heartbeat/authkeys

haresources

The second file that needs changing is the 'haresources' file. As with the authkeys, this needs to be exactly the same on BOTH Slices:

sudo nano /etc/heartbeat/haresources

The contents are very simple:

master 123.45.67.890/24

The name 'master' is the hostname of the MASTER Slice and the IP address (123.45.67.890) is the IP address of the MASTER Slice.

Note the '/24' after the MASTER Slice IP address.

To drive this home, this file needs to be the same on BOTH Slices.

MASTER Slice ha.cf

The third file that needs creating is named 'ha.cf'.

This file is different on each of the Slices.

Lets deal with the contents of this file for the MASTER Slice:

sudo nano /etc/heartbeat/ha.cf

The contents would be as follows:

logfacility daemon
keepalive 2
deadtime 15
warntime 5
initdead 120
udpport 694
ucast eth1 172.0.0.0 # The Private IP address of your SLAVE slice.
auto_failback on
node master # The hostname of your MASTER Slice.
node slave # The hostname of your SLAVE slice.
respawn hacluster /usr/lib/heartbeat/ipfail
use_logd yes

Note the three lines that have comments on them:

ucast eth1 172.0.0.0 # The Private IP address of your SLAVE slice.

As this is the ha.cf on the Master Slice it references the private IP of the Slave Slice. In other words, it is telling the Master Slice where the Slave Slice is.

The other two lines to note are these:

node master # The hostname of your MASTER Slice.
node slave # The hostname of your SLAVE slice.

These must be set to the exact hostname of each of the two Slices.

If you are not sure of the hostnames, you can find out by entering this command:

hostname

on my demoslice.com Slice the answer would be:

demoslice.com

Restart the service:

sudo /etc/init.d/heartbeat restart

Ok. Done.

SLAVE Slice ha.cf

The third file (the ha.cf file) needs to be created on the Slave Slice. It is similar but has some differences that are worth noting.

Let's open the file on the Slave Slice:

sudo nano /etc/heartbeat/ha.cf

The contents will need to be:

logfacility daemon
keepalive 2
deadtime 15
warntime 5
initdead 120
udpport 694
ucast eth1 172.0.0.1 # The Private IP address of your MASTER slice.
auto_failback on
node master  # The hostname of your MASTER Slice.
node slave # The hostname of your SLAVE Slice.
respawn hacluster /usr/lib/heartbeat/ipfail
use_logd yes

Hmmm, that looks familiar...

In fact, they are the same EXCEPT for this line:

ucast eth1 172.0.0.1 # The Private IP address of your MASTER slice.

Note it is the opposite to the Master Slice configuration. This time, as we are working on the Slave Slice, we need to reference the Master Slice's private IP address.

Once done, save the file and restart Heartbeat on the Slave Slice:

sudo /etc/init.d/heartbeat restart

Done.

Is that it?

Yup. Sure, we spent a bit of time in this article going through the settings in the configuration files, but it is as simple as that.

Let's have a little test of the system.

Test

Start off with both Slices running and ping the main IP (the IP we have set to be the failover) on the Master Slice:

ping -c2 123.45.67.890

The '-c2' option simply tells ping to 'ping' twice. You will get a standard reply.

For example, if I were to ping the demo Slice I would get this response (remember I am in the UK):

ping -c2 208.75.84.20
PING 208.75.84.20 (208.75.84.20): 56 data bytes
64 bytes from 208.75.84.20: icmp_seq=0 ttl=49 time=122.765 ms
64 bytes from 208.75.84.20: icmp_seq=1 ttl=49 time=122.863 ms

--- 208.75.84.20 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 122.765/122.814/122.863/0.049 ms

So my Slice is up and responding to pings.

Now shutdown the Master Slice:

sudo shutdown -h now

Without the failover IP, there would be no response from the ping request as the Slice is down.

However, assuming you have installed Heartbeat you will notice that the IP is still responding to pings (because the Slave Slice noticed the Master was down and simply took over the role of serving that IP address).

A simple, yet effective demonstration of the IP failover in action.

Summary

Installing Heartbeat on both Master and Slave Slices is very easy. Configuration files are also very easy to create. Just be careful to ensure you use the correct IPs and hostnames as shown in the examples above.

The facility to offer HA sites using failover IPs is a great boost to busy and vital websites.

PickledOnion

Article Comments:

James commented Tue Oct 28 16:58:29 UTC 2008:

Very cool tutorial. I assume that heartbeat can have more than one slice as slave. Also is there a way to easily clone slices on the slicehost network.

Adam Kocoloski commented Tue Oct 28 18:39:08 UTC 2008:

Another excellent tutorial. It looks like this configuration doesn't actually check that the Master can reach the internet, since the heartbeat is just a ucast between the private IPs. Is that right?

The really paranoid among us might want to do a heartbeat on both the public and private interfaces. I think this would require an extra IP address, though, so that the Master and Slave both have dedicated paths to the internet regardless of which one is currently serving requests directed at the virtual IP.

Carl commented Wed Oct 29 02:53:41 UTC 2008:

James,

If you enable backups you can select one of your backups to the the "distro" for the new slice. With backup enabled, you get a weekly backup, a daily backup, and one wildcard backup. So you could run the wildcard backup right before you create the new slice and then use that backup for the build.

Pedro commented Mon Nov 24 22:57:39 UTC 2008:

What about the Database?. If it is in a third slice that's the point of failure and if we set a DB on each slice, how we synchronize them?.

Joe commented Tue Dec 09 19:43:40 UTC 2008:

Pedro,

I would think that you'd have the same setup with your DB servers along with replication to keep them in synch.

if ur front end master fails, the slave takes over, if the db master fails, the synched slave takes over.

Can anyone confirm this? (looking to do something similar)

Firefly commented Wed Jan 14 04:54:08 UTC 2009:

Pedro/Joe,

Google mysql mmm. MMM (MySQL Master-Master Replication Manager. I have been trying it out for past few weeks. Looks promising way to create highly available cluster.

Kevin commented Mon Jan 19 21:31:58 UTC 2009:

Is it possible to use one slice as a failover for two different slices? I know the IP sharing isn't a problem but can heartbeat be setup to do this?

Example would be something like: Server1 192.168.0.1 Server2 192.168.0.2 HeartbeatServer 192.168.0.99

You would want HeartbeatServer to take over all services for EITHER server1 or server2 in the event one went down. Possible?

Joe commented Thu Feb 12 20:33:45 UTC 2009:

Is there anyway to have the slave notify you when it has taken over for the master? I guess you could run something separately, but it would be nice if there was a config to automatically email on failover.

Matt commented Wed Oct 21 20:31:18 UTC 2009:

"Something to keep in mind that I ran into when I was following this tutorial is iptables. If your running iptables (which you are if you followed the other tutorials on setting up your slices) you will need to allow heartbeat to talk to the other slice over the udpport that you assigned in the ha.cf (694). If you don't do this your slave will think your master died and take over. I used this on both... sudo iptables -I INPUT 2 -p udp --dport 694 -j ACCEPT"

Donald commented Sat Nov 14 20:43:46 UTC 2009:

Hi, could we get a article like this with using ldirectord for load balancing?

Steven commented Thu Dec 31 05:24:53 UTC 2009:

I have the same question as Joe, is it possible to have the slave email on failover and rollback?

cc commented Mon Mar 29 01:46:44 UTC 2010:

The Ping works when the Master is down, but the Website and Email server stay down. What am I missing?

Tyler Bye commented Fri Jul 30 15:16:20 UTC 2010:

A few folks have asked about receiving email notifications on takeover (IP is lost) and migration (IP is given back).

I changed my haresources entry, on both my slices to:

myhost 123.456.123.456/24 MailTo::operations@mydomain.com::Heartbeat

After the update to haresources I bounced the heartbeat daemon on each box. I then ran shutdown -h now on the box with the failover IP to trigger a failover. I received emails from each box telling me that a takeover was in progress.

After your slice shutdowns to halt, you need to hard reboot it in your slicehost management panel. As the slice reboots and comes up, I receive 2 additional emails, one from each slice, notifying me that the resource was being migrated back.

A little bit of background on our setup: 2 base slices both running Ubuntu 10.04 (64-bit). Aside from the change to the haresources mentioned in my comment, I followed this tutorial to the letter.

Mijastra commented Mon Apr 22 20:24:03 UTC 2013:

Hello,

Little help, how to add IP route with a virtual IP address, when you have a master server failure.

example: Master 192.168.1.11 Slave 192.168.1.12 Virtual 192.168.1.10

The idea is that each packet sent to the virtual IP address back to the virtual source IP address. I need a solution for ip pbx, each registration to the virtual IP address goes wrong, when you need to return the package is returned by real static address (in this case 192,168,1,11) in which case the SIP client next message sends the actual static IP address 192.168.1.11, not to 192.168.1.10. In this case ip pbx server announces SIP client is not available. This was resolved by adding the IP route (ip route add 192.168.1.0/24 dev eth0 src 192.168.1.10) and things like this are great. The problem is when you have a failure, you need to add this route to work without any problem.

Any idea?

rozliczenie pit przez internet commented Fri Jul 19 20:58:44 UTC 2013:

Great blog you have here but I was curious about if you knew of any user discussion forums that cover the same topics talked about here? I'd really like to be a part of group where I can get feed-back from other knowledgeable individuals that share the same interest. If you have any recommendations, please let me know. Appreciate it!

my web site: rozliczenie pit przez internet

formularze pit 36 commented Wed Jul 31 11:19:30 UTC 2013:

Feel free to surf to my homepage :: [formularze pit 36](http: //mangk.us/12y "formularze pit 36")

formularze pit 36 commented Wed Jul 31 11:19:32 UTC 2013:

Feel free to surf to my homepage :: [formularze pit 36](http: //mangk.us/12y "formularze pit 36")

formularze pit 36 commented Wed Jul 31 11:21:16 UTC 2013:

Feel free to surf to my homepage :: [formularze pit 36](http: //mangk.us/12y "formularze pit 36")

podatek pit 38 commented Wed Jul 31 18:09:34 UTC 2013:

Check out my weblog: [podatek pit 38](http://www. dd4u.ch/s/1cx "podatek pit 38")

podatek pit 38 commented Wed Jul 31 18:10:40 UTC 2013:

Check out my weblog: [podatek pit 38](http://www. dd4u.ch/s/1cx "podatek pit 38")

http://shtn.me/8aj commented Mon Aug 05 03:18:26 UTC 2013:

Also visit my blog - anchor ([http://shtn.me/8aj](http: //shtn.me/8aj "http://shtn.me/8aj"))

http://www.pepbrainin.com/ commented Wed Aug 07 06:29:01 UTC 2013:

Feel free to visit my website; pit online 2014 (http://www.pepbrainin.com/)

pit 28 druk commented Wed Aug 07 19:23:37 UTC 2013:

Here is my webpage: pit 28 druk

pit 28 druk commented Wed Aug 07 19:25:04 UTC 2013:

Here is my webpage: pit 28 druk

Want to comment?


(not made public)

(optional)

(use plain text or Markdown syntax)