Using pv-grub to run custom kernels on RHEL - enabling and troubleshooting

Make the final changes to enable pv-grub for your RHEL VPS, then check the results.


Changing more system settings

Just installing the kernel in the first article hasn't changed anything about how your RHEL VPS will boot. We'll need to make some other changes before pv-grub will boot with the new kernel.

Remember: Make sure you have a backup of any important data on your VPS before applying these changes. Backups are important when you're tinkering with system stuff.

Edit fstab

This is the change that will rig the system so it can't boot without pv-grub anymore, so you're warned.

Since the device used to load the filesystem (the virtual hard drive) is called "xvda" when pv-grub is loading the kernel, we have to change /etc/fstab. That's the file that tells the system how to access and mount the filesystem.

The device reference changes from "/dev/sda" to "/dev/xvda" (for both the root filesystem and swap). You can either make the change manually with a text editor or you can use sed:

sudo sed -i 's/sda/xvda/' /etc/fstab

That command replaces "sda" with "xvda" wherever it exists in the /etc/fstab file. If you need to revert that change later you can just run the sed command in reverse (putting "xvda" first and "sda" second).

To the SliceManager

Your VPS should be ready for pv-grub at this point. This is your last chance to back up important data and, if you're worried about your chances, make offerings to the deity or deities of your choice.

With that done, log into the SliceManager and go to the entry for your slice.

The web console

Before making the switch to pv-grub open the web console for your VPS using the "Console" option. That should give you a window where you can look at what your server is printing to the console. That includes the stuff that would scroll across the screen while the VPS boots, which is what we want to watch when the kernel change is applied.

When you load the console, if you don't see any text in the window try hitting enter a couple times. You should see a login prompt, which is what we want. You don't need to log into the VPS in the console since we're just monitoring output.

Change kernel to pv-grub

To enable pv-grub you'll need to contact support via the SliceManager and ask them to make the switch-over for your VPS.

When the change is applied, go back to the web console to see what your VPS is up to. You should see it shut down and start up again. If everything goes smoothly you'll see the grub menu and then eventually a login prompt. If you see an error instead skip down the "Troubleshooting" section.

Check the results

Once the VPS is running using pv-grub, you can ssh into it and check the kernel version it's running:

$ uname -a
Linux demo 2.6.18-194.17.1.el5xen #1 SMP Mon Sep 20 07:20:39 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux

The version should match the kernel you installed. From now on, to use a different kernel you'll just need to install it and edit the /boot/grub/menu.lst file appropriately.

To be extra sure that the fstab changes are all good, make sure your swap is using the new device too:

$ swapon -s
Filename                Type        Size    Used    Priority
/dev/xvda2                              partition   524280  0   -1

If nothing is listed when you run "swapon -s", recheck your /etc/fstab changes to make sure the designated swap device isn't still pointing to /dev/sda2.

Troubleshooting

If you get an error message in the web console instead of a login prompt (or just a blank screen with an error that flashes past really quickly before disappearing) then something has gone wrong. Pay attention to where in the process the error occurred since that can help narrow down the problem. If you missed it the first time, reboot the server so you can see it again.

If the VPS never got to the point where it displayed the grub menu that means there was a problem loading pv-grub to begin with. Hopefully that will never happen, but if it does contact support and they can help track down the problem.

If you see the grub menu before seeing the error, or get a message from grub that it couldn't display the menu, then the problem lies somewhere on your VPS filesystem. To get at those files now you'll have to either enable rescue mode for the VPS or restore a backed-up snapshot.

If you see an error message about a problem decompressing the kernel (probably mentioning bzip or LZMA), it means the kernel you tried to boot from uses a compression scheme that pv-grub doesn't support. Fixing that problem might require rebuilding the kernel to compress with gzip, or you may just want to copy the uncompressed "vmlinux" file from the kernel source directory (usually in /usr/src) to /boot and change menu.lst to use that kernel instead.

An error complaining that the root filesystem can't be found usually means either menu.lst doesn't point to the right files or that there is a mistake in the /etc/fstab file.

Rescue mode

When your VPS won't boot you can access its filesystem and make changes using "rescue mode". You can put your VPS into rescue mode via the SliceManager. Rescue mode is described in more detail in this article, but we cover the basics here.

Putting the VPS into rescue mode causes it to boot into a custom "rescue image" instead of booting with your regular configuration. If a problem is keeping your VPS from booting rescue mode is a Good Thing.

To make changes to your server in rescue mode, once you've logged in (via ssh or in the web console) you can run:

mount /dev/sda1 /mnt

Your whole filesystem will be available for changes under that /mnt directory. You can get to /boot using "/mnt/boot", /etc would be in "/mnt/etc", and so forth. Once you're done making changes you can exit rescue mode and the VPS will try to boot normally.

Double-check the changes

The first thing to check when you're looking at the files on your VPS via rescue mode is the /boot/grub/menu.lst file. Check that the kernel and initrd references point to the right files in the right locations. Typos are a fact of life.

Next look at /etc/fstab to make sure the name of the device for the root filesystem was changed from "sda1" to "xvda1".

Also check /etc/inittab for the "xvc0" TTY line you added at the end, making sure it matches the instructions.

If everything seems to be in order there may be a problem with the kernel you installed. You should roll back the change in /etc/fstab (changing the device back to /dev/sda1) then ask support to disable pv-grub. That will let you reboot normally and check the kernel that was installed. Make sure the kernel in question has Xen compatibility enabled.

In one place: Enabling pv-grub

If you're creating a new VPS or modifying several servers it's handy to just have all the commands in one place instead of digging through the full tutorial. Here's the short version of the steps you can take on an RHEL VPS to prepare it for enabling pv-grub with a stock Red Hat kernel:

sudo yum upgrade
sudo cp /etc/modprobe.conf /etc/modprobe.conf.bak
sudo echo "alias eth0 xennet" > /etc/modprobe.conf
sudo echo "alias eth1 xennet" >> /etc/modprobe.conf
sudo echo "alias scsi_hostadapter xenblk" >> /etc/modprobe.conf
sudo yum install kernel-xen
sudo mkdir -p /boot/grub
sudo echo "8:2345:respawn:/sbin/mingetty xvc0" >> /etc/inittab
sudo grep xvc0 /etc/securetty || echo "xvc0" >> /etc/securetty
sudo sed -i 's/sda/xvda/' /etc/fstab

After that look at the kernel and initrd names in /boot then edit /boot/grub/menu.lst accordingly.

Ask support to enable pv-grub and the VPS will reboot.

In one place: Disabling pv-grub

If you want to disable pv-grub you'll need to revert the change that was made to /etc/fstab:

sudo sed -i 's/xvda/sda/' /etc/fstab

It's likely that you would need to make the change while in rescue mode, so if you're in rescue mode and have your VPS filesystem mounted to "/mnt" the sed command would look like:

sed -i 's/xvda/sda/' /mnt/etc/fstab

The other changes and the installed kernel can stay in place without interfering with normal operation.

If you did disable the old TTY, remember to uncomment the line in /etc/inittab. The proper TTY device entry for a non-pv-grub kernel will look like:

1:2345:respawn:/sbin/mingetty tty1

And remember to move your old modprobe.conf file back into place:

sudo mv /etc/modprobe.conf.bak /etc/modprobe.conf

Once you've made the necessary adjustments on your VPS you can make the switch back to one of our stock kernels.

Summary

Once you have pv-grub enabled and are running the kernel you've installed you should be okay to install other kernels when you need to. Just remember to point to them in menu.lst.

If you want to compile your own kernel and run it first refer to this article about obtaining our kernel source code. Even if you aren't going to use that source code it can be useful to refer to the article to see how we configure our kernel builds.

  • -- Jered
Want to comment?


(not made public)

(optional)

(use plain text or Markdown syntax)