Using pv-grub to run custom kernels on Gentoo - enabling and troubleshooting

Make the final changes to enable pv-grub for your Gentoo Linux VPS, then check the results.


Final adjustments

Just installing the kernel in the first article hasn't changed anything about how your Gentoo Linux VPS will boot. We'll need to make some other changes before pv-grub will boot with the new kernel.

Remember: Make sure you have a backup of any important data on your VPS before applying these changes. Backups are important when you're tinkering with system stuff.

A look at /boot

To tell pv-grub where to look for the new kernel you'll need to know what it's called and where it is. The where part is straightforward — it's in /boot. Check a directory listing of /boot and you'll see something like:

$ ls /boot
System.map-genkernel-x86_64-2.6.34-gentoo-r12-simple
boot
initramfs-genkernel-x86_64-2.6.34-gentoo-r12-simple
kernel-genkernel-x86_64-2.6.34-gentoo-r12-simple

The two files you need to take note of are the one that starts with "initramfs" and the one that starts with "kernel". The former is the initial ramdisk (and may not always be present, or may start with "initrd") and the latter is the actual kernel (which might start with "vmlinuz" instead of "kernel"). If you see more than one of each it means you have more than one kernel version installed. Pick the files that match the version of the kernel you are trying to install (probably the ones with the newest version number).

To tell pv-grub to use the new kernel we'll be editing a file in the "grub" subdirectory of /boot.

The grub directory

To tell pv-grub to use the new kernel we'll be editing a file in the "grub" subdirectory of /boot.

The grub package is a bootloader, which is to say, it lets you pick how you want your system to start up from a menu at boot time. If you've set up a linux installation on a workstation you may have used it before. For your VPS editing the grub menu works similarly to editing it on a workstation.

The menu configuration file needs to be in /boot/grub, which may already exist. If it doesn't, create it:

sudo mkdir -p /boot/grub

Create /boot/grub/menu.lst

There's one last step before telling the VPS to use pv-grub: create the grub menu.lst file. If you've used grub on a workstation you may or may not have looked at a menu.lst file before. Many newer distributions install grub 2, which uses a different configuration scheme from the original (and uses a different file, "grub.cfg"). Pv-grub uses the "grub legacy" format.

Tell your favorite text editor to open menu.lst for editing:

sudo nano /boot/grub/menu.lst

Your favorite text editor doesn't have to be "nano" of course. That's just an example.

Editing menu.lst

We'll want the contents to look something like:

default=0
timeout=5

title=Gentoo kernel 2.6.34-gentoo-r12-simple
    root (hd0)
    kernel /boot/kernel-genkernel-x86_64-2.6.34-gentoo-r12-simple ro console=hvc0 root=/dev/xvda1
    initrd /boot/initramfs-genkernel-x86_64-2.6.34-gentoo-r12-simple

The most important parts to change are the "kernel" and "initrd" lines. Those lines in particular will need to be changed from the above example. The version installed by the kernel package will probably be different, and the name can differ depending on whether you're running a 32-bit or 64-bit Linux image.

Change the "kernel" reference in the example above to point to the kernel on your filesystem. Don't forget the "/boot" part. Leave the rest of the line the same.

If there's an "initrd" or "initramfs" file in /boot, or one that ends in ".img", edit the initrd line above to point to that file. Again, remember the "/boot" part, pv-grub needs to know exactly where to find the file.

You may want to change the "title=" line. You can have anything you like after the equals sign. The title just reflects what text will appear in the grub menu. You're probably the only person who will ever see the title, so indulge your whimsy. Name the kernel after where you got it and the version number like I did above, or pick your favorite Flintstones character.

The "timeout=5" line tells pv-grub how long to linger on the grub menu before proceeding to use the default kernel. It should be fine at 5 seconds.

The "default" line tells pv-grub which kernel to use as the default. The "0" refers to the first one listed, which in this case is also the only one listed. If you had two items in your menu.lst file and wanted the second to boot by default you would change the "0" to "1".

Adding more choices

You may want to add more items to the grub menu if you have more than one kernel available. To do so just copy that block of text that starts with "title=" and edit the new menu item settings for each other kernel. For example:

default=0
timeout=5

title=My custom 2.6.35.7 kernel
    root (hd0)
    kernel /boot/vmlinuz-2.6.35.7-mine ro console=hvc0 root=/dev/xvda1
    initrd /boot/initrd.img-2.6.35.7-mine

title=Gentoo kernel 2.6.34-gentoo-r12-simple (works)
    root (hd0)
    kernel /boot/kernel-genkernel-x86_64-2.6.34-gentoo-r12-simple ro console=hvc0 root=/dev/xvda1
    initrd /boot/initramfs-genkernel-x86_64-2.6.34-gentoo-r12-simple

Having more than one choice in the menu.lst can be especially useful when you're upgrading to a new kernel. You can keep the original kernel in the menu (the one you know works) so you can easily switch back if there's a problem. You'll be able to access the grub menu at boot via the web console for your VPS.

Remember that "default=0" means that the first kernel in the list is the one pv-grub will use if you don't select a different choice from the grub menu.

To the SliceManager

Your VPS should be ready for pv-grub at this point. This is your last chance to back up important data and, if you're worried about your chances, make offerings to the deity or deities of your choice.

With that done, log into the SliceManager and go to the entry for your slice.

The web console

Before making the switch to pv-grub open the web console for your VPS using the "Console" option. That should give you a window where you can look at what your server is printing to the console. That includes the stuff that would scroll across the screen while the VPS boots, which is what we want to watch when the kernel change is applied.

When you load the console, if you don't see any text in the window try hitting enter a couple times. You should see a login prompt, which is what we want. You don't need to log into the VPS in the console since we're just monitoring output.

Change kernel to pv-grub

To enable pv-grub you'll need to contact support via the SliceManager and ask them to make the switch-over for your VPS.

When the change is applied, go back to the web console to see what your VPS is up to. You should see it shut down and start up again. If everything goes smoothly you'll see the grub menu and then eventually a login prompt. If you see an error instead skip down the "Troubleshooting" section.

Check the results

Once the VPS is running using pv-grub, you can ssh into it and check the kernel version it's running:

$ uname -a
Linux demo 2.6.34-gentoo-r12-simple #1 SMP Thu Oct 28 02:26:33 GMT 2010 x86_64 Quad-Core AMD Opteron(tm) Processor 2347 HE AuthenticAMD GNU/Linux

The version should match the kernel you installed. From now on, to use a different kernel you'll just need to install it and edit the /boot/grub/menu.lst file appropriately.

To be extra sure that the fstab changes are all good, make sure your swap is using the new device too:

$ swapon -s
Filename                Type        Size    Used    Priority
/dev/xvda2                              partition   524280  0   -1

If nothing is listed when you run "swapon -s", recheck your /etc/fstab changes to make sure the designated swap device isn't still pointing to /dev/sda2.

Troubleshooting

If you get an error message in the web console instead of a login prompt (or just a blank screen with an error that flashes past really quickly before disappearing) then something has gone wrong. Pay attention to where in the process the error occurred since that can help narrow down the problem. If you missed it the first time, reboot the server so you can see it again.

If the VPS never got to the point where it displayed the grub menu that means there was a problem loading pv-grub to begin with. Hopefully that will never happen, but if it does contact support and they can help track down the problem.

If you see the grub menu before seeing the error, or get a message from grub that it couldn't display the menu, then the problem lies somewhere on your VPS filesystem. To get at those files now you'll have to either enable rescue mode for the VPS or restore a backed-up snapshot.

If you see an error message about a problem decompressing the kernel (probably mentioning bzip or LZMA), it means the kernel you tried to boot from uses a compression scheme that pv-grub doesn't support. Fixing that problem might require rebuilding the kernel to compress with gzip, or you may just want to copy the uncompressed "vmlinux" file from the kernel source directory (usually in /usr/src) to /boot and change menu.lst to use that kernel instead.

An error complaining that the root filesystem can't be found usually means either menu.lst doesn't point to the right files or that there is a mistake in the /etc/fstab file.

Rescue mode

When your VPS won't boot you can access its filesystem and make changes using "rescue mode". You can put your VPS into rescue mode via the SliceManager. Rescue mode is described in more detail in this article, but we cover the basics here.

Putting the VPS into rescue mode causes it to boot into a custom "rescue image" instead of booting with your regular configuration. If a problem is keeping your VPS from booting rescue mode is a Good Thing.

To make changes to your server in rescue mode, once you've logged in (via ssh or in the web console) you can run:

mount /dev/sda1 /mnt

Your whole filesystem will be available for changes under that /mnt directory. You can get to /boot using "/mnt/boot", /etc would be in "/mnt/etc", and so forth. Once you're done making changes you can exit rescue mode and the VPS will try to boot normally.

Double-check the changes

The first thing to check when you're looking at the files on your VPS via rescue mode is the /boot/grub/menu.lst file. Check that the kernel and initrd references point to the right files in the right locations. Typos are a fact of life.

Next look at /etc/fstab to make sure the name of the device for the root filesystem was changed from "sda1" to "xvda1".

Also check /etc/inittab for the "hvc0" TTY line you added at the end, making sure it matches the instructions.

If everything seems to be in order there may be a problem with the kernel you installed. You should roll back the change in /etc/fstab (changing the device back to /dev/sda1) then disable pv-grub in the SliceManager. That will let you reboot normally and check the kernel that was installed. Make sure it's a kernel with Xen compatibility enabled.

In one place: Enabling pv-grub

If you're creating a new VPS or modifying several servers it's handy to just have all the commands in one place instead of digging through the full tutorial. Here's the short version of the steps you can take on a Gentoo VPS to prepare it for enabling pv-grub with a stock Gentoo kernel source package:

sudo emerge --sync
sudo emerge --update portage
sudo sed -i 's/sda/xvda/' /etc/fstab
sudo echo "h0:12345:respawn:/sbin/agetty 38400 hvc0 linux" >> /etc/inittab
sudo grep hvc0 /etc/securetty || echo "hvc0" >> /etc/securetty
sudo emerge gentoo-sources genkernel
sudo zcat /proc/config.gz > /usr/src/linux/.config
sudo sed -i 's/rscloud/simple/' /usr/src/linux/.config
sudo genkernel --oldconfig all
sudo mkdir -p /boot/grub

After that look at the kernel and initrd names in /boot then edit /boot/grub/menu.lst accordingly.

Ask support to enable pv-grub and the VPS will reboot.

In one place: Disabling pv-grub

If you want to disable pv-grub you'll need to revert the change that was made to /etc/fstab:

sudo sed -i 's/xvda/sda/' /etc/fstab

It's likely that you would need to make the change while in rescue mode, so if you're in rescue mode and have your VPS filesystem mounted to "/mnt" the sed command would look like:

sed -i 's/xvda/sda/' /mnt/etc/fstab

The other changes and the installed kernel can stay in place without interfering with normal operation.

If you did disable the old TTY, remember to uncomment the line in /etc/inittab. The proper TTY device entry for a non-pv-grub kernel will look like:

c1:12345:respawn:/sbin/agetty 38400 tty1 linux

Once you've made the necessary adjustments on your VPS you can make the switch back to one of our stock kernels.

Summary

Once you have pv-grub enabled and are running the kernel you've installed you should be okay to install other kernels when you need to. Just remember to point to them in menu.lst.

If you want to compile your own kernel and run it first refer to this article about obtaining our kernel source code. Even if you aren't going to use that source code it can be useful to refer to the article to see how we configure our kernel builds.

  • -- Jered
Want to comment?


(not made public)

(optional)

(use plain text or Markdown syntax)