Using pv-grub to run custom kernels on RHEL - preparing the slice

The pv-grub kernel option allows your RHEL slice to boot from your own kernel instead of one of ours. Before you can try it out you need to prepare the slice with some config changes.


Your own kernel

The Linux kernel manages processes and how they carry out their tasks. The kernel is what makes Linux "Linux". Every distribution has a Linux kernel at the core, often with their own tweaks included but still using the standard kernel as the base.

We provide several kernels that have been tested with our Linux images and are known to work well with them. Most users will want to stick to those kernels. It's safe, it's stable, and it's easy.

There are times you'll want to install a different kernel from what we offer. Maybe a new kernel version has a feature you really want (or an older version lacks one you don't want), or perhaps you want to change some aspects of the configuration of our stock kernels. That's where pv-grub comes in.

These instructions are for Red Hat Enterprise Linux 5.4 and higher. The process will probably work with earlier versions but has not been tested with them.

pv-grub

Normally the kernel running on your server is selected by the virtualization layer that creates your VPS instance (in our case, a program called "Xen"). That means your kernel is not in your VPS filesystem.

If you tell Xen to use a feature called "pv-grub" (by contacting support through the SliceManager) you can change that behavior. Instead of the virtualization layer selecting your kernel from its own list it will look for the kernel on the VPS filesystem. That kernel can be a distribution's stock kernel (with Xen support enabled) or it can be one you've compiled yourself.

There are several changes that need to be made in order to make pv-grub work with your system. We'll cover those changes in this article and its sequel. We'll also talk a bit about how to recover if something goes wrong.

Warning

The kernel is really important to a Linux installation. If there's something about the kernel that doesn't work with the (virtual) hardware you're running on, the system just plain won't start. So installing your own kernels carries some risk.

Take precautions before diving into kernel experiments. Back up important data or take a backup snapshot of the VPS. Test the process on a new VPS before trying to implement it on a production instance. Keep track of the changes you make so you can backtrack and revert them later if necessary.

It may be useful to be familiar with the kernel in general and how it works. There's good information to be found at the official kernel archive site and in your distribution's documentation as well.

Kernel config

More advanced users may just want the skinny on what's required for a custom kernel. This section is for them. If you want to start with a stock kernel (recommended, since that gets you a kernel for pv-grub that you know works and lets you tinker from there), skip ahead.

The important points to keep in mind:

Use Xen config options. You can narrow down the ones you absolutely need by running a "make localmodconfig" to generate an initial config file based on the kernel you're already running on your VPS, or get the config for your current kernel from /proc/config.gz. At the least, "Xen guest support" must be enabled for the kernel.

Use gzip compression for the kernel. There are problems trying to load a kernel that's been compressed with bzip or LZMA through pv-grub. Make sure to set the compression type in the config to "gzip". Alternately, the uncompressed "vmlinux" version of the kernel will work with pv-grub (found in the root of the kernel source directory after compiling).

Change the TTY. Details on the changes that need to be made to enable the different TTY device are later in this article. Remember to make sure the new TTY device is in /etc/securetty too.

Change the fstab. The disk devices change when using pv-grub, so before booting with pv-grub the /etc/fstab file will need to be updated to reflect the new device names (from /dev/sda to /dev/xvda, basically).

Other interface names change. This generally is not an issue if your kernel has udev enabled, but you should skim ahead and check for changes to module aliases just in case.

Set up /boot/grub/menu.lst. Instructions for setting up the menu.lst file are listed later in this article. You won't be able to boot with pv-grub unless there's a menu.lst file telling pv-grub what kernel choices are available.

Using a stock kernel

This article will describe the process of switching to one of your distribution's stock kernels. It is possible to compile your own kernel from source and use it instead.

We're using a stock kernel here for convenience and for practicality. Having pv-grub enabled for your VPS means that if pv-grub can't boot from a working kernel on your filesystem it won't boot at all. It's a good idea to have something you know will work in place (like a stock Xen-enabled kernel) before you introduce another variable into the experiment like a custom-built kernel.

Besides, compiling a kernel from source is a whole different bag of weasels. One thing at a time.

Update the system

Before downloading a stock kernel from the RHEL repository it's a good idea to upgrade your installed software to the latest versions, particularly if you're moving to a newer kernel release:

sudo yum upgrade

Module aliases

To tell the system to use the right references for the network and virtual disk interfaces we'll want to add some lines to /etc/modprobe.conf. Before we do that, make a backup in case you want to revert the changes later:

sudo cp /etc/modprobe.conf /etc/modprobe.conf.bak

Next we want to replace the contents of /etc/modprobe.conf with the following three lines:

alias eth0 xennet
alias eth1 xennet
alias scsi_hostadapter xenblk

To skip the text editor, you can make this change by running these commands:

sudo echo "alias eth0 xennet" > /etc/modprobe.conf
sudo echo "alias eth1 xennet" >> /etc/modprobe.conf
sudo echo "alias scsi_hostadapter xenblk" >> /etc/modprobe.conf

This change won't necessarily keep your VPS from booting with a non-pv-grub kernel, but if you do decide not to stick with pv-grub you'll want to move the backed-up modprobe.conf.bak file back to modprobe.conf.

Install the kernel package

There are a few kernel packages in the RHEL repositories. The one you want is the one with Xen support enabled, which should be "kernel-xen". To install it run:

sudo yum install kernel-xen

Installing the kernel package won't reboot your slice or configure it to use that kernel, it just puts the files we need in the right places.

Make sure you've made the change to /etc/modprobe.conf before installing the kernel package. The package manager uses that file when building the initrd image (the kernel's initial ramdisk).

A look at /boot

To tell pv-grub where to look for the new kernel you'll need to know what it's called and where it is. The where part is straightforward — it's in /boot. Check a directory listing of /boot and you'll see something like:

$ ls /boot
config-2.6.18-194.17.1.el5xen      vmlinuz-2.6.18-194.17.1.el5xen
initrd-2.6.18-194.17.1.el5xen.img  xen.gz-2.6.18-194.17.1.el5
symvers-2.6.18-194.17.1.el5xen.gz  xen-syms-2.6.18-194.17.1.el5
System.map-2.6.18-194.17.1.el5xen

The two files you need to take note of are the one that starts with "initrd" and the one that starts with "vmlinuz". The former is the initial ramdisk (and may not always be present, or may start with "initramfs") and the latter is the actual kernel. If you see more than one of each it means you have more than one kernel version installed. Pick the files that match the version of the kernel you are trying to install (probably the ones with the newest version number).

To tell pv-grub to use the new kernel we'll be editing a file in the "grub" subdirectory of /boot.

The grub directory

To tell pv-grub to use the new kernel we'll be editing a file in the "grub" subdirectory of /boot.

The grub package is a bootloader, which is to say, it lets you pick how you want your system to start up from a menu at boot time. If you've set up a linux installation on a workstation you may have used it before. For your VPS editing the grub menu works similarly to editing it on a workstation.

The menu configuration file needs to be in /boot/grub, which may already exist. If it doesn't, create it:

sudo mkdir -p /boot/grub

Create /boot/grub/menu.lst

There's one last step before telling the VPS to use pv-grub: create the grub menu.lst file. If you've used grub on a workstation you may or may not have looked at a menu.lst file before. Many newer distributions install grub 2, which uses a different configuration scheme from the original (and uses a different file, "grub.cfg"). Pv-grub uses the "grub legacy" format.

Tell your favorite text editor to open menu.lst for editing:

sudo nano /boot/grub/menu.lst

Your favorite text editor doesn't have to be "nano" of course. That's just an example.

Editing menu.lst

We'll want the contents to look something like:

default=0
timeout=5

title=RHEL kernel 2.6.18-194.17.1.el5xen
    root (hd0)
    kernel /boot/vmlinuz-2.6.18-194.17.1.el5xen ro console=hvc0 root=/dev/xvda1
    initrd /boot/initrd-2.6.18-194.17.1.el5xen.img

The most important parts to change are the "kernel" and "initrd" lines. Those lines in particular will need to be changed from the above example. The version installed by the kernel package will probably be different, and the name can differ depending on whether you're running a 32-bit or 64-bit Linux image.

Change the "kernel" reference in the example above to point to the kernel on your filesystem. Don't forget the "/boot" part. Leave the rest of the line the same.

If there's an "initrd" or "initramfs" file in /boot, or one that ends in ".img", edit the initrd line above to point to that file. Again, remember the "/boot" part, pv-grub needs to know exactly where to find the file.

You may want to change the "title=" line. You can have anything you like after the equals sign. The title just reflects what text will appear in the grub menu. You're probably the only person who will ever see the title, so indulge your whimsy. Name the kernel after where you got it and the version number like I did above, or pick your favorite Flintstones character.

The "timeout=5" line tells pv-grub how long to linger on the grub menu before proceeding to use the default kernel. It should be fine at 5 seconds.

The "default" line tells pv-grub which kernel to use as the default. The "0" refers to the first one listed, which in this case is also the only one listed. If you had two items in your menu.lst file and wanted the second to boot by default you would change the "0" to "1".

Adding more choices

You may want to add more items to the grub menu if you have more than one kernel available. To do so just copy that block of text that starts with "title=" and edit the new menu item settings for each other kernel. For example:

default=0
timeout=5

title=My custom 2.6.35.7 kernel
    root (hd0)
    kernel /boot/vmlinuz-2.6.35.7-mine ro console=hvc0 root=/dev/xvda1
    initrd /boot/initrd.img-2.6.35.7-mine

title=RHEL kernel 2.6.18-194.17.1.el5xen (works)
    root (hd0)
    kernel /boot/vmlinuz-2.6.18-194.17.1.el5xen ro console=hvc0 root=/dev/xvda1
    initrd /boot/initrd.img-2.6.18-194.17.1.el5xen.img

Having more than one choice in the menu.lst can be especially useful when you're upgrading to a new kernel. You can keep the original kernel in the menu (the one you know works) so you can easily switch back if there's a problem. You'll be able to access the grub menu at boot via the web console for your VPS.

Remember that "default=0" means that the first kernel in the list is the one pv-grub will use if you don't select a different choice from the grub menu.

Add the TTY

The name of the TTY device, which represents your ability to interact with the system (via ssh or the console), is different when booting through pv-grub. To prepare the system for that change we can add to the VPS configuration without breaking the non-pv-grub configuration.

RHEL keeps the configuration for the TTY in /etc/inittab. We just need to add a line for the new TTY device, xvc0.

The line we need to add to /etc/inittab can be added at the end of the file. It will look like:

8:2345:respawn:/sbin/mingetty xvc0

It's okay to leave any previous TTY device lines in place for now.

You can use a text editor to add the line, or run the following command to tack it onto the end of the file:

sudo echo "8:2345:respawn:/sbin/mingetty xvc0" >> /etc/inittab

You can make sure the line was added by checking just the end of the file:

tail /etc/inittab

Add to securetty

We may need to tell the system that the new TTY is okay for root to use by adding this line to /etc/securetty:

xvc0

If it's already in that file, great. If not, add the TTY on its own line.

Again, we can save a little time by turning the edit into a one-shot command (that adds xvc0 only if it isn't already in securetty):

sudo grep xvc0 /etc/securetty || echo "xvc0" >> /etc/securetty

Disabling the old TTY

It's possible after rebooting that you may see a bunch of statements in the console similar to:

INIT: "Id "1" respawning too fast: disabled for 5 minutes

That message is fairly harmless, it just means that the system is trying to spawn that TTY device (probably the old one) and can't. If you want to get rid of those messages comment out the TTY line or lines in /etc/inittab that correspond to that TTY by putting a "#" character in front of them.

For the example above I would look for the line that starts with "1:" (the ID of the init directive listed in the error) and comment it out like so:

# 1:2345:respawn:/sbin/mingetty tty1

I recommend waiting until you're certain that you have pv-grub working and will stick with it before commenting the TTY out. Having the line in there makes it easier to switch back to a non-pv-grub configuration if you need to troubleshoot (more on that in the next article).

Summary

The changes made in this article are safe to leave in place if you take a break or want to revert to a non-pv-grub boot later. They won't interfere with the normal operation of your VPS when pv-grub is disabled, nor will they cause a problem if the VPS is rebooted.

The more disruptive changes are reserved for the next article, which will also cover actually enabling pv-grub.

  • -- Jered
Want to comment?


(not made public)

(optional)

(use plain text or Markdown syntax)