Using pv-grub to run custom kernels on Arch - preparing the slice

The pv-grub kernel option allows your Arch Linux slice to boot from your own kernel instead of one of ours. Before you can try it out you need to prepare the slice with some config changes.


Your own kernel

The Linux kernel manages processes and how they carry out their tasks. The kernel is what makes Linux "Linux". Every distribution has a Linux kernel at the core, often with their own tweaks included but still using the standard kernel as the base.

We provide several kernels that have been tested with our Linux images and are known to work well with them. Most users will want to stick to those kernels. It's safe, it's stable, and it's easy.

There are times you'll want to install a different kernel from what we offer. Maybe a new kernel version has a feature you really want (or an older version lacks one you don't want), or perhaps you want to change some aspects of the configuration of our stock kernels. That's where pv-grub comes in.

These instructions are for Arch Linux 2010.05 and higher. The process will probably work with earlier versions but has not been tested with them.

pv-grub

Normally the kernel running on your server is selected by the virtualization layer that creates your VPS instance (in our case, a program called "Xen"). That means your kernel is not in your VPS filesystem.

If you tell Xen to use a feature called "pv-grub" (by contacting support through the SliceManager) you can change that behavior. Instead of the virtualization layer selecting your kernel from its own list it will look for the kernel on the VPS filesystem. That kernel can be a distribution's stock kernel (with Xen support enabled) or it can be one you've compiled yourself.

There are several changes that need to be made in order to make pv-grub work with your system. We'll cover those changes in this article and its sequel. We'll also talk a bit about how to recover if something goes wrong.

Warning

The kernel is really important to a Linux installation. If there's something about the kernel that doesn't work with the (virtual) hardware you're running on, the system just plain won't start. So installing your own kernels carries some risk.

Take precautions before diving into kernel experiments. Back up important data or take a backup snapshot of the VPS. Test the process on a new VPS before trying to implement it on a production instance. Keep track of the changes you make so you can backtrack and revert them later if necessary.

It may be useful to be familiar with the kernel in general and how it works. There's good information to be found at the official kernel archive site and in your distribution's documentation as well.

Kernel config

More advanced users may just want the skinny on what's required for a custom kernel. This section is for them. If you want to start with a stock kernel (recommended, since that gets you a kernel for pv-grub that you know works and lets you tinker from there), skip ahead.

The important points to keep in mind:

Use Xen config options. You can narrow down the ones you absolutely need by running a "make localmodconfig" to generate an initial config file based on the kernel you're already running on your VPS, or get the config for your current kernel from /proc/config.gz. At the least, "Xen guest support" must be enabled for the kernel.

Use gzip compression for the kernel. There are problems trying to load a kernel that's been compressed with bzip or LZMA through pv-grub. Make sure to set the compression type in the config to "gzip". Alternately, the uncompressed "vmlinux" version of the kernel will work with pv-grub (found in the root of the kernel source directory after compiling).

Change the TTY. Details on the changes that need to be made to enable the different TTY device are later in this article. Remember to make sure the new TTY device is in /etc/securetty too.

Change the fstab. The disk devices change when using pv-grub, so before booting with pv-grub the /etc/fstab file will need to be updated to reflect the new device names (from /dev/sda to /dev/xvda, basically).

Other interface names change. This generally is not an issue if your kernel has udev enabled, but you should skim ahead and check for changes to module aliases just in case.

Set up /boot/grub/menu.lst. Instructions for setting up the menu.lst file are listed later in this article. You won't be able to boot with pv-grub unless there's a menu.lst file telling pv-grub what kernel choices are available.

Using a stock kernel

This article will describe the process of switching to one of your distribution's stock kernels. It is possible to compile your own kernel from source and use it instead.

We're using a stock kernel here for convenience and for practicality. Having pv-grub enabled for your VPS means that if pv-grub can't boot from a working kernel on your filesystem it won't boot at all. It's a good idea to have something you know will work in place (like a stock Xen-enabled kernel) before you introduce another variable into the experiment like a custom-built kernel.

Besides, compiling a kernel from source is a whole different bag of weasels. One thing at a time.

Update the system

Before downloading a stock kernel from the Arch repository it's a good idea to upgrade your installed software to the latest versions, particularly if you're moving to a newer kernel release.

Start by updating the package manager:

sudo pacman -Sy pacman

Then update the packages on your system. The easiest way to do that is to run a full system update:

sudo pacman -Syu

Also, if you don't have them installed already, having the base development tools installed can be helpful:

sudo pacman -Sy base-devel

Edit fstab

This is the change that will rig the system so it can't boot without pv-grub anymore, so you're warned.

Since the device used to load the filesystem (the virtual hard drive) is called "xvda" when pv-grub is loading the kernel, we have to change /etc/fstab. That's the file that tells the system how to access and mount the filesystem.

The device reference changes from "/dev/sda" to "/dev/xvda" (for both the root filesystem and swap). You can either make the change manually with a text editor or you can use sed:

sudo sed -i 's/sda/xvda/' /etc/fstab

That command replaces "sda" with "xvda" wherever it exists in the /etc/fstab file. If you need to revert that change later you can just run the sed command in reverse (putting "xvda" first and "sda" second).

Add the TTY

The name of the TTY device, which represents your ability to interact with the system (via ssh or the console), is different when booting through pv-grub. To prepare the system for that change we can change the VPS configuration without breaking the non-pv-grub configuration.

Arch keeps the configuration for the TTY in /etc/inittab. We just need to uncomment a line for the new TTY device, hvc0.

The line we need to enable /etc/inittab is towards the end of the file. That section will look like:

# Hypervisor Virtual Console for Xen and KVM
#h0:2345:respawn:/sbin/agetty -8 38400 hvc0 linux

It's okay to leave any previous TTY device lines in place for now.

You can use a text editor to uncomment that second line, or run the following command to automate the change:

sudo sed -i 's/#h0:/h0:/' /etc/inittab

Of course, if you didn't find that line at all in the inittab, then rather than uncommenting it you'll want to add it to the file.

Add to securetty

We may need to tell the system that the new TTY is okay for root to use by adding this line to /etc/securetty:

hvc0

If it's already in that file, great. If not, add the TTY on its own line.

Again, we can save a little time by turning the edit into a one-shot command (that adds hvc0 only if it isn't already in securetty):

sudo grep hvc0 /etc/securetty || echo "hvc0" >> /etc/securetty

Disabling the old TTY

It's possible after rebooting that you may see a bunch of statements in the console similar to:

INIT: "Id "c1" respawning too fast: disabled for 5 minutes

That message is fairly harmless, it just means that the system is trying to spawn that TTY device (probably the old one) and can't. If you want to get rid of those messages comment out the TTY line or lines in /etc/inittab that correspond to that TTY by putting a "#" character in front of them.

For the example above I would look for the line that starts with "c1:" (the ID of the init directive listed in the error) and comment it out like so:

# -8 options fixes umlauts problem on login
#c1:2345:respawn:/sbin/agetty -8 38400 tty1 linux

I recommend waiting until you're certain that you have pv-grub working and will stick with it before commenting the TTY out. Having the line in there makes it easier to switch back to a non-pv-grub configuration if you need to troubleshoot (more on that in the next article).

Install the kernel package

Next we'll need to install the kernel package itself. Make sure you've made the change to /etc/fstab before installing the kernel.

At the time of this writing, the latest stock kernel is the "kernel26" package:

sudo pacman -Sy kernel26

If the kernel26 package isn't available you can look at a list of kernel-related packages by running:

pacman -Ss kernel

Note that along with the most current kernel package there will also be one labeled "lts", for "long-term support". This kernel is considered the most stable kernel for production servers. You can install that package instead by changing, for example, "kernel26" to "kernel26-lts" in the command above.

Installing the kernel package won't reboot your slice or configure it to use that kernel, it just puts the files we need in the right places.

A look at /boot

To tell pv-grub where to look for the new kernel and image you'll need to know what they're called and where they are. The where part is straightforward — they're in /boot. Check a directory listing of /boot and you'll see something like:

$ ls /boot
kernel26-fallback.img  kernel26.img  System.map26  vmlinuz26

The two files you need to take note of are the one that ends with "img" and the one that starts with "vmlinuz". The former is the initial ramdisk and the latter is the actual kernel. If you see more than one of each it means you have more than one kernel version installed. Pick the files that match the version of the kernel you are trying to install (probably the ones with the newest version number).

The "vmlinuz" file installed by the package manager is actually not one we can use with pv-grub because of the way it's compressed. Fortunately we can work around that.

Copy the uncompressed kernel

Xen has trouble with the compression used for the stock kernel installed by Arch (bzip), so we'll want to copy the uncompressed version of the kernel into /boot.

The uncompressed kernel is stored in the /usr/src directory, in a subdirectory named after the kernel version. So, check the name of the directory:

$ ls /usr/src
linux-2.6.34-ARCH

Then run the command to copy the "vmlinux" file from that directory into /boot, matching the suffix given to the "vmlinuz" kernel already there:

sudo cp /usr/src/linux-2.6.34-ARCH/vmlinux /boot/vmlinux26

The grub directory

To tell pv-grub to use the new kernel we'll be editing a file in the "grub" subdirectory of /boot.

The grub package is a bootloader, which is to say, it lets you pick how you want your system to start up from a menu at boot time. If you've set up a linux installation on a workstation you may have used it before. For your VPS editing the grub menu works similarly to editing it on a workstation.

The menu configuration file needs to be in /boot/grub, which may already exist. If it doesn't, create it:

sudo mkdir -p /boot/grub

Create /boot/grub/menu.lst

There's one last step before telling the VPS to use pv-grub: create the grub menu.lst file. If you've used grub on a workstation you may or may not have looked at a menu.lst file before. Many newer distributions install grub 2, which uses a different configuration scheme from the original (and uses a different file, "grub.cfg"). Pv-grub uses the "grub legacy" format.

Tell your favorite text editor to open menu.lst for editing:

sudo nano /boot/grub/menu.lst

Your favorite text editor doesn't have to be "nano" of course. That's just an example.

Editing menu.lst

We'll want the contents to look something like:

default=0
timeout=5

title=Arch kernel 2.6.34-ARCH
    root (hd0)
    kernel /boot/vmlinux26 ro console=hvc0 root=/dev/xvda1
    initrd /boot/kernel26.img

The most important parts to change are the "kernel" and "initrd" lines. Those lines in particular will need to be changed from the above example. The version installed by the kernel package will probably be different, and the name can differ depending on whether you're running a 32-bit or 64-bit Linux image.

Change the "kernel" reference in the example above to point to the kernel on your filesystem. Don't forget the "/boot" part. Leave the rest of the line the same.

If there's an "initrd" or "initramfs" file in /boot, or one that ends in ".img", edit the initrd line above to point to that file. Again, remember the "/boot" part, pv-grub needs to know exactly where to find the file.

You may want to change the "title=" line. You can have anything you like after the equals sign. The title just reflects what text will appear in the grub menu. You're probably the only person who will ever see the title, so indulge your whimsy. Name the kernel after where you got it and the version number like I did above, or pick your favorite Flintstones character.

The "timeout=5" line tells pv-grub how long to linger on the grub menu before proceeding to use the default kernel. It should be fine at 5 seconds.

The "default" line tells pv-grub which kernel to use as the default. The "0" refers to the first one listed, which in this case is also the only one listed. If you had two items in your menu.lst file and wanted the second to boot by default you would change the "0" to "1".

Adding more choices

You may want to add more items to the grub menu if you have more than one kernel available. To do so just copy that block of text that starts with "title=" and edit the new menu item settings for each other kernel. For example:

default=0
timeout=5

title=My custom 2.6.35.7 kernel
    root (hd0)
    kernel /boot/vmlinuz-2.6.35.7-mine ro console=hvc0 root=/dev/xvda1
    initrd /boot/initrd.img-2.6.35.7-mine

title=Arch kernel 2.6.34-ARCH (works)
    root (hd0)
    kernel /boot/vmlinux26 ro console=hvc0 root=/dev/xvda1
    initrd /boot/kernel26.img

Having more than one choice in the menu.lst can be especially useful when you're upgrading to a new kernel. You can keep the original kernel in the menu (the one you know works) so you can easily switch back if there's a problem. You'll be able to access the grub menu at boot via the web console for your VPS.

Remember that "default=0" means that the first kernel in the list is the one pv-grub will use if you don't select a different choice from the grub menu.

Summary

The changes made in this article are safe to leave in place if you take a break or want to revert to a non-pv-grub boot later, except the change to /etc/fstab. If you do need to interrupt this process make sure to set /etc/fstab to point back to /dev/sda or there could be a problem if your VPS needs to reboot.

The process of setting up your VPS to use pv-grub is continued in the next article in this series.

  • -- Jered
Want to comment?


(not made public)

(optional)

(use plain text or Markdown syntax)