Using pv-grub to run custom kernels on Ubuntu - preparing the slice

The pv-grub kernel option allows your Ubuntu slice to boot from your own kernel instead of one of ours. Before you can try it out you need to prepare the slice with some config changes.


Your own kernel

The Linux kernel manages processes and how they carry out their tasks. The kernel is what makes Linux "Linux". Every distribution has a Linux kernel at the core, often with their own tweaks included but still using the standard kernel as the base.

We provide several kernels that have been tested with our Linux images and are known to work well with them. Most users will want to stick to those kernels. It's safe, it's stable, and it's easy.

There are times you'll want to install a different kernel from what we offer. Maybe a new kernel version has a feature you really want (or an older version lacks one you don't want), or perhaps you want to change some aspects of the configuration of our stock kernels. That's where pv-grub comes in.

These instructions are for Ubuntu Karmic (9.10) and higher. Unfortunately pv-grub is not compatible with the Ubuntu Hardy (8.04) stock kernel.

pv-grub

Normally the kernel running on your server is selected by the virtualization layer that creates your VPS instance (in our case, a program called "Xen"). That means your kernel is not in your VPS filesystem.

If you tell Xen to use a feature called "pv-grub" (by contacting support through the SliceManager) you can change that behavior. Instead of the virtualization layer selecting your kernel from its own list it will look for the kernel on the VPS filesystem. That kernel can be a distribution's stock kernel (with Xen support enabled) or it can be one you've compiled yourself.

There are several changes that need to be made in order to make pv-grub work with your system. We'll cover those changes in this article and its sequel. We'll also talk a bit about how to recover if something goes wrong.

Warning

The kernel is really important to a Linux installation. If there's something about the kernel that doesn't work with the (virtual) hardware you're running on, the system just plain won't start. So installing your own kernels carries some risk.

Take precautions before diving into kernel experiments. Back up important data or take a backup snapshot of the VPS. Test the process on a new VPS before trying to implement it on a production instance. Keep track of the changes you make so you can backtrack and revert them later if necessary.

It may be useful to be familiar with the kernel in general and how it works. There's good information to be found at the official kernel archive site and in your distribution's documentation as well.

Kernel config

More advanced users may just want the skinny on what's required for a custom kernel. This section is for them. If you want to start with a stock kernel (recommended, since that gets you a kernel for pv-grub that you know works and lets you tinker from there), skip ahead.

The important points to keep in mind:

Use Xen config options. You can narrow down the ones you absolutely need by running a "make localmodconfig" to generate an initial config file based on the kernel you're already running on your VPS, or get the config for your current kernel from /proc/config.gz. At the least, "Xen guest support" must be enabled for the kernel.

Use gzip compression for the kernel. There are problems trying to load a kernel that's been compressed with bzip or LZMA through pv-grub. Make sure to set the compression type in the config to "gzip". Alternately, the uncompressed "vmlinux" version of the kernel will work with pv-grub (found in the root of the kernel source directory after compiling).

Change the TTY. Details on the changes that need to be made to enable the different TTY device are later in this article. Remember to make sure the new TTY device is in /etc/securetty too.

Change the fstab. The disk devices change when using pv-grub, so before booting with pv-grub the /etc/fstab file will need to be updated to reflect the new device names (from /dev/sda to /dev/xvda, basically).

Other interface names change. This generally is not an issue if your kernel has udev enabled, but you should skim ahead and check for changes to module aliases just in case.

Set up /boot/grub/menu.lst. Instructions for setting up the menu.lst file are listed later in this article. You won't be able to boot with pv-grub unless there's a menu.lst file telling pv-grub what kernel choices are available.

Using a stock kernel

This article will describe the process of switching to one of your distribution's stock kernels. It is possible to compile your own kernel from source and use it instead.

We're using a stock kernel here for convenience and for practicality. Having pv-grub enabled for your VPS means that if pv-grub can't boot from a working kernel on your filesystem it won't boot at all. It's a good idea to have something you know will work in place (like a stock Xen-enabled kernel) before you introduce another variable into the experiment like a custom-built kernel.

Besides, compiling a kernel from source is a whole different bag of weasels. One thing at a time.

Update the system

Before downloading a stock kernel from the Ubuntu repository it's a good idea to update aptitude's package database:

sudo aptitude update

You should also update your installed software to the latest versions, particularly if you're upgrading to a newer kernel release:

sudo aptitude safe-upgrade

Install the kernel package

There are a few kernel packages in the Ubuntu repositories. The one you want is the one with Xen support enabled, which should be "linux-virtual". To install it run:

sudo aptitude install linux-virtual

The installer will list a few dependencies and a couple of them might be related to the bootloader "grub". That's okay; go ahead and say yes to the list of packages it offers to install.

Once the installation starts you'll get a screen asking for any "linux commands" you want to run at boot time. Leave that blank and continue.

Next the installation may give you a big warning about grub not being compatible with your installation: Are you sure you want to install the kernel without installing grub? Say "yes" to that question and it should finish installing the stuff you need.

Installing the kernel package won't reboot your slice or configure it to use that kernel, it just puts the files we need in the right places.

A look at /boot

To tell pv-grub where to look for the new kernel you'll need to know what it's called and where it is. The where part is straightforward — it's in /boot. Check a directory listing of /boot and you'll see something like:

$ ls /boot
System.map-2.6.32-25-server  initrd.img-2.6.32-25-server
config-2.6.32-25-server      vmlinuz-2.6.32-25-server
grub

The two files you need to take note of are the one that starts with "initrd" and the one that starts with "vmlinuz". The former is the initial ramdisk (and may not always be present, or may start with "initramfs") and the latter is the actual kernel. If you see more than one of each it means you have more than one kernel version installed. Pick the files that match the version of the kernel you are trying to install (probably the ones with the newest version number).

Note that the example above shows the kernel names for a 64-bit Ubuntu installation. For some reason Ubuntu uses different names for their 32-bit kernels, so instead of "server" you'd see "generic-pae" in the kernel filenames.

To tell pv-grub to use the new kernel we'll be editing a file in the "grub" subdirectory of /boot.

The grub directory

To tell pv-grub to use the new kernel we'll be editing a file in the "grub" subdirectory of /boot.

The grub package is a bootloader, which is to say, it lets you pick how you want your system to start up from a menu at boot time. If you've set up a linux installation on a workstation you may have used it before. For your VPS editing the grub menu works similarly to editing it on a workstation.

The menu configuration file needs to be in /boot/grub, which may already exist. If it doesn't, create it:

sudo mkdir -p /boot/grub

Create /boot/grub/menu.lst

There's one last step before telling the VPS to use pv-grub: create the grub menu.lst file. If you've used grub on a workstation you may or may not have looked at a menu.lst file before. Many newer distributions install grub 2, which uses a different configuration scheme from the original (and uses a different file, "grub.cfg"). Pv-grub uses the "grub legacy" format.

Tell your favorite text editor to open menu.lst for editing:

sudo nano /boot/grub/menu.lst

Your favorite text editor doesn't have to be "nano" of course. That's just an example.

Editing menu.lst

We'll want the contents to look something like:

default=0
timeout=5

title=Ubuntu kernel 2.6.32-25
    root (hd0)
    kernel /boot/vmlinuz-2.6.32-25-server ro console=hvc0 root=/dev/xvda1
    initrd /boot/initrd.img-2.6.32-25-server

Pay attention to the "kernel" and "initrd" lines. Those lines in particular will need to be changed from the above example. The version installed by the kernel package will probably be different, and the name can differ depending on whether you're running a 32-bit or 64-bit Linux image.

Change the "kernel" reference in the example above to point to the kernel on your filesystem. Don't forget the "/boot" part. Leave the rest of the line the same.

If there's an "initrd" or "initramfs" file in /boot, or one that ends in ".img", edit the initrd line above to point to that file. Again, remember the "/boot" part, pv-grub needs to know exactly where to find the file.

If you are running Ubuntu 10.10 (Maverick) or later then change the "root=/dev/xvda1" part of the "kernel" line to "root=/dev/sda1".

You may want to change the "title=" line. You can have anything you like after the equals sign. The title just reflects what text will appear in the grub menu. You're probably the only person who will ever see the title, so indulge your whimsy. Name the kernel after where you got it and the version number like I did above, or pick your favorite Flintstones character.

The "timeout=5" line tells pv-grub how long to linger on the grub menu before proceeding to use the default kernel. It should be fine at 5 seconds.

The "default" line tells pv-grub which kernel to use as the default. The "0" refers to the first one listed, which in this case is also the only one listed. If you had two items in your menu.lst file and wanted the second to boot by default you would change the "0" to "1".

Adding more choices

You may want to add more items to the grub menu if you have more than one kernel available. To do so just copy that block of text that starts with "title=" and edit the new menu item settings for each other kernel. For example:

default=0
timeout=5

title=My custom 2.6.35.7 kernel
    root (hd0)
    kernel /boot/vmlinuz-2.6.35.7-mine ro console=hvc0 root=/dev/xvda1
    initrd /boot/initrd.img-2.6.35.7-mine

title=Ubuntu kernel 2.6.32-25 (works)
    root (hd0)
    kernel /boot/vmlinuz-2.6.32-25-server ro console=hvc0 root=/dev/xvda1
    initrd /boot/initrd.img-2.6.32-25-server

Having more than one choice in the menu.lst can be especially useful when you're upgrading to a new kernel. You can keep the original kernel in the menu (the one you know works) so you can easily switch back if there's a problem. You'll be able to access the grub menu at boot via the web console for your VPS.

Remember that "default=0" means that the first kernel in the list is the one pv-grub will use if you don't select a different choice from the grub menu.

Add the TTY

The name of the TTY device, which represents your ability to interact with the system (via ssh or the console), is different when booting through pv-grub. To prepare the system for that change we can just add to the VPS configuration without breaking the non-pv-grub configuration.

Ubuntu keeps the configuration for the TTY in /etc/init. We just need to make a copy of that configuration and change it so we have support for the pv-grub TTY, hvc0.

First, copy the TTY config file:

sudo cp /etc/init/tty1.conf /etc/init/hvc0.conf

Next change the new file so it refers to the "hvc0" device instead of the "tty1" device. You can do that manually in a text editor, but to save some time and effort you can use a command called "sed" to do a quick find-and-replace on the file:

sudo sed -i 's/tty1/hvc0/' /etc/init/hvc0.conf

Breaking that command down, the "-i 's/tty1/hvc0'" is the part that tells sed it will perform a substitution (the "s" tells it that), turning "tty1" (the next part) into "hvc0" (the second part). The last argument to sed tells it what file it will target to carry out its instructions.

Add to securetty

We may need to tell the system that the new TTY is okay for root to use by adding this line to /etc/securetty:

hvc0

If it's already in that file, great. If not, add the TTY on its own line.

Again, we can save a little time by turning the edit into a one-shot command (that adds hvc0 only if it isn't already in securetty):

sudo grep hvc0 /etc/securetty || echo "hvc0" >> /etc/securetty

Disabling the old TTY

It's possible after rebooting that you may see a bunch of statements in the console similar to:

INIT: "Id "1" respawning too fast: disabled for 5 minutes

That message is fairly harmless, it just means that the system is trying to spawn that TTY device (probably the old one) and can't. If you want to get rid of those messages, remove the old TTY config file from /etc/init like so:

sudo rm /etc/init/tty1.conf

I recommend waiting until you're certain that you have pv-grub working and will stick with it before removing the old TTY. Having the config file in there makes it easier to switch back to a non-pv-grub configuration if you need to troubleshoot (more on that in the next article).

Summary

The changes made in this article are safe to leave in place if you take a break or want to revert to a non-pv-grub boot later. They won't interfere with the normal operation of your VPS when pv-grub is disabled, nor will they cause a problem if the VPS is rebooted.

The more disruptive changes are reserved for the next article, which will also cover actually enabling pv-grub.

  • -- Jered

Article Comments:

Shodan commented Thu Oct 28 06:40:42 UTC 2010:

Yum ??? Looks like this article is of Redhat.Please correct the article.

Jered commented Thu Oct 28 14:20:54 UTC 2010:

That was definitely a paste fail. Article's been restored to its intended Ubuntu state. Sorry about that.

Want to comment?


(not made public)

(optional)

(use plain text or Markdown syntax)