Using pv-grub to run custom kernels on Gentoo - preparing the slice

The pv-grub kernel option allows your Gentoo slice to boot from your own kernel instead of one of ours. Before you can try it out you need to prepare the slice with some config changes.


Your own kernel

The Linux kernel manages processes and how they carry out their tasks. The kernel is what makes Linux "Linux". Every distribution has a Linux kernel at the core, often with their own tweaks included but still using the standard kernel as the base.

We provide several kernels that have been tested with our Linux images and are known to work well with them. Most users will want to stick to those kernels. It's safe, it's stable, and it's easy.

There are times you'll want to install a different kernel from what we offer. Maybe a new kernel version has a feature you really want (or an older version lacks one you don't want), or perhaps you want to change some aspects of the configuration of our stock kernels. That's where pv-grub comes in.

These instructions are for Gentoo 10.1 and higher. The process will probably work with earlier versions but has not been tested with them.

pv-grub

Normally the kernel running on your server is selected by the virtualization layer that creates your VPS instance (in our case, a program called "Xen"). That means your kernel is not in your VPS filesystem.

If you tell Xen to use a feature called "pv-grub" (by contacting support through the SliceManager) you can change that behavior. Instead of the virtualization layer selecting your kernel from its own list it will look for the kernel on the VPS filesystem. That kernel can be a distribution's stock kernel (with Xen support enabled) or it can be one you've compiled yourself.

There are several changes that need to be made in order to make pv-grub work with your system. We'll cover those changes in this article and its sequel. We'll also talk a bit about how to recover if something goes wrong.

Warning

The kernel is really important to a Linux installation. If there's something about the kernel that doesn't work with the (virtual) hardware you're running on, the system just plain won't start. So installing your own kernels carries some risk.

Take precautions before diving into kernel experiments. Back up important data or take a backup snapshot of the VPS. Test the process on a new VPS before trying to implement it on a production instance. Keep track of the changes you make so you can backtrack and revert them later if necessary.

It may be useful to be familiar with the kernel in general and how it works. There's good information to be found at the official kernel archive site and in your distribution's documentation as well.

Kernel config

More advanced users may just want the skinny on what's required for a custom kernel. This section is for them. If you want to start with a stock kernel (recommended, since that gets you a kernel for pv-grub that you know works and lets you tinker from there), skip ahead.

The important points to keep in mind:

Use Xen config options. You can narrow down the ones you absolutely need by running a "make localmodconfig" to generate an initial config file based on the kernel you're already running on your VPS, or get the config for your current kernel from /proc/config.gz. At the least, "Xen guest support" must be enabled for the kernel.

Use gzip compression for the kernel. There are problems trying to load a kernel that's been compressed with bzip or LZMA through pv-grub. Make sure to set the compression type in the config to "gzip". Alternately, the uncompressed "vmlinux" version of the kernel will work with pv-grub (found in the root of the kernel source directory after compiling).

Change the TTY. Details on the changes that need to be made to enable the different TTY device are later in this article. Remember to make sure the new TTY device is in /etc/securetty too.

Change the fstab. The disk devices change when using pv-grub, so before booting with pv-grub the /etc/fstab file will need to be updated to reflect the new device names (from /dev/sda to /dev/xvda, basically).

Other interface names change. This generally is not an issue if your kernel has udev enabled, but you should skim ahead and check for changes to module aliases just in case.

Set up /boot/grub/menu.lst. Instructions for setting up the menu.lst file are listed later in this article. You won't be able to boot with pv-grub unless there's a menu.lst file telling pv-grub what kernel choices are available.

Using a stock kernel

This article will describe the process of switching to one of your distribution's stock kernels. It is possible to compile your own kernel from source and use it instead.

We're using a stock kernel here for convenience and for practicality. Having pv-grub enabled for your VPS means that if pv-grub can't boot from a working kernel on your filesystem it won't boot at all. It's a good idea to have something you know will work in place (like a stock Xen-enabled kernel) before you introduce another variable into the experiment like a custom-built kernel.

Besides, compiling a kernel from source is a whole different bag of weasels. One thing at a time.

Update the system

Before downloading stock kernel source from the Gentoo repository it's a good idea to upgrade your installed software to the latest versions, particularly if you're moving to a newer kernel release.

First, sync the portage data and make sure you have the latest portage version:

sudo emerge --sync
sudo emerge --update portage

If you want to update the rest of the software on your system too, run:

sudo emerge --update world

Edit fstab

This is the change that will rig the system so it can't boot without pv-grub anymore, so you're warned.

Since the device used to load the filesystem (the virtual hard drive) is called "xvda" when pv-grub is loading the kernel, we have to change /etc/fstab. That's the file that tells the system how to access and mount the filesystem.

The device reference changes from "/dev/sda" to "/dev/xvda" (for both the root filesystem and swap). You can either make the change manually with a text editor or you can use sed:

sudo sed -i 's/sda/xvda/' /etc/fstab

That command replaces "sda" with "xvda" wherever it exists in the /etc/fstab file. If you need to revert that change later you can just run the sed command in reverse (putting "xvda" first and "sda" second).

Add the TTY

The name of the TTY device, which represents your ability to interact with the system (via ssh or the console), is different when booting through pv-grub. To prepare the system for that change we can add to the VPS configuration without breaking the non-pv-grub configuration.

Gentoo keeps the configuration for the TTY in /etc/inittab. We just need to add a line for the new TTY device, hvc0.

The line we need to add to /etc/inittab can be added at the end of the file. It will look like:

h0:12345:respawn:/sbin/agetty 38400 hvc0 linux

It's okay to leave any previous TTY device lines in place for now.

You can use a text editor to add the line, or run the following command to tack it onto the end of the file:

sudo echo "h0:12345:respawn:/sbin/agetty 38400 hvc0 linux" >> /etc/inittab

You can make sure the line was added by checking just the end of the file:

tail /etc/inittab

Add to securetty

We may need to tell the system that the new TTY is okay for root to use by adding this line to /etc/securetty:

hvc0

If it's already in that file, great. If not, add the TTY on its own line.

Again, we can save a little time by turning the edit into a one-shot command (that adds hvc0 only if it isn't already in securetty):

sudo grep hvc0 /etc/securetty || echo "hvc0" >> /etc/securetty

Disabling the old TTY

It's possible after rebooting that you may see a bunch of statements in the console similar to:

INIT: "Id "c1" respawning too fast: disabled for 5 minutes

That message is fairly harmless, it just means that the system is trying to spawn that TTY device (probably the old one) and can't. If you want to get rid of those messages comment out the TTY line or lines in /etc/inittab that correspond to that TTY by putting a "#" character in front of them.

For the example above I would look for the line that starts with "c1:" (the ID of the init directive listed in the error) and comment it out like so:

#c1:12345:respawn:/sbin/agetty 38400 tty1 linux

I recommend waiting until you're certain that you have pv-grub working and will stick with it before commenting the TTY out. Having the line in there makes it easier to switch back to a non-pv-grub configuration if you need to troubleshoot (more on that in the next article).

Install the kernel package

There are a few kernel source packages in the Gentoo repositories. You can run "emerge -s sources" to see them all, but for our purposes the standard "gentoo-sources" will work:

sudo emerge gentoo-sources

There's a "xen-sources" in there too, but it's an older kernel version. More recent standard kernel sources include virtualization support, which is why we're using the gentoo-sources package.

To build the kernel we'll use a tool called "genkernel". The kernels produced by genkernel generally include more modules and such than you need, but they have the advantage of being easy to build and very compatible. Our goal here is a working kernel we can use as a starting point, after all.

sudo emerge genkernel

Linux source symlink

So far all we've installed is the source code for the kernel we want and a tool we can use to build it. Soon, we'll build it.

First, take a look at a long directory listing of the /usr/src directory:

$ ls -l /usr/src
total 4
lrwxrwxrwx  1 root root   23 Oct 28 02:06 linux -> linux-2.6.34-gentoo-r12
drwxr-xr-x 24 root root 4096 Oct 28 03:08 linux-2.6.34-gentoo-r12

Gentoo requires a "/usr/src/linux" symlink that points to the current kernel version. When genkernel runs it will use the kernel source referenced by that symlink.

If you haven't installed any kernel sources before the package manager should have taken care of that symlink for you. If you had some kernel sources in /usr/src already, then the "linux" symlink wouldn't have been changed and needs to be pointed to the new kernel source.

If you need to change the symlink, run the following commands (changing the target name to point to the source you installed):

cd /usr/src
sudo rm linux
sudo ln -s linux-2.6.34-gentoo-r12 linux

Copy the config

Now we'll run genkernel to build the kernel. The default configuration for the kernel source doesn't enable the Xen stuff we need, so before we begin let's copy the config from the currently-running kernel over to our source directory:

sudo zcat /proc/config.gz > /usr/src/linux/.config

Note the period at the beginning of ".config".

That config file will work, but it's also set up to append "-rscloud" to the end of the created file names. If you're okay with that you can leave that as-is. If you want to change it, edit .config and look for this line:

CONFIG_LOCALVERSION="-rscloud"

Change that "-rscloud" to whatever you like.

To save some searching you can also use sed to make the change. If you wanted to change that "rscloud" to "simple", you can run:

sudo sed -i 's/rscloud/simple/' /usr/src/linux/.config

Run genkernel

At last, with the config in place, let's build the kernel:

sudo genkernel --oldconfig all

It will take a while to complete. Usually fifteen to twenty minutes.

When it's done you'll have a new kernel and initial ramdisk installed in /boot.

Summary

The changes made in this article are safe to leave in place if you take a break or want to revert to a non-pv-grub boot later, except the change to /etc/fstab. If you do need to interrupt this process make sure to set /etc/fstab to point back to /dev/sda or there could be a problem if your VPS needs to reboot.

The process of setting up your VPS to use pv-grub is continued in the next article in this series.

  • -- Jered
Want to comment?


(not made public)

(optional)

(use plain text or Markdown syntax)