Xen on Debian

Dom0

Install Debian Sarge using the Sarge Debian Installer.

If you are using the "testing" distribution of Debian, some very preliminary notes are on Xen Dom0 on Debian Testing. They should only be treated as hints and used with the much better instructions on this page.

Building

Install packages necessary to build and run xen,

aptitude install libcurl3-dev python-dev bzip2 libncurses5-dev
aptitude install bridge-utils iproute python-twisted

Grab the Xen source and unpack it.

Clean and build the Xen system and kernels.

make world

(optional) Customize the kernels.

make kclean
cd linux-2.6.11-xen0
make ARCH=xen menuconfig
cd ..
cd linux-2.6.11-xenU
make ARCH=xen menuconfig
cd ..

This is also a good time to apply any patches you may need. (See the DRBD warning below.)

Rebuild the kernels.

make kernels

Installing

If you don't mind installing things into /usr and such:

sudo make install

However, I wanted the binaries and libraries to go into /usr/local. You can tweak the layout of dist/install to mirror how you want files installed into your system. I moved /usr to /usr/local and I modified where the python packages went.

cd dist/install/usr
mkdir local
mv * local
cd local/lib
mkdir -p python2.3/site-packages
mv python/xen python2.3/site-packages
rmdir python
cd ../..
sudo ./install.sh

Since I installed to /usr/local I needed to make sure the libraries would be found.

echo "/usr/local/lib" >> /etc/ld.so.conf
ldconfig -v

Also, I had to modify /usr/local/lib/python2.3/site-packages/xen/xend/server/SrvDaemon.py. In it, there is a hardcoded call to /usr/sbin/xfrd. This needs to be changed to /usr/local/sbin/xfrd.

Move the /lib/tls directory.

mv /lib/tls /lib/tls.disabled

In /boot/grub/menu.lst add a section for Xen above the automagic section. Have it boot into runlevel 4 so we can have Xen specific services startup which won't run if we ever boot traditionally (into runlevel 2).

title Xen 2.0 / XenLinux 2.6
  root   (hd0,0)
  kernel /boot/xen-2.0.gz dom0_mem=131072 
  module /boot/vmlinuz-2.6-xen0 root=/dev/sda1 ro console=tty0 4

Rebooterate.

reboot

Start the Xen control daemon.

/etc/init.d/xend start

You should check dmesg and /var/log/xend*.log and make sure nothing evil was incanted. Also, you should now be able to see Domain-0 is running.

xm list

Should produce something similar to:

Name              Id  Mem(MB)  CPU  State  Time(s)  Console
Domain-0           0      123    0  r----     19.7

If everything is working, you should have xend start automatically.

update-rc.d xend start 70 3 . stop 8 0 1 6 .

If xend is in /usr/local, add a PATH environment to /etc/init.d/xend.

PATH="${PATH}:/usr/local/sbin"

I modified /etc/init.d/xend to display start/stop messages similar to other Debian daemons. My version is here: xend initscript.

If you want some domains to automatically start/stop when you boot/shutdown (outside of something like heartbeat), install the xendomains scripts as well. Whatever domains are in /etc/xen/auto will automatically startup.

aptitude install lsb-base
update-rc.d xendomains start 98 3 . stop 2 0 1 6 .

DRBD on Dom0

Warning

There is a bug in the 2.6.11.12 kernel which will cause some nasty messages to get printed when you mount a DRBD device.

Badness in blk_plug_device at drivers/block/ll_rw_blk.c:1217
Badness in blk_remove_plug at drivers/block/ll_rw_blk.c:1238

You will need to apply the one-line patch to fix the bio_clone bug before using DRBD. As a side note, I was able to run DRBD 0.7.14 without this problem, but I don't trust it. See this post for more information about the bio_clone bug and DRBD.

Build/Install

Install the source and utils.

aptitude install module-assistant drbd0.7-module-source drbd0.7-utils

Build and install the module.

cd /usr/local/src/xen/xen-2.0 
ARCH=xen module-assistant --kernel-dir=`pwd`/linux-2.6.11-xen0 build drbd0.7-module
dpkg -i drbd0.7-module-2.6.11.12-xen0_0.7.10-1_i386.deb
update-modules
modprobe drbd

Check everything is kosher.

dmesg | tail -n 25 ; cat /proc/drbd

Configure

If you are going to be placing LVM on top of DRBD, you must filter out the physical device DRBD is running on or LVM will find labels on both the physical device and the DRBD device. In /etc/lvm.conf look for "filter" and add something like:

filter = [ "r|/dev/sda2|" ]

Both nodes must have /etc/hosts entries for their hostname to ensure they can get to one another.

Create identical drbd.conf files on both nodes.

# /etc/drbd.conf
#
global { minor-count 8; }
resource test {
    protocol C;
    #incon-degr-cmd "echo '!DRBD! pri on incon-degr' | wall ; sleep 60 ; halt -f";
    startup  { wfc-timeout 60; degr-wfc-timeout 120; }
    disk     { on-io-error detach; }
    syncer   { rate 700000K; group 1; }
    on xena {
        device      /dev/drbd0;
        disk        /dev/mapper/xen-drbdtest;
        address     172.16.0.67:7788;
        meta-disk   internal;
    }
    on xeno {
        device      /dev/drbd0;
        disk        /dev/mapper/xen-drbdtest;
        address     172.16.0.68:7788;
        meta-disk   internal;
    }
}

If you are using a pre-existing filesystem (i.e., you will not run mkfs on the drbd device once it's created) and "meta-disk internal", you must shrink the filesystem by 128M for the DRBD metadata and to avoid "access beyond end of device" errors.

Copy the drbd.conf file to both nodes and start drbd.

/etc/init.d/drbd start

DRBD will come up on both nodes in "secondary" mode.

Make your "primary" node the primary for all drbd devices.

drbdadm -- --do-what-I-say primary all

Check DRBD's status.

dmesg | tail ; cat /proc/drbd

After DRBD has synchronized, test out your new partition.

mkreiserfs /dev/drbd0
mkdir /mnt/drbd0
mount /dev/drbd0 /mnt/drbd0
rsync -aW /var/cache/apt/archives/. /mnt/drbd0/.
dmesg | tail ; cat /proc/drbd

Heartbeat on Dom0

For my experiences with Heartbeat-2 and its CRM see Heartbeat-2 on Dom0.

Install

I'm using heartbeat-2 because heartbeat versions 1.2.3 and 1.2.3.cvs.20050927 have a bug where it will constantly restart if you are using ping nodes and bcast. This bug is gone from v2 (though no one knows why).

Grab the latest Ultra Monkey heartbeat-2 backport for Debian Sarge.

wget [1]

Install it and then run aptitude to install the dependencies.

dpkg -i heartbeat-2_2.0.2-4bpo1_i386.deb
aptitude

Configure

There are three files you must create in /etc/ha.d:

authkeys -- heartbeat authentication
ha.cf -- heartbeat configuration
haresources -- resource configuration

Duplicates of these files need to appear on both heartbeat nodes.

I'm not going to explain all the options. There are plenty of resources explaining these files. See the Linux-HA homepage as a starting place. For reference, I'm pasting my versions of these files below (with hosts and IPs made generic). Hopefully they will help you get up and running quickly.

The /etc/ha.d/authkeys file should be owned by root and chmod 400.

# /etc/ha.d/authkeys
auth 1
1 sha1 SuperSecretKey

# /etc/ha.d/ha.cf

use_logd    yes

keepalive   2
warntime    5
deadtime    10
initdead    90

bcast           eth0 eth1
auto_failback   off
node            node1
node            node2

deadping 30
ping_group group1 10.1.2.3 10.4.5.6
respawn hacluster /usr/lib/heartbeat/ipfail

# /etc/ha.d/haresources
xena IPaddr2::192.168.1.100/26/eth0:100 drbddisk::r0 LVM::vg0 xendomains::domU

Additionally, since I'm using the heartbeat logging daemon, I created a configuration for it as well (it is not mandatory).

# /etc/logd.cf: heartbeat logging daemon
debugfile   /var/log/ha-debug.log
logfile     /var/log/ha.log
entity ha_logd

Lastly, to stop heartbeat bitching about core files:

sysctl -w kernel.core_uses_pid=1
cat >> /etc/sysctl.conf
kernel.core_uses_pid = 1
EOF

Test

The best way to test heartbeat is to start with just the IPaddr2 resource in the haresources files. Start up heartbeat, and then restart heartbeat on the machine owning the resources to transfer it over to the other.

node1: /etc/init.d/heartbeat start
node2: /etc/init.d/heartbeat start
node1: ifconfig
node1: /etc/init.d/heartbeat restart
node2: ifconfig

If you see the IP coming up and moving over, then heartbeat is working. Check the logs to make sure everything is kosher.

Test your resources running the scripts in /etc/ha.d/resource.d manually. For example to test a DRBD resource you would say something like:

cat /proc/drbd
/etc/ha.d/resource.d/drbddisk r0 start 
cat /proc/drbd
/etc/ha.d/resource.d/drbddisk r0 status
/etc/ha.d/resource.d/drbddisk r0 stop
cat /proc/drbd

First DomU

Setup the storage:

lvcreate -L 384M -n domU_swap xen
lvcreate -L 2048M -n domU_root xen
mkswap /dev/mapper/xen-domU_swap
mkfs.reiserfs /dev/mapper/xen-domU_root

Mount target and installation media:

mkdir /mnt/target
mount /dev/mapper/xen-domU_root /mnt/target
mkdir /mnt/netinst
mount -o loop /root/debian-31r0a-i386-netinst.iso /mnt/netinst

Install Debian Sarge:

aptitude install debootstrap
debootstrap sarge /mnt/target file:///mnt/netinst

Move tls libraries:

cd /mnt/target
mv /lib/tls /lib/tls.disabled

Edit files in /mnt/target/etc:

# /etc/hosts: static table lookup for host names
#
127.0.0.1     localhost.localdomain  localhost
10.1.2.3      domu

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

# /etc/fstab: static file system information.
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
/dev/sda1   /       reiserfs    defaults        0   1
/dev/sda2   swap    swap        defaults        0   0
proc        /proc   proc        defaults        0   0

hostname
network/interfaces
resolv.conf
apt/sources.list
mailname
postfix/main.cf

If you need any extra packages before booting (such as reiserfsprog or xfsprogs) you can chroot /mnt/target and then apt-get install them.

unmount the target filesystem

umount /mnt/target

create a xen config

# /etc/xen/dom
name   = "domU"
memory = 128
disk   = [ 'phy:/dev/mapper/xen-domU_root,sda1,w', 'phy:/dev/mapper/xen-domU_swap,sda2,w' ]
root   = "/dev/sda1 ro"
kernel = "/boot/vmlinuz-2.6-xenU"

boot into your new system

xm create -c domU

this should create a console for you and boot into the system. to exit the console press ctrl-].

login as root (no password) and update the system

nano apt/sources.list
aptitude update

if you are upgrading to etch from sarge, the newer apt uses gpg authentication. do the following to avoid some annoying warning messages about missing keys:

aptitude install gnupg
aptitude install apt
wget http://secure-testing-master.debian.net/ziyi-2005-7.asc
apt-key add ziyi-2005-7.asc
rm ziyi-2005-7.asc
aptitude update

upgrade the system

aptitude upgrade

move lib/tls again if a new lib6 was installed

rm -rf /lib/tls.disabled
mv /lib/tls /lib/tls.disabled

reboot and configure

reboot
base-config

Another DomU

I keep two pristine VMs called "etch" and "sarge" which serve as templates for getting a new VM up and running. Below is example steps for setting up a new sarge VM.

Create swap and storage.

lvcreate -L 500M -n domU_root vg0
lvcreate -L 2G   -n domU_var  vg0
lvcreate -L 384M -n domU_swap vg0
mkreiserfs /dev/mapper/vg0-domU_root
mkreiserfs /dev/mapper/vg0-domU_var
mkswap     /dev/mapper/vg0-domU_swap

Mount and copy.

mkdir /mnt/sarge
mkdir /mnt/domU
mount /dev/mapper/vg0-sarge     /mnt/sarge
mount /dev/mapper/vg0-domU_root /mnt/domU
mkdir /mnt/domU/var
mount /dev/mapper/vg0-domU-var  /mnt/domU/var
rsync -a --numeric-ids /mnt/sarge/. /mnt/domU/.

Configure the new system by modifying the following files:

etc/hostname
etc/hosts
etc/network/interfaces
etc/mailname
etc/fstab

Create/Edit a new xen configuration file.

cp /etc/xen/sarge /etc/xen/domU
edit /etc/xen/domU

Unmount it and you can boot in.

umount /mnt/domU/usr
umount /mnt/domU
xm create -c newvm

It's a good idea to update the packages list and make sure you have all the recent security updates installed.

Xen on Debian

Contents

Dom0

Building

Installing

DRBD on Dom0

Warning

Build/Install

Configure

Heartbeat on Dom0

Install

Configure

Test

First DomU

Another DomU

Navigation menu

Search