Installation notes working with UEC / Eucalyptus

We just bought a couple new servers on which we were going to run a number of XEN virtual machines. After checking out Eucalyptus and Ubuntu's UEC (Ubuntu Enterprise Cloud), I decided to give that a shot instead - after all being able to automatically launch instances from a EC2-like command interface as well as being able to take snapshots and have my own personal S3 store seemed pretty darn nice.

Getting the UEC CloudController and NodeController's up and running turned out to be pretty easy - The UEC docs , while they're aren't that many of them - are pretty good. I ended up doing an install from both the CD and from packages and both went fine.

Where I ran into some problems was in actually deploying and running the provided Debian Lenny image (I'm guessing straight Debian is less popular than running Ubuntu, but I like the slow pace of updates Debian) that I thought I'd document here in case they are of use to other people getting started. I'm sure some of these are already fixed upstream or in the latest update, but here are my notes from deploying early last month:

Root image size

 Root eucalyptus image too small for what you're doing? Debian lenny image wasn't big enough for all the packages I need to install. Make a bigger image (run this on your box and upload the bundle to your CC):

# create a new file 5 GB large
$ dd if=/dev/zero of=debian-large.5-0.x86.img bs=1k count=5000000
5000000+0 records in
5000000+0 records out
5120000000 bytes (5.1 GB) copied, 92.4922 s, 55.4 MB/s

# copy the current image to the new file
$ dd if=debian.5-0.x86.img of=debian-large.5-0.x86.img conv=notrunc
2050048+0 records in
2050048+0 records out
1049624576 bytes (1.0 GB) copied, 86.6303 s, 12.1 MB/s

# setup loopback device
$ sudo losetup /dev/loop1 debian-large.5-0.x86.img
$ sudo e2fsck -f /dev/loop1

e2fsck 1.41.9 (22-Aug-2009)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/loop1: 20021/128256 files (0.6% non-contiguous), 127180/256256 blocks

# resize the file system to the new size
$ sudo resize2fs /dev/loop1
resize2fs 1.41.9 (22-Aug-2009)
Resizing the filesystem on /dev/loop1 to 1250000 (4k) blocks.
The filesystem on /dev/loop1 is now 1250000 blocks long.

# Now rebundle and uploaded (linking to existing kernel images):
$ euca-bundle-image -i debian-large.5-0.x86.img  --kernel eki-CF9C1839 --ramdisk eri-1BBF191F

Debian lenny image problems

(All these fixes need to be done from inside of the image after the image is run for the first time)

 /tmp isn't writable (not a eucalpytus problem - but this causes a bunch of problems esp. w/ mysql install)

 chmod a+w /tmp

DNS server is messed up on the VM's - add your DNS manually to   /etc/dhcp3/dhclient.conf by adding the line (near the top)

 prepend domain-name-servers DNS-SERVER-IP;

(Don't forget the semicolon) Where DNS-SERVER-IP is your nameserver, then run  /etc/init.d/networking restart to restart networking

Mounted file system problems - no swap or transient storage, run the following:

swapon /dev/sda3
fdisk -l
mke2fs -t ext3 /dev/sda2
mount /dev/sda2 /mnt

I added the following to /etc/fstab to make the mounts survive a reboot:

/dev/sda2        /mnt          ext3     defaults                   0 0
/dev/sda3        none          swap     sw                         0 0

nosegneg issue - receive an error message from apt upgrade like:

 ldconfig: /etc/ld.so.conf.d/libc6-xen.conf:6: hwcap index 0 already defined as nosegneg
dpkg: error processing libc6 (--configure):
 subprocess post-installation script returned error exit status 1
Errors were encountered while processing:
 libc6

edit  /etc/ld.so.conf.d/libc6-xen.conf and change 1 to 0