Version 2 (modified by yxin, 11 years ago)


== Eucalyptus Notes (from Anirban)==

Eucalyptus Installation notes for the doc cluster @RENCI

Following is a step-by-step guide to install, configure and deploy Eucalyptus, the public cloud-computing infrastructure ( from source. System

Head node: - uname -a Linux 2.6.9-34.0.2.EL #1 Fri Jul 7 17:58:49 CDT 2006 x86_64 x86_64 x86_64 GNU/Linux Compute node: - uname -a Linux compute-0-1.local 2.6.18-xen #1 SMP Fri May 18 16:01:42 BST 2007 x86_64 x86_64 x86_64 GNU/Linux

Rocks 4.2 with RHEL 4. Each compute node has a dual core AMD Opteron (1.9 GHz) and 2GB memory.

GNU C/C++ compilers exist.

Prerequisites Java update

Need jdk-1.6 for head node. Downloaded jdk 6 update 7 from cp jdk-6u7-linux-x64.bin /usr/java/. To install java in /usr/java do ./jdk-6u7-linux-x64.bin Update /etc/profile.d/ and /etc/profile.d/java.csh to reflect 1.6 version

Java Unlimited strength policy files:

Download unlimited strength policy files from

(Look for other downloads at the bottom: Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files 6)

Unpack the zip file in $JAVA_HOME/lib/security and $JAVA_HOME/jre/lib/security (Move the old security files to a different name; Basically, we need the following files in there - local_policy.jar , US_export_policy.jar and cacerts (which was already there).


Downloaded maven from mkdir /usr/loca/apache-maven cp apache-maven-2.0.9-bin.tar.gz /usr/local/apache-maven/ cd /usr/local/apache-maven tar -zxvf apache-maven-2.0.9-bin.tar.gz Create and apache-maven.csh in /etc/profile.d : - export M2_HOME=/usr/local/apache-maven/apache-maven-2.0.9 export M2=$M2_HOME/bin export PATH=$M2:$PATH

apache-maven.csh - setenv M2_HOME /usr/local/apache-maven/apache-maven-2.0.9 setenv M2 /usr/local/apache-maven/apache-maven-2.0.9/bin set path = ( $M2 $path)


Exists in /opt/local/bin/ant


Xen on compute nodes:

Need brctl as a prerequisite for Xen –

Download bridge-utils-1.0.4-4.x86_64.rpm sudo rpm -ivh bridge-utils-1.0.4-4.x86_64.rpm

Download xen-3.1.0-install-x86_64.tgz mkdir Xen; cd Xen; tar -zxvf xen-3.1.0-install-x86_64.tgz cd dist; sudo ./

This will install necessary Xen files in different directories including /boot

cd /boot; sudo /sbin/depmod -a 2.6.18-xen sudo /sbin/mkinitrd /boot/initrd-2.6.18-xen.img 2.6.18-xen

This will create a initrd-XXX.img in the /boot directory

Modify /etc/grub.conf:

sudo chmod 644 /etc/grub.conf

Add the following to /etc/grub.conf (Make sure to boot with it by default)

title Xen3.1-RHEL4 (2.6.18-xen)

root (hd0,0) kernel /boot/xen-3.1.gz module /boot/vmlinuz-2.6.18-xen ro root=LABEL=/ module /boot/initrd-2.6.18-xen.img


To get "xend" and "xm" commands in PATH export PATH=/usr/sbin:$PATH

Create symbolic link for by doing cd /lib64; sudo ln -s /lib64/ /lib64/

Start xend daemon sudo /etc/init.d/xend start

xm list This will list Dom0 and other things now

Xen for head node:

The Xen header files were required on the head node because of libvirt's dependency on the Xen header files.

So, I installed Xen on the head node. Alternatively, only installing xen-devel would suffice.

Made the following change in /usr/include/xen/xen.h


This was required because the libvirt library uses this header file and fails to compile without this change. Primarily this error occurs because of how macros and #defines work for a specific compiler. Under buggy xen.h, DEFINE_XEN_GUEST_HANDLE(<>) was expanding to something which was syntactically wrong.

/* Turn a plain number into a C unsigned long constant. */ #define mk_unsigned_long(x) x ## UL #define mk_unsigned_long(x) mk_unsigned_long(x) /* DEFINE_XEN_GUEST_HANDLE(uint8_t); DEFINE_XEN_GUEST_HANDLE(uint16_t); DEFINE_XEN_GUEST_HANDLE(uint32_t); DEFINE_XEN_GUEST_HANDLE(uint64_t); */ DEFINE_XEN_GUEST_HANDLE(uint8_t, uint8_t); DEFINE_XEN_GUEST_HANDLE(uint16_t, uint16_t); DEFINE_XEN_GUEST_HANDLE(uint32_t, uint32_t); DEFINE_XEN_GUEST_HANDLE(uint64_t, uint64_t);

Eucalyptus Installation

Download Eucalyptus and set some environment vars:

Download eucalyptus-1.3-src.tar.gz and eucalyptus-1.3-src-deps.tar.gz from

tar zxvf eucalyptus-1.3-src.tar.gz cd eucalyptus-1.3-src tar zxvf ../eucalyptus-1.3-src-deps.tar.gz

Edit Makedefs and also add in .bashrc the following:

export EUCALYPTUS_SRC=/home/anirban/Euca/src/eucalyptus-1.3-src export EUCALYPTUS=/opt/eucalyptus export AXIS2_HOME=${EUCALYPTUS}/packages/axis2-1.4 export AXIS2C_HOME=${EUCALYPTUS}/packages/axis2c-1.4.0 export LIBVIRT_HOME=${EUCALYPTUS}/packages/libvirt-0.4.2 export GWT_HOME=${EUCALYPTUS}/packages/gwt-1.4.62

source Makedefs mkdir -p $EUCALYPTUS/packages/

Compile dependencies: (mostly, as mentioned on the webpage except libvirt)

1. GWT

cd $EUCALYPTUS/packages tar zvxf $EUCALYPTUS_SRC/eucalyptus-src-deps/gwt-1.4.62.tar.gz

2. Axis2

cd $EUCALYPTUS/packages unzip $EUCALYPTUS_SRC/eucalyptus-src-deps/

3. Apache (httpd)

cd $EUCALYPTUS_SRC/eucalyptus-src-deps tar zvxf httpd-2.2.8.tar.gz cd httpd-2.2.8 CFLAGS="-DBIG_SECURITY_HOLE" ./configure --prefix=$EUCALYPTUS/packages/httpd-2.2.8 make ; make install

4. Axis2/C

cd $EUCALYPTUS_SRC/eucalyptus-src-deps tar zvxf axis2c-src-1.4.0.tar.gz cd axis2c-src-1.4.0 ./configure --with-apache2=$EUCALYPTUS/packages/httpd-2.2.8/include --prefix=${AXIS2C_HOME} make ; make install

5. Libvirt

Try these steps to see if it works:

cd $EUCALYPTUS_SRC/eucalyptus-src-deps tar zvxf libvirt-0.4.2.tar.gz cd libvirt-0.4.2 ./configure --prefix=$EUCALYPTUS/packages/libvirt-0.4.2 --without-storage-disk make ; make install

These failed for me for the following dependencies. Download and compile following in this order:

a. Xen header files and libxenstore (so install Xen as described earlier) b. libtasn1

c. libgpg-error-1.6

d. libgcrypt-1.4.1

e. gnupg-1.4.9

f. gnutls-2.5.4

Now, try cd $EUCALYPTUS_SRC/eucalyptus-src-deps cd libvirt-0.4.2 ./configure --prefix=$EUCALYPTUS/packages/libvirt-0.4.2 --without-storage-disk make ; make install

6. Rampart/C

cd $EUCALYPTUS_SRC/eucalyptus-src-deps tar zvxf rampartc-src-1.2.0.tar.gz cd rampartc-src-1.2.0 export LD_LIBRARY_PATH=${AXIS2C_HOME}/lib:$LD_LIBRARY_PATH ./configure --prefix=${AXIS2C_HOME} --enable-static=no --with-axis2=${AXIS2C_HOME}/include/axis2-1.4.0 make ; make install

7. VDE

cd $EUCALYPTUS_SRC/eucalyptus-src-deps tar zvxf vde2-2.2.2.tar.gz cd vde2-2.2.2 ./configure --prefix=$EUCALYPTUS/packages/vde2-2.2.2 --enable-cryptcab --enable-experimental make ; make install

Put this in your .bashrc:

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib

Mirror /usr/local/lib on the compute nodes (scp /usr/local/lib/* root@compute-0-1:/usr/local/lib/.)

Build Eucalyptus: (as in webpage)

cd $EUCALYPTUS_SRC source Makedefs cd clc ant clean deps build jce setup

cd $EUCALYPTUS_SRC make build make deploy

Eucalyptus Setup

source $EUCALYPTUS_SRC/Makedefs

Axis2/C configuration

Edit $EUCALYPTUS/packages/axis2c-1.4.0/axis2.xml by changing <!--phase name="Security"/--> to <phase name="Security"/>

Apache2 configuration

cp $EUCALYPTUS/packages/axis2c-1.4.0/lib/ $EUCALYPTUS/packages/httpd-2.2.8/modules/

cp $EUCALYPTUS_SRC/tools/httpd.conf $EUCALYPTUS/etc/eucalyptus/

Edit $EUCALYPTUS/etc/eucalyptus/httpd.conf

a. LoadModule? axis2_module modules/ Axis2RepoPath /opt/eucalyptus/packages/axis2c-1.4.0 Axis2LogFile /opt/eucalyptus/packages/axis2c-1.4.0/logs/axis2.log Axis2MaxLogFileSize 128 Axis2LogLevel trace <Location /axis2>

SetHandler? axis2_module


b. <Directory />

Order deny,allow Allow from all Deny from all


c. PidFile? /opt/eucalyptus/var/run/eucalyptus/

d. ServerRoot? "/opt/eucalyptus/packages/httpd-2.2.8"

e. Listen 9090 for head node and Listen 9091 for compute node (make this change after we copy over everything in /opt/eucalyptus on head node to /opt/eucalyptus on the compute node)

Keys, directories, paths

cd $EUCALYPTUS_SRC ssh-keygen -t rsa1 -b 768 -f $EUCALYPTUS/var/eucalyptus/keys/vdekey -N

mkdir -p /etc/default mkdir -p $EUCALYPTUS/etc/init.d mkdir -p $EUCALYPTUS/bin mkdir -p $EUCALYPTUS/lib mkdir -p $EUCALYPTUS/usr/sbin mkdir -p $EUCALYPTUS/var/run/eucalyptus

cp $EUCALYPTUS_SRC/tools/eucalyptus.conf $EUCALYPTUS/etc/eucalyptus/ cp $EUCALYPTUS_SRC/tools/ $EUCALYPTUS/usr/share/eucalyptus/ cp $EUCALYPTUS_SRC/tools/euca_conf $EUCALYPTUS/usr/sbin/ cp $EUCALYPTUS_SRC/tools/euca $EUCALYPTUS/usr/sbin/ cp $EUCALYPTUS_SRC/tools/euca_sync_key $EUCALYPTUS/usr/sbin/ cp $EUCALYPTUS_SRC/tools/eucalyptus $EUCALYPTUS/etc/init.d/ cp $EUCALYPTUS_SRC/clc/cloud.xml $EUCALYPTUS/etc/eucalyptus/cloud-ant.xml

ln -sf $EUCALYPTUS/etc/eucalyptus/eucalyptus.conf /etc/default/eucalyptus ln -sf $EUCALYPTUS/packages/*/bin/* $EUCALYPTUS/bin/ ln -sf $EUCALYPTUS/packages/*/lib/* $EUCALYPTUS/lib/

Edit .bashrc :

export PATH=/opt/eucalyptus/bin:$PATH:/sbin:/usr/sbin export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/eucalyptus/lib

$EUCALYPTUS/usr/sbin/euca_conf -d $EUCALYPTUS $EUCALYPTUS/usr/sbin/euca_conf -ccp 9090 -ncp 9091

Now, mirror the /opt/eucalyptus tree to the compute node:

scp -r /opt/eucalyptus/* root@compute-0-1:/opt/eucalyptus/.

On the head node: $EUCALYPTUS/usr/sbin/euca_conf -cloud Y $EUCALYPTUS/usr/sbin/euca_conf -cc Y

On the compute node: $EUCALYPTUS/usr/sbin/euca_conf -nc Y

Eucalyptus Configuration

On both head and compute node, do

export NFS_SHARE=/home/anirban/Euca-images export LOCAL_IMAGES=/home/anirban/Euca-local-images

Ideally, LOCAL_IMAGES should be /usr/local/eucalyptus ; but since we didn't have enough disk space in /usr/local , we put it in /home/anirban/Euca-local-images

mkdir -p $NFS_SHARE/templates/ mkdir -p $NFS_SHARE/registered/ mkdir -p $LOCAL_IMAGES/instances/

$EUCALYPTUS/usr/sbin/euca_conf -templates $NFS_SHARE/templates $EUCALYPTUS/usr/sbin/euca_conf -registered $NFS_SHARE/registered $EUCALYPTUS/usr/sbin/euca_conf -instances $LOCAL_IMAGES/instances

On the front end, do

$EUCALYPTUS/usr/sbin/euca_conf -nodes "" $EUCALYPTUS/usr/sbin/euca_conf -setup $EUCALYPTUS/usr/sbin/euca_sync_key -f -v

Configure Xen with networking

Edit /etc/xen/xend-config.sxp on compute node to enable unix-server - Uncomment and put yes in the line which says xend-unix-server.

(xend-unix-server yes) Network bridging

Ideally, the following should work for network bridging for the given system.

Have a line in /etc/xen/xend-config.sxp in the section for network-script

(network-script 'network-bridge bridge=xenbr1 netdev=eth1')

Unfortunately, this didn't work for me. I wrote a custom network-script in /etc/xen/scripts/my-network-bridge chmod 700 /etc/xen/scripts/my-network-bridge Include the following in the section for network-bridging in /etc/xen/xend-config.sxp (network-script my-network-bridge)

These are the contents of my-network-bridge #------------ #!/bin/bash

echo "Start setting up xenbr1" >> /home/anirban/tmp/log-xen-bridge

ifdown eth1 ip link set eth1 down arp off brctl addbr xenbr1 ip link set xenbr1 arp on ip link set xenbr1 multicast off ifconfig xenbr1 broadcast netmask up brctl addif xenbr1 eth1 ip link set eth1 up arp off

echo "Done setting up xenbr1" >> /home/anirban/tmp/log-xen-bridge #-----------------

I also had to set forwarding to ACCEPT in iptables for the compute node by editing /etc/sysconfig/iptables - change :FORWARD DROP [0:0] to :FORWARD ACCEPT [0:0] This is equivalent to manually doing 'iptables -P FORWARD ACCEPT' Test Xen with networking

Testing Xen with ttylinux - make sure that Xen works with ttylinux and networking on the compute node. Have the images in /home/anirban/tmp/ttylinux-image/ttylinux Need ttylinux.img vmlinuz-2.6.18-xen and ttylinux.conf ttylinux.conf has:

kernel="vmlinuz-2.6.18-xen" disk=[ 'file:/home/anirban/tmp/ttylinux-image/ttylinux/ttylinux.img,sda1,w' ] vif=[ 'mac=AA:DD:11:CE:FE:AA, bridge=xenbr1' ] root="/dev/sda1 ro" extra="4" name="tty"

Issue 'xm create -c ttylinux.conf' This should boot up ttylinux. Login with root , root. Check whether the VM gets an IP using dhcp by doing 'ifconfig -a'. Check if you can ping this IP from host, head node and other machines. Also verify whether you can ping host, head node and other machines from inside VM. Try to scp a file from VM to somewhere else and the opposite. If all these succeed, Xen is working with networking. 'xm list' should list this new domain. To shutdown ttylinux, do 'xm shutdown tty'

Eucalyptus Deployment

Start Xen on compute node

xend start

'xm list' should display domain 0. 'brctl show' should show eth1 attached to xenbr1 Verify interfaces using 'ifconfig -a' Verify iptables using 'iptables -L'

Start Eucalyptus

Start Eucalyptus Cloud, CC (Cluster controller) and NC (Node Controller)

On head node: To start cluster controller: $EUCALYPTUS/bin/httpd -f $EUCALYPTUS/etc/eucalyptus/httpd.conf To start cloud: ant -f $EUCALYPTUS/etc/eucalyptus/cloud-ant.xml run

On compute node: To start node controller: $EUCALYPTUS/bin/httpd -f $EUCALYPTUS/etc/eucalyptus/httpd.conf

Open up web browser to access the cloud

Image management

Adding a new image - go to the image directory (eg. ttylinux)

${EUCALYPTUS}/usr/sbin/euca add_image --disk-image ttylinux.img --kernel-image vmlinuz-2.6.18-xen --image-name ttylinuxWorking

The default xen bridge in the config.xml created is xenbr0. Since, we are using xenbr1, we have to change xenbr0 to xenbr1in config.xml in $NFS_SHARE/registered/eucalyptus/<emi-id> for all registered images and also cache ( $LOCAL_IMAGES/instances/eucalyptus/cache), if there was an attempt to run that image before.

User guide

User Guide @ for a. creating accounts, b. generating and storing the keys/certs, c. installing EC2 command-line tools, d.setting up environment variables to use ec2 comand line tools.

Important EC2 commands: ec2-describe-availability-zones, ec2-describe-images, ec2-run-instances <emi-id>, ec2-describe-instances, ec2-terminate-instances <instance-id>


On the compute node, fixed the stat command and put a 'stat' file in /opt/eucalyptus/bin (because of incompatibilities with old version of stat on the machine (compute node). stat contents: #------- #!/bin/bash

ARGS=echo $@ | sed "s/file-system/filesystem/g" ARGS=echo $ARGS | sed "s/%S/%s/g" /usr/bin/stat $ARGS #--------

Don't need vde on the head node. So, moved back vde from /opt/eucalyptus/packages/vde2-2.2.2 to ~anirban/vde2-2.2.2; Added 4 scripts (vdecmd, vde_cryptcab, vde_plug2tap, vde_switch) in /opt/eucalyptus/bin to return 0 (#!/bin/bash exit 0;)