Changes between Version 1 and Version 2 of GoEucalyptus

11/13/08 22:34:19 (11 years ago)
yxin (IP:



  • GoEucalyptus

    v1 v2  
    1 == Eucalyptus Notes == 
     1== Eucalyptus Notes (from Anirban)== 
     3Eucalyptus Installation notes for the doc cluster @RENCI 
     5Following is a step-by-step guide to install, configure and deploy Eucalyptus, the public cloud-computing infrastructure ( from source. 
     8Head node: - uname -a 
     9Linux 2.6.9-34.0.2.EL #1 Fri Jul 7 17:58:49 CDT 2006 x86_64 x86_64 x86_64 GNU/Linux 
     10Compute node: - uname -a 
     11Linux compute-0-1.local 2.6.18-xen #1 SMP Fri May 18 16:01:42 BST 2007 x86_64 x86_64 x86_64 GNU/Linux 
     13Rocks 4.2 with RHEL 4. Each compute node has a dual core AMD Opteron (1.9 GHz) and 2GB memory. 
     15GNU C/C++ compilers exist. 
     18Java update 
     20Need jdk-1.6 for head node. 
     21Downloaded jdk 6 update 7 from 
     22cp jdk-6u7-linux-x64.bin /usr/java/. 
     23To install java in /usr/java do ./jdk-6u7-linux-x64.bin 
     24Update /etc/profile.d/ and /etc/profile.d/java.csh to reflect 1.6 version 
     26Java Unlimited strength policy files: 
     28Download unlimited strength policy files from  
     31(Look for other downloads at the bottom: Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files 6) 
     33Unpack the zip file in $JAVA_HOME/lib/security and $JAVA_HOME/jre/lib/security (Move the old security files to a different name; Basically, we need the following files in there - local_policy.jar , US_export_policy.jar and cacerts (which was already there). 
     37Downloaded maven from 
     38mkdir /usr/loca/apache-maven 
     39cp apache-maven-2.0.9-bin.tar.gz /usr/local/apache-maven/ 
     40cd /usr/local/apache-maven 
     41tar -zxvf apache-maven-2.0.9-bin.tar.gz 
     42Create and apache-maven.csh in /etc/profile.d : - 
     44export M2_HOME=/usr/local/apache-maven/apache-maven-2.0.9 
     45export M2=$M2_HOME/bin 
     46export PATH=$M2:$PATH 
     48apache-maven.csh - 
     49setenv  M2_HOME /usr/local/apache-maven/apache-maven-2.0.9 
     50setenv M2 /usr/local/apache-maven/apache-maven-2.0.9/bin 
     51set path = ( $M2 $path) 
     55Exists in /opt/local/bin/ant 
     59Xen on compute nodes: 
     61Need brctl as a prerequisite for Xen – 
     63Download bridge-utils-1.0.4-4.x86_64.rpm 
     64sudo rpm -ivh bridge-utils-1.0.4-4.x86_64.rpm 
     66Download xen-3.1.0-install-x86_64.tgz 
     67mkdir Xen; cd Xen; 
     68tar -zxvf xen-3.1.0-install-x86_64.tgz 
     69cd dist; 
     70sudo ./ 
     72This will install necessary Xen files in different directories including /boot 
     74cd /boot; 
     75sudo /sbin/depmod -a 2.6.18-xen 
     76sudo /sbin/mkinitrd /boot/initrd-2.6.18-xen.img 2.6.18-xen 
     78This will create a initrd-XXX.img in the /boot directory 
     80Modify /etc/grub.conf: 
     82sudo chmod 644 /etc/grub.conf 
     84Add the following to /etc/grub.conf  (Make sure to boot with it by default) 
     86title Xen3.1-RHEL4 (2.6.18-xen) 
     87        root (hd0,0) 
     88        kernel /boot/xen-3.1.gz 
     89        module /boot/vmlinuz-2.6.18-xen ro root=LABEL=/ 
     90        module /boot/initrd-2.6.18-xen.img 
     94To get "xend" and "xm" commands in PATH 
     95export PATH=/usr/sbin:$PATH 
     97Create symbolic link for by doing 
     98cd /lib64; 
     99sudo ln -s /lib64/ /lib64/ 
     101Start xend daemon 
     102sudo /etc/init.d/xend start 
     104xm list 
     105This will list Dom0 and other things now 
     107Xen for head node: 
     109The Xen header files were required on the head node because of libvirt's dependency on the Xen header files. 
     111So, I installed Xen on the head node. Alternatively, only installing xen-devel would suffice. 
     113Made the following change in /usr/include/xen/xen.h 
     115Changed line 578-581 (DEFINE_XEN_GUEST_HANDLE(<>) to __DEFINE_XEN_GUEST_HANDLE(<>)) 
     117This was required because the libvirt library uses this header file and fails to compile without this change. Primarily this error occurs because of how macros and #defines work for a specific compiler. Under buggy xen.h, DEFINE_XEN_GUEST_HANDLE(<>) was expanding to something which was syntactically wrong. 
     119/* Turn a plain number into a C unsigned long constant. */ 
     120#define __mk_unsigned_long(x) x ## UL 
     121#define mk_unsigned_long(x) __mk_unsigned_long(x) 
     128__DEFINE_XEN_GUEST_HANDLE(uint8_t, uint8_t); 
     129__DEFINE_XEN_GUEST_HANDLE(uint16_t, uint16_t); 
     130__DEFINE_XEN_GUEST_HANDLE(uint32_t, uint32_t); 
     131__DEFINE_XEN_GUEST_HANDLE(uint64_t, uint64_t); 
     134Eucalyptus Installation 
     136Download Eucalyptus and set some environment vars: 
     138Download eucalyptus-1.3-src.tar.gz and eucalyptus-1.3-src-deps.tar.gz from  
     141tar zxvf  eucalyptus-1.3-src.tar.gz 
     142cd eucalyptus-1.3-src 
     143tar zxvf ../eucalyptus-1.3-src-deps.tar.gz 
     145Edit Makedefs and also add in .bashrc the following: 
     147export EUCALYPTUS_SRC=/home/anirban/Euca/src/eucalyptus-1.3-src 
     148export EUCALYPTUS=/opt/eucalyptus 
     149export AXIS2_HOME=${EUCALYPTUS}/packages/axis2-1.4 
     150export AXIS2C_HOME=${EUCALYPTUS}/packages/axis2c-1.4.0 
     151export LIBVIRT_HOME=${EUCALYPTUS}/packages/libvirt-0.4.2 
     152export GWT_HOME=${EUCALYPTUS}/packages/gwt-1.4.62 
     154source Makedefs 
     155mkdir -p $EUCALYPTUS/packages/ 
     157Compile dependencies: (mostly, as mentioned on the webpage except libvirt) 
     1591. GWT  
     161cd $EUCALYPTUS/packages 
     162tar zvxf $EUCALYPTUS_SRC/eucalyptus-src-deps/gwt-1.4.62.tar.gz 
     1642. Axis2 
     166cd $EUCALYPTUS/packages 
     167unzip $EUCALYPTUS_SRC/eucalyptus-src-deps/ 
     1693. Apache (httpd) 
     171cd $EUCALYPTUS_SRC/eucalyptus-src-deps 
     172tar zvxf httpd-2.2.8.tar.gz 
     173cd httpd-2.2.8 
     174CFLAGS="-DBIG_SECURITY_HOLE" ./configure --prefix=$EUCALYPTUS/packages/httpd-2.2.8 
     175make ; make install 
     1774. Axis2/C 
     179cd $EUCALYPTUS_SRC/eucalyptus-src-deps 
     180tar zvxf axis2c-src-1.4.0.tar.gz 
     181cd axis2c-src-1.4.0 
     182./configure --with-apache2=$EUCALYPTUS/packages/httpd-2.2.8/include --prefix=${AXIS2C_HOME} 
     183make ; make install 
     1855. Libvirt 
     187Try these steps to see if it works: 
     189cd $EUCALYPTUS_SRC/eucalyptus-src-deps 
     190tar zvxf libvirt-0.4.2.tar.gz 
     191cd libvirt-0.4.2 
     192./configure --prefix=$EUCALYPTUS/packages/libvirt-0.4.2 --without-storage-disk 
     193make ; make install 
     195These failed for me for the following dependencies. Download and compile following in this order: 
     197a. Xen header files and libxenstore (so install Xen as described earlier) 
     198b. libtasn1 
     199        - Download from 
     200        - tar zxvf libtasn1-1.5.tar.gz; cd libtasn1-1.5; ./configure; make; make install 
     201c. libgpg-error-1.6 
     202        - Download from 
     203        - bunzip2 libgpg-error-1.6.tar.bz2; tar xvf libgpg-error-1.6.tar; cd libgpg-error-1.6; ./configure; make; make install 
     204d. libgcrypt-1.4.1 
     205        - Download from 
     206        - bunzip2 libgcrypt-1.4.1.tar.bz2; tar xvf libgcrypt-1.4.1.tar; cd libgcrypt-1.4.1; ./configure; make; make install 
     207e. gnupg-1.4.9 
     208        - Download from 
     209        - bunzip2 gnupg-1.4.9.tar.bz2; tar xvf gnupg-1.4.9.tar; cd gnupg-1.4.9; ./configure; make; make install 
     210f. gnutls-2.5.4 
     211        - Download from 
     212        - bunzip2 gnutls-2.5.4.tar.bz2; tar xvf gnutls-2.5.4.tar; cd gnutls-2.5.4; ./configure; make; make install 
     214Now, try  
     215cd $EUCALYPTUS_SRC/eucalyptus-src-deps 
     216cd libvirt-0.4.2 
     217./configure --prefix=$EUCALYPTUS/packages/libvirt-0.4.2 --without-storage-disk 
     218make ; make install 
     2206. Rampart/C 
     222cd $EUCALYPTUS_SRC/eucalyptus-src-deps 
     223tar zvxf rampartc-src-1.2.0.tar.gz 
     224cd rampartc-src-1.2.0 
     226./configure --prefix=${AXIS2C_HOME} --enable-static=no --with-axis2=${AXIS2C_HOME}/include/axis2-1.4.0 
     227make ; make install 
     2297. VDE 
     231cd $EUCALYPTUS_SRC/eucalyptus-src-deps 
     232tar zvxf vde2-2.2.2.tar.gz 
     233cd vde2-2.2.2 
     234./configure --prefix=$EUCALYPTUS/packages/vde2-2.2.2 --enable-cryptcab --enable-experimental 
     235make ; make install 
     237Put this in your .bashrc: 
     239export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib 
     241Mirror /usr/local/lib on the compute nodes (scp /usr/local/lib/* root@compute-0-1:/usr/local/lib/.) 
     244Build Eucalyptus: (as in webpage) 
     246cd $EUCALYPTUS_SRC 
     247source Makedefs 
     248cd clc 
     249ant clean deps build jce setup 
     251cd $EUCALYPTUS_SRC 
     252make build 
     253make deploy 
     255Eucalyptus Setup 
     257source $EUCALYPTUS_SRC/Makedefs 
     259Axis2/C configuration 
     261Edit $EUCALYPTUS/packages/axis2c-1.4.0/axis2.xml by changing <!--phase name="Security"/--> to <phase name="Security"/> 
     263Apache2 configuration 
     265cp $EUCALYPTUS/packages/axis2c-1.4.0/lib/ $EUCALYPTUS/packages/httpd-2.2.8/modules/ 
     267cp $EUCALYPTUS_SRC/tools/httpd.conf $EUCALYPTUS/etc/eucalyptus/ 
     269Edit $EUCALYPTUS/etc/eucalyptus/httpd.conf 
     272LoadModule axis2_module modules/ 
     273Axis2RepoPath /opt/eucalyptus/packages/axis2c-1.4.0 
     274Axis2LogFile /opt/eucalyptus/packages/axis2c-1.4.0/logs/axis2.log 
     275Axis2MaxLogFileSize 128 
     276Axis2LogLevel trace 
     277<Location /axis2> 
     278        SetHandler axis2_module 
     282<Directory /> 
     283        Order deny,allow 
     284        Allow from all 
     285        Deny from all 
     289PidFile /opt/eucalyptus/var/run/eucalyptus/ 
     292ServerRoot "/opt/eucalyptus/packages/httpd-2.2.8" 
     295Listen 9090 for head node and Listen 9091 for compute node (make this change after we copy over everything in /opt/eucalyptus on head node to /opt/eucalyptus on the compute node) 
     297Keys, directories, paths  
     299cd $EUCALYPTUS_SRC 
     300ssh-keygen -t rsa1 -b 768 -f $EUCALYPTUS/var/eucalyptus/keys/vdekey -N '' 
     302mkdir -p /etc/default 
     303mkdir -p $EUCALYPTUS/etc/init.d 
     304mkdir -p $EUCALYPTUS/bin 
     305mkdir -p $EUCALYPTUS/lib 
     306mkdir -p $EUCALYPTUS/usr/sbin 
     307mkdir -p $EUCALYPTUS/var/run/eucalyptus 
     309cp $EUCALYPTUS_SRC/tools/eucalyptus.conf $EUCALYPTUS/etc/eucalyptus/ 
     310cp $EUCALYPTUS_SRC/tools/ $EUCALYPTUS/usr/share/eucalyptus/ 
     311cp $EUCALYPTUS_SRC/tools/euca_conf $EUCALYPTUS/usr/sbin/ 
     312cp $EUCALYPTUS_SRC/tools/euca $EUCALYPTUS/usr/sbin/ 
     313cp $EUCALYPTUS_SRC/tools/euca_sync_key $EUCALYPTUS/usr/sbin/ 
     314cp $EUCALYPTUS_SRC/tools/eucalyptus $EUCALYPTUS/etc/init.d/ 
     315cp $EUCALYPTUS_SRC/clc/cloud.xml $EUCALYPTUS/etc/eucalyptus/cloud-ant.xml 
     317ln -sf $EUCALYPTUS/etc/eucalyptus/eucalyptus.conf /etc/default/eucalyptus 
     318ln -sf $EUCALYPTUS/packages/*/bin/* $EUCALYPTUS/bin/ 
     319ln -sf $EUCALYPTUS/packages/*/lib/* $EUCALYPTUS/lib/ 
     321Edit .bashrc : 
     323export PATH=/opt/eucalyptus/bin:$PATH:/sbin:/usr/sbin 
     324export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/eucalyptus/lib 
     326$EUCALYPTUS/usr/sbin/euca_conf -d $EUCALYPTUS 
     327$EUCALYPTUS/usr/sbin/euca_conf -ccp 9090 -ncp 9091 
     329Now, mirror the /opt/eucalyptus tree to the compute node: 
     331scp -r /opt/eucalyptus/* root@compute-0-1:/opt/eucalyptus/. 
     333On the head node: 
     334$EUCALYPTUS/usr/sbin/euca_conf -cloud Y 
     335$EUCALYPTUS/usr/sbin/euca_conf -cc Y 
     337On the compute node: 
     338$EUCALYPTUS/usr/sbin/euca_conf -nc Y 
     340Eucalyptus Configuration 
     342On both head and compute node, do 
     344export NFS_SHARE=/home/anirban/Euca-images 
     345export LOCAL_IMAGES=/home/anirban/Euca-local-images 
     347Ideally, LOCAL_IMAGES should be /usr/local/eucalyptus ; but since we didn't have enough disk space in /usr/local , we put it in /home/anirban/Euca-local-images 
     349mkdir -p $NFS_SHARE/templates/ 
     350mkdir -p $NFS_SHARE/registered/ 
     351mkdir -p $LOCAL_IMAGES/instances/ 
     353$EUCALYPTUS/usr/sbin/euca_conf -templates $NFS_SHARE/templates 
     354$EUCALYPTUS/usr/sbin/euca_conf -registered $NFS_SHARE/registered 
     355$EUCALYPTUS/usr/sbin/euca_conf -instances $LOCAL_IMAGES/instances 
     357On the front end, do 
     359$EUCALYPTUS/usr/sbin/euca_conf -nodes "" 
     360$EUCALYPTUS/usr/sbin/euca_conf -setup 
     361$EUCALYPTUS/usr/sbin/euca_sync_key -f -v 
     364Configure Xen with networking 
     366Edit /etc/xen/xend-config.sxp on compute node to enable unix-server - Uncomment and put yes in the line which says xend-unix-server. 
     368(xend-unix-server yes) 
     369Network bridging  
     371Ideally, the following should work for network bridging for the given system. 
     373Have a line in /etc/xen/xend-config.sxp in the section for network-script  
     375(network-script 'network-bridge bridge=xenbr1 netdev=eth1')  
     377Unfortunately, this didn't work for me. I wrote a custom network-script in /etc/xen/scripts/my-network-bridge 
     378chmod 700 /etc/xen/scripts/my-network-bridge 
     379Include the following in the section for network-bridging in /etc/xen/xend-config.sxp 
     380(network-script my-network-bridge) 
     382These are the contents of my-network-bridge 
     386echo "Start setting up xenbr1" >> /home/anirban/tmp/log-xen-bridge 
     388ifdown eth1 
     389ip link set eth1 down arp off 
     390brctl addbr xenbr1 
     391ip link set xenbr1 arp on 
     392ip link set xenbr1 multicast off 
     393ifconfig xenbr1 broadcast netmask up 
     394brctl addif xenbr1 eth1 
     395ip link set eth1 up arp off 
     397echo "Done setting up xenbr1" >> /home/anirban/tmp/log-xen-bridge 
     400I also had to set forwarding to ACCEPT in iptables for the compute node by editing /etc/sysconfig/iptables - change :FORWARD DROP [0:0] to :FORWARD ACCEPT [0:0] 
     401This is equivalent to manually doing 'iptables -P FORWARD ACCEPT' 
     402Test Xen with networking 
     404Testing Xen with ttylinux - make sure that Xen works with ttylinux and networking on the compute node. 
     405Have the images in /home/anirban/tmp/ttylinux-image/ttylinux 
     406Need ttylinux.img vmlinuz-2.6.18-xen and ttylinux.conf 
     407ttylinux.conf has: 
     410disk=[ 'file:/home/anirban/tmp/ttylinux-image/ttylinux/ttylinux.img,sda1,w' ] 
     411vif=[ 'mac=AA:DD:11:CE:FE:AA, bridge=xenbr1' ] 
     412root="/dev/sda1 ro" 
     416Issue 'xm create -c ttylinux.conf' 
     417This should boot up ttylinux. Login with root , root. Check whether the VM gets an IP using dhcp by doing 'ifconfig -a'. Check if you can ping this IP from host, head node and other machines. Also verify whether you can ping host, head node and other machines from inside VM. Try to scp a file from VM to somewhere else and the opposite. If all these succeed, Xen is working with networking.  
     418'xm list' should list this new domain. 
     419To shutdown ttylinux, do 'xm shutdown tty' 
     421Eucalyptus Deployment 
     423Start Xen on compute node  
     425xend start 
     427'xm list' should display domain 0. 
     428'brctl show' should show eth1 attached to xenbr1 
     429Verify interfaces using 'ifconfig -a' 
     430Verify iptables using 'iptables -L' 
     432Start Eucalyptus 
     434Start Eucalyptus Cloud, CC (Cluster controller) and NC (Node Controller) 
     436On head node: 
     437To start cluster controller: 
     438$EUCALYPTUS/bin/httpd -f $EUCALYPTUS/etc/eucalyptus/httpd.conf 
     439To start cloud: 
     440ant -f $EUCALYPTUS/etc/eucalyptus/cloud-ant.xml run 
     442On compute node: 
     443To start node controller: 
     444$EUCALYPTUS/bin/httpd -f $EUCALYPTUS/etc/eucalyptus/httpd.conf 
     446Open up web browser to access the cloud 
     450Image management 
     452Adding a new image - go to the image directory (eg. ttylinux) 
     454${EUCALYPTUS}/usr/sbin/euca add_image  --disk-image ttylinux.img --kernel-image vmlinuz-2.6.18-xen --image-name ttylinuxWorking 
     456The default xen bridge in the config.xml created is xenbr0. Since, we are using xenbr1, we have to change xenbr0 to xenbr1in config.xml in $NFS_SHARE/registered/eucalyptus/<emi-id>  for all registered images and also cache ( $LOCAL_IMAGES/instances/eucalyptus/cache), if there was an attempt to run that image before. 
     458User guide 
     460User Guide @ 
     461for a. creating accounts, b. generating and storing the keys/certs, c. installing EC2 command-line tools, d.setting up environment variables to use ec2 comand line tools. 
     463Important EC2 commands: 
     464ec2-describe-availability-zones, ec2-describe-images, ec2-run-instances <emi-id>, ec2-describe-instances, ec2-terminate-instances <instance-id> 
     469On the compute node, fixed the stat command and put a 'stat' file in /opt/eucalyptus/bin (because of incompatibilities with old version of stat on the machine (compute node). stat contents: 
     473ARGS=`echo $@ | sed "s/file-system/filesystem/g"` 
     474ARGS=`echo $ARGS | sed "s/%S/%s/g"` 
     475/usr/bin/stat $ARGS 
     478Don't need vde on the head node. So, moved back vde from /opt/eucalyptus/packages/vde2-2.2.2 to ~anirban/vde2-2.2.2; Added 4 scripts (vdecmd, vde_cryptcab, vde_plug2tap, vde_switch) in /opt/eucalyptus/bin to return 0  
     480exit 0;)