Changes between Initial Version and Version 1 of Eucalyptus-1.6.2-Setup

06/03/10 16:41:30 (9 years ago)
shuang (IP:



  • Eucalyptus-1.6.2-Setup

    v1 v1  
     2= Setting up Eucalyptus 1.6.2 with ORCA 2.x = 
     4This page explains how to setup and test Euca 1.6.2 for ORCA. Most of the steps are similar with Since the servers do not have hardware virtualization support, we assume XEN is used.  
     6== XEN installation on Debian Lenny == 
     7According to [], Euca 1.6.2 package is only available on Debian Squeeze. Unfortunately, Debian Squeeze does not support XEN well. Therefore we chose to use Lenny and install Euca from source.  
     9The idea is the Eucalyptus will set up slivers consisting of one or more VMs hanging off vlans, whose IDs orca specified at creation time. Then these slivers with vlans can be dynamically stitched to other slivers into a complete slice. Using Eucalyptus simplifies substrate setup compared to Xen. 
     11== Setting up stock Eucalyptus 1.5.2 == 
     13=== Hardware setup === 
     15You will need several hosts with dual interfaces - one interface on 'management' network and one on 'data plane' that will be stitched into ORCA slices. At RENCI this setup is implemented by having each host its eth1 on BEN management network (192.168.xx.xx address space) and eth0 connected into BEN with unassigned IP addresses. 
     17Each host should support hardware virtualization and be able to run KVM or Xen. The cluster will consist of a single head node and multiple compute nodes. The head node requires substantial disk space to store all VM filesystem images (if you plan to support many options).  
     19The dataplane interfaces of the cluster should be plugged in into an ORCA-controllable switch (a Cisco 6509 in our case) to allow for the mapping of Euca-created vlans to other vlan segments. 
     21=== Software pre-requisutes === 
     23 1. Ubuntu jaunty basic server install 
     24 1. kvm and libvirt (including libvirt-bin). Ubuntu favors kvm over Xen. KVM requires hardware virtualization support in your CPU! 
     25 1. ntp (Euca instructions suggest using open-ntp, however there is no reason not to use the Ubuntu 9.04 stock ntpd3 server 
     26 1. vconfig tools (to enable creating tagged interface) 
     27 1. brctl tools (to enable creating bridges) 
     29==== Testing software pre-requisites ==== 
     31The notes here are either for the head node [HN], the compute nodes [CN], or for all [ALL] 
     33 1. [ALL] Install and test ntp. Run ntpdc and verify the output is sane (substitute your own NTP server):  
     35$ apt-get install ntp 
     36$ echo server >> /etc/ntp.conf 
     37$ /etc/init.d/ntp restart 
     38$ ntpdc 
     40line to /etc/ntp.conf and restarting ntpd 
     41 1. [ALL] Test vconfig and brctl: 
     43$ vconfig add eth0 10 
     44$ ifconfig eth0.10 
     45$ vconfig rem eth0.10 
     46$ brctl show  
     48 1. [CN] Make sure kvm is OK. If you receive a message about a problem with a kernel module either your CPU does not support hardware virtualization, or it is disabled in the BIOS. In the latter case, edit the BIOS setting and try again: 
     50$ /etc/init.d/kvm restart 
     52 1. [CN] Make sure libvirtd is running: 
     54$ /etc/init.d/libvirt-bin restart 
     55$ virsh list 
     57 1. Make sure the dataplane interface (although unconfigured) is UP 
     59$ ifconfig eth0 
     61 1. [CN] identify or create a default bridge for kvm/xen to use. Xen by default creates a bridge (xenbr0). KVM requires that a bridge is manually setup. On Ubuntu this means adding 
     63auto br0 
     64iface br0 inet manual 
     65        bridge_ports eth0 
     66        bridge_stp off 
     67        bridge_maxwait 0 
     69to /etc/network/interfaces. NOTE: in this setup eth0 is the dataplane interface facing into BEN. It remains unconfigured. The management interface is eth1 and is not shown here - it has a static configuration. Restart networking and verify that bridge br0 exists and eth0 is part of it, verify that br0 and eth0 are in the UP: 
     71$ brctl show 
     72$ ifconfig br0 
     73$ ifconfig eth0 
     75Refer to the discussion [ here] about the significance of having the correct bridge setup.  
     76 1. [HN] Install DHCP server. It does not have to be configured or running. Euca will start it when needed. 
     78$ apt-get install dhcp3-server 
     81=== Installing Eucalyptus === 
     83Follow the instructions [ here]. Be sure to select the right packages for your architecture. BEN cluster uses amd64 packages for eucalyptus and euca2ools.  
     85=== Configuring Eucalyptus === 
     87We will configure Euca to run in MANAGED network mode to enable dynamic VLAN creation. This section only identifies entries in the /etc/eucalyptus.conf that differ from the default or need to be verified: 
     100#VNET_PUBLICIPS="your-free-public-ip-1 your-free-public-ip-2 ..." 
     112=== Running Eucalyptus === 
     114 1. [HN]  
     116$ /etc/init.d/eucalyptus-cloud restart 
     117$ /etc/init.d/eucalyptus-cc restart 
     119 1. [CN] 
     121$ /etc/init.d/eucalyptus-nc restart 
     123 1. Login to the head node and perform initial configuration. It will be running on !https://hostname:8443/. Follow the instructions [ here]. 
     124 1. Eucalyptus attempts to guess your Walrus URL (incorrectly in our case). The correct URL for it should be ! 
     125 1. [HN] Setup passwordless login for users 'root' and 'eucalyptus' from head node to compute nodes.  
     127# ssh-keygen 
     128# cd /root/.ssh 
     129# cp authorized_keys 
     130# scp authorized_keys 
     131# su - eucalyptus 
     132$ ssh-keygen 
     133$ cp authorized_keys 
     134$ scp authorized_keys 
     135$ exit 
     137 1. [HN] Create a cluster 
     139# euca_conf -addcluster <cluster name> 
     141 1. [HN] Add compute nodes to it 
     143# euca_conf -addnode 
     146=== How Eucalyptus allocates VLAN tags === 
     148Eucalyptus uses vlans as an isolation mechanism between security groups. When a security group is created (euca-add-group) and then instances (VMs) are created within the group, Euca allocates a VLAN for this group, creates bridges in individual hosts (head and worker/client nodes) and attaches VMs to those bridges, instead of the default bridge. VLAN allocation in Eucalyptus is simple. In the configuration file there is a VNET_NETMASK parameter that indicates to it the width of the mask for address assignment, and, consequently, the total number of VMs that can be created (let's call this MAXHOSTS = 2^(32-MASK_WIDTH)^-2 ). The VNET_ADDRSPERNET parameter dictates the maximum number of VMs per security group/VLAN. Therefore the total number of VLANs that the system will use will be MAXHOSTS/VNET_ADDRSPERNET and the maximum VLAN tag Eucalyptus will use appears to be MAXHOSTS/VNET_ADDRSPERNET -1. For example for a 24 bit mask, MAXHOSTS=254 and for VNET_ADDRSPERNET=16, the maximum VLAN tag Eucalyptus will use will be 15. Default VLAN tag is 10 (for a default security group). 
     150=== Testing Eucalyptus === 
     152 1. [HN as root]  Download and install [ euca2ools 1.0] 
     153 1. Register a user through the head node portal and [ acquire credentials] 
     154 1. [HN as regular user] Login, install the credentials and try 
     156euca-describe-availability-zones verbose 
     158 1. [HN as root] If client machines aren't showing up, try restarting cloud controller and portal 
     160# /etc/init.d/eucalyptus-cc restart 
     161# /etc/init.d/eucalyptus-cloud restart 
     163 1. [HN as regular user] Install a stock kernel, filesystem and ramdisk into walrus using these [ instructions] 
     165$ tar -zxf euca-ubuntu-9.04-x86_64.tar.gz 
     166$ cd euca-ubuntu-9.04-x86_64/kvm-kernel 
     167$ euca-bundle-image -i vmlinuz-2.6.28-11-generic --kernel true 
     168$ euca-upload-bundle -b kernels -m /tmp/vmlinuz-2.6.28-11-generic.manifest.xml 
     169$ euca-register kernels/vmlinuz-2.6.28-11-generic.manifest.xml 
     170$ euca-bundle-image -i initrd.img-2.6.28-11-generic --ramdisk true 
     171$ euca-upload-bundle -b ramdisks -m /tmp/initrd.img-2.6.28-11-generic.manifest.xml 
     172$ euca-register ramdisks/initrd.img-2.6.28-11-generic.manifest.xml 
     173$ cd .. 
     174$ euca-bundle-image -i ubuntu.9-04.x86-64.img 
     175$ euca-upload-bundle -b images -m /tmp/ubuntu.9-04.x86-64.img.manifest.xml 
     176$ euca-register images/ubuntu.9-04.x86-64.img.manifest.xml 
     177$ euca-describe-images 
     179 1. Generate ssh credentials for logging into the VMs 
     181euca-add-keypair mykey >mykey.private 
     183 1. Attempt to create some vms 
     185euca-run-instances --addressing private -k mykey -n <number of instances to start> <emi-id>  
     189=== Troubleshooting === 
     191 1. Check if the bridges are created (eucabrXX) in the compute nodes: 
     193$ brctl show 
     195 1. Check the VMs are created in machines (may have to hunt for them since you don't know which specific compute node a VM will be created on): 
     197$ virsh list 
     199 1. Check the logs 
     200 * [HN] /var/log/eucalyptus/cc.log 
     201 * [CN] /var/log/eucalyptus/nc.log 
     203== Modifying networking setup to work with ORCA == 
     205In order to use Eucalyptus with ORCA each physical host must have two interfaces: one to the switch that is the dataplane (Cisco 6509 in RENCI's case) and one that leads either to a management network or to the public internet, to allow connection with ORCA actors. ORCA site authority for Euca will be deployed on the Eucalyptus master node and it must have 
     206 1. connectivity to other ORCA actors 
     207 1. connectivity to Euca slivers so it can install guests 
     211This is achieved by creating a bridge on each node with a known name. This example uses 'sliverbr' although the name is not important, as it is not known to ORCA and is hard-wired into Eucalyptus through a patch. The following procedure must be performed on each node (master and client). This presumes the eth1 on the physical host is the interface that leads into the management network or to the public internet. It must not have a configured IP address. It can be an 802.1q VLAN interface. 
     213 1. Create a bridge and add eth1 into it, then configure the bridge to be the default interface 
     215$ brctl add sliverbr 
     216$ brctl addif sliverbr eth1 
     217$ ifconfig sliverbr <public or management IP address> netmask <netmask>  
     218$ route add default gw <default gw via the bridge interface> 
     221In Ubuntu this can be accomplished by replacing eth1 configuration in /etc/network/interfaces file with the following: 
     223auto sliverbr 
     224iface sliverbr inet static 
     225        bridge_ports eth1 
     226        bridge_stp off 
     227        bridge_maxwait 0 
     228        address <ip address> 
     229        netmask <netmak> 
     230        gateway <gateway> 
     233and rebooting. 
     235== Install Eucalyptus on master node from source == 
     236Now that everything is working it is time to re-install the Eucalyptus master from source. Download the source code for 1.5.2 and follow the build instructions. It is advisable to build it in $EUCALYPTUS=/opt/eucalyptus to keep it out of the way of a packaged install. Pay attention to dependencies required to build it. Once built, install it, restart it and test access to the portal, then VM creation again. You can reuse the configuration file from the stock install by moving it to $EUCALYPTUS/etc/eucalyptus/eucalyptus.conf.  
     238Note that this procedure invalidates any previous configuration you had, so you have to establish new user credentials and upload new images from which VMs are created. 
     240On Ubuntu 9.04 we had an issue with stock DHCP server that would not start properly after installing Eucalyptus master from source. It manifested itself by VMs being unreachable (in 'running' state). Log inspection (cc.log on master) revealed that dhcpd would not start when required. Our solution was to build a [ dhcp server] from source and install it in a different location from the stock dhcpd. Then eucalyptus.conf had to be modified to reflect the new location of dhcpd.   
     242== Installing ORCA-related patches on master node == 
     244There are two patches - one for the VM creation template (to allow creation of VMs with more than one interface), the other to enable to specify the VLAN tag to be used for a particular security group.  
     245 1. Install the updated VM creation template on client nodes by replacing files gen_kvm_libvirt_xml  and gen_libvirt_xml in Eucalyptus. In Ubuntu/Debian they can be found under $EUCALYPTUS/usr/share/eucalyptus. The two files are attached to this page. 
     246 1. Install the patch (vlan.patch attached to this page) for Eucalyptus security group VLAN forcing on master node. Note that the user doing make and make install must have $JAVA_HOME, $EUCALYPTUS and $EUCALYPTUS_SRC defined and ant and java executables must be on the $PATH. 
     248$ cd eucalyptus-1.5.2/clc 
     249$ patch -p2 < vlan.patch 
     250$ make; make install 
     252 Restart the cloud controller and the portal, try the following as a regular user: 
     254$ euca-add-group -d testvlan vlan22 
     255$ euca-run-instances -g vlan22 <usual parameters from above> 
     257If this works, you should see that 'eucabr22' bridge has been created on every host and a 802.1q tagged interface (typycally eth0.22) was created and is part of that bridge. If VLAN id 22 is enabled on the switch between all hosts, then you should be able to reach the new VM on the IP address indicated by Eucalyptus and it will be on the private VLAN 22. 
     259== Configuring ORCA to control the Eucalyptus cluster == 
     261ORCA site authority must run from a container running on the Euca master node (otherwise the site authority has no access to the newly created VMs). Stand up an ORCA container with at least the Euca site authority. Here is the relevant sample piece of the actor_configs/config.xml: 
     263                <actor> 
     264                        <type>site</type> 
     265                        <name>duke-vm-site</name> 
     266                        <guid>9b12d036-23e7-11df-b3a3-000c29b1c193</guid> 
     267                        <pools> 
     268                                <pool> 
     269                                        <type>duke.vm</type> 
     270                                        <label>Eucalyptus Virtual Machine (DUKE)</label> 
     271                                        <description>A virtual machine</description> 
     272                                        <units>10</units> 
     273                                        <start>2010-01-30T00:00:00</start> 
     274                                        <end>2011-01-30T00:00:00</end> 
     275                                        <handler path="ec2/handler.xml" /> 
     276                                        <attributes> 
     277                                                <attribute> 
     278                                                        <key>resource.memory</key> 
     279                                                        <label>Memory</label> 
     280                                                        <value>128</value> 
     281                                                        <unit>MB</unit> 
     282                                                        <type>integer</type> 
     283                                                </attribute> 
     284                                                <attribute> 
     285                                                        <key>resource.cpu</key> 
     286                                                        <label>CPU</label> 
     287                                                        <value>1/2 of 2GHz Intel Xeon</value> 
     288                                                        <type>String</type> 
     289                                                </attribute> 
     290                                        </attributes> 
     291                                        <properties> 
     292                                                <property name="ip.list" value="" /> 
     293                                                <property name="ip.subnet" value="" /> 
     294                                                <property name="ip.gateway" value="" /> 
     295                                                <property name="data.subnet" value="" /> 
     296                                        </properties> 
     297                                </pool> 
     298                        </pools> 
     299                        <controls> 
     300                                <control type="duke.vm" class="orca.policy.core.SimpleVMControl" /> 
     301                        </controls> 
     302                </actor> 
     305Note that this presumes an install where $ORCA_HOME contains the configuration files and they are not packaged in the webapp. 
     307Once the container is up and running, you need to acquire credentials for ORCA from Eucalyptus. Login to the Eucalyptus portal, create a user for ORCA, export its credentials, which come in a zip file. 
     309First test the credentials by unzipping them into $HOME/.euca, sourcing the .euca/XXX/eucarc file and making sure you can communicate with Eucalyptus using euca- tools. Create a keypair that ORCA will use (euca-add-keypair).  
     311Now place the contents of the zip file under $ORCA_HOME/ec2 on the head node. Note that the zip file has a structure to it, which needs to be ignored. Simply copy the files from the lowest level of the zip file hierarchy into the $ORCA_HOME/ec2. Copy the generated ssh key (from euca-add-keypair) into the same directory. Modify the $ORCA_HOME/ec2/eucarc file as follows: 
     314#EUCA_KEY_DIR=$(dirname $(readlink -f ${BASH_SOURCE})) 
     315export AMI_NAME=emi-6E7412EE 
     316export EC2_SSH_KEY=orca-key-renci 
     317export EC2_INSTANCE_TYPE=m1.small 
     320(comment out the first line, add $AMI_NAME - the image to be used, $EC2_SSH_KEY and $EC2_INSTANCE_TYPE for ORCA to use). Note that AMI_NAME must have a default kernel and initrd image associated with it in Eucalyptus - they are currently not specified explicitly. 
     322NOTE: For Bella 2.0 ORCA Euca authority logs into the VM, turns off DHCP and installs  BEN DNS server into /etc/resolv.conf. This may need to be modified in handlers/ec2/resources/scripts/ 
     324== Running Eucalyptus/EC2 handler tests == 
     326= Undoing a packaged install = 
     328When things don't seem to work, fear not, there is a way to start from scratch (note this is ONLY for DEB packaged installs, not installs from source): 
     330 1. Stop the euca daemons: 
     333$ /etc/init.d/eucalyptus-cc stop 
     334$ /etc/init.d/eucalyptus-cloud stop 
     338$ /etc/init.d/eucalyptus-nc stop 
     340 1. Remove eucalyptus packages (including config directories, if possible) 
     343$ dpkg --purge eucalyptus-cloud 
     344$ dpkg --purge eucalyptus-cc 
     345$ dpkg --purge eucalyptus-gl 
     346$ dpkg --purge eucalyptus-common 
     347$ dpkg --purge eucalyptus-javadeps 
     351$ dpkg --purge eucalyptus-nc 
     352$ dpkg --purge eucalyptus-gl 
     353$ dpkg --purge eucalyptus-common 
     355 1. Remove user eucalyptus from the system 
     357$ userdel -r eucalyptus 
     358$ groupdel eucalyptus 
     360 1. Remove remnants of config and log directories 
     362$ rm -rf /etc/eucalyptus 
     363$ rm -rf /var/log/eucalyptus 
     365 1. Sometimes you may need to fix dpkg state 
     367$ vi /var/lib/dpkg/statoverride 
     369and remove the line that mentions 'eucalyptus' 
     370 1. Start over 
     372= References =  
     374[ KVM Networking] 
     376[ Euca install on Jaunty] 
     378[ euca-group-add bug] 
     380[ Getting started with Eucalyptus]