Changes between Initial Version and Version 1 of Eucalyptus-1.6.2-Setup

Show
Ignore:
Timestamp:
06/03/10 16:41:30 (9 years ago)
Author:
shuang (IP: 152.54.6.53)
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • Eucalyptus-1.6.2-Setup

    v1 v1  
     1[[TOC(depth=99)]] 
     2= Setting up Eucalyptus 1.6.2 with ORCA 2.x = 
     3 
     4This page explains how to setup and test Euca 1.6.2 for ORCA. Most of the steps are similar with Since the servers do not have hardware virtualization support, we assume XEN is used.  
     5 
     6== XEN installation on Debian Lenny == 
     7According to [http://open.eucalyptus.com/wiki/EucalyptusInstall_v1.6], Euca 1.6.2 package is only available on Debian Squeeze. Unfortunately, Debian Squeeze does not support XEN well. Therefore we chose to use Lenny and install Euca from source.  
     8 
     9The idea is the Eucalyptus will set up slivers consisting of one or more VMs hanging off vlans, whose IDs orca specified at creation time. Then these slivers with vlans can be dynamically stitched to other slivers into a complete slice. Using Eucalyptus simplifies substrate setup compared to Xen. 
     10 
     11== Setting up stock Eucalyptus 1.5.2 == 
     12 
     13=== Hardware setup === 
     14 
     15You will need several hosts with dual interfaces - one interface on 'management' network and one on 'data plane' that will be stitched into ORCA slices. At RENCI this setup is implemented by having each host its eth1 on BEN management network (192.168.xx.xx address space) and eth0 connected into BEN with unassigned IP addresses. 
     16 
     17Each host should support hardware virtualization and be able to run KVM or Xen. The cluster will consist of a single head node and multiple compute nodes. The head node requires substantial disk space to store all VM filesystem images (if you plan to support many options).  
     18 
     19The dataplane interfaces of the cluster should be plugged in into an ORCA-controllable switch (a Cisco 6509 in our case) to allow for the mapping of Euca-created vlans to other vlan segments. 
     20 
     21=== Software pre-requisutes === 
     22  
     23 1. Ubuntu jaunty basic server install 
     24 1. kvm and libvirt (including libvirt-bin). Ubuntu favors kvm over Xen. KVM requires hardware virtualization support in your CPU! 
     25 1. ntp (Euca instructions suggest using open-ntp, however there is no reason not to use the Ubuntu 9.04 stock ntpd3 server 
     26 1. vconfig tools (to enable creating tagged interface) 
     27 1. brctl tools (to enable creating bridges) 
     28 
     29==== Testing software pre-requisites ==== 
     30 
     31The notes here are either for the head node [HN], the compute nodes [CN], or for all [ALL] 
     32 
     33 1. [ALL] Install and test ntp. Run ntpdc and verify the output is sane (substitute your own NTP server):  
     34{{{ 
     35$ apt-get install ntp 
     36$ echo server clock3.unc.edu >> /etc/ntp.conf 
     37$ /etc/init.d/ntp restart 
     38$ ntpdc 
     39}}} 
     40line to /etc/ntp.conf and restarting ntpd 
     41 1. [ALL] Test vconfig and brctl: 
     42{{{ 
     43$ vconfig add eth0 10 
     44$ ifconfig eth0.10 
     45$ vconfig rem eth0.10 
     46$ brctl show  
     47}}} 
     48 1. [CN] Make sure kvm is OK. If you receive a message about a problem with a kernel module either your CPU does not support hardware virtualization, or it is disabled in the BIOS. In the latter case, edit the BIOS setting and try again: 
     49{{{ 
     50$ /etc/init.d/kvm restart 
     51}}} 
     52 1. [CN] Make sure libvirtd is running: 
     53{{{ 
     54$ /etc/init.d/libvirt-bin restart 
     55$ virsh list 
     56}}} 
     57 1. Make sure the dataplane interface (although unconfigured) is UP 
     58{{{ 
     59$ ifconfig eth0 
     60}}} 
     61 1. [CN] identify or create a default bridge for kvm/xen to use. Xen by default creates a bridge (xenbr0). KVM requires that a bridge is manually setup. On Ubuntu this means adding 
     62{{{ 
     63auto br0 
     64iface br0 inet manual 
     65        bridge_ports eth0 
     66        bridge_stp off 
     67        bridge_maxwait 0 
     68}}}  
     69to /etc/network/interfaces. NOTE: in this setup eth0 is the dataplane interface facing into BEN. It remains unconfigured. The management interface is eth1 and is not shown here - it has a static configuration. Restart networking and verify that bridge br0 exists and eth0 is part of it, verify that br0 and eth0 are in the UP: 
     70{{{ 
     71$ brctl show 
     72$ ifconfig br0 
     73$ ifconfig eth0 
     74}}} 
     75Refer to the discussion [http://open.eucalyptus.com/wiki/EucalyptusNetworking_v1.5.2 here] about the significance of having the correct bridge setup.  
     76 1. [HN] Install DHCP server. It does not have to be configured or running. Euca will start it when needed. 
     77{{{ 
     78$ apt-get install dhcp3-server 
     79}}} 
     80 
     81=== Installing Eucalyptus === 
     82 
     83Follow the instructions [http://open.eucalyptus.com/wiki/EucalyptusInstallationUbuntuJaunty_v1.5.2 here]. Be sure to select the right packages for your architecture. BEN cluster uses amd64 packages for eucalyptus and euca2ools.  
     84 
     85=== Configuring Eucalyptus === 
     86 
     87We will configure Euca to run in MANAGED network mode to enable dynamic VLAN creation. This section only identifies entries in the /etc/eucalyptus.conf that differ from the default or need to be verified: 
     88 
     89[HN] 
     90{{{ 
     91VNET_INTERFACE="eth0" 
     92VNET_BRIDGE="br0" 
     93VNET_DHCPDAEMON="/usr/sbin/dhcpd3" 
     94VNET_DHCPUSER="dhcpd" 
     95VNET_MODE="MANAGED" 
     96VNET_SUBNET="172.16.0.0" 
     97VNET_NETMASK="255.255.0.0" 
     98VNET_DNS="192.168.201.254" 
     99VNET_ADDRSPERNET="32" 
     100#VNET_PUBLICIPS="your-free-public-ip-1 your-free-public-ip-2 ..." 
     101#VNET_MODE="SYSTEM" 
     102}}} 
     103 
     104[CN] 
     105{{{ 
     106VNET_INTERFACE="eth0" 
     107VNET_BRIDGE="br0" 
     108VNET_MODE="MANAGED" 
     109#VNET_MODE="SYSTEM" 
     110}}} 
     111 
     112=== Running Eucalyptus === 
     113 
     114 1. [HN]  
     115{{{ 
     116$ /etc/init.d/eucalyptus-cloud restart 
     117$ /etc/init.d/eucalyptus-cc restart 
     118}}} 
     119 1. [CN] 
     120{{{ 
     121$ /etc/init.d/eucalyptus-nc restart 
     122}}} 
     123 1. Login to the head node and perform initial configuration. It will be running on !https://hostname:8443/. Follow the instructions [http://open.eucalyptus.com/wiki/EucalyptusConfiguration_v1.5.2 here]. 
     124 1. Eucalyptus attempts to guess your Walrus URL (incorrectly in our case). The correct URL for it should be !http://head-node.name.or.ip:8773/services/Walrus 
     125 1. [HN] Setup passwordless login for users 'root' and 'eucalyptus' from head node to compute nodes.  
     126{{{ 
     127# ssh-keygen 
     128# cd /root/.ssh 
     129# cp id_rsa.pub authorized_keys 
     130# scp authorized_keys root@compute-node.name.or.ip:/root/.ssh/ 
     131# su - eucalyptus 
     132$ ssh-keygen 
     133$ cp id_rsa.pub authorized_keys 
     134$ scp authorized_keys root@compute-node.name.or.ip:/var/lib/eucalyptus/.ssh/ 
     135$ exit 
     136}}} 
     137 1. [HN] Create a cluster 
     138{{{ 
     139# euca_conf -addcluster <cluster name> head-node.name.or.ip 
     140}}} 
     141 1. [HN] Add compute nodes to it 
     142{{{ 
     143# euca_conf -addnode compute-node.name.or.ip 
     144}}} 
     145 
     146=== How Eucalyptus allocates VLAN tags === 
     147 
     148Eucalyptus uses vlans as an isolation mechanism between security groups. When a security group is created (euca-add-group) and then instances (VMs) are created within the group, Euca allocates a VLAN for this group, creates bridges in individual hosts (head and worker/client nodes) and attaches VMs to those bridges, instead of the default bridge. VLAN allocation in Eucalyptus is simple. In the configuration file there is a VNET_NETMASK parameter that indicates to it the width of the mask for address assignment, and, consequently, the total number of VMs that can be created (let's call this MAXHOSTS = 2^(32-MASK_WIDTH)^-2 ). The VNET_ADDRSPERNET parameter dictates the maximum number of VMs per security group/VLAN. Therefore the total number of VLANs that the system will use will be MAXHOSTS/VNET_ADDRSPERNET and the maximum VLAN tag Eucalyptus will use appears to be MAXHOSTS/VNET_ADDRSPERNET -1. For example for a 24 bit mask, MAXHOSTS=254 and for VNET_ADDRSPERNET=16, the maximum VLAN tag Eucalyptus will use will be 15. Default VLAN tag is 10 (for a default security group). 
     149 
     150=== Testing Eucalyptus === 
     151 
     152 1. [HN as root]  Download and install [http://open.eucalyptus.com/downloads euca2ools 1.0] 
     153 1. Register a user through the head node portal and [http://open.eucalyptus.com/wiki/EucalyptusGettingStarted_v1.5.2 acquire credentials] 
     154 1. [HN as regular user] Login, install the credentials and try 
     155{{{ 
     156euca-describe-availability-zones verbose 
     157}}} 
     158 1. [HN as root] If client machines aren't showing up, try restarting cloud controller and portal 
     159{{{ 
     160# /etc/init.d/eucalyptus-cc restart 
     161# /etc/init.d/eucalyptus-cloud restart 
     162}}} 
     163 1. [HN as regular user] Install a stock kernel, filesystem and ramdisk into walrus using these [http://open.eucalyptus.com/wiki/EucalyptusImageManagement_v1.5.2 instructions] 
     164{{{ 
     165$ tar -zxf euca-ubuntu-9.04-x86_64.tar.gz 
     166$ cd euca-ubuntu-9.04-x86_64/kvm-kernel 
     167$ euca-bundle-image -i vmlinuz-2.6.28-11-generic --kernel true 
     168$ euca-upload-bundle -b kernels -m /tmp/vmlinuz-2.6.28-11-generic.manifest.xml 
     169$ euca-register kernels/vmlinuz-2.6.28-11-generic.manifest.xml 
     170$ euca-bundle-image -i initrd.img-2.6.28-11-generic --ramdisk true 
     171$ euca-upload-bundle -b ramdisks -m /tmp/initrd.img-2.6.28-11-generic.manifest.xml 
     172$ euca-register ramdisks/initrd.img-2.6.28-11-generic.manifest.xml 
     173$ cd .. 
     174$ euca-bundle-image -i ubuntu.9-04.x86-64.img 
     175$ euca-upload-bundle -b images -m /tmp/ubuntu.9-04.x86-64.img.manifest.xml 
     176$ euca-register images/ubuntu.9-04.x86-64.img.manifest.xml 
     177$ euca-describe-images 
     178}}} 
     179 1. Generate ssh credentials for logging into the VMs 
     180{{{ 
     181euca-add-keypair mykey >mykey.private 
     182}}} 
     183 1. Attempt to create some vms 
     184{{{ 
     185euca-run-instances --addressing private -k mykey -n <number of instances to start> <emi-id>  
     186}}} 
     187 
     188 
     189=== Troubleshooting === 
     190 
     191 1. Check if the bridges are created (eucabrXX) in the compute nodes: 
     192{{{ 
     193$ brctl show 
     194}}} 
     195 1. Check the VMs are created in machines (may have to hunt for them since you don't know which specific compute node a VM will be created on): 
     196{{{ 
     197$ virsh list 
     198}}} 
     199 1. Check the logs 
     200 * [HN] /var/log/eucalyptus/cc.log 
     201 * [CN] /var/log/eucalyptus/nc.log 
     202 
     203== Modifying networking setup to work with ORCA == 
     204 
     205In order to use Eucalyptus with ORCA each physical host must have two interfaces: one to the switch that is the dataplane (Cisco 6509 in RENCI's case) and one that leads either to a management network or to the public internet, to allow connection with ORCA actors. ORCA site authority for Euca will be deployed on the Eucalyptus master node and it must have 
     206 1. connectivity to other ORCA actors 
     207 1. connectivity to Euca slivers so it can install guests 
     208 
     209[[Image(RENCI-Euca.png)]] 
     210 
     211This is achieved by creating a bridge on each node with a known name. This example uses 'sliverbr' although the name is not important, as it is not known to ORCA and is hard-wired into Eucalyptus through a patch. The following procedure must be performed on each node (master and client). This presumes the eth1 on the physical host is the interface that leads into the management network or to the public internet. It must not have a configured IP address. It can be an 802.1q VLAN interface. 
     212 
     213 1. Create a bridge and add eth1 into it, then configure the bridge to be the default interface 
     214{{{ 
     215$ brctl add sliverbr 
     216$ brctl addif sliverbr eth1 
     217$ ifconfig sliverbr <public or management IP address> netmask <netmask>  
     218$ route add default gw <default gw via the bridge interface> 
     219}}} 
     220 
     221In Ubuntu this can be accomplished by replacing eth1 configuration in /etc/network/interfaces file with the following: 
     222{{{ 
     223auto sliverbr 
     224iface sliverbr inet static 
     225        bridge_ports eth1 
     226        bridge_stp off 
     227        bridge_maxwait 0 
     228        address <ip address> 
     229        netmask <netmak> 
     230        gateway <gateway> 
     231}}} 
     232 
     233and rebooting. 
     234 
     235== Install Eucalyptus on master node from source == 
     236Now that everything is working it is time to re-install the Eucalyptus master from source. Download the source code for 1.5.2 and follow the build instructions. It is advisable to build it in $EUCALYPTUS=/opt/eucalyptus to keep it out of the way of a packaged install. Pay attention to dependencies required to build it. Once built, install it, restart it and test access to the portal, then VM creation again. You can reuse the configuration file from the stock install by moving it to $EUCALYPTUS/etc/eucalyptus/eucalyptus.conf.  
     237 
     238Note that this procedure invalidates any previous configuration you had, so you have to establish new user credentials and upload new images from which VMs are created. 
     239 
     240On Ubuntu 9.04 we had an issue with stock DHCP server that would not start properly after installing Eucalyptus master from source. It manifested itself by VMs being unreachable (in 'running' state). Log inspection (cc.log on master) revealed that dhcpd would not start when required. Our solution was to build a [http://www.isc.org/software/dhcp/313/download/dhcp-313targz dhcp server] from source and install it in a different location from the stock dhcpd. Then eucalyptus.conf had to be modified to reflect the new location of dhcpd.   
     241 
     242== Installing ORCA-related patches on master node == 
     243 
     244There are two patches - one for the VM creation template (to allow creation of VMs with more than one interface), the other to enable to specify the VLAN tag to be used for a particular security group.  
     245 1. Install the updated VM creation template on client nodes by replacing files gen_kvm_libvirt_xml  and gen_libvirt_xml in Eucalyptus. In Ubuntu/Debian they can be found under $EUCALYPTUS/usr/share/eucalyptus. The two files are attached to this page. 
     246 1. Install the patch (vlan.patch attached to this page) for Eucalyptus security group VLAN forcing on master node. Note that the user doing make and make install must have $JAVA_HOME, $EUCALYPTUS and $EUCALYPTUS_SRC defined and ant and java executables must be on the $PATH. 
     247{{{ 
     248$ cd eucalyptus-1.5.2/clc 
     249$ patch -p2 < vlan.patch 
     250$ make; make install 
     251}}} 
     252 Restart the cloud controller and the portal, try the following as a regular user: 
     253{{{ 
     254$ euca-add-group -d testvlan vlan22 
     255$ euca-run-instances -g vlan22 <usual parameters from above> 
     256}}} 
     257If this works, you should see that 'eucabr22' bridge has been created on every host and a 802.1q tagged interface (typycally eth0.22) was created and is part of that bridge. If VLAN id 22 is enabled on the switch between all hosts, then you should be able to reach the new VM on the IP address indicated by Eucalyptus and it will be on the private VLAN 22. 
     258 
     259== Configuring ORCA to control the Eucalyptus cluster == 
     260 
     261ORCA site authority must run from a container running on the Euca master node (otherwise the site authority has no access to the newly created VMs). Stand up an ORCA container with at least the Euca site authority. Here is the relevant sample piece of the actor_configs/config.xml: 
     262{{{ 
     263                <actor> 
     264                        <type>site</type> 
     265                        <name>duke-vm-site</name> 
     266                        <guid>9b12d036-23e7-11df-b3a3-000c29b1c193</guid> 
     267                        <pools> 
     268                                <pool> 
     269                                        <type>duke.vm</type> 
     270                                        <label>Eucalyptus Virtual Machine (DUKE)</label> 
     271                                        <description>A virtual machine</description> 
     272                                        <units>10</units> 
     273                                        <start>2010-01-30T00:00:00</start> 
     274                                        <end>2011-01-30T00:00:00</end> 
     275                                        <handler path="ec2/handler.xml" /> 
     276                                        <attributes> 
     277                                                <attribute> 
     278                                                        <key>resource.memory</key> 
     279                                                        <label>Memory</label> 
     280                                                        <value>128</value> 
     281                                                        <unit>MB</unit> 
     282                                                        <type>integer</type> 
     283                                                </attribute> 
     284                                                <attribute> 
     285                                                        <key>resource.cpu</key> 
     286                                                        <label>CPU</label> 
     287                                                        <value>1/2 of 2GHz Intel Xeon</value> 
     288                                                        <type>String</type> 
     289                                                </attribute> 
     290                                        </attributes> 
     291                                        <properties> 
     292                                                <property name="ip.list" value="192.168.206.3/24" /> 
     293                                                <property name="ip.subnet" value="255.255.255.0" /> 
     294                                                <property name="ip.gateway" value="192.168.206.1" /> 
     295                                                <property name="data.subnet" value="255.255.0.0" /> 
     296                                        </properties> 
     297                                </pool> 
     298                        </pools> 
     299                        <controls> 
     300                                <control type="duke.vm" class="orca.policy.core.SimpleVMControl" /> 
     301                        </controls> 
     302                </actor> 
     303}}} 
     304 
     305Note that this presumes an install where $ORCA_HOME contains the configuration files and they are not packaged in the webapp. 
     306 
     307Once the container is up and running, you need to acquire credentials for ORCA from Eucalyptus. Login to the Eucalyptus portal, create a user for ORCA, export its credentials, which come in a zip file. 
     308 
     309First test the credentials by unzipping them into $HOME/.euca, sourcing the .euca/XXX/eucarc file and making sure you can communicate with Eucalyptus using euca- tools. Create a keypair that ORCA will use (euca-add-keypair).  
     310 
     311Now place the contents of the zip file under $ORCA_HOME/ec2 on the head node. Note that the zip file has a structure to it, which needs to be ignored. Simply copy the files from the lowest level of the zip file hierarchy into the $ORCA_HOME/ec2. Copy the generated ssh key (from euca-add-keypair) into the same directory. Modify the $ORCA_HOME/ec2/eucarc file as follows: 
     312 
     313{{{ 
     314#EUCA_KEY_DIR=$(dirname $(readlink -f ${BASH_SOURCE})) 
     315export AMI_NAME=emi-6E7412EE 
     316export EC2_SSH_KEY=orca-key-renci 
     317export EC2_INSTANCE_TYPE=m1.small 
     318}}} 
     319 
     320(comment out the first line, add $AMI_NAME - the image to be used, $EC2_SSH_KEY and $EC2_INSTANCE_TYPE for ORCA to use). Note that AMI_NAME must have a default kernel and initrd image associated with it in Eucalyptus - they are currently not specified explicitly. 
     321 
     322NOTE: For Bella 2.0 ORCA Euca authority logs into the VM, turns off DHCP and installs  BEN DNS server into /etc/resolv.conf. This may need to be modified in handlers/ec2/resources/scripts/prepare-net.sh 
     323 
     324== Running Eucalyptus/EC2 handler tests == 
     325 
     326= Undoing a packaged install = 
     327 
     328When things don't seem to work, fear not, there is a way to start from scratch (note this is ONLY for DEB packaged installs, not installs from source): 
     329 
     330 1. Stop the euca daemons: 
     331[HN] 
     332{{{ 
     333$ /etc/init.d/eucalyptus-cc stop 
     334$ /etc/init.d/eucalyptus-cloud stop 
     335}}} 
     336[CN] 
     337{{{ 
     338$ /etc/init.d/eucalyptus-nc stop 
     339}}} 
     340 1. Remove eucalyptus packages (including config directories, if possible) 
     341[HN] 
     342{{{ 
     343$ dpkg --purge eucalyptus-cloud 
     344$ dpkg --purge eucalyptus-cc 
     345$ dpkg --purge eucalyptus-gl 
     346$ dpkg --purge eucalyptus-common 
     347$ dpkg --purge eucalyptus-javadeps 
     348}}} 
     349[CN] 
     350{{{ 
     351$ dpkg --purge eucalyptus-nc 
     352$ dpkg --purge eucalyptus-gl 
     353$ dpkg --purge eucalyptus-common 
     354}}} 
     355 1. Remove user eucalyptus from the system 
     356{{{ 
     357$ userdel -r eucalyptus 
     358$ groupdel eucalyptus 
     359}}} 
     360 1. Remove remnants of config and log directories 
     361{{{ 
     362$ rm -rf /etc/eucalyptus 
     363$ rm -rf /var/log/eucalyptus 
     364}}} 
     365 1. Sometimes you may need to fix dpkg state 
     366{{{  
     367$ vi /var/lib/dpkg/statoverride 
     368}}}  
     369and remove the line that mentions 'eucalyptus' 
     370 1. Start over 
     371 
     372= References =  
     373 
     374[https://help.ubuntu.com/community/KVM/Networking KVM Networking] 
     375 
     376[http://open.eucalyptus.com/wiki/EucalyptusInstallationUbuntuJaunty_v1.5.2 Euca install on Jaunty] 
     377 
     378[https://bugs.launchpad.net/eucalyptus/+bug/417217 euca-group-add bug] 
     379 
     380[http://open.eucalyptus.com/wiki/EucalyptusGettingStarted_v1.5.2 Getting started with Eucalyptus]