Version 62 (modified by ibaldin, 7 years ago)

--

Using Flukes

Overview

Flukes is a Java-based GUI that allows an experimenter to graphically inspect the state of ORCA substrates, create request topologies, submit requests to ORCA and inspect the returned substrate information (called 'manifest'). Several key features of Flukes:

  • Graphical intuitive interface
  • Uses native ORCA interfaces and resource descriptions (NDL-OWL) for maximum flexibility, unlike other interfaces to ORCA which use GENI RSpecs and GENI AM API and rely on conversions between these resource formats.
  • Allows the user to submit requests and inspect the output manifest in a graphical environment
  • Allows the user to login to provisioned resources

To launch Flukes, be sure Java 6 JRE is installed and double-click on this link. Flukes is a Java Webstart application, which means it will update itself every time it is launched. It also requires Internet access to operate (off-line mode is not supported). If Flukes fails to launch automatically from your browser, execute the following command from commandline:

$ javaws /path/to/downloaded/flukes.jnlp

'javaws' executable comes as part of JRE or JDK 1.6.

Flukes will ask for permission to access the filesystem, which you should grant. Flukes code is digitally signed by a certificate issued to IliaBaldine from BEN@RENCI with a certificate that has the following fingerprints:

SHA1: 3D 24 35 3E 57 72 C0 0C 40 B9 C6 4F B2 ED 62 C2 87 91 0C A8
MD5: 83 08 1C D4 C8 75 DB C4 36 21 B9 B2 92 98 D6 E0

FLUKES Screenshot

Advantages of using Flukes

Flukes uses ORCA native interfaces and resource descriptions. ORCA natively allows the user to specify a few things not available in RSpecs

  • Various node groupings (individual nodes, node groups)
  • Permits group properties common to all nodes in a group
  • Permits ORCA-driven splitting of groups between provider domains without user input
  • Powerful stitching - requests submitted to Fluke can span multiple geographic domains and ORCA will determine and create the proper stitching between slice components at Layer 2 across multiple national and regional research networks.
  • Allows the user to specify custom boot images (filesystem, kernel, ramdisk) to be installed into provisioned resources
  • Allows post-boot script templates (see section below)
  • Allows the user to specify functional dependencies between nodes (node A depends on node B == node B must boot before node A).

Using Flukes

Workflow

Flukes follows a simple workflow:

  • Create or load existing request file in the request pane
  • Type in a name of the slice (must be unique)
  • Click submit button
  • Verify the return status has no errors
  • Switch to Manifest pane
  • Type in slice request into the field
  • Continue querying for manifest until slice is ready (all reservations are up)

NOTE: the time it takes to create a slice is highly variable. It depends on a large number of factors. The dominant delays are associated with downloading and installing custom images at selected sites. If an image is new, it must be downloaded and registered at a site and then copied to a worker node on which a VM is being stood up. If the image has been used in the past, the delays are much shorter because images are cached in a number of places. Obviously, the larger the image, the longer the delays. It is not uncommon for slices to take 10 minutes to be stood up.

User preferences

Flukes relies on a $HOME/.flukes.properties file to customize its behavior. Properties that are not specified in this file use default values. To inspect the current values of preference properties, click on the 'Help' menu and select 'Preference Settings'. The popup window will display the current settings.

If you wish to overwrite any of the values:

  • create $HOME/.flukes.properties file if it doesn't exist
  • copy/paste the content of the preferences window into a text editor
  • modify the values of properties you need to change
  • save the file
  • restart Flukes.

Flukes GUI

Flukes GUI opens with several tabs:

  • Resources pane - for inspecting available resources
  • Request pane - for creating the request graph for a slice and submitting the request to ORCA
  • Manifest pane - for inspecting the manifest of a slice (either previously saved or queried from ORCA)

The slice request can be created in the Request pane and either submitted to ORCA or saved for future use. Once the slice request is submitted, the user can switch to the Manifest pane and query ORCA for slice status. Once the valid manifest is received, the user can inspect the elements of the slice and login to the provisioned nodes.

Left click places the nodes or node groups (in request mode). Clicking on one node and dragging (while holding the mouse button) to another node creates a link. Right click displays a context menu for either

  • A node (when clicked on a node)
  • A link (when clicked on a link)
  • A mouse mode (when clicked on empty space)

All panes support some subset of the following 3 mouse modes:

  • Edit - for creating nodes and edges [Request only]
  • Pick - for picking one or more nodes (holding the left mouse button while dragging selects multiple nodes) [Request only]
  • Pan - panning the graph image within the pane [All modes]

In pick mode the user can select multiple nodes (by clicking on a node or clicking and dragging to create a select area) and edit their common properties (e.g. image, domain binding etc) by right clicking anywhere except node or link and selecting 'Edit Selected Nodes'.

Nodes

Nodes are a main element of the topology. A node group is a special kind of a node that represents some number of nodes sharing common properties. Node and node groups can

  • Have IP addresses/IP address ranges within the dataplane assigned by the user
  • Have functional dependencies on other nodes (node A depending on node B will not boot until node B boots)
  • Nodes can have post boot scripts or post boot script templates.

Nodes are accessed by users over the management IP network which has access to the public Internet. In addition nodes may have other interfaces connecting them to other nodes, which are created by ORCA from Layer 2 circuits either within one domain or between domains, depending on the requested topology. IP address assignment on the primary/management interfaces is the responsibility of the hosting domain and these IP addresses are reported as part of the manifest (they cannot be set or determined at request time by the user). IP addresses for all other interfaces can be specified by the user. When IP addresses are left unspecified, the interfaces in the hosts are created, but remain unconfigured.

Two node types are possible - plain node (a VM with either a default image or an image specified by the user) or a node group. A node group is a set of nodes with common attributes (e.g. size, boot image, post boot script, binding). A node group can be connected to other node groups or individual nodes over a Layer 2 connection (e.g. a VLAN). Nodes within a node group that has a connection to another node or node group will be put on the same vlan.

Similar parameters can be specified for nodes and node groups. For node groups, the number of nodes should be specified. The IP address specified for the node group is the address to be assigned to the first node in the node group, with other addresses sequentially selected from the same address set limited by the netmask. E.g. if there are 3 nodes in a cluster and a user specifies 172.16.1.10/16 then the first node will have 172.16.1.10/16, second 172.16.1.11/16 etc.

There are two types of links in Flukes - point-to-point and broadcast. Point-to-point links can connect nodes and nodegroups to each other. In the case of a nodegroup, a point-to-point link leading to another node or node group implies that all nodes in the nodegroup must have an interface on this link.

Broadcast links can be used to create isolated VLANs inside a nodegroup (replacing the now obsoleted 'private vlan' feature in flukes) or to connect multiple nodes together. Currently nodes or nodegroups connected to a broadcast link cannot be bound to different domains (however you can connect two nodegroups with a point-to-point link and bind those nodegroups to different domains).

Any link can be given an explicit label (VLAN tag). This tag must come from a specially defined pool of tags known to ORCA that lead to e.g. other substrates. For example, to put a node on a mesoscale VLAN 1750 at a particular site

  1. create a node and a broadcast link
  2. connect the two together
  3. bind the node to a particular site (if submitting via ExoSM)
  4. right-cick on the broadcast link and fill in the tag '1750' for it
  5. submit this request.

Node dependencies

ORCA performs its own dependency analysis on resources to determine the specific order in which they need to be configured/booted. In some cases additional functional dependencies can be introduced by the user to reflect e.g. the semantics of the application (for example an application may have a head node, which must be booted prior to worker nodes).

Node properties dialog allows for these additional dependencies to be specified. A dependency implies that the dependent node will not boot until its dependency target has booted. Dependencies may span multiple domains.

Domain binding

One of the powerful features of ORCA is automatic domain binding of nodes or node groups to specific provider domains based on resource availability. There are scenarios in which a user may explicitly indicate a binding of nodes within a request to specific domains. This can be done at either reservation level (within reservation properties), which makes all nodes within this request bound to a single domain, or it can be done on the individual node or node group level, where a binding of a specific node to a domain can be selected through node properties.

Nodes and node groups can be bound to resource provider domains by selecting one of the domains in the menu. Unbound nodes or domains in a bound reservation will be automatically bound to the selected domain.

An unbound node group may be split by the system among multiple domains depending on the availability of resources. A bound node group is never split and is allocated from a single domain. A user can indicate the splittability of the group by checking the appropriate box in node group properties.

Client disk images

ORCA allows experimenters to create their own filesystem, kernel and ramdisk images to be used for a slice. As a starting point an experimenter can use a number of pre-existing NEuca-enabled images or start with a base Eucalyptus image and add NEuca tools to them to enable post boot configuration of network interfaces and the execution of post-boot scripts. An experimenter can then create an ImageProxy metafile describing the images, place them on a web server and use the URL of the metafile (and its SHA-1 signature) in the slice request.

VM Images are defined within Flukes using the 'Images' button. The user can add new images or edit the properties of existing images. Each image must have:

  • A unique short name
  • A URL of the ImageProxy metafile from which it can be downloaded
  • A SHA-1 hash of the metafile

Images can be specified per reservation (as part of reservation properties) or per node. Images must first be defined using the 'Images|New' dialog and then can be referenced from pull down menus in reservation and node properties.

If no images are specified in the request, the sites to which the slivers are bound will select default images which are not guaranteed to be the same across all sites.

Note that there are two ways to customize a boot image. One is to create your own filesystem/kernel/ramdisk images, describe them in a metafile and use this metafile in the slice request. Another is to use the a stock image and use a post-boot script described below to load additional software onto the image after the VM boots.

In order to allow reusing the same image descriptions across multiple requests, Flukes users can edit ~/.flukes.properties file to add their favorite images:

image1.name=regression-i386-debian
image1.url=http://geni-images.renci.org/images/regression/regression-deb5-i386.xml
image1.hash=ea80af6601a2a000ec5b050d7e7701f26db096fc

More images can be added by specifying image2.name, image2.url, image2.hash and so on. The images will be visible in the "Client Images' window and can be selected for individual nodes or the entire reservation.

Post Boot Scripts and templates

Flukes allows a post boot script to be associated with each node or node group. Post boot scripts are the preferred way to create customized instances from a shared image. Specifically, a post boot script can contain an arbitrary script that is a executed immediately after the instance has booted.

Simple example:

echo "Hello from post boot script"
apt-get install my-favorite-app
echo node1 > /etc/hostname

For added functionality, ORCA uses velocity templates to create the scripts. See the user's guide for details about Velocity.

ORCA templates follow the Velocity syntax which can be used to replace several key pieces of data that are assigned by ORCA. Specifically, there are one or more arrays that contains the IP addresses assigned to each instance. There is one array of IP addresses for each group. Each array of IP addresses is referenced using the name provided to Flukes for the group. In addition, there is a variable called $MY_ADDR_INDEX that is the index of the IP address of the current instance within its group's array of IP addresses. All arrays of IP addresses are accessible from any post boot script in the current request.

The following is an example post boot script that adds an entry to the /etc/hosts file for each of the instances in the request and sets the hostname of each instance to an appropriate value. In this particular example, the first instance is assumed to be a master node named "master" while the remaining nodes are named "worker1", "worker2", etc.. The string "Group1" was used as the group name.

#!/bin/bash
   # Test script 
   echo "Hello from post boot script"
   echo $Group1.get(0) master  >> /etc/hosts 
#set ( $size = $Group1.size() - 1 )
#foreach ( $i in [1..$size] )
   echo $Group1.get($i) worker$i >> /etc/hosts
#end
#if ( $MY_ADDR_INDEX == 0)
   echo master > /etc/hostname
#else
   echo worker$MY_ADDR_INDEX > /etc/hostname
#end

WARNING: due to a known bug, all script lines (not template lines) except for the first (which typically just says "#!/bin/bash" or similar) must start with white space (e.g. a tab) in order to be properly interpreted (as in the example above).

OpenFlow

You can create simple slices that attach to your OpenFlow controller. The easiest type of slice is a NodeGroup? with broadcast link (you can assign IP addresses to nodes on this VLAN). At reservation level (click on 'Reservation' button) be sure to bind the request to one of the XO racks (other racks will not support OpenFlow).

  • Start your openflow controller on some internet-reachable host
  • Click on 'OpenFlow Reservation'
  • In new fields fill out your email, your slice password and the url of your controller (you have to start it outside the slice). The url of the controller typically looks like 'tcp:<hostname>:<port number>'. When starting a controller on some host be sure the port on which the controller runs is not blocked by a firewall. Remember that it is the slice that connects to your controller, not the other way around.
  • Submit your request

Once the slice is created you will have a switch with some number of nodes attached to it, with the switch controlled by your controller.

Graph Layouts

There are several graph layouts available from a pull-down menu in Flukes. When a graph is saved in NDL-OWL, its layout is not preserved, so this feature is provided as a convenience. After loading a graph a user can experiment with different layouts to find the most convenient one.

User authorization

When using Flukes, the experimenter can use one of the following credentials:

  • A private key/certificate pem file provided by the GENI Project Office
  • A private key/certificate pem file provided by Emulab/ProtoGENI
  • A private key and a certificate provided by BEN

This section explains how to convert these credentials for use with Flukes. It presumes you have one of the credentials described above. A Flukes keystore can contain multiple private key entries. A user can select the key to be used for submitting a specific request at the time of the submission.

In order to make these credentials usable by Flukes, a user must create a Java JKS keystore and import at least one of the credentials into it. Java's command-line tool for manipulating keystores does not permit importing an existing private key. The easiest way to import a key/certificate is to download a tool like Portecle. Portecle is Java-based and works across multiple platforms. Importing a .pem file containing a key and a certificate issued by either Emulab or GPO is straightforward and shown in this screen capture. The process is as follows:

  • Obtain a .pem file
  • Check it using openssl:
    $ openssl x509 -text -in mycredential.pem
    
  • Start Portecle
  • Create a new keystore of type JKS
  • Click on "Import Key Pair"
  • Locate the .pem file
  • Enter the password protecting the .pem file
  • Import the key and certificate, but change the key alias to something simpler than a GUID (e.g. firstnamelastname-emulab with no spaces)
  • Enter and confirm a new key password
  • Click on "Save Keystore"
  • Enter and confirm the new keystore password (use same as the key password)
  • Enter the path to the keystore file into .flukes.properties "user.keystore" property

A note about keystore passwords: a java keystore always has a password protecting its integrity. Each key within a keystore can also have a password. Flukes currently assumes both the keystore and key passwords are the same.

When you submit a request through Flukes, it will ask you for the key alias and password. You should use the alias you assigned to this new key and the password used for keystore and keys. Note that you can have multiple key/certificate pairs under different aliases within the same keystore (e.g. one from GPO and one from BEN).

Using BEN credentials

For credentials issued through BEN (usually as part of BEN VPN access) the user must first create a single PKCS#12 file combining the private key and a BEN certificate (they are otherwise supplied separately). Then the resulting .p12 file can be imported into a Java keystore the same way that a .pem file from GPO or Emulab is imported:

$ openssl pkcs12 -export -in username.crt -inkey username.key -out username.p12

In this case username.crt and username.key are your BEN certificate and private key, respectively and username.p12 is the new file combining the two.

Attachments