Best practices for setting up a distributed production environment

Introduction

Setting up a production ORCA configuration requires careful and meticulous approach to avoid common problems. This document attempts to summarize best practices used at RENCI for deploying ORCA.

Prerequisites

Software prerequisites

Build ORCA from source

Understanding container configuration: ORCA_HOME, ORCA_LOCAL, and all that

Actor configuration

Preparation

Preparing the infrastructure

  1. You must decide and document the topology of the ORCA actors in the future deployment - how many Tomcat containers you will have, which hosts they will be located on and which actors will be deployed in which container.
    • BIG FAT NOTE: current design prevents a service manager from talking to remote authorities if it is co-located with the broker in the same container. If your service manager is expected to redeem on remote sites, do not put it in the same container as the broker.
  2. Setup ORCA Configuration directories on each host
  3. Setup tomcat on each of the hosts.
  4. Setup MySQL database on each of the hosts
  5. Verify that tomcat starts and stops properly without ORCA
    $ cd $ORCA_HOME/tomcat
    $ ./start.sh
    $ ./stop.sh
    

Preparing the configuration and deploying

You can deploy from a binary release or from source. If deploying from source the deployment can use a single source tree located on the machine, where you build ORCA, which has Java, Ant and Maven. The hosts with Tomcat containers should have the same version of Java as is used on the build machine.

Bella 2.2

  1. Build ORCA as usual (no binary webapp option is available)
    $ cd $ORCA_SRC
    $ mvn install
    
  2. Prepare the directory structure with configuration files on the build host. Basically for each host with a container you should have a separate copy of $ORCA_SRC/webapp (or $ORCA_SRC/webapp2, if present). One way to do this:
    $ cd $HOME
    $ mkdir host1 host2 host3
    $ cd $ORCA_SRC/webapp2
    $ tar -cf - . | tar -xf - -C $HOME/host1
    $ tar -cf - . | tar -xf - -C $HOME/host2
    $ tar -cf - . | tar -xf - -C $HOME/host3
    
  3. For each host, generate and write down a new container GUID. This can be done in a number of ways.
  4. Edit hostX/config/container.properties
    1. replace the container.guid property with a new value.
    2. Update the container URL property to the host on which it will be deployed.
    3. Be sure to set emulation=false
    4. To make sure actors from this container are registered with ORCA actor registry, add or edit the following statements at the bottom of container.properties:
      registry.url=http://geni.renci.org:11080/registry/
      registry.method=registryService.insert
      
    5. Be sure the property values in container.properties for MySQL credentials on the host are valid
  5. Copy hostX/config/container.properties to $ORCA_HOME/config/ on hostX
  6. Generate a new security configuration (if not yet done) and a guid and certificate for each new actor (take note of the GUID)
    $ cd $ORCA_SRC/tools/config
    $ ant security.create.admin.config (THIS ONLY NEEDS TO BE DONE ONCE!!!)
    $ ant guid
    $ ant security.create.actor.config -Dactor=<Actor GUID>
    
  7. Copy the $ORCA_SRC/tools/config/runtime directory to $ORCA_HOME on the host where the actors will reside. In the instructions below if ORCA_HOME is not defined as an environment variable for the user, you have to replace it with an explicit path.
    $ cd $ORCA_SRC/tools/config
    $ tar -cf - runtime/ | ssh user@hostX tar -xf - -C $ORCA_HOME
    $ tar -cf - scripts/ | ssh user@hostX tar -xf - -C $ORCA_HOME
    
    1. CAVEAT: this method obviously accumulates certificates of all actors across containers. This is OK for some definition of OK. The important thing is that the container has a version of the keystore that contains the certificates for actors in that container. Any extra certificates will not harm but present a potential security loophole. A more involved method would create separate runtime/ directories for each container.
  8. Create the actor configuration for the container by editing $HOME/hostX/actor_configs/config.xml for each host. You will need the GUIDs you have issued to the actors. At the bottom of this page is a working example of a configuration file.
  9. To lookup certificates for actors in other containers (if they are already running), consult ORCA actor registry. You can extract them from your own keystores as described here
  10. Edit the $HOME/hostX/ant/build.properties to point to the URL of the container on hostX
  11. Package and deploy for each host (note that Tomcat must be running for the deploy step too succeed as it uses Tomcat Manager webapp to remotely install ORCA webapp into Tomcat).
    $ cd $HOME/hostX
    $ mvn package
    $ ant deploy
    
  12. Check that your actors are properly registered by visiting the ORCA actor registry.

Camano 3.0

Use the binary release for deployment. No need to build from source unless you've made changes to the source code.

  1. Generate GUIDs and certificates for all actors in $ORCA_SRC/tools/config, note the GUIDs, copy the contents of $ORCA_SRC/tools/config/runtime to each host's $ORCA_HOME/config/runtime
  2. Create $ORCA_HOME/config/container.properties and $ORCA_HOME/config/config.xml actor configuration file for each host
  3. Add any additional configuration, like e.g. for an authority actor
  4. Deploy the binary webapp
  5. Check that your actors are properly registered by visiting the ORCA actor registry.

Help

Consult configuration files in this directory for real working examples of multi-actor deployments.

Attachments