Version 24 (modified by ibaldin, 9 years ago)

--

Best practices for setting up a distributed production environment

Introduction

Setting up a production ORCA configuration requires careful and meticulous approach to avoid common problems. This document attempts to summarize best practices used at RENCI for deploying ORCA.

Prerequisites

Software prerequisites

Build ORCA from source

Understanding container configuration: ORCA_HOME, ORCA_LOCAL, and all that

Actor configuration

Preparation

Preparing the infrastructure

  1. You must decide and document the topology of the ORCA actors in the future deployment - how many Tomcat containers you will have, which hosts they will be located on and which actors will be deployed in which container.
  2. Setup tomcat on each of the hosts. The canonical way, in which RENCI sets up ORCA is to have $ORCA_HOME=/opt/orca on each of the deployment hosts. ORCA-modified tomcat is installed under $ORCA_HOME/tomcat and $ORCA_HOME/tomcat/start.sh and $ORCA_HOME/tomcat/stop.sh scripts are modified to reflect the $ORCA_HOME setting by prepending them with
    export ORCA_HOME=/opt/orca
    export CATALINA_HOME=$ORCA_HOME/tomcat
    
  3. MySQL database on each of the hosts must be initialized with the schema and initial data. Loading inventory files is optional.
  4. Verify that tomcat starts and stops properly without ORCA
    $ cd $ORCA_HOME/tomcat
    $ ./start.sh
    $ ./stop.sh
    

Preparing the configuration

The deployment can use a single source tree located on the machine, where you build ORCA, which has Java, Ant and Maven. The hosts with Tomcat containers should have the same version of Java as is used on the build machine.

  1. Build ORCA as usual
    $ cd $ORCA_SRC
    $ mvn install
    
  2. Prepare the directory structure with configuration files on the build host. Basically for each host with a container you should have a separate copy of $ORCA_SRC/webapp (or $ORCA_SRC/webapp2, if present). One way to do this:
    $ cd $HOME
    $ mkdir host1 host2 host3
    $ cd $ORCA_SRC/webapp2
    $ tar -cf - . | tar -xf - -C $HOME/host1
    $ tar -cf - . | tar -xf - -C $HOME/host2
    $ tar -cf - . | tar -xf - -C $HOME/host3
    
  3. For each host, generate and write down a new container GUID. This can be done in a number of ways. Edit hostX/config/container.properties to replace the container.guid property with a new value. Also update the container URL property to the host on which it will be deployed.
  4. To make sure actors from this container are registered with ORCA actor registry, add or edit the following statements at the bottom of container.properties:
    registry.url=http://geni.renci.org:8080/registry/
    registry.method=registryService.insert
    
  5. Be sure the property values in container.properties for MySQL credentials on the host are valid
  6. Copy hostX/config/container.properties to $ORCA_HOME/config/ on hostX
  7. Generate a new security configuration (if not yet done) and a guid and certificate for each new actor (take note of the GUID)
    $ cd $ORCA_SRC/tools/config
    $ ant security.create.admin.config (THIS ONLY NEEDS TO BE DONE ONCE!!!)
    $ ant guid
    $ ant security.create.actor.config -Dactor=<Actor GUID>
    
  8. Copy the $ORCA_SRC/tools/config/runtime directory to $ORCA_HOME on the host where the actors will reside. In the instructions below if ORCA_HOME is not defined as an environment variable for the user, you have to replace it with an explicit path.
    $ cd $ORCA_SRC/tools/config
    $ tar -cf - runtime/ | ssh user@hostX tar -xf - -C $ORCA_HOME
    $ tar -cf - scripts/ | ssh user@hostX tar -xf - -C $ORCA_HOME
    

CAVEAT: this method obviously accumulates certificates of all actors across containers. This is OK for some definition of OK. The important thing is that the container has a version of the keystore that contains the certificates for actors in that container. Any extra certificates will not harm but present a potential security loophole. A more involved method would create separate runtime/ directories for each container.

Deploying

  1. Create the actor configuration for the container by editing $HOME/hostX/actor_configs/config.xml for each host. You will need the GUIDs you have issued to the actors. At the bottom of this page is a working example of a configuration file.
  2. To lookup certificates for actors in other containers (if they are already running), consult ORCA actor registry. You can extract them from your own keystores as described here
  3. Edit the $HOME/hostX/ant/build.properties to point to the URL of the container on hostX
  4. Package and deploy for each host (note that Tomcat must be running for the deploy step too succeed as it uses Tomcat Manager webapp to remotely install ORCA webapp into Tomcat).
    $ cd $HOME/hostX
    $ mvn package
    $ ant deploy
    
  5. Check that your actors are properly registered by visiting the ORCA actor registry.

Attachments