Setting up SSH proxy tunneling

This document is a collection of notes on creating SSH proxy tunnels for reaching substrate (e.g. VMs) that are located in private address spaces from the public address spaces

SSH Reverse Tunnels

SSH tunneling presents a generic mechanism for making TCP-based services located in private address spaces available securely to hosts located in public address space. This situation arises when, for example, VMs are created on computational substrate that is behind a NAT firewall and external experimenter tools need to connect to these VMs to get e.g. a command shell.

The proposed setup involves a domain with a NAT firewall and a internal private address space. The hosts inside the domain can originate connections to the public Internet. One or more hosts with public IP addresses must be chosen to be SSH 'proxies'. It could be the NAT host or could be a different host, depending on hardware. The idea is to provide access to hosts inside the domain (internal hosts) by tunneling their SSH connections from the SSH proxy. Since NAT generally prevents incoming connections and the internal hosts can be dynamically created, it is more convenient to have these hosts open a 'reverse' SSH tunnel to the proxy host, rather than having the proxy host open a forward tunnel to the internal hosts.

The control framework must treat reverse tunnel ports (endpoints) as an allocatable resource.

SSH keys

(At least) Two sets of keys must be used in this scenario:

  1. User keys - needed to provide the user access to the internal host. The user public key must be installed on the internal host (e.g. .ssh/authorized_keys for root user). This is normally done by the control framework even today
  2. Proxy keys - needed to establish a connection between the internal host and a proxy. This SSH keypair(s) must be generated ahead of time, the public key installed in the .ssh/authorized_keys for the account used on the proxy host for SSH tunneling. The private key must be installed into the internal host. If it is a VM it can be installed at the time of VM creation. Since this private key becomes known to the user due to being installed into the VM, it may be necessary to generate a per-user/per slice dynamic key pair used for all the hosts in the user's slice.

SSHD Configuration

By default SSHD binds reverse tunnel connections on the proxyhost to 'localhost' only as a security measure. The following stanza must be added to the proxy hosts /etc/ssh/sshd_config file (and sshd restarted) to allow clients to specify the bind address for the tunnel remote endpoint:

# to allow reverse tunnels to work
GatewayPorts clientspecified

Control Framework Workflow

  1. Generate a proxy SSH keypair.
  2. Install the public key into the proxy host account used for reverse tunnels (call it rev-tunnel-user):
    $ cat newkey >> ~rev-tunnel-user/.ssh/authorized_keys
    
  3. Boot the internal host (VM)
  4. Install the user public key into the root account
  5. Install the proxy key into the root account
  6. Allocate the port on the proxy host (assume port 12345)
  7. Create a reverse tunnel (in this example for SSH, replace 22 with a port of a different service as needed; to limit the interfaces on which port 12345 is bound to the tunnel replace 0.0.0.0 with the address of the specific public interface on the proxy) to the proxy host using its private address/name:
    $ ssh -nNT -R 0.0.0.0:12345:localhost:22 rev-tunnel-user@proxy.host.name.priv
    
  8. Now the internal host will be accessible via SSH on proxy.host.name.pub (proxy's public name/ip) on port 12345. For example to ssh to the new host from outside, the user can simply
    $ ssh -i user.key -p 12345 root@proxy.host.name.pub