Skip to content

libre / metro

Go to a project
Toggle navigation
Toggle navigation pinning
M

metro

The IPSec-protected back-end server-to-server connection with IPv6 support.

Star
0

IPsec Metro line

The IPSec-protected back-end server-to-server connection with IPv6 support, based on docker and racoon.

Setup

The IPsec Metro Line supports certificate based and pre-shared key based peer authentication, discussed below. If you just want the checklist to get started, there's a usage example available below.

Network configuration

Under the hood, Metro sets up an IPsec connection in transport mode. Since IPsec encapsulation makes the maximum encapsulated packet size smaller, Metro also sets the MSS to 1200 bytes, by inserting an iptables rule into the FORWARD chain in the mangle table during ./metro up. This rule is automatically removed during ./metro down.

Running, stopping, testing

The metro command has several actions available:

  • up [--no-build|--debug|--test|--ignore-alternative-posts|--help]

Makes sure that the image is (re)-built, and runs the container (which handles all the setkey magic, generates racoon configs, etc). The --no-build flag makes metro not rebuild image; the --debug flag makes the container run in foreground, and runs racoon with more verbose output; --ignore-alternative-ports flag makes metro not set-up the alternative ports iptables redirects even if configured using envvars (this is useful for removing redirects once they are not needed without doing this manually using iptables). The --test flag runs tests immediately after bringing metro up.

Flags do not mix.

  • down [--enable-clear-text|--help]

Stops and removes the container, optionally (when --enable-clear-text command-line option is used) clearing the ipsec config, so that servers can communicate even though the tunnel is down (it does this by running setkey in the container -- so there's no need to install ipsec-tools on the host system itself).

  • restart [--debug|--debug|--test|--ignore-alternative-posts|--help]

Runs down, and then up actions. The --debug, --test, and --ignore-alternative-posts flags are passed to up.

  • test

Runs tests. These are meant to test if the config is sane, Internet connectivity is available, peers themselves are accessible, and communication is encrypted.

Configuration

The scripts are configurable via these environment variables:

  • METRO_PARENT_HOSTNAME (default: /etc/parent_hostname or /etc/hostname)

Hostname of the server Metro is being run on; used to decide which server is the "local party" in IPsec configuration. If not explicitly set, taken from /etc/parent_hostname within the container -- host's /etc/hostname is automatically volume-mounted in the METRO_DOCKER_CONTAINER container as /etc/parent_hostname by up.sh and down.sh scripts for this very purpose.

If /etc/parent_hostname is unavailable in the container, /etc/hostname is used. This might be useful when using the container without the scripts (for instance, via docker-compose as part of a larger set-up).

  • METRO_ISAKMP_IKE_PORT (default: 500)
  • METRO_IPSEC_NAT_TRAVERSAL_PORT (default: 4500)

Ports racoon is supposed to listen on, and on which IKE is initiated. These have to be the same across all servers in the metro system. Although racoon itself supports specifying a port per remote (i.e. racoon locally running on port x, remote1 having racoon on port y, configuration on both reflecting that), this is not exposed in the metro system for simplicity's sake.

To facilitate such non-standard configs to some extent, and to help with seamless transition of the whole metro system to a different port should such need arise, *_ALTERNATIVE_PORT envvars are provided.

  • METRO_ISAKMP_IKE_ALTERNATIVE_PORT (default: disabled)
  • METRO_IPSEC_NAT_TRAVERSAL_ALTERNATIVE_PORT (default: disabled)

Alternative ports that will be redirected (using iptables) to ports racoon is listening on.

This is provided to facilitate non-standard configs (two remotes using different ports), and to help with seamless transitioning of the whole metro system from one set of ports to another without bringing down metro on all hosts simultaneously. More details can be found in Ports section.

NOTICE: relevant iptables rules are inserted upon metro up, and removed during metro down (so, metro restart will remove them and then add them again). The rules redirect are always built based on current contents of *_ALTERNATIVE_PORT envvars in config.env. Hence, changin the ports in config.env should only be done between metro down, and metro up; otherwise results can be unpredictable due to old rules being left in place!

  • METRO_DOCKER_IMAGE (default: ipsec_metro)

The name of the docker image that will be created and used by the scripts.

  • METRO_DOCKER_CONTAINER (default: ipsec_metro)

The name of the docker container that will be created and run.

  • METRO_DIR (default: the directory containing the metro script)

The directory containing the whole Metro code.

  • METRO_DOMAIN (default: derived from certificate file names; used only when using cert-based peer auth)

The domain name for all the servers, used in certificates. Server name is <server>.$METRO_DOMAIN. If not set, it will be derived from the names of *.crt files found in METRO_CREDS_DIR directory.

  • METRO_CRT_SERVERS (default: list of servers derived from certificate file names; used only when using cert-based peer auth)

A space-separated list of server names (e.g. server01 server02) to be included in the IPsec Metro system. If not set, it will be derived from the names of *.crt files found in METRO_CREDS_DIR directory.

  • METRO_CREDS_DIR (default: /etc/metro; used only when using cert-based peer auth)

Directory containing credentials files (in particular, *.crt and *.key files) to be used. This directory will be volume-mounted (read-only) into the METRO_DOCKER_CONTAINER and its path passed to the entrypoint script therein.

  • METRO_CA_CERT (default: $METRO_CREDS_DIR/ca.crt; used only when using cert-based peer auth)

Path to the Certificate Authority root certificate file. All server certificates are assumed to be signed with this CA's key.

NOTICE: If using the default settings (METRO_DOMAIN and METRO_CRT_SERVERS derived from server certificates filenames in METRO_CREDS_DIR), make sure that the CA certificate file name is either exactly ca.crt, or METRO_CA_CERT is explicitly set to its full path -- otherwise it will be treated as a server cert file, breaking METRO_DOMAIN and METRO_CRT_SERVERS derivation.

Ports

There are two environment variables controlling which port racoon runs on (METRO_ISAKMP_IKE_PORT and METRO_IPSEC_NAT_TRAVERSAL_PORT), and two additional envvars controlling which ports are additionally redirected to the racoon ports (METRO_ISAKMP_IKE_ALTERNATIVE_PORT and METRO_IPSEC_NAT_TRAVERSAL_ALTERNATIVE_PORT).

If none are set, default ports 500 and 4500 (both UDP) are used. When setting up metro, any two ports can be used.

*_ALTERNATIVE_PORT envvars are provided mainly to make it possbible to move a metro system already in place to a different port gradually. Assuming old ports are 500 and 4500, and new ports are 600 and 4600, the procedure would be:

  1. On each metro node, add METRO_ISAKMP_IKE_ALTERNATIVE_PORT=600 and METRO_IPSEC_NAT_TRAVERSAL_ALTERNATIVE_PORT=4600 to config.env, and restart metro.
  2. On each metro node, bring metro down, then remove *_ALTERNATIVE_PORT variables from config.env, and set METRO_ISAKMP_IKE_PORT and METRO_IPSEC_NAT_TRAVERSAL_PORT to the new port numbers (600, 4600, respectively); bring metro up.

This way there is no downtime of the whole metro system, and interruptions are limited to two short intervals of a few seconds on each host.

If you need to add *_ALTERNATIVE_PORT config, just add it to config.env and restart metro. Removing is a bit more tricky, first restart metro with the --ignore-alternative-ports flag, and only then remove the entries from config.env.

Config files

All the variables can be configured in an config.env file in the main metro directory. If the file exists, metro will load from it environment variables that are defined in it. The format is is just a regular envvar file format, example is provided as the config.example file. The config.env filename is also added to .gitignore.

Cert-based peer authentication

Cert-based peer authentitcation is used when server certificates (*.crt) files are found in METRO_CREDS_DIR. If METRO_DOMAIN is explicitly configured, only *.crt files matching that domain are taken into account. IPsec Metro expects *.crt to have file names in the form of:

servername.domain.name.of.any.length.crt

The servername part cannot contain any dots; domain.name.of.any.length will be used in full as the METRO_DOMAIN; server certificate file names have to end in .crt. These assumptions are as valid for automatic METRO_DOMAIN and METRO_CRT_SERVERS derivation, as for when both of these are explicitly set. Similar scheme is used for key file of a given server:

servername.domain.name.of.any.length.key

At this time, IPsec Metro supports only a single domain. If *.crt files with different domains are found, Metro will exit with an error. If a METRO_DOMAIN is explicitly configured, only certificate files matching that domain will be used, however.

Additionally, for cert-based auth Metro requires a Certificate Authority (CA) certificate file in METRO_CA_CERT. Creating a CA and managing the certificates themselves is however out of Metro's scope.

Pre-shared key peer authentication

Pre-shared key peer authentitcation is used when *.psk files are found in METRO_CREDS_DIR. Any filename ending in *.psk is permissible. The files are expected to be at least two lines, including exactly two non-comment lines. These rules apply:

  • first non-comment line contains a whitespace (space, tab)-separated list of IP addresses
  • second (and last) non-comment line contains the pre-shared key only
  • empty lines are ignored
  • lines starting with # (hash) caracter are considered comments and ignored

For example, these are valid *.psk files:

198.51.100.25 203.0.113.11
some-pre-shared-key

# a comment
198.51.100.25 2001:DB8::1 203.0.113.11

# another comment after an empty line; there's an empty line at the end of the file, too!
some-pre-shared-key



2001:DB8::15 198.51.100.25 203.0.113.11 192.0.2.87 2001:DB8::32 203.0.113.221

some-pre-shared-key

# some comment at the end

The IPs and keys are then assembled into a single racoon psk.txt file (as defined in the "Pre-shared key file" section here), and the PSKs are handled by racoon. A warning will be issued if a single IP (other than the IP of the host Metro is being run on) shows more than once -- in such a case behaviour is undefined.

Usage example

Setting up metro is not complicated. All the steps below should obviously be done on all participating servers. However, please be aware of metro's limitations.

1. Clone the repository

git clone https://git.occrp.org/libre/metro.git
cd metro/

2. Check if config is sane

The defaults should be fine, but if you're using a different METRO_CREDS_DIR, make sure config.env reflects that:

cp config.example config.env
vim config.env

3. Put the certificates or PSK config files in the METRO_CREDS_DIR

Remember that in case of PSK-based auth you should set up the same PSK on all servers participating in a connection. You can have the same PSK for two or more servers. or you can also have different PSKs for different groups of servers communicating with the same server, by having many *.psk files in the METRO_CREDS_DIR.

In case of cert-based auth, all the participating servers have to have the CA certificate, participating server ceritificates, and their own certificate key file.

You can mix PSK- and cert-based auth as you whish, although using both for the same set of servers at the same time doe snot make sense, is not supported, and have undefined results.

4. Fire up metro on all participating servers

./metro up

This will build the docker image, run the docker container, and set everything up.

5. Run tests:

./metro test

If some tests fail the first time around, re-run them (setting-up IPsec takes some time and might go over the metro test timeouts).

6. PROFIT!!1!

You should now have a secure IPsec-based backbone link between the servers.

Limitations

IPsec Metro has the following limitations:

  • it only supports a single (local) domain for cert-based peer authentication.
  • cert-based authentication has to be hostname/domain name based
  • pre-shared key authentication has to be IP (v4 or v6) address based
  • it only supports a single IPv4 and a single IPv6 local adddress

TODO

There's a lot to be done still:

  • make it possible to control te MSS size, and to disable it using envvars
  • detect when to pretty-format BATS output, and when to use the TAP-compatible formatter