IPsec Metro line
The IPSec-protected back-end server-to-server connection with IPv6 support, based on
The IPsec Metro Line supports certificate based and pre-shared key based peer authentication, discussed below. If you just want the checklist to get started, there's a usage example available below.
Under the hood, Metro sets up an IPsec connection in transport mode. Since IPsec encapsulation makes the maximum encapsulated packet size smaller, Metro also sets the MSS to 1200 bytes, by inserting an
iptables rule into the
FORWARD chain in the
mangle table during
./metro up. This rule is automatically removed during
Running, stopping, testing
metro command has several actions available:
Makes sure that the image is (re)-built, and runs the container (which handles all the
setkey magic, generates
racoon configs, etc). The
--no-build flag makes metro not rebuild image; the
--debug flag makes the container run in foreground, and runs
racoon with more verbose output;
--ignore-alternative-ports flag makes metro not set-up the alternative ports
iptables redirects even if configured using envvars (this is useful for removing redirects once they are not needed without doing this manually using iptables). The
--test flag runs tests immediately after bringing metro up.
Flags do not mix.
Stops and removes the container, optionally (when
--enable-clear-text command-line option is used) clearing the
ipsec config, so that servers can communicate even though the tunnel is down (it does this by running
setkey in the container -- so there's no need to install
ipsec-tools on the host system itself).
down, and then
up actions. The
--ignore-alternative-posts flags are passed to
Runs tests. These are meant to test if the config is sane, Internet connectivity is available, peers themselves are accessible, and communication is encrypted.
The scripts are configurable via these environment variables:
Hostname of the server Metro is being run on; used to decide which server is the "local party" in
IPsec configuration. If not explicitly set, taken from
/etc/parent_hostname within the container -- host's
/etc/hostname is automatically volume-mounted in the
METRO_DOCKER_CONTAINER container as
down.sh scripts for this very purpose.
/etc/parent_hostname is unavailable in the container,
/etc/hostname is used. This might be useful when using the container without the scripts (for instance, via
docker-compose as part of a larger set-up).
racoon is supposed to listen on, and on which IKE is initiated. These have to be the same across all servers in the metro system. Although
racoon itself supports specifying a port per remote (i.e.
racoon locally running on port
remote1 having racoon on port
y, configuration on both reflecting that), this is not exposed in the metro system for simplicity's sake.
To facilitate such non-standard configs to some extent, and to help with seamless transition of the whole metro system to a different port should such need arise,
*_ALTERNATIVE_PORT envvars are provided.
Alternative ports that will be redirected (using
iptables) to ports
racoon is listening on.
This is provided to facilitate non-standard configs (two remotes using different ports), and to help with seamless transitioning of the whole metro system from one set of ports to another without bringing down metro on all hosts simultaneously. More details can be found in Ports section.
iptables rules are inserted upon
metro up, and removed during
metro down (so,
metro restart will remove them and then add them again). The rules redirect are always built based on current contents of
*_ALTERNATIVE_PORT envvars in
config.env. Hence, changin the ports in
config.env should only be done between
metro down, and
metro up; otherwise results can be unpredictable due to old rules being left in place!
The name of the docker image that will be created and used by the scripts.
The name of the docker container that will be created and run.
METRO_DIR(default: the directory containing the
The directory containing the whole Metro code.
METRO_DOMAIN(default: derived from certificate file names; used only when using cert-based peer auth)
The domain name for all the servers, used in certificates. Server name is
<server>.$METRO_DOMAIN. If not set, it will be derived from the names of
*.crt files found in
METRO_CRT_SERVERS(default: list of servers derived from certificate file names; used only when using cert-based peer auth)
A space-separated list of server names (e.g.
server01 server02) to be included in the IPsec Metro system. If not set, it will be derived from the names of
*.crt files found in
/etc/metro; used only when using cert-based peer auth)
Directory containing credentials files (in particular,
*.key files) to be used. This directory will be volume-mounted (read-only) into the
METRO_DOCKER_CONTAINER and its path passed to the entrypoint script therein.
$METRO_CREDS_DIR/ca.crt; used only when using cert-based peer auth)
Path to the Certificate Authority root certificate file. All server certificates are assumed to be signed with this CA's key.
NOTICE: If using the default settings (
METRO_CRT_SERVERS derived from server certificates filenames in
METRO_CREDS_DIR), make sure that the CA certificate file name is either exactly
METRO_CA_CERT is explicitly set to its full path -- otherwise it will be treated as a server cert file, breaking
There are two environment variables controlling which port
racoon runs on (
METRO_IPSEC_NAT_TRAVERSAL_PORT), and two additional envvars controlling which ports are additionally redirected to the
racoon ports (
If none are set, default ports
4500 (both UDP) are used. When setting up metro, any two ports can be used.
*_ALTERNATIVE_PORT envvars are provided mainly to make it possbible to move a metro system already in place to a different port gradually. Assuming old ports are
4500, and new ports are
4600, the procedure would be:
- On each metro node, add
config.env, and restart metro.
- On each metro node, bring metro down, then remove
config.env, and set
METRO_IPSEC_NAT_TRAVERSAL_PORTto the new port numbers (
4600, respectively); bring metro up.
This way there is no downtime of the whole metro system, and interruptions are limited to two short intervals of a few seconds on each host.
If you need to add
*_ALTERNATIVE_PORT config, just add it to
config.env and restart metro. Removing is a bit more tricky, first restart metro with the
--ignore-alternative-ports flag, and only then remove the entries from
All the variables can be configured in an
config.env file in the main metro directory. If the file exists, metro will load from it environment variables that are defined in it. The format is is just a regular envvar file format, example is provided as the
config.example file. The
config.env filename is also added to
Cert-based peer authentication
Cert-based peer authentitcation is used when server certificates (
*.crt) files are found in
METRO_DOMAIN is explicitly configured, only
*.crt files matching that domain are taken into account. IPsec Metro expects
*.crt to have file names in the form of:
servername part cannot contain any dots;
domain.name.of.any.length will be used in full as the
METRO_DOMAIN; server certificate file names have to end in
.crt. These assumptions are as valid for automatic
METRO_CRT_SERVERS derivation, as for when both of these are explicitly set. Similar scheme is used for key file of a given server:
At this time, IPsec Metro supports only a single domain. If
*.crt files with different domains are found, Metro will exit with an error. If a
METRO_DOMAIN is explicitly configured, only certificate files matching that domain will be used, however.
Additionally, for cert-based auth Metro requires a Certificate Authority (CA) certificate file in
METRO_CA_CERT. Creating a CA and managing the certificates themselves is however out of Metro's scope.
Pre-shared key peer authentication
Pre-shared key peer authentitcation is used when
*.psk files are found in
METRO_CREDS_DIR. Any filename ending in
*.psk is permissible. The files are expected to be at least two lines, including exactly two non-comment lines. These rules apply:
- first non-comment line contains a whitespace (space, tab)-separated list of IP addresses
- second (and last) non-comment line contains the pre-shared key only
- empty lines are ignored
- lines starting with # (hash) caracter are considered comments and ignored
For example, these are valid
198.51.100.25 203.0.113.11 some-pre-shared-key
# a comment 198.51.100.25 2001:DB8::1 203.0.113.11 # another comment after an empty line; there's an empty line at the end of the file, too! some-pre-shared-key
2001:DB8::15 198.51.100.25 203.0.113.11 192.0.2.87 2001:DB8::32 203.0.113.221 some-pre-shared-key # some comment at the end
The IPs and keys are then assembled into a single
psk.txt file (as defined in the "Pre-shared key file" section here), and the PSKs are handled by
racoon. A warning will be issued if a single IP (other than the IP of the host Metro is being run on) shows more than once -- in such a case behaviour is undefined.
Setting up metro is not complicated. All the steps below should obviously be done on all participating servers. However, please be aware of metro's limitations.
1. Clone the repository
git clone https://git.occrp.org/libre/metro.git cd metro/
2. Check if config is sane
The defaults should be fine, but if you're using a different
METRO_CREDS_DIR, make sure
config.env reflects that:
cp config.example config.env vim config.env
3. Put the certificates or PSK config files in the
Remember that in case of PSK-based auth you should set up the same PSK on all servers participating in a connection. You can have the same PSK for two or more servers. or you can also have different PSKs for different groups of servers communicating with the same server, by having many
*.psk files in the
In case of cert-based auth, all the participating servers have to have the CA certificate, participating server ceritificates, and their own certificate key file.
You can mix PSK- and cert-based auth as you whish, although using both for the same set of servers at the same time doe snot make sense, is not supported, and have undefined results.
4. Fire up metro on all participating servers
This will build the docker image, run the docker container, and set everything up.
5. Run tests:
If some tests fail the first time around, re-run them (setting-up IPsec takes some time and might go over the metro test timeouts).
You should now have a secure IPsec-based backbone link between the servers.
IPsec Metro has the following limitations:
- it only supports a single (local) domain for cert-based peer authentication.
- cert-based authentication has to be hostname/domain name based
- pre-shared key authentication has to be IP (v4 or v6) address based
- it only supports a single IPv4 and a single IPv6 local adddress
There's a lot to be done still: