Dynamic iptables port-forwarding for NAT-ed libvirt networks

Libvirt is particularly awesome when it comes to managing virtual machines, their underlying storage and networks. However, if you happen to use NAT-ed networking and want to allow external access to services offered by your VMs, you’ve got to do some manual work. The simplest way to get access is to set up some  iptables rules to do port-forwarding. So for quite a while, I had the following in my /etc/init.d/boot.local and things just worked:

HOST_IP=10.120.4.195

function iptables_forward() {
    read HOST_PORT FORWARD_IP FORWARD_PORT BRIDGE <<< $2
    # Drop rules just in case they are already present:
    iptables -t nat -D PREROUTING -p tcp -d $HOST_IP --dport $HOST_PORT -j DNAT --to $FORWARD_IP:$FORWARD_PORT 2> /dev/null
    iptables -D FORWARD -o $BRIDGE -p tcp --dport $FORWARD_PORT -j ACCEPT 2> /dev/null
    echo "iptables_forward: Forward service $1 from $HOST_IP:$HOST_PORT to $FORWARD_IP:$FORWARD_PORT"
    #iptables -t nat -A PREROUTING -p tcp -d $HOST_IP --dport $HOST_PORT -j DNAT --to $FORWARD_IP:$FORWARD_PORT
    #iptables -I FORWARD -o $BRIDGE -p tcp --dport $FORWARD_PORT -j ACCEPT
}

declare -A service
# Declare array of forwarding rules for VM services in the following form:
#service["VM_SERVICE_NAME"]="HOST_PORT FORWARD_IP FORWARD_PORT"

# libvirt 'default' network:
#service["devstack_dashboard"]="1011 192.168.122.100 80 virbr0"
#service["obs_api"]="1011 192.168.122.80 4040 virbr0"
#service["obs_webui"]="1022 192.168.122.80 80 virbr0"
#service["quickstart_crowbar"]="1030 192.168.122.101 3000 virbr0"
#service["quickstart_dashboard"]="1031 192.168.122.101 443 virbr0"
#service["quickstart_chef_webui"]="1032 192.168.122.101 4040 virbr0"
#service["quickstart_ssh"]="1033 192.168.122.101 22 virbr0"

# libvirt 'cloud' network:
service["cloud_crowbar"]="1100 192.168.124.10 3000 virbr1"
service["cloud_dashboard"]="1101 192.168.126.2 80 virbr1"
service["cloud_dashboard_ssl"]="1102 192.168.126.2 443 virbr1"

for key in ${!service[@]} ; do
    iptables_forward "$key" "${service[$key]}"
done

Pretty simple. However, with systemd things got more complicated. Not only is /etc/init.d/boot.local not evaluated anymore, but it likes to fight with libvirt over when to create bridges (and VLANs). Thus I had to manually invoke the script after libvirt was running.  After re-reading libvirt’s awesome documentation, it was clear that this really rather belongs into a hook script. For qemu domains, the script has to be put in /etc/libvirt/hooks and named qemu. I has to comply to this rather simply interface:

/etc/libvirt/hooks/qemu VIR_DOMAIN ACTION ...

Where VIR_DOMAIN is the exact name of the libvirt domain (virtual machine) for which you want to add / remove iptables port-forwarding rules. ACTION is either “start”, “stopped” or “reconnect”. It could be something like this Python script:

#!/usr/bin/python

"""Libvirt port-forwarding hook.

Libvirt hook for setting up iptables port-forwarding rules when using NAT-ed
networking.
"""
__author__ = "Sascha Peilicke <saschpe@gmx.de>"
__version__ = "0.1.1"


import os
import json
import subprocess
import sys


CONFIG_PATH = os.path.dirname(os.path.abspath(__file__))
CONFIG_FILENAME = os.path.join(CONFIG_PATH, "qemu.json")
CONFIG_SCHEMA_FILENAME = os.path.join(CONFIG_PATH, "qemu.schema.json")
IPTABLES_BINARY = subprocess.check_output(["which", "iptables"]).strip()


def host_ip():
    """Returns the default route interface IP (if any).

    In other words, the public IP used to access the virtualization host. It
    is used as default public IP for guest forwarding rules should they not
    specify a different public IP to forward from.
    """
    if not hasattr(host_ip, "_host_ip"):
        cmd = "ip route | grep default | cut -d' ' -f5"
        default_route_interface = subprocess.check_output(cmd, shell=True).decode().strip()
        cmd = "ip addr show {0} | grep -E 'inet .*{0}' | cut -d' ' -f6 | cut -d'/' -f1".format(default_route_interface)
        host_ip._host_ip = subprocess.check_output(cmd, shell=True).decode().strip()
    return host_ip._host_ip


def config(validate=True):
    """Returns the hook configuration.

    Assumes that the file /etc/libvirt/hooks/qemu.json exists and contains
    JSON-formatted configuration data. Optionally tries to validate the
    configuration if the 'jsonschema' module is available.

    Args:
        validate: Use JSON schema validation
    """
    if not hasattr(config, "_conf"):
        with open(CONFIG_FILENAME, "r") as f:
            config._conf = json.load(f)
        if validate:
            # Try schema validation but avoid hard 'jsonschema' requirement:
            try:
                import jsonschema
                with open(CONFIG_SCHEMA_FILENAME, "r") as f:
                    config._schema = json.load(f)
                jsonschema.validate(config._conf,
                                    config._schema,
                                    format_checker=jsonschema.FormatChecker())
            except ImportError:
                pass
    return config._conf


def iptables_forward(action, domain):
    """Set iptables port-forwarding rules based on domain configuration.

    Args:
        action: iptables rule actions (one of '-I', '-A' or '-D')
        domain: Libvirt domain configuration
    """
    public_ip = domain.get("public_ip", host_ip())
    # Iterate over protocols (tcp, udp, icmp, ...)
    for protocol in domain["port_map"]:
        # Iterate over all public/private port pairs for the protocol
        for public_port, private_port in domain["port_map"].get(protocol):
            args = [IPTABLES_BINARY,
                    "-t", "nat", action, "PREROUTING",
                    "-p", protocol,
                    "-d", public_ip, "--dport", str(public_port),
                    "-j", "DNAT", "--to", "{0}:{1}".format(domain["private_ip"], str(private_port))]
            subprocess.call(args)

            args = [IPTABLES_BINARY,
                    "-t", "filter", action, "FORWARD",
                    "-p", protocol,
                    "--dport", str(private_port),
                    "-j", "ACCEPT"]
            if "interface" in domain:
                args += ["-o", domain["interface"]]
            subprocess.call(args)


if __name__ == "__main__": 
    vir_domain, action = sys.argv[1:3]
    domain = config().get(vir_domain)
    if domain is None:
        sys.exit(0)
    if action in ["stopped", "reconnect"]:
        iptables_forward("-D", domain)
    if action in ["start", "reconnect"]:
        iptables_forward("-I", domain)

It has a very simple configuration file that is expected to live at /etc/libvirt/hooks/qemu.json:

{
    "cloud-admin": {
        "interface": "virbr1",
        "private_ip": "192.168.124.10",
        "port_map": { 
            "tcp": [[1100, 3000]],
            "udp": [[1200, 163]]
        }
    },
    "cloud-node1": {
        "interface": "virbr1",
        "private_ip": "192.168.126.2",
        "port_map": {
            "tcp": [[1101, 80],
                    [1102, 443]]
        }
    }
}

With that in place, iptables rules are added and removed when the domain is started / stopped. Pretty neat, huh? You can find the full code together with some tests and documentation on the libvirt-hook-qemu Github repository.