Dynamic iptables port-forwarding for NAT-ed libvirt networks

Libvirt is particularly awesome when it comes to managing virtual machines, their underlying storage and networks. However, if you happen to use NAT-ed networking and want to allow external access to services offered by your VMs, you’ve got to do some manual work. The simplest way to get access is to set up some  iptables rules to do port-forwarding. So for quite a while, I had the following in my /etc/init.d/boot.local and things just worked:

HOST_IP=10.120.4.195

function iptables_forward() {
    read HOST_PORT FORWARD_IP FORWARD_PORT BRIDGE <<< $2
    # Drop rules just in case they are already present:
    iptables -t nat -D PREROUTING -p tcp -d $HOST_IP --dport $HOST_PORT -j DNAT --to $FORWARD_IP:$FORWARD_PORT 2> /dev/null
    iptables -D FORWARD -o $BRIDGE -p tcp --dport $FORWARD_PORT -j ACCEPT 2> /dev/null
    echo "iptables_forward: Forward service $1 from $HOST_IP:$HOST_PORT to $FORWARD_IP:$FORWARD_PORT"
    #iptables -t nat -A PREROUTING -p tcp -d $HOST_IP --dport $HOST_PORT -j DNAT --to $FORWARD_IP:$FORWARD_PORT
    #iptables -I FORWARD -o $BRIDGE -p tcp --dport $FORWARD_PORT -j ACCEPT
}

declare -A service
# Declare array of forwarding rules for VM services in the following form:
#service["VM_SERVICE_NAME"]="HOST_PORT FORWARD_IP FORWARD_PORT"

# libvirt 'default' network:
#service["devstack_dashboard"]="1011 192.168.122.100 80 virbr0"
#service["obs_api"]="1011 192.168.122.80 4040 virbr0"
#service["obs_webui"]="1022 192.168.122.80 80 virbr0"
#service["quickstart_crowbar"]="1030 192.168.122.101 3000 virbr0"
#service["quickstart_dashboard"]="1031 192.168.122.101 443 virbr0"
#service["quickstart_chef_webui"]="1032 192.168.122.101 4040 virbr0"
#service["quickstart_ssh"]="1033 192.168.122.101 22 virbr0"

# libvirt 'cloud' network:
service["cloud_crowbar"]="1100 192.168.124.10 3000 virbr1"
service["cloud_dashboard"]="1101 192.168.126.2 80 virbr1"
service["cloud_dashboard_ssl"]="1102 192.168.126.2 443 virbr1"

for key in ${!service[@]} ; do
    iptables_forward "$key" "${service[$key]}"
done

Pretty simple. However, with systemd things got more complicated. Not only is /etc/init.d/boot.local not evaluated anymore, but it likes to fight with libvirt over when to create bridges (and VLANs). Thus I had to manually invoke the script after libvirt was running.  After re-reading libvirt’s awesome documentation, it was clear that this really rather belongs into a hook script. For qemu domains, the script has to be put in /etc/libvirt/hooks and named qemu. I has to comply to this rather simply interface:

/etc/libvirt/hooks/qemu VIR_DOMAIN ACTION ...

Where VIR_DOMAIN is the exact name of the libvirt domain (virtual machine) for which you want to add / remove iptables port-forwarding rules. ACTION is either “start”, “stopped” or “reconnect”. It could be something like this Python script:

#!/usr/bin/python

"""Libvirt port-forwarding hook.

Libvirt hook for setting up iptables port-forwarding rules when using NAT-ed
networking.
"""
__author__ = "Sascha Peilicke <saschpe@gmx.de>"
__version__ = "0.1.1"


import os
import json
import subprocess
import sys


CONFIG_PATH = os.path.dirname(os.path.abspath(__file__))
CONFIG_FILENAME = os.path.join(CONFIG_PATH, "qemu.json")
CONFIG_SCHEMA_FILENAME = os.path.join(CONFIG_PATH, "qemu.schema.json")
IPTABLES_BINARY = subprocess.check_output(["which", "iptables"]).strip()


def host_ip():
    """Returns the default route interface IP (if any).

    In other words, the public IP used to access the virtualization host. It
    is used as default public IP for guest forwarding rules should they not
    specify a different public IP to forward from.
    """
    if not hasattr(host_ip, "_host_ip"):
        cmd = "ip route | grep default | cut -d' ' -f5"
        default_route_interface = subprocess.check_output(cmd, shell=True).decode().strip()
        cmd = "ip addr show {0} | grep -E 'inet .*{0}' | cut -d' ' -f6 | cut -d'/' -f1".format(default_route_interface)
        host_ip._host_ip = subprocess.check_output(cmd, shell=True).decode().strip()
    return host_ip._host_ip


def config(validate=True):
    """Returns the hook configuration.

    Assumes that the file /etc/libvirt/hooks/qemu.json exists and contains
    JSON-formatted configuration data. Optionally tries to validate the
    configuration if the 'jsonschema' module is available.

    Args:
        validate: Use JSON schema validation
    """
    if not hasattr(config, "_conf"):
        with open(CONFIG_FILENAME, "r") as f:
            config._conf = json.load(f)
        if validate:
            # Try schema validation but avoid hard 'jsonschema' requirement:
            try:
                import jsonschema
                with open(CONFIG_SCHEMA_FILENAME, "r") as f:
                    config._schema = json.load(f)
                jsonschema.validate(config._conf,
                                    config._schema,
                                    format_checker=jsonschema.FormatChecker())
            except ImportError:
                pass
    return config._conf


def iptables_forward(action, domain):
    """Set iptables port-forwarding rules based on domain configuration.

    Args:
        action: iptables rule actions (one of '-I', '-A' or '-D')
        domain: Libvirt domain configuration
    """
    public_ip = domain.get("public_ip", host_ip())
    # Iterate over protocols (tcp, udp, icmp, ...)
    for protocol in domain["port_map"]:
        # Iterate over all public/private port pairs for the protocol
        for public_port, private_port in domain["port_map"].get(protocol):
            args = [IPTABLES_BINARY,
                    "-t", "nat", action, "PREROUTING",
                    "-p", protocol,
                    "-d", public_ip, "--dport", str(public_port),
                    "-j", "DNAT", "--to", "{0}:{1}".format(domain["private_ip"], str(private_port))]
            subprocess.call(args)

            args = [IPTABLES_BINARY,
                    "-t", "filter", action, "FORWARD",
                    "-p", protocol,
                    "--dport", str(private_port),
                    "-j", "ACCEPT"]
            if "interface" in domain:
                args += ["-o", domain["interface"]]
            subprocess.call(args)


if __name__ == "__main__": 
    vir_domain, action = sys.argv[1:3]
    domain = config().get(vir_domain)
    if domain is None:
        sys.exit(0)
    if action in ["stopped", "reconnect"]:
        iptables_forward("-D", domain)
    if action in ["start", "reconnect"]:
        iptables_forward("-I", domain)

It has a very simple configuration file that is expected to live at /etc/libvirt/hooks/qemu.json:

{
    "cloud-admin": {
        "interface": "virbr1",
        "private_ip": "192.168.124.10",
        "port_map": { 
            "tcp": [[1100, 3000]],
            "udp": [[1200, 163]]
        }
    },
    "cloud-node1": {
        "interface": "virbr1",
        "private_ip": "192.168.126.2",
        "port_map": {
            "tcp": [[1101, 80],
                    [1102, 443]]
        }
    }
}

With that in place, iptables rules are added and removed when the domain is started / stopped. Pretty neat, huh? You can find the full code together with some tests and documentation on the libvirt-hook-qemu Github repository.

12 thoughts on “Dynamic iptables port-forwarding for NAT-ed libvirt networks

  1. Nice, thank you – works beautifully for 2 guests on Debian Jessie… but I’m sure the guests will multiply ;)

  2. Hi, the idea is plain awesome! Unfortunately, it seems that the rules aren’t added by iptables on CentOS Linux release 7.0.1406 (Core) and after staring at terminals all day, my patience is running kind of low with this personal project of mine.

    # cat /etc/libvirt/hooks/qemu.json
    {
    “web”: {
    “interface”: “virbr0”,
    “private_ip”: “10.0.0.10”,
    “port_map”: {
    “tcp”: [[2210, 22]]
    }
    },
    “db”: {
    “interface”: “virbr0”,
    “private_ip”: “10.0.0.11”,
    “port_map”: {
    “tcp”: [[2211, 22]]
    }
    }
    }

    I did a
    iptables-save > /tmp/iptables-rules

    and it looks that the rules aren’t added:
    # grep -c ‘0.10’ /tmp/iptables-rules
    0

    I tried
    # ./qemu web-rhel7-64 attach begin –
    as seen in the libvirt documentation, still with no luck.

    The test also doesn’t run. Can it be due to the fact that I’ve got
    # python -V
    Python 2.7.5
    installed?

    # python test_qemu.py
    Traceback (most recent call last):
    File “test_qemu.py”, line 7, in
    import qemu # Our local libvirt hooks module
    ImportError: No module named qemu

    Thanks!

  3. The hook script doesn’t use any libvirt Python bindings and it should work just fine with 2.7.5. Dunno much about Ubuntu’s libvirt but it seems like the hook isn’t invoked at all. Maybe it’s supposed to reside in another location or maybe libvirt is just to old.

  4. I have a problem with this script, it opens ports on my public IP and overrides my ufw (uncomplicated firewall) rules. Now I can not restrict access to certain IP addresses, everybody gets access.

    How come it does not just set the forwarding on virbr0 but also on em1 (Ubuntu 14 Server)?

    {
    “Maschine1”: {
    “interface”: “virbr0”,
    “private_ip”: “192.168.122.3”,
    “port_map”: {
    “tcp”: [[443, 443],
    [445, 445],
    [6262, 6262],
    [3389, 3389],
    [25856, 25856]]
    }
    },
    “Maschine2”: {
    “interface”: “virbr0”,
    “private_ip”: “192.168.122.2”,
    “port_map”: {
    “tcp”: [[53, 53],
    [33893, 33893]]
    }
    }
    }

  5. Is it a bad idea to modify the qemu script, adding a line like that (see line ADDED):

    “””Set iptables port-forwarding rules based on domain configuration.
    Args:
    action: iptables rule actions (one of ‘-I’, ‘-A’ or ‘-D’)
    domain: Libvirt domain configuration
    “””
    public_ip = domain.get(“public_ip”, host_ip())
    # Iterate over protocols (tcp, udp, icmp, …)
    for protocol in domain[“port_map”]:
    # Iterate over all public/private port pairs for the protocol
    for public_port, private_port in domain[“port_map”].get(protocol):
    args = [IPTABLES_BINARY,
    “-t”, “nat”, action, “PREROUTING”,
    “-p”, protocol,
    “-d”, public_ip, “–dport”, str(public_port),
    “-j”, “DNAT”, “–to”, “{0}:{1}”.format(domain[“private_ip”], str(private_port))]
    subprocess.call(args)

    args = [IPTABLES_BINARY,
    “-t”, “filter”, action, “FORWARD”,
    “-p”, protocol,
    “–dport”, str(private_port),
    “-j”, “ACCEPT”]
    if “interface” in domain:
    args += [“-o”, domain[“interface”]]
    ADDED args += [“-i”, domain[“interface”]]
    subprocess.call(args)

  6. Well that’s exactly the purpose. For example, I do have a big workstation hidden somewhere under my desk which runs all my VMs 24/7. I regularly directly ssh into them. Furthermore most of those VMs offer services that I want to be accessible publically (e.g. a webserver). For that to work, I obviously have to map ports on the public IP of the host machine to ports on the VMs. The script tries to be defensive when modifying iptables rules but I don’t know how ufw works and what iptables you have set up. Feel free to modify the script though (as you already seem to have).

  7. Well, port numbers depend on the services you want to offer with your VMs. For instance, if you have a webserver VM listening for HTTP and HTTPs connection, you’ve got to forward port 80 and 443 (see Wikipedia). Which port you use on the host is a matter of taste, but if you have only one VM listening on port 80, it makes a lot of sense to just forward the hosts port 80 to the guest one.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s