Dynamic iptables port-forwarding for NAT-ed libvirt networks

Libvirt is particularly awesome when it comes to managing virtual machines, their underlying storage and networks. However, if you happen to use NAT-ed networking and want to allow external access to services offered by your VMs, you’ve got to do some manual work. The simplest way to get access is to set up some  iptables rules to do port-forwarding. So for quite a while, I had the following in my /etc/init.d/boot.local and things just worked:


function iptables_forward() {
    # Drop rules just in case they are already present:
    iptables -t nat -D PREROUTING -p tcp -d $HOST_IP --dport $HOST_PORT -j DNAT --to $FORWARD_IP:$FORWARD_PORT 2> /dev/null
    iptables -D FORWARD -o $BRIDGE -p tcp --dport $FORWARD_PORT -j ACCEPT 2> /dev/null
    echo "iptables_forward: Forward service $1 from $HOST_IP:$HOST_PORT to $FORWARD_IP:$FORWARD_PORT"
    #iptables -t nat -A PREROUTING -p tcp -d $HOST_IP --dport $HOST_PORT -j DNAT --to $FORWARD_IP:$FORWARD_PORT
    #iptables -I FORWARD -o $BRIDGE -p tcp --dport $FORWARD_PORT -j ACCEPT

declare -A service
# Declare array of forwarding rules for VM services in the following form:

# libvirt 'default' network:
#service["devstack_dashboard"]="1011 80 virbr0"
#service["obs_api"]="1011 4040 virbr0"
#service["obs_webui"]="1022 80 virbr0"
#service["quickstart_crowbar"]="1030 3000 virbr0"
#service["quickstart_dashboard"]="1031 443 virbr0"
#service["quickstart_chef_webui"]="1032 4040 virbr0"
#service["quickstart_ssh"]="1033 22 virbr0"

# libvirt 'cloud' network:
service["cloud_crowbar"]="1100 3000 virbr1"
service["cloud_dashboard"]="1101 80 virbr1"
service["cloud_dashboard_ssl"]="1102 443 virbr1"

for key in ${!service[@]} ; do
    iptables_forward "$key" "${service[$key]}"

Pretty simple. However, with systemd things got more complicated. Not only is /etc/init.d/boot.local not evaluated anymore, but it likes to fight with libvirt over when to create bridges (and VLANs). Thus I had to manually invoke the script after libvirt was running.  After re-reading libvirt’s awesome documentation, it was clear that this really rather belongs into a hook script. For qemu domains, the script has to be put in /etc/libvirt/hooks and named qemu. I has to comply to this rather simply interface:

/etc/libvirt/hooks/qemu VIR_DOMAIN ACTION ...

Where VIR_DOMAIN is the exact name of the libvirt domain (virtual machine) for which you want to add / remove iptables port-forwarding rules. ACTION is either “start”, “stopped” or “reconnect”. It could be something like this Python script:


"""Libvirt port-forwarding hook.

Libvirt hook for setting up iptables port-forwarding rules when using NAT-ed
__author__ = "Sascha Peilicke <saschpe@gmx.de>"
__version__ = "0.1.1"

import os
import json
import subprocess
import sys

CONFIG_PATH = os.path.dirname(os.path.abspath(__file__))
CONFIG_FILENAME = os.path.join(CONFIG_PATH, "qemu.json")
CONFIG_SCHEMA_FILENAME = os.path.join(CONFIG_PATH, "qemu.schema.json")
IPTABLES_BINARY = subprocess.check_output(["which", "iptables"]).strip()

def host_ip():
    """Returns the default route interface IP (if any).

    In other words, the public IP used to access the virtualization host. It
    is used as default public IP for guest forwarding rules should they not
    specify a different public IP to forward from.
    if not hasattr(host_ip, "_host_ip"):
        cmd = "ip route | grep default | cut -d' ' -f5"
        default_route_interface = subprocess.check_output(cmd, shell=True).decode().strip()
        cmd = "ip addr show {0} | grep -E 'inet .*{0}' | cut -d' ' -f6 | cut -d'/' -f1".format(default_route_interface)
        host_ip._host_ip = subprocess.check_output(cmd, shell=True).decode().strip()
    return host_ip._host_ip

def config(validate=True):
    """Returns the hook configuration.

    Assumes that the file /etc/libvirt/hooks/qemu.json exists and contains
    JSON-formatted configuration data. Optionally tries to validate the
    configuration if the 'jsonschema' module is available.

        validate: Use JSON schema validation
    if not hasattr(config, "_conf"):
        with open(CONFIG_FILENAME, "r") as f:
            config._conf = json.load(f)
        if validate:
            # Try schema validation but avoid hard 'jsonschema' requirement:
                import jsonschema
                with open(CONFIG_SCHEMA_FILENAME, "r") as f:
                    config._schema = json.load(f)
            except ImportError:
    return config._conf

def iptables_forward(action, domain):
    """Set iptables port-forwarding rules based on domain configuration.

        action: iptables rule actions (one of '-I', '-A' or '-D')
        domain: Libvirt domain configuration
    public_ip = domain.get("public_ip", host_ip())
    # Iterate over protocols (tcp, udp, icmp, ...)
    for protocol in domain["port_map"]:
        # Iterate over all public/private port pairs for the protocol
        for public_port, private_port in domain["port_map"].get(protocol):
            args = [IPTABLES_BINARY,
                    "-t", "nat", action, "PREROUTING",
                    "-p", protocol,
                    "-d", public_ip, "--dport", str(public_port),
                    "-j", "DNAT", "--to", "{0}:{1}".format(domain["private_ip"], str(private_port))]

            args = [IPTABLES_BINARY,
                    "-t", "filter", action, "FORWARD",
                    "-p", protocol,
                    "--dport", str(private_port),
                    "-j", "ACCEPT"]
            if "interface" in domain:
                args += ["-o", domain["interface"]]

if __name__ == "__main__": 
    vir_domain, action = sys.argv[1:3]
    domain = config().get(vir_domain)
    if domain is None:
    if action in ["stopped", "reconnect"]:
        iptables_forward("-D", domain)
    if action in ["start", "reconnect"]:
        iptables_forward("-I", domain)

It has a very simple configuration file that is expected to live at /etc/libvirt/hooks/qemu.json:

    "cloud-admin": {
        "interface": "virbr1",
        "private_ip": "",
        "port_map": { 
            "tcp": [[1100, 3000]],
            "udp": [[1200, 163]]
    "cloud-node1": {
        "interface": "virbr1",
        "private_ip": "",
        "port_map": {
            "tcp": [[1101, 80],
                    [1102, 443]]

With that in place, iptables rules are added and removed when the domain is started / stopped. Pretty neat, huh? You can find the full code together with some tests and documentation on the libvirt-hook-qemu Github repository.

About these ads

7 thoughts on “Dynamic iptables port-forwarding for NAT-ed libvirt networks

  1. Nice, thank you – works beautifully for 2 guests on Debian Jessie… but I’m sure the guests will multiply ;)

  2. Hi, the idea is plain awesome! Unfortunately, it seems that the rules aren’t added by iptables on CentOS Linux release 7.0.1406 (Core) and after staring at terminals all day, my patience is running kind of low with this personal project of mine.

    # cat /etc/libvirt/hooks/qemu.json
    “web”: {
    “interface”: “virbr0″,
    “private_ip”: “”,
    “port_map”: {
    “tcp”: [[2210, 22]]
    “db”: {
    “interface”: “virbr0″,
    “private_ip”: “”,
    “port_map”: {
    “tcp”: [[2211, 22]]

    I did a
    iptables-save > /tmp/iptables-rules

    and it looks that the rules aren’t added:
    # grep -c ‘0.10’ /tmp/iptables-rules

    I tried
    # ./qemu web-rhel7-64 attach begin –
    as seen in the libvirt documentation, still with no luck.

    The test also doesn’t run. Can it be due to the fact that I’ve got
    # python -V
    Python 2.7.5

    # python test_qemu.py
    Traceback (most recent call last):
    File “test_qemu.py”, line 7, in
    import qemu # Our local libvirt hooks module
    ImportError: No module named qemu


  3. The hook script doesn’t use any libvirt Python bindings and it should work just fine with 2.7.5. Dunno much about Ubuntu’s libvirt but it seems like the hook isn’t invoked at all. Maybe it’s supposed to reside in another location or maybe libvirt is just to old.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s