Firewall Rules as Code: The Migration Trap

Moving host firewall rules to a new mechanism can silently lose them during clean-up. How that happens and which order of steps is safe.

Host firewalls on Linux servers are increasingly treated like application code: the rules live in a repository, changes go through a review, a tool rolls them out. That is right – and it sets a trap during the switchover that is easy to overlook. At the end stands a server that behaves completely normally and yet no longer filters anything.

This article explains the mechanism using nftables, the packet filtering framework of today’s Linux distributions, as an example, describes a safe order for the migration and adds two further pitfalls: rules derived from open sockets, and container ports that bypass the host firewall.

How the distribution service works

With the nftables package, Debian ships a systemd service nftables.service. Its unit file is short and unambiguous: on start and on reload it loads the file /etc/nftables.conf with nft -f. On stop it runs nft flush ruleset.

Added to this is the shipped configuration file, whose first statement is flush ruleset. The nftables wiki describes what this command does: it clears the entire ruleset – “flush (erase, delete, wipe) the complete ruleset”. This means not just the tables this service created, but everything loaded in the kernel.

For a server on which only this service manages rules, that is a sensible design: the ruleset is completely replaced on every load, and no remnants of earlier states are left behind.

The trap when switching mechanisms

It becomes problematic as soon as a second mechanism is added. A typical sequence of events: in future, the rules are to come from a repository and be loaded via configuration management or a dedicated service, in a table of their own. The new ruleset is loaded, checked and found to be good. The old service is left in place, since it does no harm.

At some point afterwards, one of three things happens:

  • Someone stops the old service, for example while cleaning up. The stop clears the entire ruleset – including the new table. The host is then unfiltered.
  • Someone restarts or reloads it. The old file is loaded, and its flush ruleset removes everything not contained in it. The host runs again with the state that had been replaced.
  • Someone loads the old file by hand to “quickly test something” – with the same result.

In none of these cases is there an error message. Services stay reachable, applications work, monitoring stays green. That is exactly what makes the trap dangerous: the usual tests after a change check whether something works – not whether something is blocked.

The safe order

The solution is not a new technique but an order of steps:

  1. Disable or mask the old service without stopping it. The systemctl man page is clear here: disable does not stop the unit; only the --now option does that. mask goes further and makes any start impossible, including a manual one. A stop would be precisely the command that clears the ruleset.
  2. Load the new ruleset – in a table of its own, from the repository, via the mechanism that will manage it in future.
  3. Read what is actually loaded. nft list tables shows which tables exist; nft list ruleset shows the complete state. In this phase the old and the new table should be side by side. Both apply at the same time – keep this phase short.
  4. Prove the filtering from outside – more on this in a moment.
  5. Only then remove the old table deliberately, with nft delete table, not by stopping the old service. The old configuration file is archived or changed so that loading it by accident no longer clears anything.
  6. After the next reboot, read again whether the new ruleset is loaded and the old one does not return.

One detail deserves attention: disabling and masking prevent another start, but they do not end the running state. Because the unit is set up as a one-shot service with RemainAfterExit=yes, systemd continues to show it as active until the next reboot – and its stop command remains registered. Document this as a known risk.

The negative probe from outside

After every firewall change, the check is whether permitted services are reachable. That proves nothing is broken. It does not prove that filtering takes place – a host with no rules at all passes this test just as well.

That is why every change needs a negative probe: a connection attempt from a source outside the host that must be blocked, to a port that must be closed. Only when this attempt fails is the effect proven. “Outside” is to be taken literally: connections a host makes to itself run over the loopback interface, which rulesets usually allow across the board.

Repeat the probe regularly, not only on changes: the trap described here can snap shut weeks later.

Rules from the repository, not from the shell

Ad hoc commands on the console are the natural adversary of any rules-as-code management. A rule that exists only in the running kernel disappears on the next load from the repository – or it survives where nobody expects it, because someone exports the running state to a file. Both make the ruleset unpredictable. The rule is simple: what is not in the reviewed repository does not belong on the host. Emergency changes are followed up in the repository, and a regular comparison between the repository and nft list ruleset makes deviations visible.

Not every open socket needs a rule

When building a new ruleset, it is tempting to work through the list of a host’s open sockets and create a permit for each one. That leads to unnecessary openings. One example is a UDP socket that a logging service uses only for sending, such as when forwarding syslog messages to a central server. It shows up in the socket list but needs no inbound rule.

Derive permits from the purpose: which connections should the host accept, and from whom? The socket list is a tool for checking, not the template.

Container ports bypass INPUT

The third pitfall concerns hosts running Docker. According to the Docker documentation, published ports are bound to all of the host’s addresses by default; publishing is “insecure by default”. The packets go through a destination NAT redirect and are handed on in the FORWARD chain to Docker’s own chains. For UFW, the documentation states the consequence explicitly: Docker routes container traffic in the nat table before it reaches the INPUT and OUTPUT chains that UFW uses. A carefully maintained INPUT rule therefore does not see a published container port.

If a service in a container is to be reachable only locally, bind it explicitly to 127.0.0.1, for example with -p 127.0.0.1:8080:80. Then, according to the documentation, only the host itself can access it – with one caveat: before Docker 28.0.0, hosts in the same layer 2 segment could also reach ports published to localhost. For filter rules that are also meant to apply in front of containers: with the default iptables backend, Docker provides the DOCKER-USER chain for this; with the experimental nftables backend, it does not exist. Published container ports also belong in every negative probe.

How sector7 supports you

We harden operating systems and standard services and build clean, traceable rulesets with you – more under cyber security. Recurring, automated vulnerability scans check your attack surface externally and systems and services internally; they show when a port is open that should not be. As part of our managed network services, we also take on change management.

Sources

Let's talk about your situation.