ANDREW/
MenuWriting
← Back to writing

TECHNOLOGY / REMOTE FIREWALL

New locks. Now IT can't get in.

Andrew Yuan · Engineering experiment

Imagine a company replacing all its access locks.
The new locks are installed. The administrator cannot get in.
“Ask IT to come in and fix it.”
“IT is outside too.”
Security has been very successful.
Even the people responsible for repairs are locked out.

I've been continuing work on Rulemoor, an experimental service for remote Linux firewall changes.
The question is simple to state: after changing the rules, can we still reach the management channel? If something goes wrong, how do we recover?

Two doors labeled IPv4 and IPv6 with separate badge readers; an access badge lies outside. Headline: Check before you apply. Can you recover?

Preflight: reject a known management-port block

First, check before applying.
Optional management-port preflight evaluates whether a candidate policy would block declared local IPv4/IPv6 TCP management ports. A known block in Rulemoor's own policy rejects the new change.
External nftables rules are inspected and diagnosed, not rewritten. Uncertain conditions remain risks; “no problem detected” is not a promise of reachability.

Confirmation: step outside and try your badge

Second, reconnect after applying.
A change has a confirmation deadline, and confirmation must use a new TLS management connection established after the rules were applied.
An existing connection staying alive does not prove a new connection can get in.
Imagine a manager sitting inside the office:
“I'm still here. The new locks work.”
“Could you step outside and try your badge?”
“Why?”
“You came in before we changed the locks.”
Suddenly, nobody volunteers.

In explicitly enabled dual-stack mode, IPv4 and IPv6 must each confirm before the change commits.
Success on one family cannot speak for the other. This establishes only the tested management paths, not availability for every source and service.

Recovery and Boot Guard: check policies before opening the interface

Third, test recovery under faults.
Without confirmation, the running Agent responsible for changes and recovery attempts to restore the previous policy. Management-port admission checks for new changes must not prevent required restoration of a known baseline; the existing state and ownership checks still apply.
The new, opt-in Boot Guard runs recovery in a supported isolated network configuration, checks the journal and both firewall policies, then enables the designated interface.

But the defect exposed by testing was in the close path.
The code originally wrote a CLOSING receipt before running the interface-down command.
If writing that receipt failed, the close command could be skipped.

Imagine a security guard preparing to close a door in an emergency.
The procedure requires an incident report first.
The form system is down.
“Why is the door still open?”
“I can't submit the report.”
Apparently, even an emergency closure needs paperwork approval.

After the fix, a failed receipt write still reports failure, but cannot skip the close attempt.
In a Linux VM running the installed production service, I replaced the receipt destination with a directory to cause a real EISDIR write error.
The tested cleanup and stop paths still issued the close command and read back the interface as DOWN.

An attempted close is not a guaranteed successful close.
If the interface operation itself fails, local rescue may still be required. An old OPEN receipt can also remain, so actual state must be inspected rather than trusting the last success record.

Other controlled tests killed Boot Guard before and after interface activation. Two separate orderly VM restart cases restored the baseline of a partially confirmed pending change and the current committed policies.
A surviving old executor also had to be terminated, with its exit verified, before a new change was admitted.

Evidence scope and limits

The same source passed 132 portable tests on each of macOS and Linux, plus 12 actual Linux VM validation phases, including setup and cleanup, under real systemd.
Boot Guard currently supports only a designated namespace and a static veth interface whose activation is reserved exclusively to Rulemoor.
This is not a guarantee for physical NICs or whole-host early-boot zero-gap protection. Sudden power-loss recovery remains untested.

The 12 VM phases include setup and cleanup and run the installed production code under actual systemd. They do not add together historical groups from different source snapshots.

The real VM EISDIR error came from replacing the receipt destination with a directory. Portable tests separately inject EIO/ENOSPC/EROFS at the receipt-writer boundary; real disk exhaustion and sudden power loss were not tested in the VM. An injected interface-command failure left the link UP/FAULTED until local rescue closed it. A passing test does not mean every fault closes the link.

Boot Guard runs recovery and policy readbacks before UP, then checks the interface and policies again afterward. The controlled post-UP crash still has an open window; these checks do not cover every fault or establish zero exposure. Physical NICs, general network managers, DHCP/SLAAC, WAN paths and arbitrary privileged actors are outside this MVP.

The old executor is retired before successor admission. Subsequent readbacks and network probes are finite samples, not continuous monitoring; they cannot rule out every transient change between observations.

Before deployment, decide who takes over

For me, these experiments lead back to a deployment decision:
Which management capabilities do we promise to preserve?
Which faults should the system handle automatically?
When automatic recovery cannot finish, who takes over, through which independent path?

Preflight results, confirmation receipts and process restarts each provide part of the evidence.
Acceptance needs to examine whether the entire flow completes under the faults we promise to handle, and whether the remaining limits are part of the operations plan.

Experimental evidence

Documents pinned to project revision 7076c6f.

Share this article