Failover Is Working. Why Can’t Users Get Online?

Health: someone answered. Can users get online?

“Is failover working?”
“Yes. Both gateways answer ping.”
“Then why can’t anyone get online?”
“We didn’t check quite that far.”

It is like calling customer service, getting an answer, then spending forty minutes on hold. The dashboard reports a 100% answer rate. You’re still on hold, and the problem is still unresolved.

I built Uplinkstead as a Linux IPv4 failover experiment. This article covers its explicitly enabled dual-DHCP mode: two dedicated uplink interfaces and one routed LAN, with independent lease acquisition and renewal before choosing a path for new traffic. Static dual-uplink and single-DHCP deployments are separate modes with different operating contracts.

A gateway reply answers a nearby question. An upstream blackhole, missing forwarding rule or broken return path can still prevent a user from opening a connection. These observations need different names:

On narrow screens, scroll the table sideways to read every column.

Uplinkstead binds probes to a device, source address and mark so a test of A cannot quietly borrow B or the main default route. Health depends on the configured target quorum. It measures those paths, not HTTP business behavior. The controlled fixture’s three targets are three ports on one server, not three independent failure domains.

Freshness: does this evidence still belong to the present?

Once a probe succeeds, the next question is when its result must expire. A renewed or changed lease, a changed gateway and lease loss invalidate that uplink’s health. A restarted writer cannot restore forwarding from yesterday’s green light either.

Using the old result would be like changing suppliers and submitting the previous supplier’s audit. The paperwork is complete. It describes somebody else.

Health therefore needs more than a boolean: it must belong to the current lease and authority generation, and remain within its validity period. A new writer must acquire newly admitted leases and fresh health before restoring traffic. Late requests cannot revive retired authority.

That costs availability. Even a routine renewal that appears to keep the address needs fresh proof; a path without valid supporting evidence may be withdrawn in the meantime. The design accepts temporary uncertainty rather than continuing to authorize from expired results. That trade-off still needs to fit the deployment’s tolerance for interruption, renewal cadence and probe cost.

Switching: is more movement actually more resilience?

Switching immediately can look responsive. Switching straight back after the next good result can keep interrupting the same work.

Imagine a request handed from department A to B and back three times, still unfinished. The manager studies the workflow: “Very agile.” The person waiting just wants to know who owns it now.

Uplinkstead uses success and failure rounds, candidate hold and withdrawal grace instead of letting one result decide everything. Once a healthy backup is selected, recovery of the original primary does not automatically preempt it. This sticky choice favors a currently healthy path over returning to the nominal favorite.

Waiting reduces flapping but delays reaction to a real failure. Staying on backup reduces disruption but can retain a more expensive or lower-capacity link. A controlled run cannot supply universally best timing values.

A durable switching budget and latch add another limit. Once latched, the service may deliberately refuse another failover until an authorized person intervenes. That bounds repeated changes and can prolong an outage. Restart and resume cannot clear the latch. Initial selection consumes budget; ordinary renewal itself is not a cross-uplink charge.

The promise concerns new traffic. Changing uplinks may break old NAT mappings and existing flows. Establishing a fresh TLS connection does not migrate an unfinished download or session to the other connection.

Authority: who proposes, and who can change the route?

A capability-free worker proposes a selection. The privileged writer performs it only after independently checking leases, health, authority tokens, hold/grace conditions, budget and lock state. “Choose B” is a proposal, not sufficient authority to change routing.

This limits what the proposal process can do directly. The writer, DHCP client and capture path remain trusted privileged components; process separation is not containment of a compromised root writer.

Operator actions also differ. Pause may retain a still-safe selected route while safety withdrawal continues. Stop withdraws allocations and selection while retaining guards against borrowing the main default. A crashed writer may leave old kernel state until supervision and cleanup run. Kernel failures or unknown ownership can prevent withdrawal. A status response does not guarantee every underlying action succeeded.

Evidence: what did the retained run actually observe?

These results come from retained closure run ab44c08f9940277e; no new VM experiment was run for this article. macOS and Linux each passed 295 portable tests, with matching test identities and source and no skips. Linux integration used the installed wheel, checked module and supporting-file hashes, and ran the shipped systemd units with namespace isolation.

Six active-uplink blackhole/recovery cycles produced successful fresh LAN TLS over the expected alternate path. Removing each fault did not preempt the healthy selected path. Initial selection plus six switches produced seven budget charges; intervening natural renewals added no cross-uplink charges.

The resource series retained 268 samples over 540.006 seconds, about nine minutes. It required the same writer, worker and two DHCP client identities throughout; formal stop/start was a separate operation afterward. The writer’s RSS median still rose 176 KiB in the five-minute tail. That passed a declared short-run allowance. It is not zero growth, leak freedom or evidence of a performance improvement, and supplies no throughput, latency or outage SLO.

The failed attempts matter too. Each DHCP session retains at most 64 distinct REQUEST transaction IDs; unanswered attempts count. An earlier faster-renewal run hit that cap, recreated the client session and needed fresh health. It failed the same-client resource gate. The final series used controlled T1 values of A 20 and B 15 seconds on 120-second leases, keeping this observation below the cap.

That does not fix or qualify long-term rollover. Reacquiring a session and health can interrupt traffic. Probe scheduling and renewal cadence both changed between attempts, so their RSS differences cannot establish a causal benefit from one change. Keeping the failed result is more useful than calling the final green check “long-term stability.”

Scope: how far is this from deployment?

The network namespaces share one Linux VM kernel. They do not represent two physical ISPs with independent failure domains. General DHCP interoperability, sudden power loss, boot-zero-gap protection and long-term capacity remain unqualified. Older orderly VM reboot evidence has its own source and a 25-second fixture startup timer. It is a separate historical record, not a rerun in this closure, product boot protection or a recovery-time guarantee.

This is also not a complete plug-and-play router. Forwarding, NAT, return paths, reverse-path settings and interface readiness are external deployment prerequisites. Accelerated configuration examples describe fixture conditions, not recommended production timings.

This article separates retained measurements, current implementation contracts and hypothetical management conversations. It describes a bounded systems experiment, without claiming customer deployment experience or turning a proposal or joke into a production guarantee.

Handover: what must the team decide beyond the green light?

Code can decide whether evidence is still valid. The team has to decide which work must recover, which interruptions are acceptable and who takes over. If the latch is set or the state remains unknown, “handle manually” is not a handover.

• Recovery outcome: Which new requests need to succeed first? Which LAN source, target or actual service result will confirm that? How much interruption of existing connections is acceptable?
• Switching cost: How long is worth waiting? Is remaining on a healthy backup acceptable given its cost, capacity and session impact? What would require revisiting the policy?
• Responsibility: Who can interpret lease, probe, route and budget records? Who may keep the service paused, repair it or explicitly reset the budget? When is escalation required?
• Independent access: If the main LAN cannot get out, how will that person get in? Do they have a local console or independent management link, diagnostic tools and the required authority?

Failover needs to do more than turn an indicator green. New work needs a usable path, the cost to old work needs to be understood, and automation needs to leave an executable next step when it has to stop.