The Node Went Silent. The Update May Still Be Running.

Silence: no reply does not settle the operation

“Are they still working on it?”
“No reply.”
“Then give it to someone else.”

Somewhere in that conversation, “we don’t know” became “we checked.”

I built Fleetweft, an experimental edge release manager with a single controller, to explore application updates and recovery.

An unreachable node may not have started, may still be updating, or may have finished without its confirmation reaching us.

Deployment slots: account for unresolved work

Fleetweft keeps a granted deployment slot occupied until the operation’s status is resolved. Pause stops new grants; previously authorized work still has to be accounted for.

Imagine reassigning the only update slot because its owner stopped replying.

Both people return: “Done.”
“Great ownership, both of you.”

They changed the same system.

Holding the slot has a cost: unresolved work can block a rollout indefinitely. Who gathers the evidence? Who decides when to proceed? Who takes over when automation cannot finish?

In one retained VM scenario, the controller durably authorized a commit, then the first reply was deliberately lost. No reply at the node did not mean no authorization existed. After Pause, the controller retained the slot and replayed that same stored decision so the original operation could settle, without granting new work to follower nodes.

Canary: check the service, not just the process

Before rollout proceeds, a designated canary checks the running application’s actual response. Starting a process is not enough. This is a functional gate, not statistical reliability evidence or continuous health monitoring.

Another scenario had the canary report healthy while its actual service returned the wrong result. The tested flow restored the baseline, paused the rollout and granted no follower work. That supports a functional gate for the tested case; passing it does not establish the reliability of other services or every later moment.

Recovery evidence: a request is not restored service

Recovery needs evidence too. Requesting rollback does not prove service has been restored. Unverified recovery stays UNKNOWN. Returning nodes also need checks so late operations cannot overwrite newer settings.

A separate scenario killed the restored application after the node reported FAILED_RESTORED. A failing probe kept that exact grant UNKNOWN and its slot occupied. Only after an Agent restart reconstructed a healthy baseline and reported recovery could it settle. A journal saying “restored” still needs a serving check.

mTLS protects the control channel, and bundles require signature verification. Trusted code can still be faulty; authentication and signatures do not provide a sandbox.

Validation scope: what this controlled run establishes

The recorded results include 69 portable tests passing on each of macOS and Linux, 15 Linux VM scenarios, and separate orderly VM restart validation.

The three nodes are network namespaces in one VM, sharing a kernel, disk and failure domain. They are not three independent hosts. The tested deployment scope is trusted, signed, single-file, stateless Python HTTP apps on Linux ARM64. High availability (HA), sudden power loss, persistent-data migration and production reliability are outside these conclusions.

Handover: turn uncertainty into specific work

For me, these distinctions matter:

A delivered command is not a completed update.
Losing contact does not mean an operation has stopped.
A rollback request is not restored service.
Reconnecting does not make an old command valid again.

Finally, the manager looks at the dashboard.

“Can we change Unresolved to Completed?”
“No completion evidence.”
“Failed, then?”
“No failure evidence either.”
“Could we at least change the color? Then it might look under control.”

The status is no clearer. The dashboard is more optimistic.

A system may not have an answer yet. It still needs to preserve known facts, constrain what happens next, and make clear to the people taking over what remains unconfirmed.

Before proceeding, record these three answers

• Which node holds which granted operation? What release, operation identity and serving result were last confirmed?
• What evidence is missing before the slot can be released or recovery confirmed? Who will gather it, and when will the decision be revisited?
• If the status remains unresolved, who decides to keep the rollout paused, recover or intervene? What path lets that person verify the actual state?