SYSTEMS / INCIDENT EVIDENCE / HANDOFF

The Fault Cleared. Did the Evidence Stay?

Andrew Yuan · Engineering experiment

The service recovered. The investigation is just starting.

“It definitely stopped working.” The engineer asks, “What about now?” “It seems fine again.” Any error message or log? “No, but I saw it happen.”

The investigation begins with an eyewitness account. The user may be entirely right, and the engineer may be taking it seriously. The problem is that the interval they need to examine has already passed.

I built Pathquill as a Linux network incident recorder with explicit storage and export budgets. It retains recent observations, gathers follow-up information after an incident trigger and produces deidentified JSON/HTML evidence packages. The question is whether a short-lived failure can leave inspectable context once it has cleared.

Recording something does not establish its cause. A security camera can show the office lights going out without becoming a qualified electrician. This article keeps the same distinction: observations, possible explanations and missing evidence are separate things.

An observation strip crosses an amber window around an anomalous interval. — Concept illustration: a bounded observation window, not a product interface or proof of root cause.

What did the recorder actually observe?

The formal daemon reconstructs a supported projection of Linux link, address and route state and schedules bounded TCP/DNS probes. It is not a packet recorder or a complete history of network behavior. The installed-service validation discussed here uses a controlled local TCP target.

A failed probe supports a narrow statement about that target and that attempt. It does not establish an Internet-wide outage. A later success is another observation; it cannot fill every unknown interval between probes.

State reconstruction has conditions too. The implementation requires two matching dump projections without intervening notifications and limits its attempts. Ongoing changes, loss or decoding errors can prevent publication of a trustworthy rebuilt view. This is neither an atomic cross-table snapshot nor guaranteed convergence under arbitrary churn.

For an investigator, uncertainty about coverage is useful information. Pathquill retains UNKNOWN/PARTIAL coverage states instead of treating a successfully produced package as a complete record of the scene.

Read the incident as a sequence, including its gaps

Before a failure, the service already retains recent observations in a bounded ring. A trigger can copy the observations that remain into an incident capture, then collect follow-up information. Budget-based eviction can shorten the pre-event interval, so a configured window is not a promise that every earlier observation survives.

Follow-up collection must also reach durable progress. The system does not accept a caller's claim that collection is complete as its watermark. Sealing checks actual progress and outstanding probes. Waiting for a deadline does not erase gaps caused by refused observations or a restart.

Recovery evidence must be current. Each probe result carries the state generation in which it started. Results from an old generation cannot update current health, and a success from before the incident cannot establish recovery. The tested logic requires current-generation successful probes for the configured targets, with probe start times at or after the incident began.

Exports preserve recorder write order and relative times, not causal order across sources. Legacy observations without a timestamp remain null rather than receiving an invented time. Seeing one event before another does not establish that it caused it.

The four positions below are a way to inspect an acceptance package. They are not a claim that every incident follows a fully observed sequence.

Before the triggerCheck which observations actually remain in the ring; do not infer a complete pre-event window.
Trigger and follow-upLocate the failed probe and subsequent observations; read coverage and any refusal or restart gaps.
Observed recoveryCheck current-generation successful probes and their start times, rather than relying on an earlier success.
Sealing and exportOpen the actual package and check ordering and coverage. A generated file does not determine the cause.

What do about 330 seconds and nine cycles establish?

The retained closeout used an actual installed wheel and systemd service under a dedicated unprivileged account in a newly provisioned isolated Colima Linux VM. Its base was Colima's Ubuntu 24.04.4 minimal containerd image. Pathquill itself ran as a systemd service, not in a container.

The final same-source observation lasted 329.93 seconds and completed nine real local TCP target failure, recovery, export and retirement cycles. Service PID, invocation and epoch stayed unchanged, with no restart. The report retains 161 resource samples.

Sampled warmed main-process RSS ranged from 20,860 to 21,888 KiB, about 20.4–21.4 MiB. The median increased by 872 KiB between the first and last warmed ten-sample windows. File descriptors ranged from 7 to 8, with two threads. These are observations of one process and defined windows, not whole-VM costs or proof against long-term accumulation.

The 111 distinct Linux source tests passed. Seven regression workflows reran that same suite, so the count remains 111. This article checks and cites existing records; it adds no new experimental runs.

Failed attempts remain in the evidence. An earlier capture reached 4,150 rows, exceeding the unchanged 4,096-row export-input limit. The correction coalesced redundant idle progress commits at 100ms; it did not slow every notification or probe operation to that rate. The final bounded workload passed after the correction. It was not a first-attempt success, nor a resolution of arbitrary high-event-rate workloads.

Two kinds of full need different responses

The first is observation or export content exhausting its budget. The ring can lose older pre-event observations; data refusal leaves coverage gaps such as PARTIAL. A full incident-slot pool refuses new incidents. An investigator needs to see those limits rather than an apparently complete timeline.

Exports have a 4,096-row/1 MiB input limit and a 256 KiB snapshot limit. High event rates or extended captures can still exceed those limits and fail export. Budgets constrain cost, but also constrain evidence. They do not promise complete recording at arbitrary load.

The second is the store's 32 permanent export-request history slots. Cancellation or package retirement does not make a historical ID available for reuse. The 33rd new request is refused. History admission differs from file and snapshot credits: deleting a package does not create a new request slot.

These are separate logical budgets, not a whole-filesystem quota or a hard main-process RSS cap. SQLite pages, journals, indexes and other manually retained stores also contribute to deployment costs.

Exports separate the shareable format from local administrative state. Identifying fields are transformed under an allowlisted format. Unknown fields are omitted; malformed values in supported fields fail snapshot creation. JSON and HTML therefore present a bounded, filtered record, not a complete copy of raw network data. Raw stores, administrative status and archive receipts remain private.

Deidentification is not complete anonymity. Ordering, topology and other correlatable details may still reveal information about an environment. Someone needs to review the package for its intended recipient. An export operation is not authorization to publish it.

A package hash can establish content consistency, not publisher identity or root cause. Deleting SQLite rows or retiring a package is not secure erasure of disk pages, journals or copies already given to someone else. Evidence production and data governance need their own owners.

Full history requires a handoff, not a reset button

Pathquill provides an explicit seal/archive/new-store workflow without automatic rollover. The operator must resolve captures, holds, exports and unfinished cleanup, then verify the store identity and current epoch before sealing the old store.

After sealing, the daemon exits. Offline archival checks the state and known contents, preserves raw data and creates an integrity receipt. A new store is created at a different, previously nonexistent path with new authority. Old requests and references are not copied into it, and service configuration is not automatically changed.

An operating plan therefore needs capacity monitoring, handoff time and an owner. How long should the old archive remain? Who can read or delete it? Which observations might be absent between stopping the old service and starting the new store? This evidence does not establish a gap-free transition, and per-store budgets do not bound all archives on a host.

Open the report, then decide who takes over

I would include a full exercise in acceptance: create a declared in-scope failure, let it recover, then produce and open the package. Check failures near the trigger, subsequent successes, ordering, actual retained range and coverage. “Export succeeded” alone does not establish that the evidence supports an investigation.

Next, check how limits become visible. Can a reader recognize a shortened pre-event interval? Who learns that export failed or history slots are nearly full, and who acts? If that needs another alerting system or operating procedure, name it explicitly. A status field does not complete the handoff.

The latest installed service has not been qualified for VM-reboot recovery, sudden power loss, continuous early-boot recording or long-term stability. Orderly-reboot experiments on older versions do not qualify the current installed service. Deployment still needs validation of its environment, data sensitivity and response procedures.

Service recovery is good news. The HTML report can arrive quickly; root cause may not respect the reporting deadline. Pathquill helps make the next questions concrete: what did we observe, which explanations remain hypotheses, what evidence is missing, and who will obtain it?

Sources and retained evidence

Code: GitHub / Pathquill · MIT. Evidence is pinned to the inspected source version; this article does not add a new experiment.