Achhina's Digital Garden

2026-09-06 The Egress Discriminator Was Backwards

15 min read

2026-09-06 The Egress Discriminator Was Backwards

What I set out to do

Answer a simple question: how many times has the autoRestartDocker hook fired since I shipped it yesterday? The alert feed had been noisy with LiteLLM and Docker pages and I wanted to know whether the automation was doing anything.

What I actually did

The answer was zero, and the interesting part was why.

docker-restarts.json never got created and no Auto-remediation: line appears in the notifier log. The hook is wired and running, it just declines on its first guard every single poll. That guard is --min-probes 2: restart only when both egress probes are dark, because the rule’s own description says both probes dark is Docker guest networking while the internet probe alone is DNS or WAN.

Over eleven days the host-gateway probe never once dropped below 1. So the guard is unreachable, and the hook is structurally blind.

My first instinct was that the alerts were all sleep noise. That was half right and I overreached on it. atlas wakes about 171 times a day, mean gap 8.4 minutes, and splitting each episode into awake versus asleep time from pmset -g log showed 17 of 34 episodes had under five minutes of awake time. A “547 minute outage” had ten awake minutes. So a lot of the volume genuinely is the sleep schedule, exactly as in 2026-07-13 ClickHouse Replication Alert Was Atlas Sleeping.

Then I recommended turning sleep off, which is precisely what that July note says not to do, and got corrected. atlas is a personal machine, wake-on-network is the deliberate compromise, and the note had already reasoned it through. I had the answer in the vault and did not read it first.

The residue turned out to matter more than the noise. Today’s 15:35 episode ran 46 minutes with the machine continuously awake (sysctl -n kern.waketime = 15:29:47, no transitions after). That is not sleep. Chasing it:

  • httpcheck.error carries the Go dial error. Every failure was connect: connection refused from a Google IP. The public internet does not RST port 443.
  • The VM’s DNS resolved connectivitycheck.gstatic.com correctly every 30 seconds throughout, so not DNS.
  • The host-side control I thought was missing already existed: Docker Desktop’s own httpproxy.log records its host-side CONNECTs with bytes transferred. It completed tunnels to desktop.docker.com:443 at 15:52 and 15:55, mid-outage. Host had internet, containers did not.
  • Seven-day tally for the internet probe: ~526 connection refused, 61 EOF, one timeout, zero DNS failures.

So the rule’s partition is backwards. Internet-probe-only is not DNS or WAN. It is Docker, and it has been Docker essentially every time.

What was striking

The guard was calibrated against a failure mode that stopped happening, and blind to the one that kept happening. VM to host-gateway and VM to internet are different paths in Docker’s netstack. I wrote the “both probes or nothing” rule from the description in the alert, treated that description as ground truth, and never checked whether the gateway probe actually goes dark during a real fault. It does not. Yesterday I described that guard as the thing standing between me and 53 unnecessary restarts. Today it is the reason a real fault has gone unremediated for eleven days. Same code, opposite reading, because I had finally measured instead of quoting.

A declining hook and an unwired hook look identical. Declines log at DEBUG, so at INFO the correct-and-useless case is indistinguishable from the broken case. The only tell was the existence of the lock file.

The error string was the discriminator all along. connection refused versus i/o timeout versus no such host separates Docker from WAN from DNS cleanly, per sample, with no second probe needed. It was in the metric the whole time; the rule just counted probes instead of reading them.

Part two: the control was never a control

Fixing the discriminator meant adding a host-side probe of the same canary, which shipped and went green. But the sharper result came from the LiteLLM alert firing while I was verifying it.

LiteLLM was healthy: bound *:4000, answering localhost in 5ms, launchd agent never exited. The container’s probe to it had logged 85 connection refused to 192.168.65.254:4000 in 90 minutes. Same gateway IP as the :8082 probe that never darkens. That is when it clicked: :8082 is SigNoz’s own published port. Docker forwarding to a port Docker itself published succeeds straight through a guest-networking break. It was never a second probe; it was the one target in the set that structurally cannot see the fault.

So every probe needing a real hop out of Docker fails together, and the one that stays green is the one that never leaves. Repointed it at macmon, a plain launchd listener on *:9110. The LiteLLM rule got the same correction: “fires alone means the proxy is down” is false when the probe crosses the hop that breaks.

Then sailor died mid-deploy

The hm switch triggered the crash. panic detected in sailor: pthread_create failed: Resource temporarily unavailable, 17:41:38, four seconds before activation finished. Recreating twenty containers is a burst of new flows, and project_docker_sailor_thread_leak says each new UDP flow costs a thread pair that is never reaped. If sailor was near the 16384 ceiling, I supplied the last few thousand.

Which raises the thing I did not see coming: the egress alert may be this leak’s early-warning phase. A sailor that cannot spawn a thread pair cannot open a new flow, so new host-crossing connections get refused while established ones keep working. That is the measured signature precisely, down to DNS resolving fine throughout and 526 refusals against a single timeout in seven days. Two investigations I had been treating as separate may be one bug seen at two points on its curve.

I cannot test it retroactively. The thread sampler stopped on 2026-08-29 and nobody noticed for a week, so there is no thread count to line up against today’s probe failures. Re-armed it as a declared launchd agent rather than another ad-hoc script.

What was striking, again

Building the sampler found a bug in the sampler. The nix check failed because the sandbox has no working ps, which showed me that a PATH-resolved ps under launchd degrades to empty output and records as a thread count of zero. Forever. Silently. While the leak ran. That is the exact failure that let the last sampler die unnoticed, and I had reproduced it without noticing until the sandbox refused to cooperate. Absolute paths now.

Two wrong discriminators in one file, both written the same way. The egress rule counted probes and the LiteLLM rule read “fires alone”, and both encoded a belief about which paths are equivalent that nobody had measured. The fix in both cases was not a better threshold, it was checking whether the thing being compared was comparable at all.

I keep being saved by controls I did not plan. The host-side WAN proof came from Docker’s own httpproxy.log, which I found while looking for something else. The ps bug came from a sandbox failing for an unrelated reason. Neither was designed; both were the only reason the conclusion held.

Part three: the first real episode since the fix

Four hours after the fix shipped, egress broke again, and this time I got to watch the corrected machinery decide.

21:23 to roughly 21:26:38, call it three and a half minutes. The new discriminator called it correctly and immediately. Host side: WAN 204, macmon 200 on localhost:9110. Guest side: 000 everywhere, from a fresh container on the default bridge, a fresh container on signoz-network, --net=host in the VM’s own namespace, and the long-running open-webui and sillytavern. Every destination failed, gstatic, 1.1.1.1, example.com, TLS and plain HTTP alike. DNS resolved fine (remote_ip populated on every attempt), connect returned in 5.8ms, then empty reply or unexpected eof. Refusal, not timeout. Exactly the signature I spent the afternoon learning to read, and this time I read it off the error text in about ninety seconds instead of reasoning from probe counts.

com.docker.sailor had an unbroken 3h10m uptime with no panic, so this was the no-crash class again. Which is the shape the thread-leak hypothesis predicts for its early phase: new host-crossing flows refused while the process itself stays up. One more data point, still not a test.

Neither the alert nor the hook fired, and both were right. The rule needs about 5.5 minutes continuous; the outage ran 3.5. graceMinutes is 20 on top of that. state.json read {"firing": {}, "pending": [], "unreachable": false, "failures": 0} throughout, and both agents had been up 4h18m since 17:09 with no restarts. Yesterday I would have read that silence as the hook being blind again. Today I could tell the difference, because the guard it declines on is now one that can actually be satisfied.

The restart was mine, not the hook’s. com.docker.sailor came back as a new PID while com.docker.backend kept its 3h16m, so the VM was replaced under a surviving backend, and recovery lands on that timestamp rather than drifting in on its own. All 21 containers returned unattended on restart: always. No hm switch needed, which is the first time that change from 6c7bf93 has paid out.

What was striking, a third time

The control I built this afternoon worked on its first live fault. The host-vantage probe stayed at 1 for the entire outage while the container vantage went dark. That is the whole point of the pair and it had never been exercised against a real break. It is a strange feeling to have the falsifier work correctly and have nothing to falsify.

But the container internet probe may be under-reporting. It logged exactly one failing minute, 21:25, while my own curl from four separate containers failed continuously from roughly 21:21 through 21:26. One of those containers sat on signoz-network, the same network as the ingester whose probe was reporting success. If that gap is real, the rule keyed on that probe is desensitized and would under-fire on a long break too, which would make this a smaller version of the same mistake I just finished fixing: trusting a probe’s verdict without checking that it can see the thing it claims to watch. I restarted Docker before capturing evidence, so the window is gone. It needs a deliberate re-test on the next episode, or an induced one.

The retracted premise is still written down in one place. autoRestartDocker.enable’s option description in signoz.nix still says “Both container probes must be dark. The rule also fires when only the internet probe fails, which is DNS or WAN.” That is verbatim the belief 3c02dbd deleted from the code, and the script’s own docstring now calls it wrong. The code is fixed and the documentation still argues for the bug. Cheap to fix and worth fixing, because the next person to reason about this hook will read the option, not the source.

Part four: it was keep-alive, and it is the same bug a third time

The suspicion from part three held, and the mechanism turned out to need no induced break at all. netstat inside the ingester’s network namespace, sampled across 104 seconds and at least three scrapes:

  • 142.251.211.195:443, the internet canary: one connection, local port 35238, ESTABLISHED the entire time.
  • 192.168.65.254:9110 and :4000, the host-gateway probes: a new local port on every single scrape, all cycling through LAST_ACK.

The httpcheck receiver reuses its TCP connections. The fault refuses new flows and leaves established ones alone. So the internet probe was riding one socket opened before the break and reporting success across it, while the two probes that open a fresh flow every 30 seconds saw the outage immediately. That is the whole discrepancy, and it explains the single failing minute at 21:25 as the moment that socket finally got torn down.

Which makes this the third instance of one defect, not a new one. The :8082 published-port canary forwarded to a port Docker itself published. The LiteLLM rule read “fires alone means the proxy is down” across the hop that breaks. And now the internet canary watched a connection the fault does not touch. Every time the surface symptom was different and the shape was identical: a probe measuring a path the fault cannot reach, passing confidently for exactly that reason.

The fix is disable_keep_alives: true on every target in both vantages. It is a per-target field; setting it on the receiver is rejected outright, which I checked against the real binary rather than assuming, and I confirmed the ingester’s older signoz-otel-collector v0.144.6 accepts it by showing that a deliberately bogus key on the same receiver is refused at unmarshal. After the switch the gstatic socket churns into TIME_WAIT every probe instead of sitting ESTABLISHED, and all four series stayed green through the change.

Also corrected the option description that still taught the retracted probe-count premise. 3c02dbd deleted that belief from the code this afternoon and left it standing in the documentation, which is the one place someone deciding whether to trust this hook would actually read.

What was striking, a fourth time

I nearly waited for the next outage to test this. The plan was to catch the next break and capture evidence mid-fault, which would have meant waiting hours or days for a fault I cannot summon. The actual answer was sitting in netstat the entire time, available in ninety seconds, because the question was never “does the probe fail during a break” but “does the probe open the kind of connection a break can refuse.” Reframing it from an event to a property made it answerable immediately. The property was observable in a healthy system.

The negative control caught nothing this time and was still the reason I believe the result. The contract test asserts on rendered output, and I ran it red first: four targets flagged, both vantages reached, host store path resolved. Had the extraction silently matched nothing, the pass afterwards would have meant exactly as much as it looked like it meant, which is the failure mode this repo keeps writing controls against.

One unrelated thing surfaced and I left it alone. nix flake check fails on capability-flags, which still lists .config/jupyter/lab/settings/overrides.json as an artifact of local.dataScience.enable. Today’s JupyterLab work moved those settings to the user-settings layer and the contract entry never followed. Nothing to do with egress, so it stays a separate thing rather than getting swept into this commit.

Coda: the two checks I said I would leave alone

I did end up fixing both, and they turned out to be the same shape as everything else today.

capability-flags was asserting that local.dataScience.enable gates .config/jupyter/lab/settings/overrides.json. That file stopped existing when the Lab settings moved to the user-settings layer, so the entry was wrong on the path and on the surface: those are home.mutableFile entries now, not home.file. The contract outlived the artifact and took the whole suite down, which is this test’s own failure mode turned on itself. Re-pointed it at the user-settings mutableFile, with one plugin standing for all fifteen since they come from a single lib.mapAttrs' under one lib.mkIf. Ran the control rather than trusting the green: a bogus plugin path produces exactly the “absent from the baseline config” error that caught overrides.json.

claude-settings rejected feedbackDrafts, which I pinned off this morning. The interesting part was the question of whether the schema could learn it. Three independent checks said no: schemastore’s live copy has 142 properties and does not define the key, the pinned copy is byte-identical to live down to the sha256, and nix flake update claude-code-settings-schema produces zero lock diff. So this was never a stale pin. The key is @internal, like skipWorkflowUsageWarning sitting next to it in the exception list, and schemastore has nothing to catch up to.

The source went in the exception comment instead, taken from the CLI bundle where the authority actually lives: in 2.1.263, feedbackDrafts is an enum settings path with options:["notify","quiet","off"] defaulting to "notify", and the settings-policy table carries {path:["feedbackDrafts"],restrictive:"off"}. So the value pinned is real and the strictest one available.

The theme held to the end. Four times today the fix was not a better threshold or a fresher pin, it was checking whether the thing being consulted could answer the question at all. A probe on a path the fault does not touch. A contract naming a file that no longer exists. A schema that was never going to describe an internal key. Each looked like staleness and each was a category error, and in every case the cheap move was to go look at the artifact rather than reason about what it ought to contain.

Graph View