Achhina's Digital Garden

2026-08-26 The Alert Fired and There Was No Crash

5 min read

2026-08-26 The Alert Fired and There Was No Crash

What I set out to do

Nothing. A SigNoz alert URL landed in the session with no question attached. It pointed at “Docker container egress broken”, the rule built yesterday in 2026-08-25 Alerting on Docker’s Guest-Network Break, which had never fired for real.

What I actually did

Confirmed it as a true positive, from inside a container. Both probe series were firing, so the rule’s own label design already said “guest networking”, not “DNS” and not “the host gateway alone”. By hand: the macOS host reached LiteLLM on :4000 in 5 ms and the internet in 54 ms, while from inside open-webui every connect returned 000 with an empty remote_ip in well under a millisecond. Forcing -4 and -6 both failed, and so did the literal 192.168.65.254. Instant refusal, not a timeout. The container’s own side was healthy throughout: /proc/net/route held a correct default via its bridge gateway, DNS resolved fine, docker version answered instantly, and all 21 containers reported healthy.

Found a one-command localizer. Running the same curl from the VM’s own network namespace (docker run --rm --net=host curlimages/curl ...) also returned 000. That moves the break off per-container NAT and onto the VM to macOS boundary, which no previous episode had actually established.

Contradicted the runbook on two points. There was no sailor crash: com.docker.sailor had an unbroken 16h30m uptime, the VM never rebooted (containers still read “Up 12/16 hours”), and the logs held zero panic, semawakeup, pthread_create failed, or guest-connect ... rejected. And it flapped: four separate outages across the day (03:53 to 06:51, 07:13 to 07:30, 10:49 to 11:53, 13:02 to 13:30 EDT), three of which cleared with no restart at all.

Restarted and recovered. osascript -e 'quit app "Docker"' looked like a complete no-op, with the backend still in pgrep sixty seconds later. It had in fact moved the app to Status: stopping, which only docker desktop status reveals, but it then sat there: the user had to quit Docker Desktop by hand from the UI before the shutdown completed. docker desktop restart brought it back from there. All 21 containers came back on their own about 11 seconds after the VM returned, identical set, and hm switch then ran clean with tofu reporting 0 added, 0 changed, 0 destroyed. Egress verified over ten consecutive probes, and the alert resolved both series at 13:30:17.

What was striking

The detector paid for itself inside 24 hours, and the label design did the diagnosis. Yesterday’s argument for splitting the rules was that the combination which arrives should be the diagnosis. That held: both probes firing together, with the failing URLs in the labels, narrowed it to guest networking before I ran a single command. The hand verification confirmed a conclusion the page had already reached.

The mechanism in the runbook turned out to be incidental, not necessary. Every prior write-up of this class describes a sailor panic followed by a VM self-restart, and the recovery advice was built on top of that story. Today had neither, and the symptom was identical. That is a good argument for keying diagnosis on the observable (instant refusal from inside a container, host healthy) rather than the presumed cause. The 08-25 loose end asking whether the occurrences share a root cause is now sharper: at least one of them has no crash to point at.

“Only a full restart fixes it” was wrong in a useful way. Three of today’s four outages cleared on their own within about 20 to 60 minutes. The restart is not the only exit, it is how you stop waiting. That also means the alert cannot tell a flap that will self-clear from a break that is stuck, which is a real gap in what the page communicates.

A small tell worth remembering. While broken, host.docker.internal resolved to an IPv6 ULA (fdc4:f303:9324::254) despite Docker Desktop’s defaultNetworkingMode being ipv4only, and came back as 192.168.65.254 after recovery. Separately, Docker Desktop’s own host-side proxy was degraded in the same window, with login.docker.com and desktop.docker.com CONNECTs each taking about 90 seconds.

Loose ends

  • Root cause still unknown, and now there is not even a crash to blame. Nothing identifies what triggers the transition in either direction.
  • The rule fires the same way for a 20-minute flap and an indefinite break. Worth thinking about whether a duration or repeat-count signal belongs on the page.
  • The notification never arrived, and the reason is structural. No ntfy push landed for any of the four outages. Despite its name, the local-placeholder channel is a real webhook to the sink at athena.tail0cef90.ts.net, reached over Tailscale, so delivery has to leave a container and cross the exact path the alert exists to report on. httpproxy.log shows container-origin TCP to a tailnet address retrying through the outage window and getting nowhere. The detector is sound and its delivery hop is not: the rule fires correctly into a void. A host-side notifier polling SigNoz’s published port and pushing to ntfy over the host’s own network would survive this, since host to container port forwarding kept working throughout (the SigNoz UI was reachable in a browser the whole time).

Graph View

Backlinks