2026-09-05 Teaching the Egress Alert to Fix Itself
What I set out to do
Asked whether ntfy can run a command when a notification arrives. It can (ntfy subscribe TOPIC COMMAND, with the message fields as environment variables), but the real goal turned out to be narrower: when the Docker container egress broken alert fires, restart Docker Desktop without waiting for a human.
What I actually did
Rejected ntfy as the hook point. The remediation target is Docker Desktop on atlas, and the host-side notifier from 2026-08-27 Moving Alert Delivery Out of the VM already runs on atlas, already polls localhost:8082, and already computes the firing edge. Routing atlas to athena and back would put the tailnet, athena’s uptime and a second credential set in the recovery path for a purely local action.
Checked the vault before designing, which changed the design. 2026-08-26 The Alert Fired and There Was No Crash records that three of that day’s four outages cleared on their own inside 20 to 60 minutes. Restarting on the firing edge would have bounced 21 containers three times out of four for a fault already resolving. So the automation waits out a 20m grace window first, measured from the earliest firing series rather than from when the poller noticed.
Built it in two pieces, tests first. signoz-notifier.py gained a generic --while-firing hook: a command run once per poll for every firing group with that group’s Alertmanager body on stdin, exit 0 acted / 1 failed / 2 declined. Deliberately not edge-triggered, unlike the notification, or a grace window could never elapse. docker-egress-recover.py holds the policy and declines unless both container probes are dark (the internet probe alone is DNS or WAN, which a restart does not fix), the break has outlasted the grace window, and the attempt is outside a 60m cooldown and a 3-per-24h budget.
Two ordering details that are the whole safety story. The page is flushed before the hook runs, so a restart cannot take down the notification about it. And the attempt stamp is written before the restart executes, so a docker desktop restart that hangs still spends its cooldown instead of being retried every 30 seconds.
40 tests across the two suites, both wired as flake checks. Verified end to end against the deployed store paths: stood up a fake /api/v1/alerts serving a synthetic egress alert and watched the real notifier publish the page and then hand the real body to the real hook, which declined on the grace window. Exercised the acting path with --docker /bin/echo, the cooldown, and a failing docker binary. Shipped in bed02fa, ADR 0028.
What was striking
The alert’s own description was the specification. It already partitions the fault: both probes dark is Docker guest networking, the internet probe alone is DNS or WAN. That sentence, written for a human to read at 4am, became the first guard in the code almost verbatim. The label design done back in 2026-08-25 Alerting on Docker’s Guest-Network Break is what made automatic action safe at all; without the per-URL grouping there would be no way to tell the two faults apart programmatically.
The vault check was load-bearing, not ceremonial. I had a defensible design ready (restart on the firing edge, with a cooldown) before reading the 08-26 entry. The “three of four cleared themselves” line is not in any code comment, any ADR, or any alert description. It only exists because someone wrote down what happened that day. It inverted the core timing decision.
Guards belong in the automation, not the rule. The tempting move is to narrow the alert to “both probes, for 20 minutes” so the hook can be dumb. That would blind the operator to the DNS/WAN case and delay every page by the grace window. The page should fire immediately and describe reality; only the thing taking destructive action should wait.
A silent self-heal would be worse than no self-heal. Every attempt publishes, and a failed restart escalates to critical, because it is strictly worse news than the page that preceded it. The machine repairing itself quietly would mean losing the signal that this keeps happening at all.
Related
Homelab Services Architecture, 2026-08-27 Moving Alert Delivery Out of the VM, 2026-08-26 The Alert Fired and There Was No Crash, 2026-08-25 Alerting on Docker’s Guest-Network Break