2026-08-25 Alerting on Docker’s Guest-Network Break
What I set out to do
Nothing, at first. A docker logs sillytavern dump landed in the session with no
question attached, so I read it and flagged the only two things in it: a
read-only config.yaml write failing with EROFS on every boot, and a wall of
503s. Then: “whats going on”.
What I actually did
Diagnosed the third Docker failure class again, live. The 503s in the log
were a red herring — they were stamped 2026-08-21 and belonged to the episode
fixed two days ago (2026-08-23 The 503 Was Docker’s Guest Networking, Not LiteLLM).
SillyTavern only writes to its log when someone chats, so the log said nothing
about now. The live probe did: from inside open-webui, both 1.1.1.1:443 and
host.docker.internal:4000 refused in 0 ms, DNS timed out, and ping 1.1.1.1
lost 100% — while the container’s own routing was intact, docker version
answered in 0.02s, docker info reported no server errors, and all 20 containers
were healthy. LiteLLM was fine on the host the whole time (launchd pid 1638,
/health/liveliness 200 over loopback). Same signature as 08-23. The user fixed
it before I touched anything.
Made SillyTavern’s config self-consistent. config.yaml is bind-mounted
:ro, and app/config.yaml is a symlink onto it, so the boot-time “add missing
config values” pass fails with EROFS twice per start. The part that actually
matters: a key missing from the Nix module is not inherited from the image
default — the write that would have added it failed, so it silently takes the
code default. Added the 13 keys 1.18.0 expects (enableKeepAlive,
sso.trustedProxies, privateAddressWhitelist.*, three rateLimiting attempt
caps, forwardedHeaders.*) and verified parity by diffing flattened key paths
against the image’s own default/config.yaml in both directions: missing none,
extra none. Clean boot after, no EROFS.
Built the detector this failure class has never had. An httpcheck receiver
on the SigNoz ingester, probing three canaries every 30s from inside the
collector container: SigNoz’s own published port (out of the VM and back in
through the host gateway), a public 204 endpoint, and LiteLLM. Two rules read
them — “Docker container egress broken” grouped by probe URL, and “LiteLLM
unreachable from Docker containers” filtered to the one URL. Verified the
receiver existed in the pinned collector image by running it before writing any
Nix, and verified both the success and failure metric shapes the same way.
Proved the alert fires. Ran a throwaway collector exporting a deliberately
failing probe into the live ingester over OTLP — no change to the deployed
config. The egress rule went firing after ~5.5 min and back to inactive
~5.5 min after I stopped it, while the URL-filtered LiteLLM rule correctly
stayed inactive throughout. Two commits, just check green.
What was striking
The outage was live for at least six hours and nothing anywhere reported it. It surfaced only because a log dump happened to land in a session for an unrelated reason. Every signal that should have caught it is structurally blind: hostmetrics measures the Linux VM (whose own networking is fine), docker_stats reads the socket (which answers fine), and container-to-container traffic on the compose network keeps working, so every healthcheck stays green. The break is exactly and only container→outside, which is the one path nothing was watching.
The probe has to live inside a container, and that is the whole design. A host-side check takes a different path and passes while the guest path is dead — it would have reported everything healthy through both this outage and the 08-23 one. The results still get out because the exporter writes to ClickHouse over the compose network, which survives this failure mode. The thing being measured and the thing carrying the measurement fail independently, and that is what makes it work at all.
I shipped a rule that could never have fired, and only caught it by running the
query. I set requiredNumPoints = 6 on a 5m window reasoning from the
receiver’s 30s collection interval. But required_num_points counts
downsampled points, and SigNoz silently clamps any step under 60s up to 60s on
a range that short — the response says so in a warnings[] entry. A 5m window
therefore holds at most 5 points, so requiring 6 would have skipped every
evaluation forever, with the rule sitting inactive and looking perfectly
healthy in the UI. Nothing errors. The only way to see it is to run the rule’s
own query and count what comes back. Fixed to 3 before it ever went live.
Two rules instead of one, so the page discriminates rather than the human. A single “LLM backend unreachable” rule would fire identically whether LiteLLM died or the network did. Splitting the URL-filtered probe out means the combination that arrives is the diagnosis: both firing = network, LiteLLM alone = proxy. Confirmed live — the injected failing URL tripped the grouped rule and left the filtered one alone.
A “verified” alert is one you have watched fire. Injecting a failing series into the live collector from a throwaway container turned out to be a cheap, repeatable way to exercise the whole path — metric, query, threshold, state transition, resolve — with zero changes to the deployed config. Worth doing for every new rule, not just this one.
Loose ends
- The alert cannot fire if the collector itself is down. Deliberately no
alert_on_absent, since this host sleeps routinely — the same reasoning as the ClickHouse rule. Absent probes are not evidence of broken egress. - Whether this occurrence and 08-23’s share a root cause is still unknown; the host log had no fresh sailor panic in the window I looked at.
nix/flake.lockandcasting.yaml.inwere already modified in the tree from the 2026-08-25 Removing ClickHouse Replication Ends the Stuck-Queue Class work and are still uncommitted; I left them alone.