2026-08-23 The 503 Was Docker’s Guest Networking, Not LiteLLM
What I set out to do
Answer “is litellm and sillytavern working? i see 503”. A status check, on the face of it. The second Docker incident of the same day, and a different failure class from the morning’s (2026-08-23 Sailor Crash and the Fifth Replication Queue Repair).
What I actually did
Checked both services and found nothing wrong with either. LiteLLM is not in
Docker at all: it runs as the ai.litellm.proxy launchd agent, bound to *:4000,
and a real completion against it returned 200 with a priced usage block. Its
request log was 200 on every line. SillyTavern’s container was healthy and served
200 on :8000. Two green services and a 503 anyway.
The browser trace pointed at SillyTavern’s own server-side proxy, and its container log gave the real error:
Streaming request failed with status 503 Service Unavailable: upstream connect
error or disconnect/reset before headers. reset reason: connection termination
That is envoy’s wording, not SillyTavern’s. The path is SillyTavern to its envoy
sidecar to host.docker.internal:4000 to litellm on the host, and the last hop
was dead. From inside a container the gateway refused outright, and so did the
open internet:
| Target | Before | After |
|---|---|---|
host.docker.internal:4000 (litellm) | 000 refused | 200 |
host.docker.internal:4318 (signoz) | 000 | (n/a) |
internet (registry.opentofu.org) | 000 | 200 |
sillytavern-envoy:4000/v1/models | 503 | 200 |
Root cause in com.docker.backend.log: a sailor panic in
pkg/sailor/guestconnect_server.go:95, the VM restarting itself at 14:40:38Z,
and then sailorforward warnings that guest-connect dials were being rejected (status 1). The VM came back; its forwarding layer did not.
Fixed it the only way that works: full Docker Desktop restart, then hm switch,
which completed clean with tofu reporting 0 added, 0 changed, 0 destroyed.
All 20 containers self-restored before hm switch even ran.
Then checked the ClickHouse replication queue, since the previous five strandings
all followed a Docker event. Clean: 0 entries across two samples, num_postponed
0 and not climbing, 0 readonly of 55 replicas, max delay 0, zero detached parts
created after the restart, and 683,840 samples ingested in the last five minutes.
What was striking
Every note I have says to diagnose these by docker version hanging, and this
time it answered instantly. Both prior crash classes,
reference_docker_desktop_semawakeup_crash and the pthread one, present as a
hang with exit 124. Here the daemon returned 29.7.2 in well under a second,
docker ps listed every container as Up and healthy, and the compose stacks all
looked perfect. The written diagnostic would have cleared Docker as a suspect.
This is a third class: the VM survives its own panic and only guest egress dies.
The host is the wrong place to look, which is exactly why it hides. Nothing
host-side was wrong at any point. The one command that decides it is a curl to
host.docker.internal issued from inside a container. Container-to-container
traffic on a compose network kept working the whole time, so intra-stack health
checks stayed green and actively misled.
Host-gateway and internet failing together is the signature. Either alone would suggest something narrower. Both at once localises it to the VM’s NAT layer rather than to any service, any port binding, or any firewall rule.
The morning’s orchestrator loose end resolved itself and revealed a second
bug. That entry left orchestrator failing on tofu init being unable to
reach registry.opentofu.org, and read it as a transient connectivity window
shared with the GitHub 504. It was not transient: it was this same broken egress,
and it had been broken continuously for about ten hours. With networking restored
the tofu stage passes and Sonarr, Radarr and Seerr configure fine. It now exits 1
on Failed to apply Bazarr language profile (HTTP 500), which is a genuinely
separate and still-open problem.
A graceful restart does not strand the replication queue; an engine crash does. All the detached-part wreckage still tops out at 14:48:13 UTC, this morning’s incident. My 00:16 UTC restart added nothing, because an orderly quit lets ClickHouse flush. Five strandings, all of them following a crash rather than a clean stop. The check stays cheap and worth running regardless.
open -a Docker can silently no-op. Exit 0, no output, and zero processes
spawned, when issued too soon after the quit. I only caught it because
pgrep -f com.docker.backend | wc -l returned 0 well after the launch supposedly
succeeded. Verify the launch took rather than assuming it.
Grepping the litellm log for 503 false-positives. The hits are line numbers
inside exception_mapping_utils.py tracebacks, not status codes. Match on a
uvicorn request line instead.
Loose ends
orchestratorstill exits 1 on the Bazarr language profile HTTP 500. Unrelated to networking and untouched.- Did not re-audit this morning’s repair for data correctness. The rollup identity
and per-engine deltas were verified at the time and the
agg_30mgap was left in place deliberately; this session only established that the restart added no new damage. - The sailor panic is now the trigger for two distinct outcomes on the same day: one that killed the VM outright and one it survived. Whether they share a cause is unknown.