2026-08-28 Sailor Panic With a Clean Replication Queue

Started as “what is this alert”, ended with the root cause of weeks of Docker Desktop crashes: com.docker.sailor leaks a thread pair per UDP flow, and 92.5% of the flows are qBittorrent’s DHT traffic on port 6881.

What I set out to do

Look at a SigNoz alert-history URL for rule 01a04670-8f26-7b62-941d-c2bf1de67e03, with no other context. The rule turned out to be LiteLLM unreachable from Docker containers.

What I actually did

The SigNoz API was unreachable, which became the investigation. localhost:8082 had a LISTEN socket owned by com.docker.backend but never answered, and docker version hung at exit 124: the engine was dead while the port-forward proxy outlived it.

With the API down I recovered the rule definition from ~/.local/share/signoz-terraform/terraform.tfstate. Worth remembering: the tofu state is a complete offline mirror of every alert rule, so an alert can be explained even when SigNoz itself is what’s broken.

Timeline (EDT):

Time	Event
~16:08	httpcheck probes to LiteLLM start failing from inside the collector
16:13:21	Alert fires
16:14:20	Host-side notifier pushes ntfy
16:14:46	sailor panics, engine running -> stopping
16:19:47	VM never powers off; 5-minute deadline expires, crash dialog shown
16:24:21	Alert resolves after recovery

docker desktop start was a no-op: it reports “already running” off the app rather than the engine, while status still read stopped. docker desktop restart worked. All 21 containers self-restored and hm switch ran clean.

Root cause

The panic message was a decoy. Read in order:

runtime/cgo: pthread_create failed: Resource temporarily unavailable   <- ACTUAL error (EAGAIN)
runtime.throw(...)                                                      <- runtime tries to report it
panic during panic
runtime.sighandler(0x6, ...) / runtime.sigtrampgo(0x6)                  <- SIGABRT on the signal stack
fatal error: semasleep / semawakeup on Darwin signal stack              <- runtime DIES while reporting

semawakeup on Darwin signal stack is the Go runtime failing to print a crash. golang/go#75051 is a reporting bug that had been masking the real error for weeks, which is why my notes had it filed as the cause and split into two crash classes. One note had recorded the cause, the other the mask.

The real failure is thread exhaustion. com.docker.sailor grows ~50 threads/min and dies at kern.num_taskthreads (16,384). An A-B-A test (pause all containers, wait, unpause) isolated it: +51.8/min running, -0.9/min paused, +58.8/min resumed. Leaked threads are never released. RSS tracks it linearly at 1.488 MB × threads + 14,812, about 1.5 MB of stack per leaked thread.

sample named the mechanism, and it is read-only, so no pausing was needed after all:

1711  thread_start -> _pthread_start -> libsailor -> __recvfrom
1667  thread_start -> _pthread_start -> libsailor -> _pthread_join -> __ulock_wait

Every flow gets a reader thread parked in recvfrom plus a partner that calls pthread_join on it. The reader never returns, so the joiner blocks forever, and both pin permanently.

And sample records thread names, which gave the attribution outright:

thread class	count	share
net-stack-udp-reader	1,703	48%
net-stack-udp-<src>-><dst>	1,703	48%
net-stack-tcp-reader	8	0.2%
fs-watcher / virtio-blk / misc	~130	3.7%

Of 1,703 UDP flows, 1,575 have source port 6881 across 624 distinct destination ports. That is qBittorrent’s BitTorrent port talking to DHT peers across the internet, confirmed against its own UDP sockets (172.20.0.11:6881, 127.0.0.1:6881, plus 6771 for Local Service Discovery).

Removing BitTorrent UDP should take the leak from 50.7 to 3.8 threads/min, moving time-to-ceiling from ~4 hours to ~54 hours. A 13× improvement, and the falsification test is just toggling DHT/PeX/LSD off, which needs no restart.

What I got wrong

I measured the wrong protocol. I built a whole per-container churn harness on /proc/net/tcp and produced a confident attribution table from it. The leak is UDP. Eight of 3,547 threads are TCP. That table was worthless.

I dismissed the right answer explicitly, saying qBittorrent “shows only 6 external connections, so it isn’t swarming”. DHT is UDP, so the number I was reading could never have shown it.

I concluded “diffuse, no single culprit” and retired chatter reduction as a lever. It is one container at 92.5%, and it is the best lever available.

I undersold the upstream match. docker/desktop-feedback#633 is specifically a UDP endpoint leak in sailor on the same build (4.88.1 / engine 29.7.2), leaking ~60/min into a 16,384 ceiling, cleared by restart, not leaking when idle. Same component, same protocol, same never-reaped-per-flow shape. Probably one bug: leaked UDP endpoints on Windows, leaked thread pairs on macOS. I called it “corroboration of the leak class, not our exact bug”, which was too weak.

I asserted macOS 27 as the trigger on a confounded date. Docker 4.88.1 is ruled out for onset, but log retention does not reach 2026-08-17 and Docker 4.87.0 landed around then too. The April “control” was also weak.

I said sailor was a rename of libkrun. Both ship side by side and libsailor.dylib has zero krun strings. They are separate components.

The pattern in all of these: I read a number, formed a story, and moved on without asking what the number could not show. The thread names were in the sample output the entire time; I parsed stacks and ignored the header field on every line.

Other findings

The replication queue was clean, despite the VM failing to power off within its deadline. Every prior unclean shutdown stranded GET_PART entries. So that implication is a correlation, not a law.

Open WebUI is never idle and its traffic is its own OTEL export, round-tripping out to host.docker.internal:4317 and back because it and the ingester sit on different Docker networks. It is 0.2 connections/min and irrelevant to the leak, but its telemetry dies during exactly the outages the alerting exists to catch.

Sailor is closed source. github.com/docker/sailor and github.com/docker/pinata both 404; the dylib is stripped Rust. sample thread names are as deep as external diagnosis goes, which turned out to be deep enough.

The alerting stack held throughout, including firing and self-resolving around the deliberate quiesce window.

The remedy test: container restart frees nothing

Ran the two restarts back to back at 22:51 to settle whether a targeted container restart could reclaim the leaked threads.

Action	before	after	freed
docker restart qbittorrent	11,709	12,153 (t+45s)	0, and it cost ~440
Docker Desktop restart (kills sailor)	12,203	260	11,943

A sample taken seconds after the container restart still showed 5,845 UDP flow pairs, 85.3% of them on source port 6881. The guest-side sockets were gone but sailor’s host-side threads persisted with nothing on the other end, which is direct evidence that there is no reaper. Then qBittorrent re-bootstrapped DHT and opened a burst of new flows: ~440 threads in 45 seconds against a 20-60/min baseline, decaying back within about two minutes.

So a container restart is strictly negative, and any watchdog has to act on the engine rather than on a container. RSS confirms the engine restart is a true reset: 22.4 GB down to 12.0 GB.

One process lesson. The engine went stopping at 12,203 threads, far below the 16,384 ceiling, and I immediately started reading com.docker.backend.log for a panic. There was no panic. The user had pressed restart in the GUI. An engine state change is not evidence of a crash on its own.

Also worth recording: qBittorrent had zero torrents loaded the entire day (torrents/info returned [], 0 peers, dht_nodes=546). The whole leak came from DHT routing-table maintenance for an empty client, which removes the tradeoff argument for leaving DHT on.

Top 3 tomorrow

1. Toggle DHT / PeX / LSD off in qBittorrent and confirm the slope drops to ~4/min. Falsifies or confirms the whole attribution in minutes, with no restart. With zero torrents loaded this currently costs nothing.
2. Make the change durable in the media-stack flake rather than by hand, since qBittorrent rewrites qBittorrent.conf at shutdown.
3. Put OWUI and the ingester on a shared network so its telemetry survives the guest-networking break.

Related

2026-08-26 The Alert Fired and There Was No Crash 2026-08-25 Alerting on Docker’s Guest-Network Break 2026-08-25 macOS 27 Upgrade and the Sixth Replication Queue Repair 2026-08-23 Sailor Crash and the Fifth Replication Queue Repair 2026-08-23 The 503 Was Docker’s Guest Networking, Not LiteLLM