2026-08-21 Docker Crash Stranded the ClickHouse Replication Queue
What I set out to do
Work out what had happened to the Docker daemon on atlas. A docker version call
had hung rather than returning, which is itself the diagnostic: the CLI blocks
when the engine stops answering.
What I actually did
Docker Desktop had crashed roughly thirty seconds earlier. com.docker.backend
hit a Go runtime fatal error, semawakeup on Darwin signal stack, then sent
desktop state:ExitBadState, shut down Electron, and closed
~/.docker/run/docker.sock. All ~20 containers went with it.
Two corrections to my own first reading, both worth keeping. I said the Linux VM
was dead because no docker-vmm or qemu process existed. That was wrong: under
Apple’s Virtualization framework the VM lives inside com.docker.backend itself,
so there is no separate process to find. vm/init.log showed the engine serving
API calls normally until 22:56. I also attributed the bug to golang/go#48925,
which is a closed 2021 netpoll deadlock and simply the wrong issue. The right one
is golang/go#75051, still open,
milestone Go1.28, whose reporter’s trace carries the identical signal 6 and the
same sys_darwin_arm64.s:227 +0x4c frame. It is an upstream Go bug that Docker
Desktop only triggers by being a large Go binary, so no Docker release fixes it.
The signature appears 18 times in a single four-day backend log.
Then the part that mattered. The unclean shutdown stranded the SigNoz ClickHouse
replication queue for the fourth time: 18 GET_PART entries, all
Code: 234 NO_REPLICA_HAS_PART, across 11 tables. The chain is tight. Entries
were created 22:56:12 to 22:56:53, the backend died at 23:01:56, and the restart
broke the parts those entries needed at 23:03:36. Every previous occurrence was
triggered by an operator restart. This one was a crash, which means it will
recur on its own.
Rebuilt all 11 replicas with the usual DETACH, DROP REPLICA, ATTACH, RESTORE,
RESTART, ordered by engine: Replacing first, then Aggregating with agg_5m
before agg_30m, then the plain MergeTree tables last. Queue back to zero, all
55 replicas healthy, max delay 0, ClickHouse 14% to 8.8%.
What was striking
Low CPU is not a screening signal. The first two occurrences pinned
ClickHouse and ZooKeeper near 120% and retried about 98 times a second. This time
num_tries was frozen at 42 per entry across a ten second sample and CPU sat at
14%. ClickHouse had backed off exponentially. The queue was just as stuck; it was
simply quiet about it. Check system.replication_queue, never CPU.
A table can lose almost everything while the percentages look fine.
top_level_operations holds 81 rows but has churned through ~40,000 block
ranges, and RESTORE detached its big covering part. It went from 79 rows and ~60
distinct service/operation pairs down to 15 rows and 9 pairs, a loss of 81%. It
is a ReplacingMergeTree, which is the one engine where re-attaching the
RESTORE-detached batch is safe, so I attached today’s 133 parts and it came back
to 60 pairs. On the other Replacing tables RESTORE cost 100 rows out of 4-6
million, about 0.002%, and re-attaching was pointless. The lesson from 08-17 was
“never re-attach except on Replacing”; the refinement is that even there it is
only worth doing when the measured delta is large.
+0 on a plain MergeTree is not guaranteed. I had written down after 07-31
that samples_v4 returning exactly +0 rows is the only honest proof the
procedure preserves data. This time both plain tables lost rows:
signoz_index_v3 20,446 (0.2%) and samples_v4 1,039,182 (0.048%). Not
recoverable, since re-attaching there is what duplicated 132,402 spans last time,
and small enough to accept. But +0 is an outcome, not a rule.
The rollup rebuild needed a sharper recipe than the one I had. RESTORE
punched holes: agg_5m lost 13:00-22:59 and agg_30m lost 00:00-22:59, with raw
samples_v4 intact so both were rebuildable. Three things the old note did not
cover:
- The anti-join has to be on the full sorting key, not the bucket. A bucket can hold keys that are present but undercounted, because the MV rolled up only some blocks before the crash. Bucket 13:20 looked populated and was short 5,028 samples.
- An anti-join cannot fix those keys, since it skips anything already present.
That bucket needed
ALTER TABLE ... DELETE WHERE unix_milli = <bucket>and a wholesale re-insert. After that it matched raw exactly at 31,026. - Backfilling
agg_5mfiressamples_v4_agg_30m_mv, which wrote 65,800 partial 30m rows for the same keys. Deleting that range before rebuildingagg_30mis mandatory, not tidiness: those rows would have summed on merge and inflated it exactly as on 08-17. Detaching the MV to dodge the cascade is worse, since it gaps the live bucket instead.
Bounded every rebuild to closed buckets, ending at 23:00 while the clock read
23:50, so the MVs kept the open region. Final verification: all 21 hours of
Aug 21 read diff 0 across samples_v4, agg_5m and agg_30m. Value sums agree
to within 56 and 80 out of 3.86e16, which is Float64 accumulation drift at about
1.5e-15 relative, not corruption. The live 23:00 hour matches across all three
and is still growing.
Loose ends
- Raw metric ingestion had already stopped around 20:30 and did not resume until
23:03, so 21:00-22:59 is empty in every table including
samples_v4. Nothing was ingested, so no rebuild can recover it. Docker was unhealthy for about two and a half hours before it actually died, which is worth watching for as an early warning next time. - The alert behaved correctly: fired 23:30:14 with value 18, about 27 minutes after the queue went non-empty, matching the 30m window.
- Because the trigger is an unfixed upstream crash rather than something I did, this will happen again. The queue check now belongs in the standard recovery path after any Docker crash, not just after a deliberate restart.