2026-08-25 macOS 27 Upgrade and the Sixth Replication Queue Repair

What I set out to do

Run the routine post-upgrade check for a stuck ClickHouse replication queue on atlas, after upgrading to macOS 27.0 (build 26A5421a). The alert 019fcc74-3f22-7500-a25c-dad25cd9ca33 was already showing history worth looking at.

What I found

Sixth recurrence. Fifteen GET_PART entries carrying Code 234 NO_REPLICA_HAS_PART across eight tables, in both signoz_metrics and signoz_traces.

New trigger class, though the shape is the 2026-08-17 one: two stack starts, not a crash. The upgrade reboot brought SigNoz up around 18:41 UTC and the queue entries were all created 18:41:17-18:41:52 during that first recovery. A second start seven minutes later broke exactly the parts those entries were waiting on: 197 broken-on-start plus 1,686 covered-by-broken in seven seconds at 18:48:02-18:48:09, and the ClickHouse process that answered my queries dated to 18:48:57. No Docker Desktop panic in the log this time, so this is neither the semawakeup nor the sailor pthread_create class.

Proving it stuck mattered again. CPU was 35% ClickHouse / 2.4% ZooKeeper, nowhere near the ~120% pinning of the first two incidents, and count() sat frozen at 15. What moved was num_postponed: 225,572 to 226,376 across about ninety seconds, with num_tries creeping 1888 to 1898. Two samples, not one.

The alert had fired at 19:15:14 with value 15, thirty-four minutes after the queue went non-empty, which is the 30m window behaving correctly. No resolve row yet at that point; it went inactive at 20:55:14, so 1h40m firing in total.

The repair

Standard per-table replica rebuild driven by a scratchpad repair.sh that reads zookeeper_path and replica_name before the DETACH (the system.replicas row disappears with the table). The run_batch.sh driver was blocked by the auto-mode classifier, same as the inline loops in prior incidents, so I ran the eight tables one call at a time. Smallest first, the two plain ReplicatedMergeTree tables last.

Deltas, read by engine:

table	engine	delta
top_level_operations	Replacing	27 → 8
dependency_graph_minutes_v2	Aggregating	-845 (-0.6%)
metadata	Aggregating	-62,428 (-7.7%)
trace_summary	Aggregating	-34,094 (-0.4%)
samples_v4_agg_5m	Aggregating	-173,839 (-0.12%)
samples_v4_agg_30m	Aggregating	-211,484 (-0.65%)
signoz_index_v3	plain	+0
samples_v4	plain	-1,195,183 (-0.045%)

signoz_index_v3 at exactly +0 is the one piece of honest evidence a plain no-dedup engine can give, and samples_v4’s -0.045% sits right on top of the -0.046% and -0.048% from the two prior incidents.

metadata at -7.7% looked out of band until I checked partitions rather than totals: today’s 2026-08-25 partition holds 23,704 rows against 15k-32k on the six preceding days. The drop is merge collapse on an aggregating engine, not the whole-day loss its part name 20260825_0_18259_3723 predicted.

Verification

Queue at 0 entries, 0 readonly replicas, max(absolute_delay) 0. CPU fell to 12.8% / 0.25%. The rollup identity over the closed 19:00 hour held exactly: samples_v4 count() under bitAnd(flags,1)=0 and samples_v4_agg_5m FINAL sum(count) both 683,426, the same number I captured before touching anything.

Nicest confirmation was watching the alert’s own source metric drain in real time: ClickHouseAsyncMetrics_ReplicasSumQueueSize went 15, 15, 15, 15, 10, 7, 3, 0 across the minutes the repair ran, then held 0. The alert resolved on the next eval cycle after that, at 20:55:14.

Small trap in rule_state_history_v0: the inactive row carries value = 7, not 0. That is the sample the ruler was holding when the all_the_times condition broke, not the queue depth at resolution, so a non-zero value on a resolve row is not evidence of a partial fix.

What I decided not to do

top_level_operations came out at 13 rows over 7 distinct (serviceName, name) keys, down from 27 over 11. The 2026-08-21 incident is the one case where re-attaching the RESTORE-detached batch was worth it, and the parts were sitting right there with reason='' and today’s timestamp. I left them. Rows are approximately equal to distinct keys and the service list matches signoz_index_v3 over 24h exactly (claude-code, litellm-proxy, open-webui, sillytavern), which is the 2026-08-23 signature for sorting-key collapse rather than loss. The table was already refilling from live traces while I looked at it, 8 to 10 to 13 rows in a few minutes. The lesson that keeps holding is that the table left alone verifies clean and the table “repaired” goes corrupt.

Also left alone: dependency_graph_minutes_v2 shows nothing before the 18:00 hour today, but the host was off for the upgrade, so there was never anything there to lose.

Related: 2026-08-23 Sailor Crash and the Fifth Replication Queue Repair, 2026-08-21 Docker Crash Stranded the ClickHouse Replication Queue, 2026-08-17 SigNoz Replication Queue Third Recurrence and Rollup Repair.