Achhina's Digital Garden

2026-07-13 ClickHouse Replication Alert Was Atlas Sleeping

3 min read

2026-07-13 ClickHouse Replication Alert Was Atlas Sleeping

What I set out to do

Investigate a SigNoz alert, “ClickHouse replication queue not draining”, that fired at 2:24pm and resolved at 2:26pm EDT. The obvious hypothesis was a recurrence of the June 4th stuck GET_PART incident, where an impossible replication-queue entry retried forever and pinned ClickHouse + ZooKeeper CPU.

What I actually did

It was not the queue. ClickHouseAsyncMetrics_ReplicasSumQueueSize was flat 0 all day. The alert fired on its alertOnAbsent dead-man’s switch (state nodata), not on its threshold.

The chase went storage-layer-first, which paid off:

  • The scrape had real gaps: dense 60s samples until ~15:38 UTC, then holes of 26–52 minutes. The 51-minute gap tripped absentFor: 30m.
  • macmon_cpu_active_ratio, a completely different scrape target, gapped at identical timestamps. So did traces. Everything stopped at once.
  • Not the collector: signoz-otel-collector up 4 days, no restarts, no scrape or export errors. ClickHouse answered queries instantly, so the Docker VM wasn’t wedged either.
  • pmset -g log closed it: atlas was in a continuous Maintenance Sleep → 45-second DarkWake → sleep cycle, all on Using Batt. pmset -g custom shows sleep 0 on AC but sleep 1 on battery. The charger was out, lid closed.

The 2-minute “fire → resolve” the alert reported was an artifact: the DarkWake windows at 14:23:46 and 14:25:06 EDT line up exactly with the two notifications. SigNoz’s ruler and its ntfy webhook only run while the machine is briefly awake, so a 90-minute firing period flushed out as two pings two minutes apart.

Fix (commit f767c06): dropped alertOnAbsent/absentFor from the rule. A stuck GET_PART entry cannot self-heal, so the queue is still >0 when the scrape resumes and the threshold branch fires then. Absent data was never evidence of the failure this rule exists to catch.

What was striking

Two things.

The signature of a sleeping host is that everything gaps at the same timestamps with nothing broken. No restarts, no errors, no resource pressure — because nothing failed, the machine just wasn’t running. That’s a genuinely different shape from any application fault, and worth recognising on sight.

A dead-man’s switch that fires on a condition the host produces routinely is worse than no alert. It had been firing on a dozen prior dates and I’d been reading them as ClickHouse problems. The alert was well-intentioned and actively misleading, because its name pointed at the wrong subsystem.

Deliberately did not “fix” the sleep with pmset -b sleep 0 — that flattens the battery rather than sleeping. atlas is a MacBook doing home-server duty; the real answer is to keep it plugged in.

Graph View

Backlinks