2026-08-26 The Keeper Is Not There For Replication

What I set out to do

Two follow-ups from removing ClickHouse replication: retire the now-pointless alert, and drop the ZooKeeper container. Both turned out to be wrong as stated.

The alert stays, with inverted meaning

Adam’s instinct was right that it can still fire. I had assumed the metric would vanish once system.replicas was empty; it does not. ClickHouse computes the Replicas* async metrics regardless, so ClickHouseAsyncMetrics_ReplicasSumQueueSize sits at a steady 0 rather than going nodata. The rule is live, it just means something different now: any non-zero value can only be a Replicated*MergeTree table having been recreated. Renamed it and rewrote the runbook to lead with finding what recreated the table.

The real find was a latent defect. requiredNumPoints = 15 was tuned against a healthy window of 30 points at 1/min. But atlas drops to 1-4 points per 30m window while it sleeps on battery, so the gate made the rule structurally unable to fire for hours at a stretch. That trade made sense when the alert tracked a recurring condition and false positives were expensive; for a detector whose expected value is a permanent 0, a miss costs far more than a spurious page. Lowered it to 3.

ZooKeeper cannot be removed, and I found out the hard way

I removed it, and the migrator started crash-looping:

Error: store not ready due to non-retryable error:
code: 999, message: Cannot use any of provided ZooKeeper nodes

The keeper is not there for replication. It backs ON CLUSTER DDL:

distributed_ddl:
  path: /clickhouse/task_queue/ddl

Every SigNoz migration runs ON CLUSTER, so with no keeper the migrator cannot pass its readiness check. Ingestion was never affected, which is exactly what makes this nasty: the failure is invisible until some future upgrade cannot migrate. I reverted to a working ZooKeeper before doing anything else, then left a four-line comment on the block so nobody repeats it.

Also learned: Foundry has no “no keeper” mode. Dropping the telemetrykeeper block does not omit the component, it silently substitutes ClickHouse Keeper. My first test render looked like a success because the zookeeper patch paths failed to resolve, which I misread as “the service is gone” when it had really been renamed underneath me.

What actually shipped

Swapping ZooKeeper for ClickHouse Keeper, which retires the ZooKeeper container without breaking DDL. Same protocol, no JVM, and it drops the user: root override the bitnami image needed. Resident memory went 886MiB to 72MiB. The keeper image tracks the ClickHouse server pin at 25.12.5.

The keeper volume is deliberately not remapped onto signoz-zookeeper-1: clickhouse-keeper uses its own on-disk format and must start empty. Nothing is lost, because with no replicated tables the only state a keeper holds is the ON CLUSTER queue, and applied migrations live in ClickHouse’s schema_migrations_v2. The old volume is left in place for rollback.

Verified rather than assumed: ON CLUSTER CREATE and DROP both return status 0 against the new keeper, the migrator exits 0, system.zookeeper lists /clickhouse and /keeper, and metrics, traces and logs all keep ingesting (228,356 / 354 / 92 in three minutes).

Two things worth carrying

ClickHouse does not reload config-0-0.yaml on hm switch. It is a bind-mounted file, so compose sees no service change and leaves the container running on its old in-memory config. I hit this twice before internalising it: any keeper endpoint change needs an explicit docker restart of ClickHouse afterwards.

Adam pushed back on my comments (“i consider inline comments to be codesmell”) and was right, against feedback I already had recorded. Having just finished the incident, I wrote its history into the config: dates, “cost six manual rebuilds”, “measured 2026-08-26”, a 20-line rationale essay. Doing the work is precisely when the history feels most worth recording and precisely when it belongs in a journal entry like this one instead. Trimmed ~50 comment lines to ~22, all of the form do not re-enable this, here is what breaks.

Related: 2026-08-25 Removing ClickHouse Replication Ends the Stuck-Queue Class.