2026-07-05 SigNoz ClickHouse Replication Queue Root Cause and Graceful-Stop Fix

Context: Dotfiles (.config). Follow-on from 2026-07-04 SigNoz Alert Notification Gap and ntfy Decision — that entry left open why the single-node ReplicatedMergeTree keeps losing parts, since a manual rebuild is recovery, not a cure.

What I set out to do

Find the real, durable root cause of the recurring stuck replication queue (not just rebuild it a third time), check for open upstream issues, and decide whether we’re misconfiguring something.

What I actually did

Root cause — two layers.

1. Structural (SigNoz’s design): SigNoz runs every table as ReplicatedMergeTree backed by ZooKeeper even on a single node (total_replicas: 1). The engine’s recovery model is “fetch a missing part from another replica,” but with one replica there is no peer, so any part ZooKeeper’s log references but the node has lost becomes a permanently unfetchable GET_PART / NO_REPLICA_HAS_PART.
2. Trigger (ours): both signoz-clickhouse and signoz-zookeeper-1 had StopTimeout=1 (confirmed via docker inspect), and the hm switch activation force-recreated all SigNoz containers on any config-hash change (docker compose up -d --force-recreate, signoz.nix). So every SigNoz tweak SIGKILL’d ClickHouse ~1s after SIGTERM, mid write/merge — the block committed to ZooKeeper but the part never landed on disk. The heavy June SigNoz iteration (dockerstats, OpAMP, retention, scrape trimming, alerts, dashboards) bounced ClickHouse repeatedly, which is why it re-stuck the day after the 2026-06-04 rebuild. The recurrence was self-inflicted.

Upstream check. No open issue to track — this isn’t really an upstream bug. ClickHouse treats it as expected for single-replica Replicated + ungraceful stop, recovered via SYSTEM RESTORE REPLICA; the relevant history is closed (ClickHouse#26716 shutdown part deletion, #44865 stuck merges). SigNoz has no issue for single-node queue loss; running Replicated on one node is their deliberate design.

Fix (committed 6b0bcf5, branch signoz-clickhouse-graceful-stop off main).

• Set stop_grace_period in the compose override — clickhouse: 5m, zookeeper-1: 2m — so ClickHouse drains and shuts down cleanly instead of a 1s SIGKILL.
• Scoped the activation recreate: plain up -d --remove-orphans applies definition changes gracefully and leaves ClickHouse/ZooKeeper running unless their own definition changed; --force-recreate is narrowed to just the otel-collector (whose bind-mounted config compose can’t diff) with --no-deps so it never cascades into ClickHouse.
• Validated: docker compose config merge shows stop_grace_period: 5m0s / 2m0s on the two services; module evaluates with the scoped recreate rendered into the activation script.

What was striking

• The recurrence was self-inflicted. Not host flakiness or a ClickHouse bug: our own force-recreate on every config tweak, combined with a 1-second stop timeout, made each restart lossy. The cure is config we simply weren’t setting.
• docker inspect beat theorizing. The 1s StopTimeout was the whole story and one inspect call surfaced it; reasoning about “single-replica part loss” in the abstract would have kept pointing at ClickHouse instead of our compose.

Top 3 next

1. Merge signoz-clickhouse-graceful-stop; the next hm switch recreates ClickHouse once (now gracefully) to pick up the grace period, then stops bouncing it.
2. Consider a ClickHouse memory cap — the other ungraceful-death path (OOM under host load).
3. Wire the ntfy channel from 2026-07-04 SigNoz Alert Notification Gap and ntfy Decision so this alert actually pages next time instead of firing into the void.

Related

2026-07-04 SigNoz Alert Notification Gap and ntfy Decision, 2026-06-04 SigNoz ClickHouse Replication Queue CPU Burn, Homelab Services Architecture.