2026-08-25 Removing ClickHouse Replication Ends the Stuck-Queue Class

What I set out to do

After repairing the sixth stuck replication queue in three months, ask the obvious question: how do we get automatic recovery instead of doing this by hand every time.

What the answer turned out to be

Not automatic recovery. ClickHouse has no lost-part recovery knob for this at all — I searched system.merge_tree_settings on 25.12.5 and there is nothing. Its real mechanism, substituting an empty part for a lost one, only engages when a replica is behind other replicas. With total_replicas = 1 there is never a peer, so it can never fire. max_postpone_time_for_failed_replicated_fetches_ms = 60000 is exactly why num_postponed climbs forever: the impossible entry is retried on a 60s cap with no give-up path.

Which reframes the problem. The stack ran 55 Replicated*MergeTree tables against a one-node cluster. Replication bought nothing here — no HA, no failover, no peer — and its only effect was to create the ZooKeeper queue that wedges. SigNoz’s own docs say to disable it for a single replica. The SIGNOZ_OTEL_COLLECTOR_CLICKHOUSE_REPLICATION=true in casting.yaml.in was a historical accident: the comment said it was set to match tables the old vendored compose had already created.

And it costs no durability to remove. The rows die at crash time either way; SYSTEM RESTORE REPLICA only ever made the loss official. Plain MergeTree loses the identical rows, detaches the broken parts at boot, and carries on.

The near-miss

I proved the conversion on a throwaway table first, which is the only reason this went well. Two bugs surfaced there rather than on 2.6B rows:

DETACH TABLE ... PERMANENTLY does not rename <table>.sql. It writes an empty <table>.sql.detached marker beside the real file. My script found the marker, edited zero bytes, and aborted before ATTACH — the exact bug that would have left 55 real tables detached.

Worse, the script wrote its backup as conv_probe.sql.detached.pre-convert inside the metadata directory. ClickHouse refuses to start if a metadata directory holds any file with an unexpected extension: Code: 79 ... Incorrect file extension, thrown from iterateMetadataFiles during loadMetadata, before the server listens. The running server did not care, so it looked fine — until a docker restart from another window, and then a crash loop. Latent boot-blockers are the nastiest kind. Recovery meant docker stop, deleting the file through an alpine container mounted on the signoz-clickhouse volume, and docker start. About two minutes down, and entirely self-inflicted.

Also learned: with an Atomic database, metadata/<db> is a symlink into store/, so a metadata backup needs tar -h or you archive symlinks and get an 873-byte tarball instead of the real files.

The conversion

55 tables, cheapest first so any procedural failure would land on an empty table rather than samples_v4. Every one reported rows X -> X exactly, measured seconds apart around its own DETACH/ATTACH:

• signoz_metrics.samples_v4 2,666,666,172 rows, unchanged
• samples_v4_agg_5m 144,186,398, unchanged
• signoz_index_v3 10,669,456, unchanged

The engine rewrite preserved third arguments correctly, so ReplicatedReplacingMergeTree('...', '{replica}', computed_at) became ReplacingMergeTree(computed_at) on the seven tables that carry a version column.

Then the check I skipped the first time and will never skip again: restart ClickHouse and confirm it boots. It did.

Where it landed

system.replicas is now empty. Not “the queue is drained” — the table that holds the failure has nothing in it and structurally cannot. system.replication_queue can never have a row again. ZooKeeper dropped to 0.09% CPU because nothing talks to it.

Ingest verified flowing afterwards: 97,810 samples and 258 spans in three minutes.

One incidental finding worth keeping: the docker restart that exposed my stray file produced zero stuck entries and zero broken parts. A graceful stop goes through SIGTERM and stop_grace_period actually applies. All six incidents were ungraceful — macOS reboot, two Docker Desktop backend crashes, second-start races. The grace period was never useless, it just never got a turn.

Shipped

hm switch ran on the user’s go-ahead, generation 2249. It also carried three files already staged from another session (egress canary alerts, an httpcheck receiver, SillyTavern config) — I flagged that rather than shipping it silently, and they chose to include them.

The ingester and the migrator both now run with SIGNOZ_OTEL_COLLECTOR_CLICKHOUSE_REPLICATION=false. The migrator exited 0 with no errors against the converted plain tables, which was the main open question: whether SigNoz’s schema migrator would object. It did not, because all 40 migrations are already recorded in schema_migrations_v2 and sync up is a no-op for them.

After the full stack recreate: 0 replicated tables, 0 replicas, 0 queue entries, 0 newly detached parts. Ingest across all three signals — 147,227 samples, 411 spans, 189 logs in three minutes. ZooKeeper idling at 0.09%.

Left open

Retire the clickhouse_replication_queue_stuck alert — now a rule that can never fire — and drop the ZooKeeper container once this has soaked a few days. Both live in signoz.nix, which the other session was mid-edit on, so they are deliberately deferred rather than merged into this change.

Related: 2026-08-25 macOS 27 Upgrade and the Sixth Replication Queue Repair.