Achhina's Digital Garden

2026-07-04 SigNoz Alert Notification Gap and ntfy Decision

3 min read

2026-07-04 SigNoz Alert Notification Gap and ntfy Decision

Context: Dotfiles (.config). Started from a firing SigNoz alert link and the question “why is it firing, and why weren’t we notified.”

What I set out to do

Investigate a firing SigNoz alert (clickhouse_replication_queue_stuck) and figure out both why it fired and why no notification ever arrived.

What I actually did

Worked it end to end: diagnose the fire, fix the underlying cause, then decide and scaffold the missing notification path.

Why it fired. ClickHouseAsyncMetrics_ReplicasSumQueueSize had been stuck at ~9 (never draining to 0) for the whole 7-day window, and the alert history showed it firing continuously since 2026-06-05 16:57 — the day after the 2026-06-04 SigNoz ClickHouse Replication Queue CPU Burn rebuild. So the original fix didn’t hold: parts got lost again within a day and the queue re-stuck.

Why no notification. SigNoz had exactly one notification channel, local-placeholder — a dead webhook pointing at http://host.docker.internal:9999. All four terraform-managed alerts in signoz.nix route to it by name, so every alert in the stack was silent, which is how a month-long fire went unnoticed.

Queue rebuild (recurrence). system.replication_queue showed 6 tables with 9 GET_PART entries failing NO_REPLICA_HAS_PART, retried ~2,900+ times each. Rebuilt each replica table-by-table (DETACH ... SYNC / SYSTEM DROP REPLICA ... FROM ZKPATH / ATTACH / SYSTEM RESTORE REPLICA / SYSTEM RESTART REPLICA). Result: queue → 0, ClickHouse 117% → 6.65% CPU, ZooKeeper 120% → 0.21%.

Notification decision (ADR 0019). Compared ntfy vs Gotify vs email vs Discord/Slack. Verified against the live instance (SigNoz v0.129.0) that its webhook sends a fixed Alertmanager JSON body with basic/bearer auth only and no custom headers. First concluded ntfy would need a relay; then found ntfy’s own message templating — a built-in alertmanager template, selectable via a ?template= URL query param (no headers), able to set priority from the body — which absorbs the payload with no relay service. Confirmed the SigNoz-side gap is an open, low-traction upstream request (SigNoz/signoz#9427). Landed on self-hosted ntfy.

Scaffolding. Created worktree signoz-notify-ntfy off main containing only ADR 0019 and a disabled-by-default programs.ntfy Home Manager module (nixpkgs ntfy-sh as a launchd/systemd agent, Nix-generated server.yml, mirroring the macmon pattern). Eval-verified in both states. Committed 0d41da6, to be enabled on athena once the 2026-07-04 Home Manager athena Server Profile porting lands.

What was striking

  • Recovery is not a cure. The 2026-06-04 rebuild worked but the single-node ReplicatedMergeTree keeps losing parts and re-sticking. The durable fix (plain MergeTree? merge/fetch tuning?) is still open.
  • Verify beat reasoning from memory. I corrected myself twice on the ntfy/relay question — first “no adapter needed,” then “needs a relay,” finally “no relay, ntfy templating absorbs it” — each turn only resolved by reading the actual SigNoz webhook schema and ntfy docs rather than reasoning from priors.
  • One dead channel silenced everything. A single black-hole webhook muted all four alerts; the blast radius of a placeholder never replaced was the whole alerting surface.

Top 3 tomorrow

  1. Enable programs.ntfy on athena after the server-profile porting lands; provision ntfy auth (publish token for SigNoz, read user for the phone).
  2. Repoint the SigNoz local-placeholder channel → the ntfy topic with ?template=alertmanager (a change in signoz.nix); consider a custom severity→priority template.
  3. Investigate the durable fix for repeated part loss on the single-node ReplicatedMergeTree. Resolved 2026-07-05 — root cause was ungraceful 1s SIGKILL + blanket force-recreate; fixed with stop_grace_period + scoped recreate. See 2026-07-05 SigNoz ClickHouse Replication Queue Root Cause and Graceful-Stop Fix.

2026-06-04 SigNoz ClickHouse Replication Queue CPU Burn, 2026-07-04 Home Manager athena Server Profile, Homelab Services Architecture — and ADR 0019 in the dotfiles repo (docs/decisions/0019-signoz-alert-notification-channel.md).

Graph View

Backlinks