Achhina's Digital Garden

2026-07-05 SigNoz Alerting Deploy - Graceful-Stop on Atlas, ntfy on Athena

4 min read

2026-07-05 SigNoz Alerting Deploy - Graceful-Stop on Atlas, ntfy on Athena

Shipped the SigNoz alerting work from 2026-07-04 SigNoz Alert Notification Gap and ntfy Decision and 2026-07-05 SigNoz ClickHouse Replication Queue Root Cause and Graceful-Stop Fix to main and deployed it across both hosts.

What shipped

  • Rebased signoz-alerting onto main (no conflicts: main’s 7 intervening commits were the athena profile split, touching none of the 4 alerting files), fast-forwarded and pushed. Two commits: the graceful-stop fix and the ntfy module + ADR 0019.
  • Follow-up commit a3ce5db enables ntfy in athena’s curated home-server.nix profile (also fixed a statix inherit lint in ntfy.nix that the full pre-commit scan surfaced).

Atlas (graceful-stop fix)

hm switch on atlas regenerated the compose files and recreated ClickHouse + ZooKeeper once (gracefully; queue was at 0 beforehand). Verified: StopTimeout now 300s / 120s, replication queue stayed at 0, all containers healthy. otel-collector recreated via the new scoped --force-recreate --no-deps path rather than the old blanket recreate.

Athena (ntfy notification server)

  • Curated profile imports ntfy.nix + enables it; fronted over tailnet HTTPS by Tailscale Serve at https://athena.tail0cef90.ts.net:443 -> 127.0.0.1:2586. Running as launchd agent sh.ntfy.server, ntfy 2.25.0.
  • Provisioned deny-all auth per ADR 0019: signoz user (write-only) + phone user (read-only) on topic signoz-alerts. Token minted. Secrets stored in a chmod-600 file on athena (~/.local/share/ntfy/provisioned-secrets); deliberately kept off the session transcript (the auto-mode classifier correctly blocked echoing them).
  • Repointed SigNoz’s local-placeholder webhook channel (id 019dbfe8-...) from the dead host.docker.internal:9999 to https://athena.tail0cef90.ts.net/signoz-alerts?template=alertmanager with basic auth, via the SigNoz API on atlas (sourcing the password in-process). send_resolved=true.

Verification

  • Docker-container reachability was the real risk (SigNoz runs in Docker on atlas, ntfy on athena over the tailnet). Confirmed the signoz container resolves MagicDNS and reaches https://athena.tail0cef90.ts.net/v1/health with valid TLS.
  • SigNoz channel-test delivered end-to-end: ntfy received the alertmanager-templated message (🚨 Alert: Test Alert (local-placeholder) / Severity: critical), confirmed by polling as the read-only phone user.

Remaining / optional

  • Phone app (manual, user): install/log into the ntfy iOS app as phone (creds in athena’s secrets file), subscribe to signoz-alerts. iPhone was offline (last seen ~22h) at deploy time.
  • Priority mapping (optional): the built-in alertmanager template leaves priority: null, so criticals won’t escalate on iOS. A custom programs.ntfy.templates entry mapping severity → ntfy priority would fix that.

Follow-up: topic rotation + alertmanager hot-reload gotcha

Checking the ntfy iOS app surfaced that the topic signoz-alerts was guessable. On our server the content is safe (deny-all auth), but the upstream-base-url=https://ntfy.sh APNs relay forwards empty wake-up polls carrying the topic name to the public server, leaking alert-timing metadata. Rotated to an unguessable random topic:

  • ntfy access grants moved to the new topic for both users; old signoz-alerts explicitly denied. New topic stored in athena’s provisioned-secrets file, kept out of git (repo is private, but defense-in-depth). Fixed the now-stale topic comment in home-server.nix (commit 06bbc2a).
  • Verified: authenticated publish to the new topic → 200, anonymous → 403.

Gotcha found in the process (and a wrong turn I later corrected): SigNoz’s test endpoints are misleading. POST /api/v1/channels/{id}/test is a no-op — returns 200 but publishes nothing — which I initially misread as evidence that the running alertmanager config was stale and needed a restart. POST /api/v1/testChannel does deliver, but it uses the config in the POSTed body, so it only proves url/auth/template/network, not what the alertmanager has loaded. Neither proves real-alert delivery.

I did docker restart signoz (a safe way to force the alertmanager to reload receivers from the channel store), then verified definitively with a disposable always-firing threshold_rule on ClickHouseAsyncMetrics_ReplicasSumQueueSize (above -1, at_least_once, 1m frequency, channel local-placeholder): it went firing within ~2 min and delivered the templated 🚨 Alert: zzz-delivery-verification-temporary to the rotated topic — proving real alerts route through the loaded config to ntfy. Deleted the temp alert after. Verification rule recorded in reference_ntfy_alert_sink_athena: never trust the test endpoints; confirm with a real disposable alert firing.

Graph View

Backlinks