2026-07-07 SigNoz Alert Notifications - Template, Priority, Tailnet Dashboard, Readable Copy

Follow-on polish to the 2026-07-05 SigNoz Alerting Deploy - Graceful-Stop on Atlas, ntfy on Athena pipeline, after confirming a real litellm_empty_completions alert delivered to both phone and Mac. Three improvements, all shipped to main and verified live.

Custom ntfy template + severity → priority

The built-in alertmanager template was verbose (empty Instance/Source, mangled timestamps) and left priority null. Added programs.ntfy.templates.signoz (Go-template over the Alertmanager v4 payload): clean title (severity emoji + alertname), body = description, and a severity→ntfy-priority map (critical=5, error=4, warning=3, resolved=2) so criticals escalate on iOS. Channel URL repointed ?template=alertmanager → ?template=signoz. Verified critical → priority 5 through the real path.

Gotcha: changing a template file’s content does not restart the ntfy launchd agent (server.yml’s template-dir path is stable, so the agent’s ProgramArguments are unchanged and setupLaunchAgents skips it). Must launchctl kickstart -k gui/$UID/sh.ntfy.server to reload the template.

SigNoz dashboard over the tailnet

Atlas’s Tailscale Serve :443 root was already taken by Open WebUI, so extended tailscale-serve.nix with an extraPorts attrset (additional httpsPort → localPort mappings) and fronted SigNoz at https://atlas.tail0cef90.ts.net:8443 → 127.0.0.1:8082. Root-of-a-dedicated-port (not a /signoz subpath) avoids SPA base-path breakage. Tailnet-only. Caveat baked into the module docs: removing an extraPorts entry doesn’t tear the mapping down (tailscale serve --https=<port> off by hand).

Readable, actionable alert copy + dashboard link

Rewrote all 4 alert descriptions to a phone-friendly headline + impact line plus a → action line (exhaustive root-cause detail stays in the nix code comments). The template now leads with the description and appends 🔗 https://atlas.tail0cef90.ts.net:8443/alerts. Terraform applied the 4 changes in place (IDs unchanged). No per-alert deep link: SigNoz’s webhook payload doesn’t carry the rule ID, so a deep link would mean pinning generated IDs into config (rejected as fragile) — the alerts page is one tap away.

Verified

Fired disposable critical alerts through the real alertmanager path each time (create threshold rule on ClickHouseAsyncMetrics_ReplicasSumQueueSize above -1, poll ntfy, delete). Final render confirmed: 🔴 <name> / description / → action / 🔗 link, priority 5. User confirmed live delivery to phone + Mac.

Related

• 2026-07-05 SigNoz Alerting Deploy - Graceful-Stop on Atlas, ntfy on Athena
• reference_ntfy_alert_sink_athena
• Commits: template d249dcf, tailnet dashboard e9ddc46, alert copy 5251ca9

Follow-on: image feasibility + deep links

Image previews (Datadog-style): confirmed ntfy attachments display in the iOS notification (published a matplotlib graph PNG to the topic, user confirmed it shows). Enabling uploads needed attachment-cache-dir (added as programs.ntfy.attachmentCacheDir, commit 2ef1716) — without it, PUT-body attachments 400 “attachments not allowed”. But SigNoz has no panel-to-image API (Slack “advanced formatting” is just mrkdwn text; no channel embeds images), and the issue tracker has no feature request for it. So a graph-in-alert needs a relay service (renders PNG, PUTs to ntfy) — scoped, de-risked, deferred.

Rich webhook payload discovery: captured the raw Alertmanager JSON (flip the channel to passthrough = url without ?template=, fire a disposable alert, poll, restore). It carries commonLabels.ruleId, ruleSource/generatorURL (→ rule page), and commonAnnotations.related_traces / related_logs (→ traces/logs explorer). Host is localhost:8080 (internal port, not 8082). So per-alert deep links were possible all along — my earlier “no rule ID in payload” claim was wrong.

Deep links shipped:

• Triggered-alerts link (4df8a78): …:8443/alerts/history?ruleId={{ .commonLabels.ruleId }} — built from the payload ruleId + tailnet base (no host-rewrite). Points at /alerts/history (triggered timeline), not /alerts/overview (the rule), per user correction. Click-tested, lands correctly.
• Related traces/logs links (3865798): appends 🔍 related_traces / 📄 related_logs, conditionally (metric alerts stay clean), rewriting localhost:8080 → tailnet via Sprig replace (ntfy bundles Sprig). Verified all three alert-type renders.

Details in reference_ntfy_alert_sink_athena.

Image relay: investigated, then dropped as too much machinery

Explored attaching a Datadog-style graph to each alert. Findings:

• SigNoz has no server-side render/image API. /api/v1/render and /api/v1/snapshot are the SPA catch-all (return text/html), not endpoints; panels are client-side canvas (PR #11803 added only a browser “download PNG” button). No feature request upstream.
• Only path is a relay: webhook → GET /api/v1/rules/{ruleId} (compositeQuery + threshold) → POST /api/v5/query_range (v5 nests series at data.data.results[].aggregations[].series[].values[] = {timestamp,value} — a jq-path mistake made me briefly think the metric had no data) → render PNG → PUT to ntfy as an attachment. An attachment PUT can’t also use ?template=, so the relay would have to reproduce all the title/message/priority/deep-link formatting in code.
• ntfy attachments confirmed displaying on iOS (published a test PNG, user confirmed). Renderer chosen after a side-by-side (matplotlib vs Plotly+Kaleido vs Vega-Lite): Vega-Lite via vl-convert-python — self-contained Rust binary, no browser, nix-friendly.

Started scaffolding the relay (relay.py) but the user called it — a whole webhook→query→render→publish service + uv2nix packaging + launchd agent is too much machinery for a graph in a notification. Removed the scaffold and reverted the attachment-cache-dir config (cbb074c). The pipeline stays text + rich deep links, which is where SigNoz’s data lives anyway.

Final state of the alerting arc: graceful-stop fix live on atlas; ntfy sink on athena (deny-all, unguessable topic, iOS push); custom template with severity→priority; punchy actionable copy; per-alert triggered-alerts + related-traces/logs deep links; SigNoz dashboard on the tailnet at atlas:8443. No image relay.