Achhina's Digital Garden

2026-08-27 Moving Alert Delivery Out of the VM

5 min read

2026-08-27 Moving Alert Delivery Out of the VM

What I set out to do

Close the loose end from 2026-08-26 The Alert Fired and There Was No Crash: the egress alert fired correctly four times and no ntfy push ever arrived, because SigNoz’s Alertmanager sends from inside a container and the outage kills exactly that path. Build the host-side notifier that inverts the dependency.

Asked first whether it should replace the in-container webhook or sit beside it as a watchdog scoped to the one failure mode. Answer: replace it entirely, single path, single point of failure accepted knowingly.

What I actually did

Wrote the poller. A launchd agent on atlas reading /api/v1/alerts (SigNoz’s Alertmanager instance view, which carries per-series labels, annotations, startsAt/endsAt and fingerprint) every 30s, grouping per ruleId, and POSTing the Alertmanager v4 body to the same ?template=signoz URL. The existing ntfy template, its severity to priority map and its deep links are reused untouched. Stdlib-only Python invoked by store path, the same shape as the existing patch-otel-config.py call: this is the thing that tells you the rest is broken, so it should not depend on an environment that can rot.

Two properties are load-bearing now that it is the only path. Undelivered payloads persist in a queue and retry oldest first, stopping at the first failure so a resolved can never overtake the firing it resolves, and state survives restarts so bouncing the agent is not a notification event. And after five consecutive failed polls it publishes a synthetic “SigNoz unreachable from the host” critical, since a stack that is down cannot alert on itself.

Found that disabling SigNoz-side delivery is harder than it looks. Three cleaner options, all tried against the live server, all fail: deleting the channel is rejected the moment a rule names it, a channel declaring no notification config is rejected outright, and usePolicy: true with no channels is accepted but still routes to the same default-receiver. So a channel must exist and must declare a notifier. It now points at http://127.0.0.1:9/, the loopback discard port inside the container’s own netns.

Verified end to end. Disposable rule, watched it fire, watched the log: firing published 12s after it fired, resolved 13s after the alert aged out. 26 tests wired as a flake check, just check green, committed as cd32529.

What was striking

The bug was a dependency cycle, not a misconfiguration. The notification path ran through the subsystem being monitored. Once framed that way every variant that keeps delivery inside the VM is obviously dead, including the elegant-looking one where the webhook posts to a host-side receiver on host.docker.internal: the POST still has to leave the container. The only question worth asking was which direction survives, and the answer was already sitting in plain sight, because the SigNoz UI stayed browsable through all four outages.

I over-read an error and only caught it by running a control. My first attempt at “does a rule accept a nonexistent channel” returned 400 alert rule is not valid, which I wrote down as proof. Later a rule with a perfectly valid channel returned the same generic error, because my hand-rolled body was malformed. Re-ran it properly as a pair: identical body, only the channel name changed, one passes and the other returns 400 channels: the following channels do not exist: [...]. Same conclusion, but the first version of the evidence did not actually support it. A 400 that says “not valid” names no cause, and the temptation is to supply one.

usePolicy: true was the answer that looked right and was not. Nothing in the config would have told me; the thing that settled it was the receivers field on a live firing alert, which read default-receiver under both routing modes. Reading the routed-to receiver off a disposable fire is a cheap discriminator worth remembering.

I deliberately rebuilt the thing that caused a month of silence. ADR 0019 exists because the channel pointed at a black hole and nobody noticed for a month. The new channel is a black hole on purpose. What makes that safe is that delivery moved somewhere else and that fact is now load-bearing rather than accidental, so turning the notifier off emits a warning saying alerts have no delivery path at all. Worth being uneasy about anyway.

Removing the delivery secret from tofu fell out for free. With the channel reduced to a constant discard URL, the two ntfy_webhook_* TF variables and their TF_VAR_* injection at apply time are gone. The credentials are read at runtime from ~/.n/signoz.env instead, which is where gmailctl-drift-check.sh already reads them.

Loose ends

  • Pending confirmation that the two verification pushes actually landed on the phone. The signoz ntfy user is publish-only and 403s on read, so the log line saying ntfy returned 2xx is as far as I can verify from here.
  • Single point of failure by choice. KeepAlive restarts the agent and the queue survives a restart, but a wedged agent is now silent in a way the old webhook would not have been.
  • The blind spot from ADR 0019 is unchanged: the egress rule cannot fire if the collector is down. The unreachable synthetic covers the whole-stack case, not that one.
  • Root cause of the guest-network breaks themselves is still unknown, and there is still no crash to blame.

Graph View

Backlinks