Achhina's Digital Garden

2026-07-15 SigNoz Reconcile Robustness

3 min read

2026-07-15 SigNoz Reconcile Robustness

Follow-up to 2026-07-14 Runtime State Reconciliation Framework: cleared the two items that cutover left open in signoz.nix. One commit, 7c2504e.

What I set out to do

Fix both open follow-ups from the SigNoz Terraform consolidation:

  1. Per-switch alert churn (ignore_changes candidate).
  2. syncSigNoz silent failure handling (bounded health poll + emit outcome instead of || true).

What I actually did

Alert churn. Ran a tofu plan against live state to get ground truth first (per the “verify the question” habit) instead of trusting the summary’s field list. Two drivers, not one: clickhouse alone had a genuine condition JSON re-serialization diff (server backfills empty reduceTo/temporality/source into the metric aggregation, drops the empty groupBy), and all four drifted on notification_settings because the provider added a computed renotify.alert_states block I never author — which cascaded every computed field (id, state, create_at, …) to known-after-apply. Considered matching the server normalization at the source but it’s signal-specific and brittle, so took the dashboard pattern instead: ignore_changes = [condition, notification_settings] to swallow the noise + replace_triggered_by on a new terraform_data.alerts_content (keyed on alertsHash) so a genuine edit still recreates the alert. One consistent mechanism across the whole alert+dashboard surface.

Failure handling. Rewrote the reconcile tail: skip cleanly when SigNoz isn’t running (some hosts enable the module without the stack), else poll /api/v1/health up to 60s (a single-shot probe silently skipped the reconcile on every switch that restarted the stack), then run apply under set -o pipefail in a subshell so the pipeline carries tofu’s real exit code even though grep trims the output — a failed apply now takes a loud WARNING branch instead of || true swallowing it. Kept it best-effort (warn, don’t abort the whole switch) and added a dry-run branch so $DRY_RUN_CMD activations don’t emit spurious warnings.

What was striking

  • Validated before touching live by planning a scratch copy of the live state. Copied terraform.tfstate + .terraform into scratch, dropped in the real Nix-built main.tf.json (nix build …homeConfigurations.aarch64-darwin…main.tf.json.source — the actual emitted hash, not a jq approximation), and planed. Crucial subtlety: apply in that scratch dir would hit the live SigNoz because the providers still point at localhost, so plan-only. It showed the exact one-time 5 add / 4 destroy recreate before anything ran.
  • The replace_triggered_by reference forces a one-time recreate. Safe here because I checked all four alerts were state = "inactive" first — nothing firing to lose. Proved pre-seeding the terraform_data target in a prior targeted apply avoids even that, but it needs the exact Nix hash, so only worth it when a recreate would lose firing state.
  • Post-switch live plan = “No changes.” The churn that re-planned all four alerts every switch is gone, and the reconcile ran through the new poll+pipefail path successfully in the same switch (the Apply complete! line came through the subshell branch, confirming the failure path is live).

Next actions

  • Alert churn fixed — live plan converges to “No changes”.
  • syncSigNoz failure handling — bounded poll + pipefail outcome, ran clean.
  • Still deferred (design ideal, not urgent): the activation → supervised-unit split for the reconcile, so last-exit-status survives independent of the backend. Reconcile is still a best-effort home.activation block.
  • Still the one genuine greenfield Terraform candidate: Cloudflare Access control plane, not in repo.

2026-07-14 Runtime State Reconciliation Framework · Runtime State Reconciliation Decision Framework · Media Stack IaC Declarative Config Evaluation · Terraform · Nix

Graph View