2026-07-14 Runtime State Reconciliation Framework

What I set out to do

Started from a question about home-manager activation scripts (are they for filesystem state, not running-process state?) and walked it, Socratically, into the general question: how should runtime / in-app state that Nix can’t manage be reconciled, and when does reaching for an established tool (Terraform, Ansible) beat a hand-rolled idempotent script.

What I actually did

Worked out a full decision framework and captured it as Runtime State Reconciliation Decision Framework. The arc, roughly in order:

• Confirmed the activation-script rule: activation reconciles filesystem/unit-definition state and delegates process supervision to launchd/systemd.
• Grounded the Nix+Terraform pairing (terranix, nix.dev tutorial, Jonas Carpay’s end-to-end guide): Terraform owns external stateful APIs, Nix owns build+machine internals; they meet at a seam.
• Found the dotfiles repo already embodies the pattern: signoz.nix renders dashboards/alerts as main.tf.json and runs tofu apply against localhost from inside a home.activation block, with views/retention scripted and sqlite-seed.nix for OWUI.
• Corrected my own framing twice under push-back: “provisioning vs configuration” is a fuzzy lineage label; the real axes are shape (object/field/action) + ownership (sole/co-owned) + a provenance test.
• Nailed what state actually buys: automated provenance, not drift-fix (stateless does that) and not restore-to-prior (nobody does that for free — Terraform destroys, it doesn’t roll back). The prune pattern (live ∩ mine ∩ ¬desired) gives automatic deletion without a state file and can’t desync.
• Un-lumped Terraform and Ansible: for in-app runtime state Ansible/convergence is on-axis, Terraform only for sole-owned CRUD objects. Multi-machine/multi-arch doesn’t restore Terraform’s fleet value because each host reconciles its own isolated local instance (Topology A) — the fleet tool there is Nix, not Terraform.
• Two-gate model as the landing: gate 1 shape → CRUD-manageable; gate 2 “can I reconstruct provenance from live?” → Terraform vs stateless-prune.

What was striking

• The whole thing is a generalization of what the Media Stack IaC Declarative Config Evaluation project already decided for the media stack (“delegating shape,” one-writer-per-object, Terraform-where-resource-shaped + idempotent-curl for settings-blob stragglers, seed-don’t-reconcile). This session extended that reasoning to SigNoz and Cloudflare and made the gates explicit.
• The off-axis tell is concrete and visible in my own repo: the SigNoz dashboard resource is mostly ignore_changes + replace_triggered_by + upstream-bug links — config that tells the tool what not to do — which is the fingerprint of using Terraform slightly off its happy path (worsened by provider bug #71).
• The one genuine Terraform candidate in my world is the Cloudflare Access control plane (account/app/policy, currently hand-managed, not in repo), not the dashboards I’d been running through tofu.

Next actions

• Re-examine the config against the framework (in progress this session): whether to move SigNoz dashboards/views off tofu to stateless+prune, and the activation→supervised-unit factoring for the reconcile.
• Independent of that: fix the reconcile’s silent failure handling in syncSigNoz (bounded health poll + emit outcome instead of || true).

Related

Runtime State Reconciliation Decision Framework · Media Stack IaC Declarative Config Evaluation · Terraform · Ansible · Infrastructure as Code

Implementation — consolidated SigNoz on Terraform (later same session)

The re-examination turned into a build. Notable: the outcome went against the framework’s initial lean. The gates said SigNoz dashboards/views/alerts are fully-owned + enumerable + name-keyed → stateless+prune is sufficient and lighter. But when it came to the concrete “bring the notification channel in” ask, I chose to consolidate on Terraform instead, because:

• The native SigNoz/signoz provider has no notification_channel resource (confirmed: it models alert/dashboard/planned_maintenance/role/route_policy/service_account/user), so the channel needed some out-of-band mechanism regardless.
• I initially argued secrets made Terraform wrong for the channel, then conceded the point under push-back: the ntfy creds are managed out-of-band in ~/.n/ either way, so both a script and restapi inject them at runtime; the only real delta is secret-in-tfstate-at-rest, which is marginal on a single local box that already accepts ~/.n/ plaintext + tfstate-with-secrets on media-stack.
• With secrets neutralized, the deciding factor was consistency: restapi-for-channel-alone would leave three mechanisms (native tofu + restapi + a view script). So the coherent choice was to pull the channel and the views into tofu via the generic Mastercard/restapi provider → one tofu apply, one mechanism. Consolidating also revives the channel→alert depends_on edge (the one real pro-tofu-for-alerts argument, which had died when the native provider turned out not to model channels).

Built in signoz.nix: added Mastercard/restapi; channel as a restapi_object with the webhook URL+password as ${var.*} injected from ~/.n/signoz.env at apply (count-gated on the secret being present, so a missing file leaves it unmanaged rather than failing the apply); alerts gained depends_on on the channel; views migrated from signoz-sync-explorer-views.py (deleted) to restapi_objects via a viewToTf helper. Verified locally: nixpkgs-fmt (caught myself running nixfmt-rfc-style by mistake — this repo uses nixpkgs-fmt, reverted the whole-file reformat), parse, escaping rendered in isolation. Not committed; live apply + verification pending on atlas (the restapi round-trip unknowns: channel id_attribute=data.id/POST body, and view read-by-id vs list-search).

Cutover is recreate (delete existing channel + view, let tofu POST fresh; alerts rebind by name). The lasting lesson for Runtime State Reconciliation Decision Framework: the gates tell you what’s sufficient; a legitimate separate axis is mechanism consistency — one tool for a whole surface can beat a lighter-but-heterogeneous mix, and that can tip an otherwise stateless-sufficient case toward consolidating on the tool you’re already running.

Next actions (updated)

• Live cutover on atlas: create ~/.n/signoz.env, delete existing channel + view, tofu plan pre-check, hm switch, verify + test-fire. Fix restapi round-trip shapes if the first apply flags them.
• Commit the channel + views consolidation atomically once the live apply is confirmed.
• Still open, independent: fix syncSigNoz silent failure handling (bounded health poll + emit outcome instead of || true).

Cutover executed on atlas (complete)

Ran the consolidation live on atlas (I was already on it; SigNoz reachable locally). Committed 4a80465, created ~/.n/signoz.env (600) with the ntfy webhook URL + password read from the existing channel, and applied. Outcome: channel + view + alerts + dashboards all under one tofu state, verified idempotent (channel/view/dashboards converge; paging config confirmed intact by GET). Real learnings, several against plan:

• SigNoz enforces unique channel names → the restapi POST 400’d on the existing channel (safe: no duplicate, old channel kept paging). Also proved the POST body/path were correct (it reached the uniqueness check, not a parse error).
• SigNoz refuses to delete a channel that alerts reference (DELETE → 400; a probe channel with no alerts deleted fine at 204). So the recreate path is blocked for an in-use channel — you MUST import, not delete-recreate. Import was gap-free: channel never went down, adopted + updated in-place via PUT.
• restapi id_attribute differs per resource: channel POST returns {data:{id}} (data.id), view POST returns {data:"<id>"} (id IS data). Found via safe probes; fixed with a per-resource override on the view. This was the one code bug in the first apply (view created but orphaned from state until fixed).
• restapi import ID is the full path /api/v1/channels/<id>, not the bare id.
• A foreign explorer view existed (“CC Human Decided Tool Permissions”) that isn’t in my config. So the views surface is NOT fully-owned — which means a blind stateless prune would have deleted it, and state-tracking (managing only what it created) is vindicated over prune here. A concrete correction to my gate-2 assumption that I “own all of SigNoz.”
• Alerts show pre-existing perpetual computed-field churn (state/update_at/notification_settings → known-after-apply); benign re-PUT, not introduced by this change. ignore_changes follow-up candidate.

Next actions (updated)

• Live cutover on atlas — done, verified, committed 4a80465.
• Optional: ignore_changes on the signoz_alert computed fields to stop the per-switch churn (pre-existing, benign).
• Still open, independent: syncSigNoz silent failure handling (bounded health poll + emit instead of || true) — the failed first apply was invisible except in captured output, exactly the gap the reference note flags.