Runtime State Reconciliation Decision Framework

Decision framework for reconciling runtime / in-app state that Nix cannot manage. Nix renders files and supervises processes; it has no model for state that lives behind a running service’s API or inside its private store. This is the config-management gap, and the question is which tool fills it. Generalizes the media-stack conclusion in Media Stack IaC Declarative Config Evaluation to any service (SigNoz, Cloudflare, dotfiles).

The core reframe

“Provisioning vs configuration” is a fuzzy lineage label, not a decision rule. A dashboard is configuration, yet Terraform manages it fine. What actually picks the tool is the shape of the target, along two axes, plus a provenance test.

Gate 1 — shape

• Object (discrete thing with identity + a create/read/update/destroy lifecycle: a dashboard, a saved view, a DNS record) → CRUD-manageable.
• Field (a value inside a larger artifact: a retention setting, one key in a config blob) → converge in place; no lifecycle, no destroy.
• Action (a verb: reload, send-test, trigger) → not state at all; a script/handler.

Only objects are Terraform-eligible. Fields and actions are not, regardless of how nice a provider looks.

Gate 2 — can I reconstruct provenance from the live system?

Passing gate 1 means “manage as a CRUD resource,” not “use Terraform.” Both Terraform and stateless convergence do create/destroy. State earns its keep only when you cannot cheaply answer “what did I create, and is it still mine?” from the live system:

• Can I enumerate them all from the API? Are they naturally-keyed (stable name, not an opaque handle)? Do I own the whole target, or only a subset among foreign resources I must not touch? Is there a dependency graph / high blast radius / shared writers?
• Mostly no → stateless + prune (delete_set = live ∩ mine ∩ ¬desired). Automatic pruning, no state file, cannot desync.
• Yes → Terraform. State is the authoritative provenance + dependency record you can’t rebuild from live.

Gate 2½ — mechanism consistency (sufficiency isn’t the only axis)

The gates tell you what’s sufficient, not what’s best. A second, independent axis: running one mechanism across a whole surface can beat a lighter-but-heterogeneous mix. When several related objects live on one system and some are stateless-sufficient but others force a particular tool (an object with no stateless option, or one carrying a dependency edge), consolidating everything onto that tool can be the better call even where the gates said stateless would do — because three mechanisms for one surface (native provider + generic-REST + a hand script) is its own cost: more code paths, more failure modes, more to reason about.

Tie-breaker once sufficiency is settled: fewest mechanisms per surface, unless a specific object is actively harmed by the shared tool (a secret it would force into state, a provider that fights it). Hold that “actively harmed” bar honestly — I once raised secrets-in-tfstate as a blocker for a notification channel, then conceded it was marginal because the credential was already managed out-of-band (~/.n/, injected via TF_VAR_*), which flipped the decision back toward consolidation. The lesson: don’t let a reflexive objection veto consistency until you’ve checked the objection actually bites at your scale.

Tool → substrate

Target	Tool
Object, sole-owned, enumerable, naturally-keyed, low blast	stateless CRUD + prune (script or Ansible)
Object, foreign-mixed / opaque id / interdependent / high blast / external SaaS	Terraform
Object with no native provider resource, want it in the tofu surface	generic REST (Mastercard/restapi) — accepts round-trip drift
Field the app treats as read-only input	wholesale render (Nix owns the whole file)
Field in a co-owned file the app rewrites	augeas / format-aware idempotent patch
Field in an embedded DB (sqlite blob key)	idempotent upsert + readiness + schema-contract guard (Nix seed)
Action / verb	script / handler, not a resource

What state actually buys (and doesn’t)

Three operations people conflate:

• Enforce (drifted actual → desired): both Terraform and stateless do this every run. State not required.
• Destroy (remove declaration → delete the thing): Terraform via state, stateless via prune. Equivalent for enumerable-owned resources.
• Restore-to-prior (remove declaration → put back the pre-management value): neither tool does this. Terraform destroys, it does not roll back to what existed before. For created objects destroy ≈ restore-to-prior only because prior was absent. For a pre-existing setting you’d have to snapshot the old value yourself.

So state’s single irreplaceable benefit is automated provenance when you can’t reconstruct it from live. It is not drift-correction (stateless does that) and not undo-to-prior (nobody does that for free). And state has a cost stateless avoids: it can desync from reality (import/state rm surgery), and it proliferates one file per host.

Config convergence techniques (for fields)

• Assert the invariant, never script the transition. Write “ensure == B,” correct from any starting state, not “if A then B.” Reconciliation = repeatedly asserting the invariant. See idempotence vs convergence.
• Own the whole artifact when you can; patch only when a co-writer forces it. Wholesale render (Nix) eliminates merge logic.
• Read back and verify after writing; against a co-writing app the write can be clobbered.
• Enforce-once vs enforce-always: seed + hash-guard if the store never drifts; a supervised timer/agent re-asserting if it can (UI edits, upgrades).

Field-level ownership (the co-owned object)

When an object has identity but the app also mutates it, ownership drops from object-level to field-level. Bucket fields into declared / app-managed / contended; give every contended field exactly one authoritative writer (config-wins, app-wins, partition, or detect-only). Two authoritative writers on one field = the fight/flap loop. Kubernetes Server-Side Apply (managedFields) and Terraform ignore_changes both only express a partition that must already exist. Discover which bucket a field is in by apply → let run → read back → diff.

Activation vs supervised unit (where the reconcile runs)

Home-manager activation is for filesystem reconcile (render the desired-state artifact). The live reconcile (apply/upsert against a running service) is runtime reconcile and belongs in a supervised unit (launchd/systemd), not an activation block, because it needs retries, decoupling from switch timing, and observable failure. Clean factoring: activation writes the artifact; a supervised unit consumes it and reconciles the live system. On Darwin the unit is a launchd agent (coarser oneshot-retry; watch macOS 26 domain fragility).

Reconcile failure handling (whichever mechanism)

The reconcile step (apply/upsert against the live service) must be observable and resilient, or a broken reconcile is invisible until you notice stale state:

• Bounded readiness poll, not a one-shot skip. curl health || skip drops the whole reconcile on a slow start; poll N times, then give up loudly.
• Never swallow failures (|| true, silent else). A failed apply must surface, not vanish.
• Emit the outcome to an always-up sink so success/failure is queryable.
• The circular-emit caveat: you cannot report “the observability backend is down” to that backend. Split the cases — reachable-but-apply-failed emits to the backend; backend-unreachable goes to an independent sink (ntfy / a durable local log). Emitting a SigNoz-reconcile failure to SigNoz only works when SigNoz is up.

This is orthogonal to tool choice — it applies equally to a tofu apply, an idempotent script, or an Ansible run — and it’s the strongest concrete argument for the activation → supervised-unit split above: a unit’s last-exit-status survives the backend being down; an activation block’s || true does not.

Applied to my systems

• SigNoz dashboards / views / alerts / channel: consolidated onto Terraform in one tofu apply (native signoz provider for dashboards/alerts; Mastercard/restapi for the notification channel and saved views, which the native provider doesn’t model). By the gates alone these are stateless-sufficient (I own all of SigNoz, enumerable, name-keyed) — the deciding factor was Gate 2½ mechanism consistency: the channel had no native resource and no clean stateless-only edge once its secret was conceded out-of-band, so one tool for the whole surface beat native-tofu + restapi + a view script. The channel/views carry the accepted restapi round-trip cost, and the off-axis tell persists on both dashboards and alerts (ignore_changes + replace_triggered_by, forced by provider re-serialization / normalization). Secrets (ntfy webhook) stay in ~/.n/signoz.env → TF_VAR_*, never the store. Reconcile now runs a bounded health poll + pipefail apply that surfaces failure loudly (still from home.activation, best-effort so a hiccup doesn’t abort the whole switch; the supervised-unit split remains the ideal, not yet done).
• Open WebUI settings: field in a co-owned sqlite blob → idempotent upsert with requireColumns guard (the existing sqlite-seed.nix, done right).
• Cloudflare Access (obsidian-mcp.achhina.com): account/app/policy — discrete, external, opaque-id, interdependent, currently hand-managed in the control plane and not in repo → the genuine Terraform candidate, unlike the dashboards.

See also Infrastructure as Code, Ansible, Terraform, and the media-stack “delegating shape” decision in Media Stack IaC Declarative Config Evaluation.

Field notes (from the SigNoz cutover)

Refinements that only showed up when the framework met a live system:

• Verify “do I own the whole target?”, don’t assume it. Gate 2 asks whether you own the surface or share it with foreign objects. I assumed I owned all SigNoz explorer views; the cutover found a foreign view I didn’t manage. Consequence: a blind stateless prune would have deleted it. So when foreign objects can exist, state-tracking (manage only what you created) beats prune — a concrete case where the state file earns its keep after all. Enumerate the live surface before choosing prune.
• Migration ≠ greenfield: an in-use resource may be undeletable, forcing import. Recreate (delete → create) assumes you can delete. A notification channel that alerts reference was refused deletion by the server (a channel with no references deleted fine). So the only cutover path was import (adopt the existing object into state, reconcile in-place) — gap-free, and the correct move whenever the live object is protected or paging-critical. Recreate is fine for greenfield or unreferenced objects; import for anything load-bearing.
• Generic-REST id location is per-response-shape, not global. With Mastercard/restapi, id_attribute reads the create response, and two endpoints on the same API returned the id differently ({data:{id}} vs {data:"<id>"}). Probe each POST response and set id_attribute per-resource. A wrong path creates the object but orphans it from state (“object ID is not set, but may have been created”).
• Import IDs for restapi are the full path (/api/v1/channels/<id>), not the bare id.
• Per-refresh in-place churn is provider normalization, not a real diff — mask it, don’t chase it. Every switch re-planned all four alerts because the SigNoz server backfills empty reduceTo/temporality/source into metric-signal conditions and adds a computed renotify.alert_states block I never author; each refresh then diverged from config and cascaded every computed field to known-after-apply. Matching the server’s normalization at the source is signal-specific and brittle. The robust fix is the same one dashboards already used: ignore_changes on the drifting fields to swallow the noise, plus replace_triggered_by on a content-hash terraform_data so a genuine edit still recreates the resource. One consistent pattern across the whole alert+dashboard surface beats per-resource hand-tuning.
• Validate a lifecycle/state change by planning a scratch copy of the live state — plan only, never apply. Copy terraform.tfstate + .terraform into a scratch dir, drop in the candidate main.tf.json, and plan. plan refreshes read-only and shows the exact actions; apply in the scratch dir would mutate the real service (the providers still point at live localhost). This surfaced the exact “5 add / 4 destroy” one-time recreate before it touched anything. Build the real Nix-emitted tf.json (nix build …#homeConfigurations.<sys>.config.home.file."…/main.tf.json".source) so you plan against the actual hash, not a hand-rolled approximation.
• Introducing a replace_triggered_by reference forces a one-time recreate; it’s safe iff the resource holds no live state you’d lose. First apply that adds the reference recreates the referenced resources once (steady state is clean thereafter). Checked all four alerts were state = "inactive" first, so nothing firing was lost. Pre-seeding the terraform_data target in a prior (targeted) apply avoids even the one-time recreate — proven — but requires matching the exact hash Nix will emit, so it’s only worth it when a recreate would actually lose firing state.