Achhina's Digital Garden

Media Stack IaC Declarative Config Evaluation

30 min read

Objective

Decide a target architecture for the media-stack’s two configuration layers: whether to move from the bespoke mstack (Jinja + Python CLI) and setup-*.sh shell scripts toward declarative Infrastructure as Code, and if so, which tool owns which layer.

Context

The stack configures services in two layers:

  • Layer 1 (static): mstack (~1,311 LOC Python) renders 8 .j2 templates to config/*/config.{xml,yaml,json,conf}, generates API keys into .secrets/generated/*.env, and detects host facts (PUID/PGID, docker subnet).
  • Layer 2 (runtime): six idempotent setup-*.sh scripts (~1,763 LOC) configure services over REST after they boot, run by the orchestrator sidecar.

GitHub issue #32 originally asked whether Terraform could replace layer 2. The design session on 2026-06-27 expanded the question to layer 1 (NixOS / home-manager) and to whole-stack orchestration. See Homelab Services Architecture for the running topology.

Approach

Two evaluation RFCs on branch feature/iac-declarative-config:

  • docs/rfcs/terraform-eval.md (layer 2): partial migration is viable for Sonarr / Radarr / Prowlarr via the mature devopsarr providers. Bazarr and qBittorrent have no provider; Seerr’s provider is single-maintainer/0.x. A Sonarr PoC proved coverage, idempotency, and drift detection in isolation. Recommendation: keep the PoC, observe ~1 week, then decide.
  • docs/rfcs/nix-orchestration-eval.md (layer 1 + orchestration): the secrets reframing (ephemeral, machine-minted, network-protected) pushes secret generation out of layer 1, collapsing its main impurity. Prior-art survey shows deployment + static config is solved by Nix (arion, oci-containers, compose2nix), but app-config universally reduces to API reconciliation, and the projects that tried to own it in a Nix module (declarative-arr, nixarr’s sync) are abandoned or unstable. Recommendation: the delegating shape, not the absorbing one.

Decisions

2026-06-27 — Prefer delegating IaC shape over a single absorbing module

Status: decided and implemented (2026-06-27, commit 52b3af9 - full cutover; mstack + Jinja deleted)

Decision: Target architecture is Nix/home-manager (or a slimmed mstack) for deploy + static-file generation, Recyclarr keeps owning quality profiles / custom formats, and a thin scoped reconciler (Terraform where resource-shaped providers exist, idempotent activation curl for the settings-blob stragglers) owns runtime app-config. Secrets move out of layer 1.

Alternatives considered: (a) one Nix module owns generation + orchestration + app-config (the nixarr/declarr path) — rejected: that layer is where those projects die. (b) Status quo mstack + Jinja + shell — viable fallback, keeps host-agnostic portability. (c) CUE/Dhall for typed generation — orthogonal, doesn’t address orchestration; CUE noted as the lighter non-Nix option.

Rationale: matches the Infrastructure as Code layering principle (one writer per object; don’t bake mutable app state into a Nix closure). The stack is already ahead of the Nix media-module ecosystem on the hardest sub-problem (custom formats) by using Recyclarr. The mutable-file ownership conflict is already solved locally by the mkComposeDeploy activation pattern (see Homelab Services Architecture).

Next Actions

  • Run the Terraform Sonarr PoC through ~1 week of normal stack usage; watch for provider flapping (the only open rubric row in terraform-eval.md).
  • Decide migrate-partially vs keep on the layer-2 rubric.
  • Build Radarr + Prowlarr TF modules (done, commit 2e13b68; both tofu validate clean).
  • Decide layer 1: adopted home-manager generation (commit 52b3af9); mstack + Jinja deleted, host-coupling trade-off accepted (personal infra).
  • Run home-manager switch against the live stack - the seam is verified in isolation but has never run live (the one remaining unknown).
  • Evaluate the Josh-Archer Seerr provider (done: adopted as-is, no fork needed; terraform/seerr covers server connections + metadata; non-declarative parts stay scripted).
  • Open a fresh PR once direction is settled (PR #40, 2026-06-29; 16 commits squashed to 4 logical units, branch force-pushed over the stale pre-rebase snapshot).

Resources

Notes

The conceptual unlock was the secrets reframing: treating API keys as ephemeral, machine-minted, network-protected tokens (not durable artifacts) lets secret generation leave the static layer entirely, which is what makes any declarative generator viable for layer 1. A later correction (see the 2026-06-27 hybrid entry below): the *arr config.xml is not the irreducible wrinkle it was thought to be. pkgs.formats.xml emits genuine arbitrary-schema XML, so native typed generation reaches 7 of 8 configs. Only nginx.conf (config syntax, no data format) stays a templater.

Progress Log

2026-06-30 - PR #40 review round 12: 2 new real bugs (dead code, sh-c injection) + warnings

User opted to keep iterating (not the churny terraform-move/setup-seerr-rewrite). Two genuinely-new real fixes: (#5) deleted dead configure_qbittorrent_client (96 lines, callers gone); (#6) single-quote injection in the PUID/PGID sh -c append (a ' in deployDir → UID-0 containers) fixed via env-var+positional-arg. Plus runtime warnings: nginx non-loopback bind (basic auth gone) and Radarr colon-rename before apply; RFC status → accepted. Pushed back: prowlarr (8th, user declined the move), subnet pinned-invariant (intentional), radarr on_health_restored (provider ~>1.7 lacks the field - validate rejects it), version pins (no blind major bump). Deferred setup-seerr (untestable). Folded into 5 commits (delta = 4 files).

2026-06-30 - PR #40 review round 11: nginx-force bug + apk image; loop is recycling

Caught a real bug from my round-10 fix: the any_minted re-seed used the global force, which also clobbered the secret-free nginx.conf (wiping user custom locations). Split into seed_force (secret configs only) vs force (nginx). Also added Dockerfile.orchestrator (OpenTofu + bash/curl/jq baked in) so the orchestrator doesn’t apk-fetch tools at every start (air-gap/CDN risk, raised 2x); enhanced the .env-seed message re bazarr creds; added a prominent basic-auth migration section to security.md. Declined parallel-tofu (one-shot orchestrator). Deferred setup-seerr GET/PUT/POST + exit codes (untestable, with add_indexer). Pushed back prowlarr 7th time (offered the terraform/reference/ move if wanted). Meta: the loop is now recycling subjective/untestable items (prowlarr 7x, setup-seerr 3x); genuinely-new real bugs are nearly exhausted. Surfacing to the user whether to do the churny definitive fixes, call the loop done, or merge.

2026-06-30 - PR #40 review round 10: 4 fixed (incl. key-divergence self-heal), 3 pushed back

Finally automated the key/config divergence self-heal (Bug 1, raised in rounds 1/3/10): if any key is (re)minted - first switch or a wiped stateDir with config/ surviving - the activation now force-re-seeds the configs so config.xml and .env can’t disagree (silent 401). Also: clean up orphaned *.seed.* temp files from an interrupted switch (Bug 4); tr -d '\r\n' for CRLF .env safety (Bug 5 - confirmed only the trailing \r is reachable, not embedded \n); fixed 2 more security.md spots claiming X-Auth-User defaults to system username (missed in round 7). Pushed back: Bug 2 (-upgrade=false doesn’t block installing locked versions - the bot’s OWN round-1 review said so), Bug 3 (prowlarr required-var aborts apply - 6th time), minor (orchestrate.sh is already #!/bin/sh + apk-adds bash). Folded into 5 commits (delta = 2 files). Standing deferral: add_indexer.

2026-06-30 - PR #40 review round 9: 4 fixed (incl. a real analytics bug), 1 pushed back

Real bug: (#1) the arrXml analytics ? true param only emitted AnalyticsEnabled=False when truthy, and Prowlarr passed analytics = false, so Prowlarr’s analytics were left ENABLED (inverted logic). Dropped the param; now emitted unconditionally for all three *arr. Also (#3) read_env_cred strips single quotes too; (#4) removed the dead opensubtitles-v1 provider block from bazarr config; (#5) AGENTS.md anime Dual Audio score +100→+500 to match the generator (flagged in case 100 was the intent). Pushed back again on #2 (terraform/prowlarr accidental-apply): the required-no-default var already aborts a bare tofu apply - the machine guard exists. Folded into 5 commits (delta = 3 files); converging (5 findings this round vs 8 last). Standing deferral: add_indexer.

2026-06-30 - PR #40 review round 8: 1 real bug + docs; refuted the bot’s top finding

The bot’s #1 (“custom_format_groups is v9 syntax, silently ignored by the :8 image”) was factually wrong - verified via the Recyclarr changelog that custom_format_groups shipped in v8.0.0 (2026-02-19); v9 is still unreleased and only adds select_all. So the :8 image supports it; no change (verifying saved a risky bump to a nonexistent :9 tag). Real fix: #3 - the round-7 fail-closed subnet guard crashed switch --dry-run on a fresh host (compose file not yet installed); now wrapped in [ -f docker-compose.yml ]. Plus: secretFiles missing-path now warns (declined the bot’s lib.types.path - it would store-copy the secret), radarr colon-format migration note, TF_LOG troubleshooting warning, authUser example. Pushed back on #2 (not an ordering bug - documented env-init flow works), #6 (harmless dry-run mint), minor 127.0.0.1. Deferred #4 add_indexer (unchanged). Folded into 5 commits; all gates green.

2026-06-30 - PR #40 review round 7: 7 fixed, 2 pushed back, 1 still deferred

Fixed: (1) read_env_cred now strips surrounding " from .env values before injecting into bazarr; (2) prowlarr update jq appends syncAnimeStandardFormatSearch if an old connection lacks it; (3) added qbit_ok gate so Sonarr/Radarr tofu is skipped when qBittorrent setup fails (round-6 fix only gated prowlarr/bazarr/seerr, not the *arr tofu itself); (5) security.md authUser default corrected (4 spots: was $(id -un)/NGINX_AUTH_USER, now authUser option default admin); (7) comment on sonarr int colon_replacement_format; (8) .env.example NGINX_BASIC_AUTH_ENABLED tombstone; stale terraform/README “Sonarr only”. Pushed back: prowlarr/seerr already abort a bare tofu apply (required vars, no defaults) - the machine guard already exists; per-switch stale-cred NOTE would be noise (documented in README). Still deferred: setup-seerr GET/PUT/POST (untestable rewrite). Folded into 5 commits (delta = exactly the 7 files); all gates green.

2026-06-30 - PR #40 review round 6: 7 fixed, 2 deferred, 2 pushed back

Real bug found: (1) the *arr naming format had a malformed audio token {[Mediainfo AudioCodec}{ Mediainfo AudioChannels]} (brackets spanning two {} tokens) - fixed to [{Mediainfo AudioCodec} {Mediainfo AudioChannels}]. Also: (2) qBittorrent failure now sets arr_ok=0; (3) .key/.env writes made atomic (tmp+mv) like the seed; (6) TF_VAR_* exported only around apply not init (TF_LOG leak window); (7) subnet guard fails closed when the compose subnet can’t be parsed; (8) forceConfig documents that it wipes Seerr Plex OAuth; dead $stateDir/host.env write removed. Deferred F4/F5 (setup-seerr GET/PUT/POST + exit codes - ~150-line profile-resolution blocks, untestable without a live stack, tracked with the add_indexer gap). Pushed back: ctx.maxRatio or 0 is reachable (flake sample ctx omits maxRatio); declined moving the validate-only prowlarr module (WARNING comment suffices). Folded into 5 commits; all gates green.

2026-06-30 - PR #40 review round 5: 7 fixes (user said keep iterating to clean)

(1) non-atomic seed - install+sed wrote $dest directly, so a failed sed left a placeholder config the next switch skips; now inject into $dest.seed.$ + mv. (2) tfstate world-readable - orchestrate.sh now umask 077 before tofu (state holds *arr keys plaintext). (3) broken Sonarr cascade - arr_ok flag skips Prowlarr/Bazarr/Seerr setup when sonarr/radarr tofu fails (manga stack still runs). (4) docs discrepancy (real) - architecture.md diagram/text claimed prowlarr/seerr were tofu-applied; corrected to sonarr/radarr-only + setup-scripts own prowlarr/seerr (the modules already had round-3 WARNING comments). (5) basic-auth migration note added to CHANGELOG. (6) dry-run printed minted key - key writes now pass value via __mskey env var (echo shows argv not env; also subsumes the round-4 override-quoting fix). (7) render-templates over-exported - made substitute/render private. Folded into 5 commits (delta = exactly the 5 touched files). All gates green.

2026-06-30 - PR #40 review round 4: 4 hardening fixes

The reviewer keeps surfacing deeper edges each pass (all legit, progressively more minor). Round 4: (1) real security bug - seedSnippet used install -m644, so config/*/config.xml with injected API keys (+ bazarr’s OpenSubtitles creds) were world-readable; changed to -m600. (2) mintSnippet’s sh -c "... \"$rawkey\" ..." truncated an override key containing "; switched both .key/.env writes to positional-arg form (sh -c 'printf "%s" "$1" > "$2"' sh "$rawkey" "$path"). (3) stale TF state lock after a SIGKILL’d apply - gitignored *.tfstate.lock.info + documented force-unlock recovery (declined auto-unlock: disabling a real safety check isn’t worth it for a homelab). (4) unconstrained authUser injected into nginx - type = strMatching "[A-Za-z0-9._-]+" + quoted the 4 proxy_set_header sites. Folded into the 5 commits (delta = exactly the 4 touched files). All gates green.

2026-06-30 - PR #40 review round 3: 3 real fixes (all CI green)

Post-fold review (5-commit branch) raised 3 findings, all legitimate this time (no misreads): (A) the inject-secrets flake check applied only sedInjectArgs and grepped @[A-Z]+_API_KEY@, so it never exercised my new env-cred injection - a bug there would pass CI. Fixed: apply envCredSedArgs (vars unset → empty = first-boot state) + broaden grep to @[A-Z_]+@. (B) mintSnippet assigned override keys ($<SVC>_API_KEY / secretFiles) verbatim into the sed injection - a |/&/\ would corrupt it. Fixed by keeping a raw key for .env/.key and a separately sed-escaped key_<svc> for injection (auto-minted hex/base64 were already safe). (C) added top-of-file WARNING comments to the validate-only terraform/{prowlarr,seerr}/main.tf (manual tofu apply would duplicate setup-script-owned resources). Folded into the 5 commits (re-pushed, tree verified = prior + exactly the 4 fix files). All 4 CI checks (Nix flake check, tofu validate, pre-commit, claude-review) had passed on the prior push.

2026-06-29 - PR #40 review round 2: 3 real fixes, 4 pushed back (commit d026f56)

A fresh claude-review on the lint/cleanup push raised 7 findings (the bot re-reviews the whole PR each run). Triaged each against the actual code:

  • Fixed (3): (1) setup-qbittorrent.sh hit localhost:8080 but orchestrate.sh now calls it from the orchestrator container - real breakage (categories never created); pointed it at qbittorrent:8080. (2) Real regression - AniDB/OpenSubtitles creds never reached the generated bazarr.config.yaml (generate.nix read ctx.x or "" but nothing threaded .env in). Added them as envCreds in secrets.nix: placeholders in the store, injected at activation from the deployed .env (sed-escaped), so the real cred never enters /nix/store. Verified end-to-end incl. |/& escaping. (6) Added an activation-time guard that subnet matches the vendored docker-compose.yml ipam subnet.
  • Pushed back (4): (3) key-divergence escalation was wrong - seedSnippet re-installs the placeholder file before the sed, so forceConfig=true recovers cleanly (core issue already documented). (4) unused terraform/{prowlarr,seerr} is intentional + documented (provider gaps); the bot’s “would fight setup-seerr.sh” is why they aren’t wired in. (7) apk add already fails loudly (if ! apk add; then exit 1), not silently. (5) add_indexer create-only is a real but pre-existing gap, already tracked as Persist Prowlarr Indexer Priorities in setup-prowlarr.sh; harder (dynamic Cardigann schemas) so it stays a focused follow-up.

Posted the full triage as a PR comment. All 21 hooks + flake check green.

Then re-folded the branch from 7 commits back to 5 logical units (force-pushed, tree verified byte-identical via git diff --quiet): the review fixes went into the feature commits (qBittorrent host → feat(terraform), bazarr envCreds + subnet guard → feat(nix)), the stale-doc fixes into docs:, the Python-era cruft (.copier-answers, .gitignore block) into refactor!: - and the CI/lint gating kept as its own ci: commit (net-new tooling that doesn’t honestly belong inside “remove Python” or a feature). Whole-file bucketing after git reset --mixed origin/main; needed --no-verify since the prek hook can’t see the devShell tools mid-rebucket.

2026-06-29 - CI runs pre-commit + Nix/Terraform lint gates added

Hardened the checks now that the repo is Nix/OpenTofu. The gap: prek hooks existed locally but CI never ran them (only nix flake check + tofu validate gated PRs), and there were no formatter/linter gates for the new Nix/Terraform sources.

  • New CI pre-commit job runs prek run --all-files inside nix develop, so the existing language-agnostic hooks and the new toolchain hooks share one job with tools on PATH.
  • Added Nix gates as local pre-commit hooks: nixfmt (adopted the RFC-official formatter; one-time ~430-line reformat of all 5 hand-tuned files, agreed on by the devShell-pinned 1.2.0 and my 1.3.1), deadnix, statix; plus tofu fmt for Terraform. Backed by the flake devShell (added nixfmt/deadnix/statix; formatter = pkgs.nixfmt).
  • statix flagged ~6 real nits (useless parens, assign-vs-inherit, repeated dotted keys). Fixed at source - nesting the repeated home.* and lib.* keys into single blocks cleared the repeated-key lint, so no statix.toml global-disable needed.
  • DRY split kept: pre-commit owns style/lint; the existing flake-check / validate jobs own correctness (no double-run). just prek and a new just fmt now wrap nix develop so local == CI.

Then a second pass on “what else / what tighter”: added gitleaks (secret scan + .gitleaks.toml allowlisting the 2 fake flake.nix sample keys), shfmt (-i 2 -ci -bn, reformatted 6 of 13 scripts), hadolint (.hadolint.yaml ignoring DL3007 :latest + DL3018; merged Dockerfile.bazarr’s two RUNs to clear DL3059). Tightened shellcheck with -x (follows sourced common.sh/constants.sh, clean) and added check-executables-have-shebangs + mixed-line-ending. Rejected on evidence: shellcheck --enable=all (818 findings, ~all stylistic SC2250/SC2292), and check-shebang-scripts-are-executable (would wrongly demand the sourced libs be executable).

Verified: all 21 hooks green through the devShell; nix flake check green. Folded into PR #40 as a 5th logical commit (696bd69, ci: run pre-commit in CI and add nix/terraform/shell/docker lint gates), pushed onto the existing 4. Also fixed the docs that drifted: CONTRIBUTING.md listed prettier/markdownlint hooks that were never configured - replaced with the real hook set; docs/development.md + the devShell tool list updated for the new toolchain + just fmt.

Then a dead docs/code audit (6th commit 3363dee). The migration deleted cleanly (no orphaned code, no broken md links, deadnix/statix already clean), but found stale refs: AGENTS.md linked docs/SECURITY.md (uppercase) where the tracked file is lowercase - a real broken link on case-sensitive FS, masked by macOS; nix/README still said the module “reconciles” layer-2 (removed) and called the orchestrator a hashicorp/terraform container (it’s OpenTofu); terraform apply→tofu apply drift in 3 docs; dead .yamllint ignores. Removed dead cruft: .copier-answers.yml (python-template provenance, local abs path, all Python gone) and the .gitignore Python/uv/sphinx block (incl. the !nix/lib/ exemption that only existed to rescue nix/lib from the dead Python lib/ rule).

2026-06-29 - PR #40 review cycle: fixes + dropped nginx basic auth

Two automated review rounds on PR #40; addressed all findings, then re-squashed to 4 commits (verified byte-identical tree each time).

Round 1 caught a real deploy-blocker: the orchestrator ran hashicorp/terraform:1.9 but the lock files pin registry.opentofu.org providers (the repo standardized on OpenTofu) - so Sonarr/Radarr layer-2 would silently never apply. Switched the orchestrator image to ghcr.io/opentofu/opentofu:1.9 + terraform→tofu. Also: wired setup-qbittorrent.sh (categories were never created - pre-existing, not a regression), restored the qBittorrent ratio knob as services.mediaStack.maxRatio (lost in the Jinja→Nix move), fixed a VideeBitDepth typo, and deleted the orphaned setup-sonarr/radarr scripts.

Round 2’s notable decision: removed nginx basic auth entirely. Tracing the threat model showed it was non-load-bearing - the actual auth is the X-Auth-User External-Auth header injection (kept, driven by authUser) plus localhost binding; basic auth was optional, off-by-default defense-in-depth that had generated review churn twice. Dropped gen-htpasswd.sh, the basicAuth option, the two on/off nginx.conf variants (now one rendered config), the dead .htpasswd mount, and all docs. The key distinction that made this safe: authUser feeds the External-Auth identity header and is independent of the basic-auth credential. Key-rotation 401 risk documented in nix/README rather than adding per-switch reconciliation (keeps seed-don’t-reconcile). Also fixed the pre-existing prowlarr app-connection idempotency gap: setup-prowlarr.sh’s Sonarr/Radarr blocks now GET/PUT/POST (rebuild desired config from the existing resource, keep its id, update managed fields) instead of create-if-absent, so a rotated *arr key reconciles into Prowlarr. All folded back to 4 commits.

2026-06-29 - Single-command switch: autoStart + vendored deploy dir (commits 2f38b23, fd61483)

Closed the deploy layer (compose→Nix), the last unconverted piece. Two steps, prompted by the question “why can’t activation bring up the stack like signoz, then apply the rest?”

First (2f38b23): corrected an overstatement I’d made earlier - that the host seam fundamentally can’t start the stack. It can. Added autoStart: after layer-1 the activation runs an idempotent docker compose up -d, guarded on docker info so a stopped Docker Desktop skips cleanly instead of failing the switch. Crucially this only starts the stack; layer-2 stays with the orchestrator (the real reasons it can’t move to the host hold: unpublished *arr ports, and blocking switch on full-stack health is fragile).

Then the better answer emerged from “why must I pass projectDir?”: because the module’s own path is a read-only /nix/store copy, the wrong dir to run compose from - but the deployment location is a real fact the module can own, exactly like the signoz module’s vendored-compose pattern. So (fd61483) I dropped the four externally-supplied path options (configRoot/composeSecretsDir/envFile/projectDir) for one deployDir (default ~/.local/share/media-stack). On switch the module now vendors the runtime stack out of the flake source (self): docker-compose.yml, Dockerfile.bazarr, .dockerignore, scripts/, terraform/ modules, static config - then seeds layer-1 and .env (from .env.example) alongside it and runs compose there. The module became a factory closed over self (mediaStackSrc). A consumer now imports homeManagerModules.mediaStack and never clones the repo.

Deploy subtleties worth keeping: scripts/ and terraform/ refresh with rsync --delete, but terraform excludes *.tfstate/.terraform/ so the orchestrator’s local state survives a redeploy; the config overlay (custom-cont-init.d, nginx html) runs without --delete and --excludes the generator-owned paths (komf, sonarr config.xml, nginx.conf) so the seam stays authoritative. Added .dockerignore because the bazarr build context is now deployDir (holds .secrets/config); its only COPY is a multi-stage --from, so excluding config/.secrets/terraform keeps secrets and MediaCover out of the build tar.

Verified: nix flake check + prek pass; rendered the autoStart=true activation and bash -n’d it clean; smoke-tested the deploy commands into a scratch dir (scripts + tf modules with lockfiles, state excluded, custom-cont-init.d present, generator-owned paths excluded). Same standing caveat: never run against a live home-manager switch. This makes all three layers (deploy + layer-1 + layer-2) convertible end-to-end in the repo.

2026-06-27 - Clean layer split: seam = layer-1, orchestrator = layer-2 (commit 78ca1a3)

Two refinements while thinking through the live setup. (1) PUID/PGID into the seam: new envFile option upserts the detected PUID/PGID into the compose .env (managing only those two lines). Required because compose interpolates ${PUID} from .env and recyclarr’s user: field needs that interpolation - an env_file can’t feed it. (2) Resolved the “where does layer-2 live” question. The host-side seam can’t do layer-2: it runs at home-manager switch (before the stack is up) and the host can’t reach the unpublished *arr ports (only nginx :8080 / seerr :5055). The key realization: the orchestrator sidecar is already a hashicorp/terraform container on the docker network, already running terraform apply for Sonarr - it’s the correct position (in-network, post-health). So I extended it to run Terraform for Radarr too, kept Prowlarr/Seerr on their setup scripts (provider gaps: no prowlarr_indexer resource; seerr permissions/watchlist not expressible), and removed the seam’s host-side reconcile entirely (it was the wrong position and a competing second layer-2 path). Clean split now: seam owns layer-1, orchestrator owns layer-2.

Worth keeping: I’d built the seam’s host-reconcile earlier this session; tracing the actual network topology (internal ports unpublished, orchestrator already in-network + already a TF image) showed it was the wrong design. The orchestrator was the answer all along.

2026-06-27 - Adopted: full cutover, mstack + Jinja deleted (commit 52b3af9)

The evaluation resolved to adopt, and this commit cuts over fully. The decision logic: the parity oracle had already proven generate.nix produces the same configs as mstack, and that oracle was only ever a migration aid - once you commit to the Nix generator as the source of truth, the old path is dead weight. So Jinja and mstack go.

Deleted (net −5,174 LOC): the 7 data .j2 templates (generate.nix produces them natively; nginx.conf.j2 renamed .tmpl as the one templater source); the parity oracle (render-expected.py, semantic-equiv.py) + the config-parity check; the redundant single-service PoC modules (sonarr.nix, sonarr-nixos.nix); and the repo’s entire Python project - src/mstack/ (1,316 LOC) + tests/ (1,568), pyproject.toml, uv.lock, the ruff/ty pre-commit hooks (ending the recurring uv.lock self-rewrite friction), the sphinx docs build, and the PyPI-publish + Python-lint CI workflows.

The one piece of new work: the seam now emits .secrets/generated/<svc>.env (<SVC>_API_KEY=..., mode 600) for docker-compose’s env_file - mstack’s secrets.py used to write those, so the seam took it over (verified against a temp dir). The two residual helpers were re-homed: htpasswd → scripts/gen-htpasswd.sh, .env bootstrap → just env-init. CI rewritten to nix flake check + tofu validate. All docs rewritten (used a subagent fan-out for the bulk: architecture/development/CONTRIBUTING/README/etc.); the RFC flipped to “accepted, implemented”.

Trade-off accepted deliberately: config generation now requires a Nix/home-manager host - the repo is no longer git clone && uv run mstack && docker compose up for non-Nix users. Justified: personal infra, the structured-config win (8 of 9 configs typed) plus deleting ~2,900 LOC of bespoke Python outweighs the portability. Still pending: a live home-manager switch against the real stack - the seam is built/verified in isolation but has never run live.

2026-06-27 - Brought the manga stack into the conversion (commit 182aac4)

After rebasing onto main (which had merged the manga stack, PRs 36-39), extended the Nix conversion to Suwayomi/Komga/KOMF. Each mapped onto the established pattern:

  • KOMF is layer-1 only: its static config/komf/application.yml is now a typed pkgs.formats.yaml value in generate.nix. It has no .j2 (mstack leaves it static), so the oracle for the parity check is the committed file itself - the flake check copies it into expected/ and semantic-equiv compares. Nice side effect: the check now enforces the Nix attrset and the committed file can’t drift. 8 data configs total.
  • Komga + Suwayomi are layer-2 with no Terraform providers (niche apps), so their existing idempotent setup-*.sh run from the reconcile seam like bazarr/qbittorrent. They authenticate with user-owned credentials (KOMGA_EMAIL/KOMGA_PASSWORD, SUWAYOMI_*) read from the environment - not minted. This validated the secrets reframing again: no new machine-minted keys, the manga services use the keep-separate credential class, and both scripts skip gracefully when unset.

The rebase itself was clean (zero conflicts across 11 commits): our subnet pin lives in compose’s networks: block, the manga services in services:, no overlap. Verified: config-parity PASS (8 data configs incl. komf), inject-secrets PASS, generated activation seeds komf + invokes both manga scripts, flake check + prek clean.

2026-06-27 - Converted layer-2 for all remaining services (commit 2e13b68)

Completed the layer-2 (runtime API config) conversion the Sonarr TF PoC started, for every service. The whole-stack seam’s reconcile (off by default) now drives all six, split by Terraform-provider availability. Built three new TF modules and wired the two no-provider services’ existing curl scripts into the seam.

Used a subagent fan-out to extract precise API specs from the three large setup scripts (prowlarr 511 / seerr 493 / bazarr 300 LOC) in parallel, then authored against them. Key technique: tofu validate against the real downloaded provider schemas verifies every resource/attribute without a live stack - I dumped each provider’s JSON schema (tofu providers schema -json) and authored from it, which caught wrong attribute names immediately (radarr copy_using_hardlinks not hardlinks_copy, recycle_bin not recycle_bin_path, etc.).

Findings worth keeping:

  • Seerr’s provider is real and good - correcting the RFC’s “single-maintainer 0.x, adopt/fork” hedge. Josh-Archer/seerr (0.19.3) has typed seerr_sonarr_server/seerr_radarr_server resources AND seerr_*_quality_profile data sources that resolve a profile ID by name (querying Sonarr/Radarr directly) - the exact runtime lookup the bash script did with curl+jq, now declarative. extra_payload_json covers anime-profile fields the typed resource omits.
  • Two genuine provider gaps, both documented in-module: (1) devopsarr/prowlarr has no prowlarr_indexer resource (indexer schemas are dynamic Cardigann defs), so the 6 indexers stay in setup-prowlarr.sh; the tag/proxy/app wiring converts cleanly. (2) devopsarr/radarr’s colon_replacement_format lacks Radarr’s “smart” (4) value the script uses; spaceDashSpace is the nearest equivalent.
  • Seerr’s non-declarative remainder stays scripted: Plex OAuth bootstrap, default/per-user permissions (provider’s main_settings has no default_permissions; per-user needs enumerating runtime Plex users), discover sliders. These are genuinely not declarative, exactly as the spec analysis predicted.
  • Bazarr/qBittorrent need no provider - settings-blob APIs are why none exists. Their existing idempotent setup-*.sh are the terminal answer, now invoked by the seam (minted keys exported as <SVC>_API_KEY) instead of the orchestrator.

All 4 TF modules tofu validate clean; the generated reconcile shell builds (exercised via reconcile=true in the PoC, built not switched); nix flake check + prek pass. Reconcile is unrun against the live stack (needs the media-stack network position), same caveat as the Sonarr PoC. With this, both layers are convertible end-to-end; what remains unproven is a live run and the deploy layer (compose→Nix).

2026-06-27 - Generalized the secret seam to mint all five keys (commit 9b25650)

Closed the gap between “Sonarr-only seam” and “full layer-1 + secrets.” One home-manager activation pass (nix/modules/media-stack.nix) now mints all five *arr/seerr API keys into a shared state dir and injects them across every config.

The conceptual move that made it clean: the five keys are machine-minted tokens with no external meaning (they only need to exist, match the format, and be consistent between an app and its clients), so the store holds only placeholders (@SONARR_API_KEY@ …) and the real value is minted at activation and sed-injected. The external credentials (OpenSubtitles, AniDB, basic-auth password) are categorically not mintable and stay user-owned.

The elegant part: mint-once-share-everywhere falls out structurally. A key like SONARR_API_KEY is referenced by three configs (sonarr.xml, bazarr.yaml, recyclarr.yml - the generator cross-refs proved this), so a single sed sweep across every seeded file injects the same value into all three. No coordination logic needed; the shared-placeholder + one-sweep design enforces consistency by construction.

A Nix purity point worth recording: you cannot mint a secret at eval or build time (both pure/reproducible, and the output would land in the world-readable store anyway). “Generate in Nix” means in the Nix-managed activation step, which runs imperatively on the host. The existing sonarr.nix mint-or-read seam was already in the right place; this just generalized it.

Verification was thorough: a new inject-secrets check (placeholders seeded, keys injected via the module’s own sed args, asserts no placeholder survives + Sonarr key identical across its 3 configs), plus I ran the generated activation script against a temp dir and confirmed it mints all 5, persists them, is idempotent on re-run, honors a $SONARR_API_KEY env override, and leaves nothing behind. nix flake check clean, all prek hooks pass.

sonarr.nix stays as the vertical seam (it keeps the layer-2 Terraform reconcile the whole-stack module omits); media-stack.nix is the horizontal one (all configs + secrets, no per-service reconcile). Decision timeline still gated on the Terraform observation week.

2026-06-27 - Reworked layer-1 to a hybrid native-generator + templater (commit 6f20a56)

Reversed the previous entry’s central finding. Instead of string-templating all 8 configs for byte parity, the generator (nix/lib/generate.nix) now emits each config in the representation Nix is best at: pkgs.formats.{xml,json,ini,yaml} for the seven data configs, a small textual templater (nix/lib/render-templates.nix) for nginx.conf only. The old renderers.nix string-list approach is deleted.

What changed my mind:

  • pkgs.formats.xml exists and emits real arbitrary-schema XML. The earlier “no typed XML backend” claim (and the whole “byte parity requires text-templating” conclusion) was wrong. Native typed generation reaches 7 of 8, not 4. The settings now live as typed Nix attrsets - no hand-escaped backslashes, no config-as-quoted-strings.
  • The cost is reformatting, not correctness. Native generators sort keys, drop comments, add <?xml?>, so the bytes differ from mstack even when the data is identical. So the oracle changed: nix/parity/semantic-equiv.py parses both sides (JSON/YAML/INI/XML) and deep-compares. Verdict: 7 data configs semantically identical, nginx still byte-exact via the templater. This is a strictly better deal - the comment-bearing intent (recyclarr’s TRaSH-id rationale) moves into the Nix source as named bindings, which beats a comment in a generated file.
  • One subtle compare gotcha: bazarr writes its apikey unquoted, so an all-digit test key parsed as a YAML int on one side and a string on the other. Fixed by using a realistic hex key (contains letters) - which also makes the test context more faithful.

render-templates.nix is trimmed to the two cases that still need text (nginx, and arr for the modules’ writable-seed path). Verified: config-parity PASS, nix flake check clean, all prek hooks green (committed --no-verify only for the recurring uv.lock self-rewrite). Docs updated: nix/README.md and the RFC’s PoC-evidence section now describe the hybrid and retract the byte-parity/pkgs.formats claims.

Decision timeline still unchanged (gated on the Terraform observation week). This is an ergonomics/correctness improvement to the eval PoC, not a commitment to migrate.

2026-06-27 - Nix Sonarr PoC built (commit f29d87d)

Made the evaluation symmetric: the Nix side now has a PoC with evidence, matching the Terraform side. Lives in nix/ on feature/iac-declarative-config.

  • Shared pure renderer (nix/lib/render-sonarr-config.nix) feeds two backends: a runnable home-manager module (nix/modules/sonarr.nix) and a NixOS sketch (nix/modules/sonarr-nixos.nix, the backend seam for a future Linux host). Same generation output across both.
  • Seam proven: generate → mint-or-read secret (never in /nix/store) → seed writable config.xml → detect host facts → tofu apply against the existing terraform/sonarr module. Reconcile is off by default so hm switch never touches a live Sonarr.
  • Verified (aarch64-darwin, nix 2.32.1): sonarr-parity check PASS (Nix render == config.xml.j2 byte-for-byte, which also closes a gap the Terraform RFC left open), sonarr-activation builds, nix flake check green.
  • Settled the “NixOS module on my Mac?” question: a NixOS module compiles to systemd (macOS can’t run it); the macOS-native equivalent is a home-manager module. Same module-system ergonomics, Docker-on-mac backend. The backend seam keeps a future NixOS host cheap.
  • Confirmed the mutable-file ownership concern is a non-issue: the home-manager backend is a slim reimplementation of the mkComposeDeploy pattern already in use (see Homelab Services Architecture).

Still open: the decision timeline is unchanged. The Terraform PoC still needs ~1 week of observation before the migrate-vs-keep call; the Nix PoC makes the eval symmetric but does not commit us.

2026-06-27 - Entire layer-1 generation converted to Nix, byte-for-byte (commits 89a4eb1, ae66eff)

Extended the Nix PoC from one service to all 8 mstack templates. nix/lib/renderers.nix renders sonarr/radarr/prowlarr config.xml, qBittorrent.conf, nginx.conf (incl. the {% if %} basic-auth conditional), bazarr/config.yaml, recyclarr.yml, seerr/settings.json. A parity check diffs every render against the real mstack Jinja Environment (oracle: nix/parity/render-expected.py). Result: config-parity PASS, all 9 renders (nginx rendered both auth-off and auth-on) byte-for-byte identical.

Findings worth keeping:

  • Byte parity requires text-templating, not pkgs.formats. The .j2s are hand-authored text with holes, so pkgs.formats (which reformats YAML/JSON/INI) can’t reproduce them. The RFC’s earlier “pkgs.formats covers 4 of 8 natively” is a reformatting path, not a drop-in one. Sharpened the RFC accordingly.
  • Jinja keep_trailing_newline defaults False - every mstack output has no trailing newline. My first Sonarr check was diffing the raw .j2 (which has the newline) and was subtly wrong; only diffing real Jinja output is correct. The oracle caught it.
  • nginx {% if %} whitespace: under Jinja defaults the if-line always emits one blank line; enabled adds the two auth lines plus the endif blank. Modelled exactly.

Toolchain friction (worth remembering): adding any Python file triggers the project-wide ty hook (pass_filenames: false), which surfaced a pre-existing environment.py return-type error (fixed in 89a4eb1). The ty hook’s uv run also rewrites uv.lock with an exclude-newer (from the env), self-aborting the commit - this is the recurring stray uv.lock change all session. Committed the Nix work with --no-verify after verifying uv run ty check, ruff, and parity pass directly. Also fixed .gitignore: the Python lib/ rule was silently ignoring nix/lib/ (the renderers almost didn’t get committed).

Scope now: full generation is convertible and proven; only Sonarr has the orchestration-seam module. Decision timeline unchanged (still gated on the Terraform observation week).

Graph View

Backlinks