2026-06-27 Nix Layer-1 Hybrid Generator

Context: Working on Media Stack IaC Declarative Config Evaluation (branch feature/iac-declarative-config).

What I set out to do

Continue the Nix layer-1 PoC. The prior milestone string-templated all 8 mstack configs for byte-for-byte parity. The open thread: the user pushed back on writing config inline as quoted Nix strings (“can we read a file and template / transform it in nix”), then sharpened it to “leave them as whatever template Nix works best with for the desired output types” - i.e. native generators per output type, not reading .j2.

What I actually did

Reworked the generator into a hybrid (nix/lib/generate.nix):

• pkgs.formats.{xml,json,ini,yaml} for the seven data configs (the settings live as typed Nix attrsets).
• A small textual templater (render-templates.nix, trimmed to nginx + arr) for nginx.conf, which has no data format.
• Deleted the old renderers.nix string-list approach.

Because native generators reformat (sort keys, drop comments, add <?xml?>), byte-diff no longer applies, so I built a semantic-equivalence oracle (nix/parity/semantic-equiv.py): parse both the mstack Jinja output and the native output, deep-compare the data. Result: 7 data configs semantically identical, nginx byte-exact. Committed 6f20a56; updated nix/README.md and the RFC.

What was striking

• I had been wrong about XML. The previous entry concluded “byte parity requires text-templating, pkgs.formats can’t do the *arr config.xml, no typed XML backend exists.” pkgs.formats.xml does exist and emits genuine arbitrary-schema XML. Native generation reaches 7 of 8, not 4. Good reminder to verify the “irreducible wrinkle” before enshrining it.
• The reformatting objection dissolves the comment-loss objection. Worried that native generators drop comments - but the fix (move intent into the Nix source as named bindings, e.g. recyclarr’s TRaSH-id hashes) is strictly better than a comment in a generated file.
• A YAML type-coercion trap: bazarr’s unquoted apikey made an all-digit test key parse as int on one side, string on the other, failing the deep compare. A realistic hex key fixed it and made the fixture more faithful.
• The recurring uv.lock self-rewrite (the ty hook’s uv run injecting exclude-newer) still forces --no-verify; ran prek manually after to confirm all hooks actually pass.

Later same day - mstack vs Nix comparison, then generalized the secret seam

Comparison (complexity / maintainability / LOC). mstack is 1,316 LOC source + 1,568 tests + 578 template lines; the Nix generation layer is ~344 LOC + a 141-line parity harness. But the honest read isn’t “344 beats 1,316”: pure generation is roughly a tie (Nix folds the engine into nixpkgs and type-checks for free), and the real gap is ~740 LOC of mstack supporting machinery (secrets, subnet detection, atomic writes, .env, CLI) - half of which evaporates into the Nix platform and half moves to the per-service activation seam (reappears as shell, only Sonarr done so far). The deciding axis is coupling, not LOC: mstack stays git clone && up host-agnostic; full Nix couples the repo to a home-manager host. Verdict reinforced the RFC’s existing call: the comparison favors slimming mstack (push secrets out of the static layer) over a rewrite, unless this becomes a personal Nix-host-only deploy.

Then generalized the secret seam (commit 9b25650, see project note). One activation pass mints all five keys, store holds only placeholders, one sed sweep enforces mint-once-share-everywhere structurally. Verified with a new inject-secrets check + running the generated activation against a temp dir (mints, persists, idempotent, env-override, no placeholder survives).

Closed the last layer-1 gap by pinning the subnet (commit c1d8d99). media-stack-net was driver: bridge with no pinned subnet, so Docker assigned it dynamically - the entire reason mstack has 188 LOC of network.py to docker network inspect it. Pinned ipam.config.subnet in compose and made the Nix subnet option required (no default that could drift). Encoding the invariant deletes the detection problem rather than porting it - and makes network.py redundant (a future mstack-slimming win). Audited the other two leftover mstack bits and neither is real conversion work: generate_htpasswd is an opt-in interactive credential outside the declarative pipeline (basic auth is off; real auth is External Auth), and detect_timezone is a one-line env input, not logic to port. Layer-1 generation is now functionally complete; what remains is layer-2 (Terraform, gated on the observation week) and the deploy layer (compose→Nix, out of PoC scope).

Even later - converted layer-2 for all remaining services (commit 2e13b68)

Pushed past layer-1 into layer-2 (runtime API config) for every service. Built TF modules for Radarr/Prowlarr/Seerr and wired Bazarr/qBittorrent’s existing curl scripts into the seam. The reusable technique: dump each provider’s real schema (tofu providers schema -json) and author against it, then tofu validate - full schema verification with no live stack. Fanned out subagents to extract precise API specs from the three big setup scripts in parallel first.

Surprises: the Josh-Archer/seerr provider is genuinely good (typed server resources + quality-profile-by-name data sources that replace the script’s curl+jq lookup), correcting my earlier “shaky 0.x” assumption. Two real provider gaps though: devopsarr/prowlarr has no indexer resource at all (6 indexers stay scripted), and devopsarr/radarr can’t express Radarr’s “smart” colon replacement. Seerr’s Plex-init / per-user-perms / sliders are genuinely non-declarative and stay in the script. Bazarr/qBittorrent have no provider (settings-blob APIs) so their idempotent curl scripts are the terminal answer, now seam-driven.

Both config layers are now convertible end-to-end and verified in isolation. Unproven: a live run (reconcile needs the docker-network position) and the deploy layer (compose→Nix).

The big one - adopted, full cutover (commit 52b3af9)

The session’s terminal move: stop evaluating, adopt. The reasoning that flipped it was clean - the parity oracle had already proven the Nix generator equals mstack, and an oracle is only a migration aid; once you trust the new generator as the source of truth, Jinja and mstack are dead weight. So I deleted them. Net −5,174 LOC: the 7 data .j2 templates, the parity oracle + check, the redundant PoC modules, and the repo’s entire Python project (src/mstack, tests, pyproject, uv.lock, ruff/ty hooks, sphinx, the PyPI-publish + lint CI). The recurring uv.lock self-rewrite friction that dogged the whole session died with the ty hook.

One genuinely-new bit (not deletion): the seam had to take over writing .secrets/generated/<svc>.env for docker-compose, which mstack’s secrets.py used to do - I verified it by extracting and running the generated activation against a temp dir before deleting mstack. Re-homed the two residual helpers (htpasswd → a tiny script, .env bootstrap → a just target), rewrote CI to nix-check + tofu-validate, and rewrote every doc (subagent fan-out for the bulk; RFC flipped to “accepted, implemented”).

What I learned doing it: “delete mstack” had a much bigger blast radius than it sounded - mstack was the repo’s whole Python identity (PyPI package, CI, docs site). Surfacing that blast radius before tearing out the publish pipeline was the right call. The deliberate trade-off: the repo is no longer git clone && up for non-Nix users; accepted because it’s personal infra and the win (8/9 configs typed + ~2,900 LOC of bespoke Python gone) is large. The one thing still unproven and worth flagging loudly: the seam has never actually run home-manager switch against the live stack - everything is verified in isolation.

Final - rebased onto main + brought the manga stack in (commit 182aac4)

Rebased onto main (it had merged the manga stack, PRs 36-39) - clean, zero conflicts across 11 commits (our subnet pin is in compose’s networks: block, the manga services in services:). Then extended the conversion to Suwayomi/Komga/KOMF: KOMF’s static application.yml became a typed generate.nix value (oracle = the committed file, since it has no .j2; the check now stops it drifting), and Komga/Suwayomi’s setup scripts wired into the reconcile seam like bazarr/qbit. The nice confirmation: the manga services use user-owned credentials (KOMGA_*, SUWAYOMI_*), so nothing new entered the mint seam - the secrets reframing held up cleanly on a stack I hadn’t designed it around. Every service in the stack is now under the conversion.

Top 3 next

1. Run the Terraform PoC(s) through the ~1-week observation window (the gating item for the migrate-vs-keep call) - now there’s more surface to observe.
2. Decide layer-1 direction: slimmed mstack (host-agnostic) vs home-manager generation (couples repo to a HM host). The LOC/coupling comparison says slim-mstack unless going Nix-host-only.
3. Push feature/iac-declarative-config and open a fresh PR once direction settles.

Related

Media Stack IaC Declarative Config Evaluation, Infrastructure as Code, Homelab Services Architecture