2026-08-23 Two Orchestrator Bugs That Repaired Themselves Back Into Breakage

What I set out to do

Explain two media-stack orchestrator failures I had been handed as already diagnosed: a Bazarr language-profile HTTP 500 (“likely a race, orchestrator hit Bazarr before it finished initializing”) and a Suwayomi org.h2.jdbc.JdbcSQLNonTransientException: The database is read only (“recurs across several past runs, probably not writable by the container’s user”).

Both diagnoses were wrong, and so was the framing that there were two problems.

What I actually did

docker logs keeps output across restarts, so the orchestrator container held every run since Aug 17. That one fact did most of the work.

The exit(1) I was asked about was not Bazarr. Bazarr never ran. The Aug 22 06:55 run died at tofu init failed for Sonarr, which sets arr_ok=0, and orchestrate.sh then printed Skipping Prowlarr/Bazarr/Seerr setup. Komga and Suwayomi both succeeded in that same run. The Bazarr 500 was real but came from other runs.

The Suwayomi read-only error was not recurring. Exactly four lines, all at 2026-08-17T18:48:16, in a log going back to container creation 35 seconds earlier. The mount is rw, the file is 0644 owned by uid 501 in a 0755 dir, the container runs as 501:20, and database.mv.db had been written that morning. Today’s run refreshed 1375 extensions without complaint. It happened once, 27 seconds after the whole stack was recreated, which is consistent with H2 opening the DB while the outgoing container still held the file lock over a macOS bind mount. Nothing to fix.

That left two real bugs, and the interesting thing is that they are the same shape: both repair themselves and then get re-broken, forever.

Sonarr: tofu heals the lock, home-manager switch reverts it

The committed terraform/sonarr/.terraform.lock.hcl carries one h1: hash. Radarr, Prowlarr and Seerr carry four each. h1: hashes are per-platform, and Sonarr’s single one does not cover linux_arm64, so tofu cannot verify the cached provider, re-downloads it from GitHub, and writes the correct hash into the deploy-dir lock in place. Init succeeds. Then the next home-manager switch re-vendors terraform/ from the flake: the rsync excludes .terraform/ but not .terraform.lock.hcl, so the repaired lock is reverted to the committed one-hash version, and the next run needs the network again.

Aug 22 is simply where that download hit a connection refused blip. The provider binary’s mtime told the story before I understood it: terraform-provider-sonarr_v3.4.2 dated to the last run, ..._radarr_v2.4.0 still dated to July 14.

The reason only Sonarr is affected is a nice bit of history. Commit 27f96bc regenerated the other three locks because it bumped their provider majors. Sonarr was skipped there for a stated reason: it was “already pinned to a current major.” Being correct is what left it stale.

Bazarr: the script strips a key, Bazarr backfills it, repeat

Not a race, and not timing-dependent at all. From bazarr.log:

Exception on /bazarr/api/system/settings [POST]
  api/system/settings.py:109  list_missing_subtitles_movies()
  subtitles/indexer/movies.py:183  if language['audio_only_include'] == "True":
KeyError: 'audio_only_include'

Bazarr’s profile-item schema gained audio_only_include. setup-bazarr.sh built items without it, so desired never equalled existing and it POSTed every run. The POST succeeds at writing, then the post-save reindex reads the flag unconditionally (language['...'], not .get()), throws, and returns 500 with the stripped profile already persisted. Bazarr backfills the default on restart, and the loop closes.

The runs between Aug 17 22:11 and Aug 19 07:43 that printed 'Subtitles' profile already up to date are the window where the stored profile was still stripped, so no POST was issued. A Watchtower Bazarr update reintroduced the key and the failures resumed.

Collateral damage worth noting: the stripped profile also killed Bazarr’s own nightly movies_full_scan_subtitles with the same KeyError at 04:00. And the UI-created English profile kept the key the whole time, while our Subtitles profile lost it, which is a clean tell for “this was written by our script.”

What was striking

My verification was contaminated, and being wrong about it is what found the real mechanism.

I regenerated the Sonarr lock, ran an A/B, and the “before” case passed, which should have meant my whole theory was wrong. It did not, because the deploy-dir lock I had backed up as the “old” one already had two h1: hashes: tofu had self-repaired it on a previous run. I had been diffing against a healed artifact and calling it the baseline. Re-running against git show HEAD: gave the honest result immediately (Installing devopsarr/sonarr v3.4.2..., lock going 1 to 2 hashes), and that in-place rewrite is the exact half of the loop I had not understood yet.

I also burned a test on --network none, concluding the fix had failed. It had not: tofu always fetches the registry discovery document during init regardless of cache, so every module fails offline. Radarr as a control took ten seconds and killed the whole line of reasoning. Run the control before trusting the experiment, not after it confuses you.

The other thing worth keeping: I got a live reproduction for free. An orchestrator run at 00:21 today hit the Bazarr 500 and re-stripped the profile while I was mid-investigation, which is why the two profiles disagreed when I fetched them. Fastest confirmation of the cycle I could have asked for.

The fix

• terraform/sonarr/.terraform.lock.hcl: regenerated via tofu providers lock -platform=linux_amd64 -platform=linux_arm64 -platform=darwin_amd64 -platform=darwin_arm64. Pure addition of 13 hashes, version and constraints untouched.
• scripts/setup-bazarr.sh:97: added audio_only_include: "False" to the item map, with a comment on why the key set must be complete.

Verified end to end. Sonarr’s init now says Using previously-installed with the provider binary mtime unchanged and the lock no longer rewritten; the Bazarr script POSTed once to heal the stored profile, then converged to already up to date on the second run. Full orchestrator run exits 0, === All services configured successfully ===, the first green run since Aug 19. /code-review low returned no findings.

Not committed yet. Both files are working-tree only, and I hand-copied them into ~/.local/share/media-stack so the stack is healthy right now, which means the next home-manager switch will revert both until they are committed and the flake input bumped. That is the same three-part-change trap as 2026-08-08 The Indexer Prowlarr Never Asked FlareSolverr About: module, consumer config, and the lock.

The Bazarr fix is targeted, not structural. It patches the one key Bazarr added; the next schema addition reopens it, because the script builds items from scratch instead of merging desired values onto what Bazarr returns. Merging would be immune to new keys and is the better fix if this recurs.

Related

Sits with 2026-08-08 The Indexer Prowlarr Never Asked FlareSolverr About, 2026-07-14 Media Stack Nix Cutover (which is where the vendor rsync that reverts the lock comes from), and Homelab Services Architecture. Adjacent project notes: Media Stack IaC Declarative Config Evaluation and Add Manga Stack to Media Stack.

Landed (2026-08-24)

Merged as media-stack#57, commit 5b088fe, all four checks green.

A third fix went into the same PR. The Terraform CI job was already red on main and had been since Aug 13: josh-archer/seerr re-signed its already-published release assets on Aug 21-22 with an RSA key the OpenTofu registry does not advertise, so every version now fails install with authentication signature from unknown issuer. Verified by hand with gpg --verify against the key the registry itself serves:

version	signed	key	verifies
0.2.5	2026-03-05	EdDSA 71ACA6D4...E5B2	yes
0.20.7	2026-08-21	RSA F3D8E9C6...751C	no
0.39.1	2026-08-22	RSA F3D8E9C6...751C	no

Retroactively re-signing old releases is what broke a pin that had worked for months with nothing changing on our side. Integrity was never in doubt, only provenance: all 14 zh: hashes in the committed lock still match the currently published SHA256SUMS byte for byte. Reported upstream as terraform-provider-seerr#217. The job used set -e and seerr sorts before sonarr, so it had been aborting before sonarr was ever validated; the lock fix was not actually covered by CI until this landed.

Two process notes worth keeping.

The automated review was half right, and the half that was wrong was confident. It correctly caught that my PR body claimed a 4-platform lock command while sonarr ended up with 14 h1: hashes against the siblings’ 4. The answer is a tofu version behavior change (current tofu derives an h1: for every platform in the signed SHA256SUMS), which I confirmed by re-running the identical command against radarr and getting 13 where #44 committed 4. A second review then asserted the siblings “each have 14”, which is simply false. Two reviews of the same file, contradicting each other, one right.

I dirtied files by verifying. Simulating the CI loop ran tofu init across all four modules in the repo, which rewrote the radarr and prowlarr locks in the working tree. It never reached the PR, but a reflexive git add -A would have widened the diff silently. Reverted.

Deploying it (2026-08-24)

Pulled main, bumped media-stack in ~/.config/nix/flake.lock from 4f7604b to 5b088fe, and ran home-manager switch --flake .#aarch64-darwin.

The revert mechanism demonstrated itself on the way. Before the switch the deploy dir had already lost both fixes (audio_only_include count 0, sonarr lock back to the self-repaired 2 hashes) because an intervening switch had re-vendored the pre-merge flake over my hand-copied files. That is exactly the loop the PR describes, observed live rather than reasoned about, and it is the argument for never patching the deploy dir as a shortcut.

After the switch, both files are byte-identical to the repo, and the orchestrator (via autoStart) exits 0 with All services configured successfully:

• Sonarr: Using previously-installed devopsarr/sonarr v3.4.2, lock still 14 hashes and unmodified. No re-download, no in-place rewrite. Loop broken.
• Bazarr: ✓ Applied language profile 'Subtitles', HTTP 200. It POSTed because the intervening old-script run had re-stripped the key; the last 500 in bazarr.log is 21:05:27 UTC, three minutes before this run at 21:08:54.

~/.config still has an uncommitted nix/flake.lock. The repo sets nix/flake.lock -diff in .gitattributes, so git renders it as binary in git diff by design.

Follow-up: narrowing the carve-out (2026-08-24)

Merged as media-stack#58, commit 69efd38, CI green on main.

The carve-out from #57 swallowed any seerr tofu init failure while its comment promised a narrow exemption for one upstream break. A corrupt lock or a real provider regression would have produced the same warning and left CI green, masking exactly the class of breakage the carve-out was supposed to be narrower than. Now the init output is captured and only unknown issuer / not signed with a valid signing key is tolerated; anything else emits ::error and exits 1. Verified all three paths against a scratch copy of terraform/: signature error tolerates, unrelated error fails, and a successful init falls through to validate (the self-retiring path).

Confirmed a version bump cannot fix this. Every published version through 0.40.0-rc.4 is signed with the unregistered RSA key; tofu init against >= 0.39.0 still fails. 0.2.5 is the only one that verifies and predates the ~> 0.20 bump in #44. It is a property of the publisher’s release pipeline, not of any version, so a new release is just one more artifact from the same broken pipeline.

Switched the annotation to the @upstream-issue: convention documented in ~/.config/docs/reference/check-upstream-issues.md. My original “Tracked upstream:” line was ad-hoc and invisible to the CLI that scans for these. Now check-upstream-issues reports it, and will flag the workaround as deletable the moment #217 closes. First use of the tag in media-stack.

A green check is not evidence of a review

Worth keeping. The review workflow ran on the final commit, reported success, and posted nothing. I nearly reported that as “reviewed, no findings.” The run data said otherwise:

run	turns	cost	posted
first commit	16	$0.31	yes
final commit	4	$0.78	no

Four turns is not enough to read a diff and post a comment. gh pr comment was allowlisted, so nothing blocked it; the run just did a fraction of the work and exited success anyway, because the workflow’s exit status does not depend on whether a review was produced. Re-running it produced a real review. The only reliable signal is whether a comment exists, not whether the check is green.

The real review found no bugs, independently traced the two echo "$out" calls to confirm they are mutually exclusive, and raised one thing I had not considered: capturing output loses real-time streaming for seerr’s init, so a hypothetical hang would show an empty log until timeout. Left as-is; it is inherent to needing the text to match against, and the block is meant to be deleted when #217 closes.

Upstream #217 remains open with no response.