2026-08-26 The Outage Was Thirty Seconds, The Damage Was Twelve Hours

What I set out to do

Read the media-stack logs and find anything that needed handling. Open-ended triage, no specific complaint.

What I found

Sonarr was blind and had been for hours. Not degraded, blind: an RSS sync I triggered by hand returned Reports found: 0, and sonarr.db showed all five indexers disabled with escalation levels 5 through 9.

The cause was not the indexers. I tested all six through Prowlarr and every one came back HTTP 200. The failure had already passed; only the punishment was left.

The mechanism is a cascade, and that is the part worth keeping. When outbound networking drops, every indexer request fails at once and Prowlarr escalates its per-indexer backoff on the hardcoded upstream ladder: 1h, 3h, 6h, 12h, 24h. But Sonarr and Radarr reach their indexers through Prowlarr, so once Prowlarr is backed off it answers them with 429 TooManyRequests, and they count that as a fresh failure and escalate on top of it. The logs show the doubling plainly:

07:56 Torznab: API Request Limit reached for 1337x. Disabled for 01:00:00
08:57 Torznab: API Request Limit reached for 1337x. Disabled for 03:00:00
12:11 Torznab: API Request Limit reached for 1337x. Disabled for 06:00:00
18:11 Torznab: API Request Limit reached for 1337x. Disabled for 12:00:00

A thirty-second blip buys twelve hours of blindness, and nothing recovers on its own.

The blip is the host sleeping. Every service logged the same instant of total outbound failure, which is what made it obvious it was not any one service’s problem:

sonarr        00:33:31  Connection refused (services.sonarr.tv:443)
radarr        00:33:31  Connection refused (radarr.servarr.com:443)
prowlarr      00:34:01  Connection refused (prowlarr.servarr.com:443)  x4 addresses
flaresolverr  00:34:01  net::ERR_CONNECTION_REFUSED

pmset -g log confirmed it: Entering Sleep state due to 'Maintenance Sleep' several times a night. The Docker Desktop VM freezes with the Mac. Bazarr made it explicit in a way the *arrs did not, by reporting the gap directly: Run time of job "Sync with Sonarr" was missed by 0:56:15.

Four more things fell out of the same outage:

• Recyclarr had been dead since 04:06 on the 25th, and had corrupted itself doing it. It deletes its TRaSH clone and re-clones on any git error, so a run that starts while the host is offline leaves an empty repo and then fatals on git reset FETCH_HEAD. I confirmed the wreckage: fatal: your current branch 'master' does not have any commits yet. On @daily, at midnight, on a laptop, that is a 24h gap by construction.
• Seerr wrote 4388 identical warnings, one every 10 seconds for 12 hours, retrying a GitHub version check. Nothing capped it, because there is no log rotation anywhere: no logging: block in compose, no log-opts in daemon.json, and json-file defaults to unbounded.
• missing-search was merged but never deployed. The module vendors from the flake input, not the checkout, and ~/.config/nix/flake.lock still pinned 69efd38 while the feature was in 06e839c. Exactly the trap from 2026-08-23 Two Orchestrator Bugs That Repaired Themselves Back Into Breakage, hit again from the other direction.
• Prowlarr’s [Error] FlareSolverr: Proxy validation failed and SystemTimeCheck: Unable to verify system time are the same blip, not separate faults. Every 6h health check landed inside a sleep window.

What was striking

The fix was hiding in the failure mode. I went looking for how to clear the backoff and assumed it meant writing to IndexerStatus in three sqlite files under running services. It does not. POST /api/v{1,3}/indexer/testall already does it: a passing test clears DisabledTill and decrements EscalationLevel, a failing test leaves the indexer disabled and escalates it.

That asymmetry is the whole design. A test sweep can only restore indexers that work right now, so it is structurally incapable of falsely re-enabling a broken one. It also resolves the standing AGENTS.md rule about not clearing The Pirate Bay’s counter, without needing to encode TPB as a special case: TPB’s test fails on its real Cloudflare block, so the sweep leaves it disabled. I watched it happen live in the same run that cleared the other four.

I found it by measuring rather than reasoning. I ran testall on Sonarr purely to see what it did, dumped IndexerStatus before and after, and the answer was in the diff. Nyaa 5→4, TokyoTosho 9→8, 1337x 8→7, EZTV 9→8, all DisabledTill cleared, and TPB alone escalated to a fresh 24h. Sonarr went from 0 reports to 289 immediately after.

Order turned out to be load-bearing in a way I would not have guessed. Sweeping Sonarr before Prowlarr is worse than doing nothing: its test proxies through Prowlarr, collects the 429, and escalates. Same for the connectivity gate. Sweeping while the network is still down fails every test and deepens the exact hole being dug out of. Both are now unit-tested, because both are silent when wrong.

Running it for real caught what the unit tests could not. The integration run printed Prowlarr’s API key into the log, because Sonarr and Radarr report an upstream failure by quoting the whole URL, apikey parameter and all. Redaction plus a test for it went in immediately. Green unit tests, real leak.

The fix

media-stack#59, one PR:

• scripts/recover-indexers.sh: the sweep, Prowlarr first, gated on a connectivity probe, with API keys redacted out of upstream error text.
• scripts/search-missing.sh: freeze detection. The interval is slept in short ticks and elapsed compared to requested; an overshoot past 300s means the host was suspended, which is otherwise invisible from inside a container. On a freeze it recovers first, then searches, because searching through backed-off indexers finds nothing and then waits another full interval.
• scripts/recover-indexers.test.sh: 18 cases wired into CI as the indexer-recovery flake check. date and sleep are shell functions, so a simulated one-hour suspend runs instantly with no clock and no network.
• Recyclarr @daily → every 6h, an x-logging anchor on all 14 services, and a just deploy that bumps the flake input before switching.

Deployed missing-search first, separately, by bumping the lock and running hm. It is running now.

What I would do differently

I spent a while treating the 6-hourly Prowlarr error bursts as a scheduled failure before checking pmset. The timestamps were the tell the entire time (00:33, 06:33, 12:33, 18:33, drifting ~25s each cycle), and I read them as “a timer fires and fails” rather than “a timer fires into a sleep window”. One pmset -g log at the start would have reframed everything downstream.

Also worth noting: the user told me mid-task that they consider inline comments a code smell. My first draft was dense with them. Stripped to file headers plus # Usage: docstrings, with the reasoning moved into the AGENTS.md decision log and the two real constraints turned into names instead (RECOVERY_TARGETS_PROWLARR_FIRST, redact_api_keys). Saved as a standing preference.

Open

Nothing in the repo can stop the host sleeping. sudo pmset -c sleep 0 (never sleep on AC, battery untouched) is the host-side half, and it is still the user’s call. Everything in the PR is about surviving the outage when it happens anyway, which it will regardless of the sleep setting.

Landed (2026-08-27)

Merged as media-stack#59, commit c6c6689, CI green on main. 41 test cases.

Four review rounds, and every one found something real. That is the number worth keeping. The pattern was consistent: each round caught a silent success, not a crash.

1. sweep_service fed the response straight to jq '.[]' after only checking it was valid JSON. A stale key returns {"message":"Unauthorized"}, jq iterating an object yields empty strings rather than erroring, and the sweep printed [OK] 0 indexer(s) passed and returned success while sweeping nothing, clearing the recovery debt on the way out. Reproduced before fixing.
2. RECOVERY_OWED started false, so the only trigger was a freeze detected while the loop was running. A container recreated during an active backoff (reboot, Watchtower, docker compose up -d) would never sweep. Same failure the PR exists to fix, arriving through a different door.
3. common.sh’s api_request dumps the response body unredacted on any non-2xx. That was a third key-leaking path, and my AGENTS.md note had claimed “both paths are covered.” The claim was wrong and had to be corrected along with the code.
4. The redaction regex was only ever exercised under GNU sed, while production is Alpine/BusyBox (/bin/sed -> /bin/busybox). Ran the suite inside the real image: 41/41 pass. Verified rather than assumed, which was the whole point.

I leaked a real secret into a commit. The test fixture d9f70e8ad2... was the live Prowlarr API key, copied verbatim out of diagnostic output. Pushed in f1b50ad. Caught by gitleaks on a later run, not by me, and notably it had passed on the first commit because that occurrence sat inside a bracketed URL the rule did not match. Scrubbed via squash + force-push, but the orphaned object may persist and it went through Actions logs, so the key still wants rotating. Lesson: real output pasted into a fixture is a secret until proven otherwise.

I ran an unintended home-manager switch. Backticks around hm inside a git commit -m "..." string are command substitution. The shell ran the switch before the commit, and the commit then died on a parse error. It happened to be a no-op (hm does not bump the flake lock, so it re-vendored the same rev and the compose up changed nothing; orchestrator exited 0). Use a quoted heredoc for any commit message containing backticks.

The bug reproduced twice more while I was fixing it. Once mid-verification, where I misread the connectivity gate’s silence as a hang and spent two tool calls chasing a nonexistent bug before checking whether the network was actually up (it was not: every container had rc=7 while the host was fine, so the Docker VM’s NAT, not DNS and not the ISP). That silence became a real fix. And once overnight, where it did not self-heal for hours, which corrects what I first told the user: I had described it as a thirty-second blip that recovers on its own.

Deployed (2026-08-27)

just deploy bumped the flake input to c6c6689 and switched. Verified rather than assumed: the vendored scripts/* are sha256-identical to HEAD, every service now carries json-file max-size=10m max-file=3, and Sonarr/Radarr are back to 262 / 240 RSS reports.

The startup sweep fired on the first boot and behaved exactly as designed: 5 Prowlarr, 4 Sonarr, 3 Radarr indexers passed and had their backoff cleared, while The Pirate Bay stayed disabled on its real Cloudflare block. IndexerStatus confirms it, with DisabledTill NULL on every working indexer and set only on ProviderId 9/7/5. The sweep can only re-enable what passes a live test, so it cleared the collateral damage without touching the genuine failure.

Two things settled in production that had only been settled in tests:

The BusyBox redaction question. The last review flagged that redact_api_keys was validated only against GNU sed while production runs Alpine. Sonarr’s proxied-failure message quotes the whole upstream Prowlarr URL, and the deployed container rendered it apikey=REDACTED. Real path, real sed, real key.

The owed-debt design. The switch stopped and started the container, and the second start raced the rest of the stack still coming up: a bare docker start does not honor depends_on, so the sweep hit services that were not listening yet and failed all three with HTTP 000. RestartCount=0, so this was not the restart policy. It is benign precisely because the failed sweep leaves RECOVERY_OWED=true and retries next cycle instead of waiting for another freeze. That was one of the four review-round fixes, validated by accident within a minute of deploying.

Still open

• Rotate the Prowlarr API key.
• sudo pmset -c sleep 0 (never sleep on AC) is the host-side half. No repo change can stop the host suspending.
• Optional CI hardening the final review suggested and I did not do: run recover-indexers.test.sh inside the orchestrator image so a future BusyBox/musl regex divergence is caught automatically. The property is verified today but not kept verified.

Related

Sits with 2026-08-23 Two Orchestrator Bugs That Repaired Themselves Back Into Breakage and 2026-08-08 The Indexer Prowlarr Never Asked FlareSolverr About. Adjacent project notes: Media Stack IaC Declarative Config Evaluation.