2026-08-31 The 429 Was Never a Rate Limit

What I set out to do

Open-ended log triage on the media-stack after a few days away. Same shape as 2026-08-26 The Outage Was Thirty Seconds, The Damage Was Twelve Hours.

What I found

Sonarr and Radarr were blind again: every indexer disabled, ~21h left on the clock, escalation level 9. Prowlarr itself was healthy apart from The Pirate Bay. The recovery built in #59 existed and had not run.

The startup race is the reason. missing-search declares depends_on: {sonarr,radarr,prowlarr: service_healthy}, but depends_on only orders docker compose up. When the Docker daemon restarts, which is what a laptop wake looks like, restart: unless-stopped starts everything in parallel, and a bash+curl container is listening-ready in milliseconds while the *arrs take ~30s. Three restarts in the log show it: [FAIL] could not reach the indexer test endpoint x3, then HTTP Status: 000 on both searches.

This exact failure is already recorded in the Aug 26 entry, observed at deploy time and filed as “benign because it stays owed”. It is not benign. Owed means retried at the next 6h cycle, and a restart during an active backoff is precisely when the sweep matters most. Today it would have meant 6 more hours blind on top of 21 already lost.

What was striking

I proposed a mechanism I had not measured, and it was wrong. I saw this on Aug 28: Prowlarr’s sweep reported 5/6 indexers passing, and Sonarr’s sweep immediately after got 429 TooManyRequests on those same indexers, 0/5 passing. I read that as the sweep rate-limiting itself, and told the user the sweep might be “deepening the cascade it exists to clear”. Three checks killed it:

• The 429 body says what it is: <error code="429" description="Indexer is disabled till 09/01/2026 21:01:36 due to recent failures." /> It is Prowlarr’s own IndexerStatus.DisabledTill, not a request-rate limit.
• 50 rapid caps requests through Prowlarr, one indexer and then spread across all of them: 200 every time. No throttling exists to trigger.
• Upstream ProviderStatusServiceBase.RecordSuccess sets DisabledTill = null unconditionally when EscalationLevel > 0, decrementing the level by one. So a single passing test fully re-enables an indexer regardless of how deep the escalation is.

AGENTS.md already documented the 429 correctly (“429 TooManyRequests with ‘Indexer is disabled till …’”). My hypothesis contradicted the project’s own decision log and I did not notice. The log was right and I was reasoning past it.

The real explanation is duller and more useful. For Sonarr to get a 429 on an indexer Prowlarr had just cleared, Prowlarr must have re-disabled it in between. Sonarr’s testall does not reuse Prowlarr’s result: it issues a fresh live t=tvsearch&cat=...&limit=100 through Prowlarr, which runs a real query against the real indexer. Measured today: 2.4s, 7.4s, 11.3s, and 20.6s for 1337x via FlareSolverr. That is a far larger failure surface than a test that finished seconds earlier. On Aug 28 the network was still degraded, and 8 minutes later Prowlarr’s own sweep showed 4/6 failing on DNS/SSL ResponseEnded with FlareSolverr throwing ERR_CONNECTION_REFUSED.

So the weak point is the connectivity gate, not the ordering: api.github.com answering proves general outbound is back, not that the indexers are reachable. The sweep ran on a half-recovered network.

And the severity is lower than I first said. Because RecordSuccess nulls DisabledTill regardless of level, an early bad sweep costs one escalation step, and the next good sweep clears everything anyway. Escalation depth only bites if no further sweep happens. The 60s settle delay I used in the manual recovery was not what made it work; Prowlarr was simply healthy.

The fix

Recovered the live outage by hand first: staged testall sweep, all non-TPB indexers back, health clean. Nothing was actually lost, the 48h missing window was empty for both services.

Then, TDD, 17 new cases (58 total) in the indexer-recovery flake check:

• wait_for_services in recover-indexers.sh, probing each service’s unauthenticated /ping (honours UrlBase, needs no key), reusing the existing RECOVERY_TARGETS_PROWLARR_FIRST table. run_startup_cycle waits, then sweeps, then searches. A timeout is non-fatal: the sweep’s own failure path already leaves the debt owed.
• run_search_with_retries in search-missing.sh. run_search_cycle now reports failure instead of swallowing it with || true; both halves still run so a Sonarr outage does not cost Radarr its cycle.

My own test seams bit me twice, both the same lesson from the other side. A shell function has one namespace, so overriding run_search_cycle in an earlier test destroyed the real definition a later test needed (exit 127). And a probe counter kept in a variable read 0, because pending_services runs the probes inside a command substitution. The file-based SWEEP_LOG at the top of that test file exists for exactly that reason and I had read the comment explaining it.

Also in the logs, no action

Recyclarr hit the documented empty-clone FETCH_HEAD fatal on the 30th and self-corrected at the next 6h run, which is the whole point of that schedule change. Seerr’s GitHub version-check spam is now capped by the x-logging anchor. Bazarr errors are SignalR reconnect churn around outages.

Still open

• Rotate the Prowlarr API key. Still outstanding from Aug 26, and I pulled it into context again today by dumping IndexerStatus with LastRssSyncReleaseInfo included, which embeds the key in every downloadUrl. Select columns next time.
• Not deployed. The module vendors from the flake input, so this needs merging and then just deploy, not a bare home-manager switch.
• A guard that skips Sonarr/Radarr when Prowlarr’s own sweep shows most indexers failing would have prevented the Aug 28 16:16 escalations exactly, but not the 16:08 ones, where Prowlarr looked healthy and the heavier tvsearch failed anyway. Partial value, not yet written.
• sudo pmset -c sleep 0 remains the host-side half.

Related

2026-08-26 The Outage Was Thirty Seconds, The Damage Was Twelve Hours, 2026-08-08 The Indexer Prowlarr Never Asked FlareSolverr About.

Shipped (2026-09-01)

media-stack#61, all four checks green, 61 test cases.

The automated review caught something real, and it was the same thing twice. Both review passes independently flagged that run_search_with_retries wrapped the whole cycle, so a partial outage re-issued the healthy service’s searches up to 3x. In a stack whose entire design is about not hammering indexers, and during the exact window when indexers are most likely degraded, that was backwards. Fixed by retrying each service individually (retry_search <fn>).

The gap was in my tests as much as my code: the retry tests only ever failed search_sonarr_missing with Radarr stubbed to succeed, so nothing asserted how many times the healthy service got called. Wrote that assertion, confirmed it failed against the old code (Radarr 3, expected 1), then fixed. Same lesson as the sweep_service “silent success” from #59: the bug lived in the case no assertion looked at.

Also renamed SERVICE_READY_* to INDEXER_RECOVERY_READY_* on review. The reviewer was right that it broke the surrounding namespacing; I had reasoned myself into a generic name because the gate serves both the sweep and the search, and lost greppability for it.

Three self-inflicted friction points worth remembering:

• shfmt reformats single-line shell functions into multi-line, so a str.replace("f() { ...; }") against an already-formatted file is a silent no-op. My script printed “ok” from an unconditional print, not from an assertion, so it looked applied. Assert on the replacement, not on reaching the end of the script.
• macOS sed -i needs an empty backup arg (sed -i ''), otherwise it eats the next token as a filename.
• constants.sh marks the service URLs readonly, so a live experiment cannot override SONARR_URL to simulate an outage. Overriding SONARR_API_KEY works and is the better probe anyway, since it exercises the real failure path.

Still open

• Rotate the Prowlarr API key. Carried over from Aug 26, still not done.
• Not deployed. Needs merge, then just deploy to bump the flake input.

Deployed (2026-09-01)

just deploy bumped the flake input 50e1668 -> 1fb6d55; vendored scripts/* verified sha256-identical to HEAD. Dotfiles lock committed and pushed.

The deploy did not actually deploy, at first. hm vendored the new scripts to disk, but missing-search kept running the old logic: its compose definition was unchanged, only the contents of its bind-mounted ./scripts, so docker compose up -d had no reason to recreate it. The banner gave it away, no Retries: line. A --force-recreate fixed it.

Worth generalising: any change confined to a bind-mounted script deploys to disk but not to the running process. That is every setup-*.sh, common.sh, constants.sh, and both search/recovery scripts. just deploy will silently under-deploy them. Not fixed, and it will bite again.

The fix validated itself in production within the hour, unplanned. At 03:54:48 a real blip hit: four Prowlarr indexers failed DNS/SSL inside 33ms and Sonarr escalated all four. The deploy restarted the stack at 03:58. The recreated missing-search ran its startup sweep at 04:11 and cleared everything on the first attempt, 5 Prowlarr / 4 Sonarr / 3 Radarr. Under the old code that exact sequence produced three could not reach the indexer test endpoint failures and a 6h wait. Every non-TPB indexer is now escalation 0 with no DisabledTill across all three services, which is cleaner than at any point in the session.

Found the source of my own wrong hypothesis. Sonarr renders Prowlarr’s 429 as:

[Warn] Torznab: API Request Limit reached for Nyaa.si (Prowlarr). Disabled for 00:01:00

“API Request Limit reached” is Sonarr’s wording for any 429, including the one that actually means “this indexer is disabled till X due to recent failures”. The same phrasing is quoted in the Aug 26 entry, which is where I first read it as rate limiting. The log message is actively misleading and will mislead again; worth adding to AGENTS.md next time that file is touched.

Closed

• Prowlarr API key rotation: dropped. User’s call, no longer tracked.