2026-08-16 A DNS Outage Fakes a 2137-Path Source Build
What happened
Routine update -c (flake update → check → hm switch) died in the build phase, same opening shape as 2026-07-24 langfuse wrapt Pin Blocks hm switch After Flake Update and 2026-08-04 Obsidian’s DMG Layout Change Blocks hm switch. This time the config was never broken.
DNS dropped partway through. Nix could not resolve cache.nixos.org and could not connect to nix-community.cachix.org, so it disabled both substituters for 60 seconds and recomputed the build plan as if no binary cache existed. What should have been a trivial nixpkgs bump became [1/498/2137 built] building llvm-project-5. The chain: with no substituter, pre-commit 4.5.1 had to build from source, and nixpkgs wires dotnet-sdk in as one of its language check inputs, which drags in swift → swiftpm → llvm-project. The build then died fetching swift-crypto over the same dead network.
The misleading error
The fatal line was fatal: could not read Username for 'https://github.com': Device not configured. That reads like a credentials problem and invites a hunt for a token that was never involved. It is just git’s phrasing when the host is unreachable at all. The tell that it was network and not auth sat far above the build errors, in the Could not resolve host warnings and disabling binary cache ... for 60 seconds.
The fix
There wasn’t one to make. With the network restored, nix build --dry-run --no-link ./nix#homeConfigurations.aarch64-darwin.activationPackage planned 14 derivations + 25 substitutions — no llvm, no swift, no dotnet, no pre-commit from source. Two orders of magnitude smaller than the outage-era plan.
nix flake update and nix flake check had both already succeeded before the outage, so the lockfiles were valid the whole time. Re-ran hm switch: exit 0, containers healthy, activation clean. Committed the nixpkgs bump (6b5e5b7 → 8be7bd0, both root and nix/ flakes) as 4a54ba1.
What was striking
The failure mode is dangerous precisely because it is so legible. A 2137-path plan naming llvm, swift, and dotnet looks exactly like a real dependency regression, and every instinct says start pinning inputs or patching derivations. Acting on that plan would have meant “fixing” a config that had nothing wrong with it — and the fix would have been permanent while the outage was transient.
The reusable rule: before diagnosing any absurd from-source build plan, scroll up past the build errors and look for substituter resolution failures. If they are there, restore the network and re-run the dry-run before theorizing. Captured to project memory as reference_nix_dns_outage_phantom_source_build.
A worthwhile side observation on .gitattributes: nix/flake.lock -diff deliberately collapses lock diffs, so git diff --stat reports the file as Bin 19842 -> 19842 bytes. That is intentional noise-suppression, not corruption — git diff --text shows the real three-line change.
Follow-up: the stdenv.isDarwin straggler
The same nix flake check kept emitting evaluation warning: stdenv.isDarwin is deprecated even though 3d5d159 had migrated all 51 repo uses to stdenv.hostPlatform. I assumed a flake input; wrong. It was nix/tests/capability-flags.nix:28 — 3d5d159 simply never covered nix/tests/.
The technique that found it in one shot: the warning comes from builtins.warn, so nix flake check ./nix --show-trace --option abort-on-warn true converts it into an aborting stack trace naming the exact call site. Far better than grepping a needle that could have lived in any input’s tree. Fixed in 6bf12a4; the check now exits 0 under abort-on-warn, which proves no evaluation warning fires anywhere, not just this one.
Follow-up: auditing prek project discovery
rg had been erroring on nix/.pre-commit-config.yaml, a tracked symlink into a garbage-collected store path, swept into git by 0c95ea2 alongside the two project configs that commit meant to add. It turned out to be inert rather than broken: prek does not discover nix/ as a project, and nix/scripts/prek-update already skips it under a dedicated regression test, test_dangling_symlink_config_does_not_abort. Dead weight, not a bug.
The audit did surface a real gap, though — the same class 0c95ea2 fixed, recurring. prek list reported 8 projects where nix/projects/ holds 10. permission-suggestion (47 tracked files, src/ and tests/) and obsidian-mcp-proxy had no .pre-commit-config.yaml at all. Their Python was still linted, since the root hooks’ ruff reaches into nix/projects/, but the project-scoped gates — pytest-unit, ty, uv-lock, commitizen — never ran on them.
Fixed in 97a6aea: dropped the dead symlink and gave permission-suggestion a config. obsidian-mcp-proxy stays as-is on purpose — it holds only pyproject.toml and uv.lock, repackaging an upstream binary with no first-party source to gate.
The template could not be copied verbatim. All 8 sibling configs are byte-identical and gate on uv run pytest -m unit, but permission-suggestion declares no markers, so that selector deselects all 233 tests and pytest exits 5 — the hook would have failed every push. Its suite runs in ~2.5s, so its hook runs unfiltered instead; everything else matches the siblings. The project is not copier-managed (no .copier-answers.yml), which is why it was born without a config in the first place.
The prek discovery rule I had was wrong
Adding the config did not make prek list see the project. Not when staged, not after committing it to HEAD — and flake.nix presence, which cleanly separates the 8 discovered projects from the 2 undiscovered ones, turned out to be coincidence rather than cause. prek list caches the workspace scan. prek list --refresh immediately reported 9. prek’s own error text names the flag; I burned three wrong hypotheses before reading it.
This matters because it makes the control itself unreliable. My note said “compare prek list against ls nix/projects/”, and a stale cache makes that comparison lie in the reassuring direction — it under-reports exactly when you have just added the thing you are checking for. Memory corrected to always pass --refresh.
Verified after the fact: just check now shows ✓ nix/projects/permission-suggestion with ty passing, a full prek run --all-files inside the project passes every hook with zero files modified, and the pre-push pytest gate passes. I had warned that tracking a new config would drag the project into yamlfmt/ruff scope with a batch of complaints, as happened in 0c95ea2. It didn’t — the project was already clean.
Closing the deviation: markers from layout
Rather than leave permission-suggestion with a one-off hook, brought it fully in line with its siblings (0ada974). The interesting part is that the sibling convention is not hand-applied decorators at all — claude-ops/tests/conftest.py derives the marker from the test’s directory in pytest_collection_modifyitems, and its docstring gives the reason: “a hand-applied marker that someone forgets silently shrinks the suite instead of failing. Deriving from the directory keeps -m unit equal to the contents of tests/unit/ by construction.” A gate that quietly narrows is worse than one that breaks, and this makes narrowing structurally impossible.
So the fix was layout, not annotation: the 14 unmarked modules moved to tests/unit/ beside the existing tests/integration/, plus the same conftest hook and the two marker definitions. It was safe to move them because the suite has no intra-test imports and the shared fixtures sit in tests/conftest.py, which still applies to subdirectories. Git recorded all 14 as renames.
-m unit now selects 223, -m integration 10, together exactly the 233 — a complete, disjoint partition with nothing left unmarked, which is the property worth checking after this kind of move. The config is byte-identical to its eight siblings again. ruff-format did reformat the conftest signature on the first just check, since this project’s line length differs from claude-ops; restaging and re-running cleared it.
Ten of ten
Gave obsidian-mcp-proxy the config too (f61a3f6), overriding my own recommendation to skip it. I had argued a dependency-only project has nothing to gate, which was the wrong frame: it buys check-toml, gitleaks, and above all uv-lock. For a project whose entire purpose is that the lock is the pin — it exists because uvx mcp-proxy used to resolve at launchd start time and an upstream release could break the service with no rebuild — a hook that catches a pyproject.toml edit which never reached uv.lock is precisely the gate that matters.
No deviation was needed, and the reason is the same mechanism that bit permission-suggestion from the other side: the source-assuming hooks filter on types: [python], find nothing, and skip. So ruff skips and the pre-push pytest -m unit gate skips rather than exiting 5. Byte-identical to its nine siblings.
prek list --refresh now reports ten projects against ten on disk. The gap that opened this thread is closed.