Achhina's Digital Garden

2026-08-04 A Drift Check Is Only Useful If Silence Means Something

3 min read

2026-08-04 A Drift Check Is Only Useful If Silence Means Something

What I set out to do

Answer whether the update script covered copier template updates. It did not: nix/scripts/update bumps flake inputs, uv tools, Claude Code marketplaces and Neovim plugins, but nothing in it or the root justfile ever touches the eight copier-generated projects under nix/projects/. Then add a check for it.

What I actually did

Deliberately a check, not an update step. Flake inputs and nvim plugins are lockfile bumps that either build or do not, so automating them is safe. copier update rewrites tracked source and can stop on merge conflicts, which is exactly what happened earlier today: all eight projects conflicted on .envrc and flake.nix, plus a hand-fix to claude-skill-sync’s .copier-answers.yml. Putting that in the middle of an unattended run that ends in home-manager switch would be a mistake.

So get_copier_drift reads each .copier-answers.yml, compares its _commit against the newest version tag on the template remote, and groups the projects that share a gap. Tag discovery is git ls-remote --tags --sort=-v:refname, which sorts by version rather than lexically, so v1.10.0 will beat v1.9.0 when that day comes. git was already a declared runtime input, so this added no dependency. One ls-remote per template rather than per project, memoized in a plain tab-separated string because macOS ships bash 3.2 and associative arrays are bash 4.

TDD throughout, sourcing the raw script with git stubbed on PATH, the way the existing test_update_render.py does. Fifteen tests, and the suite in nix/tests/update-changelogs/ is now 56.

What was striking

The design flaw surfaced from a test-harness bug, not from thinking. My end-to-end check pointed XDG_CONFIG_HOME at a scratch repo, which meant git lost ~/.config/git/config and with it the gh auth git-credential helper. The template is a private repo, so ls-remote failed, and my 2>/dev/null swallowed it. Output: nothing.

Nothing is precisely what “no drift” looks like. And in --preview I had it print a green “Projects are on the newest template tag”. A missing credential helper would have rendered as a passing check, indefinitely.

The fix is small: a template whose tags cannot be listed reports could not list tags, drift unknown instead of being skipped. Silence now means “checked, and there is no drift”, which is the only thing that makes the green check honest. Same shape as A Null Result Needs a Control — an absent result is not evidence of absence until you have shown the measurement can produce a present one.

Worth noting the accident was load-bearing. Had my harness kept the real config home, the check would have worked on the first try, I would have shipped the silent-failure version, and the private-repo dependency would have stayed invisible until a credential expired.

Verified the deployed wrapper under env -i with PATH=/nonexistent, since writeShellApplication prefixes runtime inputs and a missing dependency would otherwise hide behind my ambient PATH. Checked the side effect (real drift output), not the exit code.

What was left

The reference doc had drifted on its own: docs/reference/update.md still described a nix-update pass over hand-pinned derivations that the script no longer performs. Corrected the pipeline list while adding the copier section.

Graph View

Backlinks