2026-07-01 IPython Config Test Matrix in Nix Flake Checks

Context: Local dotfiles session, direct follow-up to 2026-07-01 Dotfiles IPython Config Fixes and Claude Attribution Settings. The question: the incorrect IPython version-guard fix was only caught by running on a machine that still had IPython 7. How do we catch that class of bug deterministically in CI?

What I set out to do

Build a test harness that loads the rendered ipython_config.py under every supported IPython major (7/8/9), fails on any warning, and runs under nix flake check - TDD style, proving the harness catches both historical bugs before trusting it.

What I actually did

• pytest harness (nix/tests/ipython-config/test_ipython_config.py): initializes TerminalIPythonApp in-process against the rendered config. Fails on both warning channels: Python warnings promoted to errors during initialize(), and WARNING+ log records via a handler attached directly to app.log (traitlets apps don’t propagate to the root logger, so pytest’s caplog never sees them). Expected shell traits injected per leg via IPYTHON_CONFIG_EXPECT JSON so the test doesn’t mirror the config’s own version branch.
• Version matrix: three uv virtual projects (package = false) with committed uv.lock files pinning IPython 7.34.0 (Python 3.11), 8.39.0, 9.15.0; built via uv2nix, run as checks.<system>.ipython-config-ipython{7,8,9} derived by mapping over homeConfigurations so each check tests the exact deployed artifact.
• RED/GREEN: reverting to the pre-guard-fix config (d9312a9^) failed legs 7/8 with the historical TraitError (colors = "gruvbox-dark" invalid on <9); the pre-theming-fix state (894f0c9^) failed leg 9 with the lowercase-theme DeprecationWarning. HEAD passes all legs; nix flake check ./nix exits 0. Committed as 8c60c03.

What was striking

• IPython swallows config errors: bad config → CRITICAL log + sys.exit(1) at startup, or just a warning with exit 0 in earlier flows. A naive smoke test passes on a broken config; the harness must assert on log records and effective shell state, not exit codes. Also SystemExit escapes except Exception - the harness catches it explicitly.
• pytest fixtures choke on half-initialized IPython singletons: a failing app.initialize() inside a module-scoped fixture trips pytest’s fixture-finalizer bookkeeping (assert not self._finalizers) and buries the real error. A lazily-cached plain helper avoids the fixture machinery entirely.
• uv can express a version matrix in one lockfile (conflicting extras via [tool.uv] conflicts), but uv2nix can’t build from it - fails with “Conflict resolution selected more than one conflict specifier”. Hence one tiny pyproject+lock per IPython major.
• IPython 7.x needs Python ≤3.11 (imports imp, removed in 3.12); python310 is gone from nixpkgs-unstable, so the leg runs on 3.11.
• The locally deployed ~/.config/ipython config is stale - it still has the broken hasattr guard from before the fixes; hm switch hasn’t been run since the PRs merged.

Top 3 next

1. Run hm switch to deploy the fixed IPython config (and the attribution settings from the cloud session).
2. Consider a GitHub Actions workflow running nix flake check ./nix on a linux runner - the rendered config text is system-independent, so one leg covers all machines.
3. If uv2nix gains conflict support, collapse the three env projects into one pyproject with conflicting extras.

Related

2026-07-01 Dotfiles IPython Config Fixes and Claude Attribution Settings

Addendum

Follow-up refactor (6c74b06): the per-cell Python interpreter is now parsed from each env’s requires-python at eval time instead of being duplicated in default.nix, so every env-specific fact lives only in its env directory and the nix matrix reduces to name → expected shell state. Also clarified in discussion: a uv.lock can’t stand alone (it must pair with the pyproject it was resolved from, and one lock resolves one version per package), which is why the matrix can’t collapse further than one tiny pyproject+lock per cell until uv2nix supports conflicting extras.

Correction: uv2nix does support conflicting extras

Adam pointed at uv2nix PR #281 (merged 2025-11-25, well inside our locked rev). My earlier “uv2nix can’t build from conflicting extras” conclusion was an API misuse: mkPyprojectOverlay defaults to resolving the whole workspace, which selects all conflict branches at once; passing the selected extra as its dependencies spec resolves cleanly. Consolidated the matrix accordingly (f2469e8): one env/ pyproject with mutually exclusive ipython7/8/9 extras and a single uv.lock, one venv per extra, plus a new guard test asserting each leg’s IPython major (TDD’d red/green) since a mis-selected extra would now silently test the wrong IPython. Interpreter per leg lives in default.nix again because requires-python is project-wide. Memory note rewritten to record the correct API usage.

Addendum: prek vs git-hooks-nix drift

Root nix flake check failed on ruff-format drift in a claude-ops test file that just check had always passed. Root cause: two hook systems, two ruffs. prek delegates the claude-ops/gh-review-preview subtrees to their nested .pre-commit-config.yaml files, which pinned ruff-pre-commit at v0.6.8 (14 months stale, honestly passing), while git-hooks-nix’s sandboxed all-files pre-commit-run uses nixpkgs ruff 0.15.20, which joins implicit string concatenations. Fixed the formatting (223f01d), then just prek-update to bring nested pins current: ruff v0.15.20, pre-commit-hooks v6, mdformat 1.0, yamlfmt, codespell (which flagged one “Re-declare” in a CSS comment), commitizen (0e980bd). Both nix flake check . and nix flake check ./nix green.

Addendum: extending the check suite (uv project tests + Claude settings)

Surveyed the dotfiles for other test-worthy surfaces; picked two of four candidates (skipped tmux parse and zsh startup checks for now):

• uv project pytest suites in flake checks (nix/tests/uv-projects.nix): bash-command-validator, claude-ops, gh-review-preview, permission-suggestion (854 tests) now run hermetically from their committed uv.locks. Sandbox scaffolding needed: git + hermetic identity, real CA bundle (nix’s stub SSL_CERT_FILE=/no-cert-file.crt crashes httpx at client construction), --override-ini=addopts= (coverage/HTML reports write into read-only store cwd), and deselecting bash-command-validator’s wall-clock latency tests (flaky under build load).
• Claude settings validation (nix/tests/claude-settings/): rendered ~/.claude/settings.json (home.mutableFile source) must parse, contain no deprecated keys (the includeCoAuthoredBy class from the 2026-07-01 Dotfiles IPython Config Fixes and Claude Attribution Settings session), and keep the attribution object shape. TDD’d red against a bad fixture.
• The sandbox immediately caught a real packaging bug: hatchling only auto-detects src/permission_suggestion, so non-editable installs of permission-suggestion omitted src/evals and its four console entry points. Local uv run masks this via editable install. Fixed with explicit [tool.hatch.build.targets.wheel] packages.
• buildSystemOverrides extracted to nix/lib/uv-build-system-overrides.nix, shared by mkUvProject and the test envs.

Committed as 2b019ae; nix flake check ./nix and root both green.

Addendum: schema-driven Claude settings validation

Adam rightly objected to the hand-seeded deprecation list. Investigated runtime alternatives: claude doctor does validate settings but is a TTY-only TUI (verified empirically on 2.1.198 under a forced pty; non-interactive mode is claude-code#26487, closed as duplicate), and the headless -p path prints settings warnings to stderr but needs auth, so nothing hermetic exists. Settled on the schemastore settings schema as a file+https flake input (nix flake update bumps it): checks are now schema-derived (no deprecated-marked keys; all keys known to schema or in a self-cleaning exceptions list that fails once the schema learns a listed key). Full jsonschema document validation deliberately omitted: the schema’s permission-rule tool pattern rejects valid open-namespace tools like AskUserQuestion.

The unknown-key test immediately caught three real stale keys in claude.nix, all confirmed invalid in settings.json (agent-researched with sources): autoUpdates and theme belong in the CLI-managed ~/.claude.json (claude-code#60956, #6962), defaultPermissionMode is a wrong name whose documented form permissions.defaultMode was already set. Removed them; exceptions list ships empty. Also confirmed the HM module never validates settings (programs.claude-code.settings is a freeform JSON value; the $schema line is only an editor hint). Committed as dbcdd00.

Addendum: update script gates on flake checks

Wired nix flake check into the update script (bcc1394): after nix flake update bumps nix/flake.lock, the checks run before home-manager switch; under set -e a red check aborts the update before anything deploys. Skipped when the lock is unchanged. This closes the loop on the schema input: schema refresh and conformance check now happen in the same run. Note: the deployed update script is a store symlink, so the gate goes live at the next hm switch.

Addendum: zsh startup check

Added the zsh startup flake check (c92b995): boots the full rendered chain (stub ~/.zshenv → ZDOTDIR .zshenv → .zprofile → .zshrc, interactive login shell) in an isolated HOME, syntax-checks each file, fails on any stderr. Harness lessons: the rendered files hardcode the absolute home dir at three layers (stub source path, ZDOTDIR re-export, hm-session-vars XDG_* exports), so everything gets sed-relocated to the sandbox HOME - my first prototype silently validated the deployed files instead of the copies because of the stub’s absolute path. Also: oh-my-zsh.sh silently swaps ZSH_CACHE_DIR for $XDG_CACHE_HOME/oh-my-zsh when the dir isn’t writable (HM guarantees it via a .keep file; the harness must mkdir it), and atuin’s store-pathed init emits code calling bare atuin (goes on the check’s PATH). Only allowlisted stderr: zsh’s “can’t change option: zle” from -i without a tty. TDD’d: injected source /nonexistent into initContent → check failed with exactly that line; real config green.

Addendum: tmux config check

Added the tmux flake check (2363a7b), completing the survey’s candidate list. Design against brittleness: tmux started with -f <conf> queues config errors as client messages and exits 0 (same silent-tolerance disease as IPython and Claude settings), so the check starts a bare server and uses tmux source-file - the only channel reporting config errors via exit status. Plugin run-shell lines execute async during sourcing and swallow failures, so each store plugin script is re-run synchronously for a real exit code. One minimal did-it-apply oracle (prefix = C-a); status-line #() output and keybinding behavior deliberately out of scope. Gotcha: tmux sockets live under TMUX_TMPDIR and hit the sockaddr_un length limit with long paths. TDD’d: injected set -g definitely-not-a-tmux-option on failed the check with exactly “invalid option”; real config green. Check suite now: ipython 7/8/9, four uv project pytest suites, claude-settings (schema-driven), zsh startup, tmux.

Wrap-up

Deployed and verified with hm switch: the fixed IPython config (class_traits guard live in ~/.config/ipython), the cleaned Claude settings (stale keys gone from the deployed file), and the update script’s flake-check gate. Pushed through bcc1394; the zsh (c92b995) and tmux (2363a7b) checks are committed locally, not yet pushed.

Final state of the check suite (nix flake check ./nix, gating update before hm switch):

• ipython-config-ipython{7,8,9} - rendered IPython config under each supported major, warnings-as-failures, one uv.lock via conflicting extras
• {bash-command-validator,claude-ops,gh-review-preview,permission-suggestion}-tests - ~850 pytest tests, hermetic via uv2nix
• claude-settings - schema-driven validation against the pinned schemastore schema (flake input, bumped by nix flake update)
• zsh-config - full startup chain in isolated HOME, stderr-clean
• tmux-config - source-file exit codes + synchronous plugin runs + prefix oracle

Recurring theme of the day: every consumer tested (IPython, Claude Code, tmux, zsh-under-omz) tolerates broken config silently, so each check had to find the one channel with real failure semantics (live shell state, log records, schema keys, source-file exit codes) rather than trusting startup exit codes. Second theme: hermetic harnesses keep finding real bugs - the permission-suggestion packaging gap, three invalid Claude settings keys, the prek/nixpkgs formatter drift.

Top 3 next

1. Push c92b995 + 2363a7b.
2. GitHub Actions workflow running nix flake check ./nix on a Linux runner (rendered configs are system-independent, so one leg covers all machines).
3. Consider a scheduled non-blocking lock-upgrade job (uv lock --upgrade per env + re-run checks) to surface dependency drift without unpinning the blocking checks.