Achhina's Digital Garden

2026-08-03 Garnix Shutdown and the Darwin Strip Bug in litellm

7 min read

2026-08-03 Garnix Shutdown and the Darwin Strip Bug in litellm

What I set out to do

Read the garnix shutdown announcement and work out what it meant for this config. Nothing more than that.

What I actually did

Garnix is gone, and yesterday’s “outage” was the shutdown. The post is dated 2026-05-28: the team joined Shopify, the hosted service ended 2026-07-15, and all build artifacts were deleted that day. The codebase is open source at garnix-io/garnix-ci. cache.garnix.io still answers 502 on every endpoint, which is exactly the signature diagnosed as a transient outage on 2026-08-02 and worked around with fallback = true. It was the shutdown, 18 days late.

Removed the substituter and its public key from base.nix, rewrote three comments that were written around garnix’s 2-hour signed URLs, updated the local.trusted description in machine.nix, fixed the same comment in the manually-managed /etc/nix/nix.conf, and added a dated update block to Dotfiles Host Profiles via Private Flake’s companion ADR 0021, which named garnix as a cache that does not mirror x86_64-darwin.

No public cache is worth adding to replace it. Measured rather than guessed: probed candidates against a random sample of the live closure. cache.nixos.org hit 51/60; nix-community, numtide, nixpkgs-unfree, pre-commit-hooks, devenv, and cache.lix.systems hit 1/60 or 0/60. nixpkgs-unfree looked promising given allowUnfree = true, but its README says it only wants to cache unfree packages in future. Every miss was a uv2nix wheel or a writeShellApplication script, i.e. things no public cache will ever hold.

Then the interesting part. Asked how expensive the uncached builds actually are. 401 of 1198 closure paths are on neither cache, totalling 7.54 GB. But rebuild times were trivial (1s to 32s each; even the 3 GB litellm took 28s), because most are wheel unpacks rather than compiles.

The size was the anomaly. litellm shipped four .ico files at 808,452,160 and 538,968,128 bytes, across two store paths: 5.93 GB of the 7.54 GB. Upstream ships them at 6,387 and 9,714 bytes.

What was striking

The bisect landed somewhere I did not expect. Checked each stage in isolation: upstream GitHub source clean, official PyPI wheel clean, the wheel our derivation builds clean (53 MB), and python -m installer run with the exact pypa-install-hook command clean (115 MB installed). Every input was fine and the output was 2.9 GB, which left only the fixup phase. The build log confirmed it:

stripping (with command strip and flags -S) in .../lib .../bin

Darwin’s cctools strip (1010.6) misidentifies an .ico as an object file and rewrites it into NUL padding. Reproduced standalone:

$ cp <src>/litellm/proxy/swagger/favicon.ico strip-test.ico   # 6,387 bytes
$ strip -S strip-test.ico
$ ls -l strip-test.ico                                        # 808,452,160 bytes
$ cmp strip-test.ico <store>/.../favicon.ico                  # identical

Byte-for-byte identical to the store file, corrupt ICO header included (planes reading 0x3030, which is ASCII "00" — the tell that something was treating binary as text). Same family as nixpkgs#218712, where Darwin strip produces invalid Rust .rlibs; dontStrip = true is the accepted workaround.

Fixed with a Darwin-gated dontStrip in the existing pythonPackagesExtensions overlay. Owned closure went 7.54 GB → 1.61 GB. Icons restored to exact upstream bytes, and file now parses them as valid ICOs with real PNG sub-images.

Two process lessons. First, three hypotheses died before the right one: uv-build 0.11.28 (tested directly, preserves binaries exactly), the install step, and store corruption (nix store verify passed). Falsifying each was what narrowed it to fixup. Second, a broken tool call nearly produced a wrong conclusion — a stat -f written in BSD syntax against GNU stat returned filesystem info, which made upstream’s icons look tiny for the right reason by accident. Re-ran with ls -l before claiming anything.

Loose ends. macmon and worktrunk both reported may not be deterministic on forced rebuild. Not investigated. Also: my three garnix files were swept into commit 2faf1fc, an unrelated obsidian-mcp commit made by a concurrent session, so that history is not atomic. Left alone rather than rewriting shared history.

Top 3 tomorrow

  1. Look at the macmon / worktrunk nondeterminism.
  2. Decide whether a self-hosted attic cache on athena is worth it now that the owned closure is only 1.6 GB — the case is much weaker than it looked this morning.
  3. Consider whether other Darwin packages shipping binary assets under lib/ are silently hit by the same strip bug.

Follow-up: the nondeterministic Rust builds

Closed the loose end above. macmon and worktrunk both differ on every rebuild because rustc bakes the absolute build directory into every binary for panic handlers and Location::caller — 78 such paths in macmon, 424 in worktrunk, all <build dir>/cargo-vendor-dir/<crate>/src/*.rs. Linux sandbox builds always run in /build, so the path is constant there; Darwin runs in /nix/var/nix/builds/nix-<pid>-<random>, which changes every time.

Two severities, and I only saw the second at first:

  1. The embedded strings differ every build, so nothing ever matches.
  2. When the random component changes digit count (39 vs 40 chars, ~1 build in 4), all those strings change length together and ld64 re-pads the Mach-O header to hold __TEXT at a constant 1,310,720 bytes. __text moves from offset 6912 to 6988 and every section shifts, so 92.6% of __text bytes differ with no code regenerated.

Confirmed by correlating repeated builds: path length 40 → offset 6912 (four times), 39 → 6988; worktrunk 40 → 16184, 39 → 16568.

It is reported upstream, and the fix I landed is the one that was abandoned. NixOS/nixpkgs#401281 proposes exactly --remap-path-prefix=$NIX_BUILD_TOP and names this exact Darwin case; it was closed unmerged 2025-09-21 because rustc’s own source paths also leak and the author wanted to wait on rust-lang/rust#140778 (since closed, but as DUPLICATE, so the blocker may not actually be resolved). Meanwhile nixpkgs does ship this technique for buildRustCrate (three --remap-path-prefix calls in build-rust-crate/build-crate.nix) and has merged recent work there, just never on the buildRustPackage hooks. Home Manager is not involved at all. Notably r13y.com and reproducible.nixos.org only measure Linux, which probably explains why a Darwin-only PR lost momentum.

Landed as a shared pkgs/rust-remap-build-path.nix imported by both derivations rather than duplicating the block. Build-dir references went 78 → 0 and 424 → 0.

Necessary but not sufficient, and worth recording honestly. macmon then rebuilt bit-identically 6/6. worktrunk still fails about 1 in 6, through a different mechanism the flag cannot reach: LLVM packs the anonymous constant pool (_anon.<hash>.<n> in __const) in a different order between runs, shifting ~4% of the binary with __text offsets identical. Fixing that would need codegen-units = 1, a real build-time and optimisation trade-off. Stopped there and wrote the limitation into the comment rather than letting it read as a clean win.

Process note worth keeping. I nearly published a wrong root cause twice. Once when a nm comparison that only looked at the name column made symbol addresses look identical (they differ across 5,392 lines, uniformly shifted). Once when a glob for *worktrunk*.check picked up a stale pre-fix artifact and I diffed the wrong pair — the giveaway was 424 build-dir refs in a binary that should have had zero. Both times the tell was a result that contradicted something already established.

Commit hygiene incident. A concurrent session staged its tmux/statusline work into the index between my git add and git commit, so my commit swallowed six unrelated files. Caught it in --stat, and since that commit was the only unpushed one, fixed with a soft reset and a git commit --only on my three files. Verified all six were preserved; the one that differed had simply grown 71 → 105 lines because the other session kept working. Lesson: in this repo the index is shared state, so git commit --only <paths> is the safe form.

Graph View

Backlinks