2026-08-20 LiteLLM Patch Stack Rebase onto 1.97.0

What I set out to do

Run the routine update -c flake bump. It aborted mid-way: the nixpkgs bump moved litellm to 1.97.0 and python3.14-litellm failed to build with 1 out of 6 hunks FAILED on litellm-25240-responses-custom-llm-provider.patch. The goal became getting hm switch green again without silently dropping any of the eight local litellm patches.

What I actually did

The visible failure was only the first domino. litellm 1.97.0 reflowed its entire source to a longer line length, so multi-line call sites collapsed onto single lines and most patch hunks either failed outright or landed with fuzz. Rebased the whole stack: unpacked python3Packages.litellm.src into a scratch tree, applied the patches in the order litellm/default.nix lists them, hand-edited every reject, and regenerated each patch as a clean diff -u against pristine source. Deliberately re-did the fuzz-applied hunks too, since fuzz means patch discarded context and the next bump could land them somewhere wrong.

Two patches changed shape rather than just position:

• litellm-16021-streaming-preserve-cost shrank from eight hunks to two. Upstream 1.97.0 now carries the chunk-builder half itself (UsagePerChunk.cost, the "cost" key in the usage helper, the setattr(returned_usage, "cost", cost) assignment). Only the stream_chunk_builder promotion onto _hidden_params["response_cost"] and the streaming_handler dict-usage branch still need patching.
• litellm-responses-tools-name had widened the function branch to a catch-all else. Upstream now has its own custom branch (convert_custom_tool_to_function_tool) and a drop-list for Responses-only tool types, both better than what the patch did, so the rebase keeps them and only widens the branch condition to accept the nested-function dialect plus typeless named tools.

Then rewrote the installCheckPhase markers in Dotfiles Host Profiles via Private Flake’s litellm module: the three markers for the now-upstreamed hunks were flipped to assert pristine source, so a future revert fails the build loudly instead of quietly regressing telemetry, and the streaming-iterator marker was retargeted at upstream’s new _model_id_from_metadata helper.

With every patch applied cleanly the build went green, but the launchd agent crash-looped. 1.97.0’s proxy extra added expression>=5.6.0, which nixpkgs does not package and the proxy imports eagerly via key_management_endpoints. Declared it in pythonForLitellm’s packageOverrides (hatchling, PyPI sdist, pydantic in nativeCheckInputs because the tests import it) and appended it to litellmPython.withPackages.

What was striking

A green Nix build proved nothing about the runtime here. The missing expression module was invisible until the agent actually started, which is the same lesson as Verify tool output before claiming success in a new costume: the build gate covers patch application, not dependency closure.

The other thing worth remembering is that upstream had adopted half of one patch. Blindly reapplying the old hunks would have duplicated the fix. Reading the pristine source before rebasing each hunk turned a mechanical conflict resolution into a real diff review, and the pristine-asserting markers make that knowledge survive into the next bump.

Verified end to end rather than at the build boundary: /v1/models lists 429 models with no leaked wildcard literal, a streamed completion returns usage.cost = 4.35e-6, and the matching litellm_request span in SigNoz reports gen_ai.cost.total_cost = 4.35e-6 rather than the zero that motivated the cost patches in the first place.

Top 3 tomorrow

1. Commit the rebase (patches + module + the two lock files the aborted update never got to commit).
2. Re-run update -c so the --cleanup GC pass that never ran actually happens.
3. Re-check litellm PR #25309 and issue #16021 on the next bump, since more of the local stack may have landed upstream.

Related

Dotfiles Host Profiles via Private Flake