Achhina's Digital Garden

2026-07-22 OWUI Usage Double-Count After merge_usage Landed

5 min read

2026-07-22 OWUI Usage Double-Count After merge_usage Landed

What I set out to do

Close out the second of two upstream issues that check-upstream-issues had flagged as resolved. The first, Home Manager [#9585], was handled earlier (see the companion work below). This entry is about open-webui [#25617] (“analytics undercounts token usage for streaming native tool-call loops”). The question was narrow: now that upstream fixed it, is our cost_usage_filter.py workaround safe to remove?

What I actually did

The answer turned out to be worse than “safe to remove” — the workaround had become actively harmful, silently inflating billing-visible analytics.

The mechanism. Our filter’s stream() half rewrote each tool-loop round-trip’s usage chunk to the running per-turn total. That was correct only while OWUI persisted the LAST chunk it saw. But upstream’s fix (utils/response.merge_usage, called at middleware.py:4176) now sums every round-trip’s usage into the stored object itself. Crucially, line 4173 reads data['usage'] — the very chunk our stream filter already mutated at line 4057 — so upstream sums our running totals a second time. For an N-call turn the stored value becomes the weighted over-sum N·P1 + (N-1)·P2 + ... + PN instead of P1 + ... + PN.

Confirmed with live data, not just the code path. I read the deployed middleware.py straight out of the image, then queried the OWUI SQLite volume read-only via a throwaway keinos/sqlite3 container (chat usage is nested in the chat JSON blob at $.history.messages[].usage, not the empty message table, which is for channels). A real 3-call turn had stored 17,439 / 1,123 tokens while the outlet status line — which reads the accumulator directly and stays correct — showed the true 11,731 / 802. That is +49% prompt, +40% completion inflation in analytics. Cost was barely off because it rides the cost key (which merge_usage sums via USAGE_COST_KEYS) and cache discounts make early-call cost tiny. Single-call turns are unaffected, which is exactly why this hid.

The fix (cost_usage_filter.py v0.4.0, commit eb6a926): stream() no longer touches per-call tokens — it lets merge_usage do the summing — and only feeds the outlet’s status-line accumulator plus normalizes each call’s cost onto the cost key (needed because merge_usage sums cost, not response_cost). Removed the @upstream-issue #25617 tag from the filter and from open-webui.nix, since the workaround is retired (our PR [#25659] closed unmerged in favour of upstream’s approach).

Tag hygiene. The image-cost patch (owui_image_cost_patch.py) also carried a #25617 tag, but that issue’s body is purely about token undercounting — it never covered the discarded x-litellm-response-cost image header or the generated-image double-display dedup the patch handles. Leaving the tag would make check-upstream-issues flag a still-needed workaround as removable now that #25617 is closed. I did an exhaustive gh issue search (asked to confirm, and to never file): nothing tracks either concern. Closest were [#15411] (headers dropped, but the non-streaming chat path, closed) and [#26746] (cumulative cost to filters, closed as duplicate). So I removed the tag and noted “not tracked upstream.”

What was striking

The outlet status line and the stored usage read from different sources — accumulator vs merge_usage — so on any multi-call turn they form a free built-in ground-truth diff. The status line was right all along; the discrepancy was sitting in the DB waiting to be measured. Also: “upstream resolved it” is never the end of the analysis. A workaround for a fixed bug isn’t automatically inert — here the fix and the workaround stacked into a new bug.

Follow-up: backfilling the inflated records

Went to backfill the already-stored inflated usage and found the blast radius was far smaller than I’d estimated. Enumerating every multi-call assistant message read-only (throwaway python:3-alpine against the volume) and diffing stored prompt tokens vs the status-line ground truth showed only 1 of 15 multi-call turns was actually inflated — 3a8c4dd2/254e984e, +48% prompt. The other 14 had stored == status line exactly.

That fits the mechanism: the double-count only happens when our running-total rewrite and upstream’s merge_usage are both live. The 14 correct turns predate merge_usage landing in our image (our workaround alone was doing its job — OWUI kept the last = running-total chunk); only turns from the window after the fix shipped and before v0.4.0 got double-counted. My earlier “~14 chats inflated” guess was wrong, and the read-only diff caught it before I touched anything.

Corrected that one record from the turn’s own status line (11,731 in / 802 out / $0.068181): set prompt/completion/total/input/output/cost, and dropped the prompt_tokens_details / completion_tokens_details sub-objects — they were over-summed by the same non-uniform weighting and can’t be reconstructed from the status line, so leaving them would show a misleadingly-inflated cache breakdown. Usage lives only in chat.history.messages[mid].usage (the messages[] list held a stub, chat_message table was empty). Stopped OWUI, backed up two ways (in-volume + host copy), patched from an inspectable script file (inline heredoc DB-writes get classifier-blocked), verified through the restarted healthy container, then deleted the backup once confirmed.

Graph View