2026-07-26 LLM Cost Accounting Audit - Enrichment Hook Fix and the claude-ops Double Count
What I set out to do
Update claude-ops pricing for newly released models (Opus 5), then audit whether the LLM cost data captured in SigNoz (both litellm spans and Claude Code OTEL) is actually correct.
What I actually did
- Added Opus 5 pricing (25) to the claude-ops
token_costview with TDD (commitee906fd0). - Audited litellm
gen_ai.cost.*span attributes against real OpenRouter per-provider pricing and found two bugs in the enrichment hook: cached tokens billed $0 (OpenRouterinput_cache_readnever mapped tocache_read_input_token_cost) and prices frozen at proxy startup (refresh loop only filled missing keys). Fixed both with 8 new tests, deployed viahm switch+ proxy restart, verified end-to-end (commit5b0843d). - Investigated whether upstream litellm makes the hook redundant: partially. litellm 1.89.0 always sends
usage: {include: true}to OpenRouter and prefers the provider-reportedusage.cost— but only on non-streaming responses. The streaming reassembler preserves the cost ontousage.cost, yet the cost calculator never reads it (open PRs #16162 and #26870 address exactly this; #22519 was closed unmerged). Since proxy traffic is essentially all streaming, andgen_ai.cost.*breakdown attributes are always computed from the model-cost map anyway, the enrichment hook stays. The bundled map covers only 96 of ~2.6k OpenRouter slugs. - Investigated the apparent ~50% loss of Claude Code spans/logs in SigNoz. The “loss” was mostly a measurement artifact: Claude Code JSONL writes one assistant entry per content block, all repeating the same
message.idand usage, and claude-ops counts rows instead of messages. 7,695 rows collapsed to 3,420 real requests over 7d.
What was striking
The denominator bug inverted the conclusion entirely. Measured against distinct message ids, SigNoz capture is ~90-100% (spans and logs at or above 100% since they include WebSearch/haiku requests that never write JSONL; the cost metric at ~90%). The only real gap was a single 10-minute window on 07-22 where two parallel burst sessions produced 81 requests and zero spans before export recovered — and the collector was provably up because open-webui data flowed throughout. The bigger consequence: claude-ops itself, the tool trusted as the authoritative spend record, overcounts tokens/cost ~2.3-2.7x (7d spend was really ~1,273). Fixed the following day (commit 27d3855): the token_usage view now dedupes to one row per API message id, keeping the earliest entry so fork/resume transcript copies attribute to the originating session. Also corrected the enrichment hook’s stale upstream reference and machine-tagged BerriAI/litellm#16021 there so check-upstream-issues watches the streaming-cost fix at both sites (commit a2a41d7).
A follow-up on 07-27 caught a third face of the same theme in the SigNoz Claude Code dashboard: the “Token Distribution by Model” pie disagreed with the input/output/cache-read/cache-write total cards. Commit fa257d5 (April) had moved all single-number totals to ClickHouse sum(max per fingerprint) because SigNoz’s increase aggregation undercounts 15-27% (first-bucket-per-fingerprint drop), but the pie was left on the builder increase query — and a pie is a single-number total per group, so it undercounted unevenly per model (opus-5 -12% through sonnet-5 -67% over 7d). Converted the pie to the same SQL pattern grouped by the model label; by-model and by-type now both sum to the identical 1,090,810,637 tokens (commit 91cf8fc).
Chasing the dashboard’s three warning banners afterwards surfaced one more silent numeric bug (commit 9817036): “Cost Leverage Over Time” divided per-bucket increase by constants sized to its pinned 1m/5m/15m stepIntervals, so when SigNoz clamped the step to 9000s at wide ranges the plotted leverage inflated 150x. Fix was switching to rate divided by the plan’s per-second cost, which is interval-independent by construction. The same commit collapsed “Leverage Over Time” to a single auto-interval A/(B+1) series (amending repo ADR-0011’s multi-window design, which degenerated to identical clamped series anyway) and qualified every dashboard service.name filter as resource.service.name to silence the ambiguity warning.
The LiteLLM dashboard’s “completely incorrect” cost turned out to be real underreporting at the emission layer, not a query bug. Reconciling every one of the day’s 333 resolvable requests against OpenRouter’s /api/v1/generation endpoint: OpenRouter billed 2.29 and the metric channel $2.32 (~25% under). Two mechanisms: ~1 in 5 requests emits gen_ai.cost.total_cost = 0 despite full usage attrs (codex-path models drop 25-31%, glm on the chat path only 4%, and requests finishing with tool_calls never drop — 158/158 costed), and OpenAI implicit cache-write surcharges (1.25x input on OpenRouter) are invisible to litellm since implicit caching reports reads only. The good news buried in it: post-enrichment-fix, map-based span costs match OpenRouter’s bill to three significant figures whenever the data is present, so 5b0843d is verified working per-request. Documented the quantified caveat on the dashboard’s Total Cost panel (commit 293fb80) and saved the reconciliation method (generation-id lookup with the standard API key; /activity needs a management key).
Then dug into the cost=0 drop itself and fixed it (commit 83abc7c). A standalone A/B repro against the deployed nix litellm (plain stop vs forced tool_calls, with a stack-tracing monkeypatch on set_cost_breakdown) showed the two cost paths were mutually exclusive: on stop-finish streams our streaming-cost patch promoted OpenRouter’s usage.cost into _hidden_params["response_cost"], which made litellm’s logging-level calculator early-return and skip the map calculation, leaving the all-zero cost_breakdown that gets stored at stream start on an empty wrapper; on tool_calls streams the promotion never fired at all (it only existed on stream_chunk_builder’s simple-text fast path), so provider cost was silently discarded in favor of map pricing. The “tool_calls never drops cost” discriminator in production data was exactly this fork. New companion patch litellm-16021-provider-cost-breakdown.patch promotes on the general assembly path too and makes response_cost_calculator run the map math for the breakdown, then override the total with the provider figure. Verified end-to-end: a streamed request through the rebuilt proxy exported gen_ai.cost.total_cost equal to OpenRouter’s billed 5.175e-05 to the digit. Pre-fix ClickHouse data keeps the undercount; remaining structural gaps are aborted streams and requests where OpenRouter omits usage.cost (map fallback). A follow-up cold-cache experiment actually disproved my initial “cache-write surcharges invisible” claim: OpenRouter’s streamed usage.cost includes the 1.25x write premium (billed streamed span cost to the digit), so surcharges are captured whenever provider cost arrives. Same session also caught the check-upstream-issues scanner false-flagging litellm#25240 as resolved — it was stale-closed not_planned with fix PR #25309 still open, so the patch stays and the machine tag became a prose note (commit 30e6414). Day’s cache economics for perspective: warm-read savings 0.18 on a $3.05 billed day.