2026-09-05 The LiteLLM Cost Dashboard Was Half the Bill

What I set out to do

Work out why SigNoz’s LiteLLM “Total Cost” panel showed ~$8forthelastsixhourswhentherealOpenRouterspendfeltcloserto$18.

What I actually did

Started by assuming a costing bug, since there is precedent for exactly that: the enrichment-hook cache-pricing bugs (fixed 2026-07-26) and the streaming-cost promotion bug (fixed 2026-07-27). Ruled it out instead.

• Confirmed the panel query is honest — sum(gen_ai.cost.total_cost) over litellm_request spans returns $8.39, the same number the dashboard renders.
• Reconciled a stratified sample of 122 spans against GET /api/v1/generation?id=<gen_ai.response.id>. Ratio 1.00 for every model. Every span that exists is priced to the digit.
• Found real ground truth in GET /api/v1/auth/key, which returns usage_daily / usage_weekly / usage_monthly on the plain key — no management key needed, unlike /api/v1/activity which 403s. Today: $\mathrm{17.61.}And‘monthly-daily‘=$0.476 against $0.466 of spans for Sep 1-4, so prior days reconcile exactly. The problem was today only.
• Ran controlled billing tests: direct-to-OpenRouter, litellm /v1/chat/completions, and litellm /v1/codex/responses streaming and non-streaming. All ratio 1.000. No duplicate upstream calls anywhere.
• Read the gap twice an hour apart. It sat at $9.16 both times while both sides grew, so it was a fixed lump of lost data, not an ongoing multiplier.
• Counted requests instead of dollars. LiteLLM’s uvicorn access log recorded 2205 POST /v1/codex/responses since the proxy started; SigNoz had 1228 spans. 977 requests, 44%, never landed.
• The cause was sitting in ~/.local/share/litellm/stderr.log the whole time: StatusCode.RESOURCE_EXHAUSTED ... grpc: received message larger than max (17184491 vs. 4194304). 135 batches permanently dropped, ~2.8 GB of payload.

Fixed in 0229382: raised the ingester’s receivers.otlp.protocols.grpc.max_recv_msg_size_mib to 32 in patch-otel-config.py, and capped OTEL_BSP_MAX_EXPORT_BATCH_SIZE at 16 in the litellm wrapper. Verified after hm switch with 16 concurrent requests carrying ~1MB bodies: 16/16 spans landed, max stored body 1,008,108 bytes, zero export errors, span cost within 1% of the OpenRouter credits delta.

Then added an alert for the recurrence, which turned out to be a bigger job than “add an alert”. The failure has no server-side witness, so there was nothing in SigNoz to alert on — litellm’s stderr had to be ingested first. Two wrinkles: /hostfs inside the ingester is the Docker Desktop Linux VM’s root, not the Mac’s filesystem (/hostfs/Users exists and is empty, which is a good way to waste ten minutes), so the log directory needed its own explicit macOS bind mount; and the filtering has to happen in the receiver’s own operators, because a filter processor sits on the whole pipeline and would have eaten every other service’s logs. Shipped in d83b18e: a filelog receiver keeping only exporter lines, tagged otel_export_result=dropped or =retry, and a logs rule firing on any drop in 5m. Verified by appending a probe line to stderr — non-exporter lines were correctly filtered out, and the rule went firing about 90 seconds later.

What was striking

The loss was invisible on the server side by construction. gRPC rejects an oversized message below the receiver, so otelcol_receiver_refused_spans never moves and the collector logs nothing. The only witness is the producer’s stderr. Every instinct said “query the observability system to find out what the observability system is missing,” and that was exactly the thing it could not tell me.

The drops skewed expensive. Span size and request cost both track context length, so the batches that blew the 4MiB cap were disproportionately the 240k-token codex turns. Losing 44% of requests cost 52% of the dollars. A sampling assumption would have led me astray here — the missing data was not a random sample of the present data. Max body size actually stored was 366KB, which in hindsight was the tell: everything bigger was systematically gone.

Cheap ground truth beat clever inference. I burned a while on credit-delta experiments that kept returning noise, including one reading of “exactly 2.00x” that was pure contamination from background agent traffic and nearly sent me chasing a double-billing theory. The thing that actually cracked it was counting lines in an access log.

“Add an alert” was really “make the failure observable at all.” The instinct is to reach for a threshold on an existing signal. There was no existing signal, and there could not be one, because the whole failure mode is that the telemetry system never learns it lost anything. Most of the work was building the witness; the rule on top was ten lines.

This started because of a config change I made three hours earlier — feat(codex): enable auto_review with glm-5.3-flash review_model. The setting was fine. It just raised traffic past the point where a latent default became load-bearing.

Related

Homelab Services Architecture, OTel Metric Temporality, Some OTel Feature Gates Are Permanent