2026-09-03 The Histogram Had One Series Per Request

What I set out to do

Rework the LiteLLM Proxy dashboard in SigNoz: Number of Requests as a bar chart, Token Usage as a stacked bar split by cache read / cache write / input / output, Total Cost as the first hero tile, Latency P95 next to the latency distribution, and a time-to-first-token panel.

What I actually did

The layout half was mechanical: a Python script over the provider-shaped JSON in nix/home-manager/files/dashboards/, cloning structures from panels that already round-trip through tofu. Three sections now: Traffic (cost hero first, then the two bar charts and the error list), Models & Cost, and a new Latency section holding P95, TTFT P95 by model, and the distribution histogram. One trap: the file is insertion-ordered, not key-sorted, so sort_keys=True would have rewritten all 1,200 lines.

The token split had to be raw ClickHouse. Cache counts exist only inside metadata.usage_object (litellm’s Python repr of the usage dict); gen_ai.usage.details.cache_read_tokens is declared in the field catalog but set on 0 of 713 litellm_request spans over 7 days. SigNoz gives ClickHouse panels the window timestamps but no interval variable, so the bucket is derived from the window (greatest(60, window/96) seconds) for a fixed bar count at any range. Over 7d: 4.90M prompt tokens of which 1.88M were cache reads, 0 cache writes on OpenRouter, 433K output.

TTFT was the real work. LiteLLM has no first-token span attribute (the bare ttft_ms and cache_read_tokens attrs in the catalog belong to the claude-code service). It records gen_ai.server.time_to_first_token as an OTel histogram, and SigNoz listed it as live. Yet p95 on the .bucket returned nothing with millions of rows scanned, and increase on the .count was 0 at every daily step for 30 days, while the .max gauge showed fresh values per service.instance.id. ClickHouse settled it in one query: every fingerprint of the .count had min=1, max=1, hundreds of fingerprints per day for ~100 requests. hidden_params (per-request response_cost) and metadata.requester_metadata (per-request x-request-id and traceparent) were metric labels, so every request minted its own series, observed once, then re-exported every 5s forever. A histogram quantile is a rate over buckets; a series that never increments has no rate.

2026-09-01 The Rename Was the Visible Half had already named these two labels as the cardinality source and left dropping them as a “worth doing, not on the critical path” item, framed as a collector-side drop. Today it moved onto the critical path, and the cleaner place turned out to be the emitter: litellm 1.97.0 validates callback_settings.otel.attributes.exclude_list (a top-level proxy config key, not under litellm_settings, and names must be in its VALID_METRIC_ATTRIBUTE_NAMES or the callback raises). Added to the litellm module, hm switch, then three streaming requests to a :free OpenRouter model. The first post-restart series carried only model/system/framework plus constant metadata labels; after the third request the count read 3 and the dashboard’s exact p95 query returned a value.

What was striking

The metric list lied by omission. signoz_list_metrics showed the TTFT histogram as active in the last hour, and it was: the stale series were being re-sent every 5s. “Active” meant “exported,” not “recording.” The .max gauge was the only signal that observations were still happening, and only a per-fingerprint min/max over the samples table said why the aggregate was blind.

A null result from the MCP quantile path needed a control. p95 on the sibling operation.duration.bucket was also null, which briefly pointed at the query shape rather than the data. Both were null for the same reason, so the control confirmed the cause instead of exonerating the tool. The control was still right to run.

The fix that costs nothing at read time is the one at the emitter. The 09-01 plan was a collector processor; an exclude list in the proxy config is one line, is validated at startup, and cuts the ~1.1B samples/week that entry measured. Old series stay unreadable by the builder panel and the description on the panel says so, with the date.

Second pass: the p95 was a constant

The panel rendered, but as a flat 4.75 s for two models whose real first-token times were 1 s and 3 s. litellm registers the histogram with no bucket advice, so the OTel SDK defaults apply (0, 5, 10, 25, 50 … seconds); every sub-5 s observation lands in the first bucket and the quantile interpolates to the same 95% of 5 s forever. Switched the panel to mean TTFT (increase(sum) / increase(count)) per model, which reads 2.6 to 3.7 s for glm-5.3-flash across the evening’s real traffic, and turned on points so a single bucket still draws. A real quantile needs semconv-shaped buckets via a metrics View in the instrumentor; parked. Also fixed the Requests legend (count() to Requests) and coloured the token stack grey / dark grey / yellow / red.

Third pass: buckets from the outside

Asked whether the exclude was the right fix at all. It is the fix upstream documents for the v1 callback we run, and their list is longer than mine, so the module now carries all seven names. The by-construction answer is litellm’s v2 OTel path, whose METRIC_ATTRIBUTE_CEILING refuses per-request labels before any operator filter and bounds hidden_params to the deployment id; but v2 is experimental, cannot coexist with v1, and renames the LLM span to chat <model>, which five panels and the empty-completion alert filter on by the old name. A migration for another day.

The bucket problem had a cleaner answer than expected. litellm’s _get_or_create_provider reuses a globally set SDK MeterProvider, so the instrumentor that already owns the TracerProvider now builds the MeterProvider too, with a View per GenAI histogram carrying the semconv-advised boundaries (extended to 80 s for TTFT and 328 s for operation duration, since this host has seen 20.8 s and 260 s). Names come from litellm’s own Metric constants so a rename breaks startup rather than a View. The installed semconv Python package turned out to ship no advice at all; the numbers live only in the semconv-genai repo’s docs. First requests after the switch: le set 0.001 to 80, p95 0.74 s against a 0.61 s mean. The panel is a p95 again. Commits f2c400e, then 524f34c for this pass. Then the p95 line became p50 and p99 by model on one line chart beside Latency P95. One server rule surfaced on the way: a panel with two standalone builder queries is rejected with a bare “Update dashboard” and no detail; the same two queries wrapped in a composite query are accepted, which is why every multi-query panel in the repo is composite. Two renderer rules then explained what the chart looked like: a series with one valid point in the window is drawn as a dot, and the default span-gaps mode bridges every gap, so two verification bursts thirty minutes apart drew a straight line that read as a second pair of glm series. The panel now bridges gaps under 10 minutes only (59bcc8f), checked in the browser: continuous where traffic is continuous, dots where it is not.

Also answered “why no cache write” against OpenRouter’s own records: Z.AI caching is automatic with free writes, and the generation endpoint carries no write field at all (only native_tokens_cached and cache_discount). Four GLM requests reconciled to the cent, including the miss that seeded the cache for the two hits after it. The segment fills in when a provider that bills writes (Anthropic, Gemini, Qwen, GPT-5.6+) goes through the proxy.

Fourth pass: layout, and three renderer rules

Errors moved to a section of its own at the bottom. Latency became p50/p95/p99 of the litellm_request span per model on one line chart (the service-wide p95 hid an order-of-magnitude gap between models), which also retired the distribution histogram. The two pies became one, tokens per model, with Cost Over Time beside it. Hero token tiles and the token stack use the short unit (636.79 K, 71.75 K).

Cost Over Time’s axis read $0 at every tick, and four-decimal precision did nothing: in the frontend the panel’s decimalPrecision reaches tooltips but the y-axis builder never receives it, so currencyUSD ticks are formatted at the two-decimal default and 0.0004 collapses to $0. The none unit takes a different formatter that keeps significant digits past the leading zeros, so the panel is Cost Over Time (USD) with unit none and ticks read 0.0005. Commits bf78769 through f582804.

Fifth pass, and what the session was actually teaching

Next day: the By Model table’s Prompt and Completion columns still read raw seven-digit counts, so they take the short unit too (8141766). That was the third separate round trip for one rule, after the hero tiles and the Token Usage y-axis, which is the tell worth keeping.

Read back over the eleven commits, almost every user turn in the session was a rendering defect I had already declared shipped: “ttft i can’t really see it there”, “hmm the dots aren’t connected”, “why do i see p99 twice”, ticks reading $0, tiles reading 636790. Each was visible in one screenshot and each cost a full hm switch plus tofu apply to correct. The other half was Adam refusing workarounds I had not justified (“i’m surprised we had to do an exclude here”, “why are we exlucding and why are we widening”), which is what turned a mean-labelled-p95 into the semconv bucket View in the instrumentor.

Both are now standing directives rather than one dashboard’s history: a panel ships when the render is legible, not when the query is right, and when a metric cannot answer the question the fix belongs in the emitter.

Top 3 tomorrow

1. Eyeball the deployed dashboard in the browser once real traffic has accumulated an hour of TTFT samples; the 09-01 lesson (SQL verified, panel blank) still applies to the stacked bar.
2. Re-run the ingestion-volume figure from 09-01 after a day with the labels gone.
3. Consider moving the cost panels back onto the metric once the per-request series have aged out of retention.

Related

2026-09-01 The Rename Was the Visible Half · 2026-08-20 LiteLLM Patch Stack Rebase onto 1.97.0

Shipped

2560516 on main, deployed and verified live: dashboard modified in place by tofu (0 added, 1 changed), proxy restarted with the exclude list in its config bundle, TTFT p95 resolving for the free model used in verification.