2026-09-01 The Rename Was the Visible Half

What I set out to do

Work out why SigNoz kept saying no data found for the metric gen_ai.client.token.cost.sum (last seen 1 week ago) on the SillyTavern → LiteLLM → OpenRouter path.

What I actually did

The surface answer took ten minutes. litellm 1.97.0 renamed the cost instrument to gen_ai.usage.cost, and moved two timing histograms onto their real semconv spellings. It arrived with the flake bump in 2026-08-20 LiteLLM Patch Stack Rebase onto 1.97.0; last sample of the old name was 2026-08-19 15:00, first of the new one 2026-08-20 23:30. The dashboard had been blank ever since and nothing failed.

The real problem was underneath. increase returned 0 on the new metric too. litellm carries hidden_params — a per-request JSON blob containing response_cost — as a metric attribute, so every request mints a unique fingerprint: 523 distinct values in 7 days against 1–3 for every other label. The same release also flipped GenAI histograms from delta to cumulative. Under delta that cardinality was invisible, because increase sums deltas and a lone sample contributes its full value. Under cumulative each fingerprint holds one constant value re-exported every 5s forever (avg 16,764 samples per fingerprint, max 38,487), so increase reads 0 and a naive sum(value) reads $30,444.83\ast \ast againstatrue\ast \ast$1.39. A latent bug became a total outage the moment temporality changed.

Rewrote the three cost panels as ClickHouse SQL on the sum(per-fingerprint max) pattern, verified against the database. Then swept everything: three more dead span attributes (gen_ai.usage.total_cost, gen_ai.usage.input_tokens.cached, ai.usage.outputTokens), all fixed.

Then built a guard so this class can’t recur: a nix check that reads metric and attribute names out of the emitter packages in the closure — litellm’s Metric and attribute classes, its legacy mapper, the CostBreakdown TypedDict, and the codegen’d OTel semconv modules — and fails the build on a name nothing declares. Wired into the activation script, not just nix flake check, so it gates the very switch that would deploy the stale dashboard.

What was striking

Checking the checker mattered more than writing it. Two bugs, both silent, both caught only because I demanded a failing case. Rust packs string literals into rodata with no terminators, so a greedy macmon_[a-z_]+ matched 28 concatenated names as one token. And pkgs.macmon inside the flake’s checks is bare-nixpkgs 0.6.1, not the overlay’s local rebuild — the guard was validating the wrong binary and passing an empty allowlist. A guard that reads the wrong package is worse than no guard, because it reports success.

No cost metric exists in OTel semconv, and I confirmed it three ways before trusting litellm’s own docstring. open-telemetry/semantic-conventions-genai#443 is trying to add one and is not converging yet: lmolkova’s CHANGES_REQUESTED from 2026-08-10 still stands unanswered, CI is red, and the total-vs-per-class argument reopened at the end of August. If it lands as written the metric becomes gen_ai.client.operation.cost — a second rename for these panels.

Schema rot and liveness are different failure classes and need different mechanisms. A build gate catches renames by emitters in the closure. It cannot catch an emitter that simply stopped: the pydantic dashboard’s attributes last flowed 2026-05-06 from a permission-suggestion service, dormant almost four months. Nor can it cover Claude Code, which is a self-updating binary outside the closure entirely — its names are greppable, but it updates without any rebuild, which is exactly when it would drift.

The ingestion number is the real story. ~1.1 billion samples a week of gen_ai.* for ~525 actual LLM requests, gen_ai.client.token.usage.bucket alone at 281M. Dropping hidden_params and metadata.requester_metadata at the collector would cut that by orders of magnitude and make the builder queries work again.

Top 3 tomorrow

1. Deploy, then eyeball the three rewritten panels. Done — and the eyeballing paid for itself twice over.
2. Drop hidden_params / metadata.requester_metadata at the collector, then revert the cost panels to builder queries with explicit cumulative temporality. Superseded: the panels read spans now, so the metric’s cardinality no longer blocks them. Dropping those two attributes is still worth doing for ingestion volume alone (~1.1B samples/week), just no longer on the critical path.
3. Decide the pydantic dashboard’s fate — revive the emitter or retire the dashboard.

Related

2026-08-20 LiteLLM Patch Stack Rebase onto 1.97.0 · 2026-08-31 The 429 Was Never a Rate Limit

Shipped

Three commits on main, deployed via two hm switch runs:

• 72064c6 repoint the dashboards at the names litellm 1.97.0 emits
• 3d731b3 the build-time schema guard, both axes
• d5658fd give Cost by Model the DateTime column a bar chart needs

That third commit is the lesson. I verified the rewritten SQL against ClickHouse, deployed, and then opened the dashboard instead of declaring victory — Total Cost showed $1.39 and Cost Over Time drew its three series, but Cost by Model said “No data in this time range”. The SQL was never wrong; it returned the right three rows the whole time. The shape was wrong for the panel type. SigNoz builds a bar chart’s series from a DateTime column, and I had given it a scalar with only a group column, because the claude-code panel I copied the pattern from is a pie, which accepts scalar+group. Bar does not. No amount of offline query verification could have caught that.

State at this point, all verified in the browser: Total Cost $1.39,CostbyModeldrawing,CostOverTimedrawing,CacheShareofInputSpend20.2$1.31. Superseded — see Third session. $1.39 turned out to be 10% over the actual bill.

Worth noting the metric channel ($1.39)andthespanchannel($1.31) disagree by ~6%. Consistent with the split in project_litellm_cost_telemetry_undercount — chased later the same day, and the answer was that neither channel was mispricing anything. See Third session.

Also swept the alerts and saved Explorer views, which the build guard does not cover at all. Every referenced name is live: httpcheck.status (157k samples), ClickHouseAsyncMetrics_ReplicasSumQueueSize (10k), gen_ai.output.messages, gen_ai.usage.output_tokens. Nothing dead there.

One caveat on my own sweep: the first pass flagged 16 names and 15 were false positives, because I queried span attributes for things that live in metric labels or resource columns. Same shape as the macmon bug — a check reading the wrong place and reporting confidently. Twice in one day.

Second session: handing supervision back to Docker

Started somewhere else entirely — does launchd have systemd’s Restart=? It does: KeepAlive in launchd.plist(5). Boolean true is Restart=always; the dict form gives SuccessfulExit (either polarity, so on-failure and on-success) and Crashed (on-abort), with ThrottleInterval standing in for RestartSec. Three asymmetries worth keeping: KeepAlive implies RunAtLoad, so there is no “restart only, never start”; dict keys are OR’d, not AND’d; and launchctl stop does not stick on a KeepAlive job, because launchd treats an admin stop like any other exit. launchctl bootout gui/$UID/<label> is the real stop. systemd explicitly excludes systemctl stop from Restart=; launchd does not. There is also no StartLimitBurst analogue and no permanent failed state — just the 10s respawn floor.

The reason I asked was the feeling that activation scripts were doing work a supervisor should do. Audited it, and the launchd half turned out to be already done: every long-running agent carries KeepAlive = true (litellm, macmon, signoz-notifier, the obsidian-mcp bridge, aerospace, glance, jankyborders, atuin). The three without it — logrotate, tldr-update, gmailctl-drift-check — are genuinely periodic StartInterval/StartCalendarInterval jobs, which is correct. And home-manager’s setupLaunchAgents already boots out and re-bootstraps a changed plist, so “updated through switches” was handled too.

The whole gap was Docker, and 2026-07-16 Docker Services Down After Docker Desktop Restart had already diagnosed it in July and named the fix in its own closing line: unattended recovery “would still need restart: always or a supervised unit.” Every compose service was on unless-stopped, which deliberately leaves gracefully-stopped containers down when the daemon returns — and Docker Desktop’s shutdown gracefully stops everything. So hm switch was the only recovery path, and mkComposeDeploy’s ensureRunning branch existed to be that path.

Flipped the six long-running services to restart: always, wired to the existing autoStart knobs so autoStart = false drops back to unless-stopped and a deliberate docker stop still survives a daemon restart. Open WebUI unconditional (it has no opt-out), SillyTavern via a restartPolicy let-binding across envoy and the app, watchtower on cfg.autoStart, and SigNoz via a new @RESTART_POLICY@ placeholder plus four JSON-patch ops in casting.yaml.in. Rewrote the autoStart docstrings and the docker/lib.nix contract note to state the split plainly: Docker owns unattended recovery, activation is the fallback for what a restart policy cannot see — config changes, and a stack that is down while the daemon is up. The docker_container_egress_broken runbook no longer tells the operator to run hm switch; a still-down stack after the daemon is up now means the policy regressed.

What was striking (second session)

The fix was subtraction, not a smarter script. 2026-07-14 Runtime State Reconciliation Framework had already set the rule — activation reconciles filesystem and unit-definition state, and delegates process supervision to the supervisor. unless-stopped was activation quietly holding supervision it should never have owned. Two months of ensureRunning branches, an autoStart option on three modules, and a Docker-daemon reachability guard, all compensating for one word in a compose file. The framework said what to do in July; it just was not obvious that the July work (which added autoStart everywhere) was building the compensation rather than the fix.

Choosing which services to exclude was the actual engineering. Three SigNoz containers are run-once: signoz-configure-retention ("no"), signoz-telemetrystore-clickhouse-user-scripts and signoz-telemetrystore-migrator (both on-failure). always would loop them forever. Used op: replace rather than op: add on the four that do get it, deliberately: a foundry bump that renames a service or drops the restart key now fails the forge loudly instead of silently reverting to switch-only recovery. That is the same tripwire instinct as this morning’s guard, and the same lesson as the macmon bug — a check that reads the wrong thing and reports success is worse than no check.

Verified rather than assumed, given how the morning went: the activation package builds, all six services render always, re-running foundryctl forge on the new casting confirms the replace ops apply and the three run-once containers keep their own policies, re-evaluating with all three autoStart flags false renders unless-stopped everywhere, and nix flake check passes.

Shipped (second session)

Both halves landed and are live. 6c7bf93 in the dotfiles, 5b682fc for the flake bump, and achhina/media-stack#62 (f6ac19c) for the stack that lives in its own repo — 13 more services there, moved via an x-restart anchor that mirrors the existing x-logging one, chosen for exactly the reason that one carries in its comment: a new service cannot silently opt out by forgetting the block.

Final live state, all 25 containers: 21 on always, 2 on no (orchestrator, signoz-configure-retention), 2 on on-failure (clickhouse-user-scripts, telemetrystore-migrator). Every container healthy after the switch. The fleet now recovers from a Docker daemon restart with no switch involved, which is the thing 2026-07-16 Docker Services Down After Docker Desktop Restart has been asking for since July.

Two process notes worth more than the change itself:

The review caught what my own sweep missed. I grepped for stale unless-stopped references and excluded AGENTS.md because I had just edited it — the exact file most likely to carry another mention. It did: the indexer-backoff section named the old policy in present tense. Fixed in a second commit, along with the observation the reviewer did not make: always makes that section’s wait_for_services gate more load-bearing, not less, because under unless-stopped the parallel-start race only happened on an ungraceful restart, and now every daemon restart brings the whole stack up at once. Third time today that a check reading the wrong place reported success (macmon rodata, span-vs-metric attributes, and now this).

I was not the only session in the repo. Five SigNoz dashboard commits landed on main between my commit and my push, with hm switch runs at 22:22, 22:35 and 22:51 — so the dotfiles half of this was already deployed by someone else’s switch before I ever ran one, since my commit was an ancestor of all three generations. My git push therefore also pushed five commits I had not reviewed. The earlier “staged dashboard change” I stopped to investigate was the same session mid-edit on that file, leaving a stale blob in the index via prek’s stash-and-restore. Checking the live containers rather than assuming my commit was undeployed is what surfaced all of it.

Still open

• No check asserts the restart policy. The test harness reads artifacts off hmConfig, but the compose files are let-bound writeText derivations with no option exposing them, so a check needs an internal-option pattern the repo does not use yet. Worth doing — this is exactly the class of thing that silently reverts, and the op: replace tripwire only covers the SigNoz quarter of it.
• Unverified in the way that matters. Nothing here has actually survived a real Docker Desktop restart yet. The policy is correct on every container, but the claim “the fleet recovers unattended” is still inference from the docs, not observation. The next daemon restart is the test.

Related (second session)

2026-07-16 Docker Services Down After Docker Desktop Restart · 2026-07-14 Runtime State Reconciliation Framework · Runtime State Reconciliation Decision Framework · Nix Launchd Agent Identity on macOS

Third session: reconciling against the actual bill

The ~6% gap between the two cost channels was worth chasing after all, because the answer was neither of them.

Priced 593 requests individually against OpenRouter, the method from project_litellm_cost_telemetry_undercount: pull gen_ai.response.id off the spans, GET /api/v1/generation?id= each one with the ordinary OPENROUTER_API_KEY.

channel	recorded	billed	gap
span (gen_ai.cost.total_cost)	$1.278423	$1.265880	+0.99%, exact on 575/592
metric (gen_ai.usage.cost)	$0.084655	$0.084655	+0.00%, exact on 26/26

Only 26 metric fingerprints carry a generation id — most hidden_params use the short header form — so that sample is small but unanimous. Per request, both channels are correct. The bug was mine, in the window.

gen_ai.usage.cost is a cumulative counter re-exported every 5s for the life of the process, so any window shorter than that uptime also collects series born before it. Of the 535 series with samples in a 7d window, 39 predated it and contributed $0.2191ofthe$1.3927 total. sum(per-fingerprint max) cannot see that: it takes each series’ whole lifetime value and attributes it to whatever window is on screen.

That pattern is right on the claude-code dashboard, where a fingerprint is a session belonging to one window, and wrong here, where a fingerprint is a request whose counter outlives it by weeks. I carried it across without re-deriving whether the premise held.

So all three cost panels moved onto spans, which carry one value per request and are window-correct by construction. They became plain builder queries — no raw SQL, no fingerprint arithmetic, no temporality caveat — and sum(per_series) over distributed_samples_v4 left the repo entirely. Total Cost now reads **$1.28\ast \ast against$1.2659 billed. The Cost Over Time chart is the visual tell: it used to open elevated at the window edge because 39 old series dumped their lifetime value into the first bucket, and that step is now flat.

Also fixed Models Info, which showed model and tokens as n/a. It filtered the Received Proxy Server Request span, but litellm stamps gen_ai.* onto litellm_request: over 7d the proxy span carried gen_ai.request.model on 0 of 596 spans, litellm_request on 593 of 594. Count and Latency are span intrinsics, so they rendered fine and hid it.

What was striking (third session)

Three times in one day I copied a pattern without re-deriving its premise. The pie query onto a bar chart. The macmon regex onto Rust rodata. The session-shaped fingerprint aggregation onto request-shaped fingerprints. Each was correct in the place I took it from. The tell is the same every time: I checked that the thing ran, not that its assumption still held.

A guard cannot catch a valid name on the wrong span. Models Info used gen_ai.request.model, which is declared, live, and correct — just absent from the span being queried. The build check passes it, and should. That is a third failure class alongside schema rot and liveness, and only looking at the rendered panel finds it.

Reconciliation reframed the question. I had been asking “which channel is right”, which has no answer, when the question was “which channel is right for a window”. Pricing individual requests is what exposed that both were fine and the aggregation was not.

Still open (third session)

• Six commits deployed, 72064c6 → e9a8ea3.
• The pydantic dashboard is still dormant — emitter silent since 2026-05-06.
• Nothing asserts the panels render. Every failure the browser caught today was invisible to both the build guard and to querying ClickHouse directly. A screenshot check would be the only thing that catches the next one.