2026-09-07 Image Cost Was in the Header and Nowhere Else

What I set out to do

Give the new litellm-mcp-server image tool a cost figure. It had one, computed client-side from /model/info rates, and I spent most of the session making that number correct instead of asking what the number was for.

What I actually did

Adam stated the actual goal in five points, the first being that costs exist to be ingested into the collector and viewed there later. One query settled it: sum(gen_ai.cost.total_cost) grouped by http.route returned chat completions and nothing else. The image routes were billing real money and contributing nothing. I had verified the response header and the tool response and never once checked SigNoz.

Root cause: LiteLLM has two independent cost channels. completion_cost returns early on the image path (cost_calculator.py:1344) before _store_cost_breakdown_in_logging_obj, and the OTel exporter emits gen_ai.cost.* only from standard_logging_payload["cost_breakdown"]. So a request can carry a correct response_cost and a correct x-litellm-response-cost header while exporting no cost at all. My earlier patch fixed the header. It never touched the telemetry.

Then a detour into the OTel GenAI conventions, which turned out to matter. They have moved to their own repository (semantic-conventions-genai); the pages under opentelemetry.io/docs/specs/semconv/gen-ai are redirect stubs, so the rendered registry I would normally cite is stale. There is no image-generation operation and no proposal for one, and the reason is structural rather than an oversight: an operation name forms the span name, is sampling_relevant, is a required metric dimension, and selects which of the eleven span-type attribute contracts applies. An image call shares the inference contract exactly, so modality rides the orthogonal gen_ai.output.type — one generateContent call can return text and an image together, which a single operation name could not describe. Cost is not in the conventions either; PR #443 (gen_ai.usage.cost.{amount,currency,source}) is open and names LiteLLM’s shape as one of three incompatible ones already shipping.

Wrote ADR 0029 (docs/decisions/0029-genai-semconv-image-observability.md) and implemented it:

• Turned on OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental, an opt-in this deployment had never set. Free: CLIENT span kind, {operation} {model} span names, gen_ai.provider.name, the gen_ai.request.* sampling attributes, choice.count, cache tokens.
• Cost breakdown in a plugin, not a patch. Verified in the installed source that every callback’s async_logging_hook receives model_call_details — which already holds the built standard_logging_object — before any callback’s async_log_success_event. So a CustomLogger can populate the breakdown ahead of the exporter with no source change and no dependence on callback ordering. litellm_image_cost_breakdown.py, 28 tests.
• One narrow patch for what no hook can reach: gen_ai.output.type, the generate_content operation mapping, and the images-API usage shape. The exporter builds its attribute set internally and ends the span in the same call, so this genuinely cannot be a plugin.
• Deleted the client-side pricing from the MCP server, with a test asserting the render path never requests /model/info — a second pricing source growing back is the failure being guarded against.
• Migrated 18 dashboard queries off name = 'litellm_request'.

Card library: one answer for who {{char}} is

Second thread, on openrouter-interceptor. Yesterday’s --nickname migration put the character’s real name in data.nickname and left the JanitorAI listing title in name, across 238 cards. Adam asked for spot checks, which is how the defect surfaced.

262 of 274 migrated cards render byte-identical before and after, and the 12 that differ are all the listing title leaving somewhere it never belonged. But all six cast nicknames were wrong: {{char}} stands for one person, and these held a roster. <Rory, Hudson, Milo> headed one woman’s sheet; Marie and Josh is {{user}}'s step-mother was one woman’s scenario.

Root cause was the recurring one in this repo. The roster grammar was spelt three times: names.ts split on [,|&/] and “and”, asCharacterName split on | alone by accident of cleanExtractedName, and name-from-capture kept a third copy without the |. So Cassidy | Tony | Sterling was Cassidy and Rory, Hudson, Milo stayed a cast, and which a card got depended on how its creator had punctuated the tag. eachName is the one answer now, and asCharacterName is the first person it finds.

Second fix, on Adam’s follow-up: asClaimedName outranks every other source so a card can keep a name an extractor would refuse to guess (“the card named Chad is named Chad”), and it was granting that claim on the cleaned candidate. personaName strips punctuation first, so Mom's Wedding? arrived as two ordinary words, passed, and beat the chat_name that names the person. isScenarioTitle now knows nobody is named a question, and the guard covers the claim itself rather than only the fallback branch it never reached.

Applied both to the live library: 212 cards, every one of them gaining {{char}} macros, 197 of the 212 rendering byte-identical afterwards and all 15 that differ being a cast or a title leaving somewhere one person’s name belongs.

Third fix, for the 14 cards left rendering a title with no capture to read: nine carried the name in the title itself with a descriptor after it, so the title proposes and the card’s own prose decides. A genre tag is read past, an excluded word ends the name rather than being dropped from it, and the hyphen is offered both ways because it separates a descriptor in “Nami-Wife Porn Star” and joins a name in “Silk-E”. Corroboration is on the exact spelling, which is what refuses “Hotwife Fucks the Porn Shop”: its only agreement is “extremely sexual and adventurous hotwife”, the common noun the title capitalised. 235 title renders down to 58, and the five left name nobody anywhere in the card.

That last claim was wrong, and Adam asking me to look through them “more seriously” is what found it. Three of the five do name their character; I had checked for a Name: field and stopped. After 3 Years at War opens its description with Dione: and writes Dione 199 times, and the decisive evidence was in the example dialogues: ten speakers, and hers are the only lines macroised. Its persona tag is Narrator, which is why nothing found her. fake doctor that's coming over opens [Name: Jenny, Role: Your Wife], and the title names Dr. Boner’s role while every macro is Jenny’s line. The three brothers introduce themselves as The Spencer Brothers. Fixed by hand, one script, each on its own evidence.

Then three more sweeps, each finding something the validation report does not check. Seven cards rendered a doubled surname ({{char}} Wong under a card named Amy Wong, “Sora Jae Jae”, “Mia Mia Harper” eight times over) from a conversion run against a shorter spelling than the field ended up with; the collapse went into the chain, keyed on the name’s own first and last word. Eleven cards shouted a name their prose writes in title case, which both mis-rendered and blocked macroisation outright, since the conversion matches case-sensitively: Akari 53 times, Liora 52, Mikaru 38. And --nickname mode had a hole I put there: the macroiser read the name field, which in that mode is deliberately the listing title, so a card whose nickname was its only source was never macroised at all.

All eleven of those render byte-identical before and after, which is the whole point of a macroisation. Title renders across the library went 235 → 58 → 1, doubling 7 → 0.

What was striking

Two silent defects, both found by looking at the span rather than the code.

The first revision set the image attributes before the existing usage block, which then overwrote the real token totals with the zeros a chat-shaped read produces — while leaving the modality breakdown beside them correct, which is the most confusing possible outcome. And gen_ai.provider.name came back as the empty string, because litellm_params carries "" rather than nothing, so the "Unknown" default never fires. Present-and-empty is worse than absent: it groups.

The replacement filter is better than the one it replaces, and by accident. gen_ai.operation.name EXISTS was chosen because the new span name is per-model and unusable as a filter. It turns out the attribute was set in the legacy shape too, so the panels keep their history across the cutover instead of restarting at zero. Confirmed by rendering the dashboard rather than reading the queries — 2026-09-01 The Rename Was the Visible Half is the reason I no longer trust a query that merely looks right.

The before/after in one query, 24h, by operation: acompletion $2.886and‘aimag{e}_{g}eneration‘\ast \ast null\ast \ast onthelegacyside;‘generat{e}_{c}ontent‘$0.134 on the new one.

I had the wrong file open for a while. LiteLLM 1.98.0 ships two OTel integrations — the legacy integrations/opentelemetry.py and a newer integrations/otel/ package gated on LITELLM_OTEL_V2. Our config references a symbol from the new one, which sent me reading it, but the flag is unset so the legacy path is live. Worth knowing that v2 defines gen_ai.output.type and never uses it, so migrating would not have avoided the patch.

Instrumenting the MCP server was nearly free, and I had talked myself out of it. I closed the session recommending against it — new OTel deps through uv2nix, exporter config, a second collector service, for correlation a one-call-per-invocation server barely needs. Adam pushed back that it was “probably fairly free with fastmcp”, and it was: FastMCP has native OpenTelemetry instrumentation, depends only on opentelemetry-api, and ships telemetry_mode="native" by default. It was already opening mcp.server spans carrying mcp.method.name, gen_ai.tool.name, mcp.session.id and mcp.protocol.version, and already propagating context. They were non-recording for want of an SDK. Two dependencies, a TracerProvider, and one httpx event hook calling propagate.inject on the outbound proxy call, and one generate_image call now produces one trace:

tools/call generate_image           litellm-mcp    SERVER    9.81s  (root)
└── Received Proxy Server Request   litellm-proxy  SERVER    9.64s
    └── generate_content google/…   litellm-proxy  CLIENT    9.64s

The span carrying the cost is a descendant of the tool call that caused it. My cost/benefit was right in the abstract and wrong on the facts — I estimated the cost of building it without checking what the framework already did.

Emitting a signal is not the same as being able to see it. I migrated the existing 18 panel queries so nothing broke, reported that as done, and left the new attributes with no panel reading them. Adam asked whether I had added image and MCP widgets; I had not. Two new sections now: Image Generation (renders, spend, cost per render, p95 render time, cost over time by model, and a per-model table pairing $/render with gen_ai.usage.image.output_tokens) and MCP Server (tool calls, errors, p95 duration, tools used, calls by tool, duration by tool). Cost per render is the one that earns its place — image output tokens bill at ~20x the text rate, so a model swap moves it far more than it moves cost per chat call.

Rendering them caught what reading the JSON would not: six columns overflowed a six-wide table, and the By Image Model row read lower than the same model’s row in By Model because renders predating today carry no gen_ai.output.type. Fixed the first, documented the second in the panel description.

Dashboard sections do not nest, and finding that out needed a control. spec.layouts is a flat list whose union has exactly one member — the server enumerates it, unknown layout kind ""; allowed values: 'Grid' — and a layout item’s content must resolve to a panel: "#/spec/layouts/1" must reference a panel as "#/spec/panels/<key>". My first three probes all failed, and every one of them would have read as “nesting rejected” if I had stopped there. They were failing on payload shape instead: the file we maintain is in the terraform provider form ({"grid": {...}}, ref, snake_case) while the API wants {"kind": "Grid"}, $ref, camelCase. The flat control failed identically each time, which is the only reason I did not draw the wrong conclusion. Fastest way to a valid probe turned out to be copying a panel verbatim out of the live dashboard’s own GET payload.

So MCP got flattened into the foot of Agentic & Tool Use rather than nested inside it, and Latency moved up into Traffic. Flattening cost that section its unambiguous names — “Tool Calls” meant both tools the models called through the proxy and tools clients called on our MCP server — so the latter carry an MCP prefix that is load-bearing rather than decorative.

Claude Desktop had been silently skipping HTTP MCP servers all along. I inferred it from missing logs — every stdio server has an mcp-server-<name>.log, neither HTTP one ever did — and Adam produced the actual toast: “The following entries in claude_desktop_config.json are not valid MCP server configurations and were skipped: jupyter, litellm.” So jupyter had been dead in Desktop for as long as it had been configured, and nobody had reason to notice.

Wrote mcp-stdio-bridge, a transport adapter speaking stdio to the client and streamable HTTP to the server. Generic by design — it takes a URL, so the one binary fixes both entries. Not a second server: one thin relay per Desktop launch against the one long-lived HTTP instance, so the HTTP-only decision survives intact.

Two bugs found by testing rather than reasoning, both in teardown:

• tg.cancel_scope.cancel() on stdin EOF leaked the process — the scope cancelled but teardown never completed, against a live endpoint as much as a dead one, so every Desktop shutdown would have left a bridge running forever with nothing to report it. await sink.aclose() instead lets the HTTP writer finish, which ends the response stream, which ends the other pump; the relay unwinds itself.
• Treating a stream Exception as a skippable bad frame hung the client. A connection failure arrives that way, and logging-and-continuing leaves the client blocked on a response that can never come.

The session test passed while the leak was live, which is the lesson: it closed cleanly enough not to notice. The regression test that pins it asserts the process exits, not that the session works.

Also fixed a merge-patch bug found on the way: transformMcpServer filters nulls out of what it emits and RFC 7396 preserves absent keys, so nothing the desktop module wrote could ever delete a field. litellm was deployed as a hybrid — stale command, args and an env pointing at the retired ~/Pictures/generated, sitting beside the correct type and url, with a store path garbage collection was free to remove.

I built a preview for an image that was already fetchable. Adam asked why the render came back downscaled and why the path was a local one. I measured the preview sizes and started proposing a bigger cap — 1024px, ~9k tokens — before he cut in: “there should be no preview. that was the point of the url.” Which is right, and I had missed the point of my own design. The render goes to Drive precisely so the caller can fetch it; re-sending a shrunken copy inline spends thousands of tokens on something already available, at a resolution too small to judge. The measurements were real and the conclusion I was drawing from them was wrong.

The path complaint had a subtler answer: the returned path is the Drive path, the local File Provider mount. But that only means something on this Mac, which is exactly the thing Drive was supposed to fix.

Turns out the URL is obtainable with no OAuth at all. Drive for Desktop keeps a SQLite catalogue at ~/Library/Application Support/Google/DriveFS/<account>/metadata_sqlite_db, and items.id is the same id the Drive API returns — checked against both for the same file. Three traps, each costing a cycle:

• Read it in place, read-only. New rows live in the write-ahead log, so a copy of the main database alone does not have them. My first probe reported “no id after 60s” for a file that had one after 1.1s.
• local-NNNNN is a placeholder, not a Drive id. It appears ~1s after the write; the real id replaces it at ~7s. A URL built from the placeholder is confidently broken.
• with sqlite3.connect(...) does not close the connection — sqlite3’s context manager only ends the transaction. That leaked a handle per poll, and surfaced as ResourceWarnings failing an unrelated test in the same run.

The last one is the good kind of test failure: it pointed at a production leak while looking like test pollution.

The naive fix was wrong in exactly three of 186 cards, and only the dry run said so. Splitting a roster at the first separator also cut titles in half: Your Extra Sweet, Not Suspicious Wife became “Your Extra Sweet”, and Surprise, you're dating the school slut! became “Surprise”, both outranking the chat_name that named the person. So a member has to name somebody, and a card claims what its field says rather than a piece of it. I would not have found any of that from the tests; I found it by counting what the change proposed to write and asking which proposals gained no macros. That number went 206 → 212 and the no-gain count went 1 → 0, which is the whole verification.

The same duplication keeps coming back in this repo, and it is what Adam has been naming all week. Yesterday it was five name resolvers folded into one. Today the roster split turned out to have three spellings, one of them in a script written after the consolidation. Writing the convention into AGENT.md did not stop a fresh copy appearing; putting the answer somewhere a caller has to go through is the only thing that has.

Top 3 tomorrow

1. Watch whether gen_ai.provider.name is empty on any other non-chat route — the fallback is general but only image was verified.
2. Decide whether the image spend deserves its own dashboard row now that gen_ai.output.type = image makes it a one-line filter.
3. Track semconv PR #443; if it lands, gen_ai.cost.* becomes gen_ai.usage.cost.amount and the panels move again.

Related

2026-09-01 The Rename Was the Visible Half, 2026-09-05 The LiteLLM Cost Dashboard Was Half the Bill, 2026-09-03 The Histogram Had One Series Per Request, 2026-08-20 LiteLLM Patch Stack Rebase onto 1.97.0, SigNoz Dashboard Design Guide, 2026-09-04 OpenRouter Interceptor Review Findings Actioned, 2026-08-28 OpenRouter Interceptor Analyses a Character It Sees Twice