Achhina's Digital Garden

2026-09-18 pi DeepSeek Session Diagnosis and LiteLLM Routing

3 min read

2026-09-18 pi DeepSeek Session Diagnosis and LiteLLM Routing

What I set out to do

Work out why a pi session on deepseek/deepseek-v4-flash-0731:free never produced a code review of the media-stack feat/plex-terraform branch, check SigNoz for configuration gaps, and route pi through LiteLLM.

What I actually did

  1. Transcript diagnosis (session 01a0b71c): 280 assistant turns, 402 tool calls, one completed answer, 45 minutes, ended aborted. 175 calls re-ran a --stat diff; the full diff was never read front to back. Reasoning restarted from scratch nearly every turn, and 29 turns claimed tool outputs were “merged/interleaved”. Context bloated to about 660K tokens on repeated diffs. The one answer was a summary built from the diffstat, not a review.
  2. Ruled out pi message formatting: tool results are separate role: "tool" messages; all 402 tool-call ids unique. The upstream was a single free fp8 deployment (OpenInference) per the OpenRouter generation API. Reasoning was 26 to 83 tokens per turn at thinking high.
  3. SigNoz gap: pi had never reported (no service in 7 days). It called openrouter.ai directly with OPENROUTER_API_KEY, bypassing LiteLLM.
  4. Routed pi through LiteLLM: new extension coding-agents/pi/extensions/litellm-route.ts overrides the built-in openrouter provider’s baseUrl to localhost:4000/v1 and rewrites the payload model to openrouter/<id> in before_provider_request. Deployed only where programs.litellm.enable.
  5. Fixed a LiteLLM conflict (TDD): litellm_deepseek_reasoning_strip dropped all structured reasoning on DeepSeek routes. DeepSeek’s thinking mode guide requires reasoning_content passed back on every request that carries tools. Now kept when tools is non-empty; <think> text stripping for SillyTavern is unchanged.
  6. Reasoning replay probe: planted a codeword in replayed reasoning. Recall was about 27% on the free endpoint and 40% on paid Flash for both pi’s shape and reasoning_details alone; control 0%. Inconclusive about cause; not specific to pi’s format or the free tier.

Verification

  • 25/25 strip-callback tests pass; prek hooks pass on the changed files.
  • hm switch applied; LiteLLM restarted on the new bundle; pi --list-models catalog intact.
  • End-to-end pi -p on the free model completed a 3-turn tool task; three chat deepseek/deepseek-v4-flash-0731:free spans landed on litellm-proxy in SigNoz.
  1. Wire capture: ran a second LiteLLM (port 4002) from a copied config bundle with OpenRouter’s api_base pointed at a logging forwarder, and a debug pi extension dumping pi’s payload. pi → LiteLLM → OpenRouter is faithful; replayed reasoning_details survives. Response stream opens with a lone " " content chunk; upstream fingerprint vllm-dev-ep-17a051ab.
  2. Root defect found via prompt_tokens deltas: the free endpoint drops replayed reasoning longer than 1024 characters (1005 kept, 1025 dropped, whole not truncated, every field shape). Paid Flash (Relace) keeps all lengths. In the original session 25 of 280 turns (the longest plans) were dropped.

Conclusions

  • The free endpoint loses the model’s longest plans between tool calls; the other 250 turns’ reasoning did arrive, so the looping is mostly the model itself (Flash, shallow reasoning, poor use of its own replayed plan: 2/8 recall even when the plan was in the prompt).
  • Recommendation: use the paid slug (about 0.12 per M tokens) or a stronger model for agentic review work.

Commits

  • 9f5f34d fix(litellm): keep reasoning_content for DeepSeek tool-carrying requests
  • 6078abc feat(pi): route pi’s OpenRouter traffic through LiteLLM

Graph View