Achhina's Digital Garden

2026-07-13 SillyTavern Continue Fixed by Turning Prefill Off

6 min read

2026-07-13 SillyTavern Continue Fixed by Turning Prefill Off

What I set out to do

Investigate the “LiteLLM empty completions (0 output tokens)” alert, which fired three times in a week on deepseek-v4-pro. Expected a regression of the June trailing-system bug (see 2026-06-17 DeepSeek Empty Replies via OpenRouter).

What I actually did

It was a different bug with an inverted signature. The June one produced usage: null and zero billing; these carried real usage (2237 input tokens) and real cost. And the role sequence ended on assistant, not system — so Prompt Post-Processing = Strict was still doing its job.

Every empty completion in 7 days was a Continue request. Of 197 DeepSeek calls, only 6 were Continue prefills, and 4 of those came back empty. Zero empties among the other 191.

Then the real discovery, from replaying the exact failing payload against the live proxy: Continue was not merely empty, it was structurally broken. Over 14 days, Continues median 18 output tokens vs 579 for fresh generations. Even the “working” ones were a closing beat, not a continuation.

Chased the cause down three levels, and got the first two wrong:

  1. Wrong: assumed Strict was trimming SillyTavern’s continue_postfix. Adam pushed back and asked for proof. The source disproved it.
  2. Half-right: ST’s continue_postfix genuinely never reaches the model — script.js appends it to cyclePrompt/continue_mag (the client-side stitch, prepended when saving) while the chat-completion prefill is built from chatMessage.content. Upstream saw it too (SillyTavern#4549, auto-closed by the stale bot). I shipped a proxy-side postfix for this (commit f0323be), which took empties from 0 → ~18 tokens. An improvement, but still not a continuation. Later removed entirely (4737e1c) — see below.
  3. Actually right (SillyTavern#4429, closed not_planned): “an EOS token is added after the assistant message regardless… continues and prefills are just a normal message instead of a completion and in essence do nothing.” On OpenRouter chat-completions the chat template closes the assistant turn. A trailing assistant message is not a prefill at all — the model sees a finished turn and must open a new one.

No knob suppresses that EOS through OpenRouter. Tested all of them: DeepSeek’s native prefix: true (needs api.deepseek.com/beta, which OpenRouter is not), continue_final_message, add_generation_prompt: false, and pinning provider.order=["DeepSeek"]. All landed in the same 11–25 token band.

The fix was to stop prefilling. Turning continue_prefill off routes through ST’s nudge path — instruct instead of prefill — which yields 668–919 tokens. Adam flipped it; his next live Continue scored 712 (confirmed via telemetry, user-agent: node-fetch).

What was striking

The reason we turned prefill on had silently expired. We enabled it in early June because the nudge arrived as a trailing system message and DeepSeek returns empty on trailing-system. Then on June 17 we enabled Strict for a completely unrelated bug (the /impersonate empties). Strict folds trailing system → user — which repaired the nudge path. The original justification was dead for weeks and nobody rechecked. The fix for bug A quietly invalidated the fix for bug B.

I was wrong twice and only caught it because Adam asked “did we confirm?” I had a tidy temporal correlation (empties start the day Strict was enabled) and reasoned straight from it to a cause. Reading the actual source took ten minutes and demolished it. Correlation plus a plausible mechanism is still not a confirmed mechanism.

Also: a fix that improves the metric can still be the wrong fix. The proxy postfix moved empties from 0 to ~18 tokens and made the alert quiet. It looked like a win. It was papering over a structural impossibility one layer up.

Cleanup

Once prefill was off, the postfix hook could no longer fire at all — ST stops sending assistant-last, so there is nothing for it to append to. Adam called it: dead code. Ripped it out (4737e1c, −157 lines), including its boundary regex, multimodal handling, and 14 tests.

I had wanted to keep it as a “guard in case prefill is ever re-enabled”. That was the wrong instinct — it means carrying untriggerable code indefinitely on the strength of a hypothetical. If prefill comes back, the right response is to reopen the ADR, not to have a dormant workaround quietly absorb the breakage so nobody notices.

What survived is the request_shape tag, which the empty-completion alert genuinely depends on, so the module got renamed litellm_continue.py → litellm_request_shape.py to stop lying about what it does. It’s now read-only: tags, never mutates.

Then Adam asked why the alert only fired on fresh_generation — and that question caught me making the same mistake a third time, in the same session.

I’d scoped the alert to exclude Continue prefills because their empties were benign noise (19 of 49). But turning prefill off deleted the prefill shape entirely. Checking the live traffic: every continue_prefill span in the last 90 minutes was one of my own test scripts; SillyTavern now sends only fresh_generation. So the filter was excluding nothing, and its sole remaining effect was a blind spot — a re-enabled prefill would return empty Continues and page nobody.

Fixing the root cause retires the mitigation. That is precisely what Strict did to the continue_prefill workaround, which is the whole subject of this entry. I wrote the ADR about that failure mode and then committed a fresh instance of it an hour later. Unscoped the alert (c2c864d).

The tag survives, but its job is triage now, not suppression: when the alert fires, branch on request_shape in the trace. A re-enabled prefill paging noisily is the point — the first page names the cause. A filter that swallows the regression silently is strictly worse than an alert that shouts.

One more catch from verifying rather than assuming: the alert’s description still read “Continue prefills are excluded” after the filter was gone. That’s the runbook text you’d read at 2am. Fixed it to branch on both shapes.

Graph View

Backlinks