Achhina's Digital Garden

2026-07-18 SigNoz Alert Fires and LiteLLM Empty-Completion Deep-Dive

4 min read

2026-07-18 SigNoz Alert Fires and LiteLLM Empty-Completion Deep-Dive

What I set out to do

Check whether any SigNoz alerts had fired recently, then investigate the ones that did.

What I actually did

Reviewed the alert rules: 5 configured, all currently inactive. Two had fired in the last 7 days.

ClickHouse replication queue not draining (fired Jul 15 14:35–15:50 UTC, ~75m). Investigated: peak queue size was 1 (a single stuck replication entry), LoadAverage1 stayed 0.5–2.3 the whole window (no CPU pin, actually higher after the queue drained), and it self-recovered without intervention. Benign transient, not the catastrophic GET_PART-retry-forever incident that needs a manual replica rebuild. No action needed. See 2026-07-05 SigNoz ClickHouse Replication Queue Root Cause and Graceful-Stop Fix and 2026-06-04 SigNoz ClickHouse Replication Queue CPU Burn for the catastrophic sibling.

LiteLLM empty completions (fired Jul 17 07:24 UTC, openrouter/deepseek/deepseek-v4-pro). Reproduced the exact empty span. Root client was SillyTavern (root ingress span). The prompt ended on a complete assistant turn (a Continue); output was finish_reason=stop, 0 output tokens.

I went down two wrong paths before the evidence settled it:

  1. First claimed continue_prefill regressed — false; it’s been false throughout (verified against the Jul-13 backup, config byte-identical to the working state).
  2. Then, after direct A/B tests on litellm (localhost:4000), claimed it was purely a reasoning-model artifact — also wrong. A benign complete-boundary Continue produced 4213 chars, so assistant-last isn’t deterministically empty.

The billing discriminator (from the 2026-07-13 SillyTavern Continue Fixed by Turning Prefill Off work / ADR 0020) settled it: the fire billed input 0. Input-billed empty = the documented Continue-empty type — an assistant-last prefill was sent despite continue_prefill=false, so the Jul-13 nudge-path fix had regressed. Config unchanged; SillyTavern auto-updated to 1.18.0 (image created Jul 16 22:57 UTC via watchtower) ~8.5h before the fire = likely trigger (unconfirmed — couldn’t read the 1.18.0 changelog, and the failure is intermittent ~4/6).

What was striking

  • gen_ai.usage.output_tokens counts reasoning tokens too, not content-only. A max_tokens=128 request that spent all 128 on reasoning recorded output_tokens=128, not 0 — so reasoning-truncation empties never trip the alert. The alert only catches true zero-completion-token responses, which is exactly the fire.
  • deepseek-v4-pro is a reasoning model: a small max_tokens yields reasoning-only, empty content, finish_reason=length — and OpenRouter bills the reasoning (~$0.00011 on the test) even with no visible output. A separate, non-alerting cost concern. ST sends max_tokens=20000, so ST itself is safe from that one.
  • The alert annotation’s request_shape triage (fresh_generation/continue_prefill) is dead for this model — the attribute isn’t present on the spans. Discriminate by billing instead.
  • The working trick to read large span attributes (message bodies, cost, tokens) is groupBy on the attribute — the group value comes back. (Corrected Jul-19: raw projection is not broken; see Update below.)
  • Two turns of confident-but-wrong diagnosis, each corrected by insisting on a test / checking the billing. “Test it first” earned its keep.

Update 2026-07-19 — landed the declarative changes

Followed through on tomorrow-item 2 plus the alert-management gap:

  • Rewrote the litellm_empty_completions alert annotation with the billing discriminator (zero-billed = our-side degenerate trailing-system empty per ADR 0018; input-billed = model ran and returned empty, then split assistant-last Continue per ADR 0020 vs a genuine decline). Notes that reasoning models never reach output_tokens=0 via reasoning-only truncation.
  • Codified the last hand-created alert into nix/terraform. “Claude raw body filename parse drift” (019dbfe8…) was the only rule lacking managedBy:terraform. Added a claudeRawBodyParseAlerts group to nix/home-manager/modules/signoz.nix (LOGS_BASED, fires on service.name='claude-code' AND body_kind='unparsed' > 0 over 5m; description points at the regex_parser in patch-otel-config.py). Deleted the unmanaged copy, hm switch recreated all 5 as managed (5 added, 1 changed, 4 destroyed). All alerts inactive during the recreate = safe window.
  • Committed b265e2d; clean-tree re-hm switch was a no-op (0 added, 0 changed, 0 destroyed) confirming no alertsHash drift.

Meta-lesson from the investigation got its own standalone memory note, reference_signoz_mcp_query_mechanics: the “raw projection returns null” belief was wrong — a zero-match raw query returns rows: null (not []) with nonzero rowsScanned; my name='raw_gen_ai_request' filter was a contradiction because the gen_ai.* attrs live on the litellm_request span. groupBy is the robust read; raw works fine when you don’t over-filter.

Every SigNoz alert is now fully declarative. Remaining open items: the ST 1.18.0 regression link is still unconfirmed (tomorrow-items 1 and 3 below stand).

Top 3 tomorrow

  1. If the LiteLLM empty recurs, pin SillyTavern off :latest and exclude it from watchtower, or re-tune prompt post-processing for 1.18.0.
  2. Update the alert annotation to add a billing-discriminator branch — done Jul-19 (see Update).
  3. Confirm (or rule out) that ST 1.18.0 changed the Continue/prompt-post-processing folding, since that’s the unconfirmed link in the regression.

Graph View