2026-07-31 SigNoz Dashboard v2 Break and Upgrade Readiness

What I set out to do

Review the update script’s ordering, then update -c blew up on hm switch.

What I actually did

The break. nix flake update bumped signoz-dashboards-src 308cfd81 → b6a6ce39. That range is one commit, aed8dd46 “Refresh 22 LLM dashboard templates” (PR #395), which rewrote all 22 LLM templates into the Perses v2 schema. pydantic-ai/pydantic-dashboard.json went from v1 keys (title, widgets, layout, panelMap, …) to image, spec, tags, with the title relocated to spec.display.name. dashboardToTf reads v1 d.title, so eval died as attribute 'title' missing buried in a home.file trace. hostmetrics and openrouter were not in that batch and are still v1.

The fix. Vendored the last v1 revision to files/dashboards/pydantic-dashboard.json, matching the existing fork pattern for claude-code and litellm. Added a guard in fetchDashboard that throws a message naming the file and the remedy when a fetched dashboard lacks title, so the next migrated template fails legibly.

Subtle trap worth remembering: my first pass added the _upstream provenance key the other two forks carry. dashboardsHash hashes toJSON dashboards wholesale, so any extra key, even a pure annotation, moves the hash, and replace_triggered_by turns that into destroy-and-recreate for all five dashboards with fresh UUIDs and dead saved links. Dropped the key, put provenance in a comment, and confirmed the rebuild produced w62fqjny…, byte-identical to the running generation. Apply reported 0 added, 0 changed, 0 destroyed.

Second, unrelated failure. After the eval fix, checkLinkTargets aborted on ~/.local/share/signoz-terraform/main.tf.json “would be clobbered”. The path is declared as home.file but existed on disk as a regular -r--r--r-- copy dated Jul 27 03:49, not a symlink. Contents were byte-identical to what the active generation declares, so moving it aside and re-switching restored HM ownership. Never established what replaced the symlink, so it may recur.

Upgrade readiness recheck

2026-07-27 SigNoz Upgrade Path Blocked on Foundry Migration decided: wait for v0.135.0, then do Foundry + ClickHouse 25.12.5 + server bump as one piece. The gate cleared, but the shape changed.

• v0.135.0 released 2026-07-29. Docker Hub’s tag-listing API still shows v0.134.0 as newest, which is stale — docker manifest inspect signoz/signoz:v0.135.0 succeeds. Controlled against v0.134.0 to prove the probe works. Another instance of A null result needs a control before it means absent.
• The new blocker did not exist on 07-27. v0.135.0 retires the v1 dashboard API, not deprecates it: 501 Not Implemented, error code dashboard_deprecated. Dashboards convert in place, irreversibly. So server bump and Terraform provider migration are now one atomic cutover, not two steps.
• Provider v0.1.0 (2026-07-26) confirmed: no alert.md at that tag, rule.md present. signoz_alert → signoz_rule is forced into the same change. Current state is 5 signoz_dashboard + 5 signoz_alert + 2 restapi_object, all ten needing rewrite and re-import.
• signoz-mcp-server is already fine. The flake input uses releases/latest/download/, and its locked lastModified is exactly the v0.10.0 release timestamp — the version v0.135.0 requires. No action.
• ClickHouse is still not a hard gate, and the 07-27 read holds. Checked collector v0.144.6 directly: PR #833 “store trace attributes as json” was reverted by #849 in that same release, so the migration needing 25.12.5’s object_serialization_version still has not shipped.

What was striking

The 07-27 plan bundled Foundry + ClickHouse + server bump because they looked like one piece. Rechecking, they separate cleanly and a different seam is the real one: server 0.135 + provider 0.1.0 + dashboards v2 + alerts→rules is atomic and unavoidable, while Foundry and the ClickHouse 25.5 → 25.12 jump are independently schedulable. Given this host has burned on ZooKeeper replication queues twice, keeping the CH jump on its own change is worth more than the tidiness of one big cutover.

Also: tonight’s break is the same v1→v2 migration arriving through a side door. Upstream shipped v2 dashboard JSON before the server that requires it was even installed here.

Decision

Vendor-and-hold for now. Do the cutover as: (1) server 0.135 + collector 0.144.6 via VERSION/OTELCOL_TAG on the pinned compose, together with provider 0.1.0, dashboards→v2, alerts→signoz_rule, plus the manual tofu state rm/import session. (2) Foundry separately. (3) ClickHouse 25.12.5 separately, before a collector release actually forces it.

Top 3 next

1. Back up the signoz-sqlite and ClickHouse volumes. Boot-time schema migrations are one-way.
2. Scope the dashboard v2 rewrite: pydantic and litellm have upstream v2 versions to crib from; claude-code is a heavy local fork and hostmetrics/openrouter are still v1 upstream, so three are hand-conversions.
3. Re-read the provider’s v1→v2 upgrade guide (PR #135) before touching tofu state.

Cutover, same night

Ran it. Server is on v0.135.0, all five dashboards converted to Perses v6, none Legacy. Two things went differently than scoped above.

ClickHouse was a hard gate after all, and I had just written that it wasn’t. Collector v0.144.6 crash-looped the migrator on ALTER TABLE signoz_traces.signoz_index_v3 MODIFY SETTING object_serialization_version → UNKNOWN_SETTING on 25.5.6, and the collector itself then blocked in migrate sync check, so ingestion stopped. My reasoning was that PR #833 was reverted by #849 in that same release, so 25.12.5 could wait. The revert removed the exporter write path; migration 1011 still shipped. Reading a changelog is not reading the migration list. Pinned the collector back to v0.144.5 and ingestion resumed; server 0.135 + collector 0.144.5 runs fine.

The dashboard cutover is blocked upstream. Harvested all five server-converted dashboards from /api/v2/dashboards and wrote a converter for the two gaps between API shape and provider shape: camelCase→snake_case, and inline kind discriminators → named union wrappers (grid, list_variable, number_panel, builder_query, …). Two traps, both caught by tofu plan rather than by reading: $ref must become ref, else every layout item plans as unknown; and the rename rule has to be identifier-only, because customColors/columnUnits are keyed by user strings like "Cache Efficiency %" and "A.count()" that a naive snake_case pass would corrupt.

Import worked, read worked, write failed: dashboard_invalid_input: invalid signal "". Isolated it instead of guessing — PUT the server’s own unmodified GET payload back to /api/v2/dashboards/<id>, and it returns success. The server round-trips its own data fine, so the fault is the provider serializing unset optional attributes as "". Stripping empty signals from my config changed nothing, which fits: the "" is generated provider-side, not carried from the payload.

Resolution. Dashboards are now unmanaged: removed from signoz.nix and state rm’d, still live and working. The provider-shaped v2 JSON is vendored in files/dashboards/ ready to tofu import when the provider is fixed. That deleted ~260 lines of v1 machinery (macmon/Apple-Silicon widget injection, fixHaving, stripStepInterval, the layout shifting) — all of it now baked into the harvested JSON, since the server converted the live dashboards including our patches. Verified before deleting: 32 macmon references survive in the harvested hostmetrics dashboard.

Alerts never needed migrating. signoz_alert on provider 0.0.11 still reads and plans fine against server 0.135 — only the v1 dashboard API was retired. So the provider stays at 0.0.11, alerts stay managed, and the entire signoz_rule rewrite evaporated.

End state verified: tofu plan → No changes, nix flake check green, switch idempotent, ingestion at 166 logs/3min.

Corrections to what I wrote earlier tonight

• “ClickHouse is still not a hard gate” — wrong for collector v0.144.6; it is the gate. Correct only for v0.144.5.
• “server 0.135 + provider 0.1.0 + dashboards v2 + alerts→rules is atomic and unavoidable” — wrong. Only dashboards were coupled to the server bump.

Top 3 next (supersedes the list above)

1. File the provider bug (signal: "" emitted for unset optional attributes) against SigNoz/terraform-provider-signoz.
2. ClickHouse 25.5.6 → 25.12.5, still deferred, now with a concrete trigger: it gates collector v0.144.6.
3. Re-adopt the five dashboards via tofu import once the provider write path works.

It was never a provider bug

Went to patch the provider. Built v0.1.0 from source, added omitempty to the three aggregation signal tags, served it through a filesystem mirror — still failed. So I stopped guessing and put a logging pass-through proxy between provider and server to capture the actual PUT body. (It first broke on \x1f, because I rewrote Content-Length while forwarding a gzip body; dropping Accept-Encoding upstream fixed it.)

The captured payload had zero empty signals, and ran 9.5KB against a 38KB known-good. Diffing the two structurally showed the whole builder_query.spec missing — aggregations, filter, signal, name, all of it. Three bugs, all mine, in the converter:

1. $ref → ref. Caught earlier, by plan output.
2. Builder-query fields are flat under spec in the API but nest under spec.<signal> (logs/metrics/traces) in the provider schema. I emitted them flat.
3. Composite sub-queries are a third union point, discriminated by type rather than kind. Unwrapped, ExpandQuerybuildertypesv5QueryEnvelope matches no member, falls through to an empty envelope, and marshals to a bare null — the diff showed "queries": [null, null, null].

With all three fixed the stock provider applies cleanly: “found no differences, so no changes are needed.” No patch, no custom build, no upstream issue to file.

What made this expensive: Terraform silently discards config attributes absent from the schema, so (2) shipped a hollow spec instead of erroring, and the server reported the symptom — a missing signal reads as "" — not the cause. My “confirmed provider-side, not server-side” claim rested on a sound experiment (the server round-trips its own payload) wired to the wrong inference: that proved the payload was fine, never that the payload was what the provider actually sent. The proxy was the step I should have reached for three hypotheses earlier. Capture the wire before theorising about the serializer.

Still not wired in, and the real blocker is the alerts. Provider 0.1.0 registers NewRuleResource and no alert resource — verified in internal/provider/provider.go, not just the docs. Adopting the v2 dashboard therefore means rewriting all five signoz_alert resources as signoz_rule (typed condition/evaluation instead of today’s JSON strings) in the same commit. That coupling is real; I was wrong to say it had evaporated. It arrives via the provider, not the server.

Left green: provider 0.0.11, alerts managed, dashboards live-but-unmanaged, converter checked in at files/dashboards/convert-dashboard.py, all five converted files verified to apply with zero drift.

Corrections, second round

• “The v2 resource in provider v0.1.0 is not a usable replacement yet: it emits signal: \"\"” — wrong. The provider is fine; my converter was dropping the query spec.
• “Alerts never needed migrating” — wrong once you want v2 dashboards. Provider 0.1.0 drops signoz_alert, so they are coupled after all.

Cutover complete

Did the alert migration and finished it. Provider 0.1.0, five signoz_rule + five signoz_dashboard under management, tofu plan → No changes.

The alerts kept their hand-authored form. All five turned out structurally uniform (one builder query, one threshold, graph/builder), so the typed scaffolding — evaluation.rolling, thresholds.basic, queries[].builder_query, signal nesting — collapsed into one mkRule helper and each definition supplies only what differs. Descriptions were lifted programmatically from the rendered old config rather than retyped, so the litellm triage runbook survived verbatim.

Two fields moved rather than being renamed, and both fail silently: severity is now a label, and preferred_channels is now per-threshold channels.

Then five rounds of enum archaeology, all the same shape — the server stores legacy values that the provider validates strictly:

Attribute	Stored	Provider wants
op	"1"	"above"
match_type	"1"	"at_least_once"
reduce_to (composite only)	"latest"	"last"
temporality (composite only)	""	omitted
default_value	atlas	"atlas" (JSON-encoded)

The composite-only asymmetry is the interesting one: the plain builder_query path accepts what the server stores, the composite path rejects it. And I over-corrected once — mapped latest→last for time_aggregation too, but that attribute has a different enum where latest is the valid spelling. The error text listing both enums is what caught it.

Last obstacle was Provider produced inconsistent result after apply on one rule: we sent step_interval = 0, the server stores no stepInterval for 0 and returns null. That error fires before the plan comparison, so lifecycle.ignore_changes cannot suppress it — the config has to omit the field.

Final: server v0.135.0, collector v0.144.5, ClickHouse 25.5.6, provider 0.1.0. 5 dashboards + 5 rules managed, plan clean, nix flake check green, switch idempotent, ingestion healthy.

ClickHouse 25.12.5, and a backup that lied

Finished the last piece. ClickHouse 25.12.5.44, collector v0.144.6, migration 1011 retried from its failed row unassisted and finished, 1012/1013 followed, ingestion healthy.

The backup was corrupt and I nearly proceeded on it. tar czf /backup/x.tar.gz writing 22GB through a Docker Desktop bind mount produced a file that tar tzf walked to exit 0 — while gzip -t and Python’s zlib both failed after 2.00 GB of 32. tar stops at the end-of-archive marker, so it never reads the damaged region; its exit code says nothing about the rest of the stream. The re-run via tar czf - > file, streaming through stdout instead of the shared filesystem, verified clean: 33.97 GB decompressed end-to-end, 394,634 entries against the corrupt archive’s 4,511. That ratio is the proof — the first backup held about 1% of the files.

Given downgrades are explicitly not guaranteed and the format rewrite happens in place on first start, that check was the difference between a recovery path and none. Verify a backup by decompressing it, never by listing it. Verify tool output before claiming success again, in its most expensive form yet.

Found while verifying: a two-day-old stuck replication queue

system.replication_queue showed 16 GET_PART entries, every one failing NO_REPLICA_HAS_PART, max 163 postponements. The instinct was that the upgrade caused it — but create_time on all 16 is 2026-07-29 11:57–11:58, two days before any of tonight’s work. It predates the upgrade and survived it.

Affected tables are metrics-side only: dependency_graph_minutes_v2 (2), metadata, samples_v4, samples_v4_agg_5m, time_series_v4_1week, top_level_operations. The trace index is clean, no readonly replicas.

The uncomfortable part: clickhouse_replication_queue_stuck alerts on a non-empty queue sustained for 30m, and this has been non-empty for two days. Either it fired and went unnoticed, or it did not fire — and the alert has silently failed before (see the 2026-06-04 entry, which notes it firing silently for a month). Worth resolving separately from the queue itself.

Replica rebuild

Cleared it. Queue 16 → 0, max replica delay 157,115s → 0, ClickHouse CPU ~120% → 12%. Ran the alert’s own runbook per table: DETACH SYNC → SYSTEM DROP REPLICA … FROM ZKPATH → ATTACH → SYSTEM RESTORE REPLICA → SYSTEM RESTART REPLICA. All ten were total_replicas = 1, which is why the parts were unrecoverable: no peer has ever existed to fetch them from.

Engine semantics are what make the verification trustworthy, and they differ per table:

• 4 × ReplicatedReplacingMergeTree → row counts bit-identical.
• 4 × ReplicatedAggregatingMergeTree → down 0.5–3%, which is merge-time state combination, lossless by design.
• 1 × ReplicatedMergeTree (samples_v4, 1.6B rows) → exactly +0. This is the one that matters: plain MergeTree has no dedup to hide a loss behind, so a preserved count there is real evidence rather than an assumption. I did it last, deliberately, after the procedure was proven on an 80-row table.

RESTORE REPLICA detached 1,152 parts (~105 MiB) along the way. That looked alarming until the same samples_v4 fact settled it: 206 of those parts came off that table while its row count moved by zero, so they were covered/inactive parts, not live data. Confirmed end-to-end by reading both metrics and traces per-day across the incident window — 2026-07-29 is intact and is in fact the busiest day in the range.

The near-miss. First attempt at the batch loop used IFS=$"\t", which is locale-translation syntax, not a tab. IFS became the literal characters \ and t, so signoz_metrics split at its t and every generated query referenced a table called rics<tab>metadata. Ten tables’ worth of DETACH/DROP REPLICA ran against garbage names. It was harmless only because every statement errored on an unknown identifier before doing anything — but a DDL loop that mangles its own table names is one bad quoting choice away from being destructive. Verify the loop variable before letting it drive DDL, and prefer pair=db:table splitting over IFS games.

The alert did fire — I was wrong about that

I assumed a silent alert and pointed at the 2026-06-04 precedent. Wrong. signoz_analytics.rule_state_history_v0 has it:

Time	State	value
2026-07-29 12:31:44	firing	16
2026-07-30 12:06:44	inactive	16
2026-07-30 14:26:44	firing	16
2026-07-30 16:31:44	inactive	16
2026-07-30 19:06:44	firing	16

First fire is 34 minutes after the queue went non-empty at 11:57, against a 30m eval window. Exactly right.

The real defect is flapping, not delivery. value = 16 on every row, including both inactive ones — the queue never drained, so it should never have cleared. Cause is telemetry gaps: 10-minute holes around the 07-30 12:06 transition, and a full hour missing 15:30–16:30 with the rule flipping inactive at 16:31 precisely as data resumed. That is the atlas off AC sleeps and gaps ALL telemetry signature. With match_type = all_the_times, a 30m window containing zero data points cannot satisfy the condition, so sleep reads as recovery.

So a stuck queue looks resolved every time the machine sleeps — which both trains you to ignore the alert and resets any renotify escalation. signoz_rule has the attributes to fix it: require_min_points / required_num_points so an empty window is not “condition not met”, and alert_on_absent / absent_for to surface the gap itself.

And the parts gave the root cause the queue alone did not. At 2026-07-29 12:07:19–12:27, ClickHouse found 141 broken-on-start parts and detached 1,166 covered-by-broken — inside 8 seconds. With telemetry gapping at 11:40 and 12:00, that is an unclean restart: precisely the hard-kill the stop_grace_period block in the compose override exists to prevent. The 11:57 queue entries are the parts committed to ZooKeeper that never reached disk. Ten minutes later the restart found the wreckage.

Lesson worth keeping: “did the alert fire” is a question with a table that answers it (rule_state_history_v0), and I theorised instead of querying it. Same failure as the provider-serialization detour earlier tonight — reason about a system only after reading what it recorded.

require_min_points does not do what I claimed

Added it (required_num_points = 15, half of the 30 points a healthy 30m window holds at 1/min) — but reading the server source first showed my own recommendation was wrong, so it is committed with that written down rather than as a fix.

func (rc *RuleCondition) ShouldEval(series *qbtypes.TimeSeries) bool {
	return !rc.RequireMinPoints || len(series.Values) >= rc.RequiredNumPoints
}

threshold_rule.go continues past a series that fails this. No result → fingerprint absent from resultFPs → threshold_rule.go:379-388 converts the firing alert to StateInactive. Identical outcome to today. ShouldEval gates evaluation, never resolution, so it guards against firing on a sparse window, not against resolving on one. Real but different value: a stuck queue now needs ~15m of actual samples before it fires, instead of one stray point as the host wakes.

The knob for absence is alert_on_absent/absent_for. HandleMissingDataAlert emits Sample{IsMissing: true} once data has been gone for absent_for minutes, which keeps a fingerprint alive through the gap — but it raises a separate missing-data alert rather than sustaining the threshold one. Left off deliberately: this host sleeps routinely, so it would page on every sleep. That tradeoff is the actual open decision, not a missing setting.

Twice tonight I proposed a mechanism from a plausible reading of a schema and was wrong both times (the signal: "" provider theory, and this). The schema tells you a field exists; only the source tells you what it does.

And the flapping is not a new defect — I overstated it. Checking memory afterwards: alertOnAbsent/absentFor was already tried and deliberately removed on 2026-07-13 (commit f767c06) for exactly the reason I re-derived from source tonight, that it pages every time atlas comes off AC. The note also records the reasoning I failed to credit: a stuck GET_PART entry cannot self-heal, so the queue is still >0 when scraping resumes and the threshold branch fires again. That is precisely what happened — re-fires at 07-30 14:26 and 19:06. So the resolve/fire cycling is an accepted tradeoff with detection intact; what it actually costs is renotify escalation and signal-to-noise, not coverage. Worth having read my own notes before calling it a defect.

Detached parts, and Foundry scoped

Cleared the detached parts: 7,674 / ~940 MiB → 215 / 0 B (only system log tables left, which rotate themselves). Dropped by partition — 104 statements instead of 7,674 — plus 20 stragglers by name, whose directories carry two prefixes (ignored_broken-on-start_…, …_try1) so partition_id is NULL and DROP DETACHED PARTITION cannot address them. Active rows verified: the only decrease was signoz_metrics.metadata −56, which is an AggregatingMergeTree with 33 merges in the preceding 10 minutes, and DROP DETACHED structurally cannot touch active parts anyway.

Third quoting bug of the night on the first attempt — ClickHouse reads "..." as an identifier, so a shell-nested IN ("a","b") became unknown-identifier errors. Nothing dropped, caught immediately. Both this and the earlier IFS=$"\t" disaster came from generating SQL/DDL inside nested shell quoting. Write the script to a file instead; the failures are silent-adjacent and the blast radius is DDL.

Foundry scoped and unblocked, cutover not done. Packaged foundryctl v0.2.17 (6e1b2cc) — prebuilt darwin-arm64, same pattern as signoz-mcp-server, though its tarball ships README.md beside bin/ so the binary is at $src/bin/, not the root. Runs, codesign -v passes.

The migration is tractable — every customization maps:

Today	Foundry
image pins, ports, env, stop_grace_period	spec.patches (JSON Patch RFC 6902 over deployment/compose.yaml)
otel-collector command override (drop --manager-config, feature gate)	same, patch /services/otel-collector/command
patch-otel-config.py (hostmetrics/docker_stats/filelog)	spec.ingester.spec.config.data — merges with generated config, so cleaner than patching
configure-retention service	patch add to /services/
volumes	preserved: MIGRATION.md says cast reattaches signoz-clickhouse/signoz-sqlite/signoz-zookeeper-1 in place. Never pass --volumes to the old down.

What remains is a rewrite of the deployment layer of a ~900-line module: author casting.yaml (needs the replica/shard macros from our ClickHouse config), translate ~8 customizations into patches, replace syncSigNoz’s rsync + every composeCmd call site (health gating, autoStart, up -d) with forge/cast, then docker compose down → foundryctl cast, then drop the signoz-src input entirely.

Deliberately stopped before the cutover. The justification for Foundry weakened tonight — image pins in the override now decouple upgrades from the signoz-src pin, so this is “stay on a supported path”, not “unblock upgrades”. Against that, the cutover takes down a freshly-upgraded healthy stack, and my error rate on complex multi-step work tonight (three wrong conclusions, three quoting bugs) argues for doing a deployment-layer rewrite with a fresh head rather than at the tail of this.

Next

2. Note rule_state_history_v0 is keyed by rule_id: tonight’s replace_triggered_by recreate orphaned all five rules’ history under their old IDs. Still in ClickHouse, invisible in the UI.
3. ~940 MiB of detached parts accumulated since April (covered-by-broken, broken-on-start) — housekeeping, not urgent, and irreversible so worth deciding deliberately.
4. Foundry migration, still independent, still deferred.

Related

2026-07-27 SigNoz Upgrade Path Blocked on Foundry Migration, Homelab Services Architecture, Runtime State Reconciliation Decision Framework