2026-08-02 SigNoz Foundry Migration

What I set out to do

Item 3 from 2026-07-31 SigNoz Dashboard v2 Break and Upgrade Readiness: “Foundry migration, still independent, still deferred”. Deferred no longer. Replace the vendored-compose deployment layer with upstream’s own installer, on a stack that was already healthy, without losing 1.6B metric samples.

What I actually did

Rewrote the deployment layer of signoz.nix. signoz-src and signoz-dashboards-src are gone from flake.nix; the module now templates files/foundry/casting.yaml.in, runs foundryctl forge at activation, patches the rendered ingester config, and hands the result to docker compose. Local deviations moved from a parallel override file to spec.patches (JSON Patch, RFC 6902) over the rendered deployment/compose.yaml.

Three commits: ce8cb15 (feat), 4a98365 (fix, the retention gate), facea87 (docs, the ADR amendment).

The four omissions, all of which fail quietly

Foundry’s output is minimal where the vendored compose was generous. The method that caught these was diffing the forged-and-patched output against the running configuration component by component (receivers, processors, exporters, connectors, pipeline wiring) rather than checking it parsed and looked plausible. Two of the four would have been invisible until something else broke.

1. No prometheus section in config-0-0.yaml. That file fully replaces the stock image config.xml via CLICKHOUSE_CONFIG, so omitting the section deletes the :9363 endpoint. With it goes ClickHouseAsyncMetrics_ReplicasSumQueueSize, which is the metric behind the stuck-replication-queue alert I spent 07-31 clearing. Losing the alert on the day after using it would have been a fitting way to lose it.
2. Derived volume names. Foundry names volumes from component identity, so without three replace patches it would have created signoz-metastore-sqlite-0-data, signoz-telemetrykeeper-0-data and signoz-telemetrystore-0-0-data beside the populated ones and come up as a clean install. MIGRATION.md’s claim that cast reattaches the old volumes in place did not survive contact with forge.
3. No user: root on zookeeper. This one did bite, at cutover: Cannot write to data directory /bitnami/zookeeper/data/version-2, exit 3, and ClickHouse then never started because its dependency was unhealthy. Found it by diffing the backed-up old compose rather than by reasoning about permissions: upstream had user: root, so the adopted volume’s /bitnami/zookeeper is root-owned while data/version-2 underneath is uid 1000. Kept running as root instead of rewriting ownership underneath a live dataset.
4. Minimal ingester config. I assumed only the prometheus receiver was missing. The diff showed resourcedetection and the entire metrics/prometheus pipeline were absent too, and since the hostmetrics and dockerstats pipelines both reference resourcedetection, the collector would simply have refused to start. patch-otel-config.py now constructs all three rather than assuming them, and takes the ClickHouse scrape host as argv[3] because Foundry renamed the container.

Restored explicitly for the same reason: the ClickHouse macros (replica: example01-01-1, shard: "01"). ReplicatedMergeTree ZooKeeper paths embed {shard}/{replica}, so Foundry’s derived names would have pointed every replicated table at a replica that has never existed. And metastore.kind: sqlite, because Foundry defaults to Postgres and every user, dashboard, rule and saved view lives in signoz-sqlite.

A retention bug that predates Foundry

logs retention FAILED on every run. Reproduced it deliberately and captured the raw HTTP response instead of theorising, which is the lesson I kept re-learning on 07-31. Manual retry returns 200, so it is a cold-start race: signoz-configure-retention gated only on signoz-signoz-0: service_healthy, and SigNoz’s /api/v1/health does not probe the telemetrystore. The v1 metrics and traces calls happen to win the race, the v2 logs call loses it, and logs retention silently stays unset while the job reports two thirds success. Now gated on ClickHouse healthy as well.

Second trap in the same area: the v1 GET ?type=logs keeps reporting -1 / expected 7 even when v2 retention is correctly set. GET /api/v2/settings/ttl?type=logs is the check that means anything, and it returned default_ttl_days: 25550.

Names changed

old	new
signoz	signoz-signoz-0
signoz-clickhouse	signoz-telemetrystore-clickhouse-0-0
signoz-otel-collector	signoz-ingester-1 (replicas, so no container_name)
signoz-zookeeper-1	signoz-telemetrykeeper-zookeeper-0
network signoz-net	signoz-network
~/.local/share/signoz/deploy/docker/	~/.local/share/signoz/pours/deployment/

Project name stays signoz and the UI port stays 8082, so cfg.url is untouched and nothing downstream (terraform, the MCP server, dashboards) noticed. Anything host-side that shells into a container by name does have to be updated.

End state

Server v0.135.0, collector v0.144.6, ClickHouse 25.12.5.44. Four long-running containers healthy. Data intact: 1,619,960,422 metric samples, 8,062,144 traces, 426,139 logs, all 8 databases, replication queue 0. All five metric receivers producing in a 2-minute window (clickhouse 824, hostmetrics 31 series / 3512 samples, docker_stats 15, macmon 29, collector self 35), OTLP traces and logs ingesting, filelog watching the Claude API bodies. Retention set cleanly. 5 dashboards, 5 rules, 1 channel, tofu plan clean.

Two false alarms in my own verification, caught before reporting either: jq '.data | length' on the dashboards endpoint returned 4 because it was counting wrapper-object keys rather than dashboards (all five are there), and tofu plan exiting 2 was the missing TF_VAR_ntfy_* secrets gating the channel resource to count = 0. Both were my instrumentation, not the system.

What was striking

Yesterday’s entry closed by arguing this should wait for a fresh head, on the grounds that my error rate on the 07-31 session was high. That was right, and the payoff showed up in method rather than in fewer mistakes: I diffed rendered output against running configuration before cutting over, and captured the retention job’s actual HTTP response instead of reasoning about why it might fail. Both are the same correction I wrote down three times yesterday (read what the system recorded before theorising about it), applied prospectively for once.

The deeper thing about Foundry is that a renderer is not a drop-in for the thing it replaces. Every one of the four omissions is a silence: no error, no warning, just a smaller document. Upstream is not wrong to render a minimal install, but “upstream’s supported path” and “what our deployment currently does” are different sets, and the diff between them is only visible if you go looking. That is now written into the ADR amendment as the thing to do before adding the next stack.

Next

1. Decide on ~/.local/share/signoz.pre-foundry (the old rsynced monorepo tree, moved aside not deleted). Pre-cutover compose backup is at ~/.local/state/signoz-backups/2026-07-31-pre-foundry/.
2. COMPOSE_STACKS no longer has entries. Upstream rendering changes are now invisible at update time and surface only as a diff in the pours/ tree after hm switch. Worth deciding whether that wants surfacing deliberately.
3. Still open from yesterday: the flapping-vs-paging tradeoff on the replication-queue alert, and rule_state_history_v0 history orphaned under the pre-recreate rule IDs.

Related

2026-07-31 SigNoz Dashboard v2 Break and Upgrade Readiness, 2026-07-27 SigNoz Upgrade Path Blocked on Foundry Migration, Homelab Services Architecture