Achhina's Digital Garden

2026-06-14 Backend-Scoped Telemetry Ownership

5 min read

2026-06-14 Backend-Scoped Telemetry Ownership

The problem

My personal dotfiles run on my work machine too. Work uses DataDog, which is delta-temporality native. My personal config hardcodes OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE = "cumulative" because that is what self-hosted SigNoz/ClickHouse prefers (see OTel Metric Temporality). On the work machine localhost:4317 is not my SigNoz, it is the corporate DataDog agent, so just launching Claude Code shipped my personal knobs straight into corp’s pipeline.

Two knobs did damage. First, temporality: corp never sets it (delta is the Claude Code default they rely on), so my cumulative override was not a conflict, it was an addition of a poison knob. claude_code.usage.cost for subagents lacks a session.id, so concurrent sessions collapse into one time-series identity; cumulative imposes a per-series monotonicity contract; the collapsed series is non-monotonic; DataDog’s cumulative→delta conversion reads each downward step as a counter reset and re-counts the full value, producing astronomical cost deltas in the corporate log sink. Second, and quieter but worse: OTEL_LOG_USER_PROMPTS / OTEL_LOG_TOOL_CONTENT / OTEL_LOG_RAW_API_BODIES were on, so prompt and body content was flowing to a sink I do not control.

The real root cause was not the missing session.id (delta is additive and immune to it). It was that endpoint, temporality, and content-logging are all properties of one backend, bound together and unconditionally on, with no model of which backend receives telemetry on a given host.

What I set out to do

Make harness telemetry (Claude Code, LiteLLM) emit OTel config only on hosts where my config actually owns the backend. Express the policy once per machine, not as literals duplicated across harness modules.

The process

I validated the diagnosis first (it held up exactly), then brainstormed the design rather than jumping to code. The key insight the user pushed me toward: separate the decisions from the encoding. Decisions (does this host own a backend, what temporality does that backend want, is content logging permitted) are cross-cutting and belong in one place; encoding (which env var name each harness uses) is irreducibly per-harness. A full programs.telemetry god-module that renders each harness’s env would be the wrong abstraction for two consumers with divergent schemas.

The dependency direction took a couple of iterations to get right. My first cut had harnesses reading programs.signoz.*, which the user correctly shot down: a harness should not know SigNoz exists. The seam that survived: backends register into a local.telemetry decision layer; harnesses read resolved outputs; no harness ever names a backend. Enablement is presence-driven (no “is this work” flag) so the work machine, registering no backend, goes silent by construction. We also worked through multi-backend (send to both local SigNoz and managed DataDog): it resolves cleanly because a harness can only emit to one endpoint, so fan-out lives behind a collector and stays out of the harness contract. Deferred it with a ≤1 backend assertion as the placeholder.

I dug through ADRs and the Obsidian vault to find where the delta→cumulative switch was decided. There was no ADR; the rationale lived only in OTel Metric Temporality and an April reflection. So part of the deliverable became finally recording that decision in-repo (ADR 0014). I wrote the spec, the ADR, and a bite-sized implementation plan, then executed it with subagent-driven development: a fresh implementer per task plus two-stage review (spec compliance, then code quality).

The work

  • nix/home-manager/modules/telemetry.nix (new): the local.telemetry decision module + resolver + the multi-backend assertion.
  • signoz.nix: added programs.signoz.enable; when enabled it registers itself as a backend (endpoint, logs-HTTP endpoint, metricsTemporality = "cumulative", contentOk). Temporality now lives next to the ClickHouse that prefers it, not as a literal in a harness. Made url a computed option default so consumers survive the stack being disabled.
  • claude.nix: OTEL env split into three gated tiers (backend-coupled / enable / content), hook emitter no-ops when inactive.
  • litellm/default.nix: gated both emission paths.

What was striking

The meta-lesson. Every per-task review passed, but the holistic final integration review caught a Critical leak that env-var gating alone missed: LiteLLM’s instrumentor Python entrypoint constructs OTLPSpanExporter() with no endpoint argument, which defaults to http://localhost:4317. On the work host that is corp’s agent, so LiteLLM would still ship FastAPI/aiohttp spans to corp even with telemetry “off.” Gating the wrapper’s env exports was necessary but not sufficient; the separate process entrypoint that builds its own exporter had to be gated too. Only a review that looked at the whole emission surface, not one diff at a time, would have found it. Strong argument for keeping a final whole-implementation review even after disciplined per-task review.

Two smaller surprises: (1) the cumulative-temporality choice was genuinely backend-scoped all along and had never been written down in the repo until now, despite causing a real incident; (2) Home Manager’s extendModules bypasses its assertion enforcement (moduleChecks), so testing an assertion fires means reading config.assertions directly, not expecting a throw on the extended config.

Resolution

Merged to main (fast-forward, 8 commits), deleted the branch, applied via home-manager switch. Verified no regression on this trusted host: live ~/.claude/settings.json still carries cumulative temporality and the full key set. On the work clone the one-line programs.signoz.enable = false makes the stack stop deploying, nothing registers, and both harnesses go silent; local.trusted = false independently disables content logging. main is unpushed (local merge by choice). Out of scope but flagged in ADR 0014: open-webui.nix and sillytavern.nix have the same unguarded otlpPort pattern and want the same treatment.

Graph View