OpenTelemetry counter metrics can be reported in two equivalent representations, and the backend’s query math must match the representation used at emission. Cumulative: each data point is the running total since process start, so rate() differentiates consecutive points and divides by elapsed time. Delta: each data point is the change since the last report, so rate() divides directly by the reporting interval. Both encode the same information; neither is inherently better. The fit depends on producer lifecycle: long-lived services (HTTP servers, daemons) suit cumulative; short-lived invocations (serverless, CLI commands, batch jobs) suit delta.
The SDK sets temporality per exporter preference (OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=cumulative|delta), and each data point carries a temporality flag. The backend is expected to honor it. Backends that don’t store the flag per-sample, or that cache a single “canonical” temporality per metric name, break as soon as an app changes its preference, either via an env-var flip or via an SDK default change across versions. The failure mode is silent: wrong formula over technically-valid samples produces empty aggregations or nonsense numbers, no exception, no log line. You only notice when a dashboard goes blank. The OTel spec permits producers to change temporality, so this is an ongoing backend correctness requirement, not “misuse.”
SigNoz #8961 is a worked example. Claude Code emitted delta through v2.1.112, then cumulative from v2.1.117 after we set the env-var preference. SigNoz stores per-row temporality correctly in samples_v4, but the metric catalog (signoz_metrics.metadata) accumulates rows for every (version, temporality) combination, and the API picks one with anyLast(temporality) GROUP BY metric_name. anyLast is non-deterministic and returned a stale Delta row; the planner then applied delta-formula rate math to cumulative samples. 1h queries returned empty. Prometheus’s cumulative-only origin and AWS CloudWatch’s cumulative-internal storage need cumulativetodelta and deltatocumulative Collector processors for the same reason: any reader that assumes one form, when the writer uses the other, needs translation somewhere.
Pick temporality deliberately and early; document it in the project’s observability config; change it only with eyes open, because each change is a soft breaking event for backends that cache. Prefer cumulative for long-lived processes targeting Prometheus, ClickHouse, or CloudWatch. Prefer delta only when the backend explicitly supports it (OTLP-native stores, Honeycomb, some New Relic configurations). When a producer version bump changes temporality, expect backend breakage and include a catalog reset or metadata purge in the upgrade procedure. Silent emptiness on a metric query whose underlying table clearly has data should immediately suggest “temporality mismatch between catalog and samples,” not “data is missing.” See Some OTel Feature Gates Are Permanent for an adjacent OTel-spec quirk that also breaks naive expectations.