2026-09-04 The Host Metrics Were the Docker VM

What I set out to do

Two things. Turn a day of dashboard feedback into a durable guide rather than eleven commits of hindsight, and then work out why the hostmetrics dashboard reported CPU at 4% while the tmux statusline showed nearly 40%.

What I actually did

The guide came first: SigNoz Dashboard Design Guide in Atlas/References. Reading back over the LiteLLM dashboard session, the corrections fell into two families. One is that almost every turn was a rendering defect I had already declared shipped (a single-point series drawn as an invisible dot, connect-all bridging an idle half hour into what looked like a second model, $0 on every cost tick, unreadable seven-digit token counts corrected three separate times). The other is refusing workarounds that were never justified, which is what turned a mean labelled p95 into semconv bucket Views in the instrumentor. So the guide leads and closes on the same rule: a panel ships when the render is legible, not when the query is right. In between it covers panel type per question, layout as an argument, statistical honesty, units chosen against real magnitudes, colour as semantics, and the renderer traps.

Then the CPU number. It was not an aggregation artifact, it was a different machine. The hostmetrics receiver runs inside the signoz-ingester-1 container with root_path: /hostfs, which on Docker Desktop is the Linux VM’s /proc, and the compose env stamps host.name=atlas, os.type=darwin on whatever comes out. So the VM’s numbers arrived wearing the Mac’s name. Four independent tells, any one of which settles it: the cpu attribute had 8 values against hw.logicalcpu = 16; the states included softirq, steal and wait, which gopsutil never emits on darwin; /hostfs/proc/meminfo said 27.4 GB against 128 GB; and the VM’s loadavg was 2.67 while macOS was at 4.68. The VM idles while macOS works, which is the whole 4% against 33%.

Adam’s read was that a dashboard called host metrics showing Docker is simply a bug, which is right and is worse than mislabelling: a container cannot see macOS at all, so that panel was never going to be correct. The fix has two halves. launchd.agents.otelcol-host now runs otelcol-contrib natively (cpu, disk, filesystem, load, memory, network, paging) and exports to the ingester’s published OTLP port, and patch-otel-config.py takes a new host-name argument that adds a resource processor to the container’s hostmetrics pipeline only, so the VM reports as atlas-docker-vm while docker_stats and httpcheck keep the Mac’s name (they describe work the Mac is doing). Linux passes an empty string and is untouched, since there /hostfs really is the host. Commit 49fb982.

What was striking

The fact was already written down. signoz.nix has carried the line “hostmetrics measures the Linux VM” inside a docker-egress alert comment for months. It was true, it was accurate, and it sat next to a dashboard confidently reporting the VM as the Mac. Knowing a thing in a comment is not the same as the system acting on it.

A resource attribute is an assertion, not a measurement. OTEL_RESOURCE_ATTRIBUTES=host.name=atlas,os.type=darwin was added so docker_stats and httpcheck would attribute to the right machine, and it silently did the same for a receiver where it was false. One env var covering three pipelines is one env var too few.

I validated before switching, for once. otelcol-contrib validate --config, then a 40 second live run, then check that cpu8..cpu15 had actually appeared in ClickHouse, and only then wire it into launchd. Yesterday’s lesson applied one day later: the deploy is not the verification.

Final numbers, which is what closing this looks like: host.name=atlas now totals 15.8 CPU-seconds/second at 23.7 to 24.9% busy against 24.1% in top, and atlas-docker-vm totals 7.98.

Sixth pass: the tiles that would not resolve

With the Mac’s own metrics finally flowing, Adam looked at a five minute window and the Apple Silicon tiles were blank. They had been fine at an hour, which is the whole shape of this pass: three separate defects that only appear on short windows, and one wrong diagnosis of my own that only appeared because I tested at the wrong window.

The first was the four macmon tiles. time_aggregation: latest returns nothing on the scalar path at five minutes; avg at the same window returns a value, and latest at ten minutes and beyond is fine. Time-series panels on the same metric and aggregation render at five minutes, so it is the scalar path alone, and the step cannot be lowered to compensate because the server clamps anything under sixty seconds. The tiles now read avg with reduce_to: last, so each still reports the most recent minute rather than smearing the window.

The second was CPU Used, and it is where I got it wrong. Its filter is split across two queries and the second one, host.name IN $host.name, returned nothing. I compared five alternative spellings, found they all worked, and shipped a commit blaming a trailing space. Every one of those comparison runs was at a sixty minute window, where the bare form works too, so the experiment could not see the real variable and I read the difference off noise. Measured properly, ten runs per case: bare IN $variable at five minutes is null ten out of ten, and parenthesising it, using =, using a literal list, adding any AND clause, or attaching an explicit limit all return a value ten out of ten. Query A had been immune the whole time because its variable is followed by AND state != 'idle' rather than ending the expression. Parenthesised repo-wide, and the second commit says plainly that the first one was wrong.

What made the difference was capturing the panel’s own request instead of reconstructing it. The MCP rewrites reduceTo on formula inputs, which is what produced the phantom “empty at fifteen minutes too” result that sent me chasing a window-length bug. Wrapping XMLHttpRequest in the panel editor (the app uses XHR, and the network tool returns no bodies), clicking Run Query, and then replaying that exact payload while varying one field at a time is the technique that actually isolates these.

Seventh pass: rates wearing the clothes of totals

Adam then said the network number looked far too small, and that he expected gigabytes per day. He was right about the magnitude and the panel was the liar: it showed a per-second rate under a plain bytes unit, so 253 KiB meant 253 KiB every second, about 21 GiB a day. Both tiles now sum the selected window with increase plus reduce sum, so a one day range answers the question the title asks.

Each was checked against something outside SigNoz before shipping. Network: ClickHouse summing per-fingerprint max-min gives 661.55 MiB in an hour excluding loopback, the builder query gives 661.80 MiB, and en0’s counters since boot are 42.19 GiB over 2.92 days, about 14.5 GiB a day. Loopback is now excluded outright, since it reports the same bytes as both receive and transmit and most of it is this stack’s own telemetry to localhost:4317. Disk: ioreg’s IOBlockStorageDriver counters moved 0.77 GiB over a 150 second window where the query reported 0.58 GiB, so increase runs about a quarter low on a window that short because it drops the first partial bucket per fingerprint, and agrees to well under a percent at an hour. The machine writes 1.46 TiB in 2.92 days at rest, so the 82 GiB an hour that looked absurd to me was simply true, and Adam said as much before the measurement confirmed it.

What the GPU was doing

The GPU Usage tile showed 90% and neither of us knew what was driving it. powermetrics confirmed the bursts are real (100% active residency at 14.5 W in one sample, 1.2% and 15 mW in another, inside the same five minute capture) and then refused to attribute them: --show-process-gpu prints 0.00 for every process in all sixty samples, including the ones at full load. The column is an Intel-era leftover with nothing behind it on Apple Silicon. ioreg’s user-client census is not a substitute; it counts display resources and WindowServer always tops it.

Activity Monitor’s GPU Time column answered it in one screenshot: WindowServer at 1:05:56 and the two Firefox GPU helpers at 39:40 and 37:24, with everything below Ghostty’s six minutes being rounding error. So the spikes are compositing and browser rendering, which also killed my earlier guess that my own Chrome automation was responsible: Chrome is nowhere in the list. Recorded in SigNoz Dashboard Design Guide’s spirit as a memory rather than a panel, since there is no per-process GPU signal to put on a dashboard.

Top 3 tomorrow

1. There is still no true macOS memory, disk, filesystem or network history: everything before today under host.name=atlas is the VM. The panels are correct going forward, the past is not, and nothing on the dashboard says so.
2. Consider whether atlas-docker-vm deserves its own row or section, now that it is an honest host rather than an impostor.
3. Check the other dashboards for the same class of error: a number that is measured somewhere other than where its label claims.

Related

SigNoz Dashboard Design Guide · 2026-09-03 The Histogram Had One Series Per Request · 2026-09-01 The Rename Was the Visible Half · Homelab Services Architecture