Homelab Services Architecture

Architecture of the self-hosted services running on the Mac (atlas). Everything below runs in Docker on atlas except LiteLLM (host launchd agent), Glance (host launchd agent), and the off-box pieces on athena (public Obsidian MCP since 2026-07-03, superseding the OCI box; ntfy). Ports listed are the host-published ones (verified against running containers, which are ground truth when they disagree with older notes).

Entry point: the Glance dashboard (:8280 local, https://atlas.tail0cef90.ts.net:8444 from the tailnet) fronts everything — per-service up/down monitors, live stats (qBittorrent, *arr queues, SigNoz alerts), the Docker container list, and click-through links that route via tailnet URLs so they work from the phone. HM module: nix/home-manager/modules/glance.nix.

Management models

Two coexist:

• Plain Docker Compose — media-stack (~/docker/media-stack) and SigNoz (~/.local/share/signoz/deploy/docker).
• Nix Home Manager → Docker — open-webui, sillytavern, sillytavern-envoy. Nix renders the docker-compose.yaml + env/config, then a mkComposeDeploy activation script copies them to real on-disk paths (Docker Desktop can’t follow /nix/store symlinks) and hash-gates container recreation on hm switch. That’s why their compose “working dir” shows /nix/store. See Backend-Scoped Telemetry Ownership for how telemetry config is gated per host.

Stack 1 — Media automation (media-stack-net bridge)

Declarative, API-first *arr stack. All 11 containers share one bridge network and address each other by service name. A run-once orchestrator (alpine) configures every service over REST at startup, so there is zero manual UI setup.

Request lifecycle:

seerr (5055, discovery/requests, Plex auth)
  └─> sonarr (TV) / radarr (movies)      ← recyclarr syncs TRaSH quality profiles @daily
        └─> prowlarr (indexer aggregator)
              └─> flaresolverr (Cloudflare/CAPTCHA bypass)
        └─> qbittorrent (downloads → shared /downloads)
  bazarr watches sonarr/radarr → fetches + alass-syncs subtitles

• nginx:8080 is the unified gateway: path-routes /sonarr /radarr /prowlarr /bazarr, rewrites /qbt/ (qBittorrent has no URL-base support), and uses Docker’s embedded resolver with variable proxy_pass so it survives container restarts. seerr is NOT proxied (no subfolder support) — hit it directly on 5055.
• Internal *arr ports (8989/7878/9696/6767) are not published; only 8080, 5055, and qBittorrent’s 6881 reach the host.
• Shared volumes under $HOME/media/{downloads,tv,movies}: qBittorrent writes downloads, sonarr/radarr import to tv/movies, bazarr reads all three. API keys live in git-ignored .secrets/generated/*.env, generated at setup and cross-sourced between services.

Runtime facts not captured in the repo:

• qBittorrent WAN reachability: Docker bridge silently drops SSDP multicast, so UPnP discovery failed and 6881 was unreachable. Worked around with direct SOAP AddPortMapping to the router gateway. See Persist qBittorrent UPnP Port Mapping in Media Stack.
• Prowlarr indexer priorities live only in prowlarr.db and revert on down -v; project to encode them is Persist Prowlarr Indexer Priorities in setup-prowlarr.sh.
• Recovery history: 2026-05-16 Media Stack Download Pipeline Recovery.

Stack 2 — SigNoz observability

Heaviest stack by far (ClickHouse alone ~14% CPU, 2.6 GiB RAM). Topology: OTel collector → ClickHouse (5 signal DBs: traces/metrics/logs/meter/metadata) ← signoz query service (UI on 8082). ZooKeeper coordinates ClickHouse’s ReplicatedMergeTree tables.

Collector ingests from five sources: OTLP gRPC/HTTP on 4317/4318 (the only published collector ports — where LiteLLM, open-webui, and the SillyTavern envoy send traces); Prometheus scrape (self, ClickHouse :9363, optional macmon); hostmetrics + docker_stats (via mounted docker.sock); and a custom filelog/claude_raw_bodies receiver reading Claude API bodies. Traces also feed signozspanmetrics to generate RED metrics.

Deliberate deviations from stock SigNoz (docker-compose.override.yaml):

• OpAMP manager disabled (dropped --manager-config) — the wrapper pushed an all-nop config on boot that neutralized every pipeline. See 2026-05-16 SigNoz Dockerstats and OpAMP Investigation.
• Retention pinned to ~70 years (25,550 days), intentionally — a prior 876,000-day setting overflowed ClickHouse’s UInt32 DateTime and destroyed all trace data. API split: v1 for metrics/traces, v2 for logs. See 2026-04-11 SigNoz ClickHouse TTL Overflow Post-Mortem.
• ClickHouse :9363 Prometheus scrape + replication-queue alert added after a single-node ReplicatedMergeTree stuck GET_PART retry pinned ClickHouse+ZooKeeper at ~120% CPU for weeks. Suspect this first if ClickHouse CPU climbs again. See 2026-06-04 SigNoz ClickHouse Replication Queue CPU Burn. Its alertOnAbsent dead-man’s switch was removed 2026-07-13: it kept firing as nodata with the queue at 0, because atlas sleeps whenever it’s off AC (see below) and gaps the scrape past 30m. A stuck queue can’t self-heal, so it’s still >0 when the scrape resumes — absent data was never evidence of it. See 2026-07-13 ClickHouse Replication Alert Was Atlas Sleeping.
• Telemetry gaps are usually atlas asleep, not a broken collector. pmset is sleep 0 on AC but sleep 1 on battery, so unplugged (lid closed) the Mac drops into maintenance sleep and only surfaces in ~45s DarkWake windows every ~10 min. The CPU halts, so Docker’s VM isn’t scheduled and every container freezes — services aren’t down, they’re intermittent. The signature: every signal gaps at identical timestamps across unrelated scrape targets, traces and logs, with no container restarts and no collector errors. Check pmset -g ps before suspecting the stack.
• macOS caveat: containerized hostmetrics reads the Docker Linux VM, not the Mac, so Plex runs natively (port 32400) rather than instrumented here.
• Telemetry temporality is cumulative (ClickHouse requirement), gated on programs.signoz.enable so it can’t poison a work DataDog backend — Backend-Scoped Telemetry Ownership.

Stack 3 — AI chat (all routed through LiteLLM)

The hub is LiteLLM on the host at :4000 (a launchd agent, not a container), exposing an OpenAI-compatible /v1 backed mainly by OpenRouter. Everything reaches it via host.docker.internal:4000.

open-webui (127.0.0.1:7878→8080): OPENAI_API_BASE_URL=…:4000/v1, default model openrouter/deepseek/deepseek-v4-flash, image gen via openrouter/google/gemini-3.1-flash-image. Runs ENABLE_PERSISTENT_CONFIG=False — all config from the env file or a post-start SQLite seed each boot. JWT key persisted via WEBUI_SECRET_KEY_FILE on the data volume (else it rotated every hm switch, killing sessions — 2026-06-21 Open WebUI Deployment and Auth Cascade). Bind-mounted boot-time Python patchers fix real bugs: image cost/double-render (2026-06-22 OWUI Per-Message Cost Tracking), mobile download (2026-06-22 OWUI Mobile Image Download), and a LiteLLM pre-call hook stripping unsupported tools from image models plus a 16 MiB SSE buffer for streamed base64 (2026-06-21 Open WebUI Image Models On Demand). Exposed to phone via Tailscale Serve (https://atlas.tail0cef90.ts.net → loopback 7878), not Cloudflare; the tailnet is the auth boundary.

sillytavern (0.0.0.0:8000) + sillytavern-envoy: SillyTavern doesn’t talk to LiteLLM directly — it goes through Envoy, whose only job is observability. Envoy listens inside the network, forwards to LiteLLM on the host, and emits an OTel span (100% sampling, service_name: sillytavern) to SigNoz :4317 for every request — tracing without touching SillyTavern’s code. Behavioral settings (reasoning.auto_parse, continue_prefill, “Prompt Post-Processing = Strict”) live in the writable data/.../settings.json volume, not the Nix-managed config.yaml — so they can drift, and nothing in the repo enforces them.

Two of them are load-bearing and coupled, which is invisible in the UI:

• Prompt Post-Processing = Strict (from 2026-06-17 DeepSeek Empty Replies via OpenRouter) fixes /impersonate returning empty, by re-roling a trailing system message to user — DeepSeek returns an empty completion on any prompt ending in system.
• continue_prefill is now OFF (inverted 2026-07-13; it used to be on). On OpenRouter chat-completions the chat template appends an EOS after a trailing assistant message, so a “prefill” is really a finished turn the model must talk past — Continues came back at a median 18 tokens vs 579 for fresh replies. Turning prefill off routes Continue through the nudge path, which lands ~700. That only works because Strict re-roles the nudge to user. Turn Strict off and you break both. See 2026-07-13 SillyTavern Continue Fixed by Turning Prefill Off.

Stack 4 — Infra & the off-box piece

• watchtower auto-updates the media containers; nginx (3 MiB) is the media gateway.
• Remote Obsidian MCP is the one thing not on the Mac: a NixOS A1.Flex box running mcp-proxy (loopback :8788) + mcpvault behind a cloudflared tunnel gated by Cloudflare Access (Google OAuth) at obsidian-mcp.achhina.com. The vault git-syncs between Mac (obsidian-git, 30s) and box (systemd timer) with a keep-both conflict policy. The Mac keeps only a loopback MCP bridge that open-webui consumes. See 2026-06-21 Remote Obsidian MCP Connector and Remote Obsidian MCP Cloud Host. Host naming follows Hostname Naming Convention.

The two cross-cutting buses

1. LiteLLM :4000 — the LLM control plane. open-webui (direct), SillyTavern (via Envoy), and Claude tooling all funnel through it; it handles OpenRouter routing, DeepSeek reasoning stripping, and image-tool stripping.
2. SigNoz :4317 — the telemetry sink. LiteLLM, open-webui, the Envoy proxy, hostmetrics, docker_stats, and Claude API bodies all land in ClickHouse, queryable at :8082.

Port reference

Service	Host port	Notes
nginx (media gateway)	8080	path-routes the *arr UIs
seerr	5055	direct, not proxied
qbittorrent	6881 (TCP/UDP)	peer/DHT; UPnP-mapped to router
SigNoz UI	8082	query service
OTel collector	4317 / 4318	OTLP gRPC / HTTP
open-webui	127.0.0.1:7878	Tailscale Serve for phone
sillytavern	8000	via Envoy → LiteLLM
LiteLLM	4000	host launchd agent, not a container
Glance dashboard	8280	host launchd agent, not a container
Plex	32400	native macOS, not containerized
Obsidian MCP (remote)	athena, Cloudflare Access	via cloudflared; OCI box kept as $0 fallback

Tailscale Serve surface (atlas.tail0cef90.ts.net, tailnet-only)

Phone-facing HTTPS fronts of the loopback services. Declared next to the service each fronts (home.nix, media-stack.nix, sillytavern.nix, glance.nix).

Serve port	Backend
443	open-webui :7878
8443	SigNoz :8082
8444	Glance :8280	the dashboard itself
8445	media gateway :8080	*arr subpaths; Komga needs trailing slash
8446	seerr :5055
8447	suwayomi :4567
8448	sillytavern :8000	whitelist needed CGNAT 100.64.0.0/10

On the gateway, reachability is authorization (nginx injects X-Auth-User); the tailnet is the auth boundary, same model as open-webui. athena’s ntfy serves at https://athena.tail0cef90.ts.net (:443).