2026-09-04 Agentic Tool Calls in the LiteLLM Proxy Dashboard

What I set out to do

Bring agentic workload visibility to the LiteLLM Proxy SigNoz dashboard (litellm-proxy-dashboard.json). The existing dashboard focused strictly on gateway-level request rates, latency histograms, and cost breakdowns, offering no visibility into agent loops, tool invocations, or multi-turn execution patterns visible in Claude Code telemetry.

What I actually did

1. Telemetry Audit: Audited ClickHouse spans (signoz_traces.distributed_signoz_index_v3) for resource_string_service$$name = 'litellm-proxy'. Discovered that LiteLLM already captures model-side tool calling choices synchronously on litellm_request spans:

   • gen_ai.completion.<N>.function_call.name
   • gen_ai.completion.<N>.function_call.arguments
   • gen_ai.response.finish_reasons (["tool_calls"])
   • metadata.user_agent (identifying callers: codex-tui, codex_exec, pydantic-ai, open-webui)
   • Multi-tool calls indexed sequentially (0, 1, 2, …)

2. Dashboard Design & Layout Ordering: Added a dedicated collapsible layout section titled Agentic & Tool Use to litellm-proxy-dashboard.json with 10 panels adhering to SigNoz Dashboard Design Guide. Placed the layout directly after Models & Cost (layout order: Traffic → Models & Cost → Agentic & Tool Use → Latency → Errors), preserving the cost-first hierarchy:

   • Hero Stat Row (y=0, h=3):
     • Tool Calls: Total tool calls unrolled across parallel completions via ClickHouse mapFilter and arrayJoin.
     • Agentic Turn Ratio: Percentage of requests where finish reason was tool_calls vs. conversational completions.
     • Distinct Tools: Count of unique tool types executed in the time range.
     • Tools per Turn: Average parallel tool calls per tool-invoking turn.
   • Activity & Trends (y=3, h=5):
     • Tool Usage Frequency: Top tools invoked (exec_command, apply_patch, obsidian_*, etc.) as a distribution pie.
     • Tool Invocations Over Time: Stacked bar chart grouped by tool name with dynamic interval derivation (toIntervalSecond(greatest(60, ...))).
   • Agent & Model Distributions (y=8, h=5):
     • Tool Calls by Client Agent: Grouped by connecting agent harness parsed from User-Agent.
     • Tool Calls by Model: Grouped by target model driving tool use.
     • Response Finish Reasons: Proportion of tool_calls vs. stop vs. length.
   • Inspection Stream (y=13, h=6):
     • Recent Tool Invocations: Full-width table with timestamp, client harness, model, tool name, argument snippet, tokens, cost, and execution latency.

3. Validation & Deployment:

   • Verified that all queries pass schema validation via nix build .#checks.aarch64-darwin.dashboard-metric-schema --no-link.
   • Ran just check to ensure pre-commit and formatting compliance.
   • Deployed via hm switch, which cleanly reconciled the updated layout order in SigNoz via OpenTofu (Apply complete! Resources: 0 added, 1 changed, 0 destroyed.).
   • Verified layout and panel registration against the live SigNoz API at http://localhost:8082/api/v2/dashboards/019fa204-64b1-7ee3-9397-0dc6919b04d2.

What was striking

• The telemetry was already in ClickHouse: LiteLLM records tool name, arguments, and finish reasons on every request out of the box. No custom logger or upstream patch was necessary to expose agentic activity; it was purely an extraction and visualization gap in the checked-in dashboard.
• Dynamic map unrolling via ClickHouse SQL: Models frequently issue parallel tool calls (completion.0, completion.1, etc.). Using mapValues(mapFilter((k, v) -> match(k, '^gen_ai\.completion\.[0-9]+\.function_call\.name$'), attributes_string)) cleanly unrolls arbitrary indices without hardcoded index limits or schema rot.