2026-09-02 Local Qwen3.5-4B Inference Baseline
What I set out to do
Get a measured tokens-per-second number for the permission-suggestion hook’s local model (huggingface.co/unsloth/qwen3.5-4b-gguf, Q4_K_M, served by Docker Model Runner on :11434). The vault had none: Small Local LLMs as Judges carries a secondhand 60-80 tok/s figure for the 4B class on an M3 Pro, nothing measured here.
What I actually did
Ran docker model bench with defaults (concurrency 1, 2, 4, 8, 30s each). It ran 26 minutes and logged six context deadline exceeded failures before I killed it. A second run at concurrency 1,2 also would not finish. The numbers were already sitting in llama.cpp’s own print_timing lines in docker model logs, so I took them from there.
M4 Max, 128 GiB, llama.cpp b9879-metal:
| Concurrent requests | Per-request tok/s | Aggregate tok/s |
|---|---|---|
| 1 | 88.3 | 88 |
| 2 | 55-56 | ~111 |
| 8 | 28.1-28.5 | ~225 |
Prompt eval on the 4-27 token prompts read 150-210 tok/s, too small a prefill to trust.
What was striking
The bench is unusable on a thinking model. The default prompt asks for a 100-word summary and qwen3.5 spends 6,400 to 12,600 tokens reasoning before writing it. The bench sets no max_tokens, so at concurrency 8 a single request outlives the 5-minute per-request timeout. Worse, after each 30-second window it launches fresh requests instead of draining, so with long completions the run has no bounded finish time.
/no_think does nothing on qwen3.5. A direct probe with the prefix still returned 7,668 reasoning tokens. Thinking is only switchable via the enable_thinking request parameter, which the bench cannot pass.
The hook’s latency is thinking, not throughput. At 88 tok/s the model takes ~100 seconds to answer a request it could answer in 2 seconds without reasoning. Disabling thinking through litellm would dwarf any model or backend change. Same shape as yesterday’s finding that 77.5% of GLM 5.3-flash completion tokens were discarded reasoning (2026-09-02 The Setting Was Set and Nothing Sent It).
llama.cpp already logs the benchmark. Every request ends with prompt eval time and eval time lines carrying tokens/second. For a single-model baseline, docker model logs | grep eval after one real request is the whole measurement.
Open
- Pass
enable_thinking: false(orreasoning_effort) for the hook’s route in litellm and measure the wall-clock change. - Whether MLX would beat 88 tok/s here (the note claims 26-30% over llama.cpp on M3 Pro). The mlx backend is not installed in Model Runner.