Core Principle
For a 128 GiB M4 Max running llama.cpp through Docker Model Runner, the question is not “what fits” (almost everything under 130B fits) but which models have independent evidence, load on the runner’s llama.cpp build, and decode fast enough to matter. Ranked 2026-09-02 from Artificial Analysis, LiveBench, Arena, OpenRouter usage, and vendor cards, with provenance kept separate per LLM Comparison Sources and the reading rules in LLM Benchmark Reference.
Full tables and sources: M4 Max Model Shortlist.
Ranking
| # | Model | Q4 size | Independent | Vendor | Fit / runner |
|---|---|---|---|---|---|
| 1 | Qwen3.8-27B (dense) | 15 GiB | AA 52.0, AA Terminal-Bench 2.1 79.8, LiveBench 75.3, Arena 1435 | SWE-bench Pro 61.7 | loads (Qwen3.5 arch); ~25-35 tok/s expected |
| 2 | Qwen3.8-Flash-Next (125B-A6B + 51B n-gram) | 87-104 GiB | AA 55.8 (above Sonnet 5), TB 2.1 86.1, LiveBench 76.2 | SWE-bench Pro 62.5, JobBench 55.7 | needs iogpu.wired_limit_mb raise; arch newer than runner build, unverified |
| 3 | Ornith 1.5 35B-A3B (MoE, MIT) | 20 GiB | none | SWE-V 79, SWE-Pro 59.6, TB 67.8, GPQA 89.2 | loads (qwen35moe arch); 3B active = 4B-class speed |
| 4 | Laguna S 2.1 (118B-A8B) | 68 GiB (unsloth) | OpenRouter #16 by tokens | TB 2.1 70.2, SWE-Pro 59.4; no general benchmarks at all | needs llama.cpp b10087; runner is b9879; Metal NaN bug open |
| 5 | Muse Glimmer 30B (Meta, dense) | 15 GiB | AA 35.1, AA-LCR 80.0, Arena 1427 | SWE-V 76 | new arch, verify load |
Dropped: Qwen3.6-35B-A3B (AA 32, usage down 48%/wk), gpt-oss-120b (AA 24, SWE-Pro 16.2 on SEAL), Gemma 4 (AA agentic 14; a roleplay model in OpenRouter practice), Nemotron 3 Super / 3.5 Lightning (AA 26 / 24), Mistral Medium 3.5 (AA 30, dense 128B, slow), GLM-5.3-Flash and DeepSeek V4 Flash (no usable quant under 96 GiB).
Why This Matters
- Qwen3.8-27B is the first open model under 30B to beat Claude Haiku 4.5 on the AA index by 20+ points and sit within 3 of Sonnet 5. The independent Terminal-Bench 2.1 run (79.8) means the coding claim is not only a card number.
- Independent coverage lags releases by months: Scale SEAL, BFCL, Aider, LiveCodeBench, and Epoch have none of the August models. Ornith and Laguna have zero independent numbers; both are unverified until run on the permission-suggestion eval set.
- Thinking defaults on everywhere (
reasoning_effortxhigh on Qwen3.8). Poolside’s ablation puts the cost of turning it off at 10 to 24 points on agentic coding, so it is a per-task choice, not a global one.
Gotchas
docker model benchnever terminates on a thinking model; llama.cpp’sprint_timinglines indocker model logsalready carry tok/s.- Model Runner’s llama.cpp build (b9879, 2026-07-06) gates which architectures load. Anything merged after that date needs a load test.
- Poolside replaced the official Laguna S 2.1 Q4_K_M with an 89 GiB mixed-precision build on 2026-07-27 while the card still says 68 GB.
- macOS caps GPU-wired memory at ~75% of RAM; only Flash-Next crosses it here.
Related Ideas
- LLM Comparison Sources
- LLM Benchmark Reference
- Small Local LLMs as Judges
- Dimensions of LLM Quality
- 2026-09-02 Local Qwen3.5-4B Inference Baseline
- 2026-09-02 Open-Weight Model Shortlist for the M4 Max
Measured on the M4 Max (2026-09-02)
Qwen3.8-27B, unsloth UD-Q4_K_M (15.3 GiB), pulled as hf.co/unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M, loads on runner llama.cpp b9879 (arch reports as qwen35).
| Mode | Completion tokens | Wall time | Generation tok/s | Prompt tok/s |
|---|---|---|---|---|
enable_thinking: false | 161 | 7.3 s | 21.9 | 80.6 |
reasoning_effort: low | 1,546 (~1,400 reasoning) | 81.8 s | 18.9 | 104.0 |
Single stream. About one quarter of the 4B’s 88 tok/s, which is the dense-27B memory-bandwidth cost. chat_template_kwargs.enable_thinking works through the runner’s OpenAI endpoint; the 4B needed the same trick.