Achhina's Digital Garden

2026-07-17 GLM 5.2 Kimi K3 and Benchmark Note Refresh

4 min read

2026-07-17 GLM 5.2, Kimi K3, and Benchmark Note Refresh

What I set out to do

Understand why GLM 5.2 was all over the news despite unremarkable coding-agent benchmarks, then look up the new Kimi model, and capture the useful bits into the vault.

What I actually did

  • Researched GLM 5.2 (Zhipu / Z.ai). Conclusion: the story isn’t a benchmark crown — it’s near-frontier coding (~90-95%) at ~1/6 the cost, open weights, landing ~48h after US export rules restricted Anthropic’s top models for foreign nationals. Wrote GLM 5.2 - Near-Frontier Open-Weight Coding Model. Also nailed down that GLM = “General Language Model” (Zhipu/Tsinghua, from the 2022 blank-infilling architecture paper).
  • Researched Kimi K3 (Moonshot, released 2026-07-16). Largest open model announced (2.8T MoE, ~A50B active, 1M ctx, Kimi Delta Attention). Trades blows with frontier on several benches but at real Sonnet-level pricing (15); weights not out until 2026-07-27. Wrote Kimi K3 - Largest Open-Weight Model, Frontier-Trading Benchmarks.
  • Cross-checked the cited benchmarks against my own LLM Benchmark Reference and LLM Comparison Sources — most headline numbers are the distrusted class (release-day self-reports, Arena Elo).
  • On a hunch that the reference note (April 2026) had gone stale, did a fresh research pass and found real errors — including one I’d just introduced. Rewrote the note current as of today.

What was striking

  • My own edit was wrong. I’d added a caveat that the 2026 successor coding benchmarks were “mostly self-report with no independent leaderboard.” False — SWE-bench Pro (Scale SEAL), FrontierSWE (Proximal Labs; LLM Stats + Epoch AI), and SWE-Marathon (Abundant AI) all have independent boards. Lesson reinforced: don’t append news-summary claims to a durable reference without verifying primary sources.
  • The independent FrontierSWE board is led by Claude Fable 5 (~0.90), not the open models — so Kimi K3’s self-reported “~81 FrontierSWE” doesn’t survive contact with the independent leaderboard. Exactly the failure mode the note warns about.
  • “Program Bench” appears to be a garbled name from a news summary; couldn’t verify it exists. Dropped it.
  • The benchmark landscape rotted in three months: SWE-bench Verified (~88%) and GPQA Diamond (~94%) are now near-saturated. New live differentiators: ARC-AGI-2, HLE, FrontierSWE, SWE-Marathon, and the METR task-completion time-horizon metric (doubling ~every 4 months, frontier past ~16h).
  • Best single reading rule captured: agent benchmark numbers measure a system, not a model — bare-model vs scaffolded vs full-system diverge 30-50 pts on the same benchmark.

Notes captured

Top 3 next

  1. Build a “China open-weight cluster” MOC tying GLM 5.2 - Near-Frontier Open-Weight Coding Model, Kimi K3 - Largest Open-Weight Model, Frontier-Trading Benchmarks, and DeepSeek.
  2. Re-check GLM/Kimi rankings on the independent boards (Scale SEAL, LLM Stats, Epoch) once Kimi K3 weights drop 2026-07-27.
  3. Consider whether a cybersecurity benchmark axis (Cybench / CyberSecEval) is worth adding to LLM Benchmark Reference.

Dimensions of LLM Quality · LLM Comparison Sources

OpenRouter Interceptor: JAI chat width + capture ancestor chains

What I did

Widened the JanitorAI chat, and root-caused a CSS mystery along the way by extending the diagnostic capture tool instead of guessing.

  • First attempt targeted the message rows (li[class*="_messageDisplayWrapper_"]). Inert: those rows are width: 100% of a parent capped elsewhere, so a max-width on them does nothing.
  • The diagnostic capture was scoped to the selected message/slider subtree, so the ancestor holding the cap was invisible. Extended DomCapture with two width-relevant layout chains: ancestors (selected element up to <body>) and inputAncestors (chat textarea up to <body>), each recording clientWidth plus computed max-width/width/margins. Mirrored in both the TS buildDomCapture and the serialized page-context template; built TDD-first against happy-dom.
  • The chains named the caps immediately: message column at main._messagesMain_ (max-width 672px) and composer at div._chatInputInner_ (max-width 672px), both JAI defaults.
  • Shipped janitorai-chat-width.ts: overrides both to a left-biased 20% / 5% offset (75% width), tunable via MARGIN_LEFT / MARGIN_RIGHT constants.

What was striking

  • Classic “fix the visible element” trap. My rule did match the message row (confirmed in the capture’s matchedCssRules) yet changed nothing, because the row’s parent was already narrower than the cap. On hashed-class UIs like JAI, guessing selectors is a dead end; instrument and read the evidence.
  • Extending the capture rather than pasting one-off console snippets paid off twice (messages, then composer) and left a reusable layout probe behind. The user explicitly pushed for this (“extend the debug capture for what you need”) and it was the right call.

Shipped

Two commits on main (local, unpushed): feat(capture) ancestor/input chains, feat(janitorai) chat width. 432 unit tests, typecheck, and lint green.

Graph View