Findings
Dated findings, written as findings rather than numbers. Full tables live in Results.
2026-11-15/16 — small models close in; a 7B beats a 36B MoE (HX 370)
Tested on the HX 370 tier: K2-Horizon-7B (dense), Recursive Flint 40B-A2.5B (int4), JetBrains Mellum2.1-12B-A2.5B (MTP), against the standing references (Qwen3.6-MTP 235, K2-MoVA-36B 226).
- K2-Horizon-7B (230/243, 151 min) beats its own 36B MoVA sibling (226/243, 147 min) — same wall time, better accuracy, ~3× less memory. The 36B is superseded on this hardware; its only remaining edge is deletion safety (clean trim-plan discipline where the 7B proposed unsafe deletes 3/3).
- Flint (230/243, 186 min) ties the 7B with an opposite personality: the best sustained decode we’ve measured on an iGPU (23–27 tok/s for hours, only ~12% droop — bytes/token drives throttle depth, not model size), near-perfect delete discipline, but an enormous reasoning appetite (240k tokens/batch) and one analysis timeout. Its entire score gap to the leaders is one timeout plus never reporting the impossible-instruction deviation.
- Mellum2.1 (158/243, 65%) is the bench’s cleanest negative result: fast decode with a working
speculative head (29 tok/s, ~46% acceptance) but it cannot reliably produce the structured
answer form (
parsed_jsonfails ~13 times), which collapses every analysis task. Vendor SWE-bench (47%) predicted a low score; the failure mode turned out to be output-form discipline, not reasoning. - Honest deviation reporting remains a K2-family trait: MoVA 3/3, K2-7B 2/3, everyone else 0/3 (now across Qwen, Flint, Mellum). If your agent will meet instructions it can’t follow, this is the trait to select for.
- The northstar for this tier (“beat Qwen3.6’s speed at Qwen3.6’s score”) remains unmet — but the gap is now fully diagnosed: it’s thinking verbosity and one timeout, not decode throughput. The dense half of the goal is bandwidth-bound closed: no current dense model smaller than Qwen3.8 (27B) reaches its score class.
2026-11-16 — GLM 5.3 Flash ties the best local MoE at a fraction of the tokens
First full 9-task pi grid of a remote model (OpenRouter, Z.AI fp8 endpoint, no skill — the
flash-zai control): 235/243 (96.7%) in 9 minutes wall, 366k prompt / 17.6k completion
tokens for the whole batch.
- Exact tie with Qwen3.6-35B-A3B-MTP (235/243 on HX 370, 236 on Halo) — same section shape:
analysis 72/75, plan 81/84 (minus the
10_pass_line_honestquirk ×3 — now a five-family pattern), code 82/84. - 8 points below the dense ceiling (Qwen3.8-27B: 243/243), with the same two failure classes that separate everyone from the dense model: deviation reporting (1/3 — marginally better than Qwen3.6’s 0/3) and one analysis discipline slip.
- 13× fewer thinking tokens than the local MoEs for the same grid (17.6k vs 230–240k completion tokens). Local MoEs pay a large verbosity tax for a matching score.
- Wall time is provider-side and deliberately not comparable; the home-page chart covers accuracy only for this reason.
Earlier — 2026-11-01/06 (both machines)
- The full original model set (Qwen3.8/Qwen3.6/Ornith/Gemma×2/Nemotron) ran on Strix Halo; Qwen3.6 + K2-MoVA re-ran on the HX 370. Rankings transferred across machines within ~1 point (see Results — Strix Halo).
- Plan tasks don’t separate strong models; analysis tasks do. The recurring weak spot of local MoEs is honest deviation reporting — only the dense Qwen3.8 and the K2 family do it reliably.
- Sustained decode droop on the HX 370 was quantified and root-caused to a firmware power skin (65 W burst → 54 W sustained), not heat. See HX 370 and Known limitations.