Changelog

Every change to published numbers, and why it happened. Corrections are never silent: if a score moves because of a harness or scorer bug, the old number stays visible here with the reason.

Two kinds of entry live here:

  • Benchmark corrections — a published score was wrong, and we say what the bug was.
  • Site and pipeline — layout, data export, version bumps.

Current site version: 1.3.7 (see the badge in the header).


1.3.7 — 2026-11-19: bars rescale when cloud models are hidden

Unticking “include cloud models” used to leave the scale untouched, so hiding two remote bars left empty track to the right of every local model. The scale is now computed from the bars actually shown:

  • Cloud hidden: renormalise to the best visible bar — the fastest local model fills the track (Qwen3.5-4B at 30.1 tok/s instead of standing at a quarter of DeepSeek’s 124.4).
  • Cloud shown: the absolute scale, unchanged — accuracy against the 243 checks, speed against the fastest model overall.
  • The printed value never changes: numbers stay absolute, bars only re-scale. Both chart footers say so.

Applies to both charts. On the accuracy chart this is currently a no-op, because the best local model (Qwen3.8-27B) is already at 100% — the renormalisation only bites where a remote model holds the top bar, which is exactly the speed chart.


1.3.6 — 2026-11-19: home page text aligns with the charts

Correction to 1.3.5: widening the whole data column was the wrong lever. The width that was there before was fine — what looked misaligned was the prose under the charts, which was capped at ~70 characters while the charts ran the full column, so the two blocks had different right edges.

  • The data column is back to 76rem.
  • The home page’s article is exempt from the prose cap and spans the same width as the charts, so headings, paragraphs and bars share both edges. Font size and line height are nudged up a little to carry the longer measure.
  • Fixed a sentence that had gone stale with 1.3.2: it said the chart covers accuracy only, when there are now two charts.

1.3.5 — 2026-11-19: results page uses the full width

The data column was capped at 76rem, which on a wide screen left dead space to the right of the tables while the tables themselves were being clipped — the last two columns ran off the edge.

  • Data pages now stretch to 104rem (capped by the viewport); prose inside them stays at ~70 characters, so the layout widens without the text doing so.
  • Tables fit again: the backend column no longer repeats ”… / llama.cpp b11846 / Vulkan” in every row (the build is constant per section and already named in the section note), so it reads “Lemonade 2026.46.2 · Vulkan”. Numeric columns are pinned to nowrap, long text columns are allowed to wrap, and the header labels get the same treatment.
  • Per-model failed checks are a responsive card grid (auto-fill, ~22rem minimum) instead of one tall stack of full-width cards, so a wide screen shows three columns and the section no longer needs scrolling to compare models.
  • Fixed the section note on the headless-CachyOS block, which rendered literal <code> tags because it is a JS string, not markup.

1.3.4 — 2026-11-19: spacing on the chart legend

The legend chips (“Strix Halo / HX 370 / Remote”) sat flush against the sentence and against each other, so they read as one run-on block. They are now a flex legend with a real gap between chips and a clear margin from the text, slightly taller chips with a touch of padding, and they wrap as a group on narrow screens instead of breaking mid-chip.


1.3.3 — 2026-11-19: chart layout pass — wider, aligned, less crowded

Both home-page charts were squeezed into a 72-character text column, which wrapped long model names, pushed tier labels onto their own lines and left the “include cloud models” checkbox floating under the subtitle. Fixed:

  • Wider container for data pages (76rem), with running prose still capped at ~70 characters so paragraphs stay readable. The home page now uses the same wide column as Results and Models.
  • Fixed four-column rows: name · tier · bar · value. Long names ellipsize instead of wrapping, the tier is a separate aligned column rather than a tag glued to the name, and values are right-aligned in a constant-width column, so bars start and end at consistent x-positions.
  • Header reorganised: title and the toggle share the top row (the toggle is now a pill on the right), the legend chips moved into the subtitle line where they belong.
  • Row hover highlight, tighter row height, and a narrow-screen fallback that moves the bar under the label instead of squeezing four columns.

No data changed. Also sharpened the speed chart’s note on the missing Strix Halo tier: the numbers exist as seed data without token counts, so the tier appears in accuracy and in the Results tables (with its grid time) but cannot be ranked on throughput until that box exports its runs.


1.3.2 — 2026-11-19: a speed chart next to the accuracy chart

The home page had one ranking (accuracy). It now has two, built from the same grids with the same one-bar-per-model-per-tier rule and the same “include cloud models” toggle, but sorted by speed.

Metric: effective output throughput — completion tokens ÷ wall-clock seconds, summed over the 27-run grid. That is output tokens of agent work produced per second, which is what you actually wait through. It is deliberately not raw decode tok/s: decode numbers exist only for the batches where llama-server metrics were reachable from the controller, so a decode-only chart would cover half the table. Where both exist they agree closely (K2-7B: 10.8 tok/s decode vs 9.8 effective).

Model Effective tok/s
DeepSeek V4.1 Flash (remote) 124.4
GLM 5.3 Flash (remote) 32.0
Qwen3.5-4B-MTP 30.1
Qwen3.6-35B-A3B-MTP 25.8
Recursive Flint 40B-A2.5B 21.6
JetBrains Mellum2.1-12B 13.3
K2-Horizon-MoVA-36B-A4B 10.6
K2-Horizon-7B (dense) 9.8

Things the chart makes visible that the accuracy ranking hides:

  • The value leader is also fast: Qwen3.5-4B is second among local models on speed and first on accuracy per gigabyte. There is no trade-off to explain.
  • The 4× dense-model penalty is explicit: K2-Horizon-7B (dense, 7B) is the slowest bar in the table, slower than a 36B MoE, because it reads every weight per token.
  • Remote is not a little faster, it is a different regime: DeepSeek is 4× the fastest local model and 12× the slowest.

Caveats stated in the chart itself: throughput rewards verbosity (a model that writes more still wins if it writes faster), remote wall time is provider-bound, and Strix Halo is absent because its rows are seed data without token counts.


1.3.1 — 2026-11-19: DeepSeek V4.1 Flash at medium effort — 239/243, for 5 cents

New remote measurement, full 9-task grid, 3 reps, pi harness, 239/243 (98.4%) in 6 minutes.

Score 239/243 (98.4%) — analysis 75/75, plan 81/84, code 83/84
Effort pi’s thinkingLevel: medium (the default; not high)
Tokens 770k prompt / 47k completion for the whole grid, 5.4 requests per run
Cost $0.0506 for all 27 runs — $0.0019 per run, as reported by the provider
Provider OpenRouter → Nebius, not pinned
Losses the universal 10_pass_line_honest ×3, one deviation_reported
  • Best score outside the dense 27B (243/243), ahead of every local MoE and ahead of GLM 5.3 Flash (235). Its analysis block is perfect — better than any local model on this benchmark.
  • It reports the code-3 deviation in 2 of 3 runs, which no local MoE except the K2 family does.
  • Cost accounting now works. pi does report real money per turn (usage.cost.total, flagged totalFromProvider), but the harness read a different field and stored None. The 27 runs were back-filled from their event logs, and new runs capture it: the export now carries cost, which makes price-per-score a publishable number instead of a guess.
  • Effort disclosure: this is medium, because baseline sends no thinking override and pi defaults to medium. High effort on this model is unmeasured — worth one batch, since the entire gap is 4 checks.

Also in this release

  • The results page’s remote section no longer hardcodes “Z.AI fp8, pinned”: provider and pinning are now disclosed per model, because the section holds two providers with opposite reproducibility properties (GLM is pinned, DeepSeek is not).
  • No DeepSeek logo asset, so its chart bar renders without one rather than shipping a broken image.

1.3.0 — 2026-11-19: a 3 GB model at 93% — the value leader

New model, first result measured under the corrected harness end to end: Qwen3.5-4B-MTP — 226/243 (93.0%) in 53 minutes, from 2.99 GB of weights, 40 tok/s with MTP.

Model Weights Score Time Score per GB Decode (MTP)
Qwen3.6-35B-A3B-MTP (cachy-tuned) 23.8 GB 234/243 (96.3%) 73 min 9.8 36 t/s
Qwen3.5-4B-MTP 3.0 GB 226/243 (93.0%) 53 min 75.3 40 t/s
JetBrains Mellum2.1-12B 9.8 GB 204/243 (84.0%) 309 min 20.8 28 t/s
K2-Horizon-MoVA-36B-A4B 22.7 GB 231/243 (95.1%) 113 min 10.2 —

Eight checks separate the 3 GB model from the 20-plus GB one, and they are the ones you would predict: hidden acceptance tests on code tasks, file-scope discipline, and plan-3 (it performed a step it had correctly decided to stop before).

Two results worth flagging:

  • It reports the code-3 deviation in 1 of 3 runs. Qwen3.6-35B-A3B has never done that in nine runs. The smallest model tested is the one that occasionally admits it could not follow the plan.
  • analysis-2 was perfect (30/30) at ~7 minutes per rep, even though this model’s MTP acceptance collapses from ~60% to ~19% past 10k tokens of context and its decode falls 40 → 16 tok/s.

Data pipeline

  • mem_gb (weights on disk) is now recorded with every run and exported. Value comparisons need the memory side of the ratio, and it cannot be reconstructed later from a score table. Backfilled for the existing local runs from the same server endpoint that reports file sizes; remote models have no such number and stay blank.
  • The Qwen3.5-4B measurement is the first full grid produced end to end under the corrected harness: 32k per-response budget, widened answer parser, hardened tool environment, 3 reps, zero timeouts, vendored sampling (temp 0.6 coding-thinking preset), 64k context, flash-attn, q8_0 KV, MTP draft.

Open question this raises

Value now punishes the big models: if 93% is available at 3 GB, the 20 GB model has to justify itself on the 8 checks it wins, not on being the default. The next measurements that would settle it are the 9B (the likely middle point) and a re-run of the 35B with its own MTP off/on to price what speculation actually buys at that size.


1.2.1 — 2026-11-19: page review — rendering fixes and corrected claims

A read-through of every published page, with each number and claim checked against data/results.json. No benchmark data changed in this release; the corrections are to presentation and to statements that had gone stale after the 1.2.0 re-measurements.

Rendering

  • Inline links and emphasis were glued to neighbouring words in several places (“measured on real hardware byllm-bench—”, “lives onthe models page”, “detail inResults”, “:234/243”). The page template drops the newline that should have separated text from an inline element, so every such place now has an explicit space.
  • The models page repeated whole cards. It was built per measurement rather than per model, so Qwen3.6 — measured on three machine/variant combinations — appeared three times with the same HTML id (invalid markup) and three copies of its configuration. The page is now one card per model, listing each measurement as a result line with its machine, variant and time.
  • Six models were missing from the models page. The Strix Halo models exist only as seed figures, so the page computed from exported runs skipped them. They are now included, and the seed figures live in one shared file instead of three copies that could drift.
  • The accuracy chart drew the same model up to four times (Qwen3.6 at 233, 234, 235 and 236). It now shows one bar per model per hardware tier — the canonical measurement named in the model metadata — with a text label for the tier, since colour alone was carrying that information.
  • The hardware page had a broken link (..//results/) and one of its anchors was wrong.

Corrected claims

Where Was Now
Home headline “a 7B dense model beats its own 36B MoE sibling” false since the 1.2.0 re-measurement (231 vs 230); rewritten around what the data supports
Models: K2-7B “supersedes its 36B sibling” it does not on score; the case for it is memory (~⅓)
Models: MoVA “best coder of the local MoEs” ties the 7B — both 82/84
Models: Mellum “drops the JSON fence about 1 run in 8” an invented frequency; now “sometimes omits the fence”
Findings: Flint “its entire score gap is one timeout plus the deviation” 13 lost checks = 7 from one timeout + 3 deviation + 3 on the 10_pass_line_honest quirk
Home headline “two thirds of models improvise around impossible instructions” measured: no model reports every time, two never do
Methodology “local models run at 64k context” one labelled measurement ran at 262k; disclosed rather than implied
Limitations scorer fixes “part of the private repo” now public, in this changelog

Backend disclosure per measurement

The results table derived its backend string from the model’s metadata, so both Qwen3.6 rows on the new box were labelled Lemonade 2026.45.3 while the runs were made on 2026.46.2. Each row now derives the server version from the runs themselves (falling back to the machine era where older runs predate that field) plus the llama.cpp build of that machine. Flint and K2-7B are correctly attributed to the 2026.46.2 batches, Qwen3.6 on Fedora 45 to 2026.45.3.

Documentation that the audit made necessary

  • Methodology gains rule 6: run each model under its vendor’s conditions — per-response output budget, thinking mode and sampling come from the model card, not one global default — and rule 7 now names harness bugs as a category of correction, not just scorer strictness. The scoring section states what the answer form accepts (fenced or bare trailing JSON).
  • Limitations now names the two defect classes found on 2026-11-18 (output budget, answer parser), says that scores published before that date include them, and updates the 10_pass_line_honest count from four to six model families.
  • Hardware page is retitled from “Fedora 45” (the box has been headless CachyOS since 2026-11-18), documents both server versions, and adds the measured desktop-overhead result: GNOME plus a Docker stack cost nothing resolvable on this workload.

1.2.0 — 2026-11-18: harness audit, and two corrected models

A review of how the harness talks to each model found three defects in our own plumbing. Two of them had been quietly flooring scores for models that behave differently from Qwen.

Corrected results

Model Was Now Machine What happened
K2-Horizon-MoVA-36B-A4B 226/243 (93.0%) 231/243 (95.1%) HX 370 / headless CachyOS Re-measured after the harness fixes; no timeouts. The old batch had one answerless run in three.
JetBrains Mellum2.1-12B-A2.5B 158/243 (65.0%) 204/243 (84.0%) HX 370 / headless CachyOS The old score was mostly a scorer artifact (see defect 2). Official JetBrains MTP checkpoint.
K2-Horizon-7B (dense) 230/243 (94.7%) unchanged HX 370 / Fedora 45 Spot-checked under the fixed harness: 26/30 vs 29/30 on analysis-2. No cap or parser effect.
Recursive Flint 40B-A2.5B 230/243 (94.7%) unchanged HX 370 / Fedora 45 Compact responses; never near the output budget.

The defects

1. Per-response output budget was too small (16384 → 32768 tokens). The harness told the agent every model had a 16k-token budget for a single response. That budget covers the whole message: reasoning, tool calls and text together. Models with long high-effort reasoning get cut off mid-thought, and a truncated reasoning block is a failed response — the model never gets to emit its tool call or its answer. K2’s own model card asks for at least 32768. Measured blast radius on this box: exactly one truncated turn in 1,134 recorded turns, so this was a real defect but a narrow one — it mostly threatened verbose reasoners, not the fleet.

2. The answer parser only accepted fenced JSON. Every task asks the model to end with a JSON block, and the scorer accepted only a fenced ```json … ``` block. Some models intermittently end with correct, valid, unfenced JSON. Those runs were scored as “no answer”, which floors the entire task: parsed_json false takes every downstream check with it. This is what happened to Mellum: its analysis-1 answers were substantively correct and scored 2/8 purely because the fence was missing. Of 123 local run outputs, 9 were unparseable before the fix and 2 of those were false negatives. The scorer now also accepts a bare JSON object that ends the reply (nothing after it), which keeps the check honest while not punishing formatting.

3. Tool calls could block on a terminal (hardened). Runs execute real shell commands with no TTY. A tool call that opened a pager or waited on stdin could stall a run for the whole timeout — the observed signature is a run that makes three requests in thirty minutes with only a few minutes of accounted work. The child environment now sets PAGER, GIT_PAGER, GIT_EDITOR, EDITOR, VISUAL, TERM=dumb and CI=1. Not proven to have caused a specific published score, but it is the best explanation for one 1800 s stall in the MoVA batch.

4. Withdrawn: the MoVA baseline variant experiment (43 runs, 29.8%). An attempt to run the K2-36B without the vendor-mandated high reasoning effort. Its collapse was almost entirely answerless runs — the same answer-form artifact as defect 2 — so the number says more about our parser than about reasoning effort. Withdrawn rather than republished.

5. Removed: an invalid MoVA pilot batch. 27 five-second runs from 2026-11-06 that never engaged thinking (one request each, ~80 tokens). They were mixed into the export and dragged its aggregate from 226 to 301/486. Dropped.

Caveats we are keeping visible

  • The MoVA improvement (226 → 231) is not attributable to one specific fix. A three-rep experiment suggested a bigger jump than the full grid delivered. Treat 231 as a cleaner re-measurement, not as “the bug cost 5 checks”.
  • The Mellum re-run mixes several changes at once: parser fix, output budget, and a switch from a third-party quant to the official JetBrains MTP checkpoint. The direction is certain, the attribution is not.
  • Mellum’s analysis-3 remains floor-scored: all three reps now hit the 30-minute timeout while writing 16–24k tokens. That is a genuine model/task mismatch, not a bug.
  • Historical runs from other machines cannot be re-scored (transcripts are not published), so any remaining answer-form losses there stay as published.

Also this release

  • New machine section: HX 370 / headless CachyOS (box reinstalled without a desktop, Lemonade 2026.46.2 at a LAN address).
  • Qwen3.6-35B-A3B-MTP measured three ways on HX 370: Fedora 45 + GNOME, headless CachyOS at server defaults, and headless CachyOS with the Fedora-era launch options (cachy-tuned). Scores are within noise of each other (233–235/243) and raw decode is identical, so the desktop environment is not a measurable cost on this hardware.

1.1.0 — 2026-11-16: remote models, machine-aware results

  • Added the first remote model: GLM 5.3 Flash (OpenRouter, Z.AI fp8 pin), 235/243 in 9 minutes — tied with the best local MoE at roughly a thirteenth of the thinking tokens.
  • Results page split per machine, with backend disclosure per section.
  • Smoke-test filter changed from time alone to time plus completion tokens: a legitimate fast remote run and a broken smoke run look the same in wall time only.
  • Accuracy chart on the home page.

1.0.0 — 2026-11-16: the site itself

  • Rebuilt as an Astro knowledge base on the LabZone design system: findings, results, per-model pages with full configuration disclosure, methodology, known limitations, hardware notes.
  • Data pipeline: sanitized export from the private llm-bench repository, committed as data/results.json, rendered at build time. Task prompts, fixtures and answer keys never leave the private repository.

Initial data (2026-11-01 → 11-16)

Scores as first published, superseded where noted above:

Machine Model Score
Strix Halo / CachyOS Qwen3.8-27B (dense) 243/243 (100%)
Strix Halo / CachyOS Qwen3.6-35B-A3B-MTP 236/243 (97%)
HX 370 / Fedora 45 Qwen3.6-35B-A3B-MTP 235/243 (96.7%)
HX 370 / Fedora 45 Recursive Flint 40B-A2.5B 230/243 (94.7%)
HX 370 / Fedora 45 K2-Horizon-7B (dense) 230/243 (94.7%)
HX 370 / Fedora 45 K2-Horizon-MoVA-36B-A4B 226/243 → 231/243
HX 370 / Fedora 45 JetBrains Mellum2.1-12B 158/243 → 204/243
Remote / OpenRouter GLM 5.3 Flash 235/243 (96.7%)
Strix Halo / CachyOS Ornith-1.5-35B-A3B 233/243 (96%)
Strix Halo / CachyOS Gemma-4-12B-it-MTP 221/243 (91%)
Strix Halo / CachyOS Gemma-4-26B-A4B-it 211/243 (87%)
Strix Halo / CachyOS Nemotron-3.5-Lightning-30B 205/243 (84%)

Also measured but not part of the standard grid: the Vulkan-vs-ROCm backend comparison, the MTP-vs-no-MTP speed comparison, and the NPU/FLM energy measurements — see Hardware.