What we test: how well large language models — local and remote — work as coding agents on real hardware, in real conditions: consumer mini-PCs and laptops, iGPUs, reasonable quantization, real-world time budgets. Not curated exam questions, not cloud GPUs.
Headlines (2026-11-19): a remote flash model scores 239/243 (98.4%) for five cents and six minutes on the full grid — at medium effort, not high — while a 3 GB dense model reaches 93% of a perfect score on the 9-task grid in 53 minutes — six and a half times the accuracy per gigabyte of the 20 GB MoE that scores 96% — and it does the one thing that model never does, report an instruction it could not follow; the best local MoE reaches 95%, and a 7B dense model holds 95% at a third of the memory; the repeated weak spot is honest deviation reporting — no model tested reports a broken instruction every time, and two never do; and our own harness was, until this week, penalising models that answer in a different shape than the reference family (see the changelog). Sustained decode on the HX 370 sits at ~50–88% of fresh speed depending on the model’s bytes-per-token.
The charts cover accuracy and throughput — see Results for per-task detail, timings and failed checks, Models for configuration disclosure and scenario recommendations, How we test for methodology, and Known limitations for what the numbers can and cannot tell you.
What we publish (and refuse to)
We publish: scores, per-check results, timings, token counts, exact model ids, quantization, sampling settings, backends, hardware, and written findings.
We withhold: the benchmark’s task prompts, fixtures and answer keys, and the specifics of the traps built into the tasks. Knowing a trap exists is a hint; knowing its content is the answer. Public pages describe trap categories, never trap content. See How we test.