What we test: how well large language models — local and remote — work as coding agents on real hardware, in real conditions: consumer mini-PCs and laptops, iGPUs, reasonable quantization, real-world time budgets. Not curated exam questions, not cloud GPUs.
Headlines (2026-11-16): a 7B dense model beats its own 36B MoE sibling at the same wall time; sustained decode on the HX 370 sits at ~50–88% of fresh speed depending on the model’s bytes-per-token; every tested model except the K2 family silently improvises around impossible instructions.
The chart covers accuracy only — see Results for per-task detail, timings and failed checks, Models for configuration disclosure and scenario recommendations, How we test for methodology, and Known limitations for what the numbers can and cannot tell you.
What we publish (and refuse to)
We publish: scores, per-check results, timings, token counts, exact model ids, quantization, sampling settings, backends, hardware, and written findings.
We withhold: the benchmark’s task prompts, fixtures and answer keys, and the specifics of the traps built into the tasks. Knowing a trap exists is a hint; knowing its content is the answer. Public pages describe trap categories, never trap content. See How we test.