LLMs as coding agents, on real hardware

Open benchmark results from consumer mini-PCs and laptops — iGPUs, reasonable quantization, thermal limits included. Script-scored, fully disclosed, no curated exam questions.

Accuracy on our benchmark

9 tasks × 3 reps, 243 checks, pi harness. Strix HaloHX 370Remote

Qwen3.8-27B (dense)Strix Halo
243/243 · 100.0%
DeepSeek V4.1 Flash 1017 (remote)Remote
239/243 · 98.4%
Qwen3.6-35B-A3B-MTPStrix Halo
236/243 · 97.1%
Qwen3.6-35B-A3B-MTPHX 370
235/243 · 96.7%
GLM 5.3 Flash (remote)Remote
235/243 · 96.7%
Ornith-1.5-35B-A3BStrix Halo
233/243 · 95.9%
K2-Horizon-MoVA-36B-A4BHX 370
231/243 · 95.1%
Recursive Flint 40B-A2.5BHX 370
230/243 · 94.7%
K2-Horizon-7B (dense)HX 370
230/243 · 94.7%
Qwen3.5-4B-MTPHX 370
226/243 · 93.0%
Gemma-4-12B-it-MTPStrix Halo
221/243 · 90.9%
Gemma-4-26B-A4B-it-MTPStrix Halo
211/243 · 86.8%
Nemotron-3.5-Lightning-30B-A3BStrix Halo
205/243 · 84.4%

Bars scale to the best model shown: untick cloud models and the best local model fills the bar. Values are always absolute (full mark = every check in every rep). Per-task detail in Results · how to read the numbers.

Speed on our benchmark

Output tokens per second of wall clock across the whole grid (completion tokens ÷ elapsed time, 27 runs). Strix HaloHX 370Remote

GLM 5.3 Flash (remote)Remote
32.0 tok/s
Qwen3.5-4B-MTPHX 370
30.1 tok/s
Qwen3.6-35B-A3B-MTPHX 370
25.8 tok/s
K2-Horizon-MoVA-36B-A4BHX 370
10.6 tok/s
K2-Horizon-7B (dense)HX 370
9.8 tok/s

Bars scale to the best model shown: untick cloud models and the fastest local model fills the bar. Values are throughput, not latency — a model that writes more still scores higher if it writes faster. Strix Halo is absent — those rows are seed data without token counts; the tier's grid times are on Results, and it joins this chart once that box exports its runs. See how to read the numbers.

What we test: how well large language models — local and remote — work as coding agents on real hardware, in real conditions: consumer mini-PCs and laptops, iGPUs, reasonable quantization, real-world time budgets. Not curated exam questions, not cloud GPUs.

Headlines (2026-11-19): a remote flash model scores 239/243 (98.4%) for five cents and six minutes on the full grid — at medium effort, not high — while a 3 GB dense model reaches 93% of a perfect score on the 9-task grid in 53 minutes — six and a half times the accuracy per gigabyte of the 20 GB MoE that scores 96% — and it does the one thing that model never does, report an instruction it could not follow; the best local MoE reaches 95%, and a 7B dense model holds 95% at a third of the memory; the repeated weak spot is honest deviation reporting — no model tested reports a broken instruction every time, and two never do; and our own harness was, until this week, penalising models that answer in a different shape than the reference family (see the changelog). Sustained decode on the HX 370 sits at ~50–88% of fresh speed depending on the model’s bytes-per-token.

The charts cover accuracy and throughput — see Results for per-task detail, timings and failed checks, Models for configuration disclosure and scenario recommendations, How we test for methodology, and Known limitations for what the numbers can and cannot tell you.

What we publish (and refuse to)

We publish: scores, per-check results, timings, token counts, exact model ids, quantization, sampling settings, backends, hardware, and written findings.

We withhold: the benchmark’s task prompts, fixtures and answer keys, and the specifics of the traps built into the tasks. Knowing a trap exists is a hint; knowing its content is the answer. Public pages describe trap categories, never trap content. See How we test.