LLMs as coding agents, on real hardware

Open benchmark results from consumer mini-PCs and laptops — iGPUs, reasonable quantization, thermal limits included. Script-scored, fully disclosed, no curated exam questions.

Accuracy on our benchmark

9 tasks × 3 reps, 243 checks, pi harness. Bars: Strix HaloHX 370 Remote.

Qwen3.8-27B (dense)
243/243 · 100.0%
Qwen3.6-35B-A3B-MTP
236/243 · 97.1%
Qwen3.6-35B-A3B-MTP
235/243 · 96.7%
GLM 5.3 Flash (remote)
235/243 · 96.7%
Ornith-1.5-35B-A3B
233/243 · 95.9%
K2-Horizon-7B (dense)
230/243 · 94.7%
K2-Horizon-MoVA-36B-A4B
226/243 · 93.0%
Gemma-4-12B-it-MTP
221/243 · 90.9%
Gemma-4-26B-A4B-it-MTP
211/243 · 86.8%

100% = every check in every rep. Per-task detail inResults · how to read the numbers.

What we test: how well large language models — local and remote — work as coding agents on real hardware, in real conditions: consumer mini-PCs and laptops, iGPUs, reasonable quantization, real-world time budgets. Not curated exam questions, not cloud GPUs.

Headlines (2026-11-16): a 7B dense model beats its own 36B MoE sibling at the same wall time; sustained decode on the HX 370 sits at ~50–88% of fresh speed depending on the model’s bytes-per-token; every tested model except the K2 family silently improvises around impossible instructions.

The chart covers accuracy only — see Results for per-task detail, timings and failed checks, Models for configuration disclosure and scenario recommendations, How we test for methodology, and Known limitations for what the numbers can and cannot tell you.

What we publish (and refuse to)

We publish: scores, per-check results, timings, token counts, exact model ids, quantization, sampling settings, backends, hardware, and written findings.

We withhold: the benchmark’s task prompts, fixtures and answer keys, and the specifics of the traps built into the tasks. Knowing a trap exists is a hint; knowing its content is the answer. Public pages describe trap categories, never trap content. See How we test.