How we test
What a run is
One run is one combination of model × harness × variant × task × rep:
- Model: any model on an OpenAI-compatible endpoint (a local inference server, or a hosted API).
- Harness: the agent loop driving the model.
piis our primary harness: a coding agent with read/write/edit/bash tools, run with an isolated config (no personal settings, skills or context leak in).minimalis a reference loop with four tools — the floor of what the model can do with almost no scaffolding. Third-party harnesses (e.g. OpenCode) are used for cross-checks. - Variant: something switched inside the harness (e.g. a thinking-level setting, a system-prompt
skill).
baselineadds nothing: harness defaults, server-side sampling. - Task: a fixture repository, a prompt, and a scorer.
- Rep: the same combination repeated (usually 3×) to expose run-to-run variance.
Every run starts from a fresh temporary copy of its fixture. Models never see previously-modified repos.
Tasks
Three task types, in escalating levels 1–3. Higher levels use bigger fixtures and add traps:
| Type | What it measures (coarse) |
|---|---|
| Analysis | Read a repo, produce exact structured facts about it; say “unknown” when a fact isn’t in the repo; propose a measured, non-destructive change plan |
| Plan | Execute a written plan exactly; update call sites; resist the temptation to “helpfully” touch things not in the plan; stop and report when a step cannot be done |
| Code | Implement a small feature or refactor so hidden acceptance tests pass; keep the public API; stay in scope; report what you did honestly |
Contamination policy: task content (prompts, fixtures, answer keys, trap specifics) is not published. This page describes categories only. We do not use public benchmark task sets.
Scoring
Scores are computed by scripts, never by another model.
- Every task asks the model to end its session with one JSON answer block.
- The scorer checks that block against reality: an answer key computed from the fixture, the git diff of what the model changed, and hidden acceptance tests copied in only after the model finishes.
- Some checks compare claims with facts (“tests pass” against the real test run, “files changed” against the diff) — misreporting costs points, not just mistakes.
- A run’s score is checks passed / checks total; task and batch scores sum these.
Check names are published as-is (e.g. tests_still_pass, report_files_honest). They hint at
categories of discipline, never at specific trap content.
Comparability rules
- Warm the model first. One short request before timing; model load time must not pollute measurements. One model’s batch at a time — the inference server keeps one LLM loaded.
- Same settings within a comparison. Same harness, variant, task set and rep count.
- Sampling comes from the server. The
baselinevariant sends no sampling parameters; models run with settings saved from their vendor’s model card (disclosed per model). - Pinned context size. Local models run at 64k context in agent runs.
- Respect thermal behavior. Sustained generation throttles on iGPUs; batches are model-order aware and cooldowns respected. See Known limitations for measured effects.
- Read the failures. Before concluding anything from a score, open the failed runs. Several apparent “model failures” turned out to be scorer strictness — those get recorded as findings.
Machines
| Machine | Class | Notes |
|---|---|---|
| Strix Halo (Ryzen AI Max) / CachyOS | 256 GB/s memory bandwidth | the “fast local” tier |
| HX 370 (Strix Point, Radeon 890M) / Fedora 45 | ~90 GB/s, power-skin throttled | the “realistic mini-PC” tier |
Scores transfer well between machines (same model ranks within ~1 point); time does not. Full details: Strix Halo, HX 370.