How we test

What a run is

One run is one combination of model × harness × variant × task × rep:

  • Model: any model on an OpenAI-compatible endpoint (a local inference server, or a hosted API).
  • Harness: the agent loop driving the model. pi is our primary harness: a coding agent with read/write/edit/bash tools, run with an isolated config (no personal settings, skills or context leak in). minimal is a reference loop with four tools — the floor of what the model can do with almost no scaffolding. Third-party harnesses (e.g. OpenCode) are used for cross-checks.
  • Variant: something switched inside the harness (e.g. a thinking-level setting, a system-prompt skill). baseline adds nothing: harness defaults, server-side sampling.
  • Task: a fixture repository, a prompt, and a scorer.
  • Rep: the same combination repeated (usually 3×) to expose run-to-run variance.

Every run starts from a fresh temporary copy of its fixture. Models never see previously-modified repos.

Tasks

Three task types, in escalating levels 1–3. Higher levels use bigger fixtures and add traps:

Type What it measures (coarse)
Analysis Read a repo, produce exact structured facts about it; say “unknown” when a fact isn’t in the repo; propose a measured, non-destructive change plan
Plan Execute a written plan exactly; update call sites; resist the temptation to “helpfully” touch things not in the plan; stop and report when a step cannot be done
Code Implement a small feature or refactor so hidden acceptance tests pass; keep the public API; stay in scope; report what you did honestly

Contamination policy: task content (prompts, fixtures, answer keys, trap specifics) is not published. This page describes categories only. We do not use public benchmark task sets.

Scoring

Scores are computed by scripts, never by another model.

  • Every task asks the model to end its session with one JSON answer block.
  • The scorer checks that block against reality: an answer key computed from the fixture, the git diff of what the model changed, and hidden acceptance tests copied in only after the model finishes.
  • Some checks compare claims with facts (“tests pass” against the real test run, “files changed” against the diff) — misreporting costs points, not just mistakes.
  • A run’s score is checks passed / checks total; task and batch scores sum these.

Check names are published as-is (e.g. tests_still_pass, report_files_honest). They hint at categories of discipline, never at specific trap content.

Comparability rules

  1. Warm the model first. One short request before timing; model load time must not pollute measurements. One model’s batch at a time — the inference server keeps one LLM loaded.
  2. Same settings within a comparison. Same harness, variant, task set and rep count.
  3. Sampling comes from the server. The baseline variant sends no sampling parameters; models run with settings saved from their vendor’s model card (disclosed per model).
  4. Pinned context size. Local models run at 64k context in agent runs.
  5. Respect thermal behavior. Sustained generation throttles on iGPUs; batches are model-order aware and cooldowns respected. See Known limitations for measured effects.
  6. Read the failures. Before concluding anything from a score, open the failed runs. Several apparent “model failures” turned out to be scorer strictness — those get recorded as findings.

Machines

Machine Class Notes
Strix Halo (Ryzen AI Max) / CachyOS 256 GB/s memory bandwidth the “fast local” tier
HX 370 (Strix Point, Radeon 890M) / Fedora 45 ~90 GB/s, power-skin throttled the “realistic mini-PC” tier

Scores transfer well between machines (same model ranks within ~1 point); time does not. Full details: Strix Halo, HX 370.