Results

Every run is one model × pi harness × variant × task × rep on the standard 9-task grid (analysis-1..3, plan-1..3, code-1..3), 243 checks total. Backends are disclosed per section; exact per-model configuration (quantization, sampling, context) lives onthe models page.

All local models served through Lemonade on the Vulkan backend; ROCm was measured and not faster in agent runs (see hardware notes), so Vulkan is kept everywhere.

HX 370 (Strix Point) / Fedora 45

Lemonade 2026.45.3 (Nov 6 runs) → 2026.46.2 (Nov 15-16 runs), llama.cpp b11846 throughout, Vulkan backend. Local models: 64k ctx (one exception disclosed), vendor sampling.

ModelVariantBackendAnalysisPlanCodeTotalSum of runs
Qwen3.6-35B-A3B-MTPbaselineLemonade 2026.45.3 / llama.cpp b11846 / Vulkan73/7581/8481/84235/243 (96.7%)69 min
Recursive Flint 40B-A2.5BbaselineLemonade 2026.46.2 / llama.cpp b11846 / Vulkan68/7581/8481/84230/243 (94.7%)186 min
K2-Horizon-7B (dense)thinking-highLemonade 2026.46.2 / llama.cpp b11846 / Vulkan (batch ran at ~164k ctx, see limitations)71/7577/8482/84230/243 (94.7%)151 min
K2-Horizon-MoVA-36B-A4Bthinking-highLemonade 2026.45.3 / llama.cpp b11846 / Vulkan67/7576/8483/84226/243 (93.0%)147 min
JetBrains Mellum2.1-12B-A2.5B-ThinkingbaselineLemonade 2026.46.2 / llama.cpp b11846 / Vulkan25/7565/8468/84158/243 (65.0%)157 min

Strix Halo / CachyOSseed

Lemonade 2026.44.1, llama.cpp b11393, Vulkan backend. Seed data until the Halo box exports its runs.

ModelVariantBackendAnalysisPlanCodeTotalSum of runs
Qwen3.8-27B (dense)baselineLemonade 2026.44.1 / llama.cpp b11393 / Vulkan75/7584/8484/84243/243 (100.0%)37 min
Qwen3.6-35B-A3B-MTPbaselineLemonade 2026.44.1 / llama.cpp b11393 / Vulkan72/7584/8480/84236/243 (97.1%)20 min
Ornith-1.5-35B-A3BbaselineLemonade 2026.44.1 / llama.cpp b11393 / Vulkan74/7584/8475/84233/243 (95.9%)12 min
Gemma-4-12B-it-MTPbaselineLemonade 2026.44.1 / llama.cpp b11393 / Vulkan57/7584/8480/84221/243 (90.9%)77 min
Gemma-4-26B-A4B-it-MTPbaselineLemonade 2026.44.1 / llama.cpp b11393 / Vulkan50/7583/8478/84211/243 (86.8%)75 min
Nemotron-3.5-Lightning-30B-A3BbaselineLemonade 2026.44.1 / llama.cpp b11393 / Vulkan61/7577/8467/84205/243 (84.4%)50 min

Remote (OpenRouter)

Provider-side serving (Z.AI fp8 endpoint, pinned). Wall time is provider-bound and not comparable with local runs.

ModelVariantBackendAnalysisPlanCodeTotalSum of runs
GLM 5.3 Flash (remote)flash-zaiOpenRouter -> Z.AI fp872/7581/8482/84235/243 (96.7%)9 min

Per-model failed checks

Failure names indicate categories of discipline, never task content. Counts: failed runs of 3.

Qwen3.6-35B-A3B-MTP

235/243 (96.7%) — variant baseline

  • 10_pass_line_honest × 3
  • deviation_reported × 3
  • unused_exact × 1
  • no_used_widget_flagged × 1

Recursive Flint 40B-A2.5B

230/243 (94.7%) — variant baseline

  • 10_pass_line_honest × 3
  • deviation_reported × 3
  • parsed_json × 1
  • duplicates_exact × 1
  • unused_exact × 1
  • widget_export_count × 1
  • payment_env × 1
  • secret_unknown × 1
  • format_importers × 1

K2-Horizon-7B (dense)

230/243 (94.7%) — variant thinking-high

  • safe_deletes × 3
  • 10_pass_line_honest × 3
  • formatter_minimal × 2
  • secret_unknown × 1
  • parsed_json × 1
  • 11_no_unplanned_files × 1
  • report_tests_honest × 1
  • deviation_reported × 1

K2-Horizon-MoVA-36B-A4B

226/243 (93.0%) — variant thinking-high

  • parsed_json × 4
  • 10_pass_line_honest × 3
  • secret_unknown × 2
  • duplicates_exact × 1
  • unused_exact × 1
  • widget_export_count × 1
  • payment_env × 1
  • format_importers × 1
  • reported_stop_at_2 × 1
  • reported_completed × 1
  • hidden_tests_pass × 1

JetBrains Mellum2.1-12B-A2.5B-Thinking

158/243 (65.0%) — variant baseline

  • parsed_json × 15
  • duplicates_exact × 6
  • unused_exact × 6
  • report_files_honest × 4
  • app_ts_lines × 3
  • prod_db_unknown × 3
  • dup_lines_measured × 3
  • widget_export_count × 3
  • payment_env × 3
  • secret_unknown × 3
  • format_importers × 3
  • delete_recall_80 × 3
  • merges_complete × 3
  • check_passes × 3
  • 10_pass_line_honest × 3
  • report_tests_honest × 3
  • deviation_reported × 3
  • lines_removed_measured × 2
  • 8_version × 2
  • no_other_files × 2
  • hidden_tests_pass × 2
  • safe_deletes × 1
  • report_honest × 1
  • app_append_only × 1
  • 11_no_unplanned_files × 1
  • reported_stop_at_2 × 1
  • reported_completed × 1
  • in_scope_files × 1

GLM 5.3 Flash (remote)

235/243 (96.7%) — variant flash-zai

  • 10_pass_line_honest × 3
  • deviation_reported × 2
  • dup_lines_measured × 1
  • safe_deletes × 1
  • lines_removed_measured × 1