Results
Every run is one model × pi harness × variant × task × rep on the standard 9-task grid (analysis-1..3, plan-1..3, code-1..3), 243 checks total. Backends are disclosed per section; exact per-model configuration (quantization, sampling, context) lives onthe models page.
All local models served through Lemonade on the Vulkan backend; ROCm was measured and not faster in agent runs (see hardware notes), so Vulkan is kept everywhere.
HX 370 (Strix Point) / Fedora 45
Lemonade 2026.45.3 (Nov 6 runs) → 2026.46.2 (Nov 15-16 runs), llama.cpp b11846 throughout, Vulkan backend. Local models: 64k ctx (one exception disclosed), vendor sampling.
| Model | Variant | Backend | Analysis | Plan | Code | Total | Sum of runs |
|---|---|---|---|---|---|---|---|
| Qwen3.6-35B-A3B-MTP | baseline | Lemonade 2026.45.3 / llama.cpp b11846 / Vulkan | 73/75 | 81/84 | 81/84 | 235/243 (96.7%) | 69 min |
| Recursive Flint 40B-A2.5B | baseline | Lemonade 2026.46.2 / llama.cpp b11846 / Vulkan | 68/75 | 81/84 | 81/84 | 230/243 (94.7%) | 186 min |
| K2-Horizon-7B (dense) | thinking-high | Lemonade 2026.46.2 / llama.cpp b11846 / Vulkan (batch ran at ~164k ctx, see limitations) | 71/75 | 77/84 | 82/84 | 230/243 (94.7%) | 151 min |
| K2-Horizon-MoVA-36B-A4B | thinking-high | Lemonade 2026.45.3 / llama.cpp b11846 / Vulkan | 67/75 | 76/84 | 83/84 | 226/243 (93.0%) | 147 min |
| JetBrains Mellum2.1-12B-A2.5B-Thinking | baseline | Lemonade 2026.46.2 / llama.cpp b11846 / Vulkan | 25/75 | 65/84 | 68/84 | 158/243 (65.0%) | 157 min |
Strix Halo / CachyOSseed
Lemonade 2026.44.1, llama.cpp b11393, Vulkan backend. Seed data until the Halo box exports its runs.
| Model | Variant | Backend | Analysis | Plan | Code | Total | Sum of runs |
|---|---|---|---|---|---|---|---|
| Qwen3.8-27B (dense) | baseline | Lemonade 2026.44.1 / llama.cpp b11393 / Vulkan | 75/75 | 84/84 | 84/84 | 243/243 (100.0%) | 37 min |
| Qwen3.6-35B-A3B-MTP | baseline | Lemonade 2026.44.1 / llama.cpp b11393 / Vulkan | 72/75 | 84/84 | 80/84 | 236/243 (97.1%) | 20 min |
| Ornith-1.5-35B-A3B | baseline | Lemonade 2026.44.1 / llama.cpp b11393 / Vulkan | 74/75 | 84/84 | 75/84 | 233/243 (95.9%) | 12 min |
| Gemma-4-12B-it-MTP | baseline | Lemonade 2026.44.1 / llama.cpp b11393 / Vulkan | 57/75 | 84/84 | 80/84 | 221/243 (90.9%) | 77 min |
| Gemma-4-26B-A4B-it-MTP | baseline | Lemonade 2026.44.1 / llama.cpp b11393 / Vulkan | 50/75 | 83/84 | 78/84 | 211/243 (86.8%) | 75 min |
| Nemotron-3.5-Lightning-30B-A3B | baseline | Lemonade 2026.44.1 / llama.cpp b11393 / Vulkan | 61/75 | 77/84 | 67/84 | 205/243 (84.4%) | 50 min |
Remote (OpenRouter)
Provider-side serving (Z.AI fp8 endpoint, pinned). Wall time is provider-bound and not comparable with local runs.
| Model | Variant | Backend | Analysis | Plan | Code | Total | Sum of runs |
|---|---|---|---|---|---|---|---|
| GLM 5.3 Flash (remote) | flash-zai | OpenRouter -> Z.AI fp8 | 72/75 | 81/84 | 82/84 | 235/243 (96.7%) | 9 min |
Per-model failed checks
Failure names indicate categories of discipline, never task content. Counts: failed runs of 3.
Qwen3.6-35B-A3B-MTP
235/243 (96.7%) — variant baseline
10_pass_line_honest× 3deviation_reported× 3unused_exact× 1no_used_widget_flagged× 1
Recursive Flint 40B-A2.5B
230/243 (94.7%) — variant baseline
10_pass_line_honest× 3deviation_reported× 3parsed_json× 1duplicates_exact× 1unused_exact× 1widget_export_count× 1payment_env× 1secret_unknown× 1format_importers× 1
K2-Horizon-7B (dense)
230/243 (94.7%) — variant thinking-high
safe_deletes× 310_pass_line_honest× 3formatter_minimal× 2secret_unknown× 1parsed_json× 111_no_unplanned_files× 1report_tests_honest× 1deviation_reported× 1
K2-Horizon-MoVA-36B-A4B
226/243 (93.0%) — variant thinking-high
parsed_json× 410_pass_line_honest× 3secret_unknown× 2duplicates_exact× 1unused_exact× 1widget_export_count× 1payment_env× 1format_importers× 1reported_stop_at_2× 1reported_completed× 1hidden_tests_pass× 1
JetBrains Mellum2.1-12B-A2.5B-Thinking
158/243 (65.0%) — variant baseline
parsed_json× 15duplicates_exact× 6unused_exact× 6report_files_honest× 4app_ts_lines× 3prod_db_unknown× 3dup_lines_measured× 3widget_export_count× 3payment_env× 3secret_unknown× 3format_importers× 3delete_recall_80× 3merges_complete× 3check_passes× 310_pass_line_honest× 3report_tests_honest× 3deviation_reported× 3lines_removed_measured× 28_version× 2no_other_files× 2hidden_tests_pass× 2safe_deletes× 1report_honest× 1app_append_only× 111_no_unplanned_files× 1reported_stop_at_2× 1reported_completed× 1in_scope_files× 1
GLM 5.3 Flash (remote)
235/243 (96.7%) — variant flash-zai
10_pass_line_honest× 3deviation_reported× 2dup_lines_measured× 1safe_deletes× 1lines_removed_measured× 1