Known limitations
What our numbers can and cannot tell you. Read this before quoting a table.
Quantization and precision
- Every local model runs quantized (typically Q4-class GGUF; exact quant disclosed per model). Quantized local models are not equivalent to their BF16 reference checkpoints; vendor-published KL-divergence metrics are quoted when available.
- One model was tested with a dynamic quant (Unsloth UD-Q4_K_XL) rather than a static one — the choice is disclosed because dynamic quants are generally closer to full precision, so treat its score as a slight upper bound for its size class.
- KV-cache precision varies per model (f16 vs q8_0) where memory forced it; this is disclosed and is a known confounder for absolute quality comparisons between models.
Thermal and power behavior (HX 370 tier)
- The mini-PC runs a 65 W burst / 54 W sustained power skin. Long batches measure the sustained regime: decode plateaus at roughly 50–90% of fresh speed depending on model bytes-per-token, reached after ~15 minutes of load. Short benchmarks do not measure this regime.
- Back-to-back model batches bias against the second model (throttle doesn’t fully recover on short pauses). Batch order is recorded; cross-model time comparisons within one session carry this caveat.
- The measured droop fraction is model-dependent (bytes/token drives power draw), so time comparisons between models on this machine are noisier than score comparisons.
Context size
- Local runs are pinned at 64k context. One batch (K2-Horizon-7B) is believed to have loaded at a larger context due to a server-side configuration anomaly; scores are unaffected (context only limits, never adds, capability) but its KV-memory headroom figures carry an asterisk until re-confirmed.
Variant differences
- Models whose vendors prescribe high reasoning effort ran with a
thinking-highvariant; the others ranbaseline. This inflates thinking-token counts (and wall time) slightly for the thinking-high models. Per-model pages disclose which variant produced which table.
Sample size
- The standard grid is 3 reps per task. Differences of ±2–3 points between models are within run-to-run noise; we label conclusions accordingly and inspect failed runs before explaining a score.
Scorer history
- Score checks are scripts, and scripts can be wrong. Where a failed run showed a scorer being
stricter than the task warranted, we fixed the scorer and re-ran; the fix history is part of
the private repo. One check (
10_pass_line_honest) has failed for every model in every rep across four model families and is under re-examination as a probable task/harness quirk — until resolved, that specific check’s 3 points are best read as constant noise.
Remote models
- Remote API runs share provider-side variation (routing, quantization on the provider’s side is disclosed when known) and network latency. Very few remote models have been tested so far; the remote section of the results is marked insufficient-data until more runs land.