Known limitations

What our numbers can and cannot tell you. Read this before quoting a table.

Quantization and precision

  • Every local model runs quantized (typically Q4-class GGUF; exact quant disclosed per model). Quantized local models are not equivalent to their BF16 reference checkpoints; vendor-published KL-divergence metrics are quoted when available.
  • One model was tested with a dynamic quant (Unsloth UD-Q4_K_XL) rather than a static one — the choice is disclosed because dynamic quants are generally closer to full precision, so treat its score as a slight upper bound for its size class.
  • KV-cache precision varies per model (f16 vs q8_0) where memory forced it; this is disclosed and is a known confounder for absolute quality comparisons between models.

Thermal and power behavior (HX 370 tier)

  • The mini-PC runs a 65 W burst / 54 W sustained power skin. Long batches measure the sustained regime: decode plateaus at roughly 50–90% of fresh speed depending on model bytes-per-token, reached after ~15 minutes of load. Short benchmarks do not measure this regime.
  • Back-to-back model batches bias against the second model (throttle doesn’t fully recover on short pauses). Batch order is recorded; cross-model time comparisons within one session carry this caveat.
  • The measured droop fraction is model-dependent (bytes/token drives power draw), so time comparisons between models on this machine are noisier than score comparisons.

Context size

  • Local runs are pinned at 64k context. One batch (K2-Horizon-7B) is believed to have loaded at a larger context due to a server-side configuration anomaly; scores are unaffected (context only limits, never adds, capability) but its KV-memory headroom figures carry an asterisk until re-confirmed.

Variant differences

  • Models whose vendors prescribe high reasoning effort ran with a thinking-high variant; the others ran baseline. This inflates thinking-token counts (and wall time) slightly for the thinking-high models. Per-model pages disclose which variant produced which table.

Sample size

  • The standard grid is 3 reps per task. Differences of ±2–3 points between models are within run-to-run noise; we label conclusions accordingly and inspect failed runs before explaining a score.

Scorer history

  • Score checks are scripts, and scripts can be wrong. Where a failed run showed a scorer being stricter than the task warranted, we fixed the scorer and re-ran; the fix history is part of the private repo. One check (10_pass_line_honest) has failed for every model in every rep across four model families and is under re-examination as a probable task/harness quirk — until resolved, that specific check’s 3 points are best read as constant noise.

Remote models

  • Remote API runs share provider-side variation (routing, quantization on the provider’s side is disclosed when known) and network latency. Very few remote models have been tested so far; the remote section of the results is marked insufficient-data until more runs land.