eval(decision): Bonsai 2 quality, calibration and latency without generated reasoning

[geisten]

Parent: #583
Depends on: #585
Use #586 for the final optimized comparison; collect the reference results before that optimization.
Planned branch: codex/decision-evaluation.

Problem / motivation

Removing generated tokens can yield a large speedup, but Bonsai's published thinking-mode scores do not establish its immediate-decision quality. We need measured results before selecting classifier training or making Jev-like speedup claims.

Scope

Confirmed rollout: start on Apple Silicon (cpu_neon and Metal), with MMLU first. Other devices/backends follow later. Pin the MMLU version/split and prompt/answer-token construction, and declare the generative quality baseline and non-inferiority margin before any quality-matched speedup comparison. The synthetic #586 timing matrix is performance-only and supplies no MMLU accuracy evidence.

Build on the current Bonsai implementation, existing eval_geist/MMLU adapters and the protocol established in #585. Keep reusable runtime evaluation mechanisms here; application policy and domain datasets belong to the consumer.

Compare on identical held-out questions:

1. Direct answer-token scores without generated reasoning.
2. An optimized single-answer-token generative baseline.
3. Ordinary generative/reasoning decisions, counting every generated reasoning and answer token.
4. The selected-row optimization once perf(decision): compute only requested output rows behind the scoring API #586 is ready.

A hosted Jev run is optional only when account access and benchmark spending are explicitly authorized. Published provider speedups are contextual references, not local measurements or acceptance thresholds.

Acceptance

• Pin Bonsai checkpoint/hash, runtime revisions, tokenizer/template, backend/device, numerical modes, candidate ordering, input lengths, question count, cache policy and concurrency.
• Recheck same-token runtime parity against a pinned Prism reference using the existing coverage from Bonsai-27B: MMLU 0.44 against the PrismML fork's 0.76 on identical tokens #432; include longer/high-entropy cases, not only a short greedy smoke test.
• Use separate development, calibration and held-out test data with documented split policy. Include representative negative/ambiguous cases and label-order sensitivity.
• Report accuracy or macro-F1, log-loss/Brier score and reliability/calibration results, plus p50/p95 latency, throughput, memory and cold/warm behavior with raw samples.
• Any post-hoc temperature scaling is fitted only on the calibration split and tied to the checkpoint, precision/backend policy and prompt construction. Uncalibrated option probabilities remain clearly identified.
• Define the quality target/non-inferiority margin before comparing speedups. Report speedup at that target; report any quality loss instead of trading it away silently.
• Count preprocessing, all model calls and any abstention/fallback route in the reported end-to-end policy.
• Conclude whether token scoring is adequate, a trained head is justified, or backbone adaptation is needed. Record negative findings too.

Existing dependencies / history

PQ2_0 and Hadamard support are already implemented. #432 and #561 are closed fixes. Link any newly reproduced backend issue to its existing owner rather than duplicating Bonsai support. Reuse #364 for performance methodology.

[geisten]

Infrastructure continuation: draft PR #608 on codex/decision-evaluation (head 403167e). #592 and #599 are merged.

Implemented offline pinned classic-MMLU preparation with explicit overlap policy and separate development/calibration/test, actual GGUF template/token-boundary validation, filtered benchmark arms with complete generation IDs/text/stop data, strict final-answer scoring, conditional log-loss/Brier/reliability and bound temperature scaling. The runner freezes the declared policy before inference and rejects pilot/subset masquerading, invalid/capped reasoning baselines and changed deterministic outputs for quality-matched claims.

Validation: format gate, release/ASan unit suites (each 70 pass / 45 supported-platform or fixture skips / 0 failures), full Python suite including 17 new cases, optimized GCC 16 driver compile, plus serial CPU NEON and Metal synthetic runner integration with exact selected/DENSE logits. These are implementation tests, not Bonsai MMLU quality or useful latency measurements.

Still open before acceptance: user-selected MMLU variant, margin and campaign scope; downstream overlap/cap policy; native pinned Prism same-token parity; representative negative/ambiguous controls; actual calibration and held-out campaigns; a measured conclusion about trained-head/backbone adaptation. No acceptance checklist is marked complete by the framework alone. Published Bonsai Redux/thinking scores are not substituted for classic five-shot greedy results. No hosted Jev service run is authorized or attempted.

[geisten]

Confirmed campaign requirements are now recorded in docs/DECISION_CAMPAIGN_PLAN.json on PR #608: classic MMLU5-shot, maximum 2 percentage points accuracy loss with a conservative paired 95% interval, small development pilot first.

The runner now supports deterministic label-independent subject limits (--per-subject 1 --subject-limit 8), and rejects bounded populations from full held-out quality runs. A development-only hypothetical BOS/TOK audit of the proposed eight-question selection checked all 96 actual candidate boundaries across 24 prefixes; prompt lengths are 399–695 tokens. No score or generation commands were used. Planning estimates with those lengths are 18–26 minutes on Metal and 88–124 minutes on CPU for two trials/all eight arms/cap512 if all reasoning outputs reach the cap; setup and shared-host load are excluded, and these are not measured campaign times.

Still awaiting downstream choices: explicit duplicate/overlap policy and exact bounded generation/pilot policy. Proposed: exclude and report cleaned subset; eight questions from eight subjects, cap512, Metal first. No MMLU quality result yet, and the broader external parity/calibration/held-out acceptance items stay open.

The first CI head had 22 successful checks, one intentional coverage skip and a Linux-arm64 Python-fixture failure from intercepted CPU discovery. This is fixed and the complete local Python suite passes, including 18 new cases; updated CI is queued/running for head 97fb9c4. C/runtime behavior is unchanged by that test correction.

[geisten]

The user approved all remaining pilot choices; the frozen development pilot is now complete on Metal and CPU NEON in #608.

• Classic cais/mmlu rev c30699e8356da336a370243923dbaf21066bb9fe, 5-shot, exact overlap exclusion, eight deterministic development questions/eight subjects, rotation0. This is a cleaned subset, not a stock-MMLU or MMLU-Redux score.
• Bonsai 2 27B PQ2_0 SHA 3907dc1658db1f78a9826bf8d5bcb8dc65db0d466388937af57f2294fae62ec1; Apple M1 Max 64 GiB, six threads, concurrency1, FP32 KV, greedy, cap512, two total trials/arm. Metal first, then CPU; 128 model calls per backend.
• Both backends: direct chat 4/8, cloze 6/8, reasoning 4/8. Reasoning has four invalid capped outputs; one-token unrestricted generation has five invalid outputs, 2/8 correct.
• All 64 selected/DENSE paired trials match exactly; reference parity and within-backend repeats pass. Prediction agreement across CPU/Metal is 8/8 in all four evaluated policies; this is not external Prism parity.
• Descriptive capped-policy reasoning/selected timing ratios: 2.168 Metal, 2.328 CPU. The conservative paired 95% accuracy-difference interval is [-57.39, 57.39]pp, so the declared 2pp loss margin is unestablished. Both reports explicitly block quality-matched eligibility.
• Full campaign wall times: 40.265min Metal / 99.224min CPU; preparation 22.029s. No concurrent owned model/build workloads during measurement; shared-host load/thermals and caches were not isolated.

Evidence: writeup, compact JSON, raw bundle and SHA-256 index.

These measurements bind clean source 13261ac73fb2841ef565bcabda5473eacd1e8ce0, with post-run model/binary/tokenizer/evaluator hash checks. The result was recorded before integrating newer main merges. Newer KV sizing/residency/snapshot changes are unmeasured by this artifact and require a newly bound acceptance run.

This completes only the approved development pilot, not #587. Next: native Prism same-token parity with full recurrent-state reset, then a development-only cap study that yields complete reasoning answers. Freeze the revised policy before calibration/held-out evaluation. Ambiguous/negative controls, label sensitivity, calibration and the full cleaned held-out quality run remain open; #588 training remains gated. No Jev-equivalence claim or production rollout follows.