Repository navigation
jev-calib: ship no calibrator parameters, ship the tools to fit your own - #1
Open
justindressler wants to merge 82 commits into
Open
justindressler wants to merge 82 commits into
justindressler wants to merge 82 commits into
Conversation
added 30 commits
October 31, 2026 19:52
… prompt (10 failing)
serve/jev.py: frozen D3 prompt, donor 422 contract, jmath readout (M1 identity), engine-turn discipline (svc.fifo), final_row_logits readout boundary (BROW on the engine side). server.py: route + import. 10/10 tests; full-suite parity vs baseline verified (0 new failures; 14 pre-existing regex-module errors, flaky sampler tests vary run to run on both trees).
…fail-closed) + jev.handle wired to real contract
…act (CUDA test target)
…, CUDA+Metal), --jev-brow flag, serve-loop BROW handler; golden battery fixture
…minism; CMake CPU guard fixed (15/15)
…allowed (review r1)
…ly order, /admin/calib hot-swap, B1.1 log rows - serve/jev_calib.py: donor src/server.cpp:518-633 surface - load (calib.json/v1.json, corrupt=identity), swap, apply (platt_v1 noul / T_choice_v1 / T_score_v1, argmax-safe, renormalized), tuple_of, async 4096-row log queue with 200ms flusher - serve/jev.py: donor order (server.cpp:1134-1142) - softmax -> round 4dp -> renorm -> calibrate -> answers_from_calibrated; calib id surfaced per answer; B1.1 rows (ts/tuple/qid/type/raw_p unrounded; label_mass omitted - BROW is sparse, fit.py never reads it) - serve/server.py: POST /admin/calib (Bearer STRATA_JEV_ADMIN_KEY, empty=open like llama-swap's control plane); startup swap() + daemon flusher + atexit drain - tests: test_jev_calib (14) math+wire+hot-reload; test_systemone re-pinned to the donor 4dp contract. 29/29 green (mock, no GPU); full-suite regression running
…he live M3.2 bench 1. question_block now the donor render_qblock verbatim (server.cpp:305-334): full Options:/Levels: list with criteria descriptions, 'Question:'/'Answer:' anchors, noul Yes/No-means lines. The M3.0 render listed NO options -> the model picked labels blind (ARC 23% = chance). 2. enable_thinking=False on the decision render (D3, frozen): the readout position must predict the label, not open <think>. 3. render_state: string states pass through VERBATIM (server.cpp:266-268); objects stay pretty JSON. The port JSON-quoted every string state. Live QFN before -> after: choice 23%->100%, noul 49%->88%, score 24%->55% (donor 27B baseline: 99/90.5/56.9). 29/29 unit tests green.
…on yes/no branches, softmax(d) - question_head + v_branch: donor render_qhead/render_vbranch verbatim (server.cpp:337-353): shared 'Question:/Options:' head, then one 'Proposed answer: <name> ... Answer:' branch per option - d_i = s_yes - s_no per branch (BROW readout), p = softmax(d), 4dp + renormalize, choice temperature applies (donor :1150-1163, calib_apply) - response: probabilities keyed by option NAME, 'choice' = name string (donor A2.6 shape); usage.readout_tokens counts one BROW per branch - X-Jev-Mode header passthrough in server.py; V scopes to choice only, noul/score fall through to the L path (donor :767) - log rows: tuple ...|V|choice with v_d (donor B1.1 V schema) - K sequential branch decodes share the state+head prefix - Strata's engine-side KV prefix mounting reuses it (M3.0 alignment, no EngineSeq) - tests: test_jev_vmode (6) - name-keyed probs, branch prompt shape, shared-head prefix, per-option readouts, v_d log rows, L unchanged. 35/35 green with calib/systemone/golden.
…n, one wire request) - POST /v1/systemone with image_b64 (+ optional vision_prompt, default 'Describe this game screen:') -> jev.handle_image - stage 1: Strata's existing per-preset vision path (svc.prepare encodes the image and expands the image pads), one bounded summary generation (<=160 tok, thinking off) - stage 2: the summary becomes the decision state; the standard L/V readout answers. Response: answers + image_summary + stages_used ['vision','decision'] (donor PLAN B V1 contract, server.cpp:2419-2603; the donor's V2 single-seq fork crashed ggml and stays NOT ported) - fail-closed: empty summary -> 502 and the decision stage never runs; no vision encoder -> 400; short b64 -> 422 - tests: test_jev_image (6) - two stages one request, summary rides into the decision state, vision_prompt accepted, fail-closed paths, validation first. 41/41 green with vmode/calib/systemone/golden.
…e jev extension
M3.9 (donor amendment 7 mode 1): "then": {"generate": {...}} on /v1/systemone -
decide first, then generate with the answers folded into the LAST USER turn as
'Jev decisions (JSON): ...' (a trailing system message throws in the chat
template; donor folds too). Sampling mirrors the chat endpoint (temp 0.6,
top_p 0.95, top_k 20, max_tokens 512 defaults; stop strings honored).
Response = decision body + then{text, finish_reason, usage}.
M3.3 (donor A3.4): "jev": {state|messages, questions} on /v1/chat/completions -
jev.questions required; state XOR messages (400 on both/neither, donor
messages-must-be-empty rule); stream refused with the jev extension; the
conversation becomes the decision state (v1-chat, donor 3.5). Response wraps
the decision body under "jev" with an OpenAI-style envelope.
Tests: test_jev_then (9) - two stages one request, answers in the gen prompt,
sampling defaults, error propagation, chat-surface validation arms, plain chat
unaffected. 50/50 green with image/vmode/calib/systemone/golden.
…ns, no cross-talk, FIFO serializes engine turns The donor's DoD arm on the serve port: 7 threads, 3 noul questions each - every client gets exactly its own 3 answers (qid-keyed, no foreign qids), 21 readout turns, and an in-flight counter on engine.generate proves the FIFO never lets two engine turns overlap (the serve-side stand-in for the donor's per-decode-step ENG.mu, which M3.4 maps to svc.fifo - already held across each request's whole readout loop). 52/52 green.
…ns, calibration, examples Wire surface (request/response/422 contract), the four extensions (V mode, image two-call, decide-then-generate, chat-surface jev), calibration math + hot-swap + the measured QFN tables (raw vs calibrated, ms/item, donor-parity numbers), concurrency arm, and curl examples for every path.
…enizer
Live QFN run caught two: the engine streams negative sentinel ids
(verification artifacts) which the byte tokenizer's token_bytes rejects, and
per-token decode re-decodes for stops. Now uses the server's own incremental
Detokenizer (split-multibyte safe), skips None/negative ids, and checks stops
against the last few decoded pieces.
Live-verified on QFN through llama-swap after the fix:
- then.generate: decision + 'Jev selects the second option ... 75.93%' (the
answers folded into the prompt, 33-token completion, finish stop)
- chat-surface jev: {object: jev.response, jev: {answers... noul 0.9995}}
…cide Justin's ask: can the image embeddings themselves drive the readout, without the summary in between? Yes on Strata - the donor's V2 needed a seq fork off image embeds (crashed ggml on hybrid attention); our readout is BROW on the final decoded row, so one prompt [image + question_block], one decode step, label logits off that row. No new engine primitive. - image_mode: 'summary' (default, two-call) | 'direct' (single readout); invalid values 422 on image_mode - the question text sits AFTER the image pad so the readout position attends the full image; svc.prepare encodes + expands pads (Strata's normal path) - no outer fifo hold: prepare takes its own engine turn (deadlock found by the new test - same lock, held twice) - response: answers + usage + stages_used ['image-direct']; calib log rows under a ...|D|<qtype> tuple (D = direct image readout) for a separate fit Tests: direct-mode shape (one turn, question after image), bad mode 422. 54/54 green.
…mage was black without them The live vision A/B caught it: chat-path control answered 'Red' (reasoning described the white box and the RED text), while both image arms saw a 'completely black' image. prepare() stages the combined embeddings file on svc.embeddings.path, but it is the CALLER that must pass embeddings=... to engine.generate - that is what turns GEN into GENI and makes the engine load the image. The decision handlers called generate() bare, so the pads attended nothing. Both handlers now take svc.embeddings.path (run()'s exact pattern), clear it, and delete the file when done, as run() does. Also the summary loop's second run-in with negative sentinel ids (fixed the same way as handle_then: Detokenizer + skip None/<0). Also why the first bench arm died at 400: same crash, earlier code. 52/52 green after; live A/B rerun in flight.
…review
P0-1 (blocker): calibration was one GLOBAL Calib - fitting an image tuple
would have silently changed text decisions. Calibs are now keyed by readout
letter ('' text legacy, L, V, D, S) with per-mode files calib-<mode>.json and
fallback to the text fit when a mode has none. state_json reports every mode.
P0-2 (blocker): summary-mode image decisions logged under the text 'L' tuple,
indistinguishable from text rows. handle() now detects an image_summary state
and logs/applies under 'S'.
P0-3 (major): direct mode silently ignored X-Jev-Mode: V. It now 422s with a
clear message (D-V does not exist; the plan's assumed behaviour), pinned by a
test.
Tests: PerTupleCalib (image fit leaves text identity; fallback; swap loads
per-mode files) + DirectVMode 422. 54/54 green.
…wapped
Found by the small image calibration suite + Opus validation (2026-10-15):
image noul raw acc 0.0% with a perfect-inversion Platt fit (a=-2.80) while
choice/score on the same images were correct. Root cause, two compounding bugs
in the noul criteria path:
1. question_block unpacked descriptions as (true_d, false_d) but they are
stored in [No, Yes] label order - so 'Yes means:' printed the FALSE
description. The model read the reversed definitions and answered
consistently - every noul with criteria was inverted.
2. The ': ' separator was baked into the stored descriptions AND re-added by
question_block -> doubled colon ('Yes means: : ...').
Text noul (BoolQ) never sends criteria, so M3.2's 88-90% never exercised the
path; the image suite was its first caller. Fixed: descriptions stored bare in
[No, Yes] order, question_block renders Yes/no lines in donor order.
Also: fit_platt in the suite now REJECTS a <= 0 (a negative slope means the
labels/prompt are wrong, never a legitimate calibrator). New golden tests pin
the exact noul question-block text (Yes means true, No means false, no doubled
colon) and the choice description shape. 54/54 green.
- calib/bench_suite.py: live fit+eval suite (text/image/real); standard top-label ECE15 + mean NLL (the first suite's gold-class-pinned ECE variant was wrong - Justin caught the tables); per-item jsonl logs for adversarial audit; refuses Platt a<=0 and flags T at search boundaries - calib/convert_benchmarks.py: POPE-adv/CV-Bench/CountBenchQA converters with the banned-field selftest (never sends prompt/text/rationales), sha256 image manifest, image-disjoint dev/test, POPE 50/50 by-image sampling - calib/calib-L.json + calib-D.json: the ACTIVE per-tuple fits (2026-10-15, corrected harness): L a/c/T from BoolQ/ARC/SST-5 dev; D from POPE/CVB/CB dev. S stays identity until its own dev run - calib/README.md: refit procedure + donor reference (jevcpp 27B calib.json, tuple p-jevcpp-qwen38-27b|q4_k_xl|timber|v1|V|all, T_choice 1.5861) Live-verified after hot-swap: TEXT-L noul platt_v1, TEXT-L score T_score_v1, IMAGE-D noul platt_v1, IMAGE-D score T_score_v1, IMAGE-S identity (own tuple pending) - every mode answers under its OWN calibrator now.
…ev-vs-jev The PR document package: - pre/post calibration tables (held-out, offline replay of identical items; includes the old-global-calib comparison showing why per-tuple matters) - benchmark vs the donor's OWN m2_baseline.json (same tasks/metrics code): QFN IQ3_XXS MoE matches the 27B within noise on noul/choice, -4.9pp on score - performance matrix measured live (p50/p95/rps: text 272ms, image-direct 275ms, image-summary 1625ms) vs donor m1_perf 71ms/14rps - honest 3.8x gap + what buys it - test strategy: 54/54 unit, live suites with guards, the 1-in-10 adversarial audit protocol + results, banned-field converter selftest, Opus vision plan - artifacts bundled in docs/bench-artifacts/ (reports, item logs, audits, convert manifest)
…er-tuple logs, B1.4 tuple refusal R1 (blocker): the per-tuple fallback contradicted itself — code fell back to the TEXT fit for V/D/S while the comment and README promised identity. Strict now: only ''/L share the legacy calib.json fit; V/D/S use their own calib-<mode>.json or identity. (The S handoff was already threaded: handle() detects image_summary states and applies/logs under 'S'.) G3 (blocker): the donor's fit.py pools a single log by type — running it on a mixed L/V/D/S log file would fit text+image rows into ONE calibrator, the exact global-fit failure the per-tuple split fixed. log_flush now writes one file per readout tuple: data/log-<mode>.jsonl. G13: B1.4 tuple refusal — a calib-<X>.json whose tuple names a different readout letter is refused at load (logged, identity), so copying calib-D.json to calib-S.json cannot apply silently. Legacy calib.json/v1.json keep no mode binding. R7 (verified, no change): the [x / t ...] before apply() in the direct path is the donor's 4dp-round renormalization (softmax -> round -> renormalize -> calibrate), NOT a second temperature. Tests: OpusPrereviewFixes (4 new) — S/D never inherit the text fit, L does; mismatched tuple refused; matching tuple accepted; log split by tuple. Prior tests updated to the new contracts (state_json modes shape, log-<mode> files). 61/61 green.
… Not-ported section R3: the donor's b13_eval ECE scores each bin at its CENTER, our harness at the bin MEAN — both top-label/15-bin but not directly comparable. Added a column recomputing our logged raw_p under the donor's exact formula (BoolQ .035, SST-5 .156 — same direction under either definition), stated the n=100 SE on the -4.9pp score gap, and said plainly that calibrated SST-5 ECE stays ~5x the donor's. Rank 8: 'Not ported / diverged' section — mode gating, fork/batch/naive differential, V2 (superseded by direct mode), chat-KV reuse, B2 LoRA, golden HTTP variants — each with its one-line rationale, plus the open pre-PR list.
…(Opus G10) G2 (security): with no ADMIN_KEY configured, /admin/calib was OPEN — anyone could hot-swap calibrators and read state. Now 404s like the donor's unconfigured --admin-key routes (404, not 401: probing must not learn the surface could exist). Keyed operation unchanged. Test added. G10: calib/token_audit.py — donor A1.3 parity for the QFN tokenizer. Loads the real pack tokenizer (same path as serve/server.py main), checks all 38 labels (A-Z, Yes/No, 0-9) for single-token, collision-free, prefix-free. Result: PASS (reports/token_audit_qfn.json, donor schema). First run failed against the byte stand-in — the audit must point at the pack, which the script now takes as its second arg. 62/62 green.
…l negation, ambiguity (Opus G8) Donor parity for reports/a211_metamorphic.json: (1) REORDER — permute the questions in one request; per-question probabilities must be invariant (Jensen-Shannon < 1e-9, argmax never flips). The port renders each question as its own prompt so this is by construction — but the Yes/No-swap bug proved construction is not evidence, so the test PROVES it. (2) NOUL NEGATION — pos/neg question pairs must flip yes-probability; the >=4/5 gate binds on the GPU arm (mock answers by token, documented). (3) AMBIGUITY — unanswerable questions must not gain confident mass; |p-0.5| proxy for the donor's label mass (BROW is sparse). 3/3 green on the mock; GPU arm reuses the same file with RUN_BASE/RUN_KEY.
… (Opus ranks 3/5/6 + R1)
calib/blockers_gpu.py + artifacts in docs/bench-artifacts/blockers/:
NOISE FLOOR (30 items x 5 runs x {idle, 8-decision, 4chat+4decision}):
- argmax flips: ZERO in every scenario (stability gate passes)
- p99 |dp|: idle noul .070 / choice .039; dec8 .043/.044; chat4dec4 .039/.022
- The MoE drift floor is ~0.04-0.07 abs-prob — 14x the donor's dense 5e-3
gate. Every suite delta must be read against THIS floor: POPE +1.6pp (1
item at n=60) is within it, exactly as Opus R6 suspected.
SOAK: text 2000 req (1200 dec1q / 400 dec3q / 400 chat) = 1 error; image
direct 200 req = 0 errors. VRAM drift 23228->23222 MB (0.026%, donor: 0.07%).
The 1 error is a 503 at the moment the engine self-stopped mid-soak
(strata 'stopping' in logs — TTL/idle unload raced the soak, NOT a jev
crash); restarted cleanly, soak continued green. Documented as infra race.
S-FIT (summary-mode dev): a=.4892 c=.8313 (Tc .05 boundary — choice n=20
degenerate, same as CVB; Ts 1.8702). calib-S.json committed + live: S answers
now platt_v1, D unchanged. Honest finding: summary raw noul acc .75 vs direct
.867 on the same POPE dev — DIRECT BEATS SUMMARY for perception.
CIs/Brier (bootstrap 1000): POPE .067->.065, CVB .200->.179 (CIs overlap
raw), CountBench .018->.005 calibrated (3.7x better, CI clean). Supports
shipping D|choice as-is with CIs stated, not identity.
…donor's dense gate Replaces the one-line 'determinism caveat' with the measured floor (p99 |dp| .04-.07, zero argmax flips in 720 repeats) AND the mechanism, verified in engine source: atomicAdd expert slot assignment (moe_fused.cu place_kernel) -> row order = warp-dispatch order -> non-associative float sums differ per run -> compounds across layers -> IQ3_XXS keeps it above dequant noise -> softmax amplifies near p .5-.9. Same floor across idle/ dec8/chat4dec4 (per-request kernel scheduling, not load). Donor's 5e-3 was a dense-model gate. Mitigation note: deterministic placement = upstream engine follow-up. Reviewer guidance: single-item deltas under the floor are not signal.
…is the 260ms 3-arm + floor decomposition (live): /health p50 1ms, /v1/models 1ms, systemone 262ms -> the serve layer is innocent; the cost is the engine turn (prefill-dominated). Fresh TCP 47ms, proxy hop 44ms (keep-alive direct p50 255ms). Streaming was never in the decision path (Content-Length bodies, donor refuses stream:true). Donor gap = no prefix amortization (it forked a cached prefix; we re-prefill) — state-KV reuse projects 60-80ms, queued as the suffix-reuse follow-up. Repeat-identical requests already hit the engine's prefix mount (278-300ms even at 470-token states).
…guard the class
calib-S.json shipped "a"/"c" while jev_calib.load() reads a_noul/c_noul, so the summary-mode Platt
was silently identity - while T_choice=0.05 was honoured and apply() only rejects <=0.01, giving
exponent 20 (near one-hot confidence). Both are now fixed from a real summary-mode run
(identity-served, 200 items): a_noul=0.435457 c_noul=0.818939 T_choice=5.027682.
T_score is deliberately left at 1.0: the dev fit (T=1.846) improved dev NLL 28% but the TEST split
got worse (NLL 0.3695->0.4067, ECE 0.081->0.156, n=20+20), which fails the documented acceptance rule
(NLL_post <= NLL_raw) - so that leg stays uncalibrated until a bigger dev split exists.
Guards, so this class cannot recur silently:
scripts/check_calib.py - loads each file through jev_calib.load() and fails on keys the loader
does not read, declared-vs-effective mismatches, degenerate T, tuple
letter mismatch
STRATA_JEV_CALIB_STRICT=1 - opt-in (default path unchanged): jev_calib.load() REFUSES such a file
and logs why, instead of half-applying it
5 new tests in serve/test_jev_calib.py cover default behaviour, strict refusal (mis-keyed,
degenerate T), a clean file still loading, and the tuple refusal
scripts/fit_tuple_from_log.py turns a suite item log into an installable calib-<letter>.json and
refuses to write anything check_calib would reject.
…refusal; field-order test leonardomurillo (Niko1221#3643): labels are 0-9 up to ten options, a-z for 11-26, more than 26 refused, and a request whose label is not a SINGLE TOKEN must be refused rather than scored on a truncated path. serve/jev.py: _label_scheme()/choice_labels() - STRATA_JEV_LABELS=upper (default) | digit-lower. Unknown value fails closed at import, like STRATA_JEV_MODES. question_block now renders qdef[render_labels] instead of assuming LETTERS, so both the option list and the readout use the same alphabet. _label_token_problem() - refuses a multi-token label with 422 questions.<id>.criteria in both handle() and handle_image(). Default stays upper: every calibrator on disk was fitted against A-Z, so the scheme is opt-in until the choice readouts are re-baselined (AGENTS.md: default path byte-identical). 6 new tests (41 in serve.test_systemone): default unchanged, digit-lower ordering, 2..26 range, block render, multi-token refusal, noul/score unaffected. scripts/order_test.py - his field-order ask. Same items twice (forward, reversed), alternating which order goes first so a cold first prefill is not mistaken for an order effect, and latency reported by request position as well as by order. Result at two fields: per-field path 199/200 agreement (1 flip, mean |dp| 0.009); one-prompt path 167/200 (33 flips, 32 yes->no, |dp| 0.152) with the reversed order ACCIDENTALLY 3.5 points more accurate - so the consolidated path needs the field order pinned.
…sume-safe row persistence
…OWAT parse, correct the calibration and label docs - src: bring row capture (set_row_capture / capture_window_rows / brow_row_captured), jevpos= parsing and the INFO jevpos advertisement into git - the shipped binary had them, the committed source did not, so the 16-commit port would have shipped a server talking to an engine with no capture - generate.cpp: BROWAT token ids range-checked before the int32 cast and a readable parse failure (trailing junk / >4096 ids), BROWROWS matched exactly instead of as a prefix - server.py: a malformed BROWROWSD answer now ends the engine like every other fail-closed read - serve/jev.py: one _row_capture() gate for both row paths, documenting the two BROWAT addressings - docs: JEV-SYSTEMONE multi-field/label sections, calibration corrected from 'argmax-preserving' to 'the Platt offset is a threshold', JEV-PR 4a-bis (re-measured, per-task not portable, arm table), test counts, calib/README status, scripts/README
Independent reviewer (fail-closed) rejected the diff on four logic errors; all are addressed:
- absolute-vs-window row addressing: the engine half (row capture) is now committed, and BROWAT's
silent fall-back to window addressing is closed by BROWCAP: the server verifies it kept exactly the
rows it named BEFORE reading them, else one prompt per field
- BROWROWS was reachable by nothing and could never pass: documented as a window-addressing
diagnostic, with captured_rows() (BROWCAP) as the real guard
- BROWAT narrowing: the row is bounded to int32 like the token ids (4294967301 used to answer row 5)
- jevpos longer than 32 was dropped silently -> a >32-field request now never takes the row path;
token ids are capped before the push, so a huge line cannot allocate first
- strict calib mode bounds a_noul/c_noul with isfinite (JSON accepts NaN, which made every answer NaN)
- the direct-image path now shares _finish_answer('D'), so the 'one place' claim is true
- label refusal is documented as the one default-path behaviour change, and tested at HTTP level
- tests: capture-dropped fallback (B path + hybrid), 33-field cap, multi-token label 422, seam move
…le-token refusal, no silent jevpos truncation
MiniMax-M3 (mm-builder) returned reject with one blocker and three majors; adopted:
- BLOCKER jev_calib._strict_problem: the temperature guard used only <=/>=, which NaN passes (every
comparison against NaN is False), so a calib file with T_choice: NaN loaded under STRICT and put NaN
probabilities, NaN margin and a NaN score into the HTTP body. isfinite now comes first, like the
Platt guard above it, with a test that loads exactly such a file.
- MAJOR the single-token label refusal skipped score questions, which render '0'..'9' and read them the
same truncating way (encode, then [:1]): now both types are covered, with an HTTP-level test.
- MINOR jevpos longer than 32 was silently truncated engine-side ('never a guessed position' said the
comment); the list is now REFUSED whole, so the server sees BROWCAP 0 and falls back.
- MINOR STRATA_JEVROWS_TRACE and BROWCAP documented in usage().
- MAJOR 'count is not a set check' refuted in the code: capture only ever takes named positions, so
captured subset-of named + equal size implies equal sets; the comment says what would break it.
…in-on-default declared GLM-5.3 (zai-builder) reached the same blocker and the same score-label major as MiniMax, plus three new findings; adopted: - MAJOR the row capture was cleared only at the read loop, so a request that failed earlier (oversize prompt, tokenization, cancel) left the previous request's rows readable by a later BROWAT - stale rows for a prompt that is no longer the last one, and captured_rows()>0 cannot tell stale from fresh. The capture is now cleared at the top of EVERY request. - MINOR the jevpos parser accepted '12x' as 12 (atoll stops at the junk); every character must be a digit. - MINOR the hybrid comments claimed the decode step was skipped - it is not (max_new stays 1, the row is read back by position); comments and docstring corrected. - MINOR MAX_NAMED_ROWS comment described a DROP; the engine now refuses the whole list. - MAJOR 'pinning is default-on but not declared' - true and deliberate: declared in JEV-SYSTEMONE next to the label refusal, with what it costs (answer identity rests on checkpoint-resume exactness, measured <=.008 on field 1, 199/200 order agreement) and how to turn it off (STRATA_JEV_PIN=0). - tests: BROWCAP wire exchange (roundtrip, zero, unexpected op, malformed) now covered; dead un-shadow line removed.
The rebased base's STRATA_ADMIT_MERGE_FIRST path moves the last tokens into a different first window. A request that named head rows (jevpos=) needs every named position to run in a window the capture sees, so that path now declines instead of silently leaving positions uncaptured.
…ld not be reproduced
The `jevpos=` field parser was a braceless for whose `else if`/`else` bound to the inner digit test, so the push ran once per DIGIT character and the 32-entry cap counted digits. Measured on the exact block, compiled: `63,128` parsed to [63,63,128,128,128], and a legal 32-position list was refused WHOLE - so the consolidated path silently fell back to one prompt per field above ~11 fields on a real prompt (3-digit positions), with only a log line as evidence. Found independently by two reviewers in round 2 (2026-10-23); it had no test on either side of the wire, which is how it survived three reviews. The parse moves to include/strata/program/jevpos.hpp (header-only, no model, no GPU) so the string -> vector rule is testable: tests/program/jevpos_test.cpp pins multi-digit positions, 32 entries (including 32 three-digit ones), the 33rd entry refused whole, and the malformed cases (`12x`, `4;9`, `-4`, " 4", a 19-digit field) plus the edges the engine always allowed (empty list, `4,,9`, `4,`, `0`). Digits are accumulated by hand, so no atoll and no overflow past kMaxDigits. The wire behaviour is otherwise unchanged: a malformed list is still no list, never a guessed position, and never a truncated one. docs/JEV-SYSTEMONE.md now states the cap and corrects a guarantee the reviewers disproved: the per-field readout still calls BROW, so an engine built without --jev-brow answers `ERR BROW:` and the server ends and restarts it rather than serving a 503. Verified: jevpos_test (24 checks) passes standalone and under ctest; split_rules_test and verify_brow_at_test still pass; serve.test_systemone + serve.test_jev_calib 84/84 OK; `cmake --build build --target strata` links clean (one pre-existing unused-variable warning at generate.cpp:11780, untouched here).
…ed on the fixed engine Every B-arm number in this document was measured while the consolidated path silently fell back to one prompt per field (the jevpos parser pushed a position once per digit character and counted its 32-entry cap in digits, fixed in f2d2621f), so the old table's B column was really a per-field column on an older base. The "981 ms / 1.33x faster at two fields" claim is withdrawn. Measured 2026-10-23 on engine a2182599 (jev-port @ f2d2621f), RTX 3090, IQ3_XXS, median of 3 (scripts/rowreadout_probe.py; artifacts in the session's bench dir): N | C (default) | H (opt-in) | B (one prefill, opt-in) | B prompt tokens 1 | 209.7 ms (86 tok) | 209.3 ms (86) | 235.2 ms (86) | same 2 | 549.1 ms (189) | 915.1 ms (189) | 1013.2 ms (134) | 29% fewer 3 | 1231.7 ms (299) | 1470.7 ms (299) | 1299.5 ms (189) | 37% fewer 4 | 1405.4 ms (387) | 1826.9 ms (387) | 1447.2 ms (222) | 43% fewer 5 | 1698.5 ms (490) | 2287.5 ms (490) | 1726.6 ms (270) | 45% fewer So B uses 29-45% fewer prompt tokens from two fields up and is slower in wall clock at every N >= 2: C's pinned checkpoint makes each extra field cheaper than B's single longer prefill. Accuracy is unchanged (banking77 200 items, 2 fields: C .660 vs B .665 card, .750 vs .755 group, McNemar both-right 2 vs 3, p = 1.000) and field order still matters on B (147/200 agreement, mean |dp| .1326, 52 flips one way against 1, McNemar p = 0.0008, first/second request latency controlled at 681.7 vs 682.4 ms). The C row also carries the numbers measured today (1/2/3/4/5 fields 210/549/1232/1405/1699 ms) instead of the unreproducible older set, and the "route by field count" advice no longer recommends B at N=2. Also fixes scripts/jevport-arm.sh: its labels check read a variable filtered to ROWS|HYBRID, so `--labels digit-lower` reported MISMATCH after setting the env correctly. Found while running the label A/B, which then measured 160/160 identical decisions between the two alphabets.
One prompt for N fields lets each later field see the earlier ones, so the caller's key order decided the answers for the same set of questions. Measured live on 200 banking77 items before this change: reversing the two fields agreed on 147/200, 52 flips one way against 1 the other, mean |dp| .1326, McNemar p=0.0008 - i.e. the opt-in path was not reproducible under a caller that reorders its fields. The fields are now sorted by qid before the consolidated prompt is rendered. The same set of questions therefore renders the same prompt and names the same cue rows whoever asks, and because answers are keyed by qid the response shape does not change. This does not remove the intrinsic "later fields see earlier questions" bias - that is what one consolidated prompt means - it removes the part that depended on the caller. Measured after, same 200 items, same run shape (scripts/order_test.py --alternate 1): is_card agreement .995 (199/200), flips 0 one way / 1 the other, mean |dp| .007, McNemar p=1.0 group agreement 1.000 (200/200), accuracy .750 forward and reversed latency 683.5 vs 676.4 ms, prompt tokens 145.0 both The per-question path is untouched (this branch is only taken with STRATA_JEV_ROWS=1). serve/test_systemone.py::RowReadout::test_field_order_is_canonical pins the contract: the same questions in either order must render an identical prompt and name identical rows.
… is not a latency path B row + PR section now carry the post-fix order numbers (199/200 under field reversal, |dp| .007, McNemar p=1.0, vs 147/200 and p=.0008 before), the fact that which field goes FIRST still moves answers (is_card .660 asked first vs .785 asked second on the same 200 items), and the reason no wall-clock win exists: the two-stage pipelined prefill keeps no head rows, so a capture request runs on the serial window path.
…sumed from Arm C pins the fields' shared prefix, so every field after the first resumes from the checkpoint the pin marks. None of them has anything of its own to park, and parking is real work: about 75 ms per field on the 3090. Measured on the 3090 (200 banking77 items, 3 reps, medians, warm), same engine, only this branch of the code changed: fields 1 2 3 4 5 park 657 1047 1050 1260 1527 ms no park 726 1119 878 985 1214 ms -20% to -24% from three fields up. Field 1 is untouched by design (it creates the checkpoint the pin marks); its 657 -> 726 is not a regression but the cold first-request effect measured separately at 747.9 ms against 569.5 ms for the second request, and it moves between runs. The suppression is confined to exactly the fields that resume: field 1 keeps its park, and a run without a pin suppresses nothing, so no other path changes. It previously lived inside the hybrid row-capture branch, which meant it only applied to callers holding both a pin and the capture hook - the plain pinned path was still paying for every park. Hoisting it one level out fixes that. Answers: 200 items, park-suppressed against the fully unpinned read - is_card 199/200 agreement (1 flip), group 198/200, mean |dp_yes| .0104 (median .0047), accuracy .665/.745 against .660/.745. The same band the pin itself was accepted on, so this is a caching change, not a readout change.
leonardomurillo's 7-arm run (Niko1221#3743) showed our per-field is_card at .766-.805 against his .870-.883 (McNemar p=0.008). The error analysis on our own logs (200 banking77 items): 65 of 67 errors are false positives, the group prompt - which shows the taxonomy - is right where is_card is wrong on 40 items, and the best threshold moves .665 to .810 while AUC stays low. Cause: a derived binary field asked without its sibling categories lets the model answer the everyday meaning; his field catalogue pins the taxonomy meaning. Fix: an opt-in top-level 'context' field (string or list of strings) rendered VERBATIM between the state and the question block, in every field's prompt - per-field, V-mode branches, and the consolidated rows path (unit 0, so the pin/jevpos math is unchanged). No context: the render is byte-identical to the donor's frozen shape, so every fitted calibrator still applies. Wrong type: 422 field 'context' (fail closed). Context-bearing requests read from a shifted distribution - calibrate for the use. 8 new tests (serve/test_jev_context.py): default render unchanged, placement per field, list join, blank-only == absent, 422 on wrong types, V branches, consolidated STRATA_JEV_ROWS=1. Suites: test_jev_context 8/8, test_systemone 55/55, jev discovery 75/75.
…fit tooling
swap() keys calibrators by READOUT LETTER, so one calib-L.json served every text
task. A fit is a domain claim; the shipped one (a_noul 0.6127, c_noul +0.6539,
fitted on BoolQ60/ARC60/SST5-60) was applied to a balanced binary grouping task
that needs c_noul ~ -1.56.
Measured on 500 gold-labelled banking77 items, one GPU, same binary, only the
calibrator differing (5-fold CV, held-out folds):
shipped calib-L acc 0.664 brier 0.2325 ECE 0.275
identity (none) acc 0.760 brier 0.1983 ECE 0.218
refit this task acc 0.816 brier 0.1315 ECE 0.109
Doing nothing beat what shipped (+9.6 points, paired McNemar p<0.001). The
calibrator admitted from raw p>0.256 and swept 187 undecided items into "yes":
specificity 0.360 vs 0.584.
Changes:
- Calib carries the `task` it was fitted for; loader reads it; it joins
_KNOWN_KEYS so STRATA_JEV_CALIB_STRICT does not reject task-tagged files.
- STRATA_JEV_TASK names the runtime task. A fit whose task does not match is NOT
applied: identity beats a stranger's intercept, so the refusal is the safe
default and it is reported as calib:"identity" plus one log line. An UNTAGGED
fit is refused too once a task tag is set. Unset tag = legacy behaviour,
byte-identical to the last release.
- scripts/refit_calib.py: collect readouts (or reuse an eval arm), stratified
dev/holdout split, damped-Newton Platt fit for noul, temperature for
choice/score, and a GATE that refuses to write a fit which does not beat
identity on accuracy AND ECE. Writes task + provenance with holdout numbers.
- docs/CALIBRATION.md + docs/examples/calib-items-example.jsonl: the operator
recipe (label ~25-100 items, fit on half, gate on the other half, deploy with a
task tag, verify twice).
Verification: serve.test_jev_calib + serve.test_jev_calib_task 39/39 OK,
serve.test_systemone 55/55 OK (container venv, /cfg/scratch-test copy).
Deployed and held-out verified separately: fit on 250 dev items served the
untouched 250 -> acc 0.816, brier 0.1285; vs shipped on the same items fixed 59 /
broke 21, McNemar p=2.5e-05; repeat pass 247/250 identical decisions.
Caveat recorded: the engine is NOT bit-exact run to run (|dp_yes| up to 5.7e-2),
so cross-run differences under ~1.2% accuracy are noise.
…r log
A static fit answers "what boundary was right last month". dynamic_calib.py moves
the boundary from LIVE traffic without traffic labels: the labelled dev set gives
the confusion matrix C[pred][y] once, the server already logs the prediction
marginal (calib/data/log-<mode>.jsonl, one row per answer with raw_p), and BBSE
solves C*pi = q for the traffic class prior, which sets the threshold.
Measured on 500 gold-labelled banking77 items (labels used only for scoring):
shipped static fit (wrong task) acc 0.664
identity, no calibration acc 0.764
BBSE prior-matched, NO traffic labels acc 0.816
oracle prior-matched (upper bound) acc 0.816
labelled offline refit acc 0.816
Unlabelled base-rate estimate 0.484 vs a true 0.500. Streaming five 50-item
batches with shrinkage + EMA: dynamic 0.828 vs static 0.764; single-batch BBSE is
noisy (n=50 mean 0.488, sd 0.108), hence shrinkage is mandatory.
On this rig's real serving log (984 rows): estimated prior 0.493 (true 0.500),
recommended c_noul -1.6186 vs the hand-fitted -1.5607 -- agreement to 0.06 from
independent information.
Deliberately a scheduled job writing a calib file, NOT an in-process control loop:
bounded step (--max-step), shrinkage toward the prior (--n0/--ema), a refusal when
the estimate is implausibly far from the expected prior (--alarm, catches a shift or
broken logging), a singularity refusal when C cannot be inverted, and every update
stamped into a `dynamic` block with n, estimate, threshold and c_prev->c_new.
docs/CALIBRATION.md gains §10: what makes it possible, the measured table, the
guard rails, and when NOT to use it (no dev set / near-binary readout / real labels
available). Verified end to end against the live log; validator clean on our stanzas.
…t files CI "Minimal serving runtime" failed on 3 tests in serve/test_systemone.py from commit b5a8f48 onward (passed at 19:55, failed on the 22:02 and 22:08 runs): test_noul_answer got 0.9570665554 expected 0.982 test_score_answer got 0.9524334890 expected 0.9887011299 test_choice_answer got 0.0219656216 expected 0.0178982102 Cause: my tearDown called jev_calib._set_for_test(None), which defaults to mode "", so the fit my test had parked in _ACTIVE["L"] stayed there. unittest discovers files alphabetically, so this file ran before test_systemone and the server those tests start applied the leftover Calib(a_noul=0.6127, c_noul=+0.6539, T_choice=1.0557, T_score=2.263) instead of identity. All three CI values reproduce exactly from that object: 0.982 -> sigmoid(0.6127*logit+0.6539) = 0.9570665554 0.9887011299 -> T_score=2.263 = 0.9524334890 0.0178982102 -> T_choice=1.0557 = 0.0219656216 Fix: setUp snapshots the whole _ACTIVE dict, the task override and the mismatch log; tearDown restores all of them. New test_restore_clears_every_mode pins the isolation, asserting active("L") and active("") are both None after restore and that apply(mode="L") then reports identity with probabilities untouched. Verified in the container venv against the repo tree: test_systemone alone 55/55 OK test_jev_calib_task + test_systemone, both orders 63/63 OK test_jev_calib + test_jev_calib_task + test_systemone 95/95 OK (pre-fix, the CI ordering reproduced the same 3 failures with the same values)
…sk fit
calib/calib-L.json shipped a_noul=0.6127, c_noul=+0.6539 - fitted on BoolQ60/ARC60/SST5-60 - and
that file is applied to EVERY text task. The task-scoping guard added in this PR only fires when
STRATA_JEV_TASK is set; with the tag unset (the default, documented as 'legacy: apply whatever
loads') the cross-task fit is applied silently. Measured cost on 500 labelled banking77 items:
yes-rate 67.6% -> 80.4%, accuracy 0.760 -> 0.664 (-9.6 points, p<0.001).
This is the one defect in the accuracy story that is still live by default, so:
* calib/calib-L.json is now an identity fit (a_noul=1.0, c_noul=0.0) with provenance saying why.
T_choice/T_score are untouched - their effect was not measured and a review patch must not guess.
* the historical fit is preserved byte-for-byte as calib/calib-L-boolq-arc-sst5.json carrying
"task": "boolq-arc-sst5", so a deployment that sets a task tag has it refused by the existing
guard, and anyone who wants the old behaviour can point STRATA_JEV_CALIB_DIR at it.
* docs/CALIBRATION.md rule 1 no longer claims the loader fails closed unconditionally - it fails
closed when a tag is set, which is the actual behaviour.
* test_jev_calib_task.py gains test_shipped_calib_L_is_identity, so reintroducing a cross-task
default fails the suite instead of costing 9.6 points in production.
No engine change, no CUDA rebuild: one data file, one doc paragraph, one regression test.
…hat actually runs The loader reads a_noul/c_noul; calib-S.json shipped "a"/"c". Those were silently dropped, so the summary-mode noul fit (a=0.4892, c=+0.8313) has NEVER been applied while T_choice/T_score were. This is exactly the defect _strict_problem() describes and refuses under STRATA_JEV_CALIB_STRICT - but strict is opt-in, so the file shipped anyway. The declared values now match what runs (a_noul=1.0, c_noul=0.0): ZERO behaviour change, since identity is what the loader was already producing. T_choice/T_score are untouched - they ARE read and changing them would be an unmeasured behaviour change. test_no_shipped_fit_declares_unread_keys guards every calib-*.json against unknown or legacy keys, so a half-fit cannot ship again. Left deliberately unresolved and documented in the file: T_choice=0.05 sits at the degenerate edge _strict_problem() refuses, so STRICT mode cannot become the default until that temperature is refitted from measurement. That is a measurement task, not a review patch.
…r own
Justin's call, and it is the right one: a fit is a claim about one data distribution, so the only
person who should be making it is the one who measured it. Removing the parameters removes a whole
class of defect instead of patching instances of it.
Deleted: calib/calib-L.json, calib-D.json, calib-S.json (and the tagged archive added earlier).
The default is identity everywhere until a deployment fits and tags its own calibrator.
Added / fixed:
* scripts/calib_precondition.py - the gate that answers 'is there anything for a fit to fix',
ahead of refit_calib.py. Verdicts on a real 500-item arm: noul mismatch/se 9.54 -> ADAPT,
choice 0.72 -> IDENTITY. It is the cheap question and it usually answers no.
* scripts/check_calib.py no longer imports the loader from a hardcoded /opt/data/builds path; it
imports from the checkout it lives in, so a user can actually run it.
* calib/README.md rewritten: no fits ship, here is the 4-step procedure (precondition -> collect
labelled items -> refit with dev/holdout + gate -> check_calib), the file format, and the old
parameter sets kept only as a documented failure example with the 9.6-point measurement.
* docs/CALIBRATION.md rule 1 now states the actual contract: nothing ships; the tag guard fires
when STRATA_JEV_TASK is set, and with no tag the loader applies whatever loads - which is why
nothing loads.
Tests (serve.test_jev_calib_task 11/11 OK):
* test_no_calibrator_files_ship - the default is identity BECAUSE nothing ships, not because a
shipped file happens to be harmless. Fails if anyone adds a calib-*.json back.
* test_missing_calibrator_yields_identity_and_says_so - absent file -> loaded=False, identity,
calib id 'identity'.
* test_legacy_keys_are_still_refused_under_strict - the a/c vs a_noul/c_noul trap still detected,
now against a temp file since no fixture exists to carry it.
…neralised-first
Justin's requirement: the generalised behaviour must work with zero configuration, and a user who
wants better results on their own data gets a clean, ordered process. This is that document.
docs/OPTIMISING.md covers, in measured-payoff order:
0. define your categories in words (criteria descriptions) - neutral on a fixed 5-bucket task
(.814 vs .822, p=0.712) but it moved 78/500 decisions, so it steers the boundary; use it when
your categories need disambiguating and measure the direction.
1. derive a yes/no field from a choice question - .746 -> .858 (p=1.2e-10), one fewer readout,
~35% less latency. Marked with an availability note: derived_from ships in the stacked
derived-fields change, not in the base port, and the grep to check for it is in the doc.
2. build a labelled set - size/balance guidance, accept sets (90 of our 500 items needed them), and
the gold-discipline warning about substring keys understating accuracy by ~8 points.
3. measure a baseline - paired tests only, plus the two traps we paid for: re-recording the same
items is not replication, and a parameter chosen on the whole set is fitted, not measured.
4. precondition gate (scripts/calib_precondition.py) - usually answers IDENTITY.
5. refit on your own data with dev/holdout and the gate (scripts/refit_calib.py).
6. validate the file against the real loader (scripts/check_calib.py), then serve it tagged.
Plus 'what not to do' with the measured reason for each: cross-task fits (-9.6 / -3.3 / -8.3),
threshold tuning without a holdout (selection bias), expecting calibration to fix confident errors
(61 of 88), question placement (p=0.0125 worse), score readouts on nominal labels (meaningless).
Every scripts/ path in the document is verified to exist in this tree and to run.
Also: docs/README.md gains a row in the Use Strata table, calib/README.md and docs/CALIBRATION.md
both point here as the practical entry point (calibration is step 5 of 6, and step 1 buys more).
Verified: zero-config path is clean - empty calibration directory starts without error, reports
loaded:false for every mode, and leaves noul and choice probabilities untouched with calib 'identity'.
serve.test_jev_calib_task 11/11 OK.
Justin's concern, stated exactly: the PR must not report numbers produced by invalid calibration.
Audited every accuracy claim and gave each one its no-fit counterpart.
docs/JEV-PR.md 4b (the donor comparison) quoted only post-calibration figures for this port, which
are not reproducible from a fresh checkout now that nothing ships. It now carries both columns -
'no fit (default)' and 'fitted on own dev split' - sourced from bench-artifacts/2026-10-23-text-ab-identity/,
the run taken with the calibrator removed. The conclusion does not move: calibration changes 4-5
answers out of 100 on BoolQ and ZERO on ARC and SST-5, so nothing in the comparison depended on it,
but a reader no longer has to infer that. SST-5 ECE is quoted fitted AND unfitted (0.292 -> 0.161),
with the fitted T_score described as damage control on a weak readout rather than a shipped parameter.
Also corrected three stale statements that still described the deleted files as shipping:
* docs/JEV-SYSTEMONE.md - the per-tuple file paragraph now says no calibrator ships, why
(-9.6 points cross-task), and points at calib/README.md + docs/OPTIMISING.md.
* docs/JEV-PR.md 4a - the 'Fitted: L a=.6127 ...' caption now marks those values as what the repo
USED to ship and names the raw columns as the quotable ones.
* docs/JEV-PR.md - 'refitting on top of the shipped calibrator' and the calib-D-served image line
now say previously shipped.
Already honest and left alone: bench-artifacts/2026-10-23-text-ab/NOTES.md marks the compounded run
SUPERSEDED in favour of the identity set; 4a-bis already withdraws the dev-split readings; the image
claims show raw->cal pairs and measure a single flip. serve.test_jev_calib_task 11/11 OK.
justindressler
force-pushed
the
jev-port-r2
branch
from
November 4, 2026 12:55
aade28f to
79ec7f9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
The jev decision surface currently ships calibrator parameters fitted on someone else's data, and
applies them to every task that uses the same readout letter. Measured on 500 labelled items from a
different task, that costs 9.6 accuracy points (.760 -> .664, p<0.001). This PR removes the
parameters and ships the tools to fit your own.
Stacked on
jev-port-r2(Niko1221#3643). Nothing in Niko1221#3643 changes unless this is merged into it.The three defects
calib/calib-L.jsoncarrieda_noul=0.6127, c_noul=+0.6539fitted on BoolQ60/ARC60/SST5-60, and it is applied to every text task. The task-scoping guard
added in [Engine] Jev System One decision endpoint (/v1/systemone): typed decisions with calibrated probabilities Niko1221/Strata#3643 only fires when
STRATA_JEV_TASKis set; the default is unset, documented as"legacy: apply whatever loads". So the defect that guard exists to close was still live by default.
calib-S.jsondeclared"a"/"c"; the loader readsa_noul/c_noul. Its Platt was never applied - for weeks - while itsT_choice=0.05was honoured(exponent 20, near one-hot).
_strict_problem()describes this exact shape and refuses it, butstrict mode is opt-in.
zeroing only
crestored both. The slope is roughly portable, the offset is a prior.The change
"calib": "identity". Verified: an empty calibration directory starts clean, reportsloaded: falsefor every mode, and leavesnoulandchoiceprobabilities untouched.scripts/calib_precondition.py(ported - answers "is there anythingfor a fit to fix", usually IDENTITY),
scripts/refit_calib.py(dev/holdout split, writes nothingunless it beats identity on both accuracy and ECE),
scripts/check_calib.py(validates a filethrough the real loader; previously imported it from a hardcoded path so only one machine could run
it).
docs/OPTIMISING.md- the ordered process for a user's own dataset, with the measured effect ofeach lever: define your categories, derive fields instead of asking for them, build a labelled set,
measure, gate, fit, validate. Plus "what not to do", each with its measured reason.
quoted only post-calibration figures, which are not reproducible from a fresh checkout. It now
carries both columns, sourced from the identity run. The conclusion does not move: calibration
changes 4-5 answers out of 100 on BoolQ and zero on ARC and SST-5.
Tests
serve.test_jev_calib_task11/11 OK, including three new guards:test_no_calibrator_files_ship- the default must be identity because nothing ships, not because afile happens to be harmless. Fails if anyone adds a
calib-*.jsonback.test_missing_calibrator_yields_identity_and_says_so- absent file ->loaded=False, identity,calib id
identity.test_legacy_keys_are_still_refused_under_strict- thea/ctrap still detected, now against atemp file since no fixture ships to carry it.
serve/test_jev_calib.pyreferencescalib-*.jsononly through temp-dir fixtures, so the deletiondoes not touch it.
Behaviour change to note in the release
A deployment relying on the shipped fits will now run identity. Measured direction: improvement on any
task other than BoolQ/ARC/SST5, and neutral on those.