[Engine] Jev System One decision endpoint (/v1/systemone): typed decisions with calibrated probabilities by justindressler · Pull Request #3643 · Niko1221/Strata

justindressler · 2026-10-21T19:58:20Z

Problem and result

Strata can answer questions, but it cannot decide. A typed decision — "is this prompt a threat?", "which of these three actions?", "how urgent, 0 to 4?" — had no surface of its own: the caller had to generate prose and parse it, which is slow, uncalibrated and impossible to automate against.

This PR adds an opt-in decision endpoint, POST /v1/systemone, plus the same readout folded into the chat surface. A question is answered from one decode step: the label logits are read off the final row of the last verify window over a new BROW engine wire line, and the ported jev readout math turns that row into a typed answer with a calibrated probability. Nothing is generated to produce it.

Before: no typed decision surface at all. After, measured on the RTX 3090 (QFN, Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS, engine built from this revision): a state carrying three questions of different types — noul, choice, score — answers each from its own readout, every answer carries its own calibrator id (platt_v1, T_choice_v1, T_score_v1), a margin (top1−top2, the cascade-routing number) and usage.readout_tokens: 3.

It is opt-in, and the default path is untouched. --jev-brow is off by default and the engine refuses BROW without it (ERR BROW, fail-closed, the session stays in step). STRATA_JEV_MODES unset means chat only: a process that never mentions jev behaves exactly like stock Strata, and the decision surfaces answer 404 not_found_error rather than advertising themselves.

Related discussion

Supersedes [Engine] Jev System One decision endpoint (/v1/systemone): typed decisions with calibrated probabilities #2639, the same change on the pre-0.2.0 base. A maintainer closed that one on 2026-10-19 in the 0.2.0-beta tracker reset, with "Closed here does not mean rejected" and a four-step recipe: rebase on main, re-run on 0.2.0-beta, area-tagged title, template filled in, then /reopen. This revision is that rebase and that validation; the /reopen comment follows.
TwoAnts (2026-10-15) asked about using jev to check whether a stop token is where it should be. That is why the readout is one decode step over the final row: it is cheap enough to run on every answer as a check, not just as a classifier.
No open issue covers this feature. The design, the API contract and the benchmark detail live in docs/JEV-SYSTEMONE.md and docs/JEV-PR.md, both in this PR.

Approach

Ported from jevcpp's src/server.cpp onto Strata's serve layer, wire-compatible with the donor (request/response shapes, the 422 with a field error contract, the calibration math).

Engine side (opt-in): one flag, --jev-brow, and one line protocol on stdin/stdout. BROW <tok> <tok> ... answers the final row of the last accepted verify window, gathered to the ids asked: BROWD <n> <tok>:<logit> .... It fails closed — flag off, token out of range, or no window yet all answer ERR and leave the engine in step. Verifier::brow_row() guards the null, empty-range, zombie-window and cudaMemcpy-failure paths separately, each with its own error string and return false; no path returns a partial or invented row.
Serve side: /v1/systemone (text, image and decide-then-generate) and the jev extension on /v1/chat/completions, both gated by STRATA_JEV_MODES. Calibration is post-hoc, monotone and argmax-preserving: it reshapes probabilities and can never pick a different label.
Compatibility: no existing request, engine flag or config gains a cost. The decision path is reachable only with the flag, the mode and a request that asks for it.

The rebase onto 011c9e60 replayed all 42 commits with zero conflicts. To show the feature code itself did not change in the move, the 38 added files are byte-identical to the pre-rebase head, and for the 5 modified files the set of added lines is identical (CMakeLists 5, verify.hpp 8, server.py 128, verify.cpp 32, generate.cpp 36).

Validation

Revision 140f0a93, rebased onto 011c9e60. Build host: WSL2, GNU 14.2.0, nvcc 13.0.88 (STRATA_ENABLE_CUDA=ON, CMAKE_CUDA_ARCHITECTURES=86, -j1). GPU: RTX 3090 (24 GiB), driver as shipped in the runtime container.

Passed:

CUDA build — cmake --build build-cuda -j1 → exit 0, binary 77,075,904 bytes, 22 warnings. Two of them are in files this PR touches and both are pre-existing lines, not added ones: verify.cpp unused f (only used in the _WIN32 branch) and generate.cpp unused w (the batch timing line). 0 warnings introduced.
ctest, 156 tests configured (-DSTRATA_BUILD_TESTS=ON): 82 passed. verify_brow_test — the C++ half of this feature, CUDA builds only — prints verify_brow_test: OK (fail-closed pre-run contract).
The same GPU tests, re-run where a driver exists (the build container cannot create a CUDA context; every one of its 71 GPU failures is the single line CUDA driver version is insufficient for CUDA runtime version). Inside the runtime container: verify_parity PASS, verify_batch_parity PASS, dequant_s2_parity bit-exact against the scalar dequantizer, mmvq_multi_parity ok, kv_q8_parity OK (worst INT8-vs-FP16 error 0.590 quantization steps), sampler_parity 0 failures, s2_gemv_parity agrees with the scalar decode, qsa_topk_parity --selftest PASS.
Terminal and Control suites (pytest tests): 1,639 passed, 16 failed, 48 skipped, 234 subtests passed in 552.75 s. All 16 failures are pre-existing and none touches this PR's files: each failing file was re-run on a clean 011c9e60 tree, where the same tests fail (test_shared_models_folder 9, test_discovery 2, test_ready, test_procs, test_private_files, test_no_acquisition, test_launch_tui — the last two give identical 1-failure results on both trees).
Installer tests (tools/test_setup_*.py, run as scripts as AGENTS.md documents): 46 of 48 exit 0; test_setup_golden.py (48 failing assertions) and test_setup_unsloth.py (4) fail identically on the clean 011c9e60 tree.
The 71 GPU-class ctest tests, re-run where a driver exists — 70 pass. The one exception is prefill_mmq_kquant_test, which needs libstrata_kq.so beside the engine; that library is not produced by this CMake configuration, so the test reports Q4_K gate/up-0: not covered by this build and the experts take the FP16 prompt path.
Decision test suite — 73 tests, all OK: test_systemone 13, test_jev_calib 26, test_jev_then 9, test_jev_image 9, test_jev_vmode 6, test_jev_golden 5, test_jev_metamorphic 3, test_jev_concurrency 2.
Serve suite — python3 -m unittest discover -s serve -p 'test_*.py': Ran 1002 tests in 328.664s, 1 failure, 14 skipped. The single failure is
test_lifeline.Lifeline.test_the_portable_watcher_ends_the_child_when_the_server_is_killed —
AssertionError: False is not true : the child outlived a killed server (watcher). On CI the same
suite can also fail on test_server_close_ends_the_sampler (AssertionError: 1 != 2) — it did on
the first run of this branch. Both are pre-existing defects in serve/, not in this PR: the first is
serve/lifeline.py building a child's identity from a ps state column that changes while the
process runs, the second a count-based assertion plus a sampler loop that dies on its first failed
sample. Both are fixed in [Engine] stabilise the serve test job: lifeline identity by start time, sampler that survives a bad source #3590, which is not part of 011c9e60 yet. With [Engine] stabilise the serve test job: lifeline identity by start time, sampler that survives a bad source #3590 in the base this
suite is green; the 14 skips are environmental by their own reason strings.
Live decision readout on the GPU — the engine started with --serve --jev-brow --gpu 1, one ordinary request to create a window, then the wire line:
```
DONE 1 8 154.2 18.2 stop 0 0 0 220 395 175 0 0.0 8 0 chunk=0 experts_streamed=0
BROWD 4 220:9.76547 13:5.95794 271:8.23788 198:11.6879
```
Real final-row logits for the four ids asked, from the CUDA build, on the 3090. A pre-rebase build of the same commits on the same card returned the same four values, so the rebase moved nothing in this path.
Live decision over HTTP on the GPU — the whole stack, not just the engine: python3 -m serve.server --engine strata --config … --port 18311 with STRATA_JEV_MODES=chat,decision and the engine started with --jev-brow --gpu 1 on the RTX 3090. POST /v1/systemone with one state and three questions of different types returned 200: noul with calib: platt_v1 and margin 0.7980, choice with calib: T_choice_v1 and margin 0.4725, score with calib: T_score_v1 and margin 0.3066, usage: {prompt_tokens: 299, readout_tokens: 3} — three typed, calibrated answers and three readout tokens. The inline jev extension on /v1/chat/completions answered the same way (T_choice_v1, margin 0.9861, one readout token) alongside a normal chat completion.

Failed:

One malformed request, on purpose, which is the error contract working: a score question written with scale instead of criteria returned 422 with {"error": {"field": "questions.urgency.criteria", "message": "must be an array of 2..10 levels"}} — a field-level refusal, not a 500 and not a silently wrong answer.
None observed beyond that in any suite that ran to completion on this revision. The 74 remaining ctest failures on the build host are environmental and named: 71 the missing-driver line above, platform_memory_test (mlock refused under the container's ulimit -l), ple_parity (the Q2_0 PLE fixture is not in this tree), ggml_tq_overlay (needs STRATA_GGML_DIR pointed at the agention llama.cpp fork for the trellis types).

Not run:

HIP, SYCL, Metal, Windows and macOS builds — no ROCm, SYCL runtime or Metal toolchain here. The engine change is guarded by the same STRATA_HAVE_CUDA_BUILD/transport seam the rest of the file uses, but I have not compiled it for those backends and do not claim they build.
The live golden battery with pinned probabilities on this revision. What ran is the invariant battery (type correctness, probability sums, bounds, determinism on repeat) against a mock engine; the pinned per-model numbers in §4 of docs/JEV-PR.md were measured before the rebase on the same model and the same code, and I am not presenting them as re-measured today.
The app's UI tests (npm test in apps/host) — this checkout has no node_modules and no build of the web UI.
A long-generation draft-acceptance measurement. The MTP draft layer loads (814 MiB resident, 106,299-token draft head), but the two-token generation used for the readout proposed no drafts (drafts accepted 0 of 0), so this revision says nothing either way about draft acceptance.

Limits

The BROW row is not bit-identical across cards. The same build, same prompt and same model on the RTX 4080 returned 220:9.72173 13:6.12896 271:8.52014 198:11.5419, about 0.4 % away from the 3090's row. Both runs are internally consistent and the difference is small enough that no label flipped, but I have not established whether it is card-dependent or run-to-run, so nothing in this PR should be read as a cross-GPU reproducibility guarantee.
calib/ is resolved relative to the server's working directory (STRATA_JEV_CALIB_DIR overrides it). A first run with no matching per-tuple file answers calib: identity — correct, but easy to mistake for a bug; the answers above are calibrated because calib-D/L/S.json sit beside the server.
The calibration files ship fitted on QFN IQ3_XXS. Another model needs its own fit (calib/README.md); a mismatched file is refused at load rather than silently applied.
Image decisions need the vision path and are covered by tests against a mock encoder; a live image decision was not part of this validation.
Windows and macOS keep the fail-closed paths for brow_row but were not executed here.
The calib/, reports/ and docs/bench-artifacts/ trees in this PR are the measurement artifacts behind §4 of docs/JEV-PR.md; they are data, not code, and are not regenerated by any build.

CI on this PR — what is failing, and why it is not this branch

Written for @leonardomurillo, who built and benchmarked this branch himself and will see the same checks. Two separate things are going on.

1. The three check suites are `action_required`, not failed

GitHub does not run workflows for a pull request opened from a fork until a maintainer of the base repository approves them. All three suites on this PR have sat at action_required since the first push, and zero check-runs have been created — nothing has run, so nothing has failed. That state is not something the author can clear.

The same workflow, dispatched inside the fork at the exact head commit, is green:

run	sha	result
`39966907103` — Minimal serving runtime, `justindressler/Strata`	`6b6618ff`	success, every step, full `python -m unittest discover -s serve` (1049 tests, 14 skipped)

2. Two tests in the shared suite are flaky, and both fail on `main` without this branch

test_lifeline.Lifeline.test_the_portable_watcher_ends_the_child_when_the_server_is_killed — serve/test_lifeline.py:98

AssertionError: False is not true : the child outlived a killed server (watcher)

The test starts a server, kills it, and asserts the watched child dies with it. On a loaded runner the child is sometimes not reaped inside the test's wait window. It failed 3 out of 3 times on the base commit 011c9e60, which predates every line of this PR, and it failed one of two runs at the same head sha where the other run passed — a flake by definition, not a regression.

test_responses.NoLeakedSampler.test_server_close_ends_the_sampler — serve/test_responses.py:284

AssertionError: 1 != 2

It counts live threads whose name ends in (_loop) and expects exactly one more than before the server started. On a loaded runner the sampler's loop thread is not always visible at the moment of the count. Seen once at a8b6d38b, absent at 6b6618ff.

Neither test imports or exercises serve/jev.py; both are pre-existing concurrency-timing assumptions in the suite.

What was actually run against this branch

check	result
`python -m unittest serve.test_systemone serve.test_jev_calib`	87 tests OK
`python tools/minimal_server_smoke.py`	`{"passed": true}`
fork CI at head (workflow above)	success
local full suite, clean checkout of base `011c9e60`	1048 tests, one failure — the lifeline watcher above, 3/3

leonardomurillo · 2026-10-22T15:10:28Z

I ran an A/B of this PR against a token-path scoring endpoint (proposed in #3743). Numbers below are ours; both directions shown.

What was tested:

This PR: commit ece6db0, built from source with the fork's GitHub Actions workflow (Windows, CUDA 13.0, archs 75;86;89;120; zip sha256 begins be133fd5), with STRATA_JEV_MODES=chat,decision.
Token paths: github.com/leonardomurillo/Strata v0.1.43-lp6 (544c32f), /v1/decision.
RTX 4080 16 GB, Windows 11. 10 interleaved pairs, one request at a time.
deepset/prompt-injections test split (116 items, 1 field); a banking77 subset (77 items, 2 fields: is_card boolean and group).

	This PR	Token paths
Injection latency	455 ms	576 ms
Injection accuracy	0.870	0.835 (McNemar p 0.39)
Injection AUC	0.961	0.984
Injection Brier	0.095	0.148
Banking latency (2 fields)	819 ms	658 ms
Banking `is_card` accuracy	0.708	0.856 (p 0.002)
Banking `group` accuracy	0.805	0.852 (p 0.34)

All paired 95 % CIs exclude 0. Run-to-run, 0-5 % of items flip.

Where this PR was better: one field is faster, with lower Brier. Where it was worse: injection AUC is lower, two fields are slower, and is_card accuracy is lower.

Engine log, injection: this PR reads a 98-token prompt in 400 ms, then 12 ms; ours reads 155 tokens in 457 ms, then 111 ms. On this card a short prompt costs about 300 ms plus ~1 ms per token, so part of the one-field gap is prompt length.

The PR is on 0.2.2, ours on 0.1.43. The official 0.2.2 engine (0.1.47) vs v0.1.43-lp6, same files and arguments, 10 pairs: same answers (McNemar p >= 0.5 on every field), JSON chat 7 ms faster on 0.2.2.

Not run: Linux, AMD, other GPUs.

justindressler · 2026-10-22T15:35:57Z

My implementation does image decision in latent space no text prompt and image summarisation. The performance isn't the best I admit. This was a adaption from another build I did based on llama.cpp with Qwen 27b. The 27b version performed alot better with typical decisions completed in 60ms. That being said it was a full weights load on a quant 6 bit model. The comparison from the Qwen 27b on llama and QFN 126b is academic. My tests on the strata version has higher accuracy and better confidence even on a MoE. It is most definitely not the fastest system 1 solution. But it offers calibrated confidence and image descion and then operation to forward to reasoning prompt on descion. These features are unique to this implementation. Your welcome to find optimisations I have committed almost 30 hours to this PR running dozens of benchmarks with A/B through image->descion, text->decision, text->decision->prompt paths. Including tests through standard text generation paths to make sure there is no regressions. QFN isn't the ideal model for a real-time jev/system 1 system. But if you need precision and accuracy it is a good compromise. Sent from Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: Leonardo Murillo ***@***.***> Sent: Thursday, 22 October 2026 23:10:51 To: Niko1221/Strata ***@***.***> Cc: justindressler ***@***.***>; Author ***@***.***> Subject: Re: [Niko1221/Strata] [Engine] Jev System One decision endpoint (/v1/systemone): typed decisions with calibrated probabilities (PR #3643) [https://avatars.githubusercontent.com/u/4869242?s=20&v=4]leonardomurillo left a comment (Niko1221/Strata#3643)<#3643 (comment)> I ran an A/B of this PR against a token-path scoring endpoint (proposed in #3743<#3743>). Numbers below are ours; both directions shown. What was tested: * This PR: commit ece6db0<ece6db0>, built from source with the fork's GitHub Actions workflow (Windows, CUDA 13.0, archs 75;86;89;120; zip sha256 begins be133fd5), with STRATA_JEV_MODES=chat,decision. * Token paths: github.com/leonardomurillo/Strata v0.1.43-lp6 (54dfb45<54dfb45>), /v1/decision. * RTX 4080 16 GB, Windows 11. 10 interleaved pairs, one request at a time. * deepset/prompt-injections test split (116 items, 1 field); a banking77 subset (77 items, 2 fields: is_card boolean and group). This PR Token paths Injection latency 455 ms 576 ms Injection accuracy 0.870 0.835 (McNemar p 0.39) Injection AUC 0.961 0.984 Injection Brier 0.095 0.148 Banking latency (2 fields) 819 ms 658 ms Banking is_card accuracy 0.708 0.856 (p 0.002) Banking group accuracy 0.805 0.852 (p 0.34) All paired 95 % CIs exclude 0. Run-to-run, 0-5 % of items flip. Where this PR was better: one field is faster, with lower Brier. Where it was worse: injection AUC is lower, two fields are slower, and is_card accuracy is lower. Engine log, injection: this PR reads a 98-token prompt in 400 ms, then 12 ms; ours reads 155 tokens in 457 ms, then 111 ms. On this card a short prompt costs about 300 ms plus ~1 ms per token, so part of the one-field gap is prompt length. The PR is on 0.2.2, ours on 0.1.43. The official 0.2.2 engine (0.1.47) vs v0.1.43-lp6, same files and arguments, 10 pairs: same answers (McNemar p >= 0.5 on every field), JSON chat 7 ms faster on 0.2.2. Not run: Linux, AMD, other GPUs. — Reply to this email directly, view it on GitHub<#3643?email_source=notifications&email_token=AETR4LJ7KSEKFCHLUMBPW5D5VIPPXA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTMMZUGQYTIMRRGEYKM4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#issuecomment-6344142110>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AETR4LLL4RKPR4IYYZ6W37L5VIPPXAVCNFSNUABGKJSXA33TNF2G64TZHMYTGOBVHE2DMMZQGE5US43TOVSTWNJZGY3TKNZRGMZTNILWAI>. Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS<https://github.com/notifications/mobile/ios/AETR4LN77QOT72OOOUQBCQD5VIPPXA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTMMZUGQYTIMRRGEYKM4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KUZTPN52GK4S7NFXXG> and Android<https://github.com/notifications/mobile/android/AETR4LOZEC557ZECWMYN6TL5VIPPXA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTMMZUGQYTIMRRGEYKM4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2K4ZTPN52GK4S7MFXGI4TPNFSA>. Download it today! You are receiving this because you authored the thread.Message ID: ***@***.***>

justindressler · 2026-10-22T15:44:12Z

Multiple choices were tested in single passes including noul, choice and score. It should have completed in a single forward pass. But maybe the latency is part of the MoE expert cache. I can review the pass again in theory the same experts will run on the same context but it is still likely to have penalties. Sent from Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: Justin Dressler ***@***.***> Sent: Thursday, 22 October 2026 23:35:46 To: Niko1221/Strata ***@***.***>; Niko1221/Strata ***@***.***> Cc: Author ***@***.***> Subject: Re: [Niko1221/Strata] [Engine] Jev System One decision endpoint (/v1/systemone): typed decisions with calibrated probabilities (PR #3643) My implementation does image decision in latent space no text prompt and image summarisation. The performance isn't the best I admit. This was a adaption from another build I did based on llama.cpp with Qwen 27b. The 27b version performed alot better with typical decisions completed in 60ms. That being said it was a full weights load on a quant 6 bit model. The comparison from the Qwen 27b on llama and QFN 126b is academic. My tests on the strata version has higher accuracy and better confidence even on a MoE. It is most definitely not the fastest system 1 solution. But it offers calibrated confidence and image descion and then operation to forward to reasoning prompt on descion. These features are unique to this implementation. Your welcome to find optimisations I have committed almost 30 hours to this PR running dozens of benchmarks with A/B through image->descion, text->decision, text->decision->prompt paths. Including tests through standard text generation paths to make sure there is no regressions. QFN isn't the ideal model for a real-time jev/system 1 system. But if you need precision and accuracy it is a good compromise. Sent from Outlook for Android<https://aka.ms/AAb9ysg>

________________________________ From: Leonardo Murillo ***@***.***> Sent: Thursday, 22 October 2026 23:10:51 To: Niko1221/Strata ***@***.***> Cc: justindressler ***@***.***>; Author ***@***.***> Subject: Re: [Niko1221/Strata] [Engine] Jev System One decision endpoint (/v1/systemone): typed decisions with calibrated probabilities (PR #3643) [https://avatars.githubusercontent.com/u/4869242?s=20&v=4]leonardomurillo left a comment (Niko1221/Strata#3643)<#3643 (comment)> I ran an A/B of this PR against a token-path scoring endpoint (proposed in #3743<#3743>). Numbers below are ours; both directions shown. What was tested: * This PR: commit ece6db0<ece6db0>, built from source with the fork's GitHub Actions workflow (Windows, CUDA 13.0, archs 75;86;89;120; zip sha256 begins be133fd5), with STRATA_JEV_MODES=chat,decision. * Token paths: github.com/leonardomurillo/Strata v0.1.43-lp6 (54dfb45<54dfb45>), /v1/decision. * RTX 4080 16 GB, Windows 11. 10 interleaved pairs, one request at a time. * deepset/prompt-injections test split (116 items, 1 field); a banking77 subset (77 items, 2 fields: is_card boolean and group). This PR Token paths Injection latency 455 ms 576 ms Injection accuracy 0.870 0.835 (McNemar p 0.39) Injection AUC 0.961 0.984 Injection Brier 0.095 0.148 Banking latency (2 fields) 819 ms 658 ms Banking is_card accuracy 0.708 0.856 (p 0.002) Banking group accuracy 0.805 0.852 (p 0.34) All paired 95 % CIs exclude 0. Run-to-run, 0-5 % of items flip. Where this PR was better: one field is faster, with lower Brier. Where it was worse: injection AUC is lower, two fields are slower, and is_card accuracy is lower. Engine log, injection: this PR reads a 98-token prompt in 400 ms, then 12 ms; ours reads 155 tokens in 457 ms, then 111 ms. On this card a short prompt costs about 300 ms plus ~1 ms per token, so part of the one-field gap is prompt length. The PR is on 0.2.2, ours on 0.1.43. The official 0.2.2 engine (0.1.47) vs v0.1.43-lp6, same files and arguments, 10 pairs: same answers (McNemar p >= 0.5 on every field), JSON chat 7 ms faster on 0.2.2. Not run: Linux, AMD, other GPUs. — Reply to this email directly, view it on GitHub<#3643?email_source=notifications&email_token=AETR4LJ7KSEKFCHLUMBPW5D5VIPPXA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTMMZUGQYTIMRRGEYKM4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#issuecomment-6344142110>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AETR4LLL4RKPR4IYYZ6W37L5VIPPXAVCNFSNUABGKJSXA33TNF2G64TZHMYTGOBVHE2DMMZQGE5US43TOVSTWNJZGY3TKNZRGMZTNILWAI>. Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS<https://github.com/notifications/mobile/ios/AETR4LN77QOT72OOOUQBCQD5VIPPXA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTMMZUGQYTIMRRGEYKM4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KUZTPN52GK4S7NFXXG> and Android<https://github.com/notifications/mobile/android/AETR4LOZEC557ZECWMYN6TL5VIPPXA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTMMZUGQYTIMRRGEYKM4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2K4ZTPN52GK4S7MFXGI4TPNFSA>. Download it today! You are receiving this because you authored the thread.Message ID: ***@***.***>

justindressler · 2026-10-22T16:22:04Z

Correction to my reply above: the two-field cost is one cold prefill per field, not the MoE expert cache — serve/jev.py:484 loops the questions, :535 re-renders the prompt, :545 runs generate() per question.

Your 98-token / 400 ms one-field log predicts 824 ms for two fields against your measured 819 ms. Details, the fix (per-position row readout) and my answer to the scope question are in #3743.

justindressler · 2026-10-23T01:06:11Z

Thanks — that's a clear brief, and the "same footing" point on Brier is the right catch: our 0.095 went through the Platt fit while yours was raw, so that row was measuring calibration, not the readout. The next comparison will be both raw, or the same calibrator on both.

Multi-field, since that's the part you flagged (and you were right that it was one cold prefill per field — 819 vs 658 ms was the honest number for that commit). There is now an opt-in path (STRATA_JEV_ROWS=1) that renders ONE prompt for the whole request and keeps the head row at each field's cue position, so N fields cost one prefill instead of N. Field 1's prompt and position are byte-identical to the per-field path.

fields	per-field	one-prompt	ratio
1	918 ms	841 ms	1.09x
2	1232 ms	981 ms	1.33x
3	1185 ms	1258 ms	0.94x
4	1406 ms	1382 ms	1.02x
5	1667 ms	1582 ms	1.05x

3090, Qwen3.8-Flash-Next IQ3_XXS, banking77 is_card+group, same states, one request at a time, ±15% run to run. usage.prompt_tokens at two fields: 193 on the per-field path, 145 on the one-prompt path.

Accuracy on the same 200 items, each path with its own fitted calibrator: is_card 0.800 vs 0.805, group 0.740 vs 0.750, both-right 0.705 vs 0.715 (paired McNemar p=1.000 / 0.688 / 0.625). No accuracy cost at this n — I'm not claiming equivalence.

Your field-order point caught something real. I ran it on both paths (same 200 items, fields forward vs reversed, alternating which order goes first so a cold prefill can't be mistaken for an order effect):

path	`is_card` agreement	flips	mean abs dp	`group` agreement
per-field	199/200 (0.995)	1	0.009	199/200 (0.995)
one-prompt	167/200 (0.835)	33 (32 yes->no, 1 no->yes)	0.152	192/200 (0.960)

So the one-prompt path IS order-sensitive at two fields: 16.5% of is_card answers move when the fields swap, and they move in one direction (the second field answers "no" more often). Accuracy happens to be slightly better in the reversed order (0.795 vs 0.760, p=0.30) — exactly the sort of accidental win that a single-order benchmark would hide. That settles it for me: the consolidated path stays opt-in and needs the field order pinned in the request rather than trusting the caller's dict.

On is_card 0.708 vs 0.856 (p=0.002): part of that is calibration, and we can now put a number on it. Fitting the noul Platt on a labelled dev split (600 banking77 items the test 200 never saw) moved held-out is_card from 0.675 to 0.805 (McNemar p=0.0034) — the Platt offset sets the yes/no threshold, and the shipped fit was trained on BoolQ/ARC/SST, which is the wrong domain for banking77. The other path gains the same ~12 points from its own fit, so it isn't a property of either readout. I'll send the fit and the dev split so both can be compared on the same footing.

What calibration does NOT explain is injection AUC 0.961 vs 0.984. That's threshold-free, so it's a genuine ranking difference and I won't argue it away. We'll keep looking at it — if you have ideas where you'd look first, I'd take them.

Labels: adopted (0-9 to ten options, a-z to 26, >26 refused), and a request whose label is not a single token is now refused rather than scored on a truncated path. One caveat — the calibrators on disk were fitted against the A-Z render, so the scheme sits behind a flag until we re-baseline the readouts. I'll report both.

Still in flight, not in the numbers above: GQA as a second labelled noul set, and MMLU as a choice set. On MMLU I'll label the caveat explicitly: a single-token letter readout is zero-shot, so it is not comparable to 5-shot/CoT MMLU and shouldn't be read as a capability claim. When those land I'll post them per field — accuracy, Brier, latency and usage.prompt_tokens — on the two public sets, one request at a time.

justindressler · 2026-10-23T02:54:26Z

Follow-up on calibration, since the harness spec asks for Brier on the same footing. All numbers below are one box (RTX 3090), one binary, identical items, default arm (per-question prompts + state parking), and are n≈100–300 per cell — treat them as directional, not as accuracy claims.

1. A calibrator fitted on one task does not transfer to another. Platt fit on banking77 (600 items, held-out 200), applied to two other tasks:

fit → eval	acc	Brier	ECE15
banking77 raw	.670	.2208	.259
banking77 ← own fit	.800	.1385	.099
POPE raw	.883	.0785	.074
POPE ← banking77 fit	.850	.1114	.145
GQA raw	.803	.1470	.054
GQA ← banking77 fit	.720	.2173	.243

Same fit that gains +13 pts in-domain actively damages both out-of-domain tasks.

2. The damage is the offset, not the slope. Keeping the fitted slope and forcing c = 0:

POPE: acc .883, Brier .0794, ECE .0882 (was .883 / .0785 / .074 raw)
GQA: acc .803, Brier .1451, ECE .0790 (was .803 / .1470 / .054 raw)

Accuracy and Brier come back; only ECE stays slightly worse. So a behaves like a legitimate global recalibration, while c is the decision threshold — on banking77 it sits at ≈2.7 logits because that set is a balanced 50/50 prior. Transferring c across tasks is really transferring a prior, and it moves decisions as well as confidence.

3. Not every task needs calibration. GQA (600 image yes/no items) is already honest: raw ECE .054, and a dev-fit on it made things worse (acc .803→.780, ECE .054→.083). The in-domain gain is largest exactly where the raw readout is most overconfident (banking77 raw ECE .259).

4. Per-tuple calibrators are at their ceiling on the image tuples. Live, served probabilities, paired against the same items served uncalibrated, 100 test items: zero argmax flips (correct behaviour — confidence only), Brier −.0046 POPE / −.0315 CV-Bench / −.0209 CountBench, ECE −.0075 / +.0100 / −.0583. Refitting on top of the calibrated output asks for ≈identity (a=1.066, c=−0.003; T=0.985, T=0.984), i.e. no residual calibration error left to recover.

Suggestions for the harness, all cheap:

Report Brier/ECE on the served probabilities, and record which calibrator id was served in the response — otherwise a harness that fits its own dev split on already-calibrated probabilities compounds the transform (that trap bit us).
Fit per (tuple, task, mode); direct and summary image modes are separate tuples with very different fits (T 1.55 vs 5.03).
Never carry c across tasks. If you want one number to compare implementations, compare the slope.
A fitted temperature within ~2% of 1.0 means "already calibrated" — don't report it as an improvement.

Happy to share the per-item logs (qid, gold, probability vector, served calibrator) if that is useful for your harness.

…refusal; field-order test leonardomurillo (Niko1221#3643): labels are 0-9 up to ten options, a-z for 11-26, more than 26 refused, and a request whose label is not a SINGLE TOKEN must be refused rather than scored on a truncated path. serve/jev.py: _label_scheme()/choice_labels() - STRATA_JEV_LABELS=upper (default) | digit-lower. Unknown value fails closed at import, like STRATA_JEV_MODES. question_block now renders qdef[render_labels] instead of assuming LETTERS, so both the option list and the readout use the same alphabet. _label_token_problem() - refuses a multi-token label with 422 questions.<id>.criteria in both handle() and handle_image(). Default stays upper: every calibrator on disk was fitted against A-Z, so the scheme is opt-in until the choice readouts are re-baselined (AGENTS.md: default path byte-identical). 6 new tests (41 in serve.test_systemone): default unchanged, digit-lower ordering, 2..26 range, block render, multi-token refusal, noul/score unaffected. scripts/order_test.py - his field-order ask. Same items twice (forward, reversed), alternating which order goes first so a cold first prefill is not mistaken for an order effect, and latency reported by request position as well as by order. Result at two fields: per-field path 199/200 agreement (1 flip, mean |dp| 0.009); one-prompt path 167/200 (33 flips, 32 yes->no, |dp| 0.152) with the reversed order ACCIDENTALLY 3.5 points more accurate - so the consolidated path needs the field order pinned.

…OWAT parse, correct the calibration and label docs - src: bring row capture (set_row_capture / capture_window_rows / brow_row_captured), jevpos= parsing and the INFO jevpos advertisement into git - the shipped binary had them, the committed source did not, so the 16-commit port would have shipped a server talking to an engine with no capture - generate.cpp: BROWAT token ids range-checked before the int32 cast and a readable parse failure (trailing junk / >4096 ids), BROWROWS matched exactly instead of as a prefix - server.py: a malformed BROWROWSD answer now ends the engine like every other fail-closed read - serve/jev.py: one _row_capture() gate for both row paths, documenting the two BROWAT addressings - docs: JEV-SYSTEMONE multi-field/label sections, calibration corrected from 'argmax-preserving' to 'the Platt offset is a threshold', JEV-PR 4a-bis (re-measured, per-task not portable, arm table), test counts, calib/README status, scripts/README

… prompt (10 failing)

serve/jev.py: frozen D3 prompt, donor 422 contract, jmath readout (M1 identity), engine-turn discipline (svc.fifo), final_row_logits readout boundary (BROW on the engine side). server.py: route + import. 10/10 tests; full-suite parity vs baseline verified (0 new failures; 14 pre-existing regex-module errors, flaky sampler tests vary run to run on both trees).

…il-closed)

…fail-closed) + jev.handle wired to real contract

…act (CUDA test target)

…, CUDA+Metal), --jev-brow flag, serve-loop BROW handler; golden battery fixture

…minism; CMake CPU guard fixed (15/15)

…allowed (review r1)

…ly order, /admin/calib hot-swap, B1.1 log rows - serve/jev_calib.py: donor src/server.cpp:518-633 surface - load (calib.json/v1.json, corrupt=identity), swap, apply (platt_v1 noul / T_choice_v1 / T_score_v1, argmax-safe, renormalized), tuple_of, async 4096-row log queue with 200ms flusher - serve/jev.py: donor order (server.cpp:1134-1142) - softmax -> round 4dp -> renorm -> calibrate -> answers_from_calibrated; calib id surfaced per answer; B1.1 rows (ts/tuple/qid/type/raw_p unrounded; label_mass omitted - BROW is sparse, fit.py never reads it) - serve/server.py: POST /admin/calib (Bearer STRATA_JEV_ADMIN_KEY, empty=open like llama-swap's control plane); startup swap() + daemon flusher + atexit drain - tests: test_jev_calib (14) math+wire+hot-reload; test_systemone re-pinned to the donor 4dp contract. 29/29 green (mock, no GPU); full-suite regression running

…he live M3.2 bench 1. question_block now the donor render_qblock verbatim (server.cpp:305-334): full Options:/Levels: list with criteria descriptions, 'Question:'/'Answer:' anchors, noul Yes/No-means lines. The M3.0 render listed NO options -> the model picked labels blind (ARC 23% = chance). 2. enable_thinking=False on the decision render (D3, frozen): the readout position must predict the label, not open <think>. 3. render_state: string states pass through VERBATIM (server.cpp:266-268); objects stay pretty JSON. The port JSON-quoted every string state. Live QFN before -> after: choice 23%->100%, noul 49%->88%, score 24%->55% (donor 27B baseline: 99/90.5/56.9). 29/29 unit tests green.

…on yes/no branches, softmax(d) - question_head + v_branch: donor render_qhead/render_vbranch verbatim (server.cpp:337-353): shared 'Question:/Options:' head, then one 'Proposed answer: <name> ... Answer:' branch per option - d_i = s_yes - s_no per branch (BROW readout), p = softmax(d), 4dp + renormalize, choice temperature applies (donor :1150-1163, calib_apply) - response: probabilities keyed by option NAME, 'choice' = name string (donor A2.6 shape); usage.readout_tokens counts one BROW per branch - X-Jev-Mode header passthrough in server.py; V scopes to choice only, noul/score fall through to the L path (donor :767) - log rows: tuple ...|V|choice with v_d (donor B1.1 V schema) - K sequential branch decodes share the state+head prefix - Strata's engine-side KV prefix mounting reuses it (M3.0 alignment, no EngineSeq) - tests: test_jev_vmode (6) - name-keyed probs, branch prompt shape, shared-head prefix, per-option readouts, v_d log rows, L unchanged. 35/35 green with calib/systemone/golden.

…n, one wire request) - POST /v1/systemone with image_b64 (+ optional vision_prompt, default 'Describe this game screen:') -> jev.handle_image - stage 1: Strata's existing per-preset vision path (svc.prepare encodes the image and expands the image pads), one bounded summary generation (<=160 tok, thinking off) - stage 2: the summary becomes the decision state; the standard L/V readout answers. Response: answers + image_summary + stages_used ['vision','decision'] (donor PLAN B V1 contract, server.cpp:2419-2603; the donor's V2 single-seq fork crashed ggml and stays NOT ported) - fail-closed: empty summary -> 502 and the decision stage never runs; no vision encoder -> 400; short b64 -> 422 - tests: test_jev_image (6) - two stages one request, summary rides into the decision state, vision_prompt accepted, fail-closed paths, validation first. 41/41 green with vmode/calib/systemone/golden.

…e jev extension M3.9 (donor amendment 7 mode 1): "then": {"generate": {...}} on /v1/systemone - decide first, then generate with the answers folded into the LAST USER turn as 'Jev decisions (JSON): ...' (a trailing system message throws in the chat template; donor folds too). Sampling mirrors the chat endpoint (temp 0.6, top_p 0.95, top_k 20, max_tokens 512 defaults; stop strings honored). Response = decision body + then{text, finish_reason, usage}. M3.3 (donor A3.4): "jev": {state|messages, questions} on /v1/chat/completions - jev.questions required; state XOR messages (400 on both/neither, donor messages-must-be-empty rule); stream refused with the jev extension; the conversation becomes the decision state (v1-chat, donor 3.5). Response wraps the decision body under "jev" with an OpenAI-style envelope. Tests: test_jev_then (9) - two stages one request, answers in the gen prompt, sampling defaults, error propagation, chat-surface validation arms, plain chat unaffected. 50/50 green with image/vmode/calib/systemone/golden.

…ns, no cross-talk, FIFO serializes engine turns The donor's DoD arm on the serve port: 7 threads, 3 noul questions each - every client gets exactly its own 3 answers (qid-keyed, no foreign qids), 21 readout turns, and an in-flight counter on engine.generate proves the FIFO never lets two engine turns overlap (the serve-side stand-in for the donor's per-decode-step ENG.mu, which M3.4 maps to svc.fifo - already held across each request's whole readout loop). 52/52 green.

…ns, calibration, examples Wire surface (request/response/422 contract), the four extensions (V mode, image two-call, decide-then-generate, chat-surface jev), calibration math + hot-swap + the measured QFN tables (raw vs calibrated, ms/item, donor-parity numbers), concurrency arm, and curl examples for every path.

…enizer Live QFN run caught two: the engine streams negative sentinel ids (verification artifacts) which the byte tokenizer's token_bytes rejects, and per-token decode re-decodes for stops. Now uses the server's own incremental Detokenizer (split-multibyte safe), skips None/negative ids, and checks stops against the last few decoded pieces. Live-verified on QFN through llama-swap after the fix: - then.generate: decision + 'Jev selects the second option ... 75.93%' (the answers folded into the prompt, 33-token completion, finish stop) - chat-surface jev: {object: jev.response, jev: {answers... noul 0.9995}}

…cide Justin's ask: can the image embeddings themselves drive the readout, without the summary in between? Yes on Strata - the donor's V2 needed a seq fork off image embeds (crashed ggml on hybrid attention); our readout is BROW on the final decoded row, so one prompt [image + question_block], one decode step, label logits off that row. No new engine primitive. - image_mode: 'summary' (default, two-call) | 'direct' (single readout); invalid values 422 on image_mode - the question text sits AFTER the image pad so the readout position attends the full image; svc.prepare encodes + expands pads (Strata's normal path) - no outer fifo hold: prepare takes its own engine turn (deadlock found by the new test - same lock, held twice) - response: answers + usage + stages_used ['image-direct']; calib log rows under a ...|D|<qtype> tuple (D = direct image readout) for a separate fit Tests: direct-mode shape (one turn, question after image), bad mode 422. 54/54 green.

…mage was black without them The live vision A/B caught it: chat-path control answered 'Red' (reasoning described the white box and the RED text), while both image arms saw a 'completely black' image. prepare() stages the combined embeddings file on svc.embeddings.path, but it is the CALLER that must pass embeddings=... to engine.generate - that is what turns GEN into GENI and makes the engine load the image. The decision handlers called generate() bare, so the pads attended nothing. Both handlers now take svc.embeddings.path (run()'s exact pattern), clear it, and delete the file when done, as run() does. Also the summary loop's second run-in with negative sentinel ids (fixed the same way as handle_then: Detokenizer + skip None/<0). Also why the first bench arm died at 400: same crash, earlier code. 52/52 green after; live A/B rerun in flight.

C on mf-600-train: is_card acc 0.673 / group 0.772 / mean_conf 0.898. Its own fit (a_noul=0.485557 c_noul=-1.638447 T_choice=2.430149) on held-out 200: is_card 0.670 -> 0.800, Brier 0.2208 -> 0.1385, group ECE 0.2078 -> 0.0889 (accuracy unchanged). ab_arms_mf.py runs both arms over the same items, each with its own calibrator, and reports the paired discordant counts + exact McNemar - the accuracy question is a paired question and an unpaired comparison throws that power away.

B and C both: BoolQ 91.0%, ARC 100%, SST-5 52/53%. Paired McNemar on identical test items: 0 discordant on BoolQ and ARC, 1 on SST-5 (p=1.000) - the consolidated-prompt arm costs no accuracy on the standard suites. The per-suite refit is ~identity (BoolQ a=1.01 c=0.00) and makes SST-5 ECE slightly WORSE, because calib-L.json was fitted on this very dev split: these suites are calibration-saturated, so the banking77 +13-point gain was domain transfer, not a broken calibrator.

…dary flags; un-compound the suite fit_temp scanned t=0.05*1.1**k and RETURNED a grid point, so every reported T was quantised to 10% steps - 1.0557 (= 0.05*1.1**32) appearing for two unrelated suites is what exposed it. On a flat likelihood it also returned a spurious non-1 T, which then made ECE worse. fit_temp_ex: coarse scan only brackets, golden-section refines in log t (~1e-4 relative); a fit must beat identity by 0.5% relative NLL or it is reported as T=1 (flat); an optimum pinned at either edge is flagged boundary so the tuple can be rejected instead of quoting the edge. fit_platt_ex gets the same identity preference (BoolQ's a=1.009/c=0.0037 was noise, now identity). The suite also warned-but-proceeded when the served probabilities were already calibrated: the refit then COMPOUNDS the installed transform (T applied twice) and the params are not installable. Now it warns explicitly, and jevport-arm.sh has --calib identity to remove the file for a clean raw->fit run. Ground truth: test_fit_temp.py recovers T in 0.6..4.0 within 5% at n=4000 (0.2-3.7%), converges with n, never returns a grid point, keeps identity flat, flags the degenerate boundary, and recovers Platt slope/offset. ALL CHECKS PASS. Identity-served suite, B vs C: BoolQ 0.890/0.900 -> 0.910 both; ARC 1.000 both; SST-5 0.500/0.520. Fitted BoolQ Platt a=0.62 c=0.67 and SST T=2.34 recover the stock Oct-15 params, i.e. the fitter now reproduces the shipped calibrator from 60 dev items. SST-5 calibration does real work once un-compounded: ECE 0.31->0.17, NLL 1.45->1.04.

…-served, arm C) suite_real gets the same treatment as suite_text: new fitters, served-calibrator warning, env-overridable data root, flags in the report. 200 items end to end, fits installable. POPE noul 60 test: acc 0.883, ECE identity 0.0993 / shipped 0.0691 / refit 0.1107 CV-Bench choice 20: acc 0.650, ECE 0.1848 / 0.0945 / 0.1928 CountBench score 20: acc 1.000, ECE 0.0992 / 0.0408 / 0.0382 Accuracy identical under every map (noul threshold barely moves; choice/score are argmax-preserving). The refit does not beat the shipped D fit here - that fit was made on this same dev split and the test splits are 60/20/20, so ECE differences of that size are a few items. transfer_check.py re-scores a log under identity / shipped / refit so 'does this calibrator still help' is answerable without another GPU run. It also caught that score and choice use DIFFERENT deployed temperatures (T_choice vs T_score) - using T_choice for score rows measures the wrong map.

…guard the class calib-S.json shipped "a"/"c" while jev_calib.load() reads a_noul/c_noul, so the summary-mode Platt was silently identity - while T_choice=0.05 was honoured and apply() only rejects <=0.01, giving exponent 20 (near one-hot confidence). Both are now fixed from a real summary-mode run (identity-served, 200 items): a_noul=0.435457 c_noul=0.818939 T_choice=5.027682. T_score is deliberately left at 1.0: the dev fit (T=1.846) improved dev NLL 28% but the TEST split got worse (NLL 0.3695->0.4067, ECE 0.081->0.156, n=20+20), which fails the documented acceptance rule (NLL_post <= NLL_raw) - so that leg stays uncalibrated until a bigger dev split exists. Guards, so this class cannot recur silently: scripts/check_calib.py - loads each file through jev_calib.load() and fails on keys the loader does not read, declared-vs-effective mismatches, degenerate T, tuple letter mismatch STRATA_JEV_CALIB_STRICT=1 - opt-in (default path unchanged): jev_calib.load() REFUSES such a file and logs why, instead of half-applying it 5 new tests in serve/test_jev_calib.py cover default behaviour, strict refusal (mis-keyed, degenerate T), a clean file still loading, and the tuple refusal scripts/fit_tuple_from_log.py turns a suite item log into an installable calib-<letter>.json and refuses to write anything check_calib would reject.

…refusal; field-order test leonardomurillo (Niko1221#3643): labels are 0-9 up to ten options, a-z for 11-26, more than 26 refused, and a request whose label is not a SINGLE TOKEN must be refused rather than scored on a truncated path. serve/jev.py: _label_scheme()/choice_labels() - STRATA_JEV_LABELS=upper (default) | digit-lower. Unknown value fails closed at import, like STRATA_JEV_MODES. question_block now renders qdef[render_labels] instead of assuming LETTERS, so both the option list and the readout use the same alphabet. _label_token_problem() - refuses a multi-token label with 422 questions.<id>.criteria in both handle() and handle_image(). Default stays upper: every calibrator on disk was fitted against A-Z, so the scheme is opt-in until the choice readouts are re-baselined (AGENTS.md: default path byte-identical). 6 new tests (41 in serve.test_systemone): default unchanged, digit-lower ordering, 2..26 range, block render, multi-token refusal, noul/score unaffected. scripts/order_test.py - his field-order ask. Same items twice (forward, reversed), alternating which order goes first so a cold first prefill is not mistaken for an order effect, and latency reported by request position as well as by order. Result at two fields: per-field path 199/200 agreement (1 flip, mean |dp| 0.009); one-prompt path 167/200 (33 flips, 32 yes->no, |dp| 0.152) with the reversed order ACCIDENTALLY 3.5 points more accurate - so the consolidated path needs the field order pinned.

…sume-safe row persistence

… decision flips)

… tweak

…m script

…OWAT parse, correct the calibration and label docs - src: bring row capture (set_row_capture / capture_window_rows / brow_row_captured), jevpos= parsing and the INFO jevpos advertisement into git - the shipped binary had them, the committed source did not, so the 16-commit port would have shipped a server talking to an engine with no capture - generate.cpp: BROWAT token ids range-checked before the int32 cast and a readable parse failure (trailing junk / >4096 ids), BROWROWS matched exactly instead of as a prefix - server.py: a malformed BROWROWSD answer now ends the engine like every other fail-closed read - serve/jev.py: one _row_capture() gate for both row paths, documenting the two BROWAT addressings - docs: JEV-SYSTEMONE multi-field/label sections, calibration corrected from 'argmax-preserving' to 'the Platt offset is a threshold', JEV-PR 4a-bis (re-measured, per-task not portable, arm table), test counts, calib/README status, scripts/README

Independent reviewer (fail-closed) rejected the diff on four logic errors; all are addressed: - absolute-vs-window row addressing: the engine half (row capture) is now committed, and BROWAT's silent fall-back to window addressing is closed by BROWCAP: the server verifies it kept exactly the rows it named BEFORE reading them, else one prompt per field - BROWROWS was reachable by nothing and could never pass: documented as a window-addressing diagnostic, with captured_rows() (BROWCAP) as the real guard - BROWAT narrowing: the row is bounded to int32 like the token ids (4294967301 used to answer row 5) - jevpos longer than 32 was dropped silently -> a >32-field request now never takes the row path; token ids are capped before the push, so a huge line cannot allocate first - strict calib mode bounds a_noul/c_noul with isfinite (JSON accepts NaN, which made every answer NaN) - the direct-image path now shares _finish_answer('D'), so the 'one place' claim is true - label refusal is documented as the one default-path behaviour change, and tested at HTTP level - tests: capture-dropped fallback (B path + hybrid), 33-field cap, multi-token label 422, seam move

…nly real risk)

…le-token refusal, no silent jevpos truncation MiniMax-M3 (mm-builder) returned reject with one blocker and three majors; adopted: - BLOCKER jev_calib._strict_problem: the temperature guard used only <=/>=, which NaN passes (every comparison against NaN is False), so a calib file with T_choice: NaN loaded under STRICT and put NaN probabilities, NaN margin and a NaN score into the HTTP body. isfinite now comes first, like the Platt guard above it, with a test that loads exactly such a file. - MAJOR the single-token label refusal skipped score questions, which render '0'..'9' and read them the same truncating way (encode, then [:1]): now both types are covered, with an HTTP-level test. - MINOR jevpos longer than 32 was silently truncated engine-side ('never a guessed position' said the comment); the list is now REFUSED whole, so the server sees BROWCAP 0 and falls back. - MINOR STRATA_JEVROWS_TRACE and BROWCAP documented in usage(). - MAJOR 'count is not a set check' refuted in the code: capture only ever takes named positions, so captured subset-of named + equal size implies equal sets; the comment says what would break it.

…in-on-default declared GLM-5.3 (zai-builder) reached the same blocker and the same score-label major as MiniMax, plus three new findings; adopted: - MAJOR the row capture was cleared only at the read loop, so a request that failed earlier (oversize prompt, tokenization, cancel) left the previous request's rows readable by a later BROWAT - stale rows for a prompt that is no longer the last one, and captured_rows()>0 cannot tell stale from fresh. The capture is now cleared at the top of EVERY request. - MINOR the jevpos parser accepted '12x' as 12 (atoll stops at the junk); every character must be a digit. - MINOR the hybrid comments claimed the decode step was skipped - it is not (max_new stays 1, the row is read back by position); comments and docstring corrected. - MINOR MAX_NAMED_ROWS comment described a DROP; the engine now refuses the whole list. - MAJOR 'pinning is default-on but not declared' - true and deliberate: declared in JEV-SYSTEMONE next to the label refusal, with what it costs (answer identity rests on checkpoint-resume exactness, measured <=.008 on field 1, 199/200 order agreement) and how to turn it off (STRATA_JEV_PIN=0). - tests: BROWCAP wire exchange (roundtrip, zero, unexpected op, malformed) now covered; dead un-shadow line removed.

The rebased base's STRATA_ADMIT_MERGE_FIRST path moves the last tokens into a different first window. A request that named head rows (jevpos=) needs every named position to run in a window the capture sees, so that path now declines instead of silently leaving positions uncaptured.

…ld not be reproduced

The `jevpos=` field parser was a braceless for whose `else if`/`else` bound to the inner digit test, so the push ran once per DIGIT character and the 32-entry cap counted digits. Measured on the exact block, compiled: `63,128` parsed to [63,63,128,128,128], and a legal 32-position list was refused WHOLE - so the consolidated path silently fell back to one prompt per field above ~11 fields on a real prompt (3-digit positions), with only a log line as evidence. Found independently by two reviewers in round 2 (2026-10-23); it had no test on either side of the wire, which is how it survived three reviews. The parse moves to include/strata/program/jevpos.hpp (header-only, no model, no GPU) so the string -> vector rule is testable: tests/program/jevpos_test.cpp pins multi-digit positions, 32 entries (including 32 three-digit ones), the 33rd entry refused whole, and the malformed cases (`12x`, `4;9`, `-4`, " 4", a 19-digit field) plus the edges the engine always allowed (empty list, `4,,9`, `4,`, `0`). Digits are accumulated by hand, so no atoll and no overflow past kMaxDigits. The wire behaviour is otherwise unchanged: a malformed list is still no list, never a guessed position, and never a truncated one. docs/JEV-SYSTEMONE.md now states the cap and corrects a guarantee the reviewers disproved: the per-field readout still calls BROW, so an engine built without --jev-brow answers `ERR BROW:` and the server ends and restarts it rather than serving a 503. Verified: jevpos_test (24 checks) passes standalone and under ctest; split_rules_test and verify_brow_at_test still pass; serve.test_systemone + serve.test_jev_calib 84/84 OK; `cmake --build build --target strata` links clean (one pre-existing unused-variable warning at generate.cpp:11780, untouched here).

…ed on the fixed engine Every B-arm number in this document was measured while the consolidated path silently fell back to one prompt per field (the jevpos parser pushed a position once per digit character and counted its 32-entry cap in digits, fixed in f2d2621f), so the old table's B column was really a per-field column on an older base. The "981 ms / 1.33x faster at two fields" claim is withdrawn. Measured 2026-10-23 on engine a2182599 (jev-port @ f2d2621f), RTX 3090, IQ3_XXS, median of 3 (scripts/rowreadout_probe.py; artifacts in the session's bench dir): N | C (default) | H (opt-in) | B (one prefill, opt-in) | B prompt tokens 1 | 209.7 ms (86 tok) | 209.3 ms (86) | 235.2 ms (86) | same 2 | 549.1 ms (189) | 915.1 ms (189) | 1013.2 ms (134) | 29% fewer 3 | 1231.7 ms (299) | 1470.7 ms (299) | 1299.5 ms (189) | 37% fewer 4 | 1405.4 ms (387) | 1826.9 ms (387) | 1447.2 ms (222) | 43% fewer 5 | 1698.5 ms (490) | 2287.5 ms (490) | 1726.6 ms (270) | 45% fewer So B uses 29-45% fewer prompt tokens from two fields up and is slower in wall clock at every N >= 2: C's pinned checkpoint makes each extra field cheaper than B's single longer prefill. Accuracy is unchanged (banking77 200 items, 2 fields: C .660 vs B .665 card, .750 vs .755 group, McNemar both-right 2 vs 3, p = 1.000) and field order still matters on B (147/200 agreement, mean |dp| .1326, 52 flips one way against 1, McNemar p = 0.0008, first/second request latency controlled at 681.7 vs 682.4 ms). The C row also carries the numbers measured today (1/2/3/4/5 fields 210/549/1232/1405/1699 ms) instead of the unreproducible older set, and the "route by field count" advice no longer recommends B at N=2. Also fixes scripts/jevport-arm.sh: its labels check read a variable filtered to ROWS|HYBRID, so `--labels digit-lower` reported MISMATCH after setting the env correctly. Found while running the label A/B, which then measured 160/160 identical decisions between the two alphabets.

One prompt for N fields lets each later field see the earlier ones, so the caller's key order decided the answers for the same set of questions. Measured live on 200 banking77 items before this change: reversing the two fields agreed on 147/200, 52 flips one way against 1 the other, mean |dp| .1326, McNemar p=0.0008 - i.e. the opt-in path was not reproducible under a caller that reorders its fields. The fields are now sorted by qid before the consolidated prompt is rendered. The same set of questions therefore renders the same prompt and names the same cue rows whoever asks, and because answers are keyed by qid the response shape does not change. This does not remove the intrinsic "later fields see earlier questions" bias - that is what one consolidated prompt means - it removes the part that depended on the caller. Measured after, same 200 items, same run shape (scripts/order_test.py --alternate 1): is_card agreement .995 (199/200), flips 0 one way / 1 the other, mean |dp| .007, McNemar p=1.0 group agreement 1.000 (200/200), accuracy .750 forward and reversed latency 683.5 vs 676.4 ms, prompt tokens 145.0 both The per-question path is untouched (this branch is only taken with STRATA_JEV_ROWS=1). serve/test_systemone.py::RowReadout::test_field_order_is_canonical pins the contract: the same questions in either order must render an identical prompt and name identical rows.

… is not a latency path B row + PR section now carry the post-fix order numbers (199/200 under field reversal, |dp| .007, McNemar p=1.0, vs 147/200 and p=.0008 before), the fact that which field goes FIRST still moves answers (is_card .660 asked first vs .785 asked second on the same 200 items), and the reason no wall-clock win exists: the two-stage pipelined prefill keeps no head rows, so a capture request runs on the serial window path.

…sumed from Arm C pins the fields' shared prefix, so every field after the first resumes from the checkpoint the pin marks. None of them has anything of its own to park, and parking is real work: about 75 ms per field on the 3090. Measured on the 3090 (200 banking77 items, 3 reps, medians, warm), same engine, only this branch of the code changed: fields 1 2 3 4 5 park 657 1047 1050 1260 1527 ms no park 726 1119 878 985 1214 ms -20% to -24% from three fields up. Field 1 is untouched by design (it creates the checkpoint the pin marks); its 657 -> 726 is not a regression but the cold first-request effect measured separately at 747.9 ms against 569.5 ms for the second request, and it moves between runs. The suppression is confined to exactly the fields that resume: field 1 keeps its park, and a run without a pin suppresses nothing, so no other path changes. It previously lived inside the hybrid row-capture branch, which meant it only applied to callers holding both a pin and the capture hook - the plain pinned path was still paying for every park. Hoisting it one level out fixes that. Answers: 200 items, park-suppressed against the fully unpinned read - is_card 199/200 agreement (1 flip), group 198/200, mean |dp_yes| .0104 (median .0047), accuracy .665/.745 against .660/.745. The same band the pin itself was accepted on, so this is a caching change, not a readout change.

… agreement

leonardomurillo's 7-arm run (Niko1221#3743) showed our per-field is_card at .766-.805 against his .870-.883 (McNemar p=0.008). The error analysis on our own logs (200 banking77 items): 65 of 67 errors are false positives, the group prompt - which shows the taxonomy - is right where is_card is wrong on 40 items, and the best threshold moves .665 to .810 while AUC stays low. Cause: a derived binary field asked without its sibling categories lets the model answer the everyday meaning; his field catalogue pins the taxonomy meaning. Fix: an opt-in top-level 'context' field (string or list of strings) rendered VERBATIM between the state and the question block, in every field's prompt - per-field, V-mode branches, and the consolidated rows path (unit 0, so the pin/jevpos math is unchanged). No context: the render is byte-identical to the donor's frozen shape, so every fitted calibrator still applies. Wrong type: 422 field 'context' (fail closed). Context-bearing requests read from a shifted distribution - calibrate for the use. 8 new tests (serve/test_jev_context.py): default render unchanged, placement per field, list join, blank-only == absent, 422 on wrong types, V branches, consolidated STRATA_JEV_ROWS=1. Suites: test_jev_context 8/8, test_systemone 55/55, jev discovery 75/75.

…fit tooling swap() keys calibrators by READOUT LETTER, so one calib-L.json served every text task. A fit is a domain claim; the shipped one (a_noul 0.6127, c_noul +0.6539, fitted on BoolQ60/ARC60/SST5-60) was applied to a balanced binary grouping task that needs c_noul ~ -1.56. Measured on 500 gold-labelled banking77 items, one GPU, same binary, only the calibrator differing (5-fold CV, held-out folds): shipped calib-L acc 0.664 brier 0.2325 ECE 0.275 identity (none) acc 0.760 brier 0.1983 ECE 0.218 refit this task acc 0.816 brier 0.1315 ECE 0.109 Doing nothing beat what shipped (+9.6 points, paired McNemar p<0.001). The calibrator admitted from raw p>0.256 and swept 187 undecided items into "yes": specificity 0.360 vs 0.584. Changes: - Calib carries the `task` it was fitted for; loader reads it; it joins _KNOWN_KEYS so STRATA_JEV_CALIB_STRICT does not reject task-tagged files. - STRATA_JEV_TASK names the runtime task. A fit whose task does not match is NOT applied: identity beats a stranger's intercept, so the refusal is the safe default and it is reported as calib:"identity" plus one log line. An UNTAGGED fit is refused too once a task tag is set. Unset tag = legacy behaviour, byte-identical to the last release. - scripts/refit_calib.py: collect readouts (or reuse an eval arm), stratified dev/holdout split, damped-Newton Platt fit for noul, temperature for choice/score, and a GATE that refuses to write a fit which does not beat identity on accuracy AND ECE. Writes task + provenance with holdout numbers. - docs/CALIBRATION.md + docs/examples/calib-items-example.jsonl: the operator recipe (label ~25-100 items, fit on half, gate on the other half, deploy with a task tag, verify twice). Verification: serve.test_jev_calib + serve.test_jev_calib_task 39/39 OK, serve.test_systemone 55/55 OK (container venv, /cfg/scratch-test copy). Deployed and held-out verified separately: fit on 250 dev items served the untouched 250 -> acc 0.816, brier 0.1285; vs shipped on the same items fixed 59 / broke 21, McNemar p=2.5e-05; repeat pass 247/250 identical decisions. Caveat recorded: the engine is NOT bit-exact run to run (|dp_yes| up to 5.7e-2), so cross-run differences under ~1.2% accuracy are noise.

…r log A static fit answers "what boundary was right last month". dynamic_calib.py moves the boundary from LIVE traffic without traffic labels: the labelled dev set gives the confusion matrix C[pred][y] once, the server already logs the prediction marginal (calib/data/log-<mode>.jsonl, one row per answer with raw_p), and BBSE solves C*pi = q for the traffic class prior, which sets the threshold. Measured on 500 gold-labelled banking77 items (labels used only for scoring): shipped static fit (wrong task) acc 0.664 identity, no calibration acc 0.764 BBSE prior-matched, NO traffic labels acc 0.816 oracle prior-matched (upper bound) acc 0.816 labelled offline refit acc 0.816 Unlabelled base-rate estimate 0.484 vs a true 0.500. Streaming five 50-item batches with shrinkage + EMA: dynamic 0.828 vs static 0.764; single-batch BBSE is noisy (n=50 mean 0.488, sd 0.108), hence shrinkage is mandatory. On this rig's real serving log (984 rows): estimated prior 0.493 (true 0.500), recommended c_noul -1.6186 vs the hand-fitted -1.5607 -- agreement to 0.06 from independent information. Deliberately a scheduled job writing a calib file, NOT an in-process control loop: bounded step (--max-step), shrinkage toward the prior (--n0/--ema), a refusal when the estimate is implausibly far from the expected prior (--alarm, catches a shift or broken logging), a singularity refusal when C cannot be inverted, and every update stamped into a `dynamic` block with n, estimate, threshold and c_prev->c_new. docs/CALIBRATION.md gains §10: what makes it possible, the measured table, the guard rails, and when NOT to use it (no dev set / near-binary readout / real labels available). Verified end to end against the live log; validator clean on our stanzas.

…t files CI "Minimal serving runtime" failed on 3 tests in serve/test_systemone.py from commit b5a8f48 onward (passed at 19:55, failed on the 22:02 and 22:08 runs): test_noul_answer got 0.9570665554 expected 0.982 test_score_answer got 0.9524334890 expected 0.9887011299 test_choice_answer got 0.0219656216 expected 0.0178982102 Cause: my tearDown called jev_calib._set_for_test(None), which defaults to mode "", so the fit my test had parked in _ACTIVE["L"] stayed there. unittest discovers files alphabetically, so this file ran before test_systemone and the server those tests start applied the leftover Calib(a_noul=0.6127, c_noul=+0.6539, T_choice=1.0557, T_score=2.263) instead of identity. All three CI values reproduce exactly from that object: 0.982 -> sigmoid(0.6127*logit+0.6539) = 0.9570665554 0.9887011299 -> T_score=2.263 = 0.9524334890 0.0178982102 -> T_choice=1.0557 = 0.0219656216 Fix: setUp snapshots the whole _ACTIVE dict, the task override and the mismatch log; tearDown restores all of them. New test_restore_clears_every_mode pins the isolation, asserting active("L") and active("") are both None after restore and that apply(mode="L") then reports identity with probabilities untouched. Verified in the container venv against the repo tree: test_systemone alone 55/55 OK test_jev_calib_task + test_systemone, both orders 63/63 OK test_jev_calib + test_jev_calib_task + test_systemone 95/95 OK (pre-fix, the CI ordering reproduced the same 3 failures with the same values)

justindressler mentioned this pull request Oct 21, 2026

[Engine] Jev System One decision endpoint (/v1/systemone): typed decisions with calibrated probabilities #2639

Closed

leonardomurillo mentioned this pull request Oct 22, 2026

[Engine] typed decisions: score a schema's token paths instead of generating them #3743

Open

This was referenced Oct 31, 2026

[Engine] Anthropic errors on /v1/messages use Anthropic's error shape, so the Vercel AI SDK shows Strata's message #4834

Open

[Engine] Direction: which vLLM and SGLang API extensions are worth adding for local coding agents? #4840

Open

Suzie (Hermes) added 18 commits October 31, 2026 19:52

test(systemone): RED - /v1/systemone validation, readout math, frozen…

45f470a

… prompt (10 failing)

test(brow): RED - StrataEngine.final_row_logits wire (BROW->BROWD, fa…

dc4c929

…il-closed)

feat(brow): GREEN - StrataEngine.final_row_logits (BROW->BROWD wire, …

d571e34

…fail-closed) + jev.handle wired to real contract

test(brow-engine): RED - Verifier::brow_row fail-closed pre-run contr…

191bb04

…act (CUDA test target)

feat(brow-engine): GREEN - Verifier::brow_row (fail-closed row gather…

fa90740

…, CUDA+Metal), --jev-brow flag, serve-loop BROW handler; golden battery fixture

test(jev-golden): battery harness - 12 items, invariants+repeat deter…

bc60016

…minism; CMake CPU guard fixed (15/15)

fix(jev-brow): restore the usage() indent the --jev-brow insertion sw…

a37c708

…allowed (review r1)

suzie and others added 25 commits October 31, 2026 19:52

bench: GQA-600 (C arm) + cross-domain calibrator transfer matrix + re…

b616bf8

…sume-safe row persistence

bench: image suite with shipped calib-D served (paired vs identity, 0…

abeb3b2

… decision flips)

chore: commit GQA set builder, mf eval, image calib comparator, probe…

63ecf30

… tweak

bench: GQA-600 arm B vs C (paired; served probs inverse-Platt'd to raw)

2826058

test(jev): label alphabet A/B (upper vs 0-9/a-z) + --labels on the ar…

b3dd100

…m script

test(jev): pin the D tuple on the direct image path (the refactor's o…

27e9e4a

…nly real risk)

docs(jev): reproduce the numbers on this PR's build, and say what cou…

aab939c

…ld not be reproduced

docs(jev): state the park fix's measured numbers, 1214 ms and 199/200…

807f169

… agreement

justindressler force-pushed the jev-port-r2 branch from 86f2ae3 to 8135af7 Compare October 31, 2026 19:55

Suzie added 3 commits October 31, 2026 22:01

justindressler mentioned this pull request Nov 1, 2026

jev-calib: ship no calibrator parameters, ship the tools to fit your own justindressler/Strata#1

Open

Navigation Menu

Repository navigation

[Engine] Jev System One decision endpoint (/v1/systemone): typed decisions with calibrated probabilities - #3643

[Engine] Jev System One decision endpoint (/v1/systemone): typed decisions with calibrated probabilities#3643
justindressler wants to merge 77 commits into
Niko1221:mainfrom
justindressler:jev-port-r2

justindressler commented Oct 21, 2026 •

edited

Loading

Uh oh!

leonardomurillo commented Oct 22, 2026 •

edited

Loading

Uh oh!

justindressler commented Oct 22, 2026 via email

Uh oh!

justindressler commented Oct 22, 2026 via email

Uh oh!

justindressler commented Oct 22, 2026

Uh oh!

justindressler commented Oct 23, 2026

Uh oh!

justindressler commented Oct 23, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

justindressler commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem and result

Related discussion

Approach

Validation

Limits

CI on this PR — what is failing, and why it is not this branch

1. The three check suites are action_required, not failed

2. Two tests in the shared suite are flaky, and both fail on main without this branch

What was actually run against this branch

Uh oh!

leonardomurillo commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

justindressler commented Oct 22, 2026 via email

Uh oh!

justindressler commented Oct 22, 2026 via email

Uh oh!

justindressler commented Oct 22, 2026

Uh oh!

justindressler commented Oct 23, 2026

Uh oh!

justindressler commented Oct 23, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

justindressler commented Oct 21, 2026 •

edited

Loading

1. The three check suites are `action_required`, not failed

2. Two tests in the shared suite are flaky, and both fail on `main` without this branch

leonardomurillo commented Oct 22, 2026 •

edited

Loading