Repository navigation
release-check: a laya-auto row, so the first request is the one Laya decided - #1071
Conversation
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at c928ce4-equivalent head — self-test passes here and the verdict helper is tested in both directions, which is the thing that makes a gate worth having.
Ran it: bash os/mini/release-check.sh --self-test → release-check self-test PASS, including the class→model mapping (prose -> ZAYA1-8B-1bit, code -> Qwen3-Coder-30B-A3B-Instruct-1bit, long_doc -> Qwen3.8-27B-1bit, not-a-class -> (empty)).
The part that matters — laya_verdict() is exercised as a gate, not as a happy path. Nine cases, six of them failures, and the failures are the ones that would otherwise pass a status-only row:
PASS the good decision
FAIL default_used:true -> fell open to default_model, so Laya did not decide
FAIL matched_rule:"long-input" -> a deterministic rule decided, not Laya
FAIL matched laya-prose when expecting code -> the wrong class's rule
FAIL routed to Qwen3-0.6B when code needs the Coder -> the wrong model
FAIL no "classifier:laya" in the trace -> no Laya score
FAIL empty body -> nothing answered
That is the "answered 200 so nothing looks wrong" case caught by construction: a request that lands on the right model by accident fails three of those six.
The row itself only fires when it can mean something. It skips (visibly) unless a 1bit.Laya-First router is registered in MODELS, and it probes one prompt per class with the strict laya_verdict "$body" "$class" "$model" form, so the strict checks are the ones in force on an appliance.
One nit, and it is the fail-open shape that has bitten this repo before: laya_verdict silently checks less when it cannot map the class. _cls/_want empty skips both "which class" and "which model", so laya_verdict "$body" '' '' returns PASS for any body that matched some laya-* rule, did not fall open, and carried a Laya score — the self-test's own line 540 asserts that behaviour. Today the row always passes a known class so _want is never empty, and laya_class_model() returns '' for an unknown one, which is exactly how an empty pair could reach the helper if a future row adds a class without touching the mapping. One line closes it: if $2 is non-empty and laya_class_model came back empty, return FAIL no model is named for class <x> in this script rather than checking only the shape. A gate should fail closed when it does not understand its own input.
Not verified: the live row on an appliance (it needs a Laya checkpoint and a running Lemonade) — everything above is the device-free half, which is the half CI runs.
|
Two defects in my own row, both fixed at
Also fixed in the same exercise: the row's class floor. Measured after all fixes, against a live lemond with a stand-in engine: |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 28c2c27e — the router name now comes from the engine instead of the script, and that is the fix that makes this row survive the name change #1069 just made.
The change is right and I checked it against the binary it parses. #1069 corrected the desugared policy's model_name from 1bit.Laya-First to user.Laya-First (the server only registers user.* definitions), and this head stops hardcoding it:
LAYA_ROUTER=user.Laya-First # the documented default, not the authority
for _b in $(command -v 1bit) /opt/1bit-engine/1bit /opt/1bit/1bit /opt/1bit-package/bin/1bit; do …
_n=$("$LAYA_BIN" laya-first | sed -n 's/.*"model_name": *"\([^"]*\)".*/\1/p' | head -1)
[ -n "$_n" ] && LAYA_ROUTER=$_n
On the rebuilt engine binary that parse yields user.Laya-First, which is exactly what the row then looks for in MODELS — so the row reads the name from the thing that produces it rather than from a copy that would drift the day the table changes. The fallback chain is sensible too: a documented default if no binary is found, and the row still skips visibly rather than passing when no such router is registered.
--self-test passes at this head (release-check self-test PASS), and auth_post → auth_post_watch is the right instrument for a row that must not block a release-check run on a slow first request.
My earlier nit stands unchanged and is still a nit: laya_verdict() checks less when it cannot map a class (laya_verdict "$b" '' '' → PASS on the shape alone, asserted by its own self-test line). Today the row always passes a known class, so nothing is wrong — but a gate that fails closed on an input it does not understand is one line better, whenever this file is next open.
|
The |
|
The fail-closed nit is fixed at |
…decided docs/laya-first.md §7 makes this the gate: on a cold Lemonade the first request must go through Laya's router, which classifies it and picks one of our models. The row sends one prompt per class through the Laya-first router and reads the router's own decision back. What makes it a row and not a chat test is laya_verdict(), which requires all four of: the class's Laya rule matched, the request went to that class's model, it did NOT fall through to default_model, and Laya's score is in the trace. The case it exists for is a request that lands on the right model by accident - a deterministic rule, or the default - which any status-only row calls healthy. That case is in the self-test, along with the wrong class, the wrong model, a missing Laya score, and an empty body. The row skips, with the fix named, when no 1bit.Laya-First router is registered: on a stock stick there is no Laya checkpoint, and a row that cannot pass must say so rather than fail the release. Self-test: 65 checks pass (10 new), and ctest's release_check_row still passes. The class -> model mapping the row asserts is mirrored from config/laya-first.json and is itself self-tested, so a table change turns into a red self-test instead of a row that checks yesterday's promise. Not run on a device yet: the row needs a board with a Laya checkpoint and the router registered, which is the release lane's window, not this one. What is proven here is the verdict logic and the skip path.
…her that works on curl-only images
Two defects found by exercising the row off the stick rather than by reading it:
1. The name is not a constant. A live lemond refuses to register anything outside `user.*`
("Registered model definitions must use a non-empty `user.*` name") - so the row now reads the name from
the engine's own desugar (`1bit laya-first` prints `model_name`) and falls back to `user.Laya-First`. A
hardcoded name would also break the row the day the table's name changes.
2. `auth_post` spells its body as wget's `--post-data`, so on an image with curl but no wget it returns
nothing and every post row reads as "no answer". This row uses `auth_post_watch`, which handles both
fetchers and hands back the body. (The other post rows have the same shape; flagging it here rather
than changing them in a release-check PR.)
Measured against a live lemond with a stand-in engine: short, prose and code each PASS through the row's own
verdict, and an engine that cannot answer FAILs with "no Laya rule matched the request" - which is the
"answered 200 on the default" case the row exists to catch.
…annot name laya_verdict() skips both the class check and the model check when it is handed an empty expectation, so a row for a class that is not in laya_class_model() would PASS on shape alone. Fail closed instead, naming the class, and assert it in --self-test. bash os/mini/release-check.sh --self-test -> 66 ok, 0 FAIL, self-test PASS
4995fc3 to
55337f3
Compare
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 4995fc31 — the author closed the fail-open I raised, and I proved the guard is load-bearing rather than decorative.
The delta since my READY (28c2c27e) is one commit, os/mini/release-check.sh +7/−0: a guard at the top of laya_verdict() that fails closed when laya_class_model has no model for the class, plus a self-test case for it. That is exactly the nit — "the helper checks less when it cannot map a class" — and it is fixed in the right place, before the rule check, so an unmapped class can no longer reach a check that is skipped by default.
Tested both versions of the function with the same body (a laya-ghost rule and a trace score, i.e. every other check satisfiable, _want empty so the model comparison would be skipped):
OLD (28c2c27e) laya_verdict <body> ghost '' -> PASS
NEW (4995fc31) laya_verdict <body> ghost '' -> FAIL no model is named for class ghost in this script
So the old code passed a row whose class has no model — the vacuous pass — and the new one refuses it by name. --self-test passes at the new head, and the added case is the one that would have failed before.
Everything from my original READY stands: --self-test PASS, and laya_verdict() exercised in both directions, the six of nine cases that must fail still failing (fell-open, deterministic rule, wrong class, wrong model, no Laya score, empty body).
Nothing else in os/mini/release-check.sh moved, and this row remains documentation-or-gate only — no engine behaviour changes.
… of them inside the split's file, and the moved block survives it main went bd7d451 -> cb5866e -> 05da4d8 while the five gate rows were being re-measured, and one of the twelve commits edits the file the split is built around: npu/forward/src/q4nx_forward.cpp +38/-5 (#1087 rope-scaling refusal by name, #1072 the batch phase counters zeroed BEFORE BatchPhases snapshots them) npu/forward/include/q4nx_rope.h +72 (new: q4nx_rope_declared_type / q4nx_rope_type_supported) npu/pack_import.cpp/.h +97 (#844 final-norm import from the .fconst tail envelope) app/, config/, docs/, tests/, tools/, os/ (laya-first #1069, determinism tool #1068, serve logging #1074, onnx pricing #1076, puppy vm-check #1080, release-check #1079/#1071) third_party/llama.cpp pin bump 6e42864a6 (#1075) Textually clean (`git merge-tree --write-tree` produced no CONFLICT), which is exactly the case that was NOT clean the last time main landed in that file — so the merge was checked by what the compiler and the tests say, not by "auto-merging" messages: - #1072 moved `BatchPhases bph` / `BatchPhaseScope _batch_scope` / `tp_bs` so the counters are zeroed before the snapshot is taken. Those symbols live in npu/forward/include/q4nx_lh.h in this branch (the move this PR exists for); the statement order is main's and is carried over verbatim, so the reordering lands on the header's definitions unchanged. - The moved set is still defined exactly once: `inline StepPhases g_ph` (q4nx_lh.h:116), `phprof_on`/`ph_ms` (q4nx_lh.h, one each), `inline bool g_in_batch` (:131), `struct BatchPhaseScope` (:132), `lh_dump_layer_host` — and q4nx_forward.cpp no longer defines any of them. - `BatchPhases` remains a local struct in q4nx_forward.cpp (one definition), which is where main keeps it. Build: `cmake --build build-p4 --target onebit -j4` under ~/lb/thermal-run.sh -> rc 0, 16 objects rebuilt (warm tree; the pin bump and the two touched TUs). Tests: `ctest` **87/87, 0 failed** (85 passed, 2 skipped: mtp_batch_all, gdn_batch_order — the same two that skip on main). Consequence, stated rather than implied: the five gate rows measured on A=bd7d451a are VOID and are being re-run against A built at this main; this commit carries no numeric claim of its own.
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
What
docs/laya-first.md§7 makes this the gate for the whole feature: on a cold Lemonade the first request mustgo through Laya's router, which classifies it and picks one of our models. This adds that as a
release-check row,
laya-auto.Why it is a verdict and not a chat test
laya_verdict()requires all four of:matched_rule: laya-<class>),default_model("default_used": false),classifier:laya).The case it exists for is a request that lands on the right model by accident — a deterministic rule, or
the default — which any status-only row calls healthy. That is exactly the failure a first-request gate has
to catch: the request answered 200, so nothing looks wrong, and the whole premise of the feature is silently
absent.
Self-test (device-free, CI)
./os/mini/release-check.sh --self-test→ 65 checks pass, 10 new, and ctest'srelease_check_rowpasses:Skip rather than fail
On a stock stick there is no Laya checkpoint, so no
1bit.Laya-Firstrouter is registered and the rowskips with the fix named (
scripts/fetch-laya.sh, then1bit laya-first --policy …) instead of failing arelease it cannot judge. A row that cannot pass on the artifact must say so — silently passing would be worse,
and failing would block on something that is not a regression.
The class → model mapping the row asserts mirrors
config/laya-first.json(engine #1069), and is itselfself-tested, so changing the table turns into a red self-test rather than a row that checks yesterday's
promise.
1bit laya-first --checkvalidates the table itself.Not run on a device
The row's own arm needs a board with a Laya checkpoint and the router registered — the release lane's window,
not this one. What is proven here is the verdict logic, the mapping, and the skip path; the board arm is
written and will run when the next stick carries a checkpoint.