Repository navigation
feat: add DecisionEngine for Laya decision models on native llama.cpp - #606
Merged
Merged
Conversation
Question and answer types, Python json.dumps parity, Laya 0.3.5 sequence assembly and answer decoding, with a parity fixture from the official checkpoint. Not exported yet; the native backend and DecisionEngine facade follow in the next PR. Design: doc/decision_engine.md.
Adds the BackendDecision contract, the native path (safetensors reader, ggml head graph with Windows ggml-base twins, private pooling-NONE encoder context, worker messages, router forwarding), LlamaEngine hooks with engine-owned head handles, and the public DecisionEngine facade. Web and LiteRT-LM report unsupported. Includes a local-only E2E against the laya 0.3.5 fixture, the decision-model-smoke runner scenario, and docs.
Move the decision encoder context setup into applyDecisionContextParams with unit tests, state that a call already sent to the backend finishes on the unloaded model, and add local F16 backbone measurements.
This was referenced Sep 23, 2026
leehack
changed the title
12 tasks done
11 of 12 tasks
…code - Stream head tensors into the upload buffer; one head load peaks about 100 MiB lower. - Allocate head weights with ggml_backend_alloc_ctx_tensors instead of a hand-written layout, dropping six GgmlGraphApi entries and their Windows twins. - Replace the fdlibm erf port with Abramowitz and Stegun 7.1.26 (within 1.4e-7). - Compute the head's last layer only for the CLS and marker rows. - Type BackendDecisionSequence.questionType as DecisionQuestionType. - Parse the head config once, with headLayers on DecisionHeadConfig. - Return the encoder output as a view, reuse the service's device matching, and test the head's device choice. - Tighten tests that could not fail and share the synthetic head and fixture helpers.
- Add ChoiceKey, ScoreKey and NoulKey: typed question handles read back with answerOf, including enum and arbitrary option values, checked against the request that produced the result. The string and JSON forms are unchanged. - Accept structured instructions in the question constructors. - Check empty question ids in DecisionRequest, tokenize each text once per call, clamp temperatures once, and reject non-string JSON keys. - Docs: typed questions guide section, Experimental label, corrected accuracy and Laya-compatibility wording, and one home for the measurements.
- systemOneBatch answers the requests as they were when it started, so changing the list mid-call can no longer mislabel answers. - DecisionEngine.load rejects a model unloaded or replaced during the capability probe instead of returning an engine for the old model. - The real-model smoke fails when the requested GPU backend fell back to CPU, and checks that a config longer than the encoder was trained for is rejected. - The guide says the head falls back to CPU when no device of the model's backend is available.
leehack
added a commit
that referenced
this pull request
Sep 23, 2026
- Merge feat/decision-native (d9abfc3): typed ChoiceKey, ScoreKey and NoulKey, the Experimental label, and BackendDecisionSequence.questionType as DecisionQuestionType. - Web passes questionType.index to the bridge; drop the int32 range check and its test, which only guarded the int field. - Keep the marker-count check next to #606's validation, without the removed question type check. - Test typed key reads and the question identity check through the Web backend; docs combine the Experimental label with Web support.
This was referenced Sep 23, 2026
leehack
marked this pull request as ready for review
September 23, 2026 17:41
Contributor
|
Chat app preview removed for |
leehack
added a commit
that referenced
this pull request
Sep 23, 2026
Re-measure the question-time table over three runs on this head; the old ranges predated the pin and the #606/#616 merge. The 12-epoch head pinned in 84b61bb is not better everywhere than the 8-epoch head it replaces (revision 83794e0b). Turn-based with "3 best + 3 random" it plays 150 pieces and 48 lines against 122 and 34, and with all legal moves 135 and 40 against 134 and 40. In real time with all legal moves, the default off Android, it cleared 36.2 lines per game over six games against 47.2; each head won three of the six seeds.
leehack
added a commit
that referenced
this pull request
Sep 23, 2026
leehack
added a commit
that referenced
this pull request
Sep 23, 2026
12 tasks done
11 of 13 tasks
leehack
added a commit
that referenced
this pull request
Sep 23, 2026
…sion model (#617) * feat: add the pure-Dart core for Laya decision models Question and answer types, Python json.dumps parity, Laya 0.3.5 sequence assembly and answer decoding, with a parity fixture from the official checkpoint. Not exported yet; the native backend and DecisionEngine facade follow in the next PR. Design: doc/decision_engine.md. * feat: run Laya decision models on native llama.cpp with DecisionEngine Adds the BackendDecision contract, the native path (safetensors reader, ggml head graph with Windows ggml-base twins, private pooling-NONE encoder context, worker messages, router forwarding), LlamaEngine hooks with engine-owned head handles, and the public DecisionEngine facade. Web and LiteRT-LM report unsupported. Includes a local-only E2E against the laya 0.3.5 fixture, the decision-model-smoke runner scenario, and docs. * fix: test decision CPU context flags and correct unload-during-call docs Move the decision encoder context setup into applyDecisionContextParams with unit tests, state that a call already sent to the backend finishes on the unloaded model, and add local F16 backbone measurements. * test: start the decision context fixture from a pooled struct * feat(basic_app): add a DecisionEngine ticket-triage example bin/llamadart_decision_example.dart downloads the pinned Laya backbone and head, answers a choice, a score and a yes/no question about a support ticket, and can print Laya's JSON response. * feat(laya_tetris): add a Flutter Tetris example played by a Laya decision model example/laya_tetris plays turn-based and real-time Tetris through DecisionEngine with heuristic, yes/no, checklist, choice and Tetris-tuned players, a benchmark screen, a headless bench and a dataset generator for fine-tuning the head. * feat(laya_tetris): download the published Tetris-tuned head by default The tuned head now resolves to LAYA_TUNED_HEAD_URL, then a file in the app's model folder, then leehack/laya-tetris-head pinned to a commit. A failed tuned head shows advice for its source, and the benchmark times the head that loaded. * refactor: stream the decision head load and trim hand-written native code - Stream head tensors into the upload buffer; one head load peaks about 100 MiB lower. - Allocate head weights with ggml_backend_alloc_ctx_tensors instead of a hand-written layout, dropping six GgmlGraphApi entries and their Windows twins. - Replace the fdlibm erf port with Abramowitz and Stegun 7.1.26 (within 1.4e-7). - Compute the head's last layer only for the CLS and marker rows. - Type BackendDecisionSequence.questionType as DecisionQuestionType. - Parse the head config once, with headLayers on DecisionHeadConfig. - Return the encoder output as a view, reuse the service's device matching, and test the head's device choice. - Tighten tests that could not fail and share the synthetic head and fixture helpers. * feat: add typed decision keys and mark DecisionEngine experimental - Add ChoiceKey, ScoreKey and NoulKey: typed question handles read back with answerOf, including enum and arbitrary option values, checked against the request that produced the result. The string and JSON forms are unchanged. - Accept structured instructions in the question constructors. - Check empty question ids in DecisionRequest, tokenize each text once per call, clamp temperatures once, and reject non-string JSON keys. - Docs: typed questions guide section, Experimental label, corrected accuracy and Laya-compatibility wording, and one home for the measurements. * refactor(basic_app): read decision example answers through typed keys The ticket triage asks ChoiceKey.enumOf over a Department enum, ScoreKey.of and NoulKey.of, and formats answers read with result.answerOf; output and exit codes are unchanged. Drop the "laya.config" error-text sniff; the library message and --config help cover it. * fix: snapshot decision batches and reject a model swapped in during load - systemOneBatch answers the requests as they were when it started, so changing the list mid-call can no longer mislabel answers. - DecisionEngine.load rejects a model unloaded or replaced during the capability probe instead of returning an engine for the old model. - The real-model smoke fails when the requested GPU backend fell back to CPU, and checks that a config longer than the encoder was trained for is rejected. - The guide says the head falls back to CPU when no device of the model's backend is available. * refactor(laya_tetris): read the choice player's answer through a typed key * test(laya_tetris): check the choice player's scores and key identity FakeLaya now keeps each request's questions as DecisionEngine does, so a rebuilt key fails the typed read, and the base-head choice test checks every option's probability in option order. Drop the delivered examples from the remaining-work sentence in the decision engine notes. * fix(basic_app): restore the --config hint and order-check triage questions A head load that fails with LlamaModelException and no --config prints the --config hint again, without matching error text; a failed backbone load does not. The questions test compares JSON text so key and option order count, and the formatter test is named for its fixed order. * feat(laya_tetris): pin the 12-epoch tuned head Pin leehack/laya-tetris-head at 465546a5 and re-measure the tuned rows with it on Metal. * docs(laya_tetris): re-measure question time and qualify two-game rows Re-measure the question-time table over three runs on this head; the old ranges predated the pin and the #606/#616 merge. The 12-epoch head pinned in 84b61bb is not better everywhere than the 8-epoch head it replaces (revision 83794e0b). Turn-based with "3 best + 3 random" it plays 150 pieces and 48 lines against 122 and 34, and with all legal moves 135 and 40 against 134 and 40. In real time with all legal moves, the default off Android, it cleared 36.2 lines per game over six games against 47.2; each head won three of the six seeds. * docs(laya_tetris): add the 40-game results for the tuned player
leehack
added a commit
that referenced
this pull request
Sep 23, 2026
…etris (#619) * feat: add the pure-Dart core for Laya decision models Question and answer types, Python json.dumps parity, Laya 0.3.5 sequence assembly and answer decoding, with a parity fixture from the official checkpoint. Not exported yet; the native backend and DecisionEngine facade follow in the next PR. Design: doc/decision_engine.md. * feat: run Laya decision models on native llama.cpp with DecisionEngine Adds the BackendDecision contract, the native path (safetensors reader, ggml head graph with Windows ggml-base twins, private pooling-NONE encoder context, worker messages, router forwarding), LlamaEngine hooks with engine-owned head handles, and the public DecisionEngine facade. Web and LiteRT-LM report unsupported. Includes a local-only E2E against the laya 0.3.5 fixture, the decision-model-smoke runner scenario, and docs. * fix: test decision CPU context flags and correct unload-during-call docs Move the decision encoder context setup into applyDecisionContextParams with unit tests, state that a call already sent to the backend finishes on the unloaded model, and add local F16 backbone measurements. * test: start the decision context fixture from a pooled struct * feat(basic_app): add a DecisionEngine ticket-triage example bin/llamadart_decision_example.dart downloads the pinned Laya backbone and head, answers a choice, a score and a yes/no question about a support ticket, and can print Laya's JSON response. * feat(laya_tetris): add a Flutter Tetris example played by a Laya decision model example/laya_tetris plays turn-based and real-time Tetris through DecisionEngine with heuristic, yes/no, checklist, choice and Tetris-tuned players, a benchmark screen, a headless bench and a dataset generator for fine-tuning the head. * docs(laya_tetris): add a notebook that fine-tunes the Laya head for Tetris example/laya_tetris/training trains Laya's decision head on the questions bin/make_dataset.dart writes, with the encoder frozen, and exports a safetensors head that DecisionEngine loads without configPath. The README also gives the recipe for building the backbone GGUF from the official checkpoint. * feat(laya_tetris): download the published Tetris-tuned head by default The tuned head now resolves to LAYA_TUNED_HEAD_URL, then a file in the app's model folder, then leehack/laya-tetris-head pinned to a commit. A failed tuned head shows advice for its source, and the benchmark times the head that loaded. * refactor: stream the decision head load and trim hand-written native code - Stream head tensors into the upload buffer; one head load peaks about 100 MiB lower. - Allocate head weights with ggml_backend_alloc_ctx_tensors instead of a hand-written layout, dropping six GgmlGraphApi entries and their Windows twins. - Replace the fdlibm erf port with Abramowitz and Stegun 7.1.26 (within 1.4e-7). - Compute the head's last layer only for the CLS and marker rows. - Type BackendDecisionSequence.questionType as DecisionQuestionType. - Parse the head config once, with headLayers on DecisionHeadConfig. - Return the encoder output as a view, reuse the service's device matching, and test the head's device choice. - Tighten tests that could not fail and share the synthetic head and fixture helpers. * feat: add typed decision keys and mark DecisionEngine experimental - Add ChoiceKey, ScoreKey and NoulKey: typed question handles read back with answerOf, including enum and arbitrary option values, checked against the request that produced the result. The string and JSON forms are unchanged. - Accept structured instructions in the question constructors. - Check empty question ids in DecisionRequest, tokenize each text once per call, clamp temperatures once, and reject non-string JSON keys. - Docs: typed questions guide section, Experimental label, corrected accuracy and Laya-compatibility wording, and one home for the measurements. * refactor(basic_app): read decision example answers through typed keys The ticket triage asks ChoiceKey.enumOf over a Department enum, ScoreKey.of and NoulKey.of, and formats answers read with result.answerOf; output and exit codes are unchanged. Drop the "laya.config" error-text sniff; the library message and --config help cover it. * fix: snapshot decision batches and reject a model swapped in during load - systemOneBatch answers the requests as they were when it started, so changing the list mid-call can no longer mislabel answers. - DecisionEngine.load rejects a model unloaded or replaced during the capability probe instead of returning an engine for the old model. - The real-model smoke fails when the requested GPU backend fell back to CPU, and checks that a config longer than the encoder was trained for is rejected. - The guide says the head falls back to CPU when no device of the model's backend is available. * fix(laya_tetris): train the head for 12 epochs so late-converging runs finish With 8 epochs, 2 of 10 MPS runs stopped at 0.705 and 0.707 accuracy after a long loss plateau; 8 runs with 12 epochs reached 0.755 to 0.785. * refactor(laya_tetris): read the choice player's answer through a typed key * test(laya_tetris): check the choice player's scores and key identity FakeLaya now keeps each request's questions as DecisionEngine does, so a rebuilt key fails the typed read, and the base-head choice test checks every option's probability in option order. Drop the delivered examples from the remaining-work sentence in the decision engine notes. * fix(basic_app): restore the --config hint and order-check triage questions A head load that fails with LlamaModelException and no --config prints the --config hint again, without matching error text; a failed backbone load does not. The questions test compares JSON text so key and option order count, and the formatter test is named for its fixed order. * feat(laya_tetris): pin the 12-epoch tuned head Pin leehack/laya-tetris-head at 465546a5 and re-measure the tuned rows with it on Metal. * docs(laya_tetris): describe the published head as the 12-epoch recipe * docs(laya_tetris): re-measure question time and qualify two-game rows Re-measure the question-time table over three runs on this head; the old ranges predated the pin and the #606/#616 merge. The 12-epoch head pinned in 84b61bb is not better everywhere than the 8-epoch head it replaces (revision 83794e0b). Turn-based with "3 best + 3 random" it plays 150 pieces and 48 lines against 122 and 34, and with all legal moves 135 and 40 against 134 and 40. In real time with all legal moves, the default off Android, it cleared 36.2 lines per game over six games against 47.2; each head won three of the six seeds. * docs(laya_tetris): compare the 12- and 8-epoch heads in real time With all legal moves in real time, the default off Android, the published 12-epoch head cleared fewer lines over six games than the earlier 8-epoch head, and each won three of the six games. State it next to the two-game table. * docs(laya_tetris): add the 40-game results for the tuned player * chore: retrigger CI after retargeting to main
This was referenced Oct 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an experimental
DecisionEngine(#604): Laya-style decision models on native llama.cpp. A ModernBERT encoder GGUF plus a safetensors head answers choice, score and yes/no questions in Laya'ssystem_oneformat. This PR carries the pure-Dart core that was #605, which exported nothing on its own; its commit038dea663is the first commit here.lib/src/core/decision/):python_json.dart: byte parity with Pythonjson.dumps(floatrepr,NaN/Infinity,ensure_ascii), because Laya formats non-string inputs with it.decision_question.dart,decision_result.dart: question, request and answer types in Laya's wire format. Question constructors accept structured instructions.decision_key.dart: typed keys (ChoiceKey,ScoreKey,NoulKey) that read answers back withanswerOf, including enum and arbitrary option values, checked against the question that built the request. The string and JSON forms are unchanged.decision_sequence.dart: Laya 0.3.5 option rendering and sequence assembly.decision_decoder.dart: temperatures, confidence, act features, answer decoding.LlamaDecisionException, the design docdoc/decision_engine.md, and a 24-row laya 0.3.5 parity fixture undertest/fixtures/decision/with the scripts that produced it.DecisionEnginewithload,capabilitiesFor,info,systemOne,systemOneBatchanddispose.DecisionCapabilities,DecisionModelInfo, the question, answer and key types and theBackendDecision*contract are exported. Marked experimental in the README, CHANGELOG, guide and support matrix.systemOneBatchanswers the requests as they were when the call started; changing the list mid-call cannot mislabel answers.loadrejects a model that was unloaded or replaced by one with a different handle during its capability probe.backendDecisionCapabilities,loadDecisionHeadBackend,runDecisionBackend,freeDecisionHeadBackend. The engine hands out its own head handles and never reuses them, so a staleDecisionEnginecannot reach a head that a restarted worker gave a recycled number. Unloading the model forgets every head.safetensors.dart: bounded header parse and ranged reads; F32, F16 and BF16.decision_head.dart: ggml head graph and act MLP. Head tensors stream into the upload buffer (one head load peaks about 100 MiB lower) and are allocated withggml_backend_alloc_ctx_tensors. The last layer runs only for the CLS and marker rows.ggml_graph_api.dart: every ggml call goes through@Nativetwins on Windows, becausellama.dllexports no graph symbols.llama_contextper head: pooling NONE,max_lentokens. The head falls back to the CPU when no device of the model's backend is available.NativeAutoBackendforwarding. The LiteRT-LM delegate reports unsupported.freeModelanddisposefree heads before contexts and models.n_ctx_train, pooling andn_ubatch. Run checks token counts againstn_ubatchbeforellama_encode. Empty question ids, non-string JSON keys and text containing U+0000 are rejected; native tokenization would cut U+0000 text at the NUL.decision_engine_e2e_test.dart, runner scenario and matrix rowdecision-model-smoke(--head-path,--config-path), and an integration test that stories15M is rejected as unsupported. The smoke fails when a requested GPU backend fell back to CPU, and checks that a config longer than the encoder's training context is rejected.guides/decision-models(with a typed-questions section), support matrix, README, CHANGELOGUnreleased, and the design doc's measurements.Production-readiness scope
laya-head.safetensors, or the officialmodel.safetensorswithconfigPath.ggml_graph_api_testanddecision_head_test, so the@Nativetwins execute there.capabilitiesForreports unsupported andloadthrowsLlamaUnsupportedException.opOffloadwiring, action head in the real-model smoke). CPU-mode context flags at the call site: #614.loadawaits its capability probe: #626.Completeness checklist
PR type guidance
LlamaUnsupportedException; happy-path, negative and lifecycle tests plus a real-model smoke. Examples come in later PRs of #604.Test Plan
All at head
d4f439b0ceebaab024e703b95e096245c7596a52, which mergesmain(ec6ad6f9b). CI ran on GitHub's merge ref65cf5e6, whose tree equals the head's.dart run tool/prepare_workspace.dartdart format --output=none --set-exit-if-changed .dart analyzedart test -p vm -j 1 --exclude-tags local-only: local (the audit's run) 2955 passed, 78 skipped. CI: Linux 3030 passed, 3 skipped; macOS 2955 passed, 78 skipped; Windows 2940 passed, 89 skipped.dart test -p chrome --exclude-tags local-only: CI, 1328 passed. Locally the audit ran the WebGPU backend, load-retry policy and decision Web files: 182 passed.run_local_e2e.dart --scenario decision-model-smokeon Q8_0, F32 and F16 backbones, CPU and Metal: 4/4 each.tool/testing/run_template_parity_suites.sh: exit 0. Detection parity skipped for missing fixtures (#618) and passed 74/74 on rerun.check_platform_boundaries.dart: OK. CI coverage: 83.62% ofliblines (threshold 70%).0848acdc6harness, re-run here, killed 104 of 112 runs. Survivors under High-risk regression review.Matrix Evidence
65cf5e6doc/decision_engine.md#measuredwere not re-run at this head.maingpt-6-astra, reasoning effort xhigh), operator-runAccuracy caveat: a broader 187-question review set found that the Q8_0 backbone can flip decisions. One noul moved from 0.694 to 0.457. A local F16 conversion matched F32 on Metal and flipped two near-ties (top-2 gaps 0.00015 and 0.0003) on CPU. The docs recommend an F32 backbone, or F16 on Metal, when answers must match Laya.
Review Notes
332bf37,0a42cdcand5c51654with 0 P1 while Codex was out of quota.gpt-6-astra, low reasoning effort) reviewed and auditedd9abfc3, then audited0848acdc6. A new Codex session (gpt-6-astra, xhigh) audited the exact head. See High-risk regression review.embed()aborts for ModernBERT GGUFs longer thann_ubatch.DisposeRequest.LlamaCppService.disposeleaks samplers and batches.'Metal', but ggml names itMTL.gguf-stop-sequencesrunner overrides the test's CPU default.d4f439b0ceebaab024e703b95e096245c7596a52: 25 checks pass, 5 skipped, all on this head.High-risk regression review
classify_high_risk_changes.dartovergit diff --name-only ec6ad6f9b...d4f439b0c(61 paths) reportsbackendRuntime,regressionPolicyandstructuredOutput.structuredOutputcomes only from thelib/llamadart.dartexport additions; no parser, template or grammar file changes.038dea663(core),332bf37fb(native),0a42cdcd1and5c5165401(earlier audit findings),fcdcd42a8(native cleanup),d9abfc306(typed keys, experimental label),0848acdc6(Codex findings),d4f439b0c(merge ofmain). The core commit had its own three-lens review as #605.main's mtmd chunk cancellation and WebGPU load-retry refactor. The PR diff is patch-identical to the0848acdc6diff; onlyllama_cpp_service.dartand the two changelogs differ from0848acdc6, bymain's own changes.gpt-6-astraat low reasoning effort (per the run logs):codex reviewand acodex execaudit ofd9abfc3, then acodex execaudit of0848acdc6given thed9abfc3report. Both audits: ACCEPT, 0 P1.0848acdc6fixed thed9abfc3findings exceptopOffloadcoverage; its audit left two P2s: the same-handle model swap (#626) andopOffloadcoverage (#625).codex execsession with no prior reports; Codex CLI 0.155.1,gpt-6-astra, reasoning effort xhigh (from its launch header). ACCEPT, 0 P1.prepare_workspace,dart analyze, the full VM suite (2955 passed, 78 skipped), 182 Chrome tests, the Q8_0 smoke on CPU and Metal, and the docs build with link validation.pythonJsonDumpsagainst Python 3.14 and 3.12json.dumpson 102,464 floats: no differences.main's merged WebGPU and mtmd files byte-identical to the base.opOffloadwiring (M30), and constant saturated action logits pass the real-model smoke (M35).high_risk_readiness.darton the exact pair exits 2 withunverifiedPrerequisites/externalPrerequisitesUnavailable: "Repository-local evidence is internally consistent, but auditor authentication, GitHub App publication, protected-environment provenance, and ruleset enforcement are not available. This is not operational merge readiness." That is the local ceiling indoc/high_risk_pre_merge_readiness.md.d4f439b0ceebaab024e703b95e096245c7596a52/ec6ad6f9bc11e44fe4502e29f685743e7ed436f7(main, also the merge base).0848acdc6harness re-run (112 runs, 104 killed) give kills in all 22 changed production files. Survivors:C22/C22Eare the sameopOffloadgap as M30.B02,WM01,EX02againstworker_messages_test, whose decision round-tripfcdcd42a8removed (worker_testalready covers it); re-targeted atworker_testordecision_engine_test, all three are killed.S05, in stream-parser code this PR does not change.decision_head_testkills it.