Skip to content

fix(llama-cpp): keep decision heads on CPU for CPU placement - #1176

Merged
leehack merged 2 commits into
mainfrom
codex/015-o-decision-head-cpu
Oct 17, 2026
Merged

leehack merged 2 commits into
mainfrom
codex/015-o-decision-head-cpu

Conversation

@leehack

@leehack leehack commented Oct 17, 2026 •

Copy link
Copy Markdown
Owner

Summary

Production-readiness scope

  • User-facing scope: A native decision head follows CPU placement when single-device mode selects no GPU. CPU context configuration disables operation offload and the head uses the CPU runtime/device name.
  • Supported platforms/paths: Native llama.cpp decision-head loading, including attachment through DecisionEngine; deterministic fake GPU registry and synthetic native ModernBERT/head checks on macOS. Existing native experimental support levels remain unchanged.
  • Unsupported or intentionally unavailable paths: LiteRT-LM decision models remain unsupported with LlamaUnsupportedException; explicit ComputeDevice.gpu plus CPU-only placement keeps its existing validation error. Web placement is unchanged. Physical iOS/simulator GPU execution was not qualified here.
  • Out of scope / follow-ups: Integration with the newer main, fresh checks after integration, and independent exact-head audit PENDING before ready. The broader simulator Metal issue remains iOS simulator (arm64): explicit llama.cpp Metal inference returns garbage #950; this change routes only the CPU-selected decision-head case.

Completeness checklist

  • Declared scope is fully implemented, or explicitly reduced above.
  • Unsupported platform/option combinations fail loudly with actionable diagnostics or are clearly documented as unavailable.
  • Public API docs, README/website docs, examples, support matrices, and changelog entries are updated where relevant. Placement prose is updated in the existing support-matrix guide; the example already prints loaded.info.deviceName and needs no code change.
  • Regression coverage covers the original issue plus key negative/version-skew paths where relevant. No runtime version floor changed.
  • Security/privacy review completed: no new logs, metadata, credential handling or secret-bearing output.
  • Follow-up work is identified above; no additional implementation follow-up is required for the declared placement scope.

PR type guidance

  • Feature PR: N/A: no new public API.
  • Bugfix PR: Head selection used stored GPU backend/layer metadata while ignoring the load's CPU-only single-device placement. A GPU-less fake registry reproduced expected null; got 12 before the fix. Both head CPU detection and GPU-device selection now consult selectsNoGpuDevice. Native-only affected path; Web and LiteRT-LM unchanged.
  • Docs-only PR: N/A: native runtime behavior changes. The existing test(llama-cpp): close capability-probe and placement audit gaps #1174 contract tests intentionally change because the maintainer approved restoring CPU placement.

Test Plan

  • dart run tool/prepare_workspace.dart (clean checkout; no tracked generated changes).
  • dart format --output=none --set-exit-if-changed . (1,140 files; 0 changed).
  • dart analyze (no issues, including zero infos in lib/).
  • dart test -p vm -j 1 --exclude-tags local-only --reporter expanded — 08:10 +6987 ~124: All tests passed!; full retry outside sandbox after two unchanged process-liveness tests were denied ps.
  • dart test -p chrome --exclude-tags local-only --reporter expanded — 01:38 +3332: All tests passed!; sandboxed Chrome aborted on launch, and the unchanged full command passed outside the sandbox.
  • Other targeted validation: platform-boundary check, 368 targeted tests (2 model-dependent skips), 22 restored regression tests, and all 3 deliberate mutations caught. ./tool/docs/build_site.sh --port 43839 passed with 2,292 pages and no broken internal links.
  • N/A: template parity runner (no template/parser/grammar changes); real Laya model/device smoke (weights/device execution excluded by task). No standalone link validator was run.

Matrix Evidence

Matrix row Scope covered Platform / model / backend Result Evidence / notes
static-format-analyze format, analyzer, web/native import boundaries pinned Flutter 3.47.1 / Dart, macOS PASS Format 1,140 files unchanged; analyzer no issues; boundary check OK.
root-vm native-safe unit/integration coverage on the Dart VM macOS VM; isolated TMPDIR; GGML_METAL_DEVICES=0; existing CI model directory PASS 08:10 +6987 ~124: All tests passed!
root-chrome browser-compatible unit/integration coverage headless Chrome / macOS PASS 01:38 +3332: All tests passed!; retry after sandbox launch failure.
high-risk-exact-head-independent-qa independent blocking-only review of exact head/current base, live threads, production call sites, adversarial negatives, and tests sensitive to production-branch deletion, bypass, or miswiring Native backend routing PASS Independent audit PASS at 3b1a5129546cfb93d72165d6ed0b33a43d8a9c4c against 591dfd844: no P1/P2. Real Metal probe: none/-1 and none/-2 heads now run on CPU with confidence identical to pure CPU; explicit mainGpu and layer split stay on MTL0. Full VM +6990 ~124 (GGML_METAL_DEVICES=0); decision suites with Metal visible +928; decision-model-smoke E2E passes on Metal and CPU; 8/10 mutations killed (2 survivors are redundant guards, P3).
docs-site website build, docs metadata, generated docs routes Jaspr / port 43839 PASS 2,292 pages, 51 sitemap entries, zero broken internal links; search index generated. Standalone validate_links.sh omitted per task.
decision-model-smoke real native ModernBERT GGUF plus Laya decision head parity and CPU placement Laya / ModernBERT / native llama.cpp N/A Real weights prohibited for this task. Fake mixed GPU registry plus generated ModernBERT and seeded synthetic head verify routing and reported CPU name; not a real-model numerical-parity claim.

Review Notes

  • Independent review status: Independent audit PASS at 3b1a5129546cfb93d72165d6ed0b33a43d8a9c4c against 591dfd844: no P1/P2. Real Metal probe: none/-1 and none/-2 heads now run on CPU with confidence identical to pure CPU; explicit mainGpu and layer split stay on MTL0. Full VM +6990 ~124 (GGML_METAL_DEVICES=0); decision suites with Metal visible +928; decision-model-smoke E2E passes on Metal and CPU; 8/10 mutations killed (2 survivors are redundant guards, P3).
  • CI status / head SHA: not pushed, so external CI is N/A. Local checks cover 3b1a5129546cfb93d72165d6ed0b33a43d8a9c4c (d515a65 + merge of main); current local origin/main is 591dfd844005b7582ad25e4018d9db88ed68f6ba.

High-risk regression review

  • Classification: high-risk; surface backendRuntime.
  • Implementation task: Task 015-O, branch codex/015-o-decision-head-cpu.
  • Independent blocking QA task: PASS at exact head 3b1a5129546cfb93d72165d6ed0b33a43d8a9c4c (fresh auditor).
  • Exact head / current base: 3b1a5129546cfb93d72165d6ed0b33a43d8a9c4c (d515a65 + merge of main) / 591dfd844005b7582ad25e4018d9db88ed68f6ba. Tested branch base is c5dc87136a73965c427599d1438548478d4bf15b; current main advanced during validation. Integrate with main and rerun checks before independent audit; refresh the remote-tracking base then.
  • Production-branch deletion, bypass, or miswire proof: Author controls only: remove the decisionHeadRunsOnCpu placement condition -> 2 failures; remove the decisionHeadDeviceIndex guard -> fake-registry failure; omit forwarding splitMode -> fake-registry failure. All run with GPU devices disabled, all restored, and 22 regression cases then passed. Independent assessment PENDING.
  • Affected-family real-model/artifact evidence: Generated one-layer ModernBERT GGUF plus seeded synthetic Laya head loads and reports CPU; fake registry contains Vulkan and two Metal devices. These are deterministic routing artifacts, not real Laya weights or device numerical qualification.
  • Explicit unavailable-family or other N/A evidence: Real Laya/ModernBERT model/head parity and physical iOS/simulator execution unavailable within the task's no-weights/device scope. Templates, parsing, streaming and grammar are unchanged; independent reviewer must confirm those exclusions. Readiness evaluator deferred until a PR and independent audit exist; no accepted audit evidence is fabricated.
  • Known PR-caused P1 regressions: 0 (independent audit).
  • Unresolved review threads: 0 (live query).

…x/015-o-decision-head-cpu

# Conflicts:
#	CHANGELOG.md
#	website/docs/changelog/recent-releases.md
@leehack
leehack marked this pull request as ready for review October 17, 2026 14:53
@leehack
leehack merged commit 80d90cc into main Oct 17, 2026
31 checks passed
@leehack
leehack deleted the codex/015-o-decision-head-cpu branch October 17, 2026 14:53
@github-actions

Copy link
Copy Markdown
Contributor

Chat app preview removed for leehack/llamadart-chat-pr-1176.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant