[leehack]

Summary

For #599. A llama.cpp cancel only sets the per-generation cancel byte, and the worker read it only at the top of the token loop. An image or audio prompt was ingested by one mtmd_helper_eval_chunks call, so a cancel during the ingest waited for every remaining chunk.

The multimodal ingest now runs that chunk loop in Dart (lib/src/backends/llama_cpp/mtmd_chunk_eval.dart) and reads the byte:

• before each chunk;
• after an image or audio chunk's mtmd_encode_chunk, before its embedding decode.

Text chunks still go through mtmd_helper_eval_chunk_single, with new_n_past seeded first because its text branch adds to it. Image and audio chunks call mtmd_encode_chunk, mtmd_get_output_embd and mtmd_helper_decode_image_chunk with the arguments mtmd_helper_eval_chunk_single passes them. On a cancel the loop returns with n_past at the last complete chunk. generate then continues through the existing path: llama_get_logits synchronizes, and the token loop's first check ends the generation with no tokens. A nonzero native result still throws the existing error, which now also names the failed call and chunk, for example Multimodal prompt evaluation failed: 1 (failed to decode audio chunk 4).

• No abort callback is registered. No native code calls into Dart, so nothing depends on which thread a backend polls on.
• Windows split bundles resolve mtmd through _MtmdApi. It now also looks up the seven chunk-level symbols as one all-or-nothing MtmdChunkEvalApi. If any is missing, that fallback keeps the single mtmd_helper_eval_chunks call. Primary or fallback is chosen from _mtmdPrimarySymbolsUnavailable, which the preceding mtmd_tokenize has already settled, so a chunk is never evaluated twice.
• Chunk type is read through a new raw binding in llama_cpp_raw_bindings.dart. The generated enum wrapper throws on unknown values; the raw value sends an unknown type to mtmd_helper_eval_chunk_single, as upstream does.
• Text prompts are unchanged. No text-path code is edited, and text prefill stays uncancellable.

Not included: an early return after a cancelled ingest. Returning from generate straight after the ingest skips the llama_get_logits synchronize. Built that way, the e2e below saw a generate that decoded 172 prompt tokens report promptEvalTokens 1. With a synchronize added, deleting the return turned no test red, because the token loop's first check already ends the generation. So the return was dropped.

Measured

Apple M4 Max, macOS arm64, llamadart-native v0.4.1. Qwen3-ASR-0.6B Q8_0 with its Q8_0 mmproj, jfk.wav (11 s). Stock is aed0bd2b5; this PR is 85ccb4fd8, and later commits change only tests, test-only service methods and dartdoc. SpeechToTextTask.cancel() is called f × (previous full transcription) after transcribe() returns; at f=0 the harness awaits a zero-length timer and then cancels. Both heavy-work locks were held, and stock and PR runs alternated. The batch first waited for the 1-min loadavg to drop below 6. Recorded after each cancel, it was then 6.18-10.40. GPU utilization read 0% before every run.

A cancel issued synchronously right after await transcribe() ended the task cancelled in 0.58-1.04 ms on stock and on a14ffbc80 (n=4 each, CPU). cancelGeneration() issued synchronously after listen() is dropped on both: a stories15M generate() gave the uncancelled output (n=3 each, #602).

Cancel latency in ms, as min-max (median):

backend	lead f	n	stock (ms)	this PR (ms)
CPU	0	6	393.9-405.2 (399.0)	6.1-10.0 (6.9)
CPU	0.1	6	332.7-347.1 (339.9)	106.6-109.6 (108.0)
CPU	0.3	6	216.6-221.5 (218.8)	80.3-90.1 (87.7)
CPU	0.5 (#594's lead)	6	100.7-137.3 (103.7)	39.1-51.0 (45.8)
Metal	0	6	56.7-58.8 (57.1)	6.1-8.0 (7.9)
Metal	0.05	6	47.2-51.3 (48.5)	14.0-14.8 (14.5)
Metal	0.15	6	30.8-33.3 (32.7)	13.6-17.2 (16.6)
Metal	0.3	6	7.3-10.0 (8.6)	7.8-11.1 (9.4)

• All 96 cancels ended cancelled.
• 16 engine.create streams were cancelled with cancelGeneration(), n=2 per cell, and all 16 ended with onDone and no error. CPU f 0.1/0.3: stock 336.3-340.0 / 222.5-225.4 ms, this PR 105.7-112.3 / 85.3-89.8 ms. Metal f 0.05/0.15: stock 47.1-47.4 / 31.1-33.9 ms, this PR 13.1-14.3 / 13.4-16.5 ms.
• Service-level generate with the byte already set, n=4 per cell, loadavg 7.60-9.03:
  • CPU: stock 426.4-433.2 ms, this PR 22.8-27.9 ms.
  • Metal: stock 75.7-87.1 ms, this PR 6.6-18.8 ms.
  • promptEvalTokens: stock 172, this PR 1. Both returned empty output with no error.
• Uncancelled output is unchanged.
  • All 160 full transcriptions (40 per tree and backend) returned the same text. Median CPU 581.4 → 581.5 ms, Metal 155.9 → 155.6 ms.
  • validation.dart speech --pack stt, CPU and Metal, loadavg 8.29-11.76: functional_pass on both trees. Apart from timings (elapsed_ms, first_partial_ms, real_time_factor and the preparation checksum times), every field is identical between stock and this PR, including all 13 checks, the 4 edge fixtures and the transcripts.
  • Greedy 64-token generation on CPU and Metal gave byte-identical output (SHA-1) on both trees for each case: stories15M with two prompts, a two-turn Qwen2.5-0.5B-Instruct chat, and SmolVLM-500M-Instruct describing website/static/img/social-card.png, twice. Loadavg 11.32-12.26.

What remains is the native call in flight when the byte is set. The CPU leads were 58.8-62.3 ms (f=0.1), 174.5-180.6 ms (f=0.3) and 289.1-300.1 ms (f=0.5). In one stock CPU run at native log level info, the two audio encodes took 165 and 74 ms, and their embedding decodes 96 and 49 ms. At Metal f=0.3 the two ranges overlap.

Production-readiness scope

• User-facing scope: cancelling a llama.cpp native generation whose prompt carries images or audio (SpeechToTextTask.cancel(), cancelGeneration()) now stops at the next chunk boundary, or between a media chunk's encode and its decode.
• Supported platforms/paths: llama.cpp native backend on every native platform, including the Windows split-bundle mtmd fallback.
• Unsupported or intentionally unavailable paths:
  • The call in flight is not interrupted: one clip encode (llamadart-native#89), one embedding decode or one text decode.
  • Text-prompt prefill is unchanged and uncancellable, as before.
  • Web and LiteRT-LM are not touched: llama_cpp_service.dart is native-only.
• Out of scope / follow-ups (filed):
  • llamadart-native#88: no exported abort callback reads a cancel byte, so llamadart cannot stop an in-flight CPU llama_decode.
  • llamadart-native#89: no exported cb_eval stops an in-flight STT or vision clip encode on a cancel byte.
  • #601: a failed prompt decode leaves cachedPromptTokens stale. It predates this PR.
  • #602: cancelGeneration() right after listen() is dropped. It predates this PR, which does not fix it.
  • #603: multimodal generate primes the repeat penalty from an unwritten token buffer. It predates this PR.
• Behaviour differences:
  • With the chunk-level functions, media chunks skip mtmd_helper_eval_chunk_single and no chunk goes through mtmd_helper_eval_chunks. So these lines of tools/mtmd/mtmd-helper.cpp at b29c606e28 are no longer printed:
    • INFO, per image or audio chunk: encoding %s slice... (:249) and %s slice encoded in N ms (:258).
    • ERROR, on a failed chunk: failed to encode %s slice (:253), the second failed to decode %s (:263) and failed to eval chunk %zu (:295).
    • WARN no chunks to eval (:285), for an empty chunk list.
  • The thrown error now carries those facts after the native result: the failed call (encode, decode or eval), the media type for encode and decode, and the chunk index. Measured with a 256-token context, CPU, native log level warn:
    • Qwen3-ASR, two audio parts: stock printed failed to decode audio twice and failed to eval chunk 4, and threw Multimodal prompt evaluation failed: 1. The active context window …. This PR printed failed to decode audio once (:179) and threw Multimodal prompt evaluation failed: 1 (failed to decode audio chunk 4). The active context window ….
    • SmolVLM-500M, six images: the same, with image and chunk 7.
    • Qwen3-ASR with a text suffix that overflows: both printed failed to decode text (:238). Stock also printed failed to eval chunk 3; this PR threw … 1 (failed to eval chunk 3). ….
    • The fallback without chunk-level functions printed every stock line and threw the stock message.
    • No encode failure was triggered with a real model; the unit tests cover its message.
    • Through LlamaEngine at its default native log level (none), neither tree printed a native line. The LlamaInferenceException read Generation failed (Exception: Multimodal prompt evaluation failed: 1 (failed to decode audio chunk 4). …) on this PR, against … failed: 1. … on stock.
  • At native log level info, stock printed both slice lines for each audio chunk. This PR printed neither, and still printed decoding audio batch 1/1, … and audio decoded (batch 1/1) in N ms.
  • A cancelled ingest no longer counts the chunks it skipped in promptEvalTokens: with the byte already set, this PR reports 1 and stock 172.

Completeness checklist

• Declared scope is fully implemented, or explicitly reduced above.
• Unsupported platform/option combinations fail loudly with actionable diagnostics or are clearly documented as unavailable.
• Public API docs, README/website docs, examples, support matrices, and changelog entries are updated where relevant. There is no public API change; ## Unreleased entries are added to CHANGELOG.md and website/docs/changelog/recent-releases.md.
• Regression coverage covers the original issue plus key negative/version-skew paths where relevant.
• Security/privacy review completed: no secrets, bearer tokens, signed URLs, or raw secret-bearing paths leak through logs, cache keys, metadata, errors, or snapshots.
• Follow-up work that is useful but not required for this PR is tracked in GitHub Issues, or explicitly marked "None" above. Linked under follow-ups.

PR type guidance

• Bugfix PR: the root cause is above. Regression tests:
  • test/unit/backends/llama_cpp/mtmd_chunk_eval_test.dart (VM, no model): 13 tests drive the loop through a fake that applies upstream's new_n_past rules; one checks that MtmdChunkEvalApi.tryLoad returns null without the symbols. The failure tests also check the reported call, media type, chunk index and native result.
  • test/integration/backends/llama_cpp/native_symbol_integration_test.dart checks one library per platform: the split mtmd library on Linux (libmtmd*.so) and Windows (mtmd*.dll), and libllamadart.dylib on macOS, which has no split mtmd library. The library must export the seven symbols, and _MtmdApi.tryLoad(library), the call _resolveMtmdFallbackApi makes, must return chunk-level functions. It runs on CI's Linux, macOS and Windows jobs.
  • test/e2e/backends/speech_to_text_e2e_test.dart (local-only): two tests, each run on three routes.
    • A generate whose byte is already set yields nothing, and the next request's output matches the first. The primary route and the wrapper fallback decode no prompt token (promptEvalTokens ≤ 1). The fallback without chunk-level functions ingests all 172 tokens, as before.
    • Two audio parts in a 256-token context throw Multimodal prompt evaluation failed: 1 (failed to decode audio chunk 4). … on the primary route and the wrapper fallback, and the stock Multimodal prompt evaluation failed: 1. … without chunk-level functions.
    • debugUseWrapperMtmdFallbackForTesting forces the fallback. With chunk-level functions it calls _MtmdApi.tryLoad(library) with the default argument, as _resolveMtmdFallbackApi does. In an instrumented run on macOS, the seam opened .dart_tool/lib/libllamadart.dylib, mtmd_tokenize and the ingest went through that fallback, and the mtmd library lookup never ran. So the new fallback typedefs run end to end.

Test Plan

• dart run tool/prepare_workspace.dart: tracked files unchanged.
• dart format --output=none --set-exit-if-changed .: 681 files, 0 changed.
• dart analyze --fatal-infos: no issues.
• dart test -p vm -j 1 --exclude-tags local-only (macOS arm64, GGML_METAL_DEVICES=0): 2524 passed, 78 skipped, 0 failed. Coverage check: 81.48% line coverage (70% threshold).
• dart test -p chrome --exclude-tags local-only: run by CI (Test Web (Chrome)), not locally; the changed library is native-only.
• Other targeted/local-only validation:
  • run_local_e2e.dart --scenario speech-to-text-smoke: 8/8, including the new e2e tests.
  • test/e2e/backends/cancel_dispose_e2e_test.dart (local-only): 2 passed, 1 skipped.
  • The cancel tests in engine_integration_test.dart and stop_sequences_test.dart passed in the VM run.
  • validation.dart speech --pack stt on CPU and Metal.

Mutation proofs. Each was applied alone and reverted, and each turned tests red:

Mutation	Red tests
delete the check before each chunk	3 unit
delete the check between encode and decode	2 unit
do not seed new_n_past before a text chunk	7 unit
logits_last on every chunk	10 unit
send unknown chunk types down the media path	1 unit
do not return a nonzero native result	6 unit
decode after a failed encode	2 unit
tryLoad does not catch a missing symbol	1 unit
do not write new_n_past on an early stop	1 unit
do not advance n_past	8 unit
report a failed decode as an encode	2 unit
report a failed encode as an eval	2 unit
report chunk 0 for every failure	4 unit
swap the image and audio names	4 unit
name a media type for an eval failure	2 unit
report native result 1 for every failure	3 unit
misspell mtmd_encode_chunk in tryLoad	integration Verify mtmd chunk-level eval symbols are resolvable
_MtmdApi.tryLoad defaults loadChunkEval to false	integration Verify mtmd chunk-level eval symbols are resolvable; e2e fallback: cancel (172) and error message
primary route keeps mtmd_helper_eval_chunks	e2e primary: cancel (promptEvalTokens 172) and error message
generate passes a fresh zero byte to the ingest	e2e primary and fallback cancel (172)
fallback ignores its chunk-level functions	e2e fallback: cancel (172) and error message
error message drops the failed chunk	e2e primary and fallback error message

Matrix Evidence

Matrix row	Scope covered	Platform / model / backend	Result	Evidence / notes
static-format-analyze	format, analyzer	macOS arm64, Flutter 3.47.1; CI	PASS	local and Analyze & Lint
root-vm	VM unit/integration	macOS arm64; CI Linux, macOS, Windows	PASS	2524 pass / 78 skip locally; CI Linux 2599 / 3, macOS 2524 / 78, Windows 2511 / 87
root-chrome	browser suite	CI Chrome	PASS	Test Web (Chrome); native-only change
coverage-lib	≥70% lib coverage	macOS arm64	PASS	81.48%
validation-speech	locked Qwen3-ASR stt pack	macOS arm64, CPU and Metal	PASS	functional_pass both; all non-timing fields identical to stock aed0bd2b5
speech-to-text-smoke	Qwen3-ASR file/bytes, cancel, recovery, preset-cancel and overflow-error routes	macOS arm64, CPU	PASS	8/8 via run_local_e2e.dart
validation-speech-stt	locked Qwen3-ASR stt pack with #594's 500 ms cancel-latency budget	Linux x64: GCE e2-standard-8, AMD EPYC 7B12, 8 vCPUs; CPU	PASS	#594 head 3ec86add0 with this PR's 85ccb4fd8 merged. Two runs, 20/20 PASS each. cancel_latency_bound worst 137.568 and 134.186 ms against 500. The run without this PR (#594 at a992f1711) was on an Intel Broadwell VM: 628.652-703.982 ms, FAIL. So the before/after comparison crosses CPUs. Later commits change only tests, test-only service methods and dartdoc.
macos-arm64-runtime-smoke	llama.cpp CPU and Metal	M4 Max; Qwen3-ASR, stories15M, Qwen2.5-0.5B, SmolVLM-500M	PASS (llama.cpp part)	measurements above; LiteRT-LM not touched
linux-x64-ci-runtime	Linux VM tests incl. symbol check	CI ubuntu	PASS	Test Linux VM; real-model row above
windows-x64-ci-runtime	Windows VM tests incl. the mtmd library symbol check	CI windows-latest	PASS	Test Native (windows-latest)
native-prompt-reuse-parity	text prompt reuse	CI	PASS	text path unchanged
docs-site	changelog page	CI	PASS	Docs Build Check
high-risk-exact-head-independent-qa	independent exact-head review	head a14ffbc80, base aed0bd2b5	PASS	see below
all other rows	—	—	N/A	no change to Web/WebGPU, LiteRT-LM, templates/grammar/parsers, hook/bundles, release metadata, examples or device flows

Review Notes

• Independent review status: accepted at a14ffbc80bbd7c7edff20cc0bf72e35547081ecd; see below.
• CI status / head SHA: all green at a14ffbc80bbd7c7edff20cc0bf72e35547081ecd: 23 passed, 5 skipped, 0 failed. Test Linux VM, Test Native (macos-latest) and Test Native (windows-latest) ran and passed the unit tests and the chunk-level symbol check, which now includes the fallback loader default.

High-risk regression review

• Classification: high-risk. git diff --name-only aed0bd2b5...HEAD | dart run tool/testing/classify_high_risk_changes.dart reports surface backendRuntime; the only applicable row is high-risk-exact-head-independent-qa.
• Implementation task: 9b3e3dabc, b429093e9, 0f780a337, 85ccb4fd8, 290405b40, a14ffbc80
• Independent blocking QA task: accepted at a14ffbc80 / aed0bd2b5: 0 blocking findings. Both reviews of this head were by fresh reviewers from the maintainer's operator session with no part in writing this PR.
  • The full audit found no P1 and rejected on one body item: it said the in-flight clip encode had no upstream hook, but mtmd_context_params.cb_eval is one. The body now links llamadart-native#89 for it.
  • A scoped re-review confirmed that item resolved and accepted.
  • Earlier, the audit of 85ccb4fd8 found no P1 and rejected on one item: no test pinned _MtmdApi.tryLoad's loadChunkEval default, which gives the Windows fallback its chunk-level functions. 290405b40 pins it.
• Exact head / current base: a14ffbc80bbd7c7edff20cc0bf72e35547081ecd / aed0bd2b52c0f41b4b0baf734d5b456878000629 (GitHub baseRefOid). main is one commit ahead, at 49ee24594 (#594's squash merge), which changes only the validation package and its doc; this branch merges onto it cleanly.
• Production-branch deletion, bypass, or miswire proof: the 22 mutations above. Four e2e rows mutate the production call sites in _ingestMultimodalPrompt and generate; the loadChunkEval row mutates the default _resolveMtmdFallbackApi relies on. Re-run at a14ffbc80 for the readiness evidence, each alone and reverted: the check before each chunk (3 unit red), the check between encode and decode (2 unit), the fresh zero byte (2 e2e), and the loadChunkEval default against the integration test only (red). Each matches the table. A new one, the raw mtmd_input_chunk_get_type binding naming a missing symbol, turned 4 e2e tests red: both primary-route tests and both public-API transcriptions. The fallback routes stayed green.
• Readiness evaluator: dart run tool/testing/high_risk_readiness.dart, with repository, PR, author, head and base from a fresh GitHub API query: exit 2, unverifiedPrerequisites (externalPrerequisitesUnavailable). That is the best local result: the evidence is internally consistent, and the external prerequisites do not exist. With main (49ee24594) as the base it rejects with baseMismatch, because that commit is not an ancestor of the head.
• Affected-family real-model/artifact evidence: Qwen3-ASR-0.6B Q8_0 with mmproj (audio) and SmolVLM-500M-Instruct with mmproj (image), CPU and Metal, v0.4.1 macOS arm64 bundle, on the cancel, uncancelled and failed-chunk paths.
• Explicit unavailable-family or other N/A evidence:
  • Apple companion XCFramework, iOS and Android bundles were not inspected. The seven symbols are MTMD_API, like mtmd_tokenize, at the pin. A primary bundle that exported mtmd_tokenize but not one of the seven would fail the generation with the binding's ArgumentError instead of falling back.
  • Windows is covered only by CI's symbol check.
  • Linux x64 CPU latency was measured only with #594 merged, on AMD EPYC; see Matrix Evidence.
• Known PR-caused P1 regressions: 0
• Unresolved review threads: 0

[github-actions]

Chat app preview removed for leehack/llamadart-chat-pr-600.