Skip to content

[0.18] Add opt-in LFM3-Audio speech output to the TTS wrapper - #139

Merged
leehack merged 3 commits into
mainfrom
feat/130-lfm3
Oct 30, 2026
Merged

leehack merged 3 commits into
mainfrom
feat/130-lfm3

Conversation

@leehack

@leehack leehack commented Oct 29, 2026

Copy link
Copy Markdown
Owner

Adds opt-in LFM3-Audio and LFM3-Omni speech output to the TTS wrapper, for core llamadart 0.18. Refs #130.

Draft. Additive only: the 89 existing required exports keep their behaviour, and llama_dart_tts_get_info / llama_dart_tts_init still refuse LFM3-Audio. No submodule, tag, release or workflow-dispatch change. On main, it ships in the next scheduled native publish.

What upstream v0.8.0 already supports

Variant Upstream Wrapper change
Text (LFM3-1.7B) LLM_ARCH_LFM3 (#31252) None
VL (LFM3-VL-2B) lfm3vl projector None (generic mtmd exports)
Audio input / ASR (Audio, Omni) lfm3a encoder (#31257) None
Speech output (Audio, Omni) MTMD_GEN_AUDIO_TYPE_LFM3A This PR
d1 (d1-2B, d1-Omni-2B) non-causal LLM_ARCH_LFM3, lfm3-d1(-omni) decision types (#31275) None (scoring is the consumer's; no KV memory, so the whole prompt goes in one batch)
Video mtmd_helper_video_* (#31283), needs the ffmpeg binary Not wrapped

No piece needs an upstream patch.

Changes

  • New exports, in their own block: llama_dart_tts_supported_model_types (build probe), llama_dart_tts_get_info_ext and llama_dart_tts_init_ext (a mask of the model types the caller handles), and llama_dart_tts_start_with_voice. New enum values: LLAMA_DART_TTS_MODEL_TYPE_LFM3_AUDIO = 2 and LLAMA_DART_TTS_CAPABILITY_VOICE.
  • The new generator needs more than an allowlist entry. request.text is rendered through the model's tts chat template (common_chat_get_tts_prompt, with the language and voice). The text head is not sampled, so a text EOG cannot end the speech. Speaker references are refused, because upstream would ignore them. Output is complete PCM only.
  • CMake probe LLAMADART_MTMD_HAS_LFM3A_GEN. On v0.7.0 the probe is off, so the build reports Qwen3 only.
  • Tests: tts_types_test (new), extended tts_api_test, and a local llamadart_tts_lfm3_smoke that every wrapper-contract lane compiles. validate_exports.py now requires 93 exports.
  • Docs: docs/lfm3_support.md, README, docs/v080_qualification.md, AGENTS.md.

Local evidence (macOS arm64)

Check Result
Wrapper contract build (Debug, CI flags) + ctest, v0.8.0 58/58
Same on pinned v0.7.0 (LFM3A probe off) 58/58; LFM3 smoke exits with "this build cannot drive LFM3-Audio"
validate_exports.py v0.8.0 / v0.7.0 93 / 93
Failing-first tts_api_test failed to link on the four missing exports before the implementation
llamadart_tts_lfm3_smoke, LFM3-Audio-2B-GGUF 02355c1a: Q4_K_M (9a773289…) + mmproj Q8_0 (2b6648b1…), 1.4 GB PASS on Metal and on CPU. The ASR round trip returns "Hello, how are you today?". Seed repeats after cancel and reuse; voice, language and seed change the PCM; cancel lands at frame 3; max_frames truncates
Qwen3-TTS 1.7B Q4_K_M tts_smoke (Metal), regression control PASS
Mutation (16 mutants, unit tests + smoke) 14 killed, 2 survived (below)
Python suites / actionlint 279 OK + 22 OK; only the known windows-11-vs2026-arm label

Notable kill: restoring Qwen3's text-head sampling and EOG stop for LFM3 (the allowlist-only change) cuts the speech, and the transcript drops to "Hello,".

Survivors:

  • The init-time tts template check: no local model lacks the template. Without the check, start fails at render with UPSTREAM_ERROR.
  • The build-mask term in the accepted-type check: equivalent and redundant, so it was removed.

Not covered

  • LFM3-VL, LFM3-1.7B, Omni and d1 were not run. The smoke uses the LFM3-Audio backbone (arch lfm3) for text decoding and audio input.
  • Not exposed upstream features: streaming PCM, Omni mixed text/audio and sequence continuation, live ASR (mtmd_audio_stream_*), and video.
  • The tts prompt is tokenized with special-token parsing, as in upstream llama-tts, so control tokens in request.text are parsed as tokens.
  • Not qualified: Linux, Windows or Android devices, and the consumer bindings.
  • High-risk independent audit not run.

llama.cpp v0.8.0 adds the LFM3-Audio generator (MTMD_GEN_AUDIO_TYPE_LFM3A).
It cannot be allowlisted: its prompt must be the model's "tts" chat
template, its frames come from the hidden state, and sampling the text
head for an end-of-generation token cuts the speech short.

Four new exports, additive only. llama_dart_tts_get_info and
llama_dart_tts_init stay Qwen3-only and still report LFM3-Audio as
UNKNOWN/UNSUPPORTED. llama_dart_tts_get_info_ext and llama_dart_tts_init_ext
take a mask of the model types the caller handles;
llama_dart_tts_supported_model_types is the build probe (Qwen3 only on
v0.7.0); llama_dart_tts_start_with_voice selects female or male. The
wrapper renders request.text with common_chat_get_tts_prompt, refuses
speaker references and returns complete PCM.

Text, VL, audio input and d1 need no wrapper change. docs/lfm3_support.md
lists what each variant needs and what stays unwrapped.

Refs #130
The MSVC lane builds Release, where NDEBUG stopped the new contract test.

Refs #130
@leehack

leehack commented Oct 30, 2026

Copy link
Copy Markdown
Owner Author

Independent audit at exact head a378dce: no P1 or P2 findings, and 64/64 checks pass. merge-tree against current main 52303ee (#140) is clean, and the merged tree passes CTest 58/58, the LFM3 smoke and the Qwen3 smoke.

  • Additive only: exactly 4 new symbols, no struct changes, existing enum values unchanged, and TTS API version 1.
  • Legacy calls unchanged: get_info and init still refuse LFM3 with type 255 and capabilities 0. Core 0.17 bindings compare model_type as an int, so companion 0.17.0 and core 0.17.1 are safe.
  • Qwen3-TTS unchanged: its PCM is byte-identical between base and head on v0.8.0 and v0.7.0.
  • Prompt injection: passing control tokens in request.text only changes the speech. The text goes in as a Jinja variable, so it cannot inject into the template.

Non-blocking P3s are tracked on #130:

  • capability-assert gaps (mutants M08, M12, M14);
  • an unknown LFM3 language is not rejected;
  • get_info_ext reports OK when the model has no tts template;
  • LFM3-Omni is claimed but untested;
  • docs/v080_qualification.md still says 89 exports.

Evidence: llamadart-evidence/native-pr-139/.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant