Repository navigation
[0.18] Add opt-in LFM3-Audio speech output to the TTS wrapper - #139
Merged
Merged
Conversation
llama.cpp v0.8.0 adds the LFM3-Audio generator (MTMD_GEN_AUDIO_TYPE_LFM3A). It cannot be allowlisted: its prompt must be the model's "tts" chat template, its frames come from the hidden state, and sampling the text head for an end-of-generation token cuts the speech short. Four new exports, additive only. llama_dart_tts_get_info and llama_dart_tts_init stay Qwen3-only and still report LFM3-Audio as UNKNOWN/UNSUPPORTED. llama_dart_tts_get_info_ext and llama_dart_tts_init_ext take a mask of the model types the caller handles; llama_dart_tts_supported_model_types is the build probe (Qwen3 only on v0.7.0); llama_dart_tts_start_with_voice selects female or male. The wrapper renders request.text with common_chat_get_tts_prompt, refuses speaker references and returns complete PCM. Text, VL, audio input and d1 need no wrapper change. docs/lfm3_support.md lists what each variant needs and what stays unwrapped. Refs #130
The MSVC lane builds Release, where NDEBUG stopped the new contract test. Refs #130
Owner
Author
|
Independent audit at exact head
Non-blocking P3s are tracked on #130:
Evidence: |
leehack
marked this pull request as ready for review
October 30, 2026 00:48
This was referenced Oct 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds opt-in LFM3-Audio and LFM3-Omni speech output to the TTS wrapper, for core llamadart 0.18. Refs #130.
Draft. Additive only: the 89 existing required exports keep their behaviour, and
llama_dart_tts_get_info/llama_dart_tts_initstill refuse LFM3-Audio. No submodule, tag, release or workflow-dispatch change. Onmain, it ships in the next scheduled native publish.What upstream v0.8.0 already supports
LLM_ARCH_LFM3(#31252)lfm3vlprojectorlfm3aencoder (#31257)MTMD_GEN_AUDIO_TYPE_LFM3ALLM_ARCH_LFM3,lfm3-d1(-omni)decision types (#31275)mtmd_helper_video_*(#31283), needs theffmpegbinaryNo piece needs an upstream patch.
Changes
llama_dart_tts_supported_model_types(build probe),llama_dart_tts_get_info_extandllama_dart_tts_init_ext(a mask of the model types the caller handles), andllama_dart_tts_start_with_voice. New enum values:LLAMA_DART_TTS_MODEL_TYPE_LFM3_AUDIO = 2andLLAMA_DART_TTS_CAPABILITY_VOICE.request.textis rendered through the model'sttschat template (common_chat_get_tts_prompt, with the language and voice). The text head is not sampled, so a text EOG cannot end the speech. Speaker references are refused, because upstream would ignore them. Output is complete PCM only.LLAMADART_MTMD_HAS_LFM3A_GEN. On v0.7.0 the probe is off, so the build reports Qwen3 only.tts_types_test(new), extendedtts_api_test, and a localllamadart_tts_lfm3_smokethat every wrapper-contract lane compiles.validate_exports.pynow requires 93 exports.docs/lfm3_support.md, README,docs/v080_qualification.md, AGENTS.md.Local evidence (macOS arm64)
validate_exports.pyv0.8.0 / v0.7.0tts_api_testfailed to link on the four missing exports before the implementationllamadart_tts_lfm3_smoke, LFM3-Audio-2B-GGUF02355c1a: Q4_K_M (9a773289…) + mmproj Q8_0 (2b6648b1…), 1.4 GBmax_framestruncatestts_smoke(Metal), regression controlwindows-11-vs2026-armlabelNotable kill: restoring Qwen3's text-head sampling and EOG stop for LFM3 (the allowlist-only change) cuts the speech, and the transcript drops to "Hello,".
Survivors:
ttstemplate check: no local model lacks the template. Without the check,startfails at render withUPSTREAM_ERROR.Not covered
lfm3) for text decoding and audio input.mtmd_audio_stream_*), and video.ttsprompt is tokenized with special-token parsing, as in upstreamllama-tts, so control tokens inrequest.textare parsed as tokens.