[ngxson]

Overview

Follow-up #26254

Example:

curl -X POST http://127.0.0.1:9931/tts \
    -H "Content-Type: application/json" \
    -d '{"input": "Hello, this is a test."}' \
    -o output.wav

With a speaker reference uploaded as a file (multipart/form-data), instead of base64-encoding it into the JSON body:

curl -X POST http://127.0.0.1:9931/tts \
    -F "input=Hello, this is a test." \
    -F "speaker_ref=@/path/to/speaker-reference.wav;type=audio/wav" \
    -o output.wav

---

• Add support for streaming output (mtmd helper)
• Add /tts endpoint to server
• Add docs
• Throw error if model cannot be used for chat (mtmd_helper_model_can_chat)
• Update qwen3-tts docs to specify about ref voice requirements
• Update router mode to reflect if model is TTS-capable

Next PRs:

• Add --tts-speaker-dir that allows specify voice name via API, without including them as base64
• Support OAI-compat TTS endpoint

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: most code written by AI, based on my plan

[mudler]

@ngxson sorry for the AI automated report, but as I was trying to port qwen3 tts to LocalAI I've met a blocker (which is reported by the bot)

[ngxson]

@mudler fwiw, the project's policy forbidden using automated bots to reply to maintainers. Please remove all bot's replies as they are not very readble and quite spam, otherwise, we will need to ban your bot, as mentioned in project's policy

[K234-1]

We've been running this PR's /tts server for a local voice assistant (Qwen3-TTS-1.7B-Base + x-vector cloning, Vulkan backend) and hit an issue with code2wav batching that others may run into, plus a fix that works well for us.

Symptom: decoding in batches smaller than wav_tfm_swa (72 frames) audibly "detunes" the voice mid-sentence. We had reduced the batch size to lower time-to-first-audio in streaming.

Cause: code2wav zero-pads every batch to 72 frames (clip.cpp input setter, and inp_codes in clip_graph_qwen3tts_gen::build()), and the padding leaks into the carried state: the KV prefix for the next batch and the causal-conv left-contexts are taken from padded positions (with 12-frame batches, 60 of the 71 prefix frames come from code-0 padding). The trimmed audio of the current batch is fine — it's the state handed to the next batch that's corrupted.

Fix: the state contract is already length-agnostic (prefix W−1, conv tails, tfm_pos += N all derive from tensor shapes), so it's enough to build the GEN_WAV sub-graph for the actual frame count instead of a fixed window — the same pattern clip_graph_pockettts_gen already uses. The banded attention mask keeps the effective window at wav_tfm_swa regardless of batch size, so short batches decode equivalently to full windows. The code-0 rear padding in the input setter and the tail-trim in the output path can then be dropped.

Results (12 sentences in 12 requests, ECAPA-TDNN speaker similarity vs. the cloning reference, identical sampling):

config	mean / min similarity
unpatched, constant 12-frame batches	0.817 / 0.777 (audible drift)
unpatched, 72-frame batches	0.862 / 0.815
patched, constant 12-frame batches	0.854 / 0.816
patched, batch ramp 12→24→48→72	0.847 / 0.800

With a short first batch (we ramp 12 → 24 → 48 → 72 in mtmd-helper-gen.cpp), streaming time-to-first-audio drops from ~1.8 s to ~0.4 s with no measurable quality loss.

patch (core fix, against 435d411)

diff --git a/tools/mtmd/clip.cpp b/tools/mtmd/clip.cpp
index 5969bf7..f8e0abc 100644
--- a/tools/mtmd/clip.cpp
+++ b/tools/mtmd/clip.cpp
@@ -1087,7 +1087,15 @@ static std::unique_ptr<clip_graph> clip_get_graph_builder(clip_ctx * ctx, const
                 const auto  gen_process = params ? params->gen_process : CLIP_GEN_PROCESS_GEN_CODE;
                 const int   top_k = params ? params->top_k : 50;
                 const float top_p = params ? params->top_p : 1.0f;
-                builder = std::make_unique<clip_graph_qwen3tts_gen>(ctx, img, gen_process, top_k, top_p);
+                // GEN_WAV: build code2wav for the actual number of frames in this call,
+                // rear-padding to a fixed window would leak padding into the carried state
+                const int64_t n_codes = ctx->model.gen_code_head_w->ne[2] + 1;
+                int n_frames = ctx->model.hparams.wav_tfm_swa;
+                if (params && params->gen_process == CLIP_GEN_PROCESS_GEN_WAV && params->codes) {
+                    GGML_ASSERT(params->codes->size() % (size_t) n_codes == 0);
+                    n_frames = (int) (params->codes->size() / n_codes);
+                }
+                builder = std::make_unique<clip_graph_qwen3tts_gen>(ctx, img, gen_process, top_k, top_p, n_frames);
             } break;
         case PROJECTOR_TYPE_YOUTUVL:
             {
@@ -5090,11 +5098,11 @@ bool clip_encode(struct clip_ctx * ctx, struct clip_encode_params * params) {
                 if (params->gen_process == CLIP_GEN_PROCESS_GEN_WAV) {
                     GGML_ASSERT(params->codes != nullptr);
 
-                    // frame-major input to group-major, rear-padded with code 0 up to one window
+                    // frame-major input to group-major; the graph is built for this
+                    // exact frame count (see the builder dispatch), so no padding
                     const int64_t n_codes  = model.gen_code_head_w->ne[2] + 1;
-                    const int64_t n_frames_w = hparams.wav_tfm_swa;
-                    const int64_t n_frames   = (int64_t) params->codes->size() / n_codes;
-                    GGML_ASSERT(n_frames > 0 && n_frames <= n_frames_w);
+                    const int64_t n_frames = (int64_t) params->codes->size() / n_codes;
+                    GGML_ASSERT(n_frames > 0);
 
                     // codes are used as ggml_get_rows indices, so check them against the codebook vocab
                     const int64_t vocab_first = model.c2w.quant_first_cb_w->ne[1];
@@ -5111,10 +5119,10 @@ bool clip_encode(struct clip_ctx * ctx, struct clip_encode_params * params) {
                         }
                     }
 
-                    std::vector<int32_t> codes(n_frames_w * n_codes, 0);
+                    std::vector<int32_t> codes(n_frames * n_codes);
                     for (int64_t f = 0; f < n_frames; f++) {
                         for (int64_t g = 0; g < n_codes; g++) {
-                            codes[g * n_frames_w + f] = (*params->codes)[f * n_codes + g];
+                            codes[g * n_frames + f] = (*params->codes)[f * n_codes + g];
                         }
                     }
                     set_input_i32("inp_codes", codes);
@@ -5628,15 +5636,8 @@ bool clip_encode(struct clip_ctx * ctx, struct clip_encode_params * params) {
         auto & out_audio = *params->out_audio;
         out_audio.resize(ggml_nelements(audio));
         ggml_backend_tensor_get(audio, out_audio.data(), 0, ggml_nbytes(audio));
-
-        // drop the tail audio that comes from the code-0 rear padding
-        const int64_t n_codes    = params->codes ? model.gen_code_head_w->ne[2] + 1 : 0;
-        const int64_t n_frames_w = hparams.wav_tfm_swa;
-        const int64_t n_frames   = params->codes ? (int64_t) params->codes->size() / n_codes : n_frames_w;
-        if (n_frames < n_frames_w) {
-            const size_t hop = out_audio.size() / n_frames_w;
-            out_audio.resize((size_t) n_frames * hop);
-        }
+        // no tail to drop: the code2wav graph is built for the actual frame
+        // count, so every emitted sample comes from a real input frame
     }
     if (params->state_out != nullptr) {
         auto & state_out = *params->state_out;
diff --git a/tools/mtmd/models/models.h b/tools/mtmd/models/models.h
index 3631d84..80181c9 100644
--- a/tools/mtmd/models/models.h
+++ b/tools/mtmd/models/models.h
@@ -233,8 +233,8 @@ struct clip_graph_qwen3tts_spkenc : clip_graph {
 };
 
 struct clip_graph_qwen3tts_gen : clip_graph {
-    clip_graph_qwen3tts_gen(clip_ctx * ctx, const clip_image_f32 & img, clip_gen_process_type gen_process, int top_k, float top_p)
-        : clip_graph(ctx, img), gen_process(gen_process), top_k(top_k), top_p(top_p) {}
+    clip_graph_qwen3tts_gen(clip_ctx * ctx, const clip_image_f32 & img, clip_gen_process_type gen_process, int top_k, float top_p, int n_frames)
+        : clip_graph(ctx, img), gen_process(gen_process), top_k(top_k), top_p(top_p), n_frames(n_frames) {}
     ggml_cgraph * build() override;
 
     // which sub-graph build() constructs, fixed at graph-build time
@@ -244,6 +244,11 @@ struct clip_graph_qwen3tts_gen : clip_graph {
     int   top_k;
     float top_p;
 
+    // GEN_WAV: number of frames this call actually decodes (the caller's batch size,
+    // hparams.wav_tfm_swa when unknown). the code2wav state contract is length-agnostic,
+    // so the sub-graph is built for the real frame count instead of a zero-padded window
+    int n_frames;
+
     //
     // code_gen: backbone hidden state + sampled code0 -> 16 RVQ codes
     // MTP-style code predictor, one token per codebook
diff --git a/tools/mtmd/models/qwen3tts-gen.cpp b/tools/mtmd/models/qwen3tts-gen.cpp
index 84c77f4..9befab6 100644
--- a/tools/mtmd/models/qwen3tts-gen.cpp
+++ b/tools/mtmd/models/qwen3tts-gen.cpp
@@ -732,7 +732,10 @@ ggml_cgraph * clip_graph_qwen3tts_gen::build() {
     cb(out_embd, "gen_audio_out", -1);
 
     // ---- CLIP_GEN_PROCESS_GEN_WAV: 16 RVQ codes -> raw PCM ----
-    const int n_frames = hparams.wav_tfm_swa; // frames per batch, == the attention window
+    // built for the caller's actual frame count: the attention stays banded to
+    // wav_tfm_swa via the mask, and every state slot is batch-size independent,
+    // so short batches (low-latency streaming) decode bit-equivalently to full windows
+    GGML_ASSERT(n_frames > 0);
 
     ggml_tensor * inp_codes = ggml_new_tensor_2d(ctx0, GGML_TYPE_I32, n_frames, n_codes);
     ggml_set_name(inp_codes, "inp_codes");

[ngxson]

@ServeurpersoCom let's also try to merge this today or tmr, I'm working on a refactoring of mtmd audio_gen API right after this

[ServeurpersoCom]

Tested it in a container with an RTX PRO 6000 and the Qwen3-TTS 1.7B Base Q8_0 GGUF. It works really well: JSON, multipart and base64 speaker refs all give clean speech (checked by transcribing the outputs with Gemma 4), streaming starts in about 0.65 s and stays correct across the code2wav windows, pcm is real float32, a client that disconnects frees its slot, and router mode reports the audio modality, autoloads and proxies /tts including streaming.

I pushed a small fix: seed and stream were missing from the multipart conversion list, so -F stream=true was rejected and you couldn't stream while uploading the speaker ref as a file.

One real issue left: a fixed seed isn't reproducible on the server. clip_ctx holds a single RNG that only reseeds when the seed changes, and the server shares that context across all requests, so two requests with seed 42 give different audio, and concurrent requests draw from the same RNG. llama-tts is only reproducible because each run is a fresh process. Letting the helper own its RNG and pass a derived seed per clip call would fix both, could fit in your audio_gen refactor.

Small things for later: the streaming get_output is duplicated in both pipelines, mtmd_helper_gen_audio_flush has no null check like reset, and K234-1's patch above would bring the first audio down from 0.65 s by decoding shorter code2wav batches.

[ServeurpersoCom]

LGTM once CI is green. The seed reproducibility can go into the audio_gen refactor.