(Planning) Support audio output in mtmd

This feature has been discussed multiple times but until now that I have the time to summarize some design choices.

Currently, there are some problems when bringing audio generation support into llama.cpp / mtmd:

Model file: Audio generation usually requires a dedicated "audio detokenizer" model. For example, sesame CSM and LFM (first version) uses Mimi, the LFM2 uses a normal text generation model as audio tokenizer, and nvidia's chatterbox uses S3Gen (token2wav)
Data passing: Some models like the qwen-omni family or sesame CSM requires passing the embedding vectors (float values) between LLM and audio generation model, instead of passing raw token IDs (int values)
Generation state: We need a way to mark the start and end of speech data. For example, a model may generates: <text>hello<audio><523><5><84>...<text>world<audio>..., that means the token IDs between <audio> and <text> markers should be piped into mtmd to generate audio wav output.

Design choices

1. Model file

Probably the best way we can do for now is:

If the audio detokenizer model is a normal causal-style decoder transformer, we can make it a normal text model (this is the case for LFM-Audio). mtmd will need to be extended to support managing this auxiliary model
If the audio detokenizer model is fundamentally different. For example, being a convolution based model, we can fully implement it in libmtmd

Alternately to the solution (1), we can also implement a mechanism such that mmproj and a text model can co-exist inside a GGUF. But I think it's better to be a future improvement.

2. Data passing

While many models simply passing the token ID between the main LLM and the audio generation model, I think it's more flexible to always pass the data as embeddings.

The implication is that text model will hold 2 set of embeddings: one for normal text model, and another to be used by audio detokenizer

The pseudo-code looks like this:

llama_token sampled_token = llama_sampling_sample(...); // llama_token is an integer
if (mtmd_token_is_audio(sampled_token)) {
  float * embeddings = llama_get_audio_embeddings(ctx, sampled_token);
  mtmd_add_audio_embeddings(mctx, embeddings);
}

3. Generation state

Making an API for managing generation state (i.e. knowing that the model is currently generating text or something else) can be quite hard to get right. Therefore, we will try to make the public API as "clean" as possible in the first iteration.

A proposal can be:

mtmd_gen_state object, which is an opaque pointer
mtmd_gen_state_track(state, generated_token) add a new generated token to tracking state
mtmd_gen_state_get(state) get the current state, return an enum: TEXT, AUDIO, ...

The underlay logic can vary from model to model

@ngxson

Thanks @ngxson for kicking off the planning here. We're working on bringing Chatterbox TTS (Resemble AI) into llama.cpp and have been prototyping against this exact design space. We wanted to share a few concrete extensions for discussion.

TL;DR — Addition to MTMD API surface

The public additions to libmtmd are grouped into 4 clusters.

Capability

bool mtmd_support_audio_output (const mtmd_context * mctx);
bool mtmd_gguf_peek_decoder    (const char * fname, /*out*/ bool * has_detok,
                                                    /*out*/ bool * has_postproc);

Per-request generation state — owns the audio accumulator and the TEXT↔AUDIO state. One per concurrent inference; never shared.

typedef enum { MTMD_GEN_STATE_TEXT, MTMD_GEN_STATE_AUDIO } mtmd_gen_state_type;
typedef enum { MTMD_AUDIO_INPUT_TOKENS, MTMD_AUDIO_INPUT_EMBEDDINGS } mtmd_audio_input_kind;
struct mtmd_gen_state;   // opaque, caller-owned

mtmd_gen_state *      mtmd_gen_state_init   (const mtmd_context * mctx);
void                  mtmd_gen_state_free   (mtmd_gen_state * state);

// state-machine — driven by the LM loop
bool                  mtmd_gen_state_track       (mtmd_gen_state *, llama_token);
mtmd_gen_state_type   mtmd_gen_state_get         (const mtmd_gen_state *);
bool                  mtmd_gen_state_transitioned(mtmd_gen_state *);   // edge-triggered, read-clear

// audio accumulator — dispatched on the loaded decoder's declared kind (queried below)
mtmd_audio_input_kind mtmd_get_audio_input_kind  (const mtmd_context *);

bool mtmd_gen_state_push_audio_tokens    (mtmd_gen_state *,
                                          const llama_token * toks, int n);
bool mtmd_gen_state_push_audio_embeddings(mtmd_gen_state *,
                                          const float * embd, int n_tokens, int n_embd);
bool mtmd_gen_state_clear_audio          (mtmd_gen_state *);

Paired llama.cpp additions for the embedding-input case — looking up the model's per-token audio-embedding row (Mimi-aware LMs ship a learned audio-embedding table; the lookup belongs in llama.cpp since that's where the model lives):

// llama.h — defined in llama.cpp, not mtmd
LLAMA_API int32_t       llama_n_audio_embd       (const struct llama_model *);   // 0 if no audio table
LLAMA_API const float * llama_get_audio_embeddings(struct llama_context *,
                                                   llama_token token);           // NULL if no audio table

Decode helpers — drive decode_a and decode_postproc end-to-end. They take a per-request gen_state (carries the accumulator) and the shared mtmd_context (carries the loaded decoders). The reference audio for speaker conditioning is passed directly as a mtmd_audio_buffer * — no chunk-container indirection, no name routing.

bool   mtmd_helper_decode_audio        (mtmd_context * mctx,
                                        mtmd_gen_state * state,
                                        const mtmd_audio_buffer * ref_wav,  // OPTIONAL — may be NULL
                                        mtmd_audio_buffer * out);
bool   mtmd_helper_decode_postproc     (mtmd_context * mctx,
                                        mtmd_audio_buffer * inout);         // in-place, stateless
size_t mtmd_get_audio_output_sample_rate(const mtmd_context * mctx);
bool   mtmd_helper_audio_save_to_file  (const mtmd_audio_buffer * audio,
                                        const char * filename);

ref_wav is optional. Non-NULL on the first decode of a session populates the decoder's "ref_wav" slot; pass NULL thereafter (or always, for arches without that slot). Mismatches — ref_wav to a decoder without the slot, or NULL to a decoder that requires it with nothing previously populated — return false with a verbose log.

Per-arch encoder→decoder wiring lives inside the helper. Some detokenizers consume outputs from the loaded encoder beyond what the LM produces. S3Gen, for instance, needs ref_speech_tokens — the reference audio's speech tokens, generated by clip_a's S3TokenizerV2 — pushed into its ref_speech_tokens input. These aren't on the LM's autoregressive path, so they don't flow through the gen_state accumulator. mtmd_helper_decode_audio carries a small arch-dispatched table (switch (decoder_get_type(dctx))) that pulls each required output out of clip_a (via clip_get_output(name)) and pushes it into the decoder (via decoder_set_input(name, …)). Arches without such requirements (Mimi-style detokenizers, plain vocoders) fall through the default branch and consume only the accumulator + ref_wav.

Typed audio container — encoder ref audio, decoder output, postproc in-place buffer, and save-to-file all share one type (see §6).

typedef struct mtmd_audio_buffer mtmd_audio_buffer;
mtmd_audio_buffer * mtmd_audio_buffer_init     (const float * samples,
                                                size_t n_samples, uint32_t sample_rate);
void                mtmd_audio_buffer_free     (mtmd_audio_buffer *);
const float *       mtmd_audio_buffer_data       (const mtmd_audio_buffer *);
size_t              mtmd_audio_buffer_n_samples  (const mtmd_audio_buffer *);
uint32_t            mtmd_audio_buffer_sample_rate(const mtmd_audio_buffer *);

That is the entire public surface. Together these form the minimum needed to run a Chatterbox-style pipeline (LM → accumulator → S3Gen → Perth → wav) entirely through mtmd_helper_*, with no caller-side knowledge of which detokenizer or postprocessor is loaded. The internal decoder_* layer that the helpers call into is in §4.

User's hot path

End-to-end usage of the public surface:

// setup once
mtmd_context * mctx = mtmd_init_from_file("chatterbox.gguf", /*params*/);
mtmd_gen_state * st = mtmd_gen_state_init(mctx);
auto kind = mtmd_get_audio_input_kind(mctx);
const int n_audio_embd = llama_n_audio_embd(model);

// optional: load ref audio once for the session
mtmd_audio_buffer * ref_wav = mtmd_audio_buffer_init(pcm, n_samples, 16000);

// LM loop
while (sampling) {
    llama_token tok = sample(...);
    mtmd_gen_state_track(st, tok);
    if (mtmd_gen_state_get(st) == MTMD_GEN_STATE_AUDIO) {
        if (kind == MTMD_AUDIO_INPUT_TOKENS) {
            mtmd_gen_state_push_audio_tokens(st, &tok, 1);
        } else {
            mtmd_gen_state_push_audio_embeddings(st,
                llama_get_audio_embeddings(lctx, tok), 1, n_audio_embd);
        }
    }
    // edge-triggered: just exited the AUDIO segment → render it
    if (mtmd_gen_state_transitioned(st) &&
        mtmd_gen_state_get(st) != MTMD_GEN_STATE_AUDIO) {
        mtmd_audio_buffer * out = mtmd_audio_buffer_init(NULL, 0, 0);
        mtmd_helper_decode_audio   (mctx, st, ref_wav, out);  // ref_wav: NULL after first call
        mtmd_helper_decode_postproc(mctx, out);
        mtmd_helper_audio_save_to_file(out, "out.wav");
        mtmd_audio_buffer_free(out);
        ref_wav = NULL;
    }
}

Chatterbox pipeline → mtmd mapping

How each stage of the Chatterbox pipeline lands in llama.cpp / mtmd. The first column is the model component, the second is what we propose to implement it as, the third is what's new vs. existing.

File layout — two GGUFs, mirroring how llama.cpp already ships LM + MMProj for vision/audio models:

chatterbox.gguf — the LM file. Holds T3's weights, hparams, vocab, and the BPE tokenizer (with the audio-cue special tokens). Loaded via llama_model_load_from_file, drives the LM through the existing llama_context.
chatterbox-mmproj.gguf — the multimodal projection / detokenizer file. Loaded via mtmd_init_from_file and routed by GGUF key namespace:
- Reference Voice Encoder → clip_a (audio encoder ctx). GGUF keys under clip.audio.*.
- S3Gen detokenizer → decode_a (detokenizer slot). GGUF keys under decoder.detokenizer.*.
- Perth watermarker → decode_postproc (postproc slot). GGUF keys under decoder.postproc.*.

Splitting this way matches the existing convention (LMs and projection models ship separately, can be quantized independently) and lets users swap the watermarker or the vocoder by replacing only the mmproj file.

Pipeline step	Implementation	Inputs	New vs. existing
HF → GGUF conversion	`convert_hf_to_gguf.py` extensions	HF Chatterbox checkpoint (T3 + voice encoder + S3Gen + Perth)	new arch entries (writes weights + arch tag + hparams; slot names live in §3's C++)
Text tokenizer	`llama_tokenize`	UTF-8 prompt text (with audio-cue specials)	existing — BPE baked into the T3 GGUF
Loudness Norm	audio utils	raw ref wav PCM (any sample rate)	small DSP helper, pre-encoder
Reference Voice Encoder	`clip_a` context	normalized ref wav PCM (mono, 16 kHz)	existing clip ctx + multi-output extension (§5)
T3 (Token-to-Token)	`llama_context`	text tokens + speaker conditioning + ref speech tokens	existing; reuses KV-cache + sampling
S3Gen	new `decoder_ctx` (`decode_a`)	speech tokens (from T3 via gen_state) + ref speech tokens (from clip_a) + `ref_wav`	§3, §4 — detokenizer slot in mtmd
Perth Watermarker	new `decoder_ctx` (`decode_postproc`)	PCM audio buffer (24 kHz, from S3Gen)	§3, §4 — postproc slot in mtmd
Audio output container	`mtmd_audio_buffer`	n/a (carrier type)	new typed container (§6); replaces `mtmd_bitmap`'s audio role

Bold rows are the additions this proposal motivates. Everything else either exists already or is a small, localized add (converter entries, DSP utility) that's not part of the mtmd-side design discussion.

Implementation Details

1. MMProj metadata for decoder routing.

Suggest pinning down concrete GGUF keys so loaders can peek and route without arch sniffing. Presence of decoder.<ns>.arch is the source of truth — no parallel has_* bool flags to keep in sync:

decoder.<ns>.arch           (str)   ; <ns> ∈ {detokenizer, postproc}; presence = enabled
decoder.<ns>.<arch>.*       (arch-specific hparams: n_mels, n_flow_layers, sample_rate, …)

Loader API:

// returns the arch string, or nullptr if this namespace isn't present
const char * mtmd_gguf_decoder_arch(const gguf_context * ctx, const char * ns);

The postproc namespace gives watermarkers / loudness-restorers a clean home without overloading the detokenizer slot. The arch's slot names, dtypes, modality, and input_kind are not stored in GGUF — they live in the arch's C++ source, since build_graph is the real authority on what a given arch consumes and produces. See §3 for the staging model.

2. mtmd_context additions — and what doesn't live there.

struct mtmd_context {
    // Loaded models — read-mostly, shared across concurrent requests
    clip_ctx          * ctx_v;
    clip_ctx          * ctx_a;
    decoder_ctx       * decode_a;        // detokenizer (e.g. S3Gen, Mimi)
    decoder_ctx       * decode_postproc; // postproc (e.g. Perth) — stateless, safe to share
    const llama_model * text_model;

};

The per-request audio token/embedding accumulator and the TEXT↔AUDIO state machine do not live on mtmd_context — they live on the caller-owned mtmd_gen_state. A server with N concurrent inferences allocates N mtmd_gen_states sharing one mtmd_context. Without this split, two concurrent requests would step on each other's accumulators or state transitions.

Stateful detokenizers (S3Gen's flow-matching session state) get their own per-request scratch held inside mtmd_gen_state rather than decoder_ctx — same reason. The decoder_ctx itself owns weights + scheduler, both of which are reentrant-safe under the assumption that runtime tensors come from the per-request scratch passed in via decoder_set_input.

Postproc (decode_postproc) is stateless by construction — mtmd_helper_decode_postproc is in-place over a caller-supplied audio buffer with no accumulator — so it doesn't need a per-request slot.

3. A generic decoder_ctx, parallel to clip_ctx. (internal to libmtmd; not on the public API)

Chatterbox needs two non-transformer post-LM modules: S3Gen (token→waveform) and a Perth watermarker (waveform→watermarked waveform). Implementing each "directly in libmtmd" works for one, but two starts to look like the same lifecycle/scheduler/backend boilerplate twice. We'd suggest a sibling context type — used only by mtmd_helper_* and the per-arch model implementations, never by callers:

struct decoder_ctx {
    decoder_model            model;
    gguf_context_ptr         ctx_gguf;
    ggml_context_ptr         ctx_data;
    std::vector<uint8_t>     buf_compute_meta;
    std::vector<ggml_backend_t>             backend_ptrs;
    std::vector<ggml_backend_buffer_type_t> backend_buft;
    ggml_backend_t           backend, backend_cpu;
    ggml_backend_buffer_ptr  buf;
    ggml_backend_sched_ptr   sched;
};

enum decoder_type {
    DECODER_TYPE_S3GEN,
    DECODER_TYPE_PERTH_WATERMARKER,
    // future audio decoders: VOCOS, SOUNDSTORM, ...
};

enum decoder_modality {
    DECODER_MODALITY_AUDIO,
};

with a polymorphic decoder_model base (load_hparams / load_weights / build_graph, status-returning where applicable) and per-arch subclasses (s3gen_model : decoder_model, perth_model : decoder_model).

decoder_model — overview.

The shape mirrors clip_model but is reversed in intent: a clip_model produces embeddings, a decoder_model consumes them and produces audio. It owns everything arch-specific so decoder_ctx can stay arch-blind.

The arch's build_graph is the single source of truth for slot names. decoder_set_input(name, data, shape, type) just stages the bytes under that name on the ctx; build_graph later picks them up by name when constructing input tensors, and tags output tensors by name so they can be read back.

ggml_tensor * is per-decode, never long-lived on the ctx. decoder_set_input doesn't allocate a ggml_tensor * — those exist only inside build_graph → compute, bound to the per-decode ggml_context and backend buffers allocated by the scheduler. The ctx carries host-side byte buffers that get memcpy'd to/from the device tensors around each compute.

decoder_ctx (long-lived state)
├─ Identity
│  ├─ type           : decoder_type            (S3GEN | PERTH_WATERMARKER | VOCOS | …)
│  ├─ modality       : decoder_modality        (AUDIO)
│  └─ input_kind     : mtmd_audio_input_kind   (TOKENS | EMBEDDINGS) — arch-implied
│
├─ Staging (host-side byte buffers, keyed by slot name)
│  ├─ staged_inputs  : map<string, staged_io>  ← populated by decoder_set_input
│  └─ output_buffers : map<string, staged_io>  ← populated by decoder_decode (read-back)
│
│     staged_io { ggml_type type; int32_t shape[GGML_MAX_DIMS]; vector<uint8_t> data; }
│
├─ Backends + scheduler (model weights + sched lifetime)
│
└─ Virtual interface (per-arch)
   ├─ load_hparams(const char * mmproj_file)               → bool
   ├─ load_weights(gguf_context_ptr)                       → bool
   └─ build_graph (ggml_context * gctx)                    → ggml_cgraph *
                                                           ← reads staged_inputs by name (creates input
                                                             tensors sized from staged shape/type, tagged
                                                             via ggml_set_name + ggml_set_input);
                                                             tags outputs via ggml_set_name + ggml_set_output

Init order is just load_hparams → load_weights → backend + scheduler setup.

Concrete subclasses supply their own hparams + weights structs and override the three virtuals. Nothing about S3Gen leaks into the base, and nothing about the base assumes an arch:

// tools/mtmd/models/s3gen.h
struct s3gen_model : decoder_model {
    struct hparams {
        int n_mels;
        int n_flow_layers;
        int n_hift_upsample;
        int sample_rate;          // 24000
        // …
    };
    struct weights {
        // UpsampleConformerEncoder
        ggml_tensor * conformer_qkv[N];
        ggml_tensor * conformer_ffn[N];
        // CausalConditionalCFM (flow matching)
        ggml_tensor * cfm_velocity_field[N];
        // HiFT vocoder
        ggml_tensor * hift_upsample[N];
        ggml_tensor * hift_resblock[N];
        // …
    };

    hparams hp;
    weights w;
    int     n_max_nodes;          // graph allocator hint

    bool          load_hparams(const char *)                          override;
    bool          load_weights(gguf_context_ptr)                      override;
    ggml_cgraph * build_graph (decoder_ctx * ctx, ggml_context * gctx) override;
};

build_graph reads staged inputs by name and tags outputs:

ggml_cgraph * s3gen_model::build_graph(decoder_ctx * ctx, ggml_context * gctx) {
    ggml_cgraph * gf = ggml_new_graph(gctx);

    auto mk_input = [&](const char * name) -> ggml_tensor * {
        const auto & s = ctx->staged_inputs.at(name);  // throws → caught + logged with `name`
        ggml_tensor * t = ggml_new_tensor(gctx, s.type, rank_of(s.shape), s.shape);
        ggml_set_name (t, name);                        // bridge: name → ggml_graph_get_tensor
        ggml_set_input(t);                              // scheduler keeps backing buffer for it
        return t;
    };
    auto * speech_tokens     = mk_input("speech_tokens");      // from T3 (gen_state accumulator)
    auto * ref_speech_tokens = mk_input("ref_speech_tokens");  // from clip_a (helper-wired)
    auto * ref_wav           = mk_input("ref_wav");            // from caller
    auto * noise             = mk_input("noise");              // flow-matching seed

    // … flow matching, HiFT, etc. …
    ggml_tensor * audio = /* … */;
    ggml_set_name  (audio, "audio");
    ggml_set_output(audio);

    ggml_build_forward_expand(gf, audio);
    return gf;
}

perth_model follows the same pattern with its own (much smaller) hparams + weights and the same three overrides; its build_graph reads audio_in and tags audio_out.

The slot names are part of each arch's public contract — documented in the arch source. mtmd_helper_* functions (the only callers expected to populate the staging maps) know what to set. End users and the LM loop never touch these names directly; they go through the helper API in the TL;DR above. For S3Gen:

direction	name	type	shape	meaning
in	`speech_tokens`	I32	`[T]`	newly generated tokens, from T3 (via gen_state accumulator)
in	`ref_speech_tokens`	I32	`[T_ref]`	speech tokens of the reference audio, from clip_a's S3TokenizerV2 output (wired by helper)
in	`ref_wav`	F32	`[N_samples_ref]`	reference audio (S3Gen derives speaker conditioning + prompt mel internally)
in	`noise`	F32	`[80, T]`	flow-matching seed
out	`audio`	F32	`[N_samples]`	24kHz waveform

Adding Vocos or SoundStorm later is a new subclass whose build_graph reads its own input names and tags its own outputs. decoder_ctx, mtmd_helper_*, and the public API are untouched.

4. Internal decoder_* API surface. (internal to libmtmd; consumed only by mtmd_helper_* impls and unit tests)

// ── Lifecycle ──────────────────────────────────────────────────────────
// decoder_init runs: dispatch arch via decoder.<ns>.arch → load_hparams →
// load_weights → backend + scheduler setup. NULL on failure (logs).
decoder_ctx *    decoder_init           (const char * fname,
                                         decoder_context_params params);
void             decoder_free           (decoder_ctx * ctx);    // idempotent on NULL
decoder_type     decoder_get_type       (const decoder_ctx * ctx);
decoder_modality decoder_get_modality   (const decoder_ctx * ctx);

// ── IO (per-request) — host-side staging by slot name ─────────────────
// Stages bytes into ctx->staged_inputs[name]. No ggml_tensor allocated
// here — those are created inside decoder_decode → build_graph, sized
// from the staged shape/type, then memcpy'd from the staged data after
// graph allocation.
bool             decoder_set_input      (decoder_ctx * ctx, const char * name,
                                         const uint8_t * data,
                                         const int32_t shape[GGML_MAX_DIMS],
                                         ggml_type type);

// Builds graph, allocates via the scheduler, copies staged inputs to
// device tensors, runs, reads outputs back into ctx->output_buffers.
// `out_optional` short-circuits the read-back for the canonical "audio"
// output if the caller wants it written straight into a buffer.
bool             decoder_decode         (decoder_ctx * ctx,
                                         mtmd_audio_buffer * out_optional);

// Reads back from ctx->output_buffers (host-side, post-decode). Copy-out —
// never aliases ctx memory. dst=NULL or dst_capacity too small returns the
// required byte count via required_bytes-style call.
bool             decoder_get_output     (decoder_ctx * ctx, const char * name,
                                         uint8_t * dst, size_t dst_capacity,
                                         int32_t shape[GGML_MAX_DIMS],
                                         ggml_type * type);

There's no introspection API — slot names live in the arch source; tools that want to know "what does this arch expect" read it from there. After at least one decode has run, ctx->staged_inputs and ctx->output_buffers give a runtime answer to "what slots have been used."

5. clip_ctx extension for multi-tensor encoder outputs. (internal to libmtmd; existing single-embedding clip_*_encode callers see no change)

Today's clip_*_encode() returns a single embedding. That breaks for Chatterbox's reference voice encoder, which produces multiple heterogeneous outputs (speech tokens consumed by both T3's cond prefix and S3Gen as ref_speech_tokens; speaker embedding consumed by T3) — different dtypes, shapes, and consumers. We propose mirroring the decoder-side staging model: the clip arch's build_graph tags each output via ggml_set_name + ggml_set_output; after clip_*_encode runs, the named outputs are read back into a clip_ctx::output_buffers map (same staged_io shape as §3). Callers fetch by name:

// Copy-out reader — looks up by name in clip_ctx::output_buffers (populated
// by the previous encode call). Returns false + log if `name` isn't there.
// dst=NULL or dst_capacity too small fills required_bytes so a single
// round-trip can size and copy.
bool clip_get_output(const clip_ctx *, const char * name,
                     uint8_t * dst, size_t dst_capacity,
                     int32_t shape[GGML_MAX_DIMS], ggml_type * type,
                     size_t * required_bytes);

Single-output encoders keep working through clip_*_encode unchanged — they tag exactly one output, and clip_get_output(ctx, "embedding", …) is equivalent to today's behavior. Multi-output encoders just tag more outputs in build_graph.

6. mtmd_audio_buffer — typed audio container, replacing the audio role of mtmd_bitmap.

mtmd_bitmap's audio path is a fiction: nx × 1 shape, no sample rate, data reinterpreted as float. We propose a purpose-built mtmd_audio_buffer (PCM samples + sample rate + id) used everywhere audio shows up — encoder ref input, decoder output, postproc, save-to-file. Public API in TL;DR. mtmd_input_chunks's audio variant switches to mtmd_audio_buffer *; the image variant keeps mtmd_bitmap *. mtmd_bitmap_init_from_audio / _is_audio / the is_audio field deprecate for one release, then go.

// tools/mtmd/mtmd.cpp
struct mtmd_audio_buffer {
    std::vector<float> samples;   // PCM, mono
    uint32_t           sample_rate = 0;
    std::string        id;        // optional, mirrors mtmd_bitmap.id
};

mtmd_audio_buffer * mtmd_audio_buffer_init(const float * samples, size_t n_samples,
                                           uint32_t sample_rate) {
    auto * b = new mtmd_audio_buffer{ {}, sample_rate, {} };
    if (samples && n_samples) b->samples.assign(samples, samples + n_samples);
    else                      b->samples.resize(n_samples);  // zero-init; "helper will fill"
    return b;
}
void                mtmd_audio_buffer_free        (mtmd_audio_buffer * b) { delete b; }
const float *       mtmd_audio_buffer_data        (const mtmd_audio_buffer * b) { return b->samples.data(); }
size_t              mtmd_audio_buffer_n_samples   (const mtmd_audio_buffer * b) { return b->samples.size(); }
uint32_t            mtmd_audio_buffer_sample_rate (const mtmd_audio_buffer * b) { return b->sample_rate; }
const char *        mtmd_audio_buffer_id          (const mtmd_audio_buffer * b) { return b->id.c_str(); }
void                mtmd_audio_buffer_set_id      (mtmd_audio_buffer * b, const char * id) { b->id = id ? id : ""; }

// internal (mtmd-internal.h) — used by mtmd_helper_decode_audio to grow the output
float * mtmd_audio_buffer_data_mut(mtmd_audio_buffer * b) { return b->samples.data(); }
bool    mtmd_audio_buffer_resize  (mtmd_audio_buffer * b, size_t n, uint32_t sr) {
    b->samples.resize(n); b->sample_rate = sr; return true;
}

The constructor's samples == NULL branch zero-allocates n_samples floats so the helper can fill the buffer in place — same pattern as mtmd_bitmap_init today.

Encoder-side reuse (follow-up, not in this PR). The same type can replace mtmd_bitmap on the encoder's audio path. clip_audio_encode would take const mtmd_audio_buffer * directly instead of reading nx as n_samples and casting data to float *; the encoder gains a real sample_rate field for free. Vision callers stay on mtmd_bitmap, audio callers move, and the is_audio discriminator goes away. Migration is mechanical (get_data + cast → _data, get_nx → _n_samples) and worth doing in a follow-up PR after the TTS work lands.

@ngxson

Hey @ngxson, dropping a note since this is the area I want to contribute to next. I work on audio in GGML and have been studying vocoder math and optimizing the inference path through acestep, omnivoice and qwentts. Two GGML primitives consistently dominate the hot path of every BigVGAN and SEANet-style decoder (Mimi included, since SEANet is its backbone): the Snake activation y = x + sin(a*x)^2 * inv_b fused from its 5 op decomposition, and transposed 1D conv reformulated as matmul + GGML_OP_COL2IM_1D, the latter being the bigger of the two when left unoptimized.

Status on llama.cpp: Snake CUDA merged via #22667, Snake Vulkan #22855 reviewed and waiting on a second approval, Snake Metal done on my fork with the PR coming next, COL2IM_1D done and validated CPU + CUDA + Vulkan + Metal on my fork with one PR per backend incoming. End to end on a real BigVGAN-like decoder, Snake fusion alone is +42.7% CUDA and +24.8% Vulkan wall clock with the optimized COL2IM_1D already in place.

The three projects below have been picked up by a broad enough range of users to validate hardware compatibility and surface edge cases on older Ampere / Turing GPUs:
https://github.com/ServeurpersoCom/acestep.cpp (music, ACE-Step)
https://github.com/ServeurpersoCom/omnivoice.cpp (TTS/voice clone)
https://github.com/ServeurpersoCom/qwentts.cpp (TTS/voice clone)

I would probably start with Qwen Omni and work my way down the layers toward a POC on llama-ui. Plenty of work ahead :)

@akarale-nv

@akarale-nv Firstly, I want to remind that the project's policy doesn't allow fully using AI to write messages or comments. Your message's content is densely packed and it took me quite a lot of time to read.

I kindly request to only have human-to-human communication (no AI-written messages) in this thread to avoid further mental strain.

For your proposal, I have some questions:

You mentioned about mtmd_gguf_peek_decoder at the beginning and then never mention about it again, is it still relevant?
I still don't get what mtmd_gen_state_push_audio_embeddings/tokens do under the hood. Can you explain futher?

Review about your proposal:

I feel like mtmd_gen_state is suddenly too complicated, we are introducing 7 calls at once here, things like _get and _transitioned can be collapsed into one call that returns a struct mtmd_gen_state_data actually we don't need that, see my code below
Seems like we don't even need to track edge transition inside mtmd, user-space code can track it easily.
On second thought, we should probably make mtmd_gen_state a C++ only interface for now, ideally in a dedicated file mtmd-gen-context.cpp and we will expose C interface later.
To avoid naming conflict "state", let's call it a context and the state will now be an enum.

I propose a new pseudo-code as below:

// setup once
mtmd_context * mctx = mtmd_init_from_file("chatterbox.gguf", /*params*/);

mtmd_gen_context gctx(lctx, mctx); // C++ only for now

// accumulator for PCM output
std::vector<float> pcm_out;

// LM loop
mtmd_gen_state st = MTMD_GEN_STATE_TEXT; // be default, the model generates text

while (sampling) {
    llama_token tok = sample(...);

    // track a new token, returns new state
    mtmd_gen_state st1 = gctx.track(tok);

    if (st1 == MTMD_GEN_STATE_AUDIO) {
        gctx.push_tokens({tok}); // it will automatically call llama_get_audio_embeddings if needed
    }

    if (st1 != st && st1 != MTMD_GEN_STATE_AUDIO) {
        // transition edge from AUDIO to another thing
        int status = gctx.decode_audio(pcm_out); // optionally check status code
        mtmd_helper_audio_save_to_file(pcm_out, "out.wav");
    }

    st = st1; // update current state
}

Let me know if the version above looks good to you.

I would like to finalize the discussion about the interface above first, then move on to other points later.

@akarale-nv

@akarale-nv on second thought, a C++ only API is a bit messy, I will see how we can have a better API

I'm working on a demo with LFM audio model, will share when it's ready. we can then extend from such API to support chatterbox

@ngxson

Hi @ngxson,
Thank you for your feedback.

I would like to note that although, the generated text may be AI-assisted, the core proposal and its design is thoroughly human-made and human-reviewed. However, I will keep this in mind going forward.

Coming to your query about the mtmd_gen_state design, the current design is split into 3 parts:

State Tracking: The first 5 APIs are for memory management, continuously reading the tokens and tracking the current gen state. Listing the C APIs below:

mtmd_gen_state *      mtmd_gen_state_init   (const mtmd_context * mctx);
void                  mtmd_gen_state_free   (mtmd_gen_state * state);

// state-machine — driven by the LM loop
bool                  mtmd_gen_state_track       (mtmd_gen_state *, llama_token);
mtmd_gen_state_type   mtmd_gen_state_get         (const mtmd_gen_state *);
bool                  mtmd_gen_state_transitioned(mtmd_gen_state *);   // edge-triggered, read-clear

Embedding/token accumulation: Since we need to accumulate the audio tokens in a buffer, I found that the mtmd_gen_state is the right place to hold that buffer as both the gen_state and buffer will be cleaned up at the same time. In essence, both these variables will have to be cleaned up after the current gen session ends. Hence, I added the below 3 APIs for the accumulator:

bool mtmd_gen_state_push_audio_tokens    (mtmd_gen_state *,
                                          const llama_token * toks, int n);
bool mtmd_gen_state_push_audio_embeddings(mtmd_gen_state *,
                                          const float * embd, int n_tokens, int n_embd);
bool mtmd_gen_state_clear_audio          (mtmd_gen_state *);

Accumulation type: In our 2 use cases we are considering for this proposal, the decoders for both expect different kinds of inputs. LFM expects the audio embeddings and Chatterbox expects audio tokens. Hence, to track what kind of data accumulator stores we have the below API:

mtmd_audio_input_kind mtmd_get_audio_input_kind  (const mtmd_context *);

Now, if you look at the "State Tracking" section, it is close to what you had originally proposed as well as the C++ interface you have explained above. However, as you mentioned we can still optimize and remove the "mtmd_gen_state_transitioned()" API and instead just rely of the "mtmd_gen_state_get()" to achieve the same results.

I'm working on a demo with LFM audio model, will share when it's ready.

It would be helpful if you can share an early prototype or a design for this so that we can try it out on our end as well.

I created a first hacky POC...

llama-ui-TTS.mp4

@akarale-nv

Hey @akarale-nv , I drafted a non-working PoC on how the API should look like: master...ngxson:llama.cpp:xsn/mtmd_gen_lfm

My implementation still missing the depthformer component to make it work, but even without that, I think I can now safely define a list of goals that I want to achieve in the first iteration:

Minimal API; as discussed in my last comment, some API that you proposed can either be fused to another, or be replaced with a few lines of code in the user code. The end goal of any implementations is to have a clean / easy API that can be called from llama-server, so I'd recommend thinking twice before adding a new API call. (Hint: in many cases, libmtmd can call directly functions in libllama, so we should make sure user code has the least extra calls to libllama)
mtmd_gen_vocoder can manage multiple sub-contexts if it needs to, that includes both llama-context and mtmd-context (one-to-many relationship)
one mtmd_gen_vocoder can be used to create multiple mtmd_gen_context --> HOWEVER, for now, let's just make it simple by allow one vocoder per context
for each new model, it should inherit from mtmd_gen_vocoder, I already did an example of mtmd_gen_vocoder_lfm2a in my code

For the next step, I invite you to review my PoC above, and let me know if you think there are anything missing (in terms of API design). Thanks!

@ServeurpersoCom appreciated if you can take a look too. Since you already had the PoC, I think it's only matter of agreeing on a good architecture now, before we can upstream your implementation

I'll examine your PoC carefully and we'll converge toward a clean implementation!

@ServeurpersoCom

@ServeurpersoCom Nice to see work on this issue. I have a - fully vibecoded - working gguf based qwen3tts implementation, not sure if useful, but maybe doesn’t hurt to have a look? https://github.com/Danmoreng/qwen3-tts.cpp

I have a - fully vibecoded - working gguf based qwen3tts implementation, not sure if useful, but maybe doesn’t hurt to have a look? https://github.com/Danmoreng/qwen3-tts.cpp

I'm going to look at how you controlled the AI and the architecture he created out of curiosity. Which model did you use? I use Opus 4.7 from its website with MCP agentic loops that connect to my rootless Podman server with a shared GPU. I made one too, but it's faster (1.7x in BF16) than the Python version of an already accelerated fork (FasterQwenTTS) on CUDA!

I have a - fully vibecoded - working gguf based qwen3tts implementation, not sure if useful, but maybe doesn’t hurt to have a look? https://github.com/Danmoreng/qwen3-tts.cpp

I'm going to look at how you controlled the AI and the architecture he created out of curiosity. Which model did you use? I use Opus 4.7 from its website with MCP agentic loops that connect to my rootless Podman server with a shared GPU. I made one too, but it's faster (1.7x in BF16) than the Python version of an already accelerated fork (FasterQwenTTS) on CUDA!

Faster that the FasterQwenTTS sounds great. Didn't manage to get it quite that fast. I used Codex CLI for this. The architecture is most likely sub-optimal, since I let the AI do basically everything.

This is a bit off-topic; it would almost require a dedicated thread for AI coding: You have to read every single change, create highly targeted LLM contexts to ensure you get only the benefits and not the slop, and completely control the architecture. If even one slop creeps into the project, it contaminates the entire session, You should always clear the context before continuing. it's like training an animal! LLM can even falsify benchmarks! Everything must be programmatic and in git, otherwise it won't hesitate to lie. And you always have to be explicit, especially about the stupidest things. LLM is a gifted child.

@ngxson

Hi @ngxson, thanks for the prototype. I now have a clearer picture of the LFM2 audio needs. Overall, our design seems to be aligning on some parts. Here are some of the parts where I think we need a bit more re-design.

Orchestration loop:

For LFM, the loop looks to be Backbone->Vocoder->Backbone->… and so on.

While for Chatterbox, it looks like the following:
backbone generates audio tokens till EOS -> accumulate tokens -> Vocoder (S3Gen) -> Watermarker -> Audio.

How would you suggest we handle this difference in the orchestration loops?

Struct mtmd_gen_vocoder :

I like the idea of having a single interface that can hold multiple types of contexts. I think this aligns with the way we were originally thinking about this problem. However, there are some caveats we need to handle to be able to generalize this:

The S3Gen part of Chatterbox (Vocoder) is a diffusion model that de-noises the mel-spectrogram tensor over 2 iterations. For this, the mtmd_gen_vocoder_s3gen context will have to implement its own ggml_ctx, ggml_cgraph and build the graph. Will this be allowed in the new design?
For loading the models, my original plan included having s3gen as part of mmproj file for this pipeline with additional gguf_metadata suggesting the presence of decoder/vocoder, something like mmproj.has_decoder=true. This should still be feasible given mtmd_gen_vocoder_init() API. But are we okay updating the mmproj metadata list to include these?
Additionally, do you want to define a set of keys for vocoder or you would leave that upto the implementation owner?
Along with the vocoder, there is a watermarking model within the Chatterbox pipeline that we would like to include for the E2E execution. This would fall under the "one mtmd_gen_vocoder can be used to create multiple mtmd_gen_context" category and I would like to nail down the details for this as well if possible.

mtmd_gen_vocoder_init() API
The current implementation assumes the existence of a llama model for the vocoder, which would not be the case for Chatterbox S3Gen. How about we move all the loading logic to mtmd_gen_vocoder_* c'tor and have a pass-through implementation in this API?

mtmd_gen_vocoder::decode() API
The following is what we need for chatterbox:

Accumulated audio tokens from the backbone: This could potentially fit into the "backbone_embd" float tensor but since these are tokens, it would be better to have it as Int32.
Additionally, the Vocoder requires 2 more additional input tensors that represent the reference audio embeddings in which the voice is to be produced. This remains unhandled. (Refer to the attached diagram)
backbone_embd_out would remain unused for the Chatterbox implementation.

Combining the above requirements with LFM's requirements, would it be possible to design a decode interface where we do not restrict the number and dtype of the I/Os? In my mind something roughly like the following should help satisfy a larger set of usecases:

		struct  staged_io 
		{ 
			ggml_type type; 
			int32_t shape[GGML_MAX_DIMS]; 
			vector<uint8_t> data; // byte-data of the tensor
		};
		map<string, staged_io> vocoder_inputs;     // Named inputs
		map<string, staged_io> vocoder_outputs;    // Named outputs

These can be used inside the mtmd_gen_decode() under a per-arch switch case similar to how it is handled in clip_image_batch_chunk().

@ngxson

@ngxson here is my POC implementation for Chatterbox TTS that implements the Backbone + S3Gen Vocoder for your reference as well.

Looking forward to your thoughts

@akarale-nv

Hi @akarale-nv sorry for the delay. Thanks for the demo, I will have a look.

To response to your questions:

mtmd_gen_vocoder_s3gen will have ggml_ctx --> maybe we can "fuse" the vocoder with the existing mmproj (if any)? this way we can reuse the same graph/ctx infrastructure already exist inside clip.cpp
s3gen has decoder --> continue my point above: yes, it should be the same as vision and audio current can co-habit inside the same mmproj. vision tensors are prefixed by v., audio prefixed by a., go maybe audio gen can be prefixed by gen.a.
"The current implementation assumes the existence of a llama model for the vocoder" --> the llama_model is the main backbone text model (usually causal, because clip.cpp only supports non-causal mode). I'm not sure if s3gen is causal or non-causal?

@ServeurpersoCom

@ServeurpersoCom did you get a chance to review this proposal?

Sorry for piggybacking on the discussion. Please check out my project audio.cpp, which aims to bridge the gap between llama.cpp and the messy world of audio models (many are non-AR). It currently supports 35 models. Consider it as llama.cpp for audio models.

I’m also considering using llama.cpp as backbone for AR TTS models instead of my hand-tuned autoregressive model template in the future. This PR may be relevent.

Repo: https://github.com/0xShug0/audio.cpp

@akarale-nv

sorry for the delay, @akarale-nv I'm implementing Qwen3-TTS in #26254 , it is still WIP but it's quite similar to Sesame-CSM (#12648) with dual autoregressive model (one backbone, one emits codes) and a code2wav to convert audio codes to waveform

I will have a look later on chatterbox to see how it fits, but I'm quite confident that most TTS models already converged into the same overall form now.

I haven't yet look into @ServeurpersoCom 's impl, but will take a look when I finish implementing all the non-standard components.

Split video follow-up to issue #450, re-confirm #439 audio no-go

Chiming in as a downstream consumer: I maintain the Inference Gateway, which proxies OpenAI-style POST /v1/audio/speech to a llamacpp provider. Our CLI just grew a gateway TTS engine on top of that, but right now there is nothing to point it at, since Qwen3-TTS (#26254) only exists in the llama-tts binary.

Two questions:

Would a minimal, model-specific /v1/audio/speech in llama-server be acceptable before the generic mtmd_gen API lands, or is server wiring strictly blocked on it? mtmd: support Qwen3-TTS (note: breaking change to llama-tts binary) #26254 took a similar shortcut for the CLI, so maybe the same is fine for the server as a first step.
OpenAI's speech schema has no field for a reference sample. Would an extension field (e.g. reference_audio, base64 wav) for zero-shot cloning be considered? That's what we forward today, and it would be a shame if the endpoint gets frozen OpenAI-strict without a way to pass a speaker reference.

I can test builds or run a server PoC against a real client if that helps.

Temporary one-shot implementation using llama-tts in the backend is done on the inference-gateway. Looking forward to see this also in llama.cpp so I can simply forward the traffic to an llama.cpp server (I think that's the natural place for this feature to land). You can use it in the meantime in the inference-gateway, but it won't handle a lot of traffic, I gave it default concurrency of 2 and I guess it depends on the hardware - for basic usage it's fine.

Here is a usage example: https://github.com/inference-gateway/inference-gateway/tree/main/examples/rest-endpoints#text-to-speech

This issue was closed because it has been inactive for 14 days since being marked as stale.

Implemented in #31269 for LFM3 models

Navigation Menu

(Planning) Support audio output in mtmd #21956

Design choices

1. Model file

2. Data passing

3. Generation state

TL;DR — Addition to MTMD API surface

User's hot path

Chatterbox pipeline → mtmd mapping

Implementation Details

7 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

(Planning) Support audio output in mtmd #21956

Description

Design choices

1. Model file

2. Data passing

3. Generation state

Activity

akarale-nv commented on Apr 30, 2026

TL;DR — Addition to MTMD API surface

User's hot path

Chatterbox pipeline → mtmd mapping

Implementation Details

ServeurpersoCom commented on May 18, 2026

ngxson commented on May 21, 2026

ngxson commented on May 21, 2026

akarale-nv commented on May 22, 2026

ServeurpersoCom commented on May 27, 2026

ngxson commented on May 27, 2026

ServeurpersoCom commented on May 27, 2026

Danmoreng commented on May 27, 2026

ServeurpersoCom commented on May 28, 2026

Danmoreng commented on May 28, 2026

ServeurpersoCom commented on May 28, 2026

akarale-nv commented on Jun 10, 2026

akarale-nv commented on Jun 11, 2026

ngxson commented on Jun 21, 2026

7 remaining items

akarale-nv commented on Jul 21, 2026

0xShug0 commented on Jul 24, 2026

ngxson commented on Jul 28, 2026

edenreich commented on Aug 31, 2026

edenreich commented on Sep 1, 2026

github-actions commented on Oct 16, 2026

tdakhran commented on Oct 27, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions