Repository navigation
(Planning) support Voxtral Mini 4B realtime ASR #20914
Description
This discussion is created to guide the direction of recent PR: #19698 , #20638
For context, libmtmd / llama-server don't yet have the capability to process audio input in realtime, and adding these can be complicated. Per our contribution guidelines, complex changes require a discussion before being implemented
Proposed API design
It is important to design a generic API that can support Voxtral and also newer models in the future. Remind: adding model-specific API in libmtmd is an anti-pattern and will be rejected.
Currently, libmtmd already supports processing audio input as a fixed-size chunks (typically 30 seconds each, shorter input will be padded with silence). The audio will then be converted to spectrogram. This step in the pipeline is done by mtmd_tokenize()
- To support pre-processing, we simply need to allow
mtmd_tokenize()to process smaller chunks. Padding tokens can be added by the the tokenizing API. - For the dual-streaming text + audio summation, the summation should be done right after encoder projection in
clip.cpp, these changes are required to make it work:- To get the token embeddings, a new API like
llama_get_tensor_token_embeddings()should be added tollama.h - New API
mtmd_bitmap_add_text_tokens()will need to be added in order to append text token into the audiomtmd_bitmapobject mtmd_bitmap,mtmd_input_chunkandclip_image_f32should be updated to store the associated text tokens- For models that doesn't support dual-streaming,
mtmd_bitmap_add_text_tokensis no-op, thus this allow having a true model-agnostic API
- To get the token embeddings, a new API like
Server implementation will need to be updated to use mtmd_bitmap_add_text_tokens(), but the changes should be minimal
Proposed timeline
Phase 1: implement all functions in libmtmd, which should allow mtmd-cli to process correctly input audio file. To facilitate reviewing, there should be NO realtime public API that exist in this stage (no changes to server, mtmd-cli uses mtmd_bitmap_add_text_tokens)
Phase 2: Implement support in llama-server, which should support realtime transcribe from static input file. It should be based on https://developers.openai.com/api/docs/guides/speech-to-text
Phase 3: Implement realtime audio streaming, which should be based on websocket API: https://developers.openai.com/api/docs/guides/speech-to-text#streaming-the-transcription-of-an-ongoing-audio-recording
Phase 4: Optional input mic capture in mtmd-cli, to be defined
Activity
github-actions commentedon Mar 23, 2026
Danmoreng commentedon Apr 1, 2026
I'm very interested in this, since I tried getting Voxtral to work in Android app but that wasn't very successful yet.
I got some questions about this:
clip_image_*and use a flagis_audioinstead of a more generic function name and concrete branches between audio and image. Should this be refactored beforehand?Kaina3 commentedon Apr 15, 2026
I'm also interested and I wanted to check the intended scope and ordering before going further.
From looking at PRs #21421 #21863 , it seems the static-file transcription path is mostly in place now.
( However, parts of phase 1 don't appear to be in place yet:
mtmd_bitmap_add_text_tokensandllama_get_tensor_token_embeddings)For Phase 3, would you prefer the first step to be a simpler WebSocket-based flow where the server buffers incoming audio and only runs transcription on commit, or do you want to aim directly for true incremental streaming as audio arrives?
Discussed a little in here
( or maybe remaining Phase 1? )
I’m asking because those feel like quite different implementation paths, the second one may require additional work on the libmtmd side first.
tdakhran commentedon Apr 16, 2026
Depending on the model, streaming STFT support might be required.
github-actions commentedon May 31, 2026
This issue was closed because it has been inactive for 14 days since being marked as stale.
github-actions commentedon Jul 17, 2026
This issue was closed because it has been inactive for 14 days since being marked as stale.
Voxtral llama.cpp STT engine core (Phase A) (#16)
Voxtral llama.cpp STT engine core (Phase A) (#16)
patrickvonplaten commentedon Jul 29, 2026
BTW Voxtral Realtime is now nicely supported in audio.cpp with high perf: https://github.com/0xShug0/audio.cpp/blob/main/docs/asr.md#voxtral-realtime if people want to try it out locally
mtmd: add dual-stream realtime ASR decode path
mtmd: add dual-stream realtime ASR decode path