[didlawowo]

Summary

Full support for Voxtral Mini 4B Realtime (February 2026), Mistral's streaming speech-to-text model, including a new OpenAI-compatible /v1/audio/transcriptions endpoint.

Pre-quantized GGUFs: https://huggingface.co/chris0173/Voxtral-Mini-4B-Realtime-2602-GGUF

What's included

1. Audio encoder + GGUF conversion (convert_hf_to_gguf.py, mtmd/)

• VoxtralRealtimeEncoderModel for Mistral-format mmproj conversion
• Causal transformer encoder (sliding window 750, RoPE, SwiGLU, RMSNorm)
• Precomputed ada_rms_norm_t_cond at conversion time (zero runtime overhead)
• PROJECTOR_TYPE_VOXTRAL_REALTIME with Whisper-compatible mel params

2. Dual-stream transcription (mtmd-helper.cpp, mtmd-cli.cpp)

• mtmd_helper_eval_voxtral_realtime() implements the dual-stream protocol
• Audio + text embeddings summed at each position (not prepended like other models)
• Token embedding table loaded from GGUF for the summation
• Full autoregressive decode with greedy sampling
• CLI support: llama-mtmd-cli --image audio.wav -p "Transcribe"

3. Server endpoint (tools/server/)

• POST /v1/audio/transcriptions — OpenAI Whisper-compatible API
• Supports both multipart/form-data (standard) and JSON with base64 (fallback)
• New task type SERVER_TASK_TYPE_AUDIO_TRANSCRIPTION
• Follows EMBEDDING pattern: process at DONE_PROMPT, clear KV cache, release slot
• Resolves feature request Feature Request: add /v1/audio/transcriptions endpoint for local openai STT support #15291

Usage

# 1. Convert from Mistral format
python3 convert_hf_to_gguf.py /path/to/Voxtral-Mini-4B-Realtime-2602 \
    --outfile voxtral-4b-rt-Q8_0.gguf --outtype q8_0 --mistral-format
python3 convert_hf_to_gguf.py /path/to/Voxtral-Mini-4B-Realtime-2602 \
    --outfile mmproj-voxtral-4b-rt-F16.gguf --outtype f16 --mmproj --mistral-format

# 2a. CLI transcription
llama-mtmd-cli \
    -m voxtral-4b-rt-Q8_0.gguf \
    --mmproj mmproj-voxtral-4b-rt-F16.gguf \
    -ngl 99 \
    -c 8192 \
    --image audio.flac \
    -p "Transcribe this audio." \
    -n 512

# 2b. Server mode
llama-server \
    -m voxtral-4b-rt-Q8_0.gguf \
    --mmproj mmproj-voxtral-4b-rt-F16.gguf \
    -ngl 99 \
    -c 8192 \
    --host 0.0.0.0 \
    --port 8080

# 3. Transcription API (OpenAI Whisper-compatible)
# multipart/form-data (standard)
curl -X POST http://localhost:8080/v1/audio/transcriptions \
    -F file=@audio.flac \
    -F language=en

# JSON with base64 (alternative)
curl -X POST http://localhost:8080/v1/audio/transcriptions \
    -H "Content-Type: application/json" \
    -d '{"file":"<base64-encoded-audio>","format":"wav","language":"en"}'

Response format:

{"task":"transcribe","language":"en","duration":0.0,"text":"Let him come in then, said the king..."}

Resource requirements

• VRAM: ~5.5 GB (3.5 GB text Q8_0 + 2 GB mmproj F16)
• RAM: 8 GB recommended (model loading + audio processing)
• GPU: any CUDA-capable GPU (tested on RTX 3090)

Testing

Tested on RTX 3090 with LibriSpeech (7127-75946-0002.flac):

Metric	Value
Audio encode	169 ms
Decode speed	145 tok/s
Total latency	~6.6 s (for 15s audio)

Deployed and verified on a Kubernetes cluster with ArgoCD GitOps.

Dual-stream protocol (architecture note)

Unlike standard multimodal models that prepend media embeddings, Voxtral Realtime sums audio and text embeddings at each position:

Standard:   [audio_embeddings] [text_tokens] → decoder
Voxtral RT: audio_embd[i] + tok_embd[prompt[i]] → decoder (at every position)

This enables streaming with <500ms latency. The prefix consists of [BOS] + [PAD×32] + [DELAY×6] = 39 tokens.

Related

• Feature request (Voxtral RT): Feature Request: Add Voxtral Realtime #19696
• Previous attempt: Add Mistral Voxtral Mini 4B Realtime 2602 4B streaming ASR support #19698 (feedback incorporated — no new binary, no mic, no streaming)
• Feature request (audio endpoint): Feature Request: add /v1/audio/transcriptions endpoint for local openai STT support #15291
• Supersedes: mtmd : add Voxtral Realtime 4B audio encoder support #20625
• Based on work by @Acceldium, refactored into clean incremental changes

What this does NOT include

• No new standalone binary
• No microphone capture / real-time streaming
• No changes to the generation loop for other models

[didlawowo]

any reviewer ?

[ngxson]

This PR contains many model-specific code paths, which are considered as anti-pattern in libmtmd design. Please refer to #20914 for a formal discussion before proceeding. For now, I'm closing the current PR.