[didlawowo]

Summary

Add basic model support for Voxtral Mini 4B Realtime (February 2026), Mistral's streaming speech-to-text model.

This PR adds Phase 1 only: GGUF conversion + audio encoder integration into mtmd. The dual-stream inference protocol (audio + text embeddings summed at each position) will be a separate follow-up PR.

Changes

GGUF conversion (convert_hf_to_gguf.py):

• VoxtralRealtimeEncoderModel for Mistral-format mmproj conversion
• Precompute ada_rms_norm_t_cond adaptive normalization at conversion time (avoids runtime overhead)
• Handle tok_embeddings.weight extraction from mm_streams_embeddings

Decoder support (src/models/llama.cpp, src/llama-model.*, src/llama-arch.*):

• Optional ffn_ada_norm_up tensor per layer (TENSOR_NOT_REQUIRED)
• Single ggml_mul in forward pass when tensor is present — zero overhead for other models

Audio encoder (tools/mtmd/models/voxtral-realtime-enc.cpp):

• Causal transformer encoder (unlike Whisper which is bidirectional)
• Sliding window attention (750), RoPE, SwiGLU FFN, RMSNorm
• Causal conv1d with left-padding, frame stacking, adapter MLP

mtmd integration (clip.cpp, mtmd-audio.*, mtmd.*):

• PROJECTOR_TYPE_VOXTRAL_REALTIME with audio preprocessing (same mel params as Whisper)
• Tensor loading, output token calculation, position IDs with sliding window causal mask
• mtmd_support_voxtral_realtime() API for downstream consumers

What this does NOT include

• No new binary (no llama-voxtral-stream)
• No microphone capture
• No streaming inference
• No changes to mtmd-cli.cpp or llama-server

The dual-stream protocol (Phase 2) will be a separate PR after discussion on the approach.

Testing

Tested on RTX 3090 with the official model:

# Convert (Mistral format)
python3 convert_hf_to_gguf.py /path/to/model --outfile voxtral-4b.gguf --outtype q8_0 --mistral-format
python3 convert_hf_to_gguf.py /path/to/model --outfile mmproj.gguf --outtype f16 --mmproj --mistral-format

# Verify model loads correctly
llama-server -m voxtral-4b.gguf --mmproj mmproj.gguf -ngl 99 -c 8192
# Output: "load_hparams: projector: voxtral_realtime" ✓
# Output: "clip_model_loader: has audio encoder" ✓

Related

• Feature request: Feature Request: Add Voxtral Realtime #19696
• Previous attempt: Add Mistral Voxtral Mini 4B Realtime 2602 4B streaming ASR support #19698 (rejected due to scope — this PR addresses the feedback by splitting into phases)
• Based on work by @Acceldium, cleaned up and refactored for Phase 1 only

[ngxson]

This PR contains a large amount of AI-generated code and/or is too extensive for thorough review. Per our contribution guidelines, we cannot accept substantial AI-generated contributions, and large changes require prior discussion in an issue.

[didlawowo]

Closing in favor of a more complete PR that includes the full pipeline (encoder + dual-stream transcription + server endpoint). See replacement PR below.