[ggerganov]

Overview

This release adds the LiquidAI LFM3 family (text, vision, audio and omni) together with a new experimental streaming-audio API in mtmd, and wires realtime audio in and out of the server through the new /v1/realtime and /tts endpoints. A batch of new architectures landed: DeepSeek V4.1 Flash, SenseNova U1, Beam, Kolibri-1, Flint, Muse Spark, Granite Switch Decision, plus decision/embedding models from LiquidAI, Baidu and NVIDIA. The core got sparse-attention (QSA/DSA) reworked onto a fused masking op, a 50/50 SLRU cache for host-resident MoE experts, and cleaner no_alloc memory handling; the server got router memory-management and tool-calling improvements; the UI can now browse and download models straight from HF Hub, search conversations and compact them. Finally, ggml was bumped to v0.28.0, bringing GGML_OP_TOP_K_MASK, -1 expert-index semantics and large backend performance work (CUDA chunked graph launches, SYCL graph replay, OpenCL Q2_K/Q3_K, Hexagon HTP rework, KleidiAI SVE kernels).

Highlights

• LFM3 family: LFM3-1.7B (text) and LFM3-VL-2B (vision), verified against the reference HF implementation llama/31252
• LFM3-Audio-2B and LFM3-Omni-2B: ASR, TTS and voice chat from a single model, with streaming audio input and output llama/31257

API changes

• DRY sampler: new LLAMA_DRY_MAX_SEQ_BREAKERS (64) caps distinct sequence breakers, each costing O(n_vocab) at init llama/31035
• mtmd: new experimental mtmd_audio_stream API turning PCM fed as it arrives into decoder-ready embedding rows llama/30629
• mtmd: audio generation is now stateful - mtmd_gen_state replaces the raw state_data buffers of the audio gen API llama/30902
• mtmd: mtmd_caps gained gen_audio; new TTS types SOPRANO and LFM3A; repetition penalty for sampled codes llama/28607 llama/31257
• mtmd_helper_get_capabilities() to query the capabilities of an mmproj file from helper code llama/31183
• Server: new /v1/realtime endpoint implementing the OpenAI realtime protocol for conversation and transcription llama/31269 llama/30636
• Server: new POST /tts endpoint for text-to-speech models llama/26603
• Server: reasoning effort levels of the chat template are exposed and selectable via the API llama/31271
• Chat API: support for deferred tools and tool_reference content parts llama/31256
• CLI: --model or -hf is now required llama/31242

New models

• LiquidAI LFM3-1.7B, LFM3-VL-2B, LFM3-Audio-2B, LFM3-Omni-2B llama/31252 llama/31257
• LiquidAI d1-2B and d1-omni-2B decision models llama/31275
• DeepSeek V4.1 Flash llama/31086
• SenseNova U1 llama/28919
• Beam (Reflection) llama/30670
• Kolibri-1 (Aleph-Alpha) llama/30673
• Flint llama/30928
• Muse Spark Open (text, vision and audio) llama/30616
• Granite Switch Decision, including the 256-way decision head llama/30676 llama/30693
• Nemotron-3(.5)-Embed, with non-standard GQA and BPE tokenizer support llama/30877 llama/23343
• H2O-Lightning-4B decision model llama/30361
• MiniCPM-OCR 1.0 and MiniCPM-OCR-Jev llama/30979 llama/30980
• Soprano TTS llama/28607
• LFM2 short-conv and FFN blocks moved into a shared LFM base reused by newer LFM models llama/30874

Core changes

• MoE experts kept in host memory now use a 50/50 SLRU cache instead of a plain LRU cache llama/30637
• Sparse attention: fused QSA mask op, QSA/DSA selection built without kq_mask, k-pool views shared between layers llama/30640 llama/31009 llama/30965 llama/31243 llama/30613
• MoE: amax rescaling enabled for laguna, and -sm tensor denied on arches that use it llama/31147 llama/31081
• Memory: no_alloc respected in all memory caches, moved out of llama_hparams, obsolete n_pad dropped llama/31087 llama/31141 llama/31200
• Vocabulary loading optimized; d2t placed next to its set_rows use so it can stay on the GPU llama/30679 llama/30912
• Tokenizer: new clean_spaces and metaspace prepend-scheme GGUF keys, hybrid DNA tokenizer fixes llama/30940 llama/30825
• Tokenizer: Tekken pre-tokenizer for Pixtral, \p{M} in Tekken/GPT-4o lookaheads, Nemotron-3.5-Embed tokenization fix llama/30960 llama/30845 llama/30974
• Grammar: recursion depth limited (and enforced) to prevent stack overflow; deep JSON rejected before walking it llama/29755 llama/31153 llama/30242
• Qwen: merge tensors on convert, 2D output reshape, Qwen3 bidirectional attention and dense_2 output projection llama/31028 llama/30823 llama/30607
• Speculative decoding: persist the dynamic ngram cache, probabilistic DFlash2 drafts, optional DFlash FC input scale llama/22055 llama/30663 llama/30945 llama/30927
• LoRA: support for prism.hadamard folded weights llama/30910
• Tensor parallel: stepfun and laguna now support 4-10 GPUs llama/24554
• GGUF tools reject truncated tensor data in split, export-lora and merge llama/30497 llama/31044
• llama-quant: fix 4D tensor quantization with ggml_nrows() and name-based expert detection llama/30820
• convert-hf: faster endian conversion with more datatypes, text-only checkpoints in pplx-decider llama/30647 llama/30575
• convert-hf: fixes for Granite speech tensors, Granite Switch MultiSwitch block count, Mistral-format DeepseekV2 llama/30628 llama/30621 llama/30669
• Common: honor HTTP_PROXY/HTTPS_PROXY, allow concurrent downloads, pin model resolution on real HF repo layouts llama/21752 llama/29312 llama/30548

Multi-modality changes

• Streaming audio input via mtmd_audio_stream, used to support voxtral-realtime llama/30629
• Soprano TTS support and MiniCPM-OCR 1.0 support llama/28607 llama/30979
• Image preprocessing aligned with transformers/torchvision: LFM2 resize, tiles and layout, Qwen-VL resize llama/30943 llama/30261 llama/30736
• LFM2 tiling parameters are read from the GGUF metadata llama/30840
• lightonocr defaults to native resolution instead of forcing a 1540 longest edge llama/30600
• build_inp_temporal_merge moved to the mtmd base class; fix Gemma-4 FFN activation llama/31065 llama/30882

Server changes

• Realtime conversation and transcription on /v1/realtime, with response.done sent after the committed audio drains llama/31269 llama/30636 llama/30994
• POST /tts endpoint for TTS models llama/26603
• Expose the reasoning effort levels of the chat template llama/31271
• Router: configurable device memory margin for dynamic unloading of models llama/21231
• Router: propagate client disconnects to the child task, stop re-adding the mmproj margin on sleep/resume llama/29707 llama/28233
• Rerank endpoint accepts typed content input and substitutes the placeholder from the original template llama/29625 llama/30864
• Anthropic /v1/messages: forward llama.cpp sampling params and pass id_slot through the conversion llama/30856 llama/28554
• Support logprobs together with speculative decoding llama/27196
• Fix usage.completion_tokens undercounting with n > 1 and missing output_index in Responses API SSE events llama/29486 llama/30863
• Fix SWA checkpoint invalidation logic llama/30666
• New --connect flag to allow p2p access to the server outside the LAN llama/28277

UI changes

• Discover and download models directly from HF Hub, with org-less models listed first llama/29584 llama/31215
• New search dialog for conversations and messages llama/31173
• Conversation compaction to keep long chats within the context llama/27401
• Models manager moved to its own route, model details open in a drawer without leaving the page llama/30988 llama/31176
• Reasoning menu built from the chat template's effort levels, with a clearer highlighted toggle llama/31273 llama/31212
• Support OGG audio attachments by converting them to WAV llama/31237
• New "Show tool call in progress" setting llama/30681
• Save and reset settings buttons shown only when there are unsaved changes llama/31248
• Sidebar resizing and general layout/UX improvements llama/31119 llama/31030
• Fixes: draft lost after "Add System Message", system row stretching in WebKit llama/31214 llama/31246
• Fixes: clipped attachment text preview, markdown tables and lists cut off on phones llama/30806 llama/30687

ggml changes

• Bumped from v0.27.0 to ggml v0.28.0
• New GGML_OP_TOP_K_MASK fused sparse-attention mask, plus accessors for the precision configured on a compute node
• -1 is now a valid expert index in MUL_MAT_ID/ADD_ID: the slot is skipped and the output row zeroed
• SCALE accepts non-contiguous row views; SUM accepts non-contiguous inputs on CPU, CUDA, Vulkan and WebGPU
• Backends: fused MoE top-k and gate/up+GLU with chunked graph launches on CUDA, graph record/replay on SYCL
• Backends: Q2_K/Q3_K weights and a lightning indexer on OpenCL, subgroup reductions and src1 precision on Vulkan
• More: restructured Hexagon HTP backend, flash attention on ZDNN, Q2_0 on WebGPU, KleidiAI SVE GEMV/GEMM kernels