[ggerganov]

Overview

llama.cpp v0.7.0 adds the LiliCorr speculative decoding technique and MTP for GLM5-Next, a new GPU cache for MoE experts kept in host memory (--moe-cache-mib, up to ~2.2x decode speedup when experts are offloaded), and a large batch of new models: K2 Horizon (dense and MoVA), Prism Bonsai 2 27B, EmbeddingGemma-2 (text+vision+audio), the LiquidAI d1/d1-omni and vLLM decision 2/d3 decision models, pplx-decider, MiniCPM-V 4.7, Nemotron Super 3.5 VL / Nemotron Bolt and maion-coder. The server gets a refactored chat session API (common_chat_session, part-based formatted output), much better slot and prompt-cache restore behavior, decision-model support for the new architectures and a new default port (9931), the web UI gains a full models manager, and ggml is updated to v0.27.0.

Highlights

• New GPU cache for host-memory MoE experts (--moe-cache-mib): experts run on the GPU from an LRU cache, up to 2.2x faster decode #29887
• The expert cache is now shared across multiple GPUs #30112
• LiliCorr speculative decoding: candidate selection with a small draft transformer on top of DFlash #30506
• GLM5-Next: MTP speculative decoding with an optimized sparse attention path #29928
• K2 Horizon support in both dense and MoVA variants: conversion, tokenizer, chat template, -sm tensor splitting #29535
• Prism Bonsai 2 27B ternary model support #29600
• EmbeddingGemma-2: text + vision + audio embedding model #30054
• New decision models: LiquidAI d1-3B, d1-omni-600M, vLLM decision 2, vLLM d3 and pplx-decider #30110 #30114 #30378 #30383 #30044
• PLaMo-3 tokenizer implements reference pre-segmentation for exact tokenization parity #30045
• RPC server supports -sm tensor (tensor-parallel RPC, RPC protocol v8) #26610
• Server now listens on port 9931 by default instead of 8080 #30159

API changes

• include/llama.h: new moe_cache_size context param for the host-expert GPU cache #29887
• include/llama.h: new LLAMA_VOCAB_TYPE_PLAMO3 vocab type #30045
• include/llama.h: llama_perf_context_data reports graph reallocations during inference #30329
• include/llama.h: embeddings now skips the LM head when a pooling type other than NONE is used #30445
• include/llama.h: llama_batch_ext_set_embd_state returns false when the context has no state embedding, M-RoPE order documented #30506
• tools/server: chat layer refactored around common_chat_session with feed() / finish() #30210
• tools/server: formatted chat output is now an array of parts instead of a single string #30389
• tools/server: default listening port changed to 9931 (Docker images keep 8080 inside the container) #30159

New models

• K2 Horizon: dense and MoVA variants with conversion, tokenizer, chat template and -sm tensor support #29535
• Prism Bonsai 2 27B (ternary) #29600
• EmbeddingGemma-2: text + vision + audio embeddings #30054
• LiquidAI d1-3B and d1-omni-600M decision models #30110 #30114
• vLLM decision 2 and vLLM d3 models #30378 #30383
• pplx-decider decision model #30044
• MiniCPM-V 4.7 multimodal model #29416
• Nemotron Super 3.5 VL and Nemotron Bolt 74B A9B #30520
• maion-coder architecture #29778
• Mellum: MoE gating parameters read from GGUF and unified RoPE handling #30578
• LiliCorr draft model conversion and graph #30506
• Rerankers: classifier_activation support, exact GELU for ModernBERT encoders #29692 #30108

Core changes

• Speculative decoding: MTP for GLM5-Next, LiliCorr drafts, correct mtmd handling in MTP graphs #29928 #30506 #30257
• Speculative decoding: shared nextn tensor flags with consolidated row cropping and sched re-reserve on flag change #30097 #30017 #30020
• Speculative decoding: fixed draft sampling and rejection handling, DFlash output head sharing #30095 #30111
• New --spec-draft-n-ubatch to size the draft context batch separately and save VRAM #30459
• MoE expert cache: host-memory experts served from a device LRU cache, supported over multiple GPUs #29887 #30112
• Embedding models: skip the logits buffer and the LM head when pooling is enabled (up to 10x less VRAM) #30255 #30445
• Embedding graphs: reordered get_rows, dropped the redundant t_embd expand and leftover ggml_cont calls #30160 #30461 #30198
• No-cache attention: padded KV dimension and KQ bias, avoided graph reallocations #30416 #30446 #30439
• k-pool models: selection built in the causal mask layout, fixed a scatter data race on shared sequences #30480 #29994
• GLM5-Next sparse attention: removed the gather path #30042
• KV cache: honor LLAMA_STATE_SEQ_FLAGS_PARTIAL_ONLY so draft checkpoints stop carrying the full KV #30514
• Prefetching: merge ranges before issuing system prefetch calls, never map foreign ranges into a backend buffer #30532 #30405
• Sampling: greedy selection for eligible temperature-zero chains, backend sampling graph kept static across ubatches #29797 #30223
• Tokenizers: PLaMo-3 pre-segmentation and FiM tokens, opt-in NFC normalization for Unigram, strict UTF-8 validation #30045 #30090 #30390 #30485
• Jinja: fixed loop variable scoping, Python-compatible tojson rendering, faster context copy #30533 #30483 #30362 #30562
• Model split: distribute layers evenly when no device reports free memory #29677

Multi-modality changes

• Added mtmd_build_inp_attn_mask for building non-causal multimodal attention masks #30259
• Added cohere2 vision encoder support #30062
• MTP drafts now handle mtmd inputs, Qwen3.8 Flash Next accepts image embeddings #30257 #30397
• New multimodal architectures: MiniCPM-V 4.7, Nemotron Super 3.5 VL, EmbeddingGemma-2 (vision + audio) #29416 #30520 #30054

Server changes

• /v1/systemone extended to the new decision models: d1-3B, d1-omni, vLLM decision 2/d3, pplx-decider #30110 #30114 #30378 #30383 #30044
• Chat refactor: session-based parsing and part-based formatted output #30210 #30389
• Generated text and tokens are now accumulated as parser input, preparing token-aware parsing #29876
• Prompt cache: only restorable slot states are saved, so rerank and embedding requests no longer fill the cache #30498
• Prompt cache: child tasks prefer empty slots and preserve the state they would overwrite #30567
• A pinned busy slot is left untouched until the request can run on it #30295
• Slot save/restore now keeps context checkpoints and the draft model state #26004 #30574
• Fixed an off-by-two in the SWA reuse threshold that discarded restored states #30580
• Fixed a stack-overflow crash when escaping special tokens with std::regex #30243
• LLAMA_ARG_SLOT_SAVE_PATH environment variable for --slot-save-path #30272
• New per-component server tests; vendored cpp-httplib updated to 0.61.0 #30312 #30494 #30134 #30081

UI changes

• New Models Manager: one table of all served models with expandable quants, search, filters, sorting and favorites #29583
• Row actions: load/unload, delete from disk, hide/unhide, resumable downloads with progress #29583
• Model configuration side drawer with model info, chat template and "start a new chat" / "use in this chat" actions #29583
• Models manager follow-ups: GGUF training context, favorite UX, filter and sort fixes #30228
• Fixed ui_settings not being applied on the first visit in router mode #29668
• Fixed text attachment preview overflow for large pasted text #29898
• Fixed the default agentic turns limit falling back to 10 instead of 100 #30582

ggml changes

• ggml updated to v0.27.0: the backend scheduler gains user callbacks for split weight copying and compute-buffer reallocation, and the RPC protocol is bumped to v8 for tensor-parallel RPC servers. Backends gained chunked gated-delta-net kernels on CUDA/HIP, KleidiAI SME2 k-quant matmuls and AVX2 Q2_0 dot products on CPU, L2-aware coopmat2 scheduling and bf16 activations on Vulkan, reordered-layout MoE kernels (MXFP4, IQ3, Q5_K, grouped XMX GEMM) on SYCL, dp4a MoE GEMMs on OpenCL, tensor-API and wider-head-dim flash attention on Metal, and a DMA/HVX copy overhaul with pooling and 3D im2col ops on Hexagon. Correctness work focused on non-contiguous and strided views, grid and workgroup size limits, out-of-bounds accesses and well-defined mul_mat_id behavior for duplicate expert ids; symbol visibility now defaults to hidden