Repository navigation
Add DeepSeek V4 Flash Vision support #2411
Description
Row: MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm
DeepSeek published deepseek-ai/DeepSeek-V4-Flash-Vision-Exp at revision 86f746b36186f0e567729a5c06a8c918caba82a9. The checkpoint reuses the DeepSeek-V4 text backbone and adds a 32-layer ViT, aligner, image sentinel embeddings, image-span attention visibility, interleaved image prompt encoding, and a DSpark tail.
vLLM at the repository parity pin has no vision path for DeepseekV4ForCausalLM; current vLLM, vLLM-Omni, SGLang, and Transformers also expose no complete implementation of this vision variant. The checkpoint repository's minimal PyTorch inference is the only complete executable reference and needs a model-specific oracle pin before it can gate a port.
Scope: add the model and oracle to the roadmap, commit the implementation spec before code, then implement the processor, vision tower and aligner, weight and quantized arms, DeepSeek image merge and visibility semantics, registered multimodal forward, OpenAI multi-image serving, and correctness/performance gates. The spec and implementation use one pull request, with the spec commit first.
CUDA, ROCm, and Vulkan are separate required device paths. Every accelerator build, correctness run, and benchmark must execute through resource-controller (rc) on a leased fleet device; no direct SSH run or CPU fallback counts. CUDA uses an eligible NVIDIA fleet device, ROCm uses strix:gpu0, and Vulkan uses the physical RADV GFX1151 device on strix:gpu0 rather than llvmpipe. A full-artifact memory blocker remains explicit per backend while reduced device-path correctness is still required.
The issue closes when the production entry point accepts the pinned checkpoint family through the documented quantized vehicle, ordinary autoregressive image generation is oracle-gated, text-only DeepSeek-V4 stays byte-identical, CUDA/ROCm/Vulkan device-path evidence is recorded from rc jobs, and the shipped checkpoint details are documented.
Activity
localai-org-maint-bot commentedon Sep 5, 2026
Scope amendment, 2026-09-05
Three facts changed since this issue was written, and the row was still planning against the old ones. The spec
.agents/specs/deepseek-v4-flash-vision.mdis amended accordingly.1. The quantized vehicle is now a published artifact
unsloth/DeepSeek-V4-Flash-Vision-Exp-GGUFat revisionb977d3c0ea2da58dbc12ddae8fb8951a7b3854d0publishes ten imatrix quantizations plus one sharedmmproj-BF16.gguf.UD-IQ1_S+mmproj-BF16is 77.65 GiB resident and fits a single 119 GiB GB10, so the single-device gate is reachable for the first time.It ships the llama.cpp two-file shape, which the spec listed as a non-goal and a stop condition. That non-goal is withdrawn by developer direction, with the reason recorded beside it: it existed to stop the row from asking users to reassemble weights, and producing a private combined artifact nobody downloads is now the thing that would do that.
src/vllm/model_executor/models/clip_mmproj_gguf.cppalready reads this shape for Qwen3-VL, so no container is invented. The revision and per-shard SHA-256 values are pinned in the spec, because unsloth re-quantizes in place under unchanged names.2. llama.cpp implements this model, and is registered as an oracle
ggml-org/llama.cppmerged ggml-org/llama.cpp#28133 (thedeepseek4vclip projector, image preprocessor, mmproj container) and ggml-org/llama.cpp#28154 (the language-side vision behaviour) on 2026-09-02. Releaseb10766is9400c8946e4da5e7694f2c26d6d4e50e14b690fa, the merge commit of the second, and the first release carryingtools/mtmd/models/deepseek4v.cpp. That path returns HTTP 404 at the stockllama-cpppinb10451, which is 315 commits behind.Registered as
llama-cpp-dsv4visionin.agents/oracles/,gateable = nountil a leased build-and-run. It is a runnable reference and a quant-matched denominator for the artifact above; it is not a mirror source and does not outrank the model author or vLLM.3. Two language-side behaviours were missing from the spec
Reading that implementation found two DeepSeek-V4 language behaviours that change under image input and that the spec never mentioned. Both are the failure mode the spec's own risk 3 names: drop either and the argmax stays plausible while a token gate stays green.
exp_probs_b_vl— a second per-layer MoE routing bias, f32[256], used when the routed token is an image token. The pinned text GGUF's entire first shard is these 43 tensors and nothing else. On the three hash layers it does not merely substitute a bias: text tokens there route through thetid2eidhash table and take no bias at all, so for an image token this bias replaces the hash routing.deepseek4.attention.sliding_window = 128while an image block can be 384 tokens, so more than half a span can be invisible while generation still reads fluently.Both are specified, given tests that cannot pass on tokens alone, and listed under
## Owed.Wave status
inference/vision.pyat the pinned revision. Fresh review with mutations is owed.deepseek4vmmproj reader) and W3B (exp_probs_b_vlaccounting) — in flight.Nothing here is a correctness or performance claim. No oracle has been built or run.
fix(MODEL-MM-deepseek-v4): name the tensor, layer and geometry the ho…
test(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): mutate the visi…
fix(MODEL-MM-deepseek-v4): make the served-image expectation device-a…
docs(MODEL-MM-deepseek-v4): close the CPU-only-premise issue, green o…
docs(MODEL-MM-deepseek-v4): record the measured outcome, and retire a…
merge: the fp8_ds_mla store/read bridge (#2455) into the DeepSeek-V4 …
spec(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): scope Flash Vis…
feat(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): port W1 prompt …
spec(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): require CUDA, R…
feat(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): port W2 vision …
spec(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): pin the artifac…
feat(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): account for and…
record(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): close the lan…
feat(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): read the deepse…
43 remaining items
fix(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): make the two ima…
docs(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): cite the pinned…
docs(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): re-measure the …
docs(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): the spec's Now …
test(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): hold the two li…
record(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): add the three…
fix(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): delete the refus…
test(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): add the W6 real…
test(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): measure the ora…
test(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): add an f32 arm …
docs(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): record W6, the …
feat(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): load the offici…
test(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): pin the vision …
fix(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): repair the W7-CU…
fix(MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm): correct five sta…