[sfallah]

Overview

Adds baidu/Unlimited-OCR to mtmd. It is DeepSeek-OCR v1 with a different decoder attention, R-SWA (Reference Sliding Window Attention): every token sees the whole prompt (image + text) plus the last 128 generated tokens. With plain SWA the image would slide out of view.

Changes:

• new LLAMA_SWA_TYPE_REFERENCE and its mask rule
• the KV cache records where the prompt ends (n_ref) when the first generated token arrives, and keeps it correct across seq_rm / seq_cp / seq_add / seq_div and session save/load (LLAMA_SESSION_VERSION and LLAMA_STATE_SEQ_VERSION bumped)
• one full KV cache, no eviction; llama_model_n_swa returns 0 for this arch so the server does not treat it as SWA
• V cache defaults to F32 for R-SWA (F16 garbles dense tables); a quantized -ctv is still honored

Only single-page parity for now.

Validation

bf16, single page, HF reference in its release config (bf16 weights, R-SWA, CUDA):

	CER
llama.cpp (Metal)	0.1658
llama.cpp (CUDA)	0.1431
HF reference	0.1591

Re-validated 2026-08-31 after merging current master.

Design note

R-SWA sits in core (swa type, mask rule, n_ref in llama_kv_cache) and is only enabled for deepseek2-ocr. It could live in the model code instead; open to moving it there.
Anyhow my immediate aim is that Unlimited-OCR is supported in llama.cpp with its original R-SWA.

Limitations

The KV cache guesses the prompt end from the first single-token decode that requests output. Where the guess is off:

• speculative decode (contexts allowing more than one output per sequence): no guess, plain causal attention
• a prompt whose last ubatch is a single token: boundary one token early
• context shift: boundary moves to the current position
• seq_cp copies the boundary as-is

An explicit prompt-length signal from the caller (mtmd knows it) would remove the guessing. Open to that as well.

How to run

GGUF models: sabafallah/Unlimited-OCR-GGUF

build/bin/llama-mtmd-cli -hf sabafallah/Unlimited-OCR-GGUF:bf16 \
  --image tools/mtmd/test-1.jpeg -p "document parsing." \
  --chat-template deepseek-ocr \
  --temp 0 --flash-attn off --no-warmup \
  -n 4096 -c 16384 \
  --dry-multiplier 0.8 --dry-base 1.75 --dry-allowed-length 35 \
  --dry-penalty-last-n 128 --dry-sequence-breaker none

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES - this is largely an AI-written first draft, opened to start the discussion. I designed the R-SWA approach (from the model's paper), directed and reviewed the implementation; the code itself is mostly AI-generated. I take responsibility for the contents.

[jason-ni]

It seems the impl in this branch has numeric precision issue:
https://huggingface.co/sabafallah/Unlimited-OCR-GGUF/discussions/1

@o7si could you please have a test on image in that discussion? Thanks.

[sfallah]

It seems the impl in this branch has numeric precision issue: https://huggingface.co/sabafallah/Unlimited-OCR-GGUF/discussions/1

@o7si could you please have a test on image in that discussion? Thanks.

@jason-ni
yes, I will.
Thanks for reporting this.

[sfallah]

@jason-ni
the root cause was found and fixed.
It was the DRY flags in my run example, not the model.
See details in the HF discussion: https://huggingface.co/sabafallah/Unlimited-OCR-GGUF/discussions/1

[twinforcesl]

Intel Arc A770/Vulkan single-page datapoint for Unlimited-OCR (not a test of this PR's R-SWA branch):

• Environment: Fedora 44, Linux 7.2.7-200.fc44.x86_64, Intel Arc A770 (DG2, i915), Mesa Vulkan 26.2.3. Clean Vulkan/Release build from llama.cpp commit 0ef4d560e12c1a46470265c1abd31dd47c777d23 (-DGGML_VULKAN=ON -DGGML_NATIVE=ON -DLLAMA_CURL=OFF); llama-mtmd-cli --version: 0.3.0-dev, build 445. Executable SHA-256 ffd2f510bf39f8054c579f9880313dcb3bd90fd61cea226b3f95ec23fe640d75.
• Weights: community Q5_K_M GGUF and F16 mmproj at 45cd66ec, SHA-256 8f5c5d7ec2c02888d905ce106c1f198cc85aae210e47b0669c28091d9299db06 and 75e2a133676a82189b10fb1c55b5b32aedc6ede796dd6d8d7c1f901d20ebc7df. This is not the BF16 reference checkpoint.
• Invocation: llama-mtmd-cli --offline --device Vulkan0 --mmproj-device Vulkan0 -m MODEL --mmproj PROJECTOR --image IMAGE -p 'document parsing.' --chat-template deepseek-ocr --temp 0 --repeat-penalty 1.0 --cache-type-v f32 --flash-attn off --no-warmup --fit on -c 8192 -n 1024 --no-perf --log-verbosity 3. One page/image per call.
• Positive smoke: the repository's tools/mtmd/test-1.jpeg yielded The New York Times and MEN WALK ON MOON. In a separately measured run, the same image was embedded on page 2 of a synthetic two-page PDF and rasterized with pdftoppm -scale-to 2000 -png; the OCR returned those markers without page-1 text. This is not a transcription-accuracy score.
• Observed for that page-2 request with the clean binary, including rasterization, weight verification and a fresh model process (filesystem caches may be warm): 13.843 s wall time; backend VmHWM/RSS 2.180 GiB; user-scope memory.peak 2.030 GiB; maximum sampled i915 drm-resident-local0 5.441 GiB (100 ms polling); user-scope swap peak 0. The RAM/VRAM counters use different accounting and must not be added. The generated output was 2,840 bytes, SHA-256 f70253a47dcfe275749f3414114995e5b9ebd2145e0efa075ac4202ce008b13b. CPU and power were not measured.
• Negative smoke, repeated with the clean binary: a separate three-line synthetic PNG (UNLIMITED OCR TEST, Factura numero 2847, Total: 123,45 EUR; SHA-256 ce38a3346fc0637ff8c0ce5dd6f63c9678ba1b0942692152dc9f2ac6e76a050f) produced empty text; our wrapper treated it as a failure. This does not isolate the cause. We can provide the public-safe fixture if useful.

The stock build uses full attention, not this PR's reference sliding-window attention. Would a paired stock-versus-PR run with a public synthetic fixture help this PR?

[sfallah]

@twinforcesl thanks for testing.

I get the same empty output with Q5_K_M on Metal, and on CUDA and Vulkan on a DGX Spark. Your exact Q5_K_M does the same on Metal. I used three-line images with your text.

• bf16 and q8_0 read it, Q5_K_M and Q4_K_M return nothing. Which mmproj doesn't matter for this.
• Q5_K_M is fine on real pages. Your Q5_K_M with a fixed mmproj vs bf16, on Metal: single-view 0.1431 vs 0.1658, dense tables 0.0107 vs 0.0107. That's CER (lower is better) from llama-mtmd-eval, scored against ground truth with pass bands set from the HF model.
• Not R-SWA. The window only kicks in after 128 generated tokens, the empty case ends long before that. Master behaves the same.

The mmproj in that HF repo is missing clip.vision.preproc_min_tiles / preproc_max_tiles, so Unlimited-OCR then uses up to 9 tiles instead of 32:
https://huggingface.co/vimalnakrani/unlimited-ocr-gguf/discussions/1
Q5_K_M with a fixed mmproj: https://huggingface.co/sabafallah/Unlimited-OCR-GGUF

A paired run on a synthetic page won't show the window for the same reason. test-deepseek-ocr.py with bf16 and a fixed mmproj on the Arc would help more. It also needs the DeepSeek-OCR and DeepSeek-OCR-2 GGUFs.

[ngxson]

imo the impl is quite too cumbersome, this is not something we can merge rn

the newly added notion of "latch" is quite out of place, I don't even fully understand how to use it

maybe just simply add a new llama_batch_ext_set_prompt() that is set for prompt token, wdyt @ggerganov ?