Repository navigation
Eval bug: LightOnOCR-1B produces degenerate output ("@@@@...") via mtmd — regression since #16764 #25652
Description
Environment
- macOS (Apple M5, arm64), tested on two builds:
- Homebrew release:
version: 9960 (a935fbffe), AppleClang 21.0.0 - Fresh
masterbuild from source:14d3ba4(CMake Release,-DMTMD_VIDEO=OFF)
- Homebrew release:
- Same behavior with Metal (
-ngl 99) and CPU-only (-ngl 0)
Model
Official GGUF conversion: ggml-org/LightOnOCR-1B-1025-GGUF (Q8_0 + mmproj-...-Q8_0), added in #16764. Community GGUFs of LightOnOCR-2-1B show the same symptom (different repeated token).
Reproduction
llama-mtmd-cli -m LightOnOCR-1B-1025-Q8_0.gguf \
--mmproj mmproj-LightOnOCR-1B-1025-Q8_0.gguf \
--image SROIE-receipt.jpeg -p "Transcribe" --temp 0.2 -n 60Test image: the public transformers OCR fixture (SROIE-receipt.jpeg). Any document image reproduces it.
Observed
Output degenerates immediately into a single repeated token, regardless of input:
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Also reproducible via llama-server (/v1/chat/completions, image as base64 data URL) using the input structure described in #16764 (empty system message, user message containing only the image), with and without --jinja, at temperature 0 and 0.2. Server log confirms the image is processed (prompt eval ~2325 tokens).
Expected
A transcription of the document. The same image through the reference implementation (transformers v5, LightOnOcrForConditionalGeneration, upstream lightonai/LightOnOCR-1B-1025) produces a correct transcription — the weights are fine, the mtmd path is not. Per the description in #16764 this worked at merge time (Oct 2025), so this looks like a later regression.
Possibly related log line
load: control-looking token: 128247 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
Activity
renealexanderrath-design commentedon Jul 19, 2026
Root cause: regression from #20962 (new Metal Tensor-API
kernel_mul_mm)Bisected this to #20962 ("metal : optimize Metal Tensor API usage for GGML_OP_MUL_MAT"). It's a Metal-only regression in the new tensor-API
kernel_mul_mmpath (gated byGGML_METAL_HAS_TENSOR).Mechanism. LightOnOCR's patch embedding is a
conv_2d→im2col+mul_matwhose contraction dim isK = patch·patch·channels = 14·14·3 = 588. Since588 % 32 != 0, the pipeline setsbc_inp = true. In the new tensor kernel the A operand is staged into threadgroup memory with zero-padding for the K-tail, but the B operand (tB) is read directly from device memory, and the last partial K-tile (tB.slice(loop_k, …)withloop_k = 576,K = 588) reads out of bounds past K. The zero-padded A doesn't save it, because0 (A) * NaN (B, OOB) = NaN. The whole image embedding becomes NaN → the LM emits a single repeated token.Verified (Apple GPU, a local build of
14d3ba4):@@@…GGML_METAL_TENSOR_DISABLE=1(legacy kernel): correct transcription-DGGML_METAL=OFF): correct transcriptionmul_mmkernel.An eval-callback confirms the first NaN node is the patch-embed
MUL_MAT, with both inputs clean (no NaN/Inf, normal value ranges). Note:-ngl 0does not avoid this, because the vision encoder still runs on Metal — a true CPU-only build does.Fix direction: bound / zero-pad
tB's K-remainder tile in the tensor kernel (as the A-staging path and the legacy kernel already do), or fall back to the legacy kernel whenbc_inp. Workaround today:GGML_METAL_TENSOR_DISABLE=1.github-actions commentedon Sep 3, 2026
This issue was closed because it has been inactive for 14 days since being marked as stale.