[BobClawblaw]

What this changes

Unlimited-OCR-2 (#2074's unlimited-ocr family) gains a CUDA backend: tensorfold-native serve --vision on an NVIDIA GPU serves page images with the release's BF16 weights as published, one stream, greedy under the publisher's page guard, as the Metal engine does. Before this, the family served on Metal only and CUDA refused unlimited-ocr checkpoints.

• Issue: Refs Unlimited-OCR-2 (document parsing, BF16) on native Metal #2073 (the family's issue, Metal; this adds its CUDA backend).
• Stacked on: Unlimited-OCR-2 on native Metal: the publisher's BF16 release, page images to structured text #2074 (head 86857ca4b, on main b331731e6). This adds:
  • 46e00f7d1 the CUDA port;
  • e1d02d708 the router's long-context speedup;
  • f385194e2 the lean rule;
  • f9e0e9b88 to 113b73040 the kernel contract: the scaffold, the five groups and the launch records;
  • cf52f953c a fix to the CUDA open: its refusals after the host is made (no driver, a checkpoint it cannot load) returned null past its errdefers, leaking the host (42,592 bytes) and, on the load path, the context and driver. They now return error.Refused as native/gemma4.zig's do; a checkpoint that fails to load no longer reports a leak;
  • e1e345a25 a refusal of a named --parallel above one ("Unlimited-OCR-2 serves one stream: serve it without --parallel"), as Unlimited-OCR-2 on native Metal: the publisher's BF16 release, page images to structured text #2074's Metal engine refuses it, with a test. Before it, CUDA served one stream whatever --parallel asked.
• atc overlap --json on this head: incomplete (pullFilesUnavailable, timeLimit). It lists 432 open pull requests sharing files, almost all through CHANGELOG.md, the contract registry and zig/build/cuda.zig: 6 with textual conflicts (server: render DeepSeek-V4.x chat with its own encoder and schema prompt #1698, Clef-Flash: joint-schema decisions on /v1/systemone (HIP) #2048, fix(picture): compare PNG tRNS keys by their low byte #2100, server: api.Structure keeps the name a refusal gives it (#2084) #2110, qwen27: canonical prompt chains follow the admitted frame on pre-M5 chips, past 128 rows #2120, glm metal: one arithmetic order for prompt windows, chunks and decode on M1-M4 (#743) #2128), 4 clean, the rest not checked. None touches families/uocr except Unlimited-OCR-2 on native Metal: the publisher's BF16 release, page images to structured text #2074, which this is stacked on.

How it is built:

• zig/kernels/cuda/uocr.cu (with uocr_common.cuh, uocr_vision.cuh and uocr_decoder.cuh) holds the Metal family's 28 own steps, each rounded at the same points as its Metal kernel:

  • the image encoder: SAM's windows, decomposed relative positions and attention, LayerNorm2d, the GELUs, CLIP's embeddings;
  • the decoder: page2d rotary, the DASH router (block keys, page-aware candidates, top 16, entmax-1.5), the block-sparse attention, the MoE gate and combine, one token's matvecs and experts picked on the GPU, and the greedy pick under the page guard.

• The projections run on cuBLASLt through cuda/vit.zig's Gemms (the bias in its epilogue, or fp32 sums and one rounding), as MOSS-Image's CUDA path does. They sum in cuBLASLt's order, not Metal's, so the replies are not bit-equal to the Metal engine's: they part at near-ties (Exactness, below).

• families/uocr/cuda_*.zig and native/uocr_cuda.zig mirror the Metal host: the weights in one device buffer, the encoder, the decoder and the served model, opened from native/cuda.zig beside the registry families.

• The family's host files are shared rather than copied. A file may belong to only one module, so two things move:

  • preprocess.With(picture) takes the picture decoder: core/picture through its relative path on Metal, the core module's on CUDA;
  • the page guard moves to guard.zig (generate.zig re-exports it), so the CUDA module never reaches the Metal decoder.

  The CUDA module is rooted at zig/src/uocr_cuda.zig for the same reason.

• The router change: a decoded token's router scored every 32-key block of the context with one warp, then took the top 16 in 16 rounds, so decode slowed as the context grew. It is now a 256-thread block. Each block's score is one thread's sum, in the same order as before. Each warp takes its own top 16 by shuffles, and warp 0 merges the eight lists in the same order: value first, then the lower block. The reads, their biases and the replies are bit for bit what they were.

Checks on this head

• Head commit: e1e345a25.
• Lean checker, python3 tools/zig/lean_check.py on Linux and macOS: 425 problems on main (b331731e6), 425 on Unlimited-OCR-2 on native Metal: the publisher's BF16 release, page images to structured text #2074's head and 425 on this head. New findings: 0; zig/build/cuda.zig and registry.zig stay at 600 lines. (On Windows the checker reads files as cp1252 and counts 15 more on every tree.)
• Tests run on this head:
  • Linux, RTX 5090 (sm_120): zig build test-cuda-host -Dsm=120 -Dnvcc=...: Build Summary: 307/307 steps succeeded; 1183/1184 tests passed (1 skipped): zig re-ran the test binaries this head changes and took the rest from its cache of 029369d16's full run (4748/4769, 21 skipped). The new refusal test fails with the check disabled. The built server refuses serve --vision --parallel 2 with that message. In the full run three passing steps print to stderr and so show a failed command: line (Dory RoPE's word table, a gfx1030 N/A line, the libtfgrammar warning); none is this change. No leak reports.
  • Linux: zig build native install -Dsm=120 -Dnvcc=... builds.
  • Linux, RTX 5090 (sm_120): tf-cuda-test uocr-contract on 029369d16 (the same kernels and contract files as this head): SUMMARY uocr: 35 kernels, 35 pass gates 1-4, 0 fail; 33 controls: 33 red as planted, 0 missed.
  • macOS, M5 Ultra: zig fmt --check build.zig zig, zig build test, test-golden, test-kernel-contract, test-server-security test-pull-security (24 passed, 0 failed), check-release-paths, fuzz-cortheon, dev-tools and zig build native install: all pass on cf52f953c (this head's parent; the last commit changes only CUDA files) and on Unlimited-OCR-2 on native Metal: the publisher's BF16 release, page images to structured text #2074's head. No leak reports.
  • Earlier base (06322491f on Unlimited-OCR-2 on native Metal: the publisher's BF16 release, page images to structured text #2074's 10fe5c459), Windows 11, RTX 4080 SUPER: test-cuda-host -Dsm=89 285/285 steps; tf-cuda-test uocr-contract gave the same SUMMARY on sm_89.
• CI: pending on the draft.

Receipt

• Environment:
  • TensorFold: this head (e1e345a25), except where marked as the RTX 4080 SUPER's earlier base.
  • Machines: one RTX 5090 (32 GB, driver 615.71, Linux, nvcc 13.3); one M5 Ultra (256 GB, macOS) for the Metal side; earlier, one RTX 4080 SUPER (16 GB, driver 610.88, Windows 11).
  • Toolchain: Zig 0.17.0.
  • Checkpoint: baidu/Unlimited-OCR-2 at revision 0480966938291445591f69f08a13b041c073a96a.
• Paths and platforms:
  • Ran: CUDA sm_120 (Linux) on this head and sm_89 (Windows) on the earlier base: prompt (image encoder, chunked prefill, MoE on gathered rows) and decode (matvecs, on-GPU experts, on-GPU pick), one stream at contexts up to 98,304 on sm_89 and 83,728 on sm_120.
  • Could not run: sm_80, sm_86, sm_121.
• Exactness, measured with the recipe harness's ocr-match plan on the same page PNGs: the recipe's five pages alone, the five as one document, and the release's 15-page paper three times over as one 45-page request.

  • Against the Metal engine (M5 Ultra, Unlimited-OCR-2 on native Metal: the publisher's BF16 release, page images to structured text #2074's code on this head): equal on the two report pages, different on the other five cases. Each engine repeats its own bytes exactly.

    case	completion tokens (CUDA / Metal)	sha256 CUDA	sha256 Metal	equal
    report-1	198 / 198	58c9ad432da973e4	58c9ad432da973e4	yes
    report-2	141 / 141	366f37903b4eec02	366f37903b4eec02	yes
    paper-1	543 / 543	9211a96c01c0dd84	9960608a870adef4	no
    paper-3	911 / 899	44da612e99c36206	357d7be06ce47b96	no
    paper-10	1959 / 1959	8d377476d6ea03c6	66767fb282460f00	no
    five pages as one document	3766 / 3764	f7fc6f672d7c67d1	a5acde0bcdabd6d1	no
    the paper's 15 pages x3 (45 pages)	43035 / 43043	3904790bb7f0ef28	6e4269decbf659ac	no

  • Against the release's own infer (modeling_unlimitedocr.py) on the CPU in torch bf16 (x86, torch 2.13.0+cpu, transformers 4.52.1), greedy, gundam mode: the text without layout boxes is identical in 5 of 6 cases; box coordinates moved in 13 of 284, by at most 2 on the 0-999 grid (the RTX 4080 SUPER on the earlier base: 13 of 284, by at most 5). report-2 and paper-10 are identical.

  • Every place the CUDA reply leaves the reference is a near-tie in the reference. Driving the reference along the CUDA reply and recording its logits after the page guard wherever its own pick (argmax, the lowest id on a tie) differs: 9 such steps over the four cases, and at each the CUDA token is the reference's second choice, either an exact tie (4) or one bf16 step below (0.125 or 0.25, 5). The sixth case's text difference (paper-3, $ [5, 6, …] $ against [5, 6, …]) is one of them: 24.50 against 24.375.

  • A repeated request gives the same bytes, and two requests at once give their solo replies (one stream: they queue). There are no drafts, so there is no "draft": false comparison.

• Decode speed (the harness's streamed decode_tps; tools/bench_openai.py not run):
  • RTX 5090, this head: one page 304-306 tok/s; the five-page document 304.4; the 45-page document's whole 43,035-token reply 275.3.
  • RTX 4080 SUPER, earlier base: one page 241-244 tok/s; the five-page document 240.9; 1,500 tokens after a 40,685-token prompt 223.8 tok/s with the router change, 140.1 before it; after 13,565 prompt tokens 234.7 against 192.4; the 45-page document's whole reply 222.2 against 117.7, byte-identical.
  • main has no CUDA path for this family to compare with.
• Prompt speed, all cold: RTX 5090, first token 0.21-0.36 s for one page (909 prompt tokens, gundam tiles), 1.05 s for five pages (4,525) and 9.39 s for 45 pages (40,685); RTX 4080 SUPER, 0.36-0.44 s, 1.73 s and 15.3 s. tools/bench_cache.py was not run: the family has no prompt cache.
• Licence of new weights: n/a.

Kernel contract, only when a GPU kernel is new or changed

• Kernels registered: all 28 tf_uocr_* kernels in uocr.cu. None replaces a CUDA kernel. The command is tf-cuda-test uocr-contract, with five groups that mirror Unlimited-OCR-2 on native Metal: the publisher's BF16 release, page images to structured text #2074's Metal groups (zig/tests/cuda/contract_uocr_*.zig):
  • rows (9 kernels, 10 registrations): add, layer_norm (SAM and CLIP eps), layer_norm2d, gelu, quick_gelu, rms_norm, rope, router_norm (stated NaN rule: nan_to_num), silu_mul;
  • moves (9): im2col (all four geometries), window, unwindow_add, rel_pos, concat, clip_embed, gather, scatter, embed;
  • route (5, 6 registrations): block_keys, route (picks bit-exact, biases in their ranges, up to 4,096 blocks), moe_gate (stated rule, ties to the lower expert), moe_combine, argmax (stated rule, plain and with EOS barred);
  • attention (2, 4 registrations): tf_uocr_attention as SAM 14x14 windows, SAM global and CLIP, and tf_uocr_attend;
  • matvec (3): matvec at every one-token shape up to the 129,280 x 1,280 lm_head, expert_gu and expert_down through a 64-entry table whose unpicked entries point at NaN rows.
• Served shapes are the Metal groups'. The bounds follow the CUDA arithmetic under the glue flags:
  • division and sqrtf correctly rounded;
  • expf within 2 ulps and logf within 1;
  • the summation depths of each kernel's own order (strided per-thread chains, warp butterflies, warp partials).
• Planted controls: 33, in zig/kernels/cuda/contract_uocr_*.cuh (each a copy of the production kernel with one break), all red as planted:
  • a dropped bf16 point in each rounding kernel;
  • swapped or shifted indices in the moves;
  • ties to the higher block, expert or id, and a short local gap or a lost last page in the router;
  • a shifted relative position, a dropped key, a key past the row, a dropped slot or a doubly counted block in attention;
  • half-precision sums in rel_pos and matvec;
  • wrong experts picked in the routed matvecs.
• SUMMARY uocr: 35 kernels, 35 pass gates 1-4, 0 fail; 33 controls: 33 red as planted, 0 missed. There are no known findings. The tightest passes:
  • kernels checked against a pointwise interval at 1.000 of it (an output on its interval's edge);
  • the bf16-output kernels at 0.988 to 0.994 of the float64 bound.
• Gate 5: the 28 launches are in uocr_cuda_*_records.zig, among cuda_pending. Their nvcc 13.0 usage is pending, because only nvcc 13.3 was available. 13.3's usage on sm_80, sm_86, sm_89, sm_120 and sm_121 is in measurements/uocr_133.zig, from tf-cuda-test kernel-contract-limits on -Dsm=80,86,89,120,121 fatbins: every launch PASS on every SM. The registry's host test checks that usage against all five SMs.
• That kernel-contract-limits run then stops at dsv41_x3ld's sm_121a-only kernel (SymbolNotInImage): this build has no sm_121a image. That is not this change.
• Gates 1-4 ran on sm_120 (RTX 5090, this head) and sm_89 (RTX 4080 SUPER, the earlier base). Chips they have not run on: sm_80, sm_86, sm_121.

Checklist

• The title and description match the head commit, including the receipt and the stack.
• New tests fail before the change, pass after it, and skip cleanly without their dependency. (The contract's 33 planted controls each fail; the production kernels pass.)
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths in the diff or its commits. No AI attribution lines.
• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off.
• Drafted output equals "draft": false (no drafts), a resumed prompt equals a fresh one (no prompt cache), and concurrent equals solo.
• No precision traded for speed. The router change keeps every bit. The CUDA port sums in fp32 as the Metal family does; its projections sum in cuBLASLt's order, not Metal's, so its replies are not bit-equal to Metal's (below).
• Decode and prompt processing are no slower than main at one stream: main does not serve this family on CUDA.

[ashhart]

Thanks @BobClawblaw. Porting all 28 steps with a five-group contract, and planting 33 controls that all go red, is thorough. The router change also keeps its picks bit for bit. This waits on #2074, so we'll review it fully once #2074 lands. Before then:

1. The port claims the Metal rounding op for op, so the bar is CUDA token ids equal to the Metal engine on the recipe's five pages and the 45-page document. We can run the Metal side on an Apple M3/M5 Ultra and post its hashes.
2. Against the torch reference, 13 of 284 boxes moved, and one page differs. For each first differing id, please show the reference's top-two logit margin, so we can see they're ties.
3. Your lean count is 439 on Unlimited-OCR-2 on native Metal: the publisher's BF16 release, page images to structured text #2074's head, while main reads 424. Could you check whether Unlimited-OCR-2 on native Metal: the publisher's BF16 release, page images to structured text #2074 adds findings, or whether the Windows run counts differently?
4. Once it's rebased, we can run the uocr-contract and the served pages on a GB10.

[BobClawblaw]

Thanks. On 3: #2074 adds no findings. On Linux, its merge base (023def309), #2074's head and this head all read 424, with identical finding lists (main is 425 today). The Windows 439 comes from lean_check.py reading files with read_text() and no encoding: on Windows that's cp1252, so each multi-byte UTF-8 character counts as two or three columns and 15 lines go past 120. Forcing cp1252 on Linux gives 439 on all three trees. A one-line encoding="utf-8" fix would make the count match on every platform; happy to open it separately.

On 1 and 2: we'll rebase #2074 and this onto main, then post the CUDA hashes for the five pages and the 45-page document next to your Metal ones, plus the reference's top-two margins at each first differing id.

[ashhart]

Thanks @BobClawblaw, that's a real find. lean_check.py reads with read_text() and no encoding, so on Windows each multi-byte UTF-8 character counts as several columns. Yes please, open the one-line encoding="utf-8" fix as its own PR and we'll land it quickly. The rebase plan for #2074 and this one sounds right, and we'll post the Metal hashes for the five pages and the 45-page document when you post the CUDA ones.

[BobClawblaw]

Rebased both onto main (8db1ee1a3): #2074's head is 4b95ba97f, this one's fa3d50cce. Not pushed yet. The new commit on this branch fixes a leak: when the CUDA open refused after creating its host (no driver, or a checkpoint it can't load), it returned null past its errdefers, and the host leaked (42,592 bytes, plus the context and driver on the load path). Those refusals now return error.Refused like native/gemma4.zig's. The leak report is gone on a checkpoint that fails to load.

1. CUDA against Metal. I ran both sides on the rebased head with the same page PNGs: Metal on an M5 Ultra, CUDA on an RTX 5090 (sm_120, Linux). Each engine repeats its own bytes exactly, but CUDA does not equal Metal:

case	completion tokens (CUDA / Metal)	sha256 CUDA	sha256 Metal	equal
report-1	198 / 198	58c9ad432da973e4	58c9ad432da973e4	yes
report-2	141 / 141	366f37903b4eec02	366f37903b4eec02	yes
paper-1	543 / 543	9211a96c01c0dd84	9960608a870adef4	no
paper-3	911 / 899	44da612e99c36206	357d7be06ce47b96	no
paper-10	1959 / 1959	8d377476d6ea03c6	66767fb282460f00	no
five pages as one document	3766 / 3764	f7fc6f672d7c67d1	a5acde0bcdabd6d1	no
the paper's 15 pages x3 (45 pages)	43035 / 43043	3904790bb7f0ef28	6e4269decbf659ac	no

The differences start at box coordinates one or a few grid points apart, and paper-3 is the $ [5, 6, …] $ near-tie the other way round. So the description's "rounded at the same points, op for op" holds for the family's own kernels but not end to end: the projections sum in cuBLASLt's order, not Metal's, and that is enough to flip near-ties. I'll correct the description. Bit-equality with Metal would need the projections in Metal's summation order on CUDA. Is that the bar you want for this PR, or is agreement with the torch reference within near-ties acceptable for a second backend?

2. Margins against the torch reference. This is the release's own infer (x86 CPU, torch 2.13.0, bf16, greedy, gundam), driven along the CUDA reply. At every step where the reference's own pick (argmax, the lowest id on a tie) differs from CUDA's token, it records the reference's logits after the page guard. There are 9 such steps across the four cases that differ from it. In every one, CUDA's token is the reference's second choice, and the gap is an exact tie or one bf16 step of the logit:

case	step	reference top	logit	CUDA's token	logit	gap
report-1	10	63	45.75	64	45.50	0.25
paper-1	66	135	38.25	134	38.25	0 (tie)
paper-1	72	155	50.25	154	50.25	0 (tie)
paper-1	464	501	40.75	506	40.75	0 (tie)
paper-3	145	[	24.50		24.375	0.125
document	36	175	42.50	176	42.50	0 (tie)
document	451	553	47.25	552	47.00	0.25
document	757	482	34.50	481	34.25	0.25
document	767	506	31.625	508	31.50	0.125

paper-10 and report-2 equal the reference exactly. The x86 summary on this build is the same as the 4080's: text without boxes identical in 5 of 6, box coordinates moved 13 of 284, by at most 2.

3. As above: no new findings. On Linux the rebased head reads 425, the same as main.

Checks on the rebased head: zig build test-cuda-host -Dsm=120: 303/303 steps, 4675/4695 tests passed (20 skipped). The Metal side was built and served for the hashes above, but zig build test on macOS has not been run on this head yet.