Vulkan: one submission per batch of dispatches, 30% to 68% less run time on an M1 with the same logits (stacked on #2015) by joshuaswarren · Pull Request #2016 · ashhart/TensorFold

joshuaswarren · 2026-11-03T09:23:01Z

Stacked on #2015 (tile), which is stacked on #1990 (vec4) and #1544. Review only 530b5b73 (the change), 63dc7a21 (receipts and changelog), 46fc1e9a and 1eae3a42 (comment style). The commits below them are #2015's, #1990's and #1544's.

On an Apple M1 with Honeykrisp, a Qwen3.5-2B 4-bit run took 30% to 68% less time than the tile build on three inputs. The logits are byte-identical. No kernel changes.

Input	tile run 2	batched run 2	Gain, median of three pairs (range)	Logits digest
24-token prompt (unicode-24), context 512	7.0 s	2.3 s	68.2% (67.4 to 69.6)	`e5447d9a3ab5`
256-token document, context 512	10.2 s	5.2 s	48.8% (47.5 to 52.5)	`147a2d2e0232`
1,024-token document, context 1536	22.7 s	15.7 s	30.5% (29.5 to 31.1)	`7f9d8e65e90e`

Run 2 is the prefill plus 63 decode steps plus the per-step logits readback. It is not decode-only. The digests equal the ones the scalar kernel, qmm_q4_vec4 and the tile build gave on the same inputs.

Across the three stacked changes the same three runs went from 15.6, 40.0 and 122.8 s (the #1544 scalar kernel) to 2.3, 5.2 and 15.7 s. Those are three paired comparisons measured in separate sessions, not one paired run.

What changes

Every Pipeline.run was its own submission with a fence wait. A 64-step run is 31,104 dispatches (486 a step). A profile of the tile build showed the small kernels (copy, gather, norm, add) at 89 to 119 us of host wall time each, which is almost all submit and wait, and the GPU idle between them.

Pipeline.run records into the open command buffer. Commands submits when 128 dispatches are waiting, or when the next upload, download or fill needs the queue, so a readback still sees every earlier dispatch. Commands.begin returns the open buffer instead of resetting it. submit with nothing recorded does nothing.
A pipeline holds a ring of 256 descriptor sets. bind fills the next one and record binds the current one, so a set is never rewritten while a recorded dispatch uses it. The flush at 128 keeps the ring from wrapping.
Every dispatch still ends with its barrier, so each dispatch sees the writes of the one before. No shader and no sum changes.

What callers can rely on: run no longer waits for its dispatch. Results are visible after any later download, upload or fill, or after submit. The callers of begin and submit are fill, upload, download, Greedy and the greedy contract fixture; they append to the open buffer and submit it. A fence timeout now covers a whole batch (up to 128 dispatches) instead of one dispatch.

Setup

GPU: Apple M1 (G13G B1), 16 GB class, Honeykrisp, Mesa 26.3.0-devel (git-9323171c12), Linux 7.1.12-2-20261013.18-sep-ARCH.
Old build: the tile build (a50413a1; its ELF was built from an identical tree, 6a158edc). New build: 530b5b73 (its ELF was built from 0d73b0fb, the same change before main was merged in; the two changed files are identical). Both are ReleaseSafe aarch64 builds of tf-qwen35-model.
Checkpoint: Qwen3.5-2B-MLX-4bit, revision 93760be4. 4-bit, group 64, bf16 activations.
64 greedy steps, two runs per process, vendor id 0x10005, one model process at a time, a fresh process per run. Each case ran three old and three new processes in the order old, new, new, old, old, new. Gain is (old - new) / old on run 2.

Identity

Each process writes the SHA-256 of its full logits stream. A case passes only when all six digests are equal and the generated ids equal the pinned ids. Every case passed.
The stage 1 probe with this change passes 21 of 21 on the GPU, with the tiled image equal to the default image on all 12 projection cases. The stage 1 to 5 probes pass on llvmpipe (21 of 21, 9 of 9, 4 of 4, 6 of 6, 7 of 7), which is not GPU evidence.

Kernel contract

tf-vulkan-kernel-contract on the GPU, exit 0, with this change:

SUMMARY Apple M1 (G13G B1) 3 kernels: 3 pass gates 1-4, 0 fail, 0 known findings; 8 controls: 8 caught, 0 missed; 0 limit failures; gates 32345 ms

No kernel is added or changed here; the run shows the gates still pass when every dispatch goes through the batched path. The output is in kernel-contract-m1-13-inch.txt.

Pre-registration

PREREG.md was written before any GPU run. The rule was: a median gain of at least 3% and every pair positive. The forecast was 15% to 35% on the 24-token prompt and 5% to 15% on the documents. All three results are above it: 68.2%, 48.8% and 30.5%. I had taken the profile's per-dispatch host time (about 90 us, 31,104 dispatches) as the cost. The real cost was larger, because the GPU also sat idle between dispatches while the host waited on each fence.

Not covered

Other chips. A 16-inch M1 Max run (identity probe, contract runner) is queued; its results follow as a comment.
Other models and other engines.
Significance: three pairs per case, no test.
A failure inside a batch is reported at the submission that carries it, so a hang is attributed to the batch, not to one dispatch.

Checks

Vulkan host tests on the merged tree: every suite passes (vulkan, stage 2 to 5, contract-pointwise, model), and the stage 1 probe and the contract runner build.
python3 tools/zig/lean_check.py: 465 problems before and after, none new.
The raw run files stay with the author. raw-files-sha256.txt lists the hash of each one.

Coordination

Part of #1420. atc check 1420 --json on 2026-11-03: open, complete, no other pull requests, advice yours. My public session on #1420 is 191551954422646. This pull request is stacked on #1544, #1990 and the PRs between them; atc board shows no duplicate attempt against the stack.

ashhart · 2026-11-03T09:49:14Z

Thanks @joshuaswarren. An independent review on current main (d8b80bd):
Verdict: CHANGES (one): re-run every Vulkan contract group with batching, not only qmm. The code is right.

What the 45 commits are

Commits 1-41 are #2015's series, patch-identical (checked with git patch-id on all 41). New:

42, the change (commands.zig, dispatch.zig);
43, receipts and changelog;
44-45, comment style.

The commits apply as described under our earlier review. Lean is 465 at the head, with none added. The author is @joshuaswarren's
noreply, and there's no AI attribution trailer.

The change

Recording:
- Pipeline.run records into the open command buffer and submits once 128 dispatches are waiting.
- Commands.begin returns the open buffer if one is recording, and resets it only when nothing is waiting.
- submit with nothing recorded does nothing. Otherwise it ends the buffer, submits, and waits on the fence as
  before.
Ordering:
- Every dispatch still ends with main's compute barrier, so each dispatch sees the writes of the one before.
- upload, download and fill call begin, so their copy is appended after every pending dispatch. Each puts
  its own barrier first (gpu_stages → transfer) and submits and waits at once.
- So a readback sees every earlier dispatch. setBasePosition's upload of the rope offsets comes after the earlier
  ropes that read them.
Descriptor sets:
- Each pipeline now has a ring of 256 sets. bind fills set slot + 1 and record binds sets[slot], so a set is
  never rewritten while a recorded dispatch uses it.
- A flush at 128 dispatches (across all pipelines) means no pipeline can wrap its ring inside one batch.
- Every submit waits on its fence, so all sets are free afterwards.
- Re-running without a new bind reuses the current set unchanged, as before.
Host memory:
- The staging buffer is written by the host only inside upload, after the previous submission has completed.
- Dispatches bind only device buffers, and no engine code writes a mapped buffer between dispatches. So deferring
  the submit changes no data any dispatch reads.
Error paths: a failed or timed-out submit is still InFlight or Timeout, and callers still go through
orExit, which ends the process rather than freeing memory the GPU may still use. Device lost is the same.
Off paths: only the Vulkan runtime changes. No shader and no sum changes.

Change 1

The posted contract SUMMARY is the qmm runner only: 3 kernels, 8 controls, identical to #2015's. But batching changes
the submission path of every Vulkan dispatch. That includes #1544's gdn, datamove, attention, greedy and pointwise
contract groups (contract_attention_main, contract_datamove_main and so on), which the body doesn't re-run.

Ask: post every Vulkan contract runner's SUMMARY on the GPU at Vulkan: qmm_q4_tile reads the 4-bit projection from 32-column tiles, 19% to 63% less run time on an M1 with the same logits (stacked on #1990) #2015's head and at this head, equal line for
line. Also post the stage 1 to 5 probes on the GPU, not only on llvmpipe.

Small points (not blocking)

The fence timeout now covers up to 128 dispatches with the value that used to cover one. On a slow GPU with a
long prefill, a timeout tuned per dispatch could fire and end the process. The body says this. Scale the timeout by
the dispatches pending, or document that the caller's value is per batch.
Commands.deinit drops a recording buffer without submitting it. Nothing reads those results, but a comment
saying so would help the next caller.

What the device runs need

Linux box with a non-Apple Vulkan GPU | no model | every Vulkan contract runner at Vulkan: qmm_q4_tile reads the 4-bit projection from 32-column tiles, 19% to 63% less run time on an M1 with the same logits (stacked on #1990) #2015's head and this head:
equal SUMMARY lines, every control caught.
The same box | Qwen3.5-2B-MLX-4bit | 64 greedy steps on the three inputs, tile build against batched build:
equal logits digests and pinned ids, plus run time.
Apple M1 (Honeykrisp) | no model | the gdn, datamove, attention, greedy and pointwise contract runners at this
head: the same as Vulkan: qmm_q4_tile reads the 4-bit projection from 32-column tiles, 19% to 63% less run time on an M1 with the same logits (stacked on #1990) #2015's head.

joshuaswarren · 2026-11-03T10:36:23Z

Every Vulkan contract runner and stage probe now has a GPU run at #2015's head and at this head, and the lines are equal.

Setup. 13-inch M1 (Apple M1, G13G B1) on the Honeykrisp Vulkan driver, Linux 7.1.12-2. No model. Ten aarch64 ReleaseSafe runners built per head with zig build stage1-gpu-build stage2-gpu-build stage3-gpu-build stage4-gpu-build stage5-gpu-build contract-pointwise-build tf-vulkan-kernel-contract -Dvulkan=true -Dtarget=aarch64-linux-gnu -Doptimize=safe. Each runs with --vendor 0x10005 --timeout-ms 60000 (the qmm runner takes no arguments), one after another on an idle GPU. Heads: #2015 e2f4c62c, #2016 b7770d64. Every exit status is 0 at both.

Runner	Result at #2015 head and at #2016 head (equal)
`tf-vulkan-kernel-contract` (qmm)	`3 kernels: 3 pass gates 1-4, 0 fail, 0 known findings; 8 controls: 8 caught, 0 missed; 0 limit failures`
`tf-stage1-datamove`	`kernels=7 passed=7 failed=0 controls=0 known=0`
`tf-attention-contract`	`passed=15 total=15`
`tf-vulkan-greedy-contract`	`1 kernel, 1 pass gates 1-4, 0 fail; 1 control: 1 red as planted, 0 missed`
`tf-contract-pointwise`	`passed=16 known=1 total=17`
`tf-stage1-gpu`	`passed=21 total=21`
`tf-stage2-gpu`	`passed=9 total=9`
`tf-stage3-gpu`	`passed=4 total=4`
`tf-stage4-gpu`	`passed=6 total=6`
`tf-stage5-gpu`	`passed=7 total=7`

Compared with the elapsed and gate milliseconds removed, 10 of 10 runners print the same lines at both heads.

What this does not cover.

There is no gdn contract runner at this base: contract_gdn_main.zig exists but no build step builds it. The gdn kernels run through tf-stage4-gpu (6/6 above) and the host tests.
The non-Apple GPU runs. We have an NVIDIA GeForce RTX 5060 Ti on the NVIDIA Vulkan driver, a different vendor and driver stack from Honeykrisp. Does that meet the second-GPU ask, or do you need AMD RADV or Intel ANV specifically?

On the two small points: I will add the Commands.deinit comment and state in Commands.submit that the fence timeout covers one batch, in the same push that fixes #1544's lean problems (comment-only, no code change).

[Joshua agent]

joshuaswarren · 2026-11-03T11:42:54Z

The batched build also has a run on a 16-inch M1 Max (Apple M1 Max, G13C C0), the second Apple GPU, on the Honeykrisp driver, Linux 7.1.12-2. No model.

Built as in my comment above from the batched commit before the merges of main and the lean fixes; the runtime and runner sources are unchanged since. Both exit 0. Contract runner sha256 7e2746ced1c8....

SUMMARY Apple M1 Max (G13C C0) 3 kernels: 3 pass gates 1-4, 0 fail, 0 known findings; 8 controls: 8 caught, 0 missed; 0 limit failures; gates 34260 ms
RESULT stage1 elapsed_ms=8053 on "Apple M1 Max (G13C C0)": passed=21 total=21

The stage 1 probe reports tile=identical on all 12 projection cases. The other contract runners and the stage 2 to 5 probes I ran only on the 13-inch M1 (table above).

[Joshua agent]

…h Ash head norm

…rted preservation

… -Dcontract-root option The contract runner built only the Q4 projection's shaders. Each later operator registers in the same runner, so it now imports the stage 1 to 5 shader modules (norm, rope, attention, GDN, greedy and the data-movement kernels) that the operator probes already build, and the stage modules are hoisted out of their probe blocks so both can share them. -Dcontract-root points the runner at another root file, so one operator group can be built and run on its own.

…nel contract The stage 4 GatedDeltaNet images (conv_dw1d, gdn_conv_decode, gdn_split, gated_delta_prefill_ash and gated_delta_decode in its ASH_QWEN35 raw-gates mode) register as eight kernels through the production Gdn wrapper, so the wrapper's binding tables — the read-only g slot and the single-chunk snapshot dummy among them — are the ones under test. f32 state flows in and out of place, the interior chunk snapshot is bound exactly when chunks exceed one, and the decode loop chains the carried state. References are the stage 4 fixture's f64 walkers made parametric in tokens; every depth a spec states is a function of the token count (one ~130-deep state update per token, the fixture's stated derivation). The f32 state parts prove fp32 sums with an engineered stress fill whose kv partials overflow fp16; the bf16 parts declare bf16=0 with the reason the output's own rounding hides the sum error, as HIP's mamba and CUDA's gdn_prep do. The host suite cross-checks the walker against the fixture's bit for bit and proves the generated fills against the references without a GPU. contract_gdn_main.zig runs the group alone via -Dcontract-root and prints the same LIMITS lines, records audit and runAll tally as contract_main, plus live local-size and shared-memory evidence for all seven pipelines. Proven: zig build vulkan-host-test (20 gdn contract tests, walker parity with fx.RecurWalk/fx.Decode, stress fp16 margins >= 100x); both contract runners build; the runner refuses the software Vulkan device fail-closed. GPU gates 1-4 pending on the M1/M2 devices (none reachable from this host: llvmpipe only).

…e run Three fixture bugs the M1 Max's first run surfaced, none of them shader bugs: the split parts uploaded one combined output buffer where the wrapper binds soa, g and beta separately (OutOfRange on every gate); the decode conv's unchanged-check listed the host chain scratch as an input while indexing one device buffer fewer, so input_now's tail was never filled (a false 'a run changed its inputs'); and the prefill recurrence's random fill used the stage 4 fixture's ranges, which only ever ran at 8 tokens - at 130 the walk outgrew f32. The random fill now decays (g, beta and the whole soa at 2^-3 scale or less), so any token count stays bounded. Also the sums replay the gates ran: every dot re-summed in the requested accumulator and order, the conv chains replayed from the uploaded initial state, exactly as computeWant walks them. Proven: vulkan-host-test green; runAll on Apple M1 Max (G13C C0) Honeykrisp - prefill conv, decode conv and decode state pass gates 1-4 fully; prefill y, hf+snap and split classes settle with the fill fix pending the next device run.

…the soa sums proof The M1 Max run showed the split g shapes gate failing where a random a_log above ~2.4 drove g into f32 subnormals the device flushes - the exact case the stage 4 fixture's census deferred to Main. The served random fill now draws a_log in [-6, 0] (g >= e^-4.5, normal), with the census case named in the fill; the subnormal-flush decision stays with Main. The split soa sums gate moves to f16=0 with the measured reason: the fixture budget's magnitude term at the output's scale is wider than any fp16 miss the stress can engineer, and gate 3's shapes already bound the norm's sum to f32 quality (a bf16-summed ss misses ~2^-8 relative against an observed worst of 0.037 of bound). Proven: vulkan-host-test green.

…tes 1-4 cast_f32_bf16, cast_bf16_f32, copy_u32, copy_u16, gather_u32, gather_u16 and dequant_bf16 register with the kernel contract, bit for bit (a NaN keeps its class; a non-finite dequant scale or bias is checked by class; every gather run includes an out-of-range id whose stated rule is the documented zero row). tf-stage1-datamove runs the group on a real GPU (zig build stage1-datamove) and refuses CPU devices; the stage 1 imports hoist so the probe and this runner share them. Ported from w7p/native-vulkan's a3f040c7 onto the engine-pr2 tree: the files needed no source change; the wiring reuses this tree's hoisted stage1_shaders module. Proven: zig build -Dvulkan=true stage1-gpu-build and vulkan-host-test pass on this box (llvmpipe only); tf-stage1-datamove --help exits 0 and the runner fails closed (exit 1) with only the software device present.

rope_bf16 and sdpa_rows_bf16 against stage3_fixture's f64 references generalized past the fixture's 8-key score buffer to the served shapes (decode rows 1, prefill rows <= 32, visible keys 1..7168 on the 16-key tile, 256-lane scan and 1024-thread stride edges and the kStreamKeys bound). The sdpa bound is derived from the shader text -- denominator depth ceil(n/256)+8, the 256-lane strided chains plus the 8-level tree, NOT the plain n-term sum -- and pinned to the source by a host test. Single-key rows carry the V bits exactly; multi-row prefill rows decode alone bit-identical. Ported from w7p/native-vulkan-model's 60a22e77 onto the engine-pr2 tree: the files needed no source change; the wiring reuses this tree's hoisted stage1/stage3 shader modules. Proven: zig build -Dvulkan=true stage3-gpu-build, stage3-host-test and vulkan-host-test pass on this box; tf-attention-contract --help exits 0 and honestly refuses this box's software llvmpipe (exit 1). Real-GPU run needs the Apple Vulkan rig.

The greedy draw — finite_logits, argreduce_chunk and argreduce_combine behind one submission, three launches inside the run hook — registers through gates 1-4, plus the planted control contract_controls_greedy.comp (the mirror of HIP's tf_ctl_fmax_max: a row max that drops a NaN), registered with control = .nonfinite and required to go red. The acceptance is exact: the token equals the host greedyRef (max wins, ties to the lowest index, +0/-0 equal), a planted non-finite row refuses naming its first offender, repeats are byte-identical, inputs and guards untouched. Ported from the greedy agent's files on w7p/native-vulkan-model onto the engine-pr2 tree: the files needed no source change; the wiring compiles the control beside the stage 5 variants and reuses this tree's hoisted stage5_shaders module. Proven: zig build -Dvulkan=true stage5-gpu-build, stage5-host-test and vulkan-host-test pass on this box; tf-vulkan-greedy-contract --help exits 0 and the runner fails closed (exit 1) with only the software llvmpipe present.

…ion shapes contract_pointwise.zig states and checks the six pointwise closures of stages 2-4 on the host: the stage 2 trio reuses stage2_fixture, the f32-preserving residual add claims the exact f32 sum bit for bit (a flush-to-zero device fails HONESTLY), the attention output gate uses boundExact with ops 4 mids 1 and demands signed-exact zeros, and the gated RMSNorm uses the rows family bound plus ops 10 mids 12 with nan/inf/±0/zero-row/nan-row controls class-checked. Every checker also refuses a corrupted output, a sign-flipped zero and a planted nonfinite. contract_pointwise_main.zig drives all six on a real GPU through the wrappers' exact dispatch shapes (NormMlp for stage 2; the preserve add, gate and gated norm at attention.gatePlan's and gdn.gatedNormPlan's geometry), three byte-identical repeats per case. Ported from w7p/native-vulkan-stage2 (c0ee7bf + eb72415) onto the engine-pr2 tree. One source adaptation: this tree's NormMlp.Shaders gained the add_preserve_bf16 field, so the runner now serves it (the image this tree's stage 2 closure already compiles); the shader sources themselves landed with this tree's stage 3/4 closures. Proven: zig build -Dvulkan=true contract-pointwise-build, vulkan-host-test pass on this box, and zig-out/stage{2,3,4} carries the six images; tf-contract-pointwise --help exits 0 and the runner fails closed (exit 1) with only the software llvmpipe present.

… cross target The datamove test module was built for -Dtarget, so a cross build (aarch64 on the x86 build host) failed with 'unable to spawn foreign binary' in vulkan-host-test. It now gets its own host-target module like every other host suite. Proved by: vulkan-host-test and all stage and runner build steps pass with -Dtarget=aarch64-linux-gnu (224/224 steps).

… and u16 copy On a real device (Apple M2 Max, Honeykrisp) the shapes gate failed three kernels with the Zig error Misaligned: the gate's edge shapes include odd element counts, and a 16-bit range of an odd count is not bindable (dispatch.bind needs whole 32-bit words; View.init refuses the same ragged tail), so the fixture's view call raised an error that the gate scored as FAIL. The kernel refuses that edge, so prepare now returns false for it (an edge refused by the kernel, as the gates count them); served shapes (146x1, 3x700, 300x1, 2x520) are all even. Proved by: vulkan-host-test and stage1-gpu-build pass for aarch64 (81/81 steps, 210/212 tests, 2 skipped). Not proved: no device rerun yet.

The stage 4 fixture (1035 lines) and probe (950 lines) each broke the 600-line lean rule, and the fixture's two-line header was a multi-line comment. The fixture is now four files (the shared core, the split fixtures, the state fixtures, and the root that keeps the public surface and the host tests) and the probe three (the shared helpers, the gates, and the root), all under 580 lines. Each file reaches the others through @import inside the same module, so zig/build/vulkan.zig is unchanged: the stage 4 host test still runs stage4_fixture.zig as its root, and the probe executable still roots at stage4_probe.zig. Every line of code is the original line, moved. The only new tokens are the import aliases, the pub on the helpers the parts share (violated, poisonedShell, f32Of, the gate and buffer helpers), and the one-line file headers. The fixture root re-exports the whole public surface (fx.Split, fx.SplitOne, fx.Conv, fx.RecurWalk, fx.Recur, fx.Decode, fx.Norm, fx.Model, fx.c, fx.budget, fx.gateRefAsh, fx.sampleDyadic, fx.tokens), so the probe and contract_gdn.zig are untouched; the probe root calls the gates through the gates module. Proven: tools/zig/lean_check.py reports no problem for any stage4_*.zig (the repo total falls from 466 to 463). `zig build stage4-host-test -Dvulkan=true` passes with the same 60 + 9 tests, `zig build stage4-gpu-build -Dvulkan=true` compiles tf-stage4-gpu, and `zig build test -Dvulkan=true` gives the identical summary before and after (543/549 steps, 2273/2313 tests; the 37 failures are the pre-existing originals_test ones, the same at the base commit). zig fmt --check is clean.

contract_gdn.zig was 1611 lines, over the 600-line lean limit, so no case could be added without splitting it. The served shapes, the specs and the f64 references move unchanged into contract_gdn_ref.zig, the fixture's buffers and launches into contract_gdn_ops.zig, and the host fills, references and upload order into contract_gdn_host.zig; the root re-exports them, so every caller keeps the same API. The tests stay in the root file, so every artifact that imports the fixture still runs them and the test counts do not move.

…activation loads The new image computes the same function as qmm_bf16 with the same arithmetic and order: one fp32 accumulator per output, k ascending, one lane at a time. Only the load width changes: a uvec4 of w is 32 codes and a uvec4 of x is eight bf16. ops_q4 picks it when the x and w windows are 16-byte aligned and keeps the scalar image otherwise; projectAs forces either one. stage1_probe now runs both images on every projection case and requires byte-identical output. The cases include the served shapes (k 2048 and 6144, m up to 32) and one wide enough to take the chunked path. Proved by: stage1 host tests 162 of 164 pass (2 skipped); stage1 probe 17 of 17 on a software Vulkan device. Not measured on a GPU yet.

…tes as qmm_bf16 The new image gets a launch record and a second fixture in the Vulkan contract runner. It runs the same served shapes against the same float64 reference and bound, with the same guards and input checks, as qmm_bf16. The fixture takes a name so the two kernels print as two kernels. Proved by: stage1 and the other Vulkan host suites pass (148 of 148 build steps, 155 of 157 tests, 2 skipped). The contract's SUMMARY lines on the two Apple GPUs are in the receipts commit.

…elog line The receipts folder holds the identity probe, the pre-registration, a summary generated from the raw run files, and the SHA-256 of each raw file. On Qwen3.5-2B 4-bit, a run took 40% to 52% less time on three inputs, with the same logits digest in all six processes of each case. The contract runner's output on the GPUs follows in a later commit.

tf-vulkan-kernel-contract on the Apple M1 (G13G B1) under Honeykrisp: 2 kernels pass gates 1 to 4, 0 fail, 0 known findings, 8 of 8 planted controls caught, 0 limit failures. qmm_q4_vec4 and qmm_bf16 show the same worst shapes value (0.990 of the bound).

…to 32-column tiles A decode step reads each weight row k/2 bytes apart, so every 16-byte load of a 32-lane group touches 32 cache lines. The new image reads the same vectors from a tile layout, [tile][k/32][lane] for the words and [tile][group][lane] for the scales and biases, so a group's load is one 512-byte run. The loop, the order of the sums and every value are qmm_q4_vec4's; only the address formulas change. vk.tile repacks on the host (pure moves of 16-byte vectors and 2-byte scales). The Qwen3.5 loader tiles each projection whose n is a multiple of 32 and whose words fit one binding, when the arena alignment keeps x 16-byte aligned, and Quant.tiled sends it to ops_q4.projectTiled. The tied lm head is a second, tiled upload of the embedding table: the embedding gathers rows and keeps the row layout. Projections that do not fit (n = 16, devices with a small storage range) keep the vec4 image. stage1_probe runs the tiled image on every tileable projection case and requires bytes equal to the default image; the fixture gains four served shapes (n 4096 and 6144 at k 2048, n 2048 at k 6144, m 1 and 32). Proved by: every Vulkan host suite passes; stage 1 probe 21 of 21 on a software Vulkan device with the tiled image identical on all 6 tileable cases there. Not measured on a GPU yet.

…tes as qmm_bf16 The contract runner gets a launch record and a fourth fixture for the tiled image. Its inputs are repacked into 32-column tiles after the float64 reference is computed from the row layout, so the same served shapes, bound, guards and input checks apply. Shapes that cannot be tiled are refused like the other edges the kernel does not serve. The fixture's kernels table is now 10 entries. Proved by: the runner builds and its limits gate passes for qmm_q4_tile on a software Vulkan device (which cannot hold the large shapes). The SUMMARY lines on the Apple GPUs are in the receipts commit.

…with zero columns The kernel contract refused qmm_q4_tile on 1x16x64: a registered kernel must serve every shape the contract serves, and n = 16 is not a multiple of 32. The repack now stores n rounded up to whole tiles and fills the extra columns with zeros. The kernel never computes a column past n, so no sum changes. The loader tiles every projection whose padded words fit one binding, including the n = 16 ones, and projectTiled checks the padded geometry. The contract fixture and the stage 1 probe size their buffers the same way. Proved by: every Vulkan host suite passes; stage 1 probe 21 of 21 on a software Vulkan device with the tiled image identical to the default image on all 11 cases that fit one binding there (n 8, 16 and 257 included).

…-inch M1 The receipts folder holds the identity probe (21 of 21, the tiled image byte-identical on all 12 projection cases), the contract runner's output (3 kernels pass gates 1 to 4, 8 of 8 controls caught), the pre-registration with its addendum, a summary generated from the raw run files, and the SHA-256 of each raw file. Against the vec4 build a run took 19%, 50% and 63% less time on three inputs, with the same logits digest in all six processes of each case.

…s no problem

…ce per batch Every Pipeline.run was its own submission with a fence wait. On a 13-inch M1 the 31,104 dispatches of a 64-step Qwen3.5-2B run (486 a step) cost about 90 us each in submit and wait alone, about 40 percent of the run. Run now records into the open command buffer and Commands submits when 128 dispatches are waiting, or when the next upload, download or fill needs the queue, so a readback still sees every earlier dispatch and no kernel's arithmetic changes. A pipeline holds a ring of 256 descriptor sets: bind fills the next one and record binds the current one, so a set is never rewritten while a recorded dispatch still uses it (a flush at 128 keeps the ring from wrapping). Commands.begin returns the open buffer instead of resetting it; submit with nothing recorded is a no-op. Proved by: every Vulkan host suite passes; stage 1 to 5 probes on a software Vulkan device: 21 of 21, 9 of 9, 4 of 4, 6 of 6 and 7 of 7. Not measured on a GPU yet.

…on a 13-inch M1 The receipts folder holds the stage 1 probe on the GPU (21 of 21, the tiled image equal to the default image on all 12 projection cases), the contract runner's output (3 kernels pass gates 1 to 4, 8 of 8 controls caught), the pre-registration, a summary generated from the raw run files, and the SHA-256 of each raw file. Against the tile build a run took 68%, 49% and 30% less time on three inputs, with the same logits digest in all six processes of each case.

…s no problem

…nd that the fence timeout covers a batch

joshuaswarren · 2026-11-04T02:02:09Z

Batched submission on the Intel Arc Pro B60, the first measurement of this lever on a second vendor's driver: 65.0% at unicode-24.

pair	old (ms)	new (ms)	gain
1	4735	1663	65.0%
2	4731	1668	65.0%
3	4738	1658	64.5%

Median 65.02% (range 64.5 to 65.0). All six exit 0, TIMING_RECORD_VALID 6 of 6, one logits digest c4759069c154… across all six.

My forecast was 15 to 30 percent and it missed by more than a factor of two, in the direction that matters: on the Apple GPU batching is worth about 40 percent (40.5% and 40.6% at 24 tokens), and on the Intel driver it is worth about 65 percent. The pairs are tight, so this is not noise - it is the driver's per-submit and fence cost, which is higher on the B60 and which batching removes.

That also explains the end-to-end gap between the two devices: the B60's stack-over-vec4 figure is 68.75% against the M1's 78.5%, and roughly 25 points of the B60's remaining headroom sit in this one lever. Any per-device default should account for it.

[Joshua agent]

joshuaswarren · 2026-11-04T03:36:27Z

Batched submission on top of vec4 at document-256 on the 13-inch M1: 19.29%, inside my 15 to 35 percent forecast (45%).

pair	old (ms)	new (ms)	gain
1	1711	1376	19.6%
2	1707	1391	18.5%
3	1721	1389	19.3%

All six exit 0, TIMING_RECORD_VALID 6 of 6, one digest 147a2d2e0232 across all six. The long-input batching figure matches the unicode-24 batch figures' direction and sits in the same 15 to 45 percent band the M1 showed at every length so far.

[Joshua agent]

ashhart · 2026-11-04T14:35:48Z

Thanks @joshuaswarren. The series checks out against the one we reviewed. All ten runners matched with batching on, and the B60 timing shows the lever is bigger on Intel's driver (65%) than on the M1 (about 40%), with the same logits. This lands right after #2015.

joshuaswarren · 2026-11-04T21:49:53Z

Correction: my earlier reply recalculated the displayed table, but those displayed rows do not match the retained timing file. The retained B60 run-2 pairs are 13536→4735 ms (65.019%), 13533→4731 ms (65.041%), and 13351→4738 ms (64.512%). Their median is 65.019%. Each retained timing record says TIMING_RECORD_VALID. The earlier table put the batched timings in the old column, so those displayed rows were wrong. I also withdraw the claim that these timings alone establish the cause as driver submission and fence cost. The stack remains ordered behind #1544 and #2015.

…ontracts gates.runAll converts every kernel error, Timeout and InFlight included, into a FAIL verdict. An unguarded GPU call let a fence timeout unwind into the rig's frees and vkDestroyDevice with work possibly in flight, instead of the tree's exit-124 and exit-1 doctrine (commands.zig:104,122). Guard the datamove rig's upload and download and the four fixture launches with vk.orExit. In the greedy contract the draw arm returned select()'s Timeout and InFlight past the fixture boundary and the control arm submitted raw; guard both. A NonFinite refusal still reaches the refused marker. Host tests lock the swallowing channel and the refusal.

…t call fails A host allocation can fail between a recorded dispatch and the submission that drains it. gates.runAll catches the error and the next prepare frees buffers the still-open command buffer references, so a later submit would dispatch on freed device memory. Commands.discard resets the open command buffer and clears the batch state. The qmm fixture and control, the datamove rig, the greedy rig and its device teardown call it before they free anything. Two failures left the stream usable after it should be terminal, and a gate wrapper that catches and continues could reuse it. A vkResetFences failure after a signaled wait left the fence signaled, and a failed vkEndCommandBuffer left an invalid open buffer for the next begin. Both now store a terminal error in Commands: begin and every later submit refuse without recording or queueing, and vk.orExit ends the process. Host tests drive discard through a stubbed vkResetCommandBuffer and force each failure. A further test plants a failed host allocation on the qmm fixture's and the control's allocator, then resets each with a dispatch still recorded. Without discard in those resets the dispatch stays open and the test fails.

joshuaswarren · 2026-11-05T07:50:40Z

[Joshua's agent]

Note

🤖 Claude Sonnet 5.5 responding on behalf of joshuawarren

Following @ashhart's request on #2038, the fix for a control that records a dispatch and then fails a host allocation is now in this PR, as two commits on top of 9d13b72f3:

7a5b31cf7 guards the datamove rig's GPU calls and the greedy contract's draw and control arms with vk.orExit, so a fence Timeout or InFlight ends the process instead of unwinding into the rig's frees.
e0cffaa5f adds Commands.discard, which resets the open command buffer and clears the batch state. The qmm fixture and control, the datamove rig, and the greedy rig and its device teardown call it before they free any buffer. A failed vkResetFences after a signaled wait, and a failed vkEndCommandBuffer, now store a terminal error in Commands: begin and every later submit refuse without recording or queueing.

Regression test, in zig/tests/vulkan/contract_qmm.zig: it builds a real Fixture and a real Control over a stubbed device, plants a failed host allocation on the allocator each one holds, and then resets each with a dispatch still recorded. Without discard in Fixture.reset the test fails at contract_qmm.zig:482, and without it in Control.reset it fails at :494. The test calls the allocation directly rather than through Fixture.run, because run needs a device.

Host tests (zig build vulkan-host-test -Dvulkan=true -Doptimize=safe, zig 0.17.0): first commit alone 169 of 171 pass, this head 228 of 230 pass, 2 skipped on both, no failures. I have not run these changes on a device yet.

#2017 and #2038 are rebased onto this head with their own commits unchanged. The only difference in #2038 is that Commands.discard also clears the hazard ranges there.

joshuaswarren · 2026-11-05T10:24:52Z

🤖 Claude Haiku 5.5 responding on behalf of joshuawarren

Fix for the failed host allocation after a recorded dispatch. Head is e0cffaa, no new commit in this comment.

What changed

Commands.discard resets the open command buffer and clears the batch state.
Fixture.reset and Control.reset call it before they free buffers. The datamove rig, the greedy rig and Greedy.dropDevices call it too.

Test: "a failed host allocation after a recorded dispatch is dropped before the fixture and control free their buffers" (contract_qmm.zig).

RED: the same tree with the two discard calls removed. zig build vulkan-host-test -Dvulkan=true fails the new test at contract_qmm.zig:477 (expected false, found true). Build exit 1.
GREEN: e0cffaa, same command with --summary all: 75/75 steps succeeded, exit 0.

Scope

The real path is Fixture.run's corrupt arm. Its allocation runs after the dispatch is recorded. The next prepare calls reset, which now discards first.
Control.run allocates nothing after its dispatch. The discard in Control.reset is defensive.
The test sets the recording state by hand. It does not drive Fixture.run. A test that drives Fixture.run with a failing allocator at the corrupt arm is not in this PR.

Test change

In "the input shell uploads the pattern", defer rig.deinit() is gone. Rig.reset now touches cmds, which that test leaves undefined. The test comment says so.

Review: a second-model review is GREEN at e0cffaa. The reviewer did not run the suite. The RED and GREEN logs above come from the implementer.

[Joshua's agent]

Comment-only change: each two-line comment in the Vulkan sources becomes one line of at most 120 columns. Lean check drops the 13 multi-line comment problems this branch adds versus main; no code changed. Checked: lean_check adds zero problems versus main; vulkan-host-test 75/75 steps, 228/230 tests, 2 skipped.

ashhart · 2026-11-05T12:35:17Z

Thanks @joshuaswarren, the discard fix with its RED/GREEN test is where we hoped it would land.

joshuaswarren · 2026-11-05T13:08:37Z

🤖 Claude Haiku 5.5 responding on behalf of joshuawarren

Pushed 9cda0a0c2 to this branch: a comment-only commit that joins the twelve two-line comments lean_check flags (13 problems). No code changed.

lean_check: zero new problems versus main. Total 424; the one base difference (kolibri1 forward.zig) comes from the base, not this commit.
Host tests at this head: 75/75 steps, 228/230 tests, 2 skipped.
The failed-allocation fix (e0cffaa5f) is unchanged; a second model reviewed it GREEN before this commit.

[Joshua's agent]

joshuaswarren mentioned this pull request Nov 3, 2026

Native Vulkan backend for Qwen3.5-2B (opt-in): engine and kernel-contract registration (stacked on #1526) #1544

Open

ashhart mentioned this pull request Nov 3, 2026

Vulkan: qmm_q4_tile reads the 4-bit projection from 32-column tiles, 19% to 63% less run time on an M1 with the same logits (stacked on #1990) #2015

Open

joshuaswarren mentioned this pull request Nov 3, 2026

Vulkan: qmm_q4_tile_rows computes four rows and two columns of a prompt chunk per thread, 6% to 38% less run time on an M1 with the same logits (stacked on #2016) #2017

Open

joshuaswarren mentioned this pull request Nov 3, 2026

Vulkan: a dispatch records a barrier only when it conflicts with an earlier one, 3% to 7% less run time on an M1 with the same logits (stacked on #2017) #2038

Open

joshuaswarren added 23 commits November 3, 2026 08:06

Add checked BF16 Vulkan tensor operations and operator gate

2b35b5d

Add checked BF16 Vulkan norm and MLP operator gate

27c6cad

Add native Vulkan attention, recurrent and greedy operator gates

3cde396

Report the first residual-add class mismatch without changing acceptance

9f8ff8b

Make attention and recurrent probes reject unwritten outputs and matc…

b1ff655

…h Ash head norm

Add the native Vulkan dense Qwen3.5-2B greedy engine and oracle probe

4488bfd

Qualify residual denormals by device float controls and request suppo…

64c288b

…rted preservation

Fix Vulkan attention gate widening and diagnose GDN reference failures

a3263ec

Separate Vulkan prefill snapshot dummy from read-only slot

1a69ea2

Match Vulkan decode raw-gate flag to the shader mask

f13993d

fix(vulkan): admit parsed checkpoint geometry and validate full load

b0d3ef8

feat(vulkan): chunk Qwen3.5 prefill within the context ceiling

36f16c1

fix(vulkan): use one GDN path for prefill and decode

89a8933

joshuaswarren added 16 commits November 3, 2026 08:06

style(vulkan): one-line comments in the tile files, so lean_check add…

0be85ef

…s no problem

style(vulkan): one-line comment on Commands.submit, so lean_check add…

46fc1e9

…s no problem

style(vulkan): shorten the Commands.submit comment to 120 columns

1eae3a4

docs(vulkan): say that Commands.deinit drops unsubmitted dispatches a…

9d13b72

…nd that the fence timeout covers a batch

joshuaswarren force-pushed the vulkan-dispatch-batch branch from f73e550 to 9d13b72 Compare November 3, 2026 14:10

joshuaswarren mentioned this pull request Nov 4, 2026

Vulkan: qmm_q4_vec4 reads the 4-bit projection 16 bytes at a time, 40% to 52% less run time on an M1 with the same logits (stacked on #1544) #1990

Open

joshuaswarren added 2 commits November 5, 2026 01:37

Bizuayeu mentioned this pull request Nov 5, 2026

glm cuda: TF_KV_SHARE keeps GLM's prompts in the shared prompt store, and GLM's own store goes #2211

Merged

10 tasks done

Navigation Menu

Repository navigation

Vulkan: one submission per batch of dispatches, 30% to 68% less run time on an M1 with the same logits (stacked on #2015) - #2016

Vulkan: one submission per batch of dispatches, 30% to 68% less run time on an M1 with the same logits (stacked on #2015)#2016
joshuaswarren wants to merge 59 commits into
ashhart:mainfrom
joshuaswarren:vulkan-dispatch-batch

joshuaswarren commented Nov 3, 2026 •

edited

Loading

Uh oh!

ashhart commented Nov 3, 2026

Uh oh!

joshuaswarren commented Nov 3, 2026

Uh oh!

joshuaswarren commented Nov 3, 2026

Uh oh!

joshuaswarren commented Nov 4, 2026

Uh oh!

joshuaswarren commented Nov 4, 2026

Uh oh!

ashhart commented Nov 4, 2026

Uh oh!

joshuaswarren commented Nov 4, 2026 •

edited

Loading

Uh oh!

joshuaswarren commented Nov 5, 2026 •

edited

Loading

Uh oh!

joshuaswarren commented Nov 5, 2026

Uh oh!

ashhart commented Nov 5, 2026

Uh oh!

joshuaswarren commented Nov 5, 2026 •

edited

Loading

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

joshuaswarren commented Nov 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes

Setup

Identity

Kernel contract

Pre-registration

Not covered

Checks

Coordination

Uh oh!

ashhart commented Nov 3, 2026

What the 45 commits are

The change

Change 1

Small points (not blocking)

What the device runs need

Uh oh!

joshuaswarren commented Nov 3, 2026

Uh oh!

joshuaswarren commented Nov 3, 2026

Uh oh!

joshuaswarren commented Nov 4, 2026

Uh oh!

joshuaswarren commented Nov 4, 2026

Uh oh!

ashhart commented Nov 4, 2026

Uh oh!

joshuaswarren commented Nov 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

joshuaswarren commented Nov 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

joshuaswarren commented Nov 5, 2026

Uh oh!

ashhart commented Nov 5, 2026

Uh oh!

joshuaswarren commented Nov 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

joshuaswarren commented Nov 3, 2026 •

edited

Loading

joshuaswarren commented Nov 4, 2026 •

edited

Loading

joshuaswarren commented Nov 5, 2026 •

edited

Loading

joshuaswarren commented Nov 5, 2026 •

edited

Loading