[totaland]

Stacked on #1431. Only the last commit is new here; the first is #1431's.

Why

#1431 adds Array.q4mv on the CPU. An earlier revision of it also had a Metal kernel, which I removed to keep comp.ts small. Without it, a program that runs on Metal still does every 4-bit product on the host. For a 2B-parameter 4-bit model, the output head alone (248320x2048) costs about 30 ms per decode step on 8 CPU threads.

Change

bend2/comp.ts only.

• When a Metal heap is live and a product has n * cols >= 2^22 weights, the host hands it to a new kernel, bend_q4mv, instead of the CPU loop.
• The kernel runs one thread per row and adds in the Bend body's order, so every row keeps the body's bits.
• Each call commits its own command buffer and waits for it. Smaller products, and programs without a Metal heap, stay on the CPU loop.
• The kernel ships in the program's Metal library, so it needs a program that already builds one (one with a bang).

Verification

Apple M1 Pro, macOS, GPU runs one at a time.

• tests/base/array_q4mv.bend passes on the interpreter, C --gpu off, C --gpu 8GB and JS. Its products are below the threshold, so the Metal lane still runs the CPU loop there.

• A product above the threshold. A scratch program builds a 4096x2048 product (8.4M weights), with a dead bang so the build carries Metal. It runs Array.q4mv 400 times and prints a hash of y's bits. The hash is 1289093127 on Array.q4mv: affine 4-bit matvec primitive (CPU 3.2-3.6x, bit-identical; over comp.ts cap) #1431 (CPU and --gpu 8GB) and on this branch (CPU and --gpu 8GB). Six alternating rounds, milliseconds per call (the 400-call run minus the 1-call run, divided by 400):

  	--gpu off	--gpu 8GB
  Array.q4mv: affine 4-bit matvec primitive (CPU 3.2-3.6x, bit-identical; over comp.ts cap) #1431	2.28	2.29 (no kernel)
  this PR	2.36	0.30 (0.27–0.50)

  The Metal time includes a command buffer and a wait per call.

• Model head. In a microbench, the 248320x2048 head takes 5.44 ms on this kernel. In a Bend inference engine (TensorFold), the same head takes 30.7 ms on the CPU with 8 threads.

• Whole model. In that engine, Qwen3.5-2B (MLX 4-bit) greedy decode with this kernel gives the same token IDs and printed logits as the CPU. On its own it does not speed up decode: --gpu 8GB had a median of 127.5 ms/step over 6 pairs, and the CPU 127 in separate runs. Most products are below the threshold, and every GPU product waits on its own. Array.q4mv sums in a fixed tree order, so NEON and one Metal SIMD group per row keep the body's bits (stacked; over comp.ts cap) #1436 and Array.q4go/Array.q4wait queue Metal Q4 products in one command buffer and wait once on a flag (stacked; over comp.ts cap) #1437 build on this.

• Regression. All 186 tests that upstream 60fa05dc has in tests/base and tests/io give byte-identical interpreter, C and JS output on upstream and on Array.q4go/Array.q4wait queue Metal Q4 products in one command buffer and wait once on a flag (stacked; over comp.ts cap) #1437, the top of this stack. The only difference is the temp path in 21 JS build-error messages. 120 of 186 match their #| lines on both trees. The new q4 tests pass on all three lanes.

uvx ttok@0.3 < bend2/comp.ts: 64,660 (#1431) → 66,466 (+1,806).

Limits / open

• Over the comp.ts cap, as Array.q4mv: affine 4-bit matvec primitive (CPU 3.2-3.6x, bit-identical; over comp.ts cap) #1431 is. See the question on Array.q4mv: affine 4-bit matvec primitive (CPU 3.2-3.6x, bit-identical; over comp.ts cap) #1431.
• One thread per row reads the weights strided, so the kernel stays well below memory bandwidth. Array.q4mv sums in a fixed tree order, so NEON and one Metal SIMD group per row keep the body's bits (stacked; over comp.ts cap) #1436 replaces it with one SIMD group per row.
• The 4M-weight break-even is fixed. Each call pays about 0.25 ms of command-buffer overhead.
• Tested only on an M1 Pro. No CUDA path. Linux is untested.

[nicolas-abril]

Thanks @totaland for the careful work and measurements on this stack.

We're closing #1435, #1436 and #1437 for the same reasons as #1431:

• They are stacked on Array.q4mv: affine 4-bit matvec primitive (CPU 3.2-3.6x, bit-identical; over comp.ts cap) #1431, which is closed, and each adds more to bend2/comp.ts, which is already over its token cap. The top of the stack is at 68,289 tokens, about 4,300 over the 64,000 cap.
• A matvec specific to the 4-bit format, and now its Metal kernel, NEON loops and q4go/q4wait API, belongs in user code, not in Base.

One difference from #1431: there, plain Bend on main already matched the CPU speedup. The measurements here suggest stock bangs don't yet match a dedicated kernel on Metal (the ~11.5x head and ~15 vs ~11 tokens/s). That gap is worth closing, but with a general mechanism (faster bangs for this kind of loop, or a general async GPU dispatch/wait) rather than one specific to the 4-bit format. If you want to explore that, please open an issue first so we can agree on the design.