Repository navigation
Conversation
…CPU and one kernel thread per row on Metal, bit-identical to the Bend body
…ne falls back to the body The C lowering is now one function: eight rows side by side, each row its own chain of adds in base.bend's order, writing only the rows asked. The Metal kernel, its host dispatch and the scalar tail macro are gone; inside a bang the same loop runs on the device. comp.ts goes from 66,465 to 64,660 ttok (ttok 0.3; upstream 63,905). The collector now reads intrinsics per lane (intr_of(k, FL.js)), so a C-only intrinsic keeps its body on the JS lane. Before, the interpreter and JS builds failed with "$Array$q4mv$ is not defined". The emitted JS of every other test is byte-identical. tests/base/array_q4mv.bend checks 13 rows (a block of eight and a tail of five) and rows 3..8 alone against an independent float32 reference, on the interpreter, C and JS lanes.
totaland
changed the title
|
A question before I take this further. This draft can't pass the repo gate as it stands, and I'd like your direction rather than guess.
I also found that plain Bend on stock Which would you prefer?
I'll keep this as a draft until you've had a look. @VictorTaelin |
|
Thanks @totaland for the detailed write-up and for asking before going further. We'll go with option 1 and close this. comp.ts is over its token cap with the change, and since plain Bend on The per-lane intr_of fix is real, but no Base op needs it today, so we'll leave it until a C-only intrinsic |
What
Adds
Array.q4mv(w, x, y, +r, +n, +cols, +rows), a matvec over affine 4-bit weights with group size 64. OneArray<U32>holds the codes, then the BF16 scales, then the BF16 biases. Activations are oneArray<F32>with the per-group sums stored after thecolsvalues. Each output row is rounded to BF16 and written toyat its index.bend2/base.bend: the pure Bend body defines the result. It sums each row group by group, left to right, then rounds to nearest-even BF16.--check-onlypasses.bend2/comp.ts: lowers the call to one C function, the same wayArray.cloneis lowered. It sums 8 rows side by side, and each row keeps its own add chain in the body's order. Clang vectorizes across rows, so every output keeps the body's bits. Only the rows asked for are written. Inside a bang, the same loop runs on the device.bend2/comp.ts: the collector now reads intrinsics per lane (intr_of(k, FL.js)). A C-only intrinsic therefore keeps its Bend body on the JS lane, and the interpreter and JS builds run the body.tests/base/array_q4mv.bend: 13 rows (a block of 8 and a tail of 5), then rows 3..8 alone. Expected bits come from an independent float32 reference. Slots that the call must not write keep their old value.Why
A Bend inference engine for a 2B-parameter 4-bit model spends its matvec time in one dependent F32 add chain per row. That is about 3 cycles per weight, or about 12 G weights/s on 10 cores. The same loop in hand-written C runs at the same speed, so codegen is not the limit. Reordering the sum would change the bits, so this PR uses independent chains across rows instead.
Results
Apple M1 Pro, macOS. Each case ran 3 palindromic rounds (stock, patched, patched, stock), with milliseconds per call measured in-process after equal warm-up. Outputs matched stock and a float32 Python reference on every run. The 1-minute load average was 40 to 47 during these runs, so treat the spreads as wide.
Bend's Mandelbrot and tree-matmul benchmarks do not call the primitive. Their outputs match stock on CPU and Metal. CPU medians are 966 vs 964 ms and 676 vs 679 ms (stock vs patched, 3 ABBA rounds). Metal fast-slot times are also equal: Mandelbrot 218-232 vs 207-233 ms and tree-matmul 1654-1735 vs 1646-1733 ms.
Every other test that imports Base produces byte-identical JS (1,281 tests). Its C output is byte-identical apart from the added function.
Limits / open
bend2/comp.tsis 64,660 ttok (ttok 0.3, as CI uses) against the 64,000 cap. Upstreammainis 63,905, so this change adds 755 tokens. No behavior-preserving cut of that size was found elsewhere incomp.ts. The gate forbids moving code out of the file, so this PR cannot merge as is.main, with no compiler change. A generated Bend kernel that sums 8 rows per pass, with one tuple per group, measured 3.0-4.1x and bit-identical in the same runs. That may make this primitive unnecessary in Base.Array.get.