Repository navigation
The test and perf gates run ! on Metal only; CUDA-specific behaviour goes unnoticed #1779
Description
What Bend should do
Run the CUDA lane in the gates, even on one small machine running the ! tests, so CUDA-specific changes and regressions show up before a release.
Why
The test gate builds every runnable test with Metal and runs its ! on the minis, and the perf gate pins its GPU cells on apple_m4; GitHub Actions runs only repo-gate. Nothing runs the CUDA lane, and it differs from Metal in ways the gates cannot see: managed memory over PCIe instead of unified memory, the shared memory reserved at launch, register counts and occupancy, and NVRTC's code generation. Three things I found on an RTX 4050 Laptop (Bend 2.0.37 and main 881cb2e) that no gate would report:
- the device loops' error poll costs up to 20x on tight loops on CUDA (CUDA: the error poll in every flat loop (
err_spun) makes ptxas yield on each iteration; tight loops run up to 20x slower #1776); - Metal: non-tail recursion under a bang faults at 1023 list elements; CPU completes through 100000 #1393's lane-stack fault reproduces on CUDA at the same depths (my comment on PR Metal: a full lane stack spills to the heap, so non-tail recursion under ! no longer faults at 1023 frames (#1393, #1394) #1439: Metal: a full lane stack spills to the heap, so non-tail recursion under ! no longer faults at 1023 frames (#1393, #1394) #1439 (comment));
- from 2.0.37 to today's main, which has PR bend_dev on CUDA declares vote[3] with 0 dynamic shared memory, so SMs keep their L1 cache #1602, CUDA kernels moved by 0.92-2.2x, both ways: 2M reads a leaf of a 32-number Array 2.02 -> 0.92 s, a tiled 512x512x4096 matmul 216 -> 234 ms (idle machine, 5 alternated rounds).
Activity