You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
test_backend_cross_device: causal conv1d fwd/update flakes intermittently on gfx1200 #3119
test_backend_cross_device.cpp:1459, TEST_CASE("causal conv1d fwd/update match the CPU oracle"), fails intermittently on isravale (RX 9060 XT, gfx1200, ROCm 7.2.3) with a real numerical mismatch, not a crash:
Reproduction: run the isolated case (--test-case="causal conv1d fwd/update match the CPU oracle") repeatedly under the GPU file mutex. Observed 3 failures in 8 consecutive runs (~37.5%), same NMSE value (0.0858971) both times it was captured. Not a timeout, not a launch failure — the kernel runs and returns a wrong number some fraction of the time.
Confirmed unrelated to any in-flight PR: the test case and the GDN conv1d kernel source are both untouched by row/KERNEL-QUANT-CIQ-GEMM-ROCM-IQUANT-w1's reconciliation work (verified present, unmodified, at that branch's pre-reconciliation head 22800690b). Not previously tracked under any issue search for conv1d/ROCm/gfx1200.
Given this project's stated position that no device-side ASan exists for RDNA/gfx12 and that green tests have twice this session alone hidden a real defect in adjacent ROCm kernel work (races, OOB reads), this deserves the same address-arithmetic-level scrutiny rather than being waved off as environment noise — but it is out of scope for the PR that found it, which does not touch this kernel.
Activity
fix(KERNEL-QUANT-CIQ-GEMM-ROCM-IQUANT): repoint the records at the re…
fix(test): MoeSiluMul bf16 compares by NMSE, which is what its title …
fix(vulkan): causal conv1d races its own state write-back across work…
fix(KERNEL-QUANT-CIQ-GEMM-ROCM-IQUANT): repoint the records at the re…