Repository navigation
fix: make the Gemma3n xla-diagnostics probe pass on CUDA past the weight-carrier and AltUp stages #2808
Description
Problem / Background
PR #2801 (issue #2795) made the eager gelu_topk kernel and the XLA tape bit-exact on the bf16 carrier and validated them with pinned_e2b_real_projection_qmv_probe on CUDA (GB10). But the probe as a whole still aborts on CUDA before it can demonstrate the full sparse pipeline on a real checkpoint. Two failures, both pre-existing and recorded in the PR #2801 body; this issue groups them because they share the probe's verification surface and one probe run covers both.
-
The weight-carrier check pins MLX CPU dequant semantics but the probe executes MLX on GPU. The Rust diagnostic
dequantize_affine_bf16_sequentialreproducesmx.dequantizeon MLX CPU exactly: 0 mismatches in the oracle venv (~/.cache/mlxcel-epic-2348/oracle-cuda, CPU device). When the probe runs MLX on GPU, every 4-bit weight's carrier check fails with about 51% 1-ulp mismatches. The comparison is CPU-reference vs GPU-execution, so it fails by construction on CUDA and abortspinned_e2b_real_projection_qmv_probebefore anylayer0_sparse_*stage runs. -
The AltUp prediction/correction coefficient stages differ by f32 noise. With item 1 bypassed (assertion temporarily relaxed during fix: Align Gemma3n gelu_topk bf16 rounding across eager, XLA, mlx-lm #2795 validation, then reverted), the probe reaches the AltUp stages and they fail strict bit comparison: prediction/correction coefficient differences of about 5.6e-6/1.5e-5 in f32 cascade one bf16 ulp into the bf16 predictions. This is f32 arithmetic noise in the coefficient math, not a formula error, but the strict comparison cannot tell.
Proposed Solution
Implement both fixes in one PR with a single verification pass at the end and as few pushes as possible.
- Make the weight-carrier check device-consistent: either build the dequant reference from the same device the probe executes on (GPU vs GPU), or make the check explicitly 1-ulp tolerant on GPU with the measured mismatch rate recorded. The check must still fail on a real dequant bug (a wrong scale/bias/group layout produces far more than 1-ulp differences), so state in the PR how the relaxed bound still catches that.
- Bring the AltUp coefficient comparison under the feat(xla): define operator-level MLX/IREE numeric contracts #932 per-op same-input parity rule (docs and precedent in the repo): record the measured f32 noise, and give the AltUp prediction/correction stages an explicit per-op threshold or 1-ulp tolerance with the measured value as its justification. Stages that are genuinely bit-exact on GPU (the
layer0_sparse_*gelu stages, verified 0 mismatches in fix: align Gemma3n gelu_topk bf16 rounding across eager, XLA and mlx-lm #2801) keep their strict comparison.
Acceptance Criteria
-
pinned_e2b_real_projection_qmv_probe(--features xla-diagnostics,IREE_CUDA_HOME,GEMMA3N_MODEL_DIR=models/gemma3n-e2b-4bit, GB10 undergpu-lock,--test-threads=1) runs to completion on CUDA without aborting at the weight-carrier check - Every
layer0_sparse_*stage andlayer0_sparse_activation_manual_vs_compiledstill reports bit_mismatches=0 (the fix: align Gemma3n gelu_topk bf16 rounding across eager, XLA and mlx-lm #2801 result is not weakened) - The AltUp stages pass under the feat(xla): define operator-level MLX/IREE numeric contracts #932-based threshold, with the measured f32 noise recorded in the PR
- The weight-carrier check still fails on a deliberately wrong dequant (a broken reference or a negative control in a test), or the PR states why that is not feasible and what bounds it instead
- The probe result on CPU (if the probe supports a CPU path) is unchanged
Verification
IREE_CUDA_HOME=$(bash scripts/iree/setup-cuda.sh --env-print IREE_CUDA_HOME) GEMMA3N_MODEL_DIR=models/gemma3n-e2b-4bit cargo test --profile test-fast --features xla-diagnostics --lib pinned_e2b_real_projection_qmv_probe -- --ignored --nocapture --test-threads=1Build unlocked; run only the printed test binary under gpu-lock run --tag gemma3n-probe -- <binary> --test-threads=1. The IREE CUDA tree at ~/.cache/mlxcel/iree-cuda-3.12.0rc20260721 is verified working on spark-101 (the "spark-101 has no IREE install" note in #2795 is stale).
Activity
pinned_e2b_real_projection_qmv_probe aborted on CUDA before any spars…