Repository navigation
ROCm gfx1200: Qwen3-0.6B produces wrong greedy output despite all-native execution #269
Description
Row: BACKEND-ROCM
ROCm gfx1200 (RX 9060 XT): Qwen3-0.6B produces wrong greedy output despite all-native execution — embedding gather cleared
Board: AMD Radeon RX 9060 XT, gfx1200 (Navi 44, RDNA4, discrete). ROCm
7.2.3, hipClang/Clang 22.0.0. M0/M1 independently verified on this board (see
my earlier comment).
Repro:
./build-hip/examples/vllm-cli --model <Qwen3-0.6B, bf16 safetensors> \
--prompt 'The capital of france is' --max-tokens 8 --temperature 0- CPU (
--device cpu):Paris. The capital of the United States— correct. - ROCm (default
auto, pickskROCM):1000000— garbage.
No crash, no missing-kernel throw, no reference-tier fallback (discrete board,
none available). VT_OP_PROVIDER_STATS=1 shows every exercised op resolving
selected=vt-native: kEmbedding, kRopeCosSinCache, kCastBf16, kRmsNorm,
kMatmulBT, kQkvSplit, kRopeFromCache, kReshapeAndCache,
kPagedAttention, kSiluAndMul, kGreedyArgmax. All of these have
cross-device coverage in test_backend_cross_device.cpp and pass there against
synthetic data — so this is either a real-shape/real-value edge case none of
that synthetic coverage hits, or a bug outside the op kernels themselves.
Isolated to one token. Capitalizing the prompt ('...of France is') gets
the correct answer ( Paris. The capital of Italy is Rome). Both prompts
tokenize to 5 tokens, identical except position 3: 47587 ("france") vs
9625 ("France"). Same shape, same op sequence — one input token id differs
and the output goes from coherent to garbage.
Ruled out:
- The
is_cuda()-vs-is_cpu()host-pointer-aliasing defect at
dense_attn_block.h:181— already fixed generically, confirmed correct for
kROCMin current source. - The embedding gather itself. Pulled
model.embed_tokens.weightdirectly
from the safetensors file and ranvt::Embeddingon CPU and ROCm at the
real row indices 47587 and 9625 (not a repacked small table — same offset
arithmetic as production). Both rows are bit-exact against the CPU
oracle. The table bytes and the gather kernel are correct for this row; the
bug is downstream, in the per-layer forward.
Not yet root-caused. Candidates: kRmsNorm (cross-lane reduction),
merged-QKV kMatmulBT (hipBLASLt GEMM), RoPE, kPagedAttention (hand-written,
least test coverage), or kGreedyArgmax (its ROCm ArgmaxK NaN handling looks
suspect by inspection — a NaN never satisfies v > best, so a fully-NaN row
would leave arg at a sentinel 0x7fffffff rather than behaving like the CPU
path; unconfirmed as the cause here, but cheap to check first since it's the
last op before the wrong token is picked).
Full writeup, evidence, and next steps: .agents/specs/rocm-gfx1200-m2-correctness.md
(this PR/branch — will link once posted).
Distinct from #201 and #132 — neither reproduced on this board/toolchain.
Does not touch the in-flight Qwen3.5 GDN kernel work (disjoint op set, disjoint
board).
Activity
joral commentedon Aug 10, 2026
Root-caused: this is a near-tie flip, not a kernel defect
Dug through the whole pipeline. Short version: not a bug to fix — the
backend computes the right thing to within ordinary cross-hardware bf16
tolerance, and lost an unusually close near-tie on this specific prompt.
The logits. Final-position top-5, same prompt, same greedy/temp=0:
ĠParis15.1216Ġlocated15.0541Ġ15.0483Ġthe14.9142Ġin14.5216Ġ15.1009ĠParis15.0907Ġlocated15.0620Ġthe14.9496Ġin14.5309Same five candidates, same magnitudes, clustered within ~0.07 logit units of
each other (max logit ≈15) on both backends. CPU resolves toward
Parisbya margin of 0.0675; ROCm resolves toward
Ġ(a bare leading-space token —content-free) by just 0.0102. Picking a content-free token derails the rest of
greedy decode into out-of-distribution territory, which is why the completion
reads as total garbage rather than a merely-different word — the mechanism is
an ordinary near-tie flip, not a broken computation.
Per-layer drift (L2 norm of the last token's hidden row after every
decoder layer): oscillates in a tight 0.01%–0.6% band across all 28 layers,
present from layer 0 (0.27%), no jump, no blow-up, sometimes CPU higher and
sometimes ROCm higher. Textbook ordinary bf16/reduction-order noise
accumulating uniformly, not a localized op bug — full table in
.agents/specs/rocm-gfx1200-m2-correctness.md.Ruled out along the way:
is_cuda()-vs-is_cpu()host-pointer defect atdense_attn_block.h:181— already fixed generically, confirmed correct for
kROCM.model.embed_tokens.weightstraight from thesafetensors file, ran the real op at the real row indices (47587/9625) on
both backends, bit-exact.
layer 0, not a break at any specific op.
Not attempted: the project's own teacher-forcing decisive-measurement
method (
scripts/qwen3-neartie-gap-transformers.py) would be the rigorousconfirmation that the oracle itself treats this as a genuine near-tie — needs
a real vLLM install, not set up on this box. Flagging as environment-blocked
rather than skipping silently; the evidence already gathered (identical top-5
set, sub-1% uniform drift, zero NaN/Inf anywhere) is strong enough that this
isn't treated as blocking closure.
Full writeup:
.agents/specs/rocm-gfx1200-m2-correctness.md(thisbranch/PR).
docs(rocm): record gfx1200 M2 as MET via the native-kernel path
record(rocm): backend-matrix + FEATURES reflect gfx1200 M0-M4
VikashLoomba commentedon Aug 11, 2026
Cross-lane corroboration for the near-tie conclusion — the same phenomenon on two other boards, reported on #41:
'The capital of France is'→Paris. The capital of Italy is Rome(deterministic 3/3), CPU on the same build →Paris. The capital of France is also. Second prompt'The sky is': 8/8 token parity with CPU. Full table.So three independent boards (two discrete RDNA, one integrated), two toolchains, produce the identical ROCm-side sequence while both CPU runs agree — exactly the signature your per-layer drift table predicts (uniform bf16/reduction-order noise, no localized break). One addition to the "why garbage" mechanism: on gfx1100 the flipped token was a content word (
Italy), so the continuation stayed coherent — which is why the gfx1100/gfx1103 runs read as "plausible but different" while yourĠflip reads as garbage. Same coin, two faces.This also bears on the M2 recording question pending on #41: with three boards in the same regime, "strict greedy parity vs the CPU backend" will keep producing attempted-not-met results on near-tie prompts. The distributional gate (ours inside the oracle's K-run set) looks like the honest bar for M2 on ROCm — maintainer's call, flagged in my #41 follow-up.
joral commentedon Aug 11, 2026
I have only seen this on Qwen3-0.6B so far. The larger qwen3 models that will fit my card (I think 1.3B and 4B? something like that) I have not run into this behavior. I haven't spent a lot of time trying to recreate it, but I wonder if it's just something about that particular model or its size.
VikashLoomba commentedon Aug 11, 2026
That size-dependence fits the mechanism: smaller models have flatter top-logit gaps (your 0.6B top-5 were all within ~0.07 units), so greedy near-ties come up far more often than at 1.3B/4B where the top token usually separates clearly. Same reason the gate-model record matters more than any single prompt: at this size some prompt in any set will be near-tied. The K-run distributional gate absorbs exactly this class — a single flipped token isn't a failure, being outside the oracle's own run-to-run set is.
docs(rocm): file the gfx1200 M2 near-tie investigation, link issue lo…
docs(rocm): record gfx1200 M2 as MET via the native-kernel path
record(rocm): backend-matrix + FEATURES reflect gfx1200 M0-M4
joral commentedon Aug 12, 2026
Here is another interesting data point. On the pinned vLLM version, enforce_eager changes the outcome.
Oracle config Output
enforce_eager=True (graphs OFF) ' Paris. The capital of the United States'
enforce_eager=False (graphs ON) ' 1000000'