Repository navigation
npu: the forward is causal-only, so a noncausal_full_attention pack (d3-lite) cannot be reproduced — measured, not inferred #847
Description
What
vllm-sr/d3-lite (the pack engine #796's route serves) declares attention_mode: noncausal_full_attention in its
decision_config.json. The pack's own runtime honours it: d3_runtime.PromptEncoder runs the language model with no
causal mask on the full-attention layers (the 18 GDN layers keep their recurrent mask). The engine has no such mode —
q4nx_forward.cpp only ever builds a causal mask — so a noncausal pack is served by a causal forward.
Measured (2026-11-02, strixhalo, host-GEMM, no device opened, reference = the pack's d3_runtime.py in a clean venv)
Running the reference both ways on the same pack (encoder on/off, one CPU run, nine contract requests) shows this
is not a cosmetic difference:
| row | result |
|---|---|
| reference noncausal vs reference causal | worst |Δp| 0.2869 (score_three_levels), choice argmax 4/5 |
And the engine's disagreement with the noncausal reference cannot be attributed to weights (it survives with the
weights held identical) or to readout scaling (the 255-way code distribution has 0/10 top codes in common and
Spearman −0.15, i.e. an unrelated ranking):
| comparison | worst |Δp| | choice argmax |
|---|---|---|
| engine(q4_1) vs reference(q4_1-rounded weights) | 0.7202 | 2/5 |
| engine(bf16) vs reference(bf16) | 0.8382 | 3/5 |
| reference sensitivity control, 9 % weight noise | 0.2636 | 3/4 |
The intuition that "last-position pooling makes causal and noncausal identical" is wrong, and the measurement is what
shows it: earlier positions' full-attention outputs differ, those feed the GDN recurrence, and the last position's
hidden inherits the difference.
What it would take
attention_mode(ornoncausal: true) carried by the plan (npu/plan/plan.h,tools/npu_plan.py) and derived from
layer_types+decision_config.json— a plan field, not another filename probe.- The full-attention path honouring it for a prompt-only forward (
step_batch/stepwith no KV reuse): a bidirectional
mask for the layers whoselayer_typesentry isfull_attention, with the GDN path untouched. Decode is causal by
construction, so a noncausal pack is a prompt-encoding mode, not a generation mode — which is also why it should be
refused, by name, when a request would use it for generation. - A test that says the mask is what it claims: same weights, same prompt, noncausal vs causal must differ (the reference
row above is the known answer), and a causal model must be byte-identical to today.
Validation is device-free: NPU_INFER_HOST_GEMM=1 on the 0.8B-GDN pack, compared against d3_runtime.py with
~/lb/d3parity (reference 10 requests in 7.7 s, engine side under one npu-hold ticket).
Not claimed
This is one cause of the parity gap, not the whole of it — 0.29 of the measured 0.72. After it lands the row must be
re-measured; the next isolation step is an engine-side last_hidden dump compared against the reference's hidden.
Activity
bong-water-water-bong commentedon Nov 2, 2026
Triaged and verified at
origin/main906f966a2(this box's pin, which already carries #846 — the bf16 converter that made the "same weights" arm cheap). Every claim below is a read of the tree or of the pack's own file, not a restatement of the report.The absence is confirmed, and it is an absence of two things, not one.
git grep -n 'attention_mode\|noncausal\|non_causal' origin/main -- npu/forward/src npu/plan tools/npu_plan.py→ no hits. The onlycausalstrings in the forward are the depthwise-conv comments (npu/forward/src/q4nx_forward.cpp:1854,:2156), i.e. the GDN path, not attention. Andnpu/plan/plan.h's v1 schema (header comment:model/container/fusion/prefill/decode/kernels/moe/layers) has no attention block at all, so item 1 of "what it would take" is not started rather than half-done.The declaration is in the pack, and so is the control.
~/models/npu/d3-lite-src/decision_config.json→{"attention_mode": "noncausal_full_attention"}(same value ind3-lite-conv/,d3-lite-bf16/). A causal sibling from the same converter already sits on this box:~/models/npu/d3-synth-0.6b/decision_config.json→{"attention_mode": "causal"}. That is the byte-identical control item 3 asks for, without generating anything.One constraint on the plan field:
plan.haccepts exactly one format string and refuses anything else by name ("a newer/older file is refused by name, never interpreted"). Soattention.modemust be an optional field defaulting to causal inside1bit-npu-plan/1, not a version bump — existingconfigs/plans/*.plan.jsonmust keep loading unchanged.Two causes, not one, for the 0.72/0.84 rows. The same pack also carries #849 (zero-point nibbles paired by halves while the codes pair even/odd), so neither number can be attributed to the mask alone. Order: #849 (weights) → this (mask) → re-measure. Both are device-free, and the reference side already exists here as
~/lb/d3parity/compare*.py.Owner pi-npufeat (it owns #796 and #832); bucket A. Closing acceptance, as the issue states it, is falsifiable in one host run: noncausal vs causal on d3-lite differing by at least the reference's own 0.2869,
d3-synth-0.6bbyte-identical to today, and a generation request against a noncausal pack refused by name.npu plan/forward: attention.mode, so a noncausal_full_attention pack …
bong-water-water-bong commentedon Nov 3, 2026
Measured again on fixed weights: the mask is mechanically right, but its magnitude is downstream of a bigger forward gap — and the numbers now say which
Re-ran the two arms on the bf16 container with engine#875's decoder fix (PR #882), so nothing here is the mis-decode that
made my earlier #861 rows unreliable. Two serves, one
npu-holdticket, host-GEMM,amdxdnaerrors 0. Reference arm =the pack's own
d3_runtime.pyon the same pack, both ways (itsencoderon/off is exactly this mask).1. The engine's causal arm is already 0.64 away from the reference's causal arm
Measured over the nine contract cases (
~/lb/d3parity/mask_cmp.pyagainstref_causal.json):noul_criteria: ref 0.0762 vs engine 0.7130)The first row is the one that matters for this issue: with the mask out of the picture entirely, the engine and the
reference disagree by 0.64. So the 0.73 in the noncausal row is not the mask, and the acceptance's
≥ 0.2869cannot beevaluated until the causal forward is faithful. Tuning the mask to reach 0.2869 would be fitting a number to a
compensating error, which is the one thing this issue should not accept.
2. The mask's mechanism is confirmed layer by layer — it is not silently dropped
NPU_INFER_DUMP_LAYERSon both arms, same prompt, at the last prompt position (layer kinds from the pack'slayer_types):That is exactly the signature the mechanism requires, and it is worth having on record: at the last position the
first full-attention layer's own output cannot change (its query has no future keys either way — its keys come from
layers 0–2, which are causal and identical), so +0.00 % at layer 3 is correct, and the effect appears from layer 4
on only through the GDN recurrence carrying the earlier positions' changed states forward. The engine reproduces that
path; what it does not reproduce is the size of it (≤1.5 % on the residual, against the reference's 7–25 % on the pooled
hidden measured earlier).
3. So what is left for this issue
do next, and the mask magnitude will follow it rather than lead it.
attention.mode, the two-pass fill, the generationrefusal and the causal path's byte-identity to the previous behaviour are all in from npu: attention.mode, so a noncausal_full_attention pack (d3-lite) is not answered causally (#847) #861 and behave as designed.
mask_cmp.py+layers_arms_cmp.py), andthe issue can then be closed on the magnitude with a number that means something.
Not closing: the acceptance's magnitude is unmet, and now with a stated reason rather than a mystery. Harness:
~/lb/d3parity/{mask_arms.sh,mask_cmp.py,mask_layers.sh,layers_arms_cmp.py}, artifactseng_eng_{causal,noncausal}_fixed.json,layers_{causal,noncausal}.log.bong-water-water-bong commentedon Nov 3, 2026
Short update, measured on fixed weights (bf16 container, causal + noncausal arms, one
npu-holdticket each step,amdxdnaerrors 0): the arm this issue's acceptance depends on moved a long way, and the acceptance still is not met —for a reason that is now a known, separate defect rather than an unexplained gap.
engine#796 found the cause of the causal arm's 0.6367: Qwen3.5's RMSNorm is zero-centered (
x/rms * (1 + w)) andthe engine applied plain
w, so every norm output in a d3 pack was scaled by the weight's own magnitude. With thatfixed (#902,
closes #901) the causal arm's worst |Δp| against the pack's ownd3_runtime.pyis 0.4140(choice argmax flips 2/5), and layers 0-2 (all GDN) now agree to cos ≥ 0.999968.
The remaining gap starts at the first full-attention layer (layer 3: cos 0.452) — partial rotary 64/256,
zero-centered
q_norm/k_norm,attn_output_gate. Until that is fixed, re-running this issue'smask_cmp.pyrowwould measure that defect, not the mask, so I am not quoting a new noncausal-vs-causal magnitude yet. The mask itself
is unchanged and still verified layer by layer (#861); the re-run is queued behind engine#796's layer-3 split.
bong-water-water-bong commentedon Nov 3, 2026
Re-measured on a faithful forward: the mask's effect is now reproduced — engine 0.2837 vs the reference's 0.2869
The precondition this issue was waiting for is met. With engine#901/#903 fixed (#902, #904) the causal arm's gap
against the pack's own
d3_runtime.pyis 0.0061 (was 0.6367 when this issue's earlier rows were taken), so themask row can finally be read on its own. Both arms, same weights (bf16 container), same 9 requests + the 3-question
alias, host-GEMM, one
npu-holdticket per arm,amdxdnaerrors 0.So the two-pass fill (
attention.modefrom #861) now reproduces the reference's noncausal behaviour rather thanmerely "differing from causal": the largest case is the same case on both sides (
score_three_levels), the per-casevalues track within ~0.004, and the causal path is unchanged to 0.0061.
On the acceptance's wording, one honest caveat: it asks for the engine's nc-vs-c to be at least 0.2869; it is
0.2837, i.e. 0.0032 short — smaller than the engine-vs-reference forward gate on the same runs (0.0044–0.0061),
so it reads as agreement rather than as an unmet threshold, but "at least" is not literally satisfied and I am not
going to call it satisfied. Whether that closes the issue is a triage call.
Device note unchanged: these are host-GEMM rows on the bf16 container; a device row still needs the six ELF designs.
Harness:
~/lb/d3parity/{parity_run.sh,mask_cmp2.py}+eng_causal_norm.json,eng_noncausal_norm.json.bong-water-water-bong commentedon Nov 3, 2026
Same re-run applies to the two arms quoted in my last comment: re-executed at 12:29-12:30 (after the silent-holder window closed), the causal and noncausal answer files are byte-identical to the ones behind that table, so the 0.2837-vs-0.2869 row stands. Ticket additions in the ledger: 622 s wait + 8 s hold for the dump recheck, 127 s for the parity recheck;
amdxdnaerrors 0.