Skip to content

hrx: the parity row compares logits, not text — text equality reports cross-backend coin flips as placement bugs (engine#674) #906

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
bong-water-water-bong wants to merge 1 commit into from
Closed

hrx: the parity row compares logits, not text — text equality reports cross-backend coin flips as placement bugs (engine#674) #906

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 7 additions & 3 deletions docs/hrx.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -974,9 +974,13 @@ same build now passes `tests/serve_e2e.sh` on both devices through `1bit serve`.
`tests/hrx_cpu_weight_parity.sh` (engine #674) is the row that compares placements instead of just checking that each one
answers: the same greedy prompt through the CPU (`-dev none`), through the device with **every layer assigned to the CPU**
(`-dev HRX0 -ngl 0`, the partial-offload path where the device still claims nodes and materializes their weights) and fully
offloaded (`-ngl 99`) must come back byte-identical. It skips without a gfx1151, and it fails rather than skips when the
machine has a GPU but the HRX runtime cannot open it — the `IREE_HAL_AMDGPU_LIBHSA_PATH` case `scripts/hrx-env.sh` now
resolves for hand-run tools.
offloaded (`-ngl 99`). It compares the **distributions**, not the text: the CPU path and the device paths are different
arithmetic (f32 CPU kernels vs HRX's kernels over an f16 KV cache), their logits differ by ~0.1 nats, and at temperature 0 a
coin flip between two candidates flips one token — measured on `ca1d42f`, where the *shipped* `-ngl 99` placement flips its
own coin 24 tokens in. So each device arm must keep the CPU's top-5 distribution (overlap >= 0.30, median |dlogprob| <=
0.50 nats, calibrated: 0.63-0.78 overlap and 0.038-0.094 nats on a correct stack, 0.00 overlap when placement actually
corrupts the answer). It skips without a gfx1151, and it fails rather than skips when the machine has a GPU but the HRX
runtime cannot open it — the `IREE_HAL_AMDGPU_LIBHSA_PATH` case `scripts/hrx-env.sh` now resolves for hand-run tools.

llama-bench, Qwen3-0.6B, pp512 / tg128 tok/s, against the previous build (April
llama.cpp with ggml-hrx2 on `HRX20`):
Expand Down
84 changes: 74 additions & 10 deletions tests/hrx_cpu_weight_parity.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,17 +18,41 @@
#
# #674 was opened as "any graph whose weight ops all run on the CPU computes garbage". The garbage itself was only ever
# produced by a do-not-merge placement patch, but the *class* is real and nothing in this repo tests for it: today
# `serve_e2e.sh` asserts each device answers Paris, and no row compares one placement's text with another's. This row
# `serve_e2e.sh` asserts each device answers Paris, and no row compares one placement's answer with another's. This row
# does, on the same binary and the same model, three ways:
#
# cpu -dev none every node on the CPU, no device at all
# weightscpu -dev HRX0 -ngl 0 the device present, every layer assigned to the CPU: the device still claims the nodes it
# supports and materializes the weights they read, i.e. the partial-offload path #674 names
# offload -dev HRX0 -ngl 99 the shipped serve placement
#
# Greedy decoding at temperature 0 must be byte-identical across all three, because placement changes where a number is
# computed, never what it is. A regression here is exactly the failure #674 reported: plausible-looking text that is not
# the CPU's answer.
# WHAT IS COMPARED, and why it is not text (2026-11-03). The first version of this row required the three greedy texts to
# be byte-identical. That is too strict to be true: the CPU path and the device paths are different arithmetic (f32 CPU
# kernels vs HRX's kernels over an f16 KV cache), so their logits differ by ~0.1 nats, and at temperature 0 a *coin-flip*
# between two candidates then flips one token. Measured on the c21 binary (`ca1d42f`), 24 greedy tokens, `n_probs 5`:
#
# pairs top-5 overlap median |dlogprob|
# cpu vs weightscpu (-ngl 0) 0.63 0.094 nats
# cpu vs offload (-ngl 99, SHIPPED) 0.78 0.084 nats <- the shipped arm flips a token too
# weightscpu vs offload 0.63 0.038 nats
#
# i.e. the text mismatch c21 hit ("also a city called" vs "also a capital city") is a near-tie at step 14 where the CPU's
# own top-1 margin is 0.17 nats, and the *shipped* placement flips its own coin three tokens later. Byte-identical text
# is therefore not achievable across backends, and a row that demands it reports coin flips as placement bugs.
#
# So the row compares the DISTRIBUTION: each device arm against the CPU arm, and the two device arms against each other,
# must still overlap and still agree. Calibration from the same two stacks (see the table above and below):
#
# correct stack overlap 0.63-0.78, median |dlogprob| 0.038-0.094 nats
# engine #674's corruption (garbage output) overlap 0.00, median undefined (nothing shared)
#
# Thresholds: overlap >= 0.30 and median |dlogprob| <= 0.50 nats. Those sit an order of magnitude clear of a correct
# stack in both metrics and cannot survive the corruption, which shares no top-5 candidate at all. The texts are still
# printed, with the step they fork at, because a human reading a failure wants them — they just do not decide the verdict.
#
# KNOWN LIMIT, stated rather than implied: this row detects placement changing the *distribution* (the #674 class: wrong
# numbers, garbage, a collapsed or shifted top-5). It does not certify bit-identical arithmetic across backends, and it
# would not catch a bug that perturbs the answer by less than the cross-backend spread measured above.
#
# The device arm is also the only row that notices the silent no-device trap: on this box the #715 trio is necessary but
# not sufficient (`IREE_HAL_AMDGPU_LIBHSA_PATH` decides whether the HAL finds a runtime that can open the device at all),
Expand Down Expand Up @@ -68,7 +92,10 @@ fi

free_port() { python3 -c 'import socket; s=socket.socket(); s.bind(("127.0.0.1", 0)); print(s.getsockname()[1])'; }

# One greedy completion from a fresh server. Prints the reply's content on stdout; logs go to stderr.
work=$(mktemp -d)
trap 'rm -rf "$work"' EXIT

# One greedy completion from a fresh server, JSON saved to $work/$tag.json; logs go to stderr.
arm() { # $1 tag, rest = llama-server args
local tag=$1; shift
local port log pid body
Expand All @@ -88,10 +115,11 @@ arm() { # $1 tag, rest = llama-server args
kill "$pid" 2>/dev/null; wait "$pid" 2>/dev/null; rm -f "$log"
return 4
fi
body='{"prompt": "What is the capital of France?", "n_predict": 16, "temperature": 0, "cache_prompt": false}'
curl -s "http://127.0.0.1:$port/completion" -H 'Content-Type: application/json' -d "$body" \
| python3 -c 'import json,sys; print(json.load(sys.stdin).get("content", ""))' 2>/dev/null
# n_probs 5: the comparison needs each step's top-5 logprobs, not only the sampled token
body='{"prompt": "What is the capital of France?", "n_predict": 24, "temperature": 0, "cache_prompt": false, "n_probs": 5}'
curl -s "http://127.0.0.1:$port/completion" -H 'Content-Type: application/json' -d "$body" >"$work/$tag.json"
kill "$pid" 2>/dev/null; wait "$pid" 2>/dev/null; rm -f "$log"
python3 -c 'import json,sys; print(json.load(open(sys.argv[1])).get("content",""))' "$work/$tag.json" 2>/dev/null
}

cpu=$(arm cpu -dev none -ngl 0 --jinja) || { echo "FAIL cpu arm did not run"; exit 1; }
Expand All @@ -102,8 +130,44 @@ echo " cpu said: $cpu"
echo " weightscpu said: $weightscpu"
echo " offload said: $offload"
check "the CPU answers at all" '[ -n "$cpu" ]'
check "device with every layer on the CPU answers the CPU's text" '[ "$weightscpu" = "$cpu" ]'
check "device with every layer offloaded answers the CPU's text" '[ "$offload" = "$cpu" ]'

# Distribution agreement, per arm pair. Prints the numbers it judged on, so a failure is readable without rerunning.
compare() { # $1 left tag, $2 right tag; prints "<overlap> <median|d|>"
python3 - "$work/$1.json" "$work/$2.json" <<'PY'
import json, sys
def load(p):
cp = json.load(open(p)).get("completion_probabilities", [])
return [[(t["token"], t["logprob"]) for t in s.get("top_logprobs", [])] for s in cp]
a, b = load(sys.argv[1]), load(sys.argv[2])
ds, shared, possible = [], 0, 0
for i in range(min(len(a), len(b))):
right = dict(b[i])
for tok, lp in a[i]:
Comment on lines +143 to +145
possible += 1
if tok in right:
shared += 1
ds.append(abs(lp - right[tok]))
Comment on lines +145 to +149
ds.sort()
overlap = shared / possible if possible else 0.0
median = ds[len(ds) // 2] if ds else float("nan")
Comment on lines +151 to +152
print(f"{overlap:.2f} {median:.4f}")
PY
}
agree() { # $1 description, $2 left, $3 right
local overlap median
read -r overlap median <<<"$(compare "$2" "$3")"
echo " $2 vs $3: top-5 overlap $overlap, median |dlogprob| $median nats"
if python3 -c "import sys; sys.exit(0 if $overlap >= 0.30 and $median <= 0.50 else 1)"; then
echo "ok $1"
else
echo "FAIL $1"
fail=1
fi
}

agree "placement onto the CPU keeps the CPU's distribution" cpu weightscpu
agree "full offload keeps the CPU's distribution" cpu offload
agree "the two device placements agree with each other" weightscpu offload

[ $fail = 0 ] && echo "PASS placement does not change the answer" || echo "FAIL placement changed the answer"
exit $fail
Loading