Skip to content

docs/hrx.md: known issue — --device hrx answers wrongly on mid-length prompts #45

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bong-water-water-bong merged 2 commits into from
Sep 24, 2026
Merged

docs/hrx.md: known issue — --device hrx answers wrongly on mid-length prompts #45

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6cedc4a
docs/hrx.md: known issue, --device hrx answers wrongly on 404-1,733-t…
bong-water-water-bong Sep 24, 2026
a4553da
docs/hrx.md: narrowed — HRX0 decode is wrong once the KV cache passes…
bong-water-water-bong Sep 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions docs/hrx.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,39 @@ Vulkan0: AMD Radeon 8060S Graphics (RADV STRIX_HALO)
- **Validation.** CI builds that PR without HRX, so run the checks below on Strix
Halo before merging.

### Known issue: `--device hrx` answers wrongly on mid-length prompts

Measured 2026-09-24 on the pinned pair (llama.cpp `f1a0aca` + our patches), Qwen3-0.6B
Q4_K_M, greedy chat through llama-server on `HRX0`:

| Prompt tokens | 146 | 404 to 1,733 | 2,167 and 2,844 |
|---|---|---|---|
| `--device vulkan` | right | right | right |
| `--device hrx` | right | **wrong** (garbled text) | right |

**Cause, narrowed:** prompt processing on `HRX0` is right at every length (last-token
logits within 1% of Vulkan at 146 to 2,040 tokens). What goes wrong is **decoding one
token at a time once the KV cache holds more than 256 tokens**: fed the same tokens as
Vulkan, a 248-token prompt decodes correctly for 8 steps, a 250-token prompt goes wrong at
the step where the cache passes 256, and 256 tokens or more are wrong from the first step
(relative logit differences 0.3-1.3, the top token different on most steps). That points
at `HRX0`'s single-query FLASH_ATTN_EXT over more than one 256-row KV block; somewhere
above 2,048 tokens another path takes over, which is why those chats read correctly.

Context size does not matter (1,536 to 8,192), the HRX diagnostic switches
(`GGML_HRX_DEBUG_SERIAL_PROGRAM_COMMANDS`, `GGML_HRX_DIAGNOSTIC_GRAPH_SYNC`,
`GGML_HRX_DIAGNOSTIC_FRESH_TRANSIENT_ARENA`, `GGML_HRX_DISABLE_QWEN_DISPATCH`) do not fix
it, and with flash attention off the graph does not run at the prompt lengths tested.
Two more HRX0 failures on the same pin: Qwen3.6-35B-A3B Q8_0 faults the GPU (`AMDGPU
memory access fault` in a graph replay), and a batch of several sequences
(`llama-perplexity` with `n_seq` > 1) stops on a FLASH_ATTN_EXT node that HRX0 claimed
but cannot run.

**The prefill split below is not affected**: `--device vulkan --prefill-device hrx`
answered the same as Vulkan alone at 1,160, 1,733 and 2,844 tokens. HRX0 only prefills
whole ubatches there; decoding is Vulkan's. Until `--device hrx` is fixed upstream, use it
for measurements, not for serving.

### Prefill on HRX, decode on Vulkan

HRX prefills faster than Vulkan and Vulkan decodes faster, on the same GPU.
Expand Down
Loading