diff --git a/docs/hrx.md b/docs/hrx.md index 14ed690f..7182d89b 100644 --- a/docs/hrx.md +++ b/docs/hrx.md @@ -54,6 +54,39 @@ Vulkan0: AMD Radeon 8060S Graphics (RADV STRIX_HALO) - **Validation.** CI builds that PR without HRX, so run the checks below on Strix Halo before merging. +### Known issue: `--device hrx` answers wrongly on mid-length prompts + +Measured 2026-09-24 on the pinned pair (llama.cpp `f1a0aca` + our patches), Qwen3-0.6B +Q4_K_M, greedy chat through llama-server on `HRX0`: + +| Prompt tokens | 146 | 404 to 1,733 | 2,167 and 2,844 | +|---|---|---|---| +| `--device vulkan` | right | right | right | +| `--device hrx` | right | **wrong** (garbled text) | right | + +**Cause, narrowed:** prompt processing on `HRX0` is right at every length (last-token +logits within 1% of Vulkan at 146 to 2,040 tokens). What goes wrong is **decoding one +token at a time once the KV cache holds more than 256 tokens**: fed the same tokens as +Vulkan, a 248-token prompt decodes correctly for 8 steps, a 250-token prompt goes wrong at +the step where the cache passes 256, and 256 tokens or more are wrong from the first step +(relative logit differences 0.3-1.3, the top token different on most steps). That points +at `HRX0`'s single-query FLASH_ATTN_EXT over more than one 256-row KV block; somewhere +above 2,048 tokens another path takes over, which is why those chats read correctly. + +Context size does not matter (1,536 to 8,192), the HRX diagnostic switches +(`GGML_HRX_DEBUG_SERIAL_PROGRAM_COMMANDS`, `GGML_HRX_DIAGNOSTIC_GRAPH_SYNC`, +`GGML_HRX_DIAGNOSTIC_FRESH_TRANSIENT_ARENA`, `GGML_HRX_DISABLE_QWEN_DISPATCH`) do not fix +it, and with flash attention off the graph does not run at the prompt lengths tested. +Two more HRX0 failures on the same pin: Qwen3.6-35B-A3B Q8_0 faults the GPU (`AMDGPU +memory access fault` in a graph replay), and a batch of several sequences +(`llama-perplexity` with `n_seq` > 1) stops on a FLASH_ATTN_EXT node that HRX0 claimed +but cannot run. + +**The prefill split below is not affected**: `--device vulkan --prefill-device hrx` +answered the same as Vulkan alone at 1,160, 1,733 and 2,844 tokens. HRX0 only prefills +whole ubatches there; decoding is Vulkan's. Until `--device hrx` is fixed upstream, use it +for measurements, not for serving. + ### Prefill on HRX, decode on Vulkan HRX prefills faster than Vulkan and Vulkan decodes faster, on the same GPU.