From 6cedc4a7bff2c9fd0545170d756a6d51ac5af84b Mon Sep 17 00:00:00 2001 From: bong-water-water-bong <277547417+bong-water-water-bong@users.noreply.github.com> Date: Thu, 24 Sep 2026 16:18:17 -0300 Subject: [PATCH 1/2] docs/hrx.md: known issue, --device hrx answers wrongly on 404-1,733-token prompts (the prefill split is fine) Co-Authored-By: Claude Opus 5.5 --- docs/hrx.md | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/docs/hrx.md b/docs/hrx.md index 14ed690f..47704aaa 100644 --- a/docs/hrx.md +++ b/docs/hrx.md @@ -54,6 +54,30 @@ Vulkan0: AMD Radeon 8060S Graphics (RADV STRIX_HALO) - **Validation.** CI builds that PR without HRX, so run the checks below on Strix Halo before merging. +### Known issue: `--device hrx` answers wrongly on mid-length prompts + +Measured 2026-09-24 on the pinned pair (llama.cpp `f1a0aca` + our patches), Qwen3-0.6B +Q4_K_M, greedy chat through llama-server on `HRX0`: + +| Prompt tokens | 146 | 404 to 1,733 | 2,167 and 2,844 | +|---|---|---|---| +| `--device vulkan` | right | right | right | +| `--device hrx` | right | **wrong** (garbled text) | right | + +Context size does not matter (1,536 to 8,192), the HRX diagnostic switches +(`GGML_HRX_DEBUG_SERIAL_PROGRAM_COMMANDS`, `GGML_HRX_DIAGNOSTIC_GRAPH_SYNC`, +`GGML_HRX_DIAGNOSTIC_FRESH_TRANSIENT_ARENA`, `GGML_HRX_DISABLE_QWEN_DISPATCH`) do not fix +it, and with flash attention off the graph does not run at the prompt lengths tested. +Two more HRX0 failures on the same pin: Qwen3.6-35B-A3B Q8_0 faults the GPU (`AMDGPU +memory access fault` in a graph replay), and a batch of several sequences +(`llama-perplexity` with `n_seq` > 1) stops on a FLASH_ATTN_EXT node that HRX0 claimed +but cannot run. + +**The prefill split below is not affected**: `--device vulkan --prefill-device hrx` +answered the same as Vulkan alone at 1,160, 1,733 and 2,844 tokens. HRX0 only prefills +whole ubatches there; decoding is Vulkan's. Until `--device hrx` is fixed upstream, use it +for measurements, not for serving. + ### Prefill on HRX, decode on Vulkan HRX prefills faster than Vulkan and Vulkan decodes faster, on the same GPU. From a4553da10fa72854c31ded3aebac1522612813a4 Mon Sep 17 00:00:00 2001 From: bong-water-water-bong <277547417+bong-water-water-bong@users.noreply.github.com> Date: Thu, 24 Sep 2026 16:27:32 -0300 Subject: [PATCH 2/2] =?UTF-8?q?docs/hrx.md:=20narrowed=20=E2=80=94=20HRX0?= =?UTF-8?q?=20decode=20is=20wrong=20once=20the=20KV=20cache=20passes=20256?= =?UTF-8?q?=20tokens=20(prefill=20is=20right)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5.5 --- docs/hrx.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/hrx.md b/docs/hrx.md index 47704aaa..7182d89b 100644 --- a/docs/hrx.md +++ b/docs/hrx.md @@ -64,6 +64,15 @@ Q4_K_M, greedy chat through llama-server on `HRX0`: | `--device vulkan` | right | right | right | | `--device hrx` | right | **wrong** (garbled text) | right | +**Cause, narrowed:** prompt processing on `HRX0` is right at every length (last-token +logits within 1% of Vulkan at 146 to 2,040 tokens). What goes wrong is **decoding one +token at a time once the KV cache holds more than 256 tokens**: fed the same tokens as +Vulkan, a 248-token prompt decodes correctly for 8 steps, a 250-token prompt goes wrong at +the step where the cache passes 256, and 256 tokens or more are wrong from the first step +(relative logit differences 0.3-1.3, the top token different on most steps). That points +at `HRX0`'s single-query FLASH_ATTN_EXT over more than one 256-row KV block; somewhere +above 2,048 tokens another path takes over, which is why those chats read correctly. + Context size does not matter (1,536 to 8,192), the HRX diagnostic switches (`GGML_HRX_DEBUG_SERIAL_PROGRAM_COMMANDS`, `GGML_HRX_DIAGNOSTIC_GRAPH_SYNC`, `GGML_HRX_DIAGNOSTIC_FRESH_TRANSIENT_ARENA`, `GGML_HRX_DISABLE_QWEN_DISPATCH`) do not fix