[bong-water-water-bong]

What this changes

One built-in recipe, hrx-graph-program-cache-128: on the hrx device with the qwen4exp architecture
(Qwen3.8-Flash-Next), 1bit serve sets GGML_HRX_GRAPH_PROGRAM_CACHE=128 instead of leaving it at the default 64.
Plus the docs table row and a tests/recipes_route.sh case that it fires on qwen4exp and on nothing else.

Root cause, in two measured parts

1. This model needs more programs than the cache holds. Flash-Next's own server log at -c 19456, 18.8k-token row,
   40-minute run: 933 trim_graph_programs: HRX graph program cache trimmed (1 of 66 programs, ... 64 max) lines
   (and 779 in the pre-rebuild run of the same row). Each trim evicts exactly one program and frees ~6 MiB, so the
   8192 MiB byte budget is nowhere near binding - the arenas sit at ~2.4 GiB - and the count limit is the whole story.

2. A program that is not resident costs ~41 ms. Measured on this box on 2026-11-04 by timing the first
   graph_compute of a fresh graph against the immediately following one (the second is served from the program cache),
   same graph, same leaves:

   graph	nodes	first	second	price of a re-JIT	steady p50
   1 GDN layer	44	41.9 ms	0.94 ms	40.9 ms	0.65 ms
   12 layers	528	52.0	6.04	46.0 ms	5.39 ms
   36 layers	1584	58.4	15.68	42.7 ms	15.29 ms
   36 layers, cache=1 x2	1584	56.8 / 59.6	15.8 / 18.5	41.0 / 41.1 ms	15.5 / 15.3 ms

   The price is flat in graph size - it is compile+load overhead, not per-command emission - and it is 3-4x the
   whole steady decode of a 36-layer GDN graph. (In a cold process the same first compute costs 110-136 ms; 41 ms is the
   warm figure, verified by the steady p50 in those same runs moving with the clocks, not with the JIT.)

What is NOT measured, and why this is a recipe and not a default change

The tok/s row at 18.8k. It needs the 76.33 GiB file with the box to itself: the load alone was measured at >= 109
GiB of GTT (engine #680's reset regime starts at 106.5 GiB, the GTT ceiling here is 120 GiB), and this box has been
held by other tenants' serves (32-40 GiB) for five days, so the row has been blocked since 2026-10-31. What the trims
cost per step therefore has a ceiling, not a value: at the observed rate of ~1 trim per 10 decode steps it is ~4
ms/step; if trims were 1 per step it would be the whole ~41 ms. The recipe claims the mechanism and the price of the
thing it removes, not a token rate - and it is scoped to this architecture rather than made the default, because the
opposite-direction evidence (a 66-program-class model at cache 128 logging 0 trims, i.e. the knob is a no-op below the
limit) is exactly why a global default is not justified yet.

Raising the count does not preallocate: the byte budget still caps the cache at 8192 MiB, and 128 slots of ~6 MiB are
~768 MiB against a 120 GiB GTT budget.

Testing

• tests/recipes_route.sh (ctest recipes_route, no GPU), 19/19 checks pass: the built-in set sets
  GGML_HRX_GRAPH_PROGRAM_CACHE=128 on a qwen4exp file served with --device hrx and names the recipe, is silent on
  cpu (it is an HRX-only variable), sets nothing for the other architectures on either device, and leaves another
  hrx architecture alone. The positive case is skipped (not failed) when the HRX runtime library is absent, since
  serve has to resolve the device; the model itself never loads - the backend is the suite's stub. The suite's run
  helper also unsets GGML_HRX_GRAPH_PROGRAM_CACHE, so an ambient value cannot mask the recipe.
• ctest -R recipes_route in the build dir: Passed (2.02 s) - this is the test CI runs (ci.yml runs the whole ctest suite, and recipes_route is registered at CMakeLists.txt:147 against the onebit target).
• tools/copyright.py --check clean.
• docs/recipes.md: the built-in table gains the row and the sentence that said "three, all qwen35" now says four,
  naming this one as the only environment setting.

Refs engine #679 (the long-context decode issue). Does not close it: the 18.8k token-rate row is still the missing
number, and the recipe says so in its own measured field.

[bong-water-water-bong]

#1006 ready for review at 6921e64. tests/recipes_route.sh 19/19 on this head, tools/copyright.py --check clean. Scope is hrx+qwen4exp and it is a recipe, not a default change: the 41 ms re-JIT price is measured, the 18.8k tok/s effect is not (that row needs the 76.33 GiB file with the box to itself).

[github-actions]

Preparing review...

[bong-water-water-bong]

Rebased onto current main f17cc61f (main gained #997, which does not touch recipes): head is now e431f2d5, and ctest -R recipes_route was re-run on that head — Passed (2.59 s), tools/copyright.py --check clean.

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit 1c95546)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 680 - Partially compliant Compliant requirements: The PR addresses the issue by increasing the graph program cache size to 128, which prevents the eviction of programs that would otherwise cause re-JIT and subsequent page faults The PR includes measurements showing that the default cache size of 64 causes 933 trim events during a 40-minute run, which is directly related to the page fault issue The PR adds a recipe that specifically targets the qwen4exp architecture on hrx devices, which is the affected configuration The PR includes documentation updates to reflect the new recipe and its purpose Non-compliant requirements: The PR does not directly address the underlying memory management issue of GTT exhaustion, but rather mitigates the symptom by increasing cache size The PR does not include a mechanism to monitor or prevent GTT from reaching the ceiling Requires further human verification: Verification that the increased cache size actually prevents the page faults in real-world usage Confirmation that the fix does not introduce performance regressions or memory issues in other scenarios

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪

🧪 PR contains tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Recipe Specificity The new recipe hrx-graph-program-cache-128 is specific to qwen4exp architecture and hrx device. While this is appropriate for the identified issue, it's important to ensure that this specificity is well-documented and that there are no other models with similar characteristics that might benefit from the same fix. The recipe should be reviewed to confirm it's not overly restrictive. { "id": "hrx-graph-program-cache-128", "match": { "device": [ "hrx" ], "architecture": [ "qwen4exp" ] }, "env": { "GGML_HRX_GRAPH_PROGRAM_CACHE": "128" }, "why": "Qwen3.8-Flash-Next's decode graph needs 66 distinct programs and the default count limit is 64, so at least one program is evicted and re-JITed for the whole run; one re-JIT costs about as much as four whole 36-layer GDN decode steps. The byte budget (8192 MiB) is not the binding constraint - the arenas sit at ~2.4 GiB - so raising only the count is the cheap half of this knob.", "measured": "Flash-Next UD-IQ3_XXS at -c 19456, 40-minute server run: 933 'trim_graph_programs: HRX graph program cache trimmed (1 of 66 programs, ... 64 max)' lines, and 779 in the pre-rebuild run of the same row, each trim evicting exactly one program and freeing ~6 MiB. On the same box (2026-11-04) a program that is not resident costs, measured as the first graph_compute of a fresh graph against the immediately following one: 40.9 / 46.0 / 42.7 ms for a 44 / 528 / 1584-node program (warm, flat in graph size), 110-136 ms first thing in a cold process, against steady p50 of 0.65 / 5.39 / 15.29 ms for those same graphs. On a 66-program-class model at cache 128 the same build logged 0 trims. NOT measured, and the reason this is a recipe rather than a default change: the tok/s effect at 18.8k, which needs the 76.33 GiB file with the box to itself (GTT >= 109 GiB measured at load). At the observed trim rate of ~1 per 10 decode steps the ceiling is ~4 ms/step; at 1 per step it would be ~41 ms/step.", "source": "engine #679, TATTN-STATE 2026-11-04" } Test Coverage The test suite includes checks for the new recipe, but it only runs on systems with HSA runtime. While this is reasonable, it would be beneficial to have a more comprehensive test that covers scenarios without HSA runtime to ensure the recipe doesn't break in those cases. hsalib=${IREE_HAL_AMDGPU_LIBHSA_PATH:-/opt/rocm-therock/lib/python3.14/site-packages/_rocm_sdk_core/lib/libhsa-runtime64.so.1} if [ -e "$hsalib" ]; then SERVE_DEVICE=hrx run "$scratch/q4exp.gguf" "$scratch/q4exp-hrx.json" check " on hrx it sets GGML_HRX_GRAPH_PROGRAM_CACHE=128 and names the recipe" \ '[ "$(field "$scratch/q4exp-hrx.json" "r[\"env\"].get(\"GGML_HRX_GRAPH_PROGRAM_CACHE\")")" = 128 ] && grep -q "recipe hrx-graph-program-cache-128" "$scratch/q4exp-hrx.json.log"' SERVE_DEVICE=hrx run "$scratch/plainq35.gguf" "$scratch/q35-hrx.json" check " and leaves another hrx architecture alone" \ '[ "$(field "$scratch/q35-hrx.json" "r[\"env\"].get(\"GGML_HRX_GRAPH_PROGRAM_CACHE\")")" = None ]' else echo "skip qwen4exp recipe on hrx (no HSA runtime at $hsalib)" fi

[bong-water-water-bong]

VERDICT: READY

Built this head (cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER=gcc-16 -DCMAKE_CXX_COMPILER=g++-16, target onebit, 32/32, 1bit links) and ran the PR's own test on it:
tests/recipes_route.sh build/1bit -> 19 ok / 0 FAIL, PASS, rc 0 — including both new hrx arms.

The two claims re-derived from primary sources, not from the body:

• 933 trims, 66 programs ... 64 max: the pre-rebuild log is on this box, ~/lb/aval/tattn-18k-topk/server.log
  -> grep -c trim_graph_programs = 779, first line verbatim:
  trim_graph_programs: HRX graph program cache trimmed (1 of 66 programs, 4212 -> 4206 MiB of arenas, budget 8192 MiB, 64 max): kept the current one, trim 1.
  So it is one program per trim, ~6 MiB freed (4212 -> 4206), and the arena figure sits far under the 8192 MiB byte
  budget — the count limit is the binding one, exactly as the why string says. The 933 figure is the cell's
  post-rebuild run (TATTN-STATE 2026-11-04, ~/lb/aval/... window).
• the arithmetic in the why string: 933 trims in 40 min at ~4.1 tok/s is ~9.8k steps, i.e. one trim per ~10.5
  decode steps, and 0.105 x 41 ms = 4.3 ms/step — the "~4 ms/step at ~1 trim per 10 steps, ~41 ms if it were 1
  per step" pair is self-consistent.

Config and docs: config/recipes.json has 4 entries; the new one is the only one with env and no args, scoped
to {"device":["hrx"],"architecture":["qwen4exp"]}, so it cannot leak to qwen35 or to cpu — and the test pins
both directions (nothing on cpu, nothing for plainq35 on hrx). docs/recipes.md now says "Four" and lists four
rows. Not measured, and the recipe string says so plainly: the tok/s effect (the 76.33 GiB file needs the box).

One nit, and it is about the test rather than the recipe:

The positive arms are gated on the HSA library existing ([ -e "$hsalib" ]), but they do not need it: the backend
is a stub and the recipe routes on the device name, so my plain non-HRX build — no HRX backend at all — ran both
hrx checks green. On a CI runner without TheRock those two checks are therefore skipped for no reason, which is the
one place this test's coverage is thinner than it looks. Dropping the guard (or replacing it with a command -v on
nothing at all) would let CI run them.

[github-actions]

Persistent review updated to latest commit 7112228

[bong-water-water-bong]

VERDICT: READY at 7112228a4 — merge-only update, own content byte-identical.

Checked with a hash of the PR's own diff at both heads rather than by eyeballing the file list: git diff <merge-base> <head> | sha256sum is the same at e431f2d5b and at 7112228a4, so nothing of this PR's moved. The branch is MERGEABLE/BEHIND; the whole delta is main advancing (#1007, #1002, and the console-TUI/prefix-cache work that landed with them).

Evidence from the review at e431f2d5b stands: the hrx-graph-program-cache-128 recipe — built and tests/recipes_route.sh 19 ok / 0 FAIL, both hrx arms included.

CI at this head: red/pending: build — pr_agent failures on engine PRs are superseded-run cancellations ("Canceling since a higher priority waiting request exists"), not findings.

[github-actions]

Persistent review updated to latest commit 614ec08

[github-actions]

Persistent review updated to latest commit 5490a2b

[github-actions]

Persistent review updated to latest commit 1c95546