[Issue]: HRX preview (v1.0.8, Linux): flm bench gemma4-it:e2b fails at 32k with 'ert state 8'; XRT completes, serve works #785

Activity

1. jtuyls commented on Oct 21, 2026

   

   jtuyls

   on Oct 21, 2026

   Collaborator

   More actions

   @mariolpantunes Thanks for reporting!

   I reproduced the reported failure and traced it to the interaction between valid long-running 32k commands and amdxdna’s default two-second TDR.

   Reproduction results

   Test environment:

   • Ubuntu 26.04 container
   • Linux 7.2.10
   • amdxdna 0.10
   • NPU firmware 1.1.2.64
   • gemma4-it:e2b
   • FastFlowLM HRX v1.0.8 and v1.0.9

   With the default tdr_timeout_ms=2000:

   • HRX v1.0.8 reproduces the reported ERT state-8 errors at 32k.
   • Repeated context resets eventually cause the later hrx_buffer_allocate failed.
   • HRX v1.0.9 still receives seven state-8 errors per 32k iteration.
   • v1.0.9 reports those errors and keeps the runtime usable, but FastFlowLM continues the benchmark and exits 0 with invalid 32k metrics.
   • XRT also receives state 8 for the same seven commands. It only appears to pass because it does not validate the final ERT packet state.

   With tdr_timeout_ms=10000, the unchanged benchmark completes on both HRX v1.0.8 and v1.0.9 without state errors or allocation failures.

   Issues identified

   1. The two-second amdxdna TDR rejects valid 32k work

   At 32k, seven NPU commands run longer than the current watchdog permits.

   Observed timings:

   • Default TDR: commands are reset after approximately 2.02–2.07 seconds.
   • HRX with a ten-second TDR: the same commands complete in approximately 3.297 seconds.
   • XRT waits approximately 2.01–2.07 seconds, but the resulting ERT packets still contain state 8 with the default TDR.

   This is therefore not exclusively an HRX failure. XRT encounters the same command aborts but does not report them.

   Selecting a new global timeout or redesigning progress detection requires coordination with the amdxdna maintainers.

   2. HRX omits the instruction BO dependency

   HRX Linux submissions omit the instruction/control BO from the EXEC_CMD dependency list.

   Registering that dependency reduces each affected HRX command from approximately 3.297 seconds to 2.02–2.06 seconds, matching XRT much more closely.

   This is a separate HRX performance and residency defect. It should be fixed independently, but it does not fully resolve the issue because the commands still narrowly exceed the current two-second watchdog. I will be working on that fix in any case.

   3. Failed commands are reported as successful benchmark results

   HRX already detects state 8 and returns an error to its caller. Keeping the runtime usable after a command-scoped failure is appropriate, but the failed operation itself must not be treated as successful.

   FastFlowLM currently logs the HRX error, continues the benchmark, publishes invalid metrics, and exits 0.

   The expected behavior is:

   • A benchmark should stop the failed iteration/run and exit nonzero.
   • A server should fail the affected request while keeping the server available.
   • XRT should validate the final ERT packet state and propagate state 8 in the same way.

   This error-handling correction is separate from making the 32k operation succeed. The amdxdna TDR and HRX BO-dependency fixes address execution; honoring the returned error prevents false-success results.

   4. Why serve succeeds

   serve configures a 4096-token maximum prefill length and chunks long prompts.

   bench exercises a monolithic 32k prefill, which reaches the long-running commands that trigger the watchdog.

   Exact reproduction

   First confirm the environment:

   uname -r
   flm validate
   sudo cat /sys/module/amdxdna/parameters/tdr_timeout_ms

   The failing environment should report:

   • Linux 7.2.10
   • amdxdna 0.10
   • firmware 1.1.2.64
   • unlimited memlock
   • tdr_timeout_ms=2000

   Reset the driver to the default two-second TDR:

   sudo modprobe -r amdxdna
   sudo modprobe amdxdna tdr_timeout_ms=2000

   Ensure no other NPU workloads are active before unloading the module.

   Set the extracted FastFlowLM package and model-cache locations:

   FLM_PACKAGE=/path/to/extracted/fastflowlm
   MODEL_CACHE=/path/to/model-cache
   OUTPUT_DIR="$PWD/flm-785-output"
   
   mkdir -p "$OUTPUT_DIR"

   Run the benchmark inside Ubuntu 26.04:

   docker run --rm \
     --device /dev/accel/accel0 \
     --ulimit memlock=-1:-1 \
     --ipc host \
     -v "$FLM_PACKAGE:/opt/flm:ro" \
     -v "$MODEL_CACHE:/app/models:ro" \
     -v "$OUTPUT_DIR:/out" \
     -w /out \
     -e FLM_MODEL_PATH=/app/models \
     -e FLM_DISABLE_UPDATE_CHECK=1 \
     ubuntu:26.04 \
     /opt/flm/flm bench gemma4-it:e2b \
       --pmode performance \
       --bench-iterations 2

   To verify the TDR diagnosis, reload amdxdna with a longer timeout:

   sudo modprobe -r amdxdna
   sudo modprobe amdxdna tdr_timeout_ms=10000
   sudo cat /sys/module/amdxdna/parameters/tdr_timeout_ms

   Re-run the identical Docker command. The benchmark should then complete without ERT state errors or allocation failures.