npu: the ~512-position wedging is boot-dependent — the appliance wedges where the fleet boot answers 661 positions clean with the same binary, kernels, module and firmware

What was measured

The appliance wedges the NPU when a reply crosses ~512 positions (c22 board, 2026-11-04: max_tokens 400/480 OK,
600 → npu runtime: invalid call (code 614): state 8, run 0 of 28 + one TDR, fresh backend each time). That does not
reproduce on this box's fleet boot — with the appliance's own binary, kernels, model, module and firmware.

Same ladder, run here under npu-hold (~/lb/c22ladder-20261104/), binary = the stick's
/opt/1bit-package/1bit/1bit, kernels = its own npu/kernels/1024x3072x16x8, model ~/models/npu/Qwen3-0.6B,
1bit unified -m <model> -p 18193, temperature 0, /no_think, 45-token prompt:

row	total positions	result
64 tok (new-ELF gate)	109	200, 0 TDR
400 tok	445	200, completion 400, 0 TDR
480 tok	525 — past 512	200, completion 480, 0 TDR
600 tok	645 — past 512	200, completion 600, 0 TDR
600 tok, streamed (`stream: true`, how Lemonade forwards it)	645	200, 598 SSE lines, 596 content deltas, `[DONE]`, 0 TDR
16 tok on the same process after	661	200
fresh process, 16 tok	61	200
dmesg for the ticket	—	`amdxdna.*ERROR` 0, `Array stopped dispatching` 0

And the board is this machine

From the c22 boot's dmesg (captured on ryzen, ~/tmp-ort/puppy-c22-hw/dmesg.txt) versus this host:

fact	c22 board boot	this host
DMI	`BOSGAME LIMITED BeyondMax Series/AXB35-02, BIOS 1.09 05/08/2026`	identical
CPU	`AMD RYZEN AI MAX+ 395 w/ Radeon 8060S`	identical
NPU PCI	`amdxdna 0000:c6:00.1`	identical (`1022:17f0`)
firmware	`Load firmware amdnpu/17f0_11/npu_7.sbin`	identical file
driver	`Initialized amdxdna_accel_driver 0.18.0`, srcversion `56AB9475…`	identical (module installed 10:23)
memory knobs	`ttm.pages_limit=31457280 ttm.page_pool_size=31457280 amdgpu.cwsr_enable=0`	identical
LAN address the owner browsed from	`http://192.168.50.110:13305`	this host's `eno1`
other fleet box (ryzen)	—	has no `Signal processing controller` in `lspci`

So "the dx design breaks above 512 positions" is not supported: the only variable left between wedge-at-629 and clean-at-661
is the boot — Puppy on usbhd (pfix=copy,nox, OS in RAM, no swap-backed pages, zram) versus this Ubuntu 7.4 fleet boot.
Same kernel release, same module, same firmware, same binary, same kernels, same weights, same silicon.

Why it matters

The appliance is the release surface, and its failures cannot be reproduced (or a fix verified) in the fleet environment.
Every "measure it on strixhalo" instruction for this defect silently measures a different system.
It re-points the search: not kernel/design, not the 0007 driver branch (which does run here), not the firmware file, but
something the appliance boot changes on the host side.

Candidate discriminators (not measured, listed to be checked in that order)

swap/zram: appliance has no swap-backed pages; ttm.pages_limit is the same but the available page pool differs.
OS-in-RAM (pfix=copy): page cache and free-memory shape at the moment a 600-token KV/BO set is live.
CPU idle/governor state during the generation (the NPU firmware's completion path is IRQ-driven; a stalled host thread
would surface as DPU PC 0xffffffff, which is what the capture shows).
Transparent hugepages / compact_unevictable_allowed / vm.min_free_kbytes — anything that changes how the runtime's
host-pinned BO ranges are backed.
console=ttyS0,115200 plus the appliance's userspace load (lemond, JWM desktop) versus this boot's.

How to close it

The appliance's own row is the gate: chat-npu-stream at ctx 4096 with a streamed reply that crosses 512 positions, plus
the chat-npu-after-tdr row for the client half (#1013, PR carrying the exit-on-614 fix). The deterministic provocation
pi-hrx-fabric built for #1005 (modprobe amdxdna tdr_timeout_ms=1 + one long dispatch) is the only way to get a 614 on the
fleet boot on demand, and it is an owner decision (a module reload).

Refs: #1005 (driver recovery), #1013 (engine client keeps serving a dead context), #1010 (evict→load race).

os/v3/puppy: ship the NPU's TDR timeout, and a release row that cross…

os/v3/puppy: ship the NPU TDR timeout and a release row that crosses …

A second, heavier fleet-boot data point — 600 positions on the i8 full-ELF device path, not the dx route

Lane pi-onebp (ZR1-1.5B), 2026-11-04 17:09–17:20 AST, this host, npu-hold -t 30, bare serve (no thermal-run). Different
model, different route and a much higher submit rate than the ladder above, so it does not duplicate it:

container ~/models/npu/ZR1-1.5B-npu-q4km-q8/model.q4nx, plan.json with decode: M=16, kernels =
~/models/npu/zr1-elfs/ (4 prefill + 4 _m16 full ELFs for H=1536 NH=12 NKV=2 HD=128 IM=8960).
1bit forward-serve, greedy, one request of 600 tokens, prompt The capital of France is.
driver srcversion 56AB94758C18FEEC838D18C (the 0007 module), Tctl 72 C at start, 78 C max.

row	result
600 tokens, one request	200, coherent through the last token (it drifts into `The capital of8 is Moscow` and a wrong-answer monologue — quality, not runtime)
device submits in that request	600 × 112 launches/token = 67,200 submits
`amdxdna` TDR / `list_add` / `list_debug` / `aie2_cmd_submit` failures, whole boot	0 (via `journalctl -k`; `dmesg` is restricted on this boot, so a `dmesg
serve state after	clean, `SIGTERM` + `wait`, no lingering process

Two caveats so this is not over-read: the wall clock (670 s) was under nice -n 19 — not a rate claim — and this is
the i8/ELF GEMM path (npu_gemm → int8 packed operands), not the dx design that wedges on the appliance. What it does add
is a third axis the ladder above does not cover: the launch rate. 67k submits at 112 per token is the heaviest submit
pattern I know of on this box, and the fleet boot took it at 78 C without a TDR.

So both fleet-boot routes now have a past-512 clean row: dx (this issue's ladder, 661 positions) and i8-ELF (this row, 600).
That keeps the discriminator list pointed at the boot, as the issue body says — my datum cannot distinguish those candidates,
only widen the "same binary+module+firmware, different boot" side of the table.

Answered by the c23 board boot (2026-11-04 16:46): the discriminating variable is `tdr_timeout_ms`, not the boot

The c23 image ships options amdxdna tdr_timeout_ms=15000 (#1020) and the board's own check now says, on the same
hardware, the same payload binary, the same kernels and the same model:

PASS  u5-tdr-timeout : the loaded module runs tdr_timeout_ms=15000 and /etc/modprobe.d/amdxdna.conf carries it
PASS  npu-length-ladder : Qwen3-0.6B-NPU-1bit on a freshly loaded backend generated 400/400 600/600 1000/1000
                          tokens with TDRs 0 -> 0

(ryzen:~/tmp-ort/puppy-c23-hw/puppy-check.log.) c22 wedged at 600 with the driver's default 2000 ms armed;
c23 answers 1000 with 15000 ms armed, same Puppy boot, same OS-in-RAM shape, same everything else. So the variable
this issue listed as "the boot" is the TDR interval the boot happened to arm — which the appendix's "how to close it"
could not see, because a longer timeout was not in the image yet.

That also supersedes the candidate-discriminator list here: swap/zram, OS-in-RAM, THP and the governor do not need to be
checked to explain the wedge. Nothing in the fleet boot's 645-clean-position ladder required a different explanation.
What remains true from this issue: the appliance's failures cannot be reproduced in the fleet environment unless the
knob matches, and the pre-15000 default is a real trap for any other image.

One caveat on the evidence, so the row is not read as stronger than it is: the same log line prints
(tdr_timeout_ms=unreadable) and /data/puppy-check.sh: line 532: TDROP: unbound variable — the ladder row's own
attribution string is broken (the u5 row read 15000 correctly, so the ladder itself ran under 15000). That is #1020's
file, reported to pi-release; it does not change the 400/600/1000 result.

Closing this as answered. If a wedge ever shows up again at a position count with 15000 armed, it is a new issue and the
capture has to come with SRC/modprobe truth for the running module, not the file on disk.

Navigation Menu

Repository navigation

npu: the ~512-position wedging is boot-dependent — the appliance wedges where the fleet boot answers 661 positions clean with the same binary, kernels, module and firmware #1017

What was measured

And the board is this machine

Why it matters

Candidate discriminators (not measured, listed to be checked in that order)

How to close it

A second, heavier fleet-boot data point — 600 positions on the i8 full-ELF device path, not the dx route

Answered by the c23 board boot (2026-11-04 16:46): the discriminating variable is `tdr_timeout_ms`, not the boot

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

npu: the ~512-position wedging is boot-dependent — the appliance wedges where the fleet boot answers 661 positions clean with the same binary, kernels, module and firmware #1017

Description

What was measured

And the board is this machine

Why it matters

Candidate discriminators (not measured, listed to be checked in that order)

How to close it

Activity

bong-water-water-bong commented on Nov 4, 2026

A second, heavier fleet-boot data point — 600 positions on the i8 full-ELF device path, not the dx route

bong-water-water-bong commented on Nov 4, 2026

Answered by the c23 board boot (2026-11-04 16:46): the discriminating variable is tdr_timeout_ms, not the boot

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Answered by the c23 board boot (2026-11-04 16:46): the discriminating variable is `tdr_timeout_ms`, not the boot