[bong-water-water-bong]

c22 board, 4 Nov 2026 — deterministic, TOS3-C23-NOTES + TC22-NPU-TDR:

fresh backend, temperature 0, /no_think, long-essay prompt, via Lemonade :13305
  400 max_tokens -> 429 total positions: 200, 0 TDR
  480 max_tokens -> 509 total positions: 200, 0 TDR
  600 max_tokens -> crosses 512: `code 614: state 8, run 0 of 28` + `aie2_tdr_recover_all: ... power-cycling NPU`

What the image was missing (both invisible in the source tree)

1. No kmod config for amdxdna. The packed c22 puppy SFS has alsa-base.conf, amdxdna-less amdxdna-nothing,
   apm.conf, ath9k.conf, puppy.conf, radeon-dpm.conf, 1bit-quiet.conf — so the module loads with the driver's
   default tdr_timeout_ms and tears the device down mid-answer. (The box reads
   /sys/module/amdxdna/parameters/tdr_timeout_ms = 15000 only because strixhalo happens to have
   /etc/modprobe.d/amdxdna-tdr.conf; a stick does not.)
2. No row that crosses the boundary. 1bit-release-check's NPU rows ask one short question (chat-npu), a
   512-token turn plus a short follow-up (chat-npu-followup) and a reload+streaming turn (chat-npu-stream). None
   of them asks the ladder the board failed at, so a wedged image could carry a green release check — which is what
   the c22 board did.

The change

file	what
os/v3/puppy/embed/etc/modprobe.d/amdxdna.conf (new)	options amdxdna tdr_timeout_ms=15000, with the board numbers and the param's 7.3 rename in the header
os/v3/puppy/container-build.sh	installs it, refuses an image whose packed sfs lacks it, and refuses any other packed modprobe.d file that names amdxdna without it (kmod merges the directory, a later options line wins — a stale copy shadows the fix and nothing fails)
os/v3/puppy/vm-check.sh	U5 npu tdr: modprobe amdxdna (it loads with no NPU hardware), then assert /sys/module/amdxdna/parameters/tdr_timeout_ms = 15000 — the value the kernel took, not the file
os/mini/release-check.sh	chat-npu-ladder: fresh backend per rung (unload first), temperature 0, /no_think, long-essay prompt at 400 / 600 / 1000 max_tokens; any new aie2_legacy_tdr_detect line fails the row, and it reports completion and prompt tokens per rung

The new row is added to the hardware-only skip list, so it prints SKIP chat-npu-ladder on a machine with no NPU
(the release check goes 21 → 22 rows).

Verification done here, and what only the board can do

• bash -n / sh -n clean; tools/copyright.py --check rc 0; shellcheck -S warning on the touched files shows
  the same 8 pre-existing warnings as origin/main's copy of release-check.sh and none new (checked by diffing
  both runs' warning sets).
• The build assertion and the two rows can only execute against a real image module combination, so they are proven by
  the c23 cut: vm-check U5 npu tdr PASS on the built .img, and the image's own 1bit-release-check printing
  SKIP chat-npu-ladder (wired, 22 checks). The runtime half — tdr_timeout_ms = 15000 on the board's loaded
  module, and the ladder answering 1000 — is the board row and is listed as such in the hand-off.
• Stated as a limit, not implied: npu: the ~512-position wedging is boot-dependent — the appliance wedges where the fleet boot answers 661 positions clean with the same binary, kernels, module and firmware #1017 shows the same ladder answering 661 positions clean on this box's fleet
  boot with the same binary, kernels, module and firmware, so "512 is a design limit of the dx route" is not
  supported. This ships the mitigation the cycle ordered and the row that decides it on hardware.
• amdxdna: one TDR leaves the NPU array unrecoverable on the appliance (hwctx_restart -22, 'restore data not reclaimed'), so a single wedged request needs a reboot #1005's after-TDR client recovery is an engine change, not merged, and is not in this PR.

closes #1019

Root cause, corrected (pi-hrx-fabric, 4 Nov 14:0x, with the c22 stick's own bytes)

The appliance shipped no modprobe override, so tdr_timeout_ms was the module default 2000 ms. The detector
needs two intervals with no aie2_tdr_signal (~4 s at 2000 ms, ~30 s at 15000 ms), and the board's failing chat took
~4 s longer for the same 629 positions than the identical stack takes on the box (12.4 s vs 8.1 s). On the box at
15000 ms the same stack answers 429 / 509 / 629 / 1029 positions with 0 TDRs, and the board's capture
(DPU PC 0xffffffff, Timed out sub command ID: 0) is shape-identical to a false positive provoked on a healthy
array. So the honest framing is the one in the hand-off: NPU chat works; a host-side pause of >=4 s trips a 2000 ms
watchdog and costs the client its backend — at 15000 ms that becomes a >=30 s pause instead. This PR is parity with
the interval every NPU number we quote comes from, plus the row that says so on hardware. The file header carries the
same correction.

[bong-water-water-bong]

VERDICT: READY — but merge it after #1016: this branch contains #1016.

Merge order first, because it is the kind of thing that costs an hour later. base says main, but both #1016
commits are ancestors of this head:

origin/main..pr1020:
  9b6eeed os/v3/puppy: ship the NPU's TDR timeout, and a release row that crosses …
  9c0b734 …a self-check for the inbound rules…            <- #1016
  9f33671 …the firewall drops every mDNS query…            <- #1016
git merge-base --is-ancestor <pr1016 head> <pr1020 head>  ->  yes
git diff --stat pr1016 pr1020  -> 4 files, +117    (its own content)

So: #1016 merges first, then #1020 (or #1020 gets rebased onto it). If #1020 merges first it silently lands #1016 as
well, and #1016 then goes CONFLICTING for no reason — the same trap as engine #983/#995 and fork #184/#178.

The parameter name is real, not assumed. This box:

$ modinfo -p amdxdna | grep tdr
tdr_timeout_ms:TDR (Timeout Detection and Recovery) timeout in milliseconds (0 = disable) (uint)
$ ls /sys/module/amdxdna/parameters/
aie2_max_col  force_cmdlist  force_iova  tdr_timeout_ms

and the header's claim about where the box's value comes from checks out — /etc/modprobe.d/amdxdna.conf on this box
opens with # Linux 7.3 amdxdna renamed timeout_in_sec to tdr_timeout_ms and carries
options amdxdna tdr_timeout_ms=15000, i.e. the box reads 15000 only because that file exists, which is exactly why a
stick without one gets the 2000 ms default. Worth noting the collision: on this box amdxdma.conf is a pre-existing
2026-09-05 file with the same name the PR installs — harmless (same value), but a good reminder that the image's copy is
not the only file that can set it.

The row asserts the right thing. vm-check's new U5 npu tdr reads the running module
(cat /sys/module/amdxdna/parameters/tdr_timeout_ms -> tdr-loaded=15000) rather than trusting the file, which is the
assertion that matters; the ladder reuses the existing tdr_count() (counts aie2_legacy_tdr_detect in dmesg) and
compares it across each rung, so a TDR is a FAIL and an unreadable dmesg is reported as "unknown" instead of "0".
container-build.sh refuses to pack the image without the option, and refuses if another packed file shadows it.

bash -n clean on all three scripts, shellcheck adds nothing: release-check.sh reports the same 5 findings as on
main (all pre-existing), container-build.sh and vm-check.sh are 0.

Three notes, none a gate:

1. The shadow scan keys on the file NAME (grep -E 'modprobe\.d/[^/]*amdxdna[^/]*$'), so a packed file that is not
   named *amdxdna* but carries options amdxdna tdr_timeout_ms=2000 would slip past it. Nothing in the c22 image does
   that (there was no amdxdna config at all, which is the bug), and U5 npu tdr reading the running value is the real
   backstop — so this is a "would be nicer to scan contents", not a hole in the guarantee.
2. The unload between rungs is not verified. The comment says a fresh backend per step is what makes the ladder
   meaningful ("a warm context would let one step hide the next"), but the POST /api/v1/unload {} result is discarded.
   Same note I made on #999's evict arm. The row would still catch the wedge with a warm context (the positions
   accumulate either way), so the claim is stronger than the check — one GET /api/v1/models-is-empty assertion per rung
   would make it true.
3. The ladder stops at the first bad rung (break), so a run that fails at 600 reports nothing about 1000. That is
   deliberate (don't hammer a wedged device) and I would keep it; just noting it so the FAIL text is not read as "the
   later rungs were fine".

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit 60e3185)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 1017 - Partially compliant Compliant requirements: The appliance's own binary, kernels, model, module and firmware are used to reproduce the issue The fix ensures the NPU TDR timeout is set to 15000 ms in the packed image A new release-check row chat-npu-ladder is added to verify the ladder of token counts Non-compliant requirements: The PR does not explicitly state that the fix has been measured on the hardware Requires further human verification: Verification that the TDR timeout is correctly set in the packed image Confirmation that the new release-check row correctly identifies the issue 1005 - Partially compliant Compliant requirements: The PR ensures that the TDR timeout is correctly set to prevent the device from resetting mid-answer Non-compliant requirements: The PR does not explicitly address the recovery mechanism for the TDR path Requires further human verification: Confirmation that the recovery mechanism works as intended 1019 - Partially compliant Compliant requirements: The PR adds the amdxdna.conf file with the correct TDR timeout setting The PR adds a new release-check row to verify the ladder of token counts The PR ensures that the packed image contains the correct configuration Non-compliant requirements: The PR does not explicitly state that the fix has been measured on the hardware Requires further human verification: Verification that the TDR timeout is correctly set in the packed image Confirmation that the new release-check row correctly identifies the issue

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪

🧪 No relevant tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Missing TDR assertion in release-check The new chat-npu-ladder row in release-check.sh does not assert that the TDR timeout is correctly set in the running module. While it checks for TDRs during the ladder, it does not verify that the module is loaded with the correct tdr_timeout_ms=15000 setting. This could lead to a false positive if the module is loaded with a different timeout value. if ls /dev/accel/accel* >/dev/null 2>&1 && [ -n "$NPU" ]; then TDRL0=$(tdr_count) TDRL1=$TDRL0 LAD_OK= LAD_BAD= for n in 400 600 1000; do got --header='Content-Type: application/json' --post-data='{}' "$API/api/v1/unload" > /dev/null 2>&1 RL=$(auth_post "{\"model\":\"$NPU\",\"messages\":[{\"role\":\"user\",\"content\":\"/no_think Write a long essay about the history of cartography, and about what map-making changed in how people understood the world.\"}],\"max_tokens\":$n,\"temperature\":0}" "$API/api/v1/chat/completions" slow) AL=$(json_str "$RL" content) CL=$(json_str "$RL" completion_tokens) PL=$(json_str "$RL" prompt_tokens) TDRL1=$(tdr_count) if [ -z "$AL" ]; then LAD_BAD="$LAD_BAD ${n}->no-answer" elif [ -n "$TDRL1" ] && [ -n "$TDRL0" ] && [ "$TDRL1" -gt "$TDRL0" ]; then LAD_BAD="$LAD_BAD ${n}->TDR(${TDRL0}->${TDRL1})" else LAD_OK="$LAD_OK $n ok (${CL:-?} tokens${PL:+, ${PL} prompt})" fi [ -z "$LAD_BAD" ] || break done _tdrs_l() { if [ -n "$TDRL0" ] && [ -n "$TDRL1" ]; then printf 'TDRs %s -> %s' "$TDRL0" "$TDRL1" else printf 'dmesg not readable, so the TDR count is unknown'; fi; } if [ -n "$LAD_BAD" ]; then fail chat-npu-ladder "$NPU ladder 400/600/1000:$LAD_BAD; clean steps:$LAD_OK ($(_tdrs_l))" else pass chat-npu-ladder "$NPU answered every rung of 400/600/1000 max_tokens with a fresh backend, no TDR:$LAD_OK ($(_tdrs_l))" fi elif ! ls /dev/accel/accel* >/dev/null 2>&1; then skip chat-npu-ladder "no /dev/accel/accel*: this machine has no NPU, so no NPU entry can run" else skip chat-npu-ladder "no onebit NPU entry is listed by Lemonade, so there is nothing to ask" fi Potential shadowing of TDR configuration The build process checks for shadowing configurations but does not ensure that the correct TDR timeout value is used in all cases. If a configuration file in the image has a different TDR timeout value, it could override the intended setting, leading to the same issue that the PR aims to fix. # every other packed modprobe.d file that names amdxdna, across every SFS, must carry the same value. Collected # into a file first: a `... | while read` loop runs in a subshell, where an exit only ends the loop. : > /tmp/1bit-tdr-shadow grep -E 'modprobe\.d/[^/]*amdxdna[^/]*$' /tmp/1bit-sfs-listing | while read -r entry; do case "$entry" in */etc/modprobe.d/amdxdna.conf) continue ;; esac sfs=${entry%%/*}; rel=${entry#*/} body=$(unsquashfs -cat "sandbox3/build/$sfs" "$rel" 2> /dev/null) || continue case "$body" in *amdxdna*tdr_timeout_ms=15000*) ;; *amdxdna*) printf '%s\n' " $entry" >> /tmp/1bit-tdr-shadow ;; esac done if [ -s /tmp/1bit-tdr-shadow ]; then echo "build: REFUSED: these packed files also configure amdxdna and would shadow the 15000 ms TDR timeout:" >&2 cat /tmp/1bit-tdr-shadow >&2 exit 1 fi echo " ok: tdr_timeout_ms=15000 for amdxdna is in the packed image, with no shadowing kmod config"

[github-actions]

Persistent review updated to latest commit 883e0bb

[bong-water-water-bong]

Follow-up at 81692e31: U5 npu tdr asserts kmod's resolved view (modprobe -c → amdxdna ... tdr_timeout_ms=15000) and prints the packed file, instead of requiring the module to load.

Measured on the cycle-23 image why the load cannot happen in this harness: modprobe amdxdna → Key was rejected by service, because sb-vars.sh enrolls ONE certificate in db (the kernel signer, /var/lib/shim-signed/mok/MOK-Kernel.pem, SKI 27:68:3E:38:…) while the modules are signed by a different certificate with the same CN (modinfo -F sig_key = 35:5C:A4:8A:…). The board loads this module, so its MOK keyring trusts the second key and the VM does not. The runtime value therefore stays the board row (/sys/module/amdxdna/parameters/tdr_timeout_ms = 15000), which is what the cycle's item 1 asks for; a VM row that is red for a harness reason would just get ignored.

Also worth a line in the hand-off for whoever re-signs modules: the two certificates share CN=strixhalo Secure Boot Kernel, O=1bit-MONSTER but not their keys, so "the MOK signs the modules" is not precise enough to reproduce trust on a fresh store.

[github-actions]

Persistent review updated to latest commit 81692e3

[bong-water-water-bong]

/review

[github-actions]

Persistent review updated to latest commit 81692e3

[bong-water-water-bong]

Rebased onto current main at 8e4a069a, so the merge-order note in your verdict is moot: #1016 merged (3002ceae) while this was in review, and the rebase dropped its two commits as already-upstream. git diff --stat origin/main..HEAD is now this PR's own four files (+120/−2), no ancestors shared with another open PR.

The U5 npu tdr row also changed in that head, for a measured reason: it asserts kmod's resolved view (modprobe -c → amdxdna ... tdr_timeout_ms=15000) rather than the loaded parameter, because in the Secure Boot harness modprobe amdxdna is refused Key was rejected by service — the db carries the kernel signer (/var/lib/shim-signed/mok/MOK-Kernel.pem, SKI 27:68:3E:38:…) while the modules are signed by a different certificate with the same CN (sig_key 35:5C:A4:8A:…). The runtime reading is the board row, as the cycle asks. bash -n, shellcheck -S warning, copyright --check clean; the mDNS self-check (now on main) still passes on this tree.

[github-actions]

Persistent review updated to latest commit 8e4a069

[bong-water-water-bong]

/review

[github-actions]

Persistent review updated to latest commit 8e4a069

[bong-water-water-bong]

#1020 ready for review at 8e4a069

[bong-water-water-bong]

VERDICT: READY at 8e4a069a3 — re-verdicted after the author's push, which changes two things and both are the right call.

1. U5 npu tdr now asserts kmod's resolved view instead of the loaded value, and the reason is worth keeping. In the
VM harness modprobe amdxdna is refused Key was rejected by service (the enrolled db cert signs the kernel,
MOK-Kernel.pem, SKI 27:68:3E:38…; the modules carry a different cert with the same CN, sig_key 35:5C:A4:8A…), so the
old row's tdr-loaded=15000 could never pass there while the board trusts the module and loads it. The replacement is
tdr-config-15000 from modprobe -c, which I checked on this box:

$ modprobe -c | grep -i amdxdna
options amdxdna tdr_timeout_ms=15000                     (the row's grep counts 1 here)

That is a genuinely better assertion than "the file exists", because modprobe -c is the resolved view — a shadowing
options line elsewhere makes the count 0, which the file-based check could not see. The runtime reading is still printed
(runtime=…) and still required on the board, and the row says so rather than pretending the VM covers it. The SKI/CN
mismatch is your measurement; I could not check it here (root-only, and this box loads the module fine).

2. The new build refusals hang off #1001, which has merged — verified, so they are satisfiable today:

git show origin/main:os/v3/puppy/embed/usr/local/bin/1bit-logs | grep -c '^mounted() '   -> 1   (the refusal's pattern)
git show origin/main:os/v3/puppy/embed/etc/init.d/06onboard-logs | grep -c 'stay in RAM'  -> 0   (the refused line is gone)

So container-build.sh's two new refusals (the 06onboard-logs give-up line, and a 1bit-logs with no mount-point
test) fail exactly on the two silent-log-loss shapes and pass on the merged fix. U3 boot logs is also stronger now
(data-is-mount + logs-daemon-running + the existing marker), and U3 logs wait is a real late-mount test: daemon
started on a non-mount, tmpfs mounted over it afterwards, assert pre=0 and post>=1 without anything restarting the
daemon — which is the 4 Nov failure reproduced as a row.

The stacking constraint is gone: #1016 merged at 18:30Z, so this branch no longer carries it (its own diff is now just
its four files) and it merges standalone. MERGEABLE/BEHIND; pr_agent fail is the cancelled-duplicate pattern.

bash -n clean on all three scripts; shellcheck adds nothing over main (release-check.sh keeps its 5 pre-existing
findings, the other two are 0).

[github-actions]

Persistent review updated to latest commit 60e3185