[bong-water-water-bong]

os/v3 puppy: the live USB logs it all, and the boot proves itself (1bit-proof), for c25

Owner brief TOS3-C25-VERBOSE: "make sure the live usb log verbose it all for proof that it works." Everything is kept
on /data/logs/<boot-id>/, survives a reboot, and nothing slows inference. This PR is the image-side half of that brief;
the per-request engine line (item 3) and Laya's route timing (item 5) are engine-side and are not here — see the end.

What each item became

brief item	what landed
1. boot proof, one file	new 1bit-proof boot → $LOGDIR/boot-proof.txt, written once per boot: cmdline, uname, Secure Boot two ways (mokutil --sb-state and the efivar byte), each driver's srcversion+vermagic, amdxdna's tdr_timeout_ms, every NPU firmware blob's sha256, amdgpu's firmware lines, PCI ids, /dev/accel*+/dev/kfd, MemTotal/TTM/VRAM, the boot timeline, and RELEASE-TAG + the sha256sum -c /data/SHA256SUMS result. Called from 1bit-logs' loop after it has waited for /data to mount, so the proof cannot land in RAM and vanish
2. Lemonade at its most verbose safe level	10lemonade seeds log_level: "debug" (+ log_max_file_size_mb: 8, log_max_files: 3) into lemond's config.json only when the key is absent, and never overwrites a level somebody set
3. engine per-request lines	not this PR — but at debug lemond prints Debug logging enabled - subprocess output will be visible, so the engine's own 1bit serve: lines reach lemond.log and /data/logs with only configuration
4. the ring must never wrap	mkimage.sh writes log_buf_len=32M into the shared cmdline for all four boot entries and refuses to build without it
5. Laya decisions	not this PR — the engine already logs the class/model/device on its laya: lines; the ms is an engine change
6. thermals while loaded	1bit-logs samples Tctl / fan rpm / package W every 10 s only while an engine child exists, in the same loop as the 15 s log copy (two cadences, one sleeper, so they cannot drift)
7. the proof bundle	1bit-proof (no argument) tars this boot's directory + SHA256SUMS into /data/proof-<tag>-<date>.tar.zst, prints the sha256 and the byte count, refuses an empty directory; offered from the console's Logs item rather than as a tenth key, because the legend is 1-9 and "the evidence" is what a person looks for under Logs
8. gates	four new vm-check rows: the ring size, the boot proof's labels (a file that exists but says nothing is what a stub produces), the thermal gate in both directions (engaged when serving, absent when idle), and the bundle being readable (zstd -dc | tar -tf) rather than merely present

Measurements that decided the code, not assumptions

• lemond has no verbosity flag and no env var. --help has only -v/--version; runtime_config.cpp accepts
  trace|debug|info|warning|error|fatal|none, default info. Proved live: a scratch config.json with log_level: debug
  starts with Log level: debug, Log max file size: 8 MB, Log max files: 3 and prints
  Debug logging enabled - subprocess output will be visible. trace is refused on purpose — per-token chatter would
  spend the whole 50 MB/hour evidence budget on itself.
• The brief's "Telemetry lines already carry most of it" does not hold here: this build starts with
  Telemetry: disabled, and its telemetry code is OTLP export, not local logging. Item 3's token/TTFT/tok-s line does
  not exist in lemond today — which is why it is an engine change, not a config change.
• The NPU firmware entries are directories of .zst blobs, not files: the first version of the proof called
  sha256sum on the directory and reported "unreadable". It now hashes each blob (12 hashes on this box).

Three false readings in my own first version, fixed rather than kept

0 bytes in the ring now when dmesg is restricted is a false zero; n/a not loaded for a module that was loaded
is a false absence (the parameter file is root-only); unreadable firmware was a wrong diagnosis. All three now say
what is actually true — n/a dmesg is empty or restricted to root, n/a no such parameter (amdxdna not loaded) vs
n/a unreadable as <user>, and per-blob hashes. A missing line and a zero look identical to a reviewer, so nothing is
omitted: unreadable values are written as n/a <reason>.

Two of my own harness bugs are fixed here too: pgrep -c (not present in every build → | wc -l), and
sha256sums ok=1 bad=2 for one deliberately-wrong file, because sha256sum -c prints a WARNING summary line that
grep -vc ': OK$' counted as an entry (now counts : FAILED$).

Proven, with the artifacts

check	result
1bit-proof boot as root on this box	real values: kernel.ring 160820, secureboot.efivar byte=1, module.amdxdna srcversion=56AB94758C18FEEC838D18C, amdxdna.tdr_timeout_ms 15000, 12 firmware hashes, memtotal_kb, ttm.pages_limit, gpu.vram
the seed's three states	absent → written; present without log_level → merged via python3; owner-set (trace) → untouched
thermal sampler	with a loaded model: tctl_c=61 fan_rpm=n/a pkg_w=104.2; with the real gate (nothing loaded): no line written
the bundle	proof-rc-2026-11-05-c25-<date>.tar.zst, 1,725 B, sha256 printed, and zstd -dc | tar -tf lists boot-proof.txt + SHA256SUMS
sha256sums counting	staged 1 good + 1 wrong → ok=1 bad=1 rc=1
the image	built from main + this branch: g3-probe --require-mok rc=0; grub.cfg carries log_buf_len=32M in all four entries; the shipped 1bit-proof and 1bit-logs are the fixed ones (hash-checked out of the sfs)
vm-check on that image	<fill in>

Checks: bash -n, shellcheck -S warning and tools/copyright.py --check clean; the vm-check row list is compiled (78
rows) rather than eyeballed.

Not in this PR, said plainly

• Item 3 (engine per-request line: route, device, positions, queue wait, tokens, tok/s, decoded NPU errors) and item 5
  (Laya's ms) are app/serve.cpp changes. They are deliberately left out of this PR because that is exactly where
  npu: a dx dispatch can hang once a context is long (614 state 8, DPU PC 0xffffffff), and the TDR then cannot restart a churned context #1051 (the NPU stall, still open with no PR) is being worked, and a conflict there would cost more than the line is
  worth. At debug, the engine's existing 1bit serve: output already lands in lemond.log.
• The 50 MB/hour volume arm is not measured yet by this PR. The idle half is measurable on a booted image; the
  "20 chats" half needs a model on the stick, which the demo staging owns. The hand-off records which half was measured and
  what the per-request cost is, rather than claiming the whole number.

[bong-water-water-bong]

VERDICT: READY at e50b1eff — I ran the new tools rather than reading them, and one of the four new rows cannot go red (that is the one thing to fix, and it is two lines).

Ran 1bit-proof here, using its own documented overrides (ONEBIT_LOG_DATA, ONEBIT_LOGS_BIN) against a scratch data dir and a stub 1bit-logs:

1bit-proof path            -> /tmp/proofdata/logs/boot-test-1
1bit-proof boot            -> writes boot-proof.txt (41 lines), rc 0
1bit-proof boot (again)    -> "already written this boot (--force rewrites it)", rc 0   <- written once per boot, as claimed
boot-proof.txt labels       cmdline uname secureboot.mokutil secureboot.efivar lockdown module.* amdxdna.*
                            firmware* pci.devices memtotal_kb timeline.dmesg release sha256sums lemond.log_level
secureboot two ways        "SecureBoot enabled" (mokutil) and "byte=1 (1 = enabled)"  <- they agree
module.amdxdna             loaded=yes srcversion=56AB94758C18FEEC838D18C vermagic=7.4.0-rc1-1bit
kernel.ring                n/a dmesg is empty or restricted to root (kernel.dmesg_restrict)   <- the false zero is gone

That srcversion is the same fingerprint my #1038 review used, so the proof's driver line is reading the real module and not a constant.

The SHA256SUMS fix is real. With a two-good/one-bad list: sha256sums ok=2 bad=1 rc=1. The previous "not OK" count would have said bad=2 for one bad file, exactly as the last commit claims.

The bundle works and is readable, not merely present: with ≥2 entries it prints the path, bytes 1595, sha256 dba5f47d…, and unzstd | tar -tf lists boot-test-1/boot-proof.txt and SHA256SUMS. With an empty directory it refuses, rc 1.

The other three changes check out by reading plus the run: mkimage.sh puts log_buf_len=32M in the shared cmdline and refuses to build without it in grub.cfg; 10lemonade seeds a fresh config only when the file is absent, uses setdefault so an existing log_level is never overwritten (and says so), and degrades to lemond's default when python3 is missing; 1bit-logs runs one sleeper with two cadences (10 s thermal gated on the engine child, 15 s copy) and calls 1bit-proof boot from the loop, after the /data wait. bash -n clean on all six files; shellcheck -S warning clean except the pre-existing ulimit -l (SC3045) in 10lemonade, which this PR does not touch.

The finding: "U5 boot proof" is an INFO row, so it cannot fail. Its expected value is None, and vm-check.sh prints INFO for want is None — never PASS or FAIL. A proof file that exists with none of the labels would show missing-cmdline missing-uname … in an INFO line and the run would still be green. That is the same class as the two rows this cycle set out to fix, and it is the one of the four that is supposed to catch "a stub would produce this". Two ways to fix, both small:

• add a positive sentinel to the row and expect it — … done; echo boot-proof-ok after the label loop, "boot-proof-ok" as the expectation (the row already loops over the exact label list, so this is one echo), or
• expect a minimum, e.g. "proof-lines=41"-style — but a count is weaker than the labels, so the sentinel is better.

Rows 1, 3 and 4 do gate (ring-sized, thermal-gate-consistent, proof-bundle-ok), so the pattern is already in the file.

Two smaller notes:

1. bundle() refuses when the boot directory holds fewer than 2 entries (n -lt 2) and says "refusing to bundle an empty proof" — I hit this with boot-proof.txt alone, which is not empty, and the message sends a reader looking for a missing proof rather than a missing second file. Either allow 1 (-lt 1) — the proof is the artifact — or word it as "holds 1 file(s), needs at least 2 (the proof plus at least one log)".
2. Your body says 12 firmware hashes on this box; I get 13 here (firmware.amdnpu.* blobs). Not a problem — it is a live count and a blob has evidently been added — just noting the number is not stable.

Not verified: the on-image behaviour (a real boot with /data mounted, the console's Logs item offering the bundle, the thermal row on a loaded box). Everything above is the tool run against a scratch directory plus the scripts' own logic.

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit ec8bb2b)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 1051 - Partially compliant Compliant requirements: The PR introduces a new mechanism to verify module source versions via --srcversion flag in g3-probe.sh, which helps ensure that the correct source code is used for building modules, thereby preventing issues related to stale or incorrect module builds. The PR enhances logging and boot verification by adding a boot-proof.txt file that captures detailed information about the boot environment, including kernel command line, Secure Boot status, module versions, firmware details, memory, and timeline. The PR adds a thermal logging mechanism that only logs when a model is loaded, preventing unnecessary writes when the system is idle. The PR ensures that boot logs are persisted to /data/logs/<boot-id>/ and that the boot proof is written once per boot, which helps in debugging and verifying the system state. Non-compliant requirements: The PR does not address the specific issue of the engine waiting on the runlist with an infinite timeout, which is a key requirement for making hung dispatches self-describing. The PR does not implement the fallback mechanism to fail the request instead of churning contexts when add_config() fails. Requires further human verification: The effectiveness of the boot proof mechanism in capturing all necessary information for debugging hangs needs to be verified through real-world testing. The thermal logging mechanism needs to be validated to ensure it correctly identifies when a model is loaded and logs accordingly.

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪

🧪 No relevant tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Missing Timeout Handling for Dispatch Hangs The PR introduces mechanisms for verifying boot proofs and logging, but does not address the core issue of infinite timeout handling for dispatch hangs. The engine still waits on the runlist with an infinite timeout, which means a hung dispatch can go unnoticed and potentially cause the whole NPU to become unusable until a driver reload. This is a critical issue that needs to be addressed to prevent the described TDR recovery failure. #!/bin/sh # Copyright 2026 bong-water-water-bong # SPDX-License-Identifier: Apache-2.0 # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # 1bit-proof - this boot's evidence, as one readable file and as one attachable archive. # # Cycle 25, owner: "make sure the live usb log verbose it all for proof that it works." 1bit-logs keeps the running # logs; this is the other half: the machine's identity as a short set of `label value` lines (so a person and a gate # read the same thing), and a bundle of the whole boot directory that can be attached to a release as evidence. # # boot-proof.txt once per boot, in this boot's log directory: what machine and entry this was, what Secure Boot # said, which drivers and firmware were loaded (with the fingerprints that name their source), the # memory the demo had, how long the boot took, and whether the release's own SHA256SUMS verified. # # Usage: # 1bit-proof boot [--force] write boot-proof.txt into this boot's directory (nothing is written twice) # 1bit-proof bundle this boot's directory + SHA256SUMS into /data/proof-<tag>-<date>.tar.zst # 1bit-proof path print this boot's log directory, or nothing when it is not persisting # # Exit: 0 ok, 1 nothing to prove (no /data mount, or an empty boot directory), 2 usage. # # A value that cannot be read is written as `n/a <reason>` and never omitted: a missing line and a zero look the same # to a reviewer, and "we could not read the fan" is a different claim from "the fan reads 0 rpm". set -u DATA=${ONEBIT_LOG_DATA:-/data} ROOT=$DATA/logs LOGS=${ONEBIT_LOGS_BIN:-/usr/local/bin/1bit-logs} say() { printf '%s\n' "$*"; } bootdir() { # 1bit-logs owns the directory (and the rule that /data must be a real mount, not a directory on the RAM root) [ -x "$LOGS" ] || return 1 "$LOGS" dir 2> /dev/null } why_no_dir() { # Two very different causes read the same from the caller, and saying the wrong one sends a person to the mount # table for a missing script. Measured here: with 1bit-logs absent the message claimed /data was unmounted. if [ ! -x "$LOGS" ]; then say "$LOGS is missing" else say "$DATA is not a mounted partition" fi } line() { # line <label> <value...> label=$1 shift say "$label $*" } firmware_section() { dir=/lib/firmware/amdnpu if [ ! -d "$dir" ]; then line firmware "n/a no /lib/firmware/amdnpu" else n=$(ls -1 "$dir" 2> /dev/null | wc -l) bytes=$(du -sk "$dir" 2> /dev/null | awk '{print $1}') line firmware "files=$n kb=${bytes:-n/a}" # The fingerprint of what is actually on the disk, not of what should be. Measured: each amdnpu entry is a # DIRECTORY of .zst blobs (17f0_11 holds four), so a plain sha256sum of the entry fails and would have been # reported as "unreadable" - a firmware section that cannot be read is not evidence. for d in $(ls -1 "$dir" 2> /dev/null | sort | head -8); do if [ -d "$dir/$d" ]; then for f in $(ls -1 "$dir/$d" 2> /dev/null | sort); do h=$(sha256sum "$dir/$d/$f" 2> /dev/null | cut -d' ' -f1) line "firmware.amdnpu.$d/$f" "${h:-n/a unreadable}" done else h=$(sha256sum "$dir/$d" 2> /dev/null | cut -d' ' -f1) line "firmware.amdnpu.$d" "${h:-n/a unreadable}" fi done fi fw=$(dmesg 2> /dev/null | grep -iE 'amdgpu.*(firmware|fw)' | head -3 | tr '\n' ';') line firmware.amdgpu "${fw:-n/a no amdgpu firmware line in dmesg}" } release_section() { tag=$(head -1 "$DATA/RELEASE-TAG" 2> /dev/null) line release "${tag:-n/a no $DATA/RELEASE-TAG}" if [ -f "$DATA/SHA256SUMS" ]; then # sha256sum -c prints one line per entry; count them instead of trusting the exit status alone # Count the per-entry markers, not the lines: `sha256sum -c` also prints a WARNING summary line, so counting # "not OK" reported 2 bad for one bad file (measured here with one good and one deliberately wrong entry). sum=$(cd "$DATA" && sha256sum -c SHA256SUMS 2>&1) ok=$(printf '%s\n' "$sum" | grep -c ': OK$') bad=$(printf '%s\n' "$sum" | grep -c ': FAILED$') rc=0 (cd "$DATA" && sha256sum -c SHA256SUMS > /dev/null 2>&1) || rc=$? line sha256sums "ok=$ok bad=$bad rc=$rc" else line sha256sums "n/a no $DATA/SHA256SUMS" fi cfg=/data/lemonade/config/config.json [ -f "$cfg" ] || cfg=/var/lib/lemonade/config/config.json lvl=$(sed -n 's/.*"log_level"[[:space:]]*:[[:space:]]*"\([a-z]*\)".*/\1/p' "$cfg" 2> /dev/null | head -1) line lemond.log_level "${lvl:-n/a no log_level in $cfg}" } write_boot_proof() { force=${1:-} d=$(bootdir) || { say "1bit-proof: nothing persists ($(why_no_dir))" >&2; return 1; } out=$d/boot-proof.txt if [ -f "$out" ] && [ "$force" != --force ]; then say "1bit-proof: $out already written this boot (--force rewrites it)" return 0 fi { line written "$(date -u +%FT%TZ 2> /dev/null)" line bootid "$(basename "$d")" line hostname "$(hostname 2> /dev/null)" line uptime_s "$(cut -d' ' -f1 /proc/uptime 2> /dev/null)" line uname "$(uname -a 2> /dev/null)" line cmdline "$(cat /proc/cmdline 2> /dev/null)" ring=$(dmesg 2> /dev/null | wc -c) if [ "${ring:-0}" -gt 0 ]; then line kernel.ring "$ring bytes in the ring now" else line kernel.ring "n/a dmesg is empty or restricted to root (kernel.dmesg_restrict)" fi # Secure Boot, read the two ways userspace reads it: the tool and the firmware variable itself line secureboot.mokutil "$(mokutil --sb-state 2>&1 | head -1)" efivar=$(ls /sys/firmware/efi/efivars/SecureBoot-* 2> /dev/null | head -1) if [ -n "$efivar" ]; then byte=$(od -An -tu1 -j4 -N1 "$efivar" 2> /dev/null | tr -d ' ') line secureboot.efivar "byte=$byte (1 = enabled)" else line secureboot.efivar "n/a no SecureBoot efivar (not an EFI boot, or efivarfs is not mounted)" fi line lockdown "$(cat /sys/kernel/security/lockdown 2> /dev/null || echo 'n/a no /sys/kernel/security/lockdown')" # the drivers, with the fingerprint that names the source they were built from (srcversion is what g3-probe # and #1038 use; a module's mere presence says nothing about which tree it came from) for m in amdxdna amdgpu ec_su_axb35; do sv=$(modinfo -F srcversion "$m" 2> /dev/null) vm=$(modinfo -F vermagic "$m" 2> /dev/null | cut -d' ' -f1) loaded=no [ -d "/sys/module/$m" ] && loaded=yes line "module.$m" "loaded=$loaded srcversion=${sv:-n/a} vermagic=${vm:-n/a}" done # Present-but-unreadable (the parameter file is root-only) is a different fact from absent, and "not loaded" # was wrong while the module WAS loaded (measured). Say which one it is. tdrp=/sys/module/amdxdna/parameters/tdr_timeout_ms if [ ! -e "$tdrp" ]; then line amdxdna.tdr_timeout_ms "n/a no such parameter (amdxdna not loaded)" else tdrv=$(cat "$tdrp" 2> /dev/null) line amdxdna.tdr_timeout_ms "${tdrv:-n/a unreadable as $(id -un 2> /dev/null)}" fi line amdgpu.cwsr_enable "$(cat /sys/module/amdgpu/parameters/cwsr_enable 2> /dev/null || echo 'n/a')" firmware_section # devices, as the kernel names them line pci.devices "$(lspci -nn 2> /dev/null | grep -iE 'vga|display|accel|npu' | tr '\n' ';' || echo 'n/a no lspci')" line dev.accel "$(ls -1 /dev/accel* 2> /dev/null | tr '\n' ' ' || true)" line dev.kfd "$([ -c /dev/kfd ] && echo present || echo absent)" # the memory a demo actually had line memtotal_kb "$(awk '/^MemTotal:/{print $2}' /proc/meminfo 2> /dev/null)" line ttm.pages_limit "$(cat /sys/module/ttm/parameters/pages_limit 2> /dev/null || echo 'n/a')" line ttm.page_pool_size "$(cat /sys/module/ttm/parameters/page_pool_size 2> /dev/null || echo 'n/a')" vram_t=$(cat /sys/class/drm/card*/device/mem_info_vram_total 2> /dev/null | head -1) vram_u=$(cat /sys/class/drm/card*/device/mem_info_vram_used 2> /dev/null | head -1) line gpu.vram "used=${vram_u:-n/a} total=${vram_t:-n/a}" # the boot's own timeline: the initrd says what it was doing and when, and dmesg anchors both ends tl=$(dmesg 2> /dev/null | grep -m3 '^\[.*1bit: ' | tr '\n' ';') line timeline.1bit "${tl:-n/a no 1bit: lines in the ring (nothing yet, or dmesg restricted)}" tfirst=$(dmesg 2> /dev/null | head -1 | cut -c1-24) tlast=$(dmesg 2> /dev/null | tail -1 | cut -c1-24) if [ -n "$tfirst" ]; then line timeline.dmesg "first=$tfirst last=$tlast" else line timeline.dmesg "n/a dmesg is empty or restricted to root" fi release_section } > "$out" 2> /dev/null say "$out" } bundle() { d=$(bootdir) || { say "1bit-proof: nothing persists ($(why_no_dir))" >&2; return 1; } n=$(ls -1 "$d" 2> /dev/null | wc -l) if [ "${n:-0}" -lt 2 ]; then say "1bit-proof: $d holds $n file(s) - refusing to bundle an empty proof" >&2 return 1 fi # the file itself is part of the proof, and so is the release's own checksum list when the stick carries one [ -f "$d/boot-proof.txt" ] || write_boot_proof > /dev/null 2>&1 tag=$(head -1 "$DATA/RELEASE-TAG" 2> /dev/null) tag=$(printf '%s' "${tag:-$(hostname 2> /dev/null)}" | tr -c 'A-Za-z0-9._-' '_') stamp=$(date -u +%Y%m%dT%H%M%SZ 2> /dev/null) out=$DATA/proof-$tag-$stamp.tar.zst if command -v zstd > /dev/null 2>&1; then { if [ -f "$DATA/SHA256SUMS" ]; then tar -C "$ROOT" -cf - "$(basename "$d")" -C "$DATA" SHA256SUMS 2> /dev/null else tar -C "$ROOT" -cf - "$(basename "$d")" 2> /dev/null fi } | zstd -q -3 -o "$out" || { say "1bit-proof: bundling failed" >&2; return 1; } else say "1bit-proof: no zstd in the image - bundling as .tar instead" >&2 out=${out%.zst} if [ -f "$DATA/SHA256SUMS" ]; then tar -C "$ROOT" -cf "$out" "$(basename "$d")" -C "$DATA" SHA256SUMS || return 1 else tar -C "$ROOT" -cf "$out" "$(basename "$d")" || return 1 fi fi bytes=$(wc -c < "$out" 2> /dev/null) sum=$(sha256sum "$out" 2> /dev/null | cut -d' ' -f1) say "$out" say "bytes ${bytes:-n/a}" say "sha256 ${sum:-n/a}" return 0 } case ${1:-} in '') bundle ;; boot) shift; write_boot_proof "${1:-}" ;; path) bootdir ;; *) say "usage: $0 [boot [--force]|path]"; exit 2 ;; esac Context Churning Not Prevented The PR does not implement the fallback mechanism to fail the request instead of churning contexts when add_config() fails. This is a key requirement to prevent the hang from causing a TDR recovery failure that can brick the array. The engine should fail the request gracefully instead of silently degrading into a mode that can cause system-wide issues. #!/bin/sh # Copyright 2026 bong-water-water-bong # SPDX-License-Identifier: Apache-2.0 # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # 1bit-proof - this boot's evidence, as one readable file and as one attachable archive. # # Cycle 25, owner: "make sure the live usb log verbose it all for proof that it works." 1bit-logs keeps the running # logs; this is the other half: the machine's identity as a short set of `label value` lines (so a person and a gate # read the same thing), and a bundle of the whole boot directory that can be attached to a release as evidence. # # boot-proof.txt once per boot, in this boot's log directory: what machine and entry this was, what Secure Boot # said, which drivers and firmware were loaded (with the fingerprints that name their source), the # memory the demo had, how long the boot took, and whether the release's own SHA256SUMS verified. # # Usage: # 1bit-proof boot [--force] write boot-proof.txt into this boot's directory (nothing is written twice) # 1bit-proof bundle this boot's directory + SHA256SUMS into /data/proof-<tag>-<date>.tar.zst # 1bit-proof path print this boot's log directory, or nothing when it is not persisting # # Exit: 0 ok, 1 nothing to prove (no /data mount, or an empty boot directory), 2 usage. # # A value that cannot be read is written as `n/a <reason>` and never omitted: a missing line and a zero look the same # to a reviewer, and "we could not read the fan" is a different claim from "the fan reads 0 rpm". set -u DATA=${ONEBIT_LOG_DATA:-/data} ROOT=$DATA/logs LOGS=${ONEBIT_LOGS_BIN:-/usr/local/bin/1bit-logs} say() { printf '%s\n' "$*"; } bootdir() { # 1bit-logs owns the directory (and the rule that /data must be a real mount, not a directory on the RAM root) [ -x "$LOGS" ] || return 1 "$LOGS" dir 2> /dev/null } why_no_dir() { # Two very different causes read the same from the caller, and saying the wrong one sends a person to the mount # table for a missing script. Measured here: with 1bit-logs absent the message claimed /data was unmounted. if [ ! -x "$LOGS" ]; then say "$LOGS is missing" else say "$DATA is not a mounted partition" fi } line() { # line <label> <value...> label=$1 shift say "$label $*" } firmware_section() { dir=/lib/firmware/amdnpu if [ ! -d "$dir" ]; then line firmware "n/a no /lib/firmware/amdnpu" else n=$(ls -1 "$dir" 2> /dev/null | wc -l) bytes=$(du -sk "$dir" 2> /dev/null | awk '{print $1}') line firmware "files=$n kb=${bytes:-n/a}" # The fingerprint of what is actually on the disk, not of what should be. Measured: each amdnpu entry is a # DIRECTORY of .zst blobs (17f0_11 holds four), so a plain sha256sum of the entry fails and would have been # reported as "unreadable" - a firmware section that cannot be read is not evidence. for d in $(ls -1 "$dir" 2> /dev/null | sort | head -8); do if [ -d "$dir/$d" ]; then for f in $(ls -1 "$dir/$d" 2> /dev/null | sort); do h=$(sha256sum "$dir/$d/$f" 2> /dev/null | cut -d' ' -f1) line "firmware.amdnpu.$d/$f" "${h:-n/a unreadable}" done else h=$(sha256sum "$dir/$d" 2> /dev/null | cut -d' ' -f1) line "firmware.amdnpu.$d" "${h:-n/a unreadable}" fi done fi fw=$(dmesg 2> /dev/null | grep -iE 'amdgpu.*(firmware|fw)' | head -3 | tr '\n' ';') line firmware.amdgpu "${fw:-n/a no amdgpu firmware line in dmesg}" } release_section() { tag=$(head -1 "$DATA/RELEASE-TAG" 2> /dev/null) line release "${tag:-n/a no $DATA/RELEASE-TAG}" if [ -f "$DATA/SHA256SUMS" ]; then # sha256sum -c prints one line per entry; count them instead of trusting the exit status alone # Count the per-entry markers, not the lines: `sha256sum -c` also prints a WARNING summary line, so counting # "not OK" reported 2 bad for one bad file (measured here with one good and one deliberately wrong entry). sum=$(cd "$DATA" && sha256sum -c SHA256SUMS 2>&1) ok=$(printf '%s\n' "$sum" | grep -c ': OK$') bad=$(printf '%s\n' "$sum" | grep -c ': FAILED$') rc=0 (cd "$DATA" && sha256sum -c SHA256SUMS > /dev/null 2>&1) || rc=$? line sha256sums "ok=$ok bad=$bad rc=$rc" else line sha256sums "n/a no $DATA/SHA256SUMS" fi cfg=/data/lemonade/config/config.json [ -f "$cfg" ] || cfg=/var/lib/lemonade/config/config.json lvl=$(sed -n 's/.*"log_level"[[:space:]]*:[[:space:]]*"\([a-z]*\)".*/\1/p' "$cfg" 2> /dev/null | head -1) line lemond.log_level "${lvl:-n/a no log_level in $cfg}" } write_boot_proof() { force=${1:-} d=$(bootdir) || { say "1bit-proof: nothing persists ($(why_no_dir))" >&2; return 1; } out=$d/boot-proof.txt if [ -f "$out" ] && [ "$force" != --force ]; then say "1bit-proof: $out already written this boot (--force rewrites it)" return 0 fi { line written "$(date -u +%FT%TZ 2> /dev/null)" line bootid "$(basename "$d")" line hostname "$(hostname 2> /dev/null)" line uptime_s "$(cut -d' ' -f1 /proc/uptime 2> /dev/null)" line uname "$(uname -a 2> /dev/null)" line cmdline "$(cat /proc/cmdline 2> /dev/null)" ring=$(dmesg 2> /dev/null | wc -c) if [ "${ring:-0}" -gt 0 ]; then line kernel.ring "$ring bytes in the ring now" else line kernel.ring "n/a dmesg is empty or restricted to root (kernel.dmesg_restrict)" fi # Secure Boot, read the two ways userspace reads it: the tool and the firmware variable itself line secureboot.mokutil "$(mokutil --sb-state 2>&1 | head -1)" efivar=$(ls /sys/firmware/efi/efivars/SecureBoot-* 2> /dev/null | head -1) if [ -n "$efivar" ]; then byte=$(od -An -tu1 -j4 -N1 "$efivar" 2> /dev/null | tr -d ' ') line secureboot.efivar "byte=$byte (1 = enabled)" else line secureboot.efivar "n/a no SecureBoot efivar (not an EFI boot, or efivarfs is not mounted)" fi line lockdown "$(cat /sys/kernel/security/lockdown 2> /dev/null || echo 'n/a no /sys/kernel/security/lockdown')" # the drivers, with the fingerprint that names the source they were built from (srcversion is what g3-probe # and #1038 use; a module's mere presence says nothing about which tree it came from) for m in amdxdna amdgpu ec_su_axb35; do sv=$(modinfo -F srcversion "$m" 2> /dev/null) vm=$(modinfo -F vermagic "$m" 2> /dev/null | cut -d' ' -f1) loaded=no [ -d "/sys/module/$m" ] && loaded=yes line "module.$m" "loaded=$loaded srcversion=${sv:-n/a} vermagic=${vm:-n/a}" done # Present-but-unreadable (the parameter file is root-only) is a different fact from absent, and "not loaded" # was wrong while the module WAS loaded (measured). Say which one it is. tdrp=/sys/module/amdxdna/parameters/tdr_timeout_ms if [ ! -e "$tdrp" ]; then line amdxdna.tdr_timeout_ms "n/a no such parameter (amdxdna not loaded)" else tdrv=$(cat "$tdrp" 2> /dev/null) line amdxdna.tdr_timeout_ms "${tdrv:-n/a unreadable as $(id -un 2> /dev/null)}" fi line amdgpu.cwsr_enable "$(cat /sys/module/amdgpu/parameters/cwsr_enable 2> /dev/null || echo 'n/a')" firmware_section # devices, as the kernel names them line pci.devices "$(lspci -nn 2> /dev/null | grep -iE 'vga|display|accel|npu' | tr '\n' ';' || echo 'n/a no lspci')" line dev.accel "$(ls -1 /dev/accel* 2> /dev/null | tr '\n' ' ' || true)" line dev.kfd "$([ -c /dev/kfd ] && echo present || echo absent)" # the memory a demo actually had line memtotal_kb "$(awk '/^MemTotal:/{print $2}' /proc/meminfo 2> /dev/null)" line ttm.pages_limit "$(cat /sys/module/ttm/parameters/pages_limit 2> /dev/null || echo 'n/a')" line ttm.page_pool_size "$(cat /sys/module/ttm/parameters/page_pool_size 2> /dev/null || echo 'n/a')" vram_t=$(cat /sys/class/drm/card*/device/mem_info_vram_total 2> /dev/null | head -1) vram_u=$(cat /sys/class/drm/card*/device/mem_info_vram_used 2> /dev/null | head -1) line gpu.vram "used=${vram_u:-n/a} total=${vram_t:-n/a}" # the boot's own timeline: the initrd says what it was doing and when, and dmesg anchors both ends tl=$(dmesg 2> /dev/null | grep -m3 '^\[.*1bit: ' | tr '\n' ';') line timeline.1bit "${tl:-n/a no 1bit: lines in the ring (nothing yet, or dmesg restricted)}" tfirst=$(dmesg 2> /dev/null | head -1 | cut -c1-24) tlast=$(dmesg 2> /dev/null | tail -1 | cut -c1-24) if [ -n "$tfirst" ]; then line timeline.dmesg "first=$tfirst last=$tlast" else line timeline.dmesg "n/a dmesg is empty or restricted to root" fi release_section } > "$out" 2> /dev/null say "$out" } bundle() { d=$(bootdir) || { say "1bit-proof: nothing persists ($(why_no_dir))" >&2; return 1; } n=$(ls -1 "$d" 2> /dev/null | wc -l) if [ "${n:-0}" -lt 2 ]; then say "1bit-proof: $d holds $n file(s) - refusing to bundle an empty proof" >&2 return 1 fi # the file itself is part of the proof, and so is the release's own checksum list when the stick carries one [ -f "$d/boot-proof.txt" ] || write_boot_proof > /dev/null 2>&1 tag=$(head -1 "$DATA/RELEASE-TAG" 2> /dev/null) tag=$(printf '%s' "${tag:-$(hostname 2> /dev/null)}" | tr -c 'A-Za-z0-9._-' '_') stamp=$(date -u +%Y%m%dT%H%M%SZ 2> /dev/null) out=$DATA/proof-$tag-$stamp.tar.zst if command -v zstd > /dev/null 2>&1; then { if [ -f "$DATA/SHA256SUMS" ]; then tar -C "$ROOT" -cf - "$(basename "$d")" -C "$DATA" SHA256SUMS 2> /dev/null else tar -C "$ROOT" -cf - "$(basename "$d")" 2> /dev/null fi } | zstd -q -3 -o "$out" || { say "1bit-proof: bundling failed" >&2; return 1; } else say "1bit-proof: no zstd in the image - bundling as .tar instead" >&2 out=${out%.zst} if [ -f "$DATA/SHA256SUMS" ]; then tar -C "$ROOT" -cf "$out" "$(basename "$d")" -C "$DATA" SHA256SUMS || return 1 else tar -C "$ROOT" -cf "$out" "$(basename "$d")" || return 1 fi fi bytes=$(wc -c < "$out" 2> /dev/null) sum=$(sha256sum "$out" 2> /dev/null | cut -d' ' -f1) say "$out" say "bytes ${bytes:-n/a}" say "sha256 ${sum:-n/a}" return 0 } case ${1:-} in '') bundle ;; boot) shift; write_boot_proof "${1:-}" ;; path) bootdir ;; *) say "usage: $0 [boot [--force]|path]"; exit 2 ;; esac

[bong-water-water-bong]

VERDICT: READY at 3f2643ac — the row I flagged now genuinely gates, and I exercised all three directions rather than reading the diff.

The row is a positive sentinel now (boot-proof-complete, emitted only when no label is missing and the proof has at least 20 lines), with that as the expected value, so want is None no longer applies to it. Ran the row's own shell against a real proof and against the two ways it should refuse:

good proof (41 lines, every label)      -> proof-lines=41 missing=none        boot-proof-complete     <- PASS
stub proof (exists, one line)           -> proof-lines=1  missing= cmdline uname secureboot.efivar
                                           module.amdxdna amdxdna.tdr_timeout_ms firmware pci.devices
                                           memtotal_kb timeline.dmesg sha256sums   (no sentinel)     <- FAIL
missing file entirely                   -> proof-lines=0  missing=<all ten>     (no sentinel)          <- FAIL
restored                                -> boot-proof-complete                                          <- PASS

So "a file that exists but says nothing is what a stub produces" is now a gate and not a transcript line, which was the one thing standing between this PR and my READY.

The line-count floor of 20 is a sensible second belt: my real proof is 41 lines on this box (host-invoked, dmesg restricted), and the smallest honest one would still be well above 20 even with several n/a <reason> values, since every label writes a line by design.

The two smaller notes from my first review stand and are unchanged in this head: bundle()'s n -lt 2 threshold wording ("refusing to bundle an empty proof" for a directory holding one file) and the firmware-hash count being live (13 here, 12 in the body). Neither blocks.

[github-actions]

Persistent review updated to latest commit ec8bb2b

[bong-water-water-bong]

VERDICT: READY at ec8bb2b2 (rebase-only for this PR; the review stands).

The rebase brought #1053's "U2 console engine" row in with the newer base — that is the only change in vm-check.sh:

git diff 3f2643ac…ec8bb2b2 -- os/v3/puppy/vm-check.sh | grep -c "^+.*U2 console engine"   ->  1  (not this PR's)

and this PR's own row is untouched, sentinel and expectation both:

"[ -z \"$miss\" ] && [ \"${n:-0}\" -ge 20 ] && echo boot-proof-complete",
"boot-proof-complete"),

so my previous verdict holds: the row gates in all three directions (good proof → sentinel; stub proof → ten missing labels, no sentinel; missing file → proof-lines=0, no sentinel), and the 1bit-proof/10lemonade/mkimage.sh/1bit-logs behaviour I exercised by running them is unchanged in this head.