Skip to content

ci(pi5): name the failing test on the Pi too - #159

Merged
geisten merged 1 commit into
mainfrom
ci/pi5-name-the-failing-test
Sep 25, 2026
Merged

geisten merged 1 commit into
mainfrom
ci/pi5-name-the-failing-test

Conversation

@geisten

@geisten geisten commented Sep 25, 2026

Copy link
Copy Markdown
Owner

#158 taught make test to announce each failure as FAILED: <path> (exit N), and taught ci.yml to turn those lines into annotations. The Pi job was left out.

The very next red Pi leg proved why that mattered — run 36169842945 on #157:

 4 success    Verify Pi 5 environment
 5 failure    Clean build and test      <<<
 6 success    Release build
 7 success    Deterministic end-to-end benchmarks

The Makefile half of #158 is on main, so make test did print the failing test's name. It went to the step log and stopped there. Actions logs are in no API, so a red Pi leg is a bare exit code to anything that is not a browser — the whole problem #158 set out to fix, still unsolved on the one machine that has now joined the flaky club.

That failure is the same intermittent make test that has hit Linux x86-64 and arm64 on GitHub-hosted runners. It is therefore not arm64-specific and not GitHub-specific; it is simply everywhere, and after five encounters its name is still unknown.

What this does

  • make test writes through tee into pi5-test.log.
  • A if: failure() step turns any FAILED: lines into annotations, falling back to the log tail — and saying that is what it is doing — when nothing named itself.
  • pi5-test.log joins the uploaded artefact, so the full log is there for whoever can read artefact bytes.

The tee needs pipefail

Adding | tee without it would have re-created the exact bug #156 fixed in the benchmark step one file over: in a pipeline only the last command's status counts, and that is tee. A failing make test would have reported success.

set -eu            sh failing.sh | tee /dev/null   ->  exit 0
set -euo pipefail  sh failing.sh | tee /dev/null   ->  exit 1

So the step is explicitly shell: bash. Avoided here rather than repeated.

Verification

Simulated against real log shapes — the pipefail shape as written exits non-zero, and all three annotation branches behave:

case output
log with a FAILED: line ::error::FAILED: build/host-debug/test/test_b (exit 3)
log without one ::error::make test failed without naming a test; tail follows + tail
no log at all exits 0, silent

The workflow itself can only be exercised on the hardware, so this PR's own Pi run is the test. Given the observed rate, there is a fair chance it names the flake on the way in.

🤖 Generated with Claude Code

https://claude.ai/code/session_01LkBr39Z2JxHU1SL63VyPC9


Generated by Claude Code

#158 taught `make test` to announce each failure as
"FAILED: <path> (exit N)" and taught ci.yml to turn those lines into
annotations. The Pi job was left out, and the very next red Pi leg
proved why that mattered: the name was printed, went to the step log,
and stopped there. Actions logs are in no API, so a red Pi leg is a bare
exit code to anything that is not a browser — which is the whole problem
#158 set out to fix, still unsolved on the one machine with a flaky
suite.

`make test` now writes through `tee` into pi5-test.log, and a failure
step turns any FAILED lines into annotations, falling back to the log
tail (and saying so) when nothing named itself.

The tee makes the step bash with `set -euo pipefail`. Without it the
pipeline's status would be tee's, and a failing `make test` would report
success — the identical trap the benchmark step was in until #156.
Avoided here rather than repeated.

pi5-test.log also joins the uploaded artefact, so the full log is
available to whoever can read artefact bytes.

Verified by simulation: the pipefail shape exits non-zero where `set -eu`
alone exits 0, and the annotation branch turns a FAILED line into one
`::error::`; the no-name and no-log branches were exercised too.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LkBr39Z2JxHU1SL63VyPC9
@github-actions

Copy link
Copy Markdown

Raspberry Pi 5 — 04c4529f

job success · run 36170824629

Eval-gated improvement


=== eval-gated self-improvement benchmark =====================
train suite     : examples/eval/bench/train.spg
held-out suite  : examples/eval/bench/holdout.spg  (5 cases)
---------------------------------------------------------------
held-out pass   : 3/5  ->  5/5   (baseline -> after learning)
lessons         : 2 proposed, 1 kept, 1 reverted
regressions kept: 0  (invariant: final >= baseline)
---------------------------------------------------------------
benchmark: PASS

Context cost

lessons  geistshell_bytes/tick  full_index_bytes/tick 
1        72                     90                    
4        72                     360                   
8        72                     720                   
16       72                     1454                  
32       72                     2233                  

@geisten
geisten merged commit 4a4aa0c into main Sep 25, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants