ci(pi5): name the failing test on the Pi too - #159
Merged
Merged
Conversation
#158 taught `make test` to announce each failure as "FAILED: <path> (exit N)" and taught ci.yml to turn those lines into annotations. The Pi job was left out, and the very next red Pi leg proved why that mattered: the name was printed, went to the step log, and stopped there. Actions logs are in no API, so a red Pi leg is a bare exit code to anything that is not a browser — which is the whole problem #158 set out to fix, still unsolved on the one machine with a flaky suite. `make test` now writes through `tee` into pi5-test.log, and a failure step turns any FAILED lines into annotations, falling back to the log tail (and saying so) when nothing named itself. The tee makes the step bash with `set -euo pipefail`. Without it the pipeline's status would be tee's, and a failing `make test` would report success — the identical trap the benchmark step was in until #156. Avoided here rather than repeated. pi5-test.log also joins the uploaded artefact, so the full log is available to whoever can read artefact bytes. Verified by simulation: the pipefail shape exits non-zero where `set -eu` alone exits 0, and the annotation branch turns a FAILED line into one `::error::`; the no-name and no-log branches were exercised too. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LkBr39Z2JxHU1SL63VyPC9
Raspberry Pi 5 —
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#158 taught
make testto announce each failure asFAILED: <path> (exit N), and taughtci.ymlto turn those lines into annotations. The Pi job was left out.The very next red Pi leg proved why that mattered — run 36169842945 on #157:
The Makefile half of #158 is on
main, somake testdid print the failing test's name. It went to the step log and stopped there. Actions logs are in no API, so a red Pi leg is a bare exit code to anything that is not a browser — the whole problem #158 set out to fix, still unsolved on the one machine that has now joined the flaky club.That failure is the same intermittent
make testthat has hit Linux x86-64 and arm64 on GitHub-hosted runners. It is therefore not arm64-specific and not GitHub-specific; it is simply everywhere, and after five encounters its name is still unknown.What this does
make testwrites throughteeintopi5-test.log.if: failure()step turns anyFAILED:lines into annotations, falling back to the log tail — and saying that is what it is doing — when nothing named itself.pi5-test.logjoins the uploaded artefact, so the full log is there for whoever can read artefact bytes.The tee needs pipefail
Adding
| teewithout it would have re-created the exact bug #156 fixed in the benchmark step one file over: in a pipeline only the last command's status counts, and that istee. A failingmake testwould have reported success.So the step is explicitly
shell: bash. Avoided here rather than repeated.Verification
Simulated against real log shapes — the pipefail shape as written exits non-zero, and all three annotation branches behave:
FAILED:line::error::FAILED: build/host-debug/test/test_b (exit 3)::error::make test failed without naming a test; tail follows+ tailThe workflow itself can only be exercised on the hardware, so this PR's own Pi run is the test. Given the observed rate, there is a fair chance it names the flake on the way in.
🤖 Generated with Claude Code
https://claude.ai/code/session_01LkBr39Z2JxHU1SL63VyPC9
Generated by Claude Code