Repository navigation
Fail provider DB test jobs when their tests fail - #74172
Merged
Merged
Conversation
zozo123
requested review from
amoghrajesh,
ashb,
bugraoz93,
choo121600,
ephraimbuddy,
gopidesupavan,
jason810496,
jedcunningham,
jscheffl,
potiuk and
vatsrahul1001
as code owners
October 3, 2026 19:13
zozo123
force-pushed
the
ci/providers-db-failures-fail
branch
from
October 3, 2026 22:35
51aae15 to
783ca0b
Compare
run_unit_tests.sh read exit code 1 from the provider DB test run as "no DB tests were collected" and passed the job. A parallel breeze run exits 1 whenever any test type fails, so failing provider DB tests have been reported green; only a timeout (exit 2) still failed the job. Core DB jobs never had the exception. A test type without DB tests is still not a failure: pytest exits 5 once --run-db-tests-only has deselected all of its tests. That can only be recognised per test type, before the parallel run folds every return code into 1, so breeze now decides it there.
Only an empty DB test selection may be normalized to success; test failures and process failures must retain their return codes.
zozo123
force-pushed
the
ci/providers-db-failures-fail
branch
from
October 4, 2026 04:58
783ca0b to
388abad
Compare
shahar1
approved these changes
Oct 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Make provider DB test jobs fail when a test fails instead of treating exit code
1from the parallel Breeze runner as “no DB tests were collected”.The empty-selection case is identified at the individual test-type boundary, where pytest returns exit code
5. Only that code is treated as success when--run-db-tests-onlyis active. All other non-zero test, worker, compose, and timeout results remain failures.Why this matters
A scan of 79 scheduled canary runs on
mainfound 29 provider DB jobs that reported real test failures as successful. This change closes that false-green path while preserving legitimate test types with no DB tests.Validation and measures
test_publish_docs_to_s3.pyreproduce without this change.The change applies consistently to core and provider DB-only runs. It preserves exit code
2for timeouts and passes through all other failure codes.Scope
This PR changes result classification only. It does not reduce test coverage, alter test selection, or change dependency/image caching.
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 5.5) following the guidelines