Repository navigation
Report EC2 Spot interruptions on pull requests and keep them out of Slack - #75615
Merged
Merged
Conversation
…lack Jobs on the self-hosted RunsOn runners fail with exit code 137 when AWS reclaims their Spot instance. Nothing in the job's log says why, so the failure looks like a broken test or an out-of-memory kill, although RunsOn re-runs the job by itself once the run finishes. A pull request should say so as soon as it happens, and the canary should not page anyone on Slack until RunsOn has given up retrying. The Slack notification also never reported how RunsOn's re-run ended, because it only ran for the first attempt. Generated-by: Claude Opus 5
potiuk
requested review from
amoghrajesh,
ashb,
bugraoz93,
choo121600,
ephraimbuddy,
gopidesupavan,
jason810496,
jedcunningham,
jscheffl and
vatsrahul1001
as code owners
October 25, 2026 18:02
boring-cyborg
Bot
added
area:dev-tools
backport-to-v3-4-test
shahar1
approved these changes
Oct 25, 2026
1 task done
Contributor
Backport failed to create: v3-4-test. View the failure log Run detailsNote: As of Merging PRs targeted for Airflow 3.X In matter of doubt please ask in #release-management Slack channel.
You can attempt to backport this manually by running: cherry_picker 4386cb1 v3-4-testThis should apply the commit to the v3-4-test branch and leave the commit in conflict state marking After you have resolved the conflicts, you can continue the backport process by running: cherry_picker --continueIf you don't have cherry-picker installed, see the installation guide. |
1 task done
potiuk
added a commit
that referenced
this pull request
Oct 25, 2026
…, #75615) (#75617) * Keep a Breeze test's --dry-run from leaking into later tests (#75589) The dry-run, verbose and forced-answer options are process-wide, so a test invoking a command with --dry-run made run_command return empty output for every later test on the same xdist worker. Which tests failed depended on how xdist distributed them, so the Breeze unit tests failed intermittently, as in the scheduled ARM run. Generated-by: Claude Opus 5 (cherry picked from commit 0c1fa8e) * Run shared distribution tests in a single CI job (#74566) * Run shared distribution tests in a single CI job The shared distributions matrix starts one runner per directory in shared/ (14 today) on every PR. Each leg spends ~15s on checkout and uv setup to run ~2-4s of tests, and the fan-out takes 14 of the 20 concurrent jobs ASF allows per workflow. Running the distributions sequentially in one job keeps the same coverage, reports every failing distribution, and frees those slots. * Run each shared distribution with uv --exact for isolation * Wrap the shared distributions test loop line to satisfy yamllint Generated-by: Claude Opus 5 --------- Co-authored-by: Jarek Potiuk <jarek@potiuk.com> (cherry picked from commit 6b42e65) * Run maintainer and canary CI on self-hosted RunsOn runners (#74889) This PR implements AIP-118 for integrating self-hosted runners in Apache Airflow's CI, utilizing the project's AWS instance and runs-on.com's runners. (cherry picked from commit b5bc253) Generated-by: Claude Opus 5 * Check newsfragment PR numbers in static checks instead of a separate workflow (#75612) * Check newsfragment PR numbers in static checks instead of a separate workflow The separate workflow ran outside the fork CI gate (AIP-120), needed its own required check, and needed its own approval for first-time contributors. As part of the existing newsfragment hook it runs wherever static checks run, in project CI and in a contributor's fork, and fails CI passed like any other static check. A newsfragment is usually written before its pull request exists, so a local run cannot know the right number and accepts any. Generated-by: Claude Opus 5 * Address review of the newsfragment PR number check A pull request inside a fork carries the fork's number, so only apache/airflow pull requests are checked. A GitHub API error is reported without a traceback, and the instructions say who re-runs the checks when CI runs in the contributor's fork. Generated-by: Claude Opus 5 (cherry picked from commit 757898c) The documentation conflicts were resolved by keeping the release branch's text and updating only its sentences about the newsfragment check. * Report EC2 Spot interruptions on pull requests and keep them out of Slack (#75615) Jobs on the self-hosted RunsOn runners fail with exit code 137 when AWS reclaims their Spot instance. Nothing in the job's log says why, so the failure looks like a broken test or an out-of-memory kill, although RunsOn re-runs the job by itself once the run finishes. A pull request should say so as soon as it happens, and the canary should not page anyone on Slack until RunsOn has given up retrying. The Slack notification also never reported how RunsOn's re-run ended, because it only ran for the first attempt. Generated-by: Claude Opus 5 (cherry picked from commit 4386cb1)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Jobs on the self-hosted RunsOn runners fail with exit code 137 when AWS reclaims their Spot instance. Nothing in the job log says why, so it looks like a broken test or an out-of-memory kill, although RunsOn re-runs the job by itself once the run finishes. Today's canary and #75612 were both hit.
spot-interruption-notice.ymlchecks the newest "Tests (AMD)" run of each PR every 10 minutes, including runs still in progress, and keeps one comment on the PR: which jobs were interrupted and that RunsOn re-runs them, then how the re-run ended. It runs on a schedule, so no PR run gets write access.notify-slackno longer posts failures that are only Spot interruptions RunsOn will retry. It posts for RunsOn's re-run instead (which it previously never ran for, so recoveries went unreported), and still posts when RunsOn gives up after attempt 3.05_workflows.md: about 2% of the repository'sGITHUB_TOKENlimit.Was generative AI tooling used to co-author this PR?
Generated-by: Claude Opus 5 following the guidelines