Repository navigation
Run maintainer and canary CI on self-hosted RunsOn runners - #74889
Merged
Merged
Conversation
boring-cyborg
Bot
added
area:dev-tools
backport-to-v3-4-test
shahar1
marked this pull request as ready for review
October 16, 2026 20:18
shahar1
requested review from
amoghrajesh,
ashb,
bugraoz93,
choo121600,
ephraimbuddy,
gopidesupavan,
jason810496,
jedcunningham,
jscheffl,
potiuk and
vatsrahul1001
as code owners
October 16, 2026 20:18
shahar1
marked this pull request as draft
October 16, 2026 20:21
shahar1
added
the
use self-hosted runners
shahar1
added
dont-merge
full tests needed
shahar1
added a commit
to shahar1/airflow
that referenced
this pull request
Oct 16, 2026
On EC2 types with local NVMe disks (the family prefixes also match the `d` variants, e.g. `m6id` or `m7gd`), RunsOn assembles them into a RAID0, mounts it at /mnt/ephemeral and /var/lib/docker, and keeps /home/runner and /tmp on it. `rm -rf /mnt/*` expanded to that mount point, emptied it and only then failed with "Device or resource busy", taking the runner's own files with it: the CI image build on apache#74889 lost its runner right after this step with "Could not find file '/home/runner/_diag/pages/...'". Whether it hit depended on which spot type the job got, which is why the apache#74714 runs passed. On self-hosted runners make_mnt_writeable.sh now keeps mount points under /mnt, keeps rm out of other filesystems mounted further down, and chowns only /mnt itself, since a recursive chown would re-own docker's store on that disk. move_docker_to_mnt.sh leaves /var/lib/docker alone when it is already a mount there: emptying it could hit the same disk, and moving docker to the EBS root volume would only make it slower. GitHub-hosted runners keep the previous commands unchanged.
1 task done
shahar1
added a commit
to shahar1/airflow
that referenced
this pull request
Oct 16, 2026
On EC2 types with local NVMe disks (the family prefixes also match the `d` variants, e.g. `m6id` or `m7gd`), RunsOn assembles them into a RAID0, mounts it at /mnt/ephemeral and /var/lib/docker, and keeps /home/runner and /tmp on it. `rm -rf /mnt/*` expanded to that mount point, emptied it and only then failed with "Device or resource busy", taking the runner's own files with it: the CI image build on apache#74889 lost its runner right after this step with "Could not find file '/home/runner/_diag/pages/...'". Whether it hit depended on which spot type the job got, which is why the apache#74714 runs passed. On self-hosted runners make_mnt_writeable.sh now keeps mount points under /mnt, keeps rm out of other filesystems mounted further down, and chowns only /mnt itself, since a recursive chown would re-own docker's store on that disk. move_docker_to_mnt.sh leaves /var/lib/docker alone when it is already a mount there: emptying it could hit the same disk, and moving docker to the EBS root volume would only make it slower. GitHub-hosted runners keep the previous commands unchanged.
shahar1
force-pushed
the
ci-runs-on-runners
branch
from
October 16, 2026 22:07
0a4e843 to
787d7fe
Compare
shahar1
marked this pull request as ready for review
October 17, 2026 04:52
shahar1
added a commit
to shahar1/airflow
that referenced
this pull request
Oct 17, 2026
On EC2 types with local NVMe disks (the family prefixes also match the `d` variants, e.g. `m6id` or `m7gd`), RunsOn assembles them into a RAID0, mounts it at /mnt/ephemeral and /var/lib/docker, and keeps /home/runner and /tmp on it. `rm -rf /mnt/*` expanded to that mount point, emptied it and only then failed with "Device or resource busy", taking the runner's own files with it: the CI image build on apache#74889 lost its runner right after this step with "Could not find file '/home/runner/_diag/pages/...'". Whether it hit depended on which spot type the job got, which is why the apache#74714 runs passed. On self-hosted runners make_mnt_writeable.sh now keeps mount points under /mnt, keeps rm out of other filesystems mounted further down, and chowns only /mnt itself, since a recursive chown would re-own docker's store on that disk. move_docker_to_mnt.sh leaves /var/lib/docker alone when it is already a mount there: emptying it could hit the same disk, and moving docker to the EBS root volume would only make it slower. GitHub-hosted runners keep the previous commands unchanged.
shahar1
force-pushed
the
ci-runs-on-runners
branch
from
October 17, 2026 04:56
787d7fe to
fcae2b9
Compare
This was referenced Oct 17, 2026
shahar1
force-pushed
the
ci-runs-on-runners
branch
from
October 19, 2026 19:22
e8d62bb to
b3c756b
Compare
shahar1
added a commit
to shahar1/airflow
that referenced
this pull request
Oct 19, 2026
On EC2 types with local NVMe disks (the family prefixes also match the `d` variants, e.g. `m6id` or `m7gd`), RunsOn assembles them into a RAID0, mounts it at /mnt/ephemeral and /var/lib/docker, and keeps /home/runner and /tmp on it. `rm -rf /mnt/*` expanded to that mount point, emptied it and only then failed with "Device or resource busy", taking the runner's own files with it: the CI image build on apache#74889 lost its runner right after this step with "Could not find file '/home/runner/_diag/pages/...'". Whether it hit depended on which spot type the job got, which is why the apache#74714 runs passed. On self-hosted runners make_mnt_writeable.sh now keeps mount points under /mnt, keeps rm out of other filesystems mounted further down, and chowns only /mnt itself, since a recursive chown would re-own docker's store on that disk. move_docker_to_mnt.sh leaves /var/lib/docker alone when it is already a mount there: emptying it could hit the same disk, and moving docker to the EBS root volume would only make it slower. GitHub-hosted runners keep the previous commands unchanged.
shahar1
added a commit
to shahar1/airflow
that referenced
this pull request
Oct 20, 2026
On EC2 types with local NVMe disks (the family prefixes also match the `d` variants, e.g. `m6id` or `m7gd`), RunsOn assembles them into a RAID0, mounts it at /mnt/ephemeral and /var/lib/docker, and keeps /home/runner and /tmp on it. `rm -rf /mnt/*` expanded to that mount point, emptied it and only then failed with "Device or resource busy", taking the runner's own files with it: the CI image build on apache#74889 lost its runner right after this step with "Could not find file '/home/runner/_diag/pages/...'". Whether it hit depended on which spot type the job got, which is why the apache#74714 runs passed. On self-hosted runners make_mnt_writeable.sh now keeps mount points under /mnt, keeps rm out of other filesystems mounted further down, and chowns only /mnt itself, since a recursive chown would re-own docker's store on that disk. move_docker_to_mnt.sh leaves /var/lib/docker alone when it is already a mount there: emptying it could hit the same disk, and moving docker to the EBS root volume would only make it slower. GitHub-hosted runners keep the previous commands unchanged.
shahar1
force-pushed
the
ci-runs-on-runners
branch
from
October 20, 2026 20:14
b3c756b to
790a094
Compare
potiuk
pushed a commit
to shahar1/airflow
that referenced
this pull request
Oct 20, 2026
On EC2 types with local NVMe disks (the family prefixes also match the `d` variants, e.g. `m6id` or `m7gd`), RunsOn assembles them into a RAID0, mounts it at /mnt/ephemeral and /var/lib/docker, and keeps /home/runner and /tmp on it. `rm -rf /mnt/*` expanded to that mount point, emptied it and only then failed with "Device or resource busy", taking the runner's own files with it: the CI image build on apache#74889 lost its runner right after this step with "Could not find file '/home/runner/_diag/pages/...'". Whether it hit depended on which spot type the job got, which is why the apache#74714 runs passed. On self-hosted runners make_mnt_writeable.sh now keeps mount points under /mnt, keeps rm out of other filesystems mounted further down, and chowns only /mnt itself, since a recursive chown would re-own docker's store on that disk. move_docker_to_mnt.sh leaves /var/lib/docker alone when it is already a mount there: emptying it could hit the same disk, and moving docker to the EBS root volume would only make it slower. GitHub-hosted runners keep the previous commands unchanged.
potiuk
force-pushed
the
ci-runs-on-runners
branch
from
October 20, 2026 22:38
790a094 to
7aa63f0
Compare
potiuk
reviewed
Oct 20, 2026
potiuk
left a comment
potiuk
left a comment
Member
There was a problem hiding this comment.
Rebased on current main (clean) and pushed; breeze tests pass on the rebased branch.
Thanks, the follow-up commit addresses everything from my previous review: the label is now bound to a committer's approval of the head commit with a standing request for changes as a veto, routing is documented as a preference, vars.USE_PUBLIC_RUNNERS gives main a way back, the S3 stash is streamed while hashed (an overwritten temporary upload only causes a digest-mismatch miss), and the runner scripts, restore path check and cost-summary action look good. The PR also merges cleanly with the AIP-120 stack and #75142 and adds no CI jobs, so the "CI passed" gate's needs stay complete.
One regression from the label change (inline): a re-run reuses its original event's labels, so after use self-hosted runners is removed, re-running an earlier run still routes to RunsOn as long as the head commit is approved. The previous _self_hosted_label_added_by_committer() caught this through the later unlabeled event. Reading the PR's current labels (GET /repos/apache/airflow/pulls/<n>, already one request away from the reviews call) for this decision would fix it.
I'm keeping my request for changes until the merge preconditions listed in the description are met: ASF Infra confirming the approval setting and the stack posture (instance role limited to the cache prefix, runner-to-runner isolation, instance type and runtime limits, S3CacheExpirationInDays >= 6), and AIP-118 being accepted. Happy to re-review as soon as those land.
Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting
shahar1
force-pushed
the
ci-runs-on-runners
branch
from
October 21, 2026 19:23
7aa63f0 to
c7396b4
Compare
shahar1
added a commit
to shahar1/airflow
that referenced
this pull request
Oct 21, 2026
On EC2 types with local NVMe disks (the family prefixes also match the `d` variants, e.g. `m6id` or `m7gd`), RunsOn assembles them into a RAID0, mounts it at /mnt/ephemeral and /var/lib/docker, and keeps /home/runner and /tmp on it. `rm -rf /mnt/*` expanded to that mount point, emptied it and only then failed with "Device or resource busy", taking the runner's own files with it: the CI image build on apache#74889 lost its runner right after this step with "Could not find file '/home/runner/_diag/pages/...'". Whether it hit depended on which spot type the job got, which is why the apache#74714 runs passed. On self-hosted runners make_mnt_writeable.sh now keeps mount points under /mnt, keeps rm out of other filesystems mounted further down, and chowns only /mnt itself, since a recursive chown would re-own docker's store on that disk. move_docker_to_mnt.sh leaves /var/lib/docker alone when it is already a mount there: emptying it could hit the same disk, and moving docker to the EBS root volume would only make it slower. GitHub-hosted runners keep the previous commands unchanged.
potiuk
reviewed
Oct 22, 2026
potiuk
left a comment
potiuk
left a comment
Member
There was a problem hiding this comment.
The re-run regression is fixed, and with AIP-118 accepted, the Infra confirmation is the only thing I'm still waiting on.
Re-reviewed c7396b4 against the last round (7aa63f0). The first three commits changed only through the rebase (the stash/save v1.0.2 and setup-java v6.1.0 pins from main). The new commit is what I asked for:
_is_self_hosted_label_approved()readsGET /pulls/<n>before the reviews and returnsFalsewhenuse self-hosted runnersis no longer on the PR. A re-run of an earlier run therefore stops routing to RunsOn once a committer removes the label, even though the event payload still lists it.- A malformed 200 (
{"message": "Not Found"}) is caught through the addedKeyErrorand counts as "no". The new test confirms that the reviews are not fetched in that case. 04_selective_checks.mdrule 6 and the docstring describe the new check.
CI is green on the head commit (139 passed).
Smaller observation (description only): the Which runs go where paragraph still says "Re-running an earlier run keeps its old labels". That is still true in general, but no longer for removing use self-hosted runners, which now takes effect on a re-run. It's worth one clause there so the description matches rule 6.
On the merge preconditions: AIP-118 passed lazy consensus today. I don't see the ASF Infra confirmation yet: the repository's approval setting, the instance role limited to the cache prefix, runner-to-runner isolation, instance-type and runtime limits, and S3CacheExpirationInDays >= 6. Once that is posted here (or linked), I'll approve. That should still fit the weekend merge.
This review was drafted by an AI-assisted tool and
confirmed by an Apache Airflow maintainer. The findings
below are observations, not blockers; an Apache Airflow
maintainer — a real person — will take the next look at the
PR. If you think a finding is mis-applied, please reply on
the PR and a maintainer will weigh in.More on how Apache Airflow handles maintainer review:
Contributing guide.
potiuk
approved these changes
Oct 22, 2026
potiuk
left a comment
potiuk
left a comment
Member
There was a problem hiding this comment.
LGTM: the re-run regression is fixed, AIP-118 is accepted and CI is green (139 passed). Details are inline.
One change to the description, not the code: the Which runs go where paragraph still says "Re-running an earlier run keeps its old labels". That no longer holds for removing use self-hosted runners, which now takes effect on a re-run. One clause there would match rule 6 in 04_selective_checks.md.
The ASF Infra merge preconditions listed in the description still apply. Please post or link Infra's confirmation here before merging.
This review was drafted by an AI-assisted tool and
confirmed by an Apache Airflow maintainer. The maintainer
approving this PR has read the findings and signed off. If
something feels off, please reply on the PR and a maintainer
will follow up.More on how Apache Airflow handles maintainer review:
Contributing guide.
GitHub-hosted runners cap every job at 4 vCPU / 16 GB, and the full provider and core suites spend most of their time waiting on that. The RunsOn experiment in apache#74714 ran the whole CI on EC2 runners in the project's AWS account and came back green, so this moves the trusted part of CI there: pushes, schedules and dispatches in apache/airflow, and PRs from apache/airflow branches. Code running on those runners can reach the RunsOn stack's AWS resources, so PRs from forks, PRs opened by bots and PRs from Dependabot branches (code nobody has reviewed yet) stay on GitHub-hosted runners. A fork PR moves only when a committer adds the `use self-hosted runners` label; triage collaborators can label PRs too, so the label counts only if the PR's events show a committer added it and nobody removed it since. `use public runners` sends any PR back to GitHub-hosted runners, e.g. when RunsOn is unavailable. The routing lives in selective checks, which already know the event, repository, labels and platform, so the decision is unit-tested instead of spread over workflow expressions. Its `runner-type` output is either the public label or a RunsOn label template, and each job picks its size class with `format(<label>, cpu, ram)`, which leaves a public label untouched. The classes are the EC2 equivalents of those measured in the EKS runner experiment: small 4 vCPU / 16 GB, medium 8 / 32, xl 32 / 128. xl gets 128 GB rather than 64 because xdist runs one worker per vCPU and the non-DB providers suite already peaked at 35 GB with 16 workers. React UI tests get the 8-CPU class, not xl: vitest starts one worker per CPU minus one, and on 32 CPUs the workers starved the `monaco-editor` import in a `beforeAll` hook past its 30s timeout (24s on the one passing run, 3.6s on the 4-CPU GitHub-hosted runner; locally 3 workers 4.5s, 7 workers 8.0s, 15 workers timed out). On RunsOn the multi-GB image tarballs go to the stack's in-region S3 cache bucket instead of leaving AWS as GitHub artifacts. Every RunsOn job can write to that bucket, so it is not trusted: a tarball is stored under its sha256 and only the digest is stashed as a GitHub artifact, which GitHub scopes to the run and branch. Without that, a PR's job could replace an image a canary run restores in the jobs that publish docs and push constraints. The small caches (mount cache, prek, docs inventory) keep using the ASF stash action on both kinds of runners, so PRs on GitHub-hosted runners still get the ones canary runs save. The image stash actions are loaded from the workflow's own checkout because ci-image-build also builds older release refs that predate them. The EC2 cost of each job is added to its summary, on RunsOn only. `amd-runners` / `arm-runners` stay on public runners: the Docker Hub release workflow consumes them and runs with release secrets.
On EC2 types with local NVMe disks (the family prefixes also match the `d` variants, e.g. `m6id` or `m7gd`), RunsOn assembles them into a RAID0, mounts it at /mnt/ephemeral and /var/lib/docker, and keeps /home/runner and /tmp on it. `rm -rf /mnt/*` expanded to that mount point, emptied it and only then failed with "Device or resource busy", taking the runner's own files with it: the CI image build on apache#74889 lost its runner right after this step with "Could not find file '/home/runner/_diag/pages/...'". Whether it hit depended on which spot type the job got, which is why the apache#74714 runs passed. On self-hosted runners make_mnt_writeable.sh now keeps mount points under /mnt, keeps rm out of other filesystems mounted further down, and chowns only /mnt itself, since a recursive chown would re-own docker's store on that disk. move_docker_to_mnt.sh leaves /var/lib/docker alone when it is already a mount there: emptying it could hit the same disk, and moving docker to the EBS root volume would only make it slower. GitHub-hosted runners keep the previous commands unchanged.
Selective checks evaluate the runner choice from the PR's own workflow files, so they cannot keep fork code off RunsOn; that is the job of the RunsOn stack and the repository's Actions settings, and the docs said otherwise. The `use self-hosted runners` label outlived the commit a committer looked at, so it vouched for every later push: it now counts only while a committer's latest review approves the PR's head commit and no committer's request for changes stands. `main` had no way back to public runners without a PR when the stack is unavailable, hence the USE_PUBLIC_RUNNERS repository variable. RunsOn instance types without local NVMe put docker on the EBS root volume, where moving it under /mnt gained nothing and wiped the runner's images. Builds of old refs ran that ref's runner scripts, whose `rm -rf /mnt/*` would empty RunsOn's own disks. The multi-GB image tarball was written, hashed and read again before upload; it is now streamed while hashed. An S3 restore ignoring `path` could let the two runner kinds diverge silently, and jobs without Breeze reported no EC2 cost.
…moved A re-run reuses the labels of its original event, so once a committer removes `use self-hosted runners`, re-running an earlier run would still go to RunsOn while the head commit stays approved. The label is now checked against the PR as it is when the run starts.
shahar1
force-pushed
the
ci-runs-on-runners
branch
from
October 23, 2026 05:38
c7396b4 to
07af0d8
Compare
The AWS CLI copies the source object's tags on an S3-to-S3 copy by default, which needs s3:GetObjectTagging. The RunsOn instance role only allows reading and writing objects, so the first run of the stash save on RunsOn failed with AccessDenied. The temporary upload has no tags or metadata to keep, so nothing is lost by skipping them.
1 task done
Contributor
Backport failed to create: v3-4-test. View the failure log Run detailsNote: As of Merging PRs targeted for Airflow 3.X In matter of doubt please ask in #release-management Slack channel.
You can attempt to backport this manually by running: cherry_picker b5bc253 v3-4-testThis should apply the commit to the v3-4-test branch and leave the commit in conflict state marking After you have resolved the conflicts, you can continue the backport process by running: cherry_picker --continueIf you don't have cherry-picker installed, see the installation guide. |
zozo123
added a commit
to zozo123/airflow
that referenced
this pull request
Oct 25, 2026
Since apache#74889 scheduled canaries run on RunsOn, where the CI image stash goes to RunsOn's S3 cache and only its digest is a GitHub artifact, so every report entry was skipped as "CI image missing". The plan now also accepts that digest artifact; the explain job then runs on a RunsOn runner, downloads exactly that tarball by its digest and checks it like stash-restore does. It prepares the disk with the workflow version's runner scripts, as the image build job now does.
1 task done
1 task done
potiuk
added a commit
that referenced
this pull request
Oct 25, 2026
…, #75615) (#75617) * Keep a Breeze test's --dry-run from leaking into later tests (#75589) The dry-run, verbose and forced-answer options are process-wide, so a test invoking a command with --dry-run made run_command return empty output for every later test on the same xdist worker. Which tests failed depended on how xdist distributed them, so the Breeze unit tests failed intermittently, as in the scheduled ARM run. Generated-by: Claude Opus 5 (cherry picked from commit 0c1fa8e) * Run shared distribution tests in a single CI job (#74566) * Run shared distribution tests in a single CI job The shared distributions matrix starts one runner per directory in shared/ (14 today) on every PR. Each leg spends ~15s on checkout and uv setup to run ~2-4s of tests, and the fan-out takes 14 of the 20 concurrent jobs ASF allows per workflow. Running the distributions sequentially in one job keeps the same coverage, reports every failing distribution, and frees those slots. * Run each shared distribution with uv --exact for isolation * Wrap the shared distributions test loop line to satisfy yamllint Generated-by: Claude Opus 5 --------- Co-authored-by: Jarek Potiuk <jarek@potiuk.com> (cherry picked from commit 6b42e65) * Run maintainer and canary CI on self-hosted RunsOn runners (#74889) This PR implements AIP-118 for integrating self-hosted runners in Apache Airflow's CI, utilizing the project's AWS instance and runs-on.com's runners. (cherry picked from commit b5bc253) Generated-by: Claude Opus 5 * Check newsfragment PR numbers in static checks instead of a separate workflow (#75612) * Check newsfragment PR numbers in static checks instead of a separate workflow The separate workflow ran outside the fork CI gate (AIP-120), needed its own required check, and needed its own approval for first-time contributors. As part of the existing newsfragment hook it runs wherever static checks run, in project CI and in a contributor's fork, and fails CI passed like any other static check. A newsfragment is usually written before its pull request exists, so a local run cannot know the right number and accepts any. Generated-by: Claude Opus 5 * Address review of the newsfragment PR number check A pull request inside a fork carries the fork's number, so only apache/airflow pull requests are checked. A GitHub API error is reported without a traceback, and the instructions say who re-runs the checks when CI runs in the contributor's fork. Generated-by: Claude Opus 5 (cherry picked from commit 757898c) The documentation conflicts were resolved by keeping the release branch's text and updating only its sentences about the newsfragment check. * Report EC2 Spot interruptions on pull requests and keep them out of Slack (#75615) Jobs on the self-hosted RunsOn runners fail with exit code 137 when AWS reclaims their Spot instance. Nothing in the job's log says why, so the failure looks like a broken test or an out-of-memory kill, although RunsOn re-runs the job by itself once the run finishes. A pull request should say so as soon as it happens, and the canary should not page anyone on Slack until RunsOn has given up retrying. The Slack notification also never reported how RunsOn's re-run ended, because it only ran for the first attempt. Generated-by: Claude Opus 5 (cherry picked from commit 4386cb1)
1 task done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR implements AIP-118 for integrating self-hosted runners in Apache Airflow's CI, utilizing the project's AWS instance and runs-on.com's runners.
Note to maintainers: please merge only after AIP-118 is approved.
AI Summary
Moves the trusted part of `Tests (AMD)` / `Tests (ARM)` onto self-hosted [RunsOn](https://runs-on.com) EC2 runners in the Airflow AWS account. Fork PRs stay on GitHub-hosted runners. This builds on the experiment in #74714 (`dont-merge`), which ran the whole CI on RunsOn and came back green. The SFTP test fix it needed on 32-CPU runners landed separately in #74888.Which runs go where
breeze ci selective-checknow outputsrunner-type, and build-info passes it on instead of hardcoding["ubuntu-22.04"]/["ubuntu-22.04-arm"]. The first matching rule wins:apache/airflow(forks' own CI)USE_PUBLIC_RUNNERSistrueuse public runnerspush/schedule/workflow_dispatchinapache/airflow(canary and release-branch pushes)pull_requestuse self-hosted runners, if a committer's (COMMITTERSin breeze) latest review of the head commit is an approval, no committer's request for changes stands, and the label is still on the PRnon committer builddependabot/branchapache/airflowbranchSelective checks read the PR and its reviews from the API. If they cannot be read, the label is ignored. A push voids the approval, so each new head commit runs on GitHub-hosted runners until a committer approves it. Labels come from the triggering event, so adding a label applies from the next push or reopen. Removing
use self-hosted runnersalso applies to a re-run, because the label is checked against the PR's current labels. "Re-run failed jobs" does not re-run selective checks, so it keeps that run's runner decision. Only users with write access can re-run.amd-runners/arm-runnersare unchanged and stay public. The Docker Hub release workflow consumes them, and it runs with release secrets.On RunsOn,
runner-typeis a label template (runs-on=<run_id>/family=…/cpu={0}/ram={1}). Each job picks its size class withformat(<label>, cpu, ram): 4/16, 8/32 or 32/128. On a public label the same call returns the label unchanged, so one workflow serves both paths. build-info, print-platform, notify-slack and the Windows smoke test keep their hardcoded GitHub-hosted labels.Steps that only make sense on RunsOn check
runner.environment == 'self-hosted'(orRUNNER_ENVIRONMENTin scripts):ci-image-buildloads the stash actions from aworkflow-version/checkout, because it also builds release refs that predate them./mntcleanup (scripts/ci/make_mnt_writeable.sh): on EC2 types with local NVMe disks, RunsOn mounts them at/mnt/ephemeraland keeps Docker,/home/runnerand/tmpthere. The oldrm -rf /mnt/*emptied that mount and killed the runner (Could not find file '/home/runner/_diag/pages/…'). On self-hosted runners, mount points under/mntare now kept and only/mntitself is chowned.move_docker_to_mnt.shalso leaves/var/lib/dockeralone when it is already such a mount. GitHub-hosted runners keep the old commands.runs-on/actionv2.5.0 (ASF-allowlisted, no expiry) adds the job's EC2 cost to the job summary.Security
The CI workflows trigger only on
pull_request,push,scheduleandworkflow_dispatch. No workflow runs onpull_request_target, and this PR adds none.pull_requestGITHUB_TOKEN(declared write permissions are downgraded), no secrets. Unchanged.use self-hosted runnerspull_requestcache/prefix.pull_requestuse self-hosted runnerspull_requestapache/airflowbranchpull_requestpackages: write(image builds),id-token: write(image checks),contents: write(update-constraints, when dependencies are upgraded),SLACK_BOT_TOKEN, andDOCS_AWS_*passed to image checks (only canary steps use them).push/schedule/workflow_dispatchDOCS_AWS_*,id-token: write), update constraints (contents: write) and push the image cache (packages: write) now run on RunsOn.cache/in the bucket. That is why image tarballs are content-addressed and checked against a GitHub-scoped digest. A PR's job can corrupt or delete a blob, which fails the job that needs it (denial of service). It cannot swap the image that a canary job with secrets restores.canarylabel does not route to RunsOn.pull_request, the code that evaluates those inputs belongs to the PR: a fork PR runs its own workflow files and breeze sources. It could therefore name RunsOn labels inruns-on:directly and skip selective checks entirely. The repository setting "Approval for running fork pull request workflows from contributors" only narrows this. Even "Require approval for all external contributors" exempts members of theapacheGitHub organization (ASF committers of any project who linked their GitHub account) and repository collaborators (including the triage collaborators in.asf.yaml). So the real isolation has to come from the AWS / RunsOn side. Merge preconditions:cache/in the cache bucket, with no access to RunsOn's config bucket.S3CacheExpirationInDaysis at least 6, the longest digest retention.Attribution
RunsOn's free license for open-source projects requires a public acknowledgement with a link to runs-on.com. This PR adds a text line to the README sponsors section. It adds no logo: the license reserves RunsOn's logos and grants no usage rights.
Verification
dev/breeze/tests/test_selective_checks.py: 330 passed. New tests cover every routing rule, the label precedence, theUSE_PUBLIC_RUNNERSvariable, and the approval gate on the label (approval of the head commit, committers' standing or withdrawn requests for changes, the label removed from the PR, unreadable PR or reviews).check-ci-workflows-in-sync.ubuntu-22.04labels and the ASF stash.use self-hosted runners): routed to RunsOn. The CI image build lost its runner on an NVMe instance type, which led to the/mntfix above.use self-hosted runners+full tests needed, with the fix): RunsOn path green. 150 checks pass, including 69 test-matrix jobs. The only failure isMergeable(dont-merge). The skipped jobs are canary-only.workflow_dispatchofTests (ARM)or the first ARM canary) and canary-only jobs on RunsOn (docs publish, constraints, image cache push).Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 5.5) following the guidelines
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.