Repository navigation
Harden the fork CI reconciler and protect the fork CI gate - #74909
Conversation
muditagrawal2007
left a comment
There was a problem hiding this comment.
Stacked-on-#74825 design is clean, and the "post the closing comment before the state comment records the new state" change is the right call — a failed post was silently lost before this PR. The gate-protection added layer is also a sensible guardrail for the same class of self-deciding risk that motivated the original gate. Read the full diff against fork-ci-5-docs; below is what stood out.
1. The gate-modified marker doesn't record gate_checked: true
The closure paths for gate_bypassed, closed (inactivity) and the regular state branches all now write Marker(..., gate_checked=gate_checked) — but the new gate_modified branch in release/classify writes Marker(ForkState.GATE_MODIFIED.value, pr.head_sha, now) without that flag:
_set_marker_comment(
plan,
pr,
state_comment(
Marker(ForkState.GATE_MODIFIED.value, pr.head_sha, now), fork=pr.head_repo, run_url=None
),
)The documentation states the opposite:
The state comment's marker records
"gate_checked": truewith the head commit only when this check ran for it
The check did run for this head commit — that's why we're in this branch. So the comment and the doc disagree.
The reason this matters in practice is maintainer-reopen-without-restore. Reading the docstring for reopened_since_closing, when marker.state == gate_modified and pr.last_reopened_at > marker.since, the "reopen since closing" path is skipped (correctly), and execution falls through to the gate-check block. With gate_checked: false on the marker, the check re-runs and closes the PR again — which matches the documented intent:
a maintainer's reopen without restoring the gate closes it again on the next run
So either:
- The code is right and the doc sentence above is overstated (e.g., add an exception for "and the close path retains
gate_checked: falseso a reopen re-runs the check"), or - The code should record
gate_checked: trueandalready_checkedshould additionally requirepr.last_reopened_at is None or pr.last_reopened_at <= marker.since. That keeps the once-per-head-commit guarantee and still re-runs after a maintainer reopen.
The second option is more consistent with the rest of the diff and with the rule "checked once per head commit". Right now the rule is held together by the same exception that makes the doc misleading.
already_checked = (
marker is not None
and marker.gate_checked
and marker.sha == pr.head_sha
and (pr.last_base_change_at is None or pr.last_base_change_at < marker.since)
and (pr.last_reopened_at is None or pr.last_reopened_at <= marker.since)
)Then record gate_checked=gate_checked on the gate-modified marker too, and add a test for the reopen-without-restore case (test_gate_change_reopens_closes_again or similar).
2. Gate-modified plans don't ask for the gate_checked sentinel that other closures carry
Same point as #1, expressed from the consumer side. test_a_change_to_the_gate_closes only asserts '"state": "gate_modified"' in plan.marker_comment, which passes whether or not the gate-checked flag is set. Tightening that to "gate_checked": true would lock in whichever resolution you pick and surface regressions in either direction. The other three tests for this branch (test_unfinished_gate_change_closure_is_retried, test_marker_records_whether_the_gate_was_checked, test_gate_change_is_checked_once_per_head_commit) sidestep the close path.
3. classify is now ~270 lines doing everything
Counting from the start of classify to its return plan, the function decides: which paths skip the gate (use project ci/use fork ci), gate tampering, gate-modified closure, "reopen since closing", fork state observation, green-trust decision (unverified vs verified), the shared-head-across-bases path, draft/labels, status, marker. None of these are individually complicated but reading them as one diff is. A small refactor — extracting _plan_closure(state, comment_factory, *, change=...) for the three close paths and _plan_record_state(state, since) for the record-only paths, with classify as a top-level orchestrator — would make the diff much easier to review and the existing tests can stay because they assert on Plan shape.
4. observe_fork_state doesn't know about gate_modified
def observe_fork_state(fork: ForkObservation, marker: Marker | None) -> tuple[ForkState | None, str | None]:
...
if fork.workflow_state is None:
if marker is not None and marker.state == ForkState.AWAITING_PUSH.value:
return ForkState.AWAITING_PUSH, None
return None, "no fork run and the fork workflow state was not read"For a re-opened gate_modified PR with no fork run, this returns (None, "no fork run...") rather than (GATE_MODIFIED, None). That's tolerable because the release/gate-check block runs before observe_fork_state and either closes again or falls through (depending on #1), but if you do end up extending the gate-check block to be re-entrant from observe_fork_state, the function needs an explicit gate_modified case, otherwise the post-#1 architecture risks an unknown line per run.
5. _other_pull_requests is re-issued per run, not cached per branch
BypassFinder._other_pull_requests calls rest_get(pulls?head=...) with pagination. The result depends only on (head_repo, head_ref, window_start) and find() is invoked once per branch by run_reconcile. Hoisting it into BypassFinder.__init__ (computed once with the same 6h window) would save one paginated call per PR per run. Trivial cost per call, but the call pattern repeats across every reconciles and shows up in the GitHub calls: {...} log line.
6. _set_status URL is hardcoded in the new branch
_set_status(
plan, pr, ForkState.GATE_MODIFIED.value, f"https://github.com/{repo}/pull/{pr.number}/files"
)vs. the helper used elsewhere — fork.run_url or fork_actions_url(pr.head_repo) and f"https://github.com/{repo}/pull/{pr.number}/checks". The hardcoded value is fine semantically (a gate_modified PR has no fork run to link to; linking the changed files is the right thing), but the inconsistency is worth a helper for consistency — something like gate_changed_files_url(pr.number) if you don't want to inline it.
7. _list_jobs raises and goes through the except GitHubError path
try:
jobs = self._list_jobs(run)
except GitHubError:
result = True
else:
result = ...That result = True on the right-hand side of the OR-test for "did work" reads the wrong way at first glance — "no jobs listed → counts as project CI" is exactly the intended semantic, but the previous shape made the same point by setting result = True before checking anything and the diff moved it under a guard. Worth a two-line comment at the assignment:
except GitHubError:
# A listing we can't make fails open: better to flag a run that turned out
# to be project CI than to miss one.
result = TrueTwo of the parametrized tests cover the False branch; there isn't a test that exercises rest_get raising on the jobs call. Adding one would lock in the "fail open on listing error" semantics.
8. base_at boundary semantics
for changed_at, previous, _ in reversed(pr.base_changes):
if when < changed_at:
base = previousFor a run whose created_at exactly equals a BaseRefChangedEvent.createdAt (when == changed_at), the new base applies. That's right for a run triggered by the retarget itself (pull_request with edited: change_base), which sees the new base. For a run whose created_at is one millisecond less than the retarget (when < changed_at), the loop sets base = previous (old base). That's also defensible — the run was triggered against the old base. The asymmetry around the boundary is intentional but worth a one-line comment; "runs at or after a retarget are considered to be against the new base".
9. Schema test for MONITORED_GATE_PROTECTED_PATHS?
GATE_PROTECTED_PATHS is a single frozenset at module scope; if a future PR renames a file or adds another gate piece (e.g., fork_ci_maintainer.py) the maintainer who lands it needs to remember to add it both here and to the four "this file is part of the gate" comments. A small assert that mirrors the comments — pulling the strings out of those four header comments and checking every protected path has a back-reference — would catch a forgotten update. Optional, but it's the kind of doc/code drift the existing comments are trying to prevent.
10. Smaller things
Marker(ForkState.GATE_MODIFIED.value, pr.head_sha, now)is built both inline (state_comment call) and via a future helper — at this PR's size, the duplication is small but a const for the per-state string would help.from dataclasses import replace—is_own_pilot_changeimports it but doesn't usereplacedirectly... wait, it does:return replace(head, pilot=base.pilot) == base and .... Fine.PILOT_BASE = "enabled = true\n[pilot]\nother = 2026-10-01T00:00:00Z\n"lives at module scope in the test file. Minor — fine for tests, but if anyone wants a second file to reference "what does an unchanged base look like" they'll copy-paste.- The "Gate protection" section in
08_fork_ci_for_external_contributors.mdis excellent and the table-row addition plus the**New comments go first.**rule are real, easily-understood improvements.
Overall the diff lines up with its stated design, the test density matches the risk of each new branch (≥6 cases for "checked once per head commit" alone), and the gate tampering and shared-head paths are clearly motivated by #74905 / #74906 / #74907. The one thing I'd want resolved before merge is the gate-modified marker flag conflict in #1/#2 — whichever way you settle it, it's worth a one-line comment where the marker is built explaining why gate_checked is on every other closure's marker and the exception.
64545b5 to
fc720e5
Compare
5ffe91d to
8196f7a
Compare
aae4714 to
b5da0ea
Compare
8196f7a to
5e243b8
Compare
b5da0ea to
7e06c66
Compare
5e243b8 to
081f95a
Compare
The state comment is what the next reconciler run compares with. Recording awaiting_push there before the nudge was posted meant a failed post was never retried, and the contributor got no prompt before the inactivity closure. Generated-by: Claude Opus 5
…ering check A workflow or trigger the base branch does not have always counted as project CI, so a pull request adding a fork-only workflow that its jobs guard off in apache/airflow was closed for gate tampering although nothing ran. Generated-by: Claude Opus 5
…aintainer The green-trust decision is stored as a commit status, which every pull request with that head commit shares. Pull requests with different bases have different changed files, so a decision made for one could mark the other ready without checking it. Generated-by: Claude Opus 5
Runs from forks carry no pull request, so they were matched by fork branch alone. Legitimate project CI against a base branch without the gate, made before a retarget, or a run of another pull request from the same branch, could close a gated pull request for gate tampering. Generated-by: Claude Opus 5
A pull request whose CI runs in the contributor's fork could change the gate that sent it there: its code, its configuration, the gate job or the monitor. Such a change decides where the pull request's own CI runs, so the monitor now closes it and explains why. Contributors can still add, change or remove their own pilot entry, which is how they join and leave the pilot. Generated-by: Claude Opus 5
A fork-only workflow run cancelled before a runner took its jobs reports them completed and cancelled, so it counted as project CI work. It now uses the same started-job test as ci-amd.yml runs. Generated-by: Claude Opus 5
The tampering check now accepts the name a cancelled ci-amd.yml run gives its CI passed job, so the docs list it with the other names of that job. Generated-by: Claude Opus 5
081f95a to
7e3d6f0
Compare
Backport failed to create: v3-4-test. View the failure log Run detailsNote: As of Merging PRs targeted for Airflow 3.X In matter of doubt please ask in #release-management Slack channel.
You can attempt to backport this manually by running: cherry_picker 8058c88 v3-4-testThis should apply the commit to the v3-4-test branch and leave the commit in conflict state marking After you have resolved the conflicts, you can continue the backport process by running: cherry_picker --continueIf you don't have cherry-picker installed, see the installation guide. |
* Bootstrap constraints generation for a Python without a baseline file (#74596) The generator always downloads the previous constraints file of the same Python from the constraints branch to diff against it, and failed with a 404 for the first generation of a new Python. When the branch carries the same constraints mode for other Pythons, a missing file means a first publication and the generator diffs against an empty baseline instead. A 404 on a branch without any file of that mode, and every other HTTP error, still fails. (cherry picked from commit d267fbd) * Check out constraints from apache/airflow when diffing them in CI (#75675) The Update constraints job checked out the constraints branch from the repository the run belongs to. Forks do not carry the constraints-* branches, so the job failed on every canary push in a fork. Diff against apache/airflow instead; the job only diffs now, so it needs neither credentials nor history. Generated-by: Claude Opus 5 (cherry picked from commit 3348602) * Always approve workflow runs of PRs from trusted stakeholder repositories (#75677) Amazon, Google and Astronomer maintain large parts of Airflow and open many PRs from their own repositories. Their PRs often touch large providers or CI and wait for a maintainer to approve every push, although only people those stakeholders gave write access can push the commits of these PRs. Generated-by: Claude Opus 5 (cherry picked from commit ffd9398) * Retry GitHub calls that fail to connect in the fork CI monitor (#75701) A monitor run failed to report job statuses for a PR because a POST to the statuses API ended in a TLS handshake timeout. Network failures are as transient as the server errors already retried, and one that happens before the request reaches GitHub cannot have taken effect, so it is safe to repeat even for calls such as creating a comment. Generated-by: Claude Opus 5 (cherry picked from commit 2ae00f1) * Stop breeze k8s commands from silently rewriting uv.lock (#75763) A v3-4-test backport bumped a workspace member's version without updating uv.lock, and the `update-uv-lock` static check still passed. The chart's prek hooks run `breeze k8s setup-env`, which ran a plain `uv sync` in kubernetes-tests. That project is a uv workspace member, so the sync re-resolved and rewrote the root uv.lock before the root hook checked it. Nested prek configs do not inherit the root UV_FROZEN=1, and prek does not report files a nested-project hook modifies outside its own project, so the rewrite went unnoticed and the stale lock was merged. It then broke every "Generate constraints" job on the branch. The `uv run pytest` calls of `breeze k8s tests` and the overlay tests had the same effect on contributors' checkouts. CI now also fails when the static checks leave tracked files modified, so any other hook writing outside its project is caught. (cherry picked from commit 7c7157b) * Keep breeze verify working when dry-run is on (#75713) With dry-run on, run_command skips every command and returns empty output. breeze verify only reads git state to list the checks a change needs, so skipping those queries made it crash with an IndexError instead of answering. (cherry picked from commit a5ff71d) * Tell contributors when their fork CI workflow stops running (#75814) The fork CI monitor read the fork's workflow state only until it first saw the workflow enabled. When the workflow was disabled afterwards, by the contributor or by GitHub after 60 days of fork inactivity, the pull request kept waiting for a push that could never run CI, while its comment still said the workflow was enabled. Contributors now get a new comment telling them why it stopped and how to turn it back on, starting with syncing their fork's main from apache/airflow. breeze ci audit also stops suggesting committers disable the workflow: a committer trying fork CI needs it, and following the hint is one way a pull request ends up stuck like this. Generated-by: Claude Opus 5 (cherry picked from commit 891e83d) * Harden the fork CI reconciler and protect the fork CI gate (#74909) * Retry the fork CI push-again nudge when posting it fails The state comment is what the next reconciler run compares with. Recording awaiting_push there before the nudge was posted meant a failed post was never retried, and the contributor got no prompt before the inactivity closure. Generated-by: Claude Opus 5 * Ignore fork-only workflows that skipped every job in the fork CI tampering check A workflow or trigger the base branch does not have always counted as project CI, so a pull request adding a fork-only workflow that its jobs guard off in apache/airflow was closed for gate tampering although nothing ran. Generated-by: Claude Opus 5 * Keep fork CI pull requests sharing a head commit across bases for a maintainer The green-trust decision is stored as a commit status, which every pull request with that head commit shares. Pull requests with different bases have different changed files, so a decision made for one could mark the other ready without checking it. Generated-by: Claude Opus 5 * Attribute fork CI runs to the right pull request in the tampering check Runs from forks carry no pull request, so they were matched by fork branch alone. Legitimate project CI against a base branch without the gate, made before a retarget, or a run of another pull request from the same branch, could close a gated pull request for gate tampering. Generated-by: Claude Opus 5 * Close fork CI pull requests that change the fork CI gate A pull request whose CI runs in the contributor's fork could change the gate that sent it there: its code, its configuration, the gate job or the monitor. Such a change decides where the pull request's own CI runs, so the monitor now closes it and explains why. Contributors can still add, change or remove their own pilot entry, which is how they join and leave the pilot. Generated-by: Claude Opus 5 * Count only started jobs of fork-only workflows as project CI work A fork-only workflow run cancelled before a runner took its jobs reports them completed and cancelled, so it counted as project CI work. It now uses the same started-job test as ci-amd.yml runs. Generated-by: Claude Opus 5 * Name the cancelled-run CI passed job in the fork CI tampering docs The tampering check now accepts the name a cancelled ci-amd.yml run gives its CI passed job, so the docs list it with the other names of that job. Generated-by: Claude Opus 5 (cherry picked from commit 8058c88) --------- Co-authored-by: Dev-iL <6509619+Dev-iL@users.noreply.github.com> Co-authored-by: Shahar Epstein <60007259+shahar1@users.noreply.github.com>
Part 6 of the AIP-120 implementation (design: #73378).
Follow-ups to the reconciler in #74822, one commit each, plus protection of the gate itself:
green_unverified, since one commit status cannot hold a decision per base.[pilot]entry, thegatejob or its wiring inci-amd.yml, or the monitor) is closed with a comment explaining why. Each of those files says so. The state comment's marker recordsgate_checkedonly when this check ran for the head commit, so pull requests reconciled before it existed are checked once.closes: #74905
closes: #74906
closes: #74907
closes: #74908
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 5) following the guidelines
🤖 Generated with Claude Code