Skip to content

Harden the fork CI reconciler and protect the fork CI gate - #74909

Merged
potiuk merged 7 commits into
mainfrom
fork-ci-6-reconciler-followups
Oct 27, 2026
Merged

potiuk merged 7 commits into
mainfrom
fork-ci-6-reconciler-followups

Conversation

@potiuk

@potiuk potiuk commented Oct 17, 2026 •

Copy link
Copy Markdown
Member

Part 6 of the AIP-120 implementation (design: #73378).

Follow-ups to the reconciler in #74822, one commit each, plus protection of the gate itself:

  • New comments (the "push again" nudge and the "fork CI stopped running" notice added in Tell contributors when their fork CI workflow stops running #75814) are posted before the state comment records the new state, so a failed post is retried.
  • A workflow or trigger the base branch does not have counts as project CI in the tampering check only if one of its jobs ran.
  • Pull requests sharing a head commit across different bases stay green_unverified, since one commit status cannot hold a decision per base.
  • Recorded runs made while the pull request targeted a base without the gate are ignored; a run that could belong to another pull request from the same branch is reported to Slack instead of closing.
  • A gated pull request that changes the fork CI gate (gate code, configuration beyond the author's own [pilot] entry, the gate job or its wiring in ci-amd.yml, or the monitor) is closed with a comment explaining why. Each of those files says so. The state comment's marker records gate_checked only when this check ran for the head commit, so pull requests reconciled before it existed are checked once.

closes: #74905
closes: #74906
closes: #74907
closes: #74908


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 5)

Generated-by: Claude Code (Opus 5) following the guidelines

🤖 Generated with Claude Code

@muditagrawal2007 muditagrawal2007 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stacked-on-#74825 design is clean, and the "post the closing comment before the state comment records the new state" change is the right call — a failed post was silently lost before this PR. The gate-protection added layer is also a sensible guardrail for the same class of self-deciding risk that motivated the original gate. Read the full diff against fork-ci-5-docs; below is what stood out.

1. The gate-modified marker doesn't record gate_checked: true

The closure paths for gate_bypassed, closed (inactivity) and the regular state branches all now write Marker(..., gate_checked=gate_checked) — but the new gate_modified branch in release/classify writes Marker(ForkState.GATE_MODIFIED.value, pr.head_sha, now) without that flag:

_set_marker_comment(
    plan,
    pr,
    state_comment(
        Marker(ForkState.GATE_MODIFIED.value, pr.head_sha, now), fork=pr.head_repo, run_url=None
    ),
)

The documentation states the opposite:

The state comment's marker records "gate_checked": true with the head commit only when this check ran for it

The check did run for this head commit — that's why we're in this branch. So the comment and the doc disagree.

The reason this matters in practice is maintainer-reopen-without-restore. Reading the docstring for reopened_since_closing, when marker.state == gate_modified and pr.last_reopened_at > marker.since, the "reopen since closing" path is skipped (correctly), and execution falls through to the gate-check block. With gate_checked: false on the marker, the check re-runs and closes the PR again — which matches the documented intent:

a maintainer's reopen without restoring the gate closes it again on the next run

So either:

  • The code is right and the doc sentence above is overstated (e.g., add an exception for "and the close path retains gate_checked: false so a reopen re-runs the check"), or
  • The code should record gate_checked: true and already_checked should additionally require pr.last_reopened_at is None or pr.last_reopened_at <= marker.since. That keeps the once-per-head-commit guarantee and still re-runs after a maintainer reopen.

The second option is more consistent with the rest of the diff and with the rule "checked once per head commit". Right now the rule is held together by the same exception that makes the doc misleading.

already_checked = (
    marker is not None
    and marker.gate_checked
    and marker.sha == pr.head_sha
    and (pr.last_base_change_at is None or pr.last_base_change_at < marker.since)
    and (pr.last_reopened_at is None or pr.last_reopened_at <= marker.since)
)

Then record gate_checked=gate_checked on the gate-modified marker too, and add a test for the reopen-without-restore case (test_gate_change_reopens_closes_again or similar).

2. Gate-modified plans don't ask for the gate_checked sentinel that other closures carry

Same point as #1, expressed from the consumer side. test_a_change_to_the_gate_closes only asserts '"state": "gate_modified"' in plan.marker_comment, which passes whether or not the gate-checked flag is set. Tightening that to "gate_checked": true would lock in whichever resolution you pick and surface regressions in either direction. The other three tests for this branch (test_unfinished_gate_change_closure_is_retried, test_marker_records_whether_the_gate_was_checked, test_gate_change_is_checked_once_per_head_commit) sidestep the close path.

3. classify is now ~270 lines doing everything

Counting from the start of classify to its return plan, the function decides: which paths skip the gate (use project ci/use fork ci), gate tampering, gate-modified closure, "reopen since closing", fork state observation, green-trust decision (unverified vs verified), the shared-head-across-bases path, draft/labels, status, marker. None of these are individually complicated but reading them as one diff is. A small refactor — extracting _plan_closure(state, comment_factory, *, change=...) for the three close paths and _plan_record_state(state, since) for the record-only paths, with classify as a top-level orchestrator — would make the diff much easier to review and the existing tests can stay because they assert on Plan shape.

4. observe_fork_state doesn't know about gate_modified

def observe_fork_state(fork: ForkObservation, marker: Marker | None) -> tuple[ForkState | None, str | None]:
    ...
    if fork.workflow_state is None:
        if marker is not None and marker.state == ForkState.AWAITING_PUSH.value:
            return ForkState.AWAITING_PUSH, None
        return None, "no fork run and the fork workflow state was not read"

For a re-opened gate_modified PR with no fork run, this returns (None, "no fork run...") rather than (GATE_MODIFIED, None). That's tolerable because the release/gate-check block runs before observe_fork_state and either closes again or falls through (depending on #1), but if you do end up extending the gate-check block to be re-entrant from observe_fork_state, the function needs an explicit gate_modified case, otherwise the post-#1 architecture risks an unknown line per run.

5. _other_pull_requests is re-issued per run, not cached per branch

BypassFinder._other_pull_requests calls rest_get(pulls?head=...) with pagination. The result depends only on (head_repo, head_ref, window_start) and find() is invoked once per branch by run_reconcile. Hoisting it into BypassFinder.__init__ (computed once with the same 6h window) would save one paginated call per PR per run. Trivial cost per call, but the call pattern repeats across every reconciles and shows up in the GitHub calls: {...} log line.

6. _set_status URL is hardcoded in the new branch

_set_status(
    plan, pr, ForkState.GATE_MODIFIED.value, f"https://github.com/{repo}/pull/{pr.number}/files"
)

vs. the helper used elsewhere — fork.run_url or fork_actions_url(pr.head_repo) and f"https://github.com/{repo}/pull/{pr.number}/checks". The hardcoded value is fine semantically (a gate_modified PR has no fork run to link to; linking the changed files is the right thing), but the inconsistency is worth a helper for consistency — something like gate_changed_files_url(pr.number) if you don't want to inline it.

7. _list_jobs raises and goes through the except GitHubError path

try:
    jobs = self._list_jobs(run)
except GitHubError:
    result = True
else:
    result = ...

That result = True on the right-hand side of the OR-test for "did work" reads the wrong way at first glance — "no jobs listed → counts as project CI" is exactly the intended semantic, but the previous shape made the same point by setting result = True before checking anything and the diff moved it under a guard. Worth a two-line comment at the assignment:

except GitHubError:
    # A listing we can't make fails open: better to flag a run that turned out
    # to be project CI than to miss one.
    result = True

Two of the parametrized tests cover the False branch; there isn't a test that exercises rest_get raising on the jobs call. Adding one would lock in the "fail open on listing error" semantics.

8. base_at boundary semantics

for changed_at, previous, _ in reversed(pr.base_changes):
    if when < changed_at:
        base = previous

For a run whose created_at exactly equals a BaseRefChangedEvent.createdAt (when == changed_at), the new base applies. That's right for a run triggered by the retarget itself (pull_request with edited: change_base), which sees the new base. For a run whose created_at is one millisecond less than the retarget (when < changed_at), the loop sets base = previous (old base). That's also defensible — the run was triggered against the old base. The asymmetry around the boundary is intentional but worth a one-line comment; "runs at or after a retarget are considered to be against the new base".

9. Schema test for MONITORED_GATE_PROTECTED_PATHS?

GATE_PROTECTED_PATHS is a single frozenset at module scope; if a future PR renames a file or adds another gate piece (e.g., fork_ci_maintainer.py) the maintainer who lands it needs to remember to add it both here and to the four "this file is part of the gate" comments. A small assert that mirrors the comments — pulling the strings out of those four header comments and checking every protected path has a back-reference — would catch a forgotten update. Optional, but it's the kind of doc/code drift the existing comments are trying to prevent.

10. Smaller things

  • Marker(ForkState.GATE_MODIFIED.value, pr.head_sha, now) is built both inline (state_comment call) and via a future helper — at this PR's size, the duplication is small but a const for the per-state string would help.
  • from dataclasses import replace — is_own_pilot_change imports it but doesn't use replace directly... wait, it does: return replace(head, pilot=base.pilot) == base and .... Fine.
  • PILOT_BASE = "enabled = true\n[pilot]\nother = 2026-10-01T00:00:00Z\n" lives at module scope in the test file. Minor — fine for tests, but if anyone wants a second file to reference "what does an unchanged base look like" they'll copy-paste.
  • The "Gate protection" section in 08_fork_ci_for_external_contributors.md is excellent and the table-row addition plus the **New comments go first.** rule are real, easily-understood improvements.

Overall the diff lines up with its stated design, the test density matches the risk of each new branch (≥6 cases for "checked once per head commit" alone), and the gate tampering and shared-head paths are clearly motivated by #74905 / #74906 / #74907. The one thing I'd want resolved before merge is the gate-modified marker flag conflict in #1/#2 — whichever way you settle it, it's worth a one-line comment where the marker is built explaining why gate_checked is on every other closure's marker and the exception.

@potiuk
potiuk force-pushed the fork-ci-6-reconciler-followups branch from 5ffe91d to 8196f7a Compare October 17, 2026 19:25
@potiuk
potiuk force-pushed the fork-ci-5-docs branch 5 times, most recently from aae4714 to b5da0ea Compare October 24, 2026 17:04
@potiuk
potiuk force-pushed the fork-ci-6-reconciler-followups branch from 8196f7a to 5e243b8 Compare October 24, 2026 17:04
@potiuk
potiuk force-pushed the fork-ci-6-reconciler-followups branch from 5e243b8 to 081f95a Compare October 24, 2026 19:37
Base automatically changed from fork-ci-5-docs to main October 24, 2026 20:35
The state comment is what the next reconciler run compares with. Recording
awaiting_push there before the nudge was posted meant a failed post was never
retried, and the contributor got no prompt before the inactivity closure.

Generated-by: Claude Opus 5
…ering check

A workflow or trigger the base branch does not have always counted as
project CI, so a pull request adding a fork-only workflow that its jobs
guard off in apache/airflow was closed for gate tampering although nothing
ran.

Generated-by: Claude Opus 5
…aintainer

The green-trust decision is stored as a commit status, which every pull
request with that head commit shares. Pull requests with different bases
have different changed files, so a decision made for one could mark the
other ready without checking it.

Generated-by: Claude Opus 5
Runs from forks carry no pull request, so they were matched by fork branch
alone. Legitimate project CI against a base branch without the gate, made
before a retarget, or a run of another pull request from the same branch,
could close a gated pull request for gate tampering.

Generated-by: Claude Opus 5
A pull request whose CI runs in the contributor's fork could change the gate
that sent it there: its code, its configuration, the gate job or the
monitor. Such a change decides where the pull request's own CI runs, so the
monitor now closes it and explains why. Contributors can still add, change
or remove their own pilot entry, which is how they join and leave the pilot.

Generated-by: Claude Opus 5
A fork-only workflow run cancelled before a runner took its jobs reports them
completed and cancelled, so it counted as project CI work. It now uses the
same started-job test as ci-amd.yml runs.

Generated-by: Claude Opus 5
The tampering check now accepts the name a cancelled ci-amd.yml run gives
its CI passed job, so the docs list it with the other names of that job.

Generated-by: Claude Opus 5
@potiuk
potiuk force-pushed the fork-ci-6-reconciler-followups branch from 081f95a to 7e3d6f0 Compare October 27, 2026 20:14
@potiuk
potiuk enabled auto-merge (squash) October 27, 2026 20:23
@potiuk
potiuk disabled auto-merge October 27, 2026 20:50
@potiuk
potiuk merged commit 8058c88 into main Oct 27, 2026
137 of 154 checks passed
@potiuk
potiuk deleted the fork-ci-6-reconciler-followups branch October 27, 2026 20:50
@potiuk potiuk added the backport-to-v3-4-test Backport to v3-4-test label Oct 27, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Backport failed to create: v3-4-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status Branch Result
❌ v3-4-test Commit Link

You can attempt to backport this manually by running:

cherry_picker 8058c88 v3-4-test

This should apply the commit to the v3-4-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

potiuk added a commit that referenced this pull request Oct 27, 2026
* Bootstrap constraints generation for a Python without a baseline file (#74596)

The generator always downloads the previous constraints file of the same Python from the constraints
branch to diff against it, and failed with a 404 for the first generation of a new Python. When the
branch carries the same constraints mode for other Pythons, a missing file means a first publication and
the generator diffs against an empty baseline instead. A 404 on a branch without any file of that mode,
and every other HTTP error, still fails.

(cherry picked from commit d267fbd)

* Check out constraints from apache/airflow when diffing them in CI (#75675)

The Update constraints job checked out the constraints branch from the
repository the run belongs to. Forks do not carry the constraints-*
branches, so the job failed on every canary push in a fork. Diff against
apache/airflow instead; the job only diffs now, so it needs neither
credentials nor history.

Generated-by: Claude Opus 5
(cherry picked from commit 3348602)

* Always approve workflow runs of PRs from trusted stakeholder repositories (#75677)

Amazon, Google and Astronomer maintain large parts of Airflow and open many
PRs from their own repositories. Their PRs often touch large providers or CI
and wait for a maintainer to approve every push, although only people those
stakeholders gave write access can push the commits of these PRs.

Generated-by: Claude Opus 5
(cherry picked from commit ffd9398)

* Retry GitHub calls that fail to connect in the fork CI monitor (#75701)

A monitor run failed to report job statuses for a PR because a POST to the statuses API ended in a TLS handshake timeout. Network failures are as transient as the server errors already retried, and one that happens before the request reaches GitHub cannot have taken effect, so it is safe to repeat even for calls such as creating a comment.

Generated-by: Claude Opus 5
(cherry picked from commit 2ae00f1)

* Stop breeze k8s commands from silently rewriting uv.lock (#75763)

A v3-4-test backport bumped a workspace member's version without updating
uv.lock, and the `update-uv-lock` static check still passed. The chart's
prek hooks run `breeze k8s setup-env`, which ran a plain `uv sync` in
kubernetes-tests. That project is a uv workspace member, so the sync
re-resolved and rewrote the root uv.lock before the root hook checked it.
Nested prek configs do not inherit the root UV_FROZEN=1, and prek does not
report files a nested-project hook modifies outside its own project, so
the rewrite went unnoticed and the stale lock was merged. It then broke
every "Generate constraints" job on the branch.

The `uv run pytest` calls of `breeze k8s tests` and the overlay tests had
the same effect on contributors' checkouts.

CI now also fails when the static checks leave tracked files modified, so
any other hook writing outside its project is caught.

(cherry picked from commit 7c7157b)

* Keep breeze verify working when dry-run is on (#75713)

With dry-run on, run_command skips every command and returns empty
output. breeze verify only reads git state to list the checks a change
needs, so skipping those queries made it crash with an IndexError
instead of answering.

(cherry picked from commit a5ff71d)

* Tell contributors when their fork CI workflow stops running (#75814)

The fork CI monitor read the fork's workflow state only until it first saw
the workflow enabled. When the workflow was disabled afterwards, by the
contributor or by GitHub after 60 days of fork inactivity, the pull request
kept waiting for a push that could never run CI, while its comment still
said the workflow was enabled. Contributors now get a new comment telling
them why it stopped and how to turn it back on, starting with syncing their
fork's main from apache/airflow.

breeze ci audit also stops suggesting committers disable the workflow: a
committer trying fork CI needs it, and following the hint is one way a pull
request ends up stuck like this.

Generated-by: Claude Opus 5
(cherry picked from commit 891e83d)

* Harden the fork CI reconciler and protect the fork CI gate (#74909)

* Retry the fork CI push-again nudge when posting it fails

The state comment is what the next reconciler run compares with. Recording
awaiting_push there before the nudge was posted meant a failed post was never
retried, and the contributor got no prompt before the inactivity closure.

Generated-by: Claude Opus 5

* Ignore fork-only workflows that skipped every job in the fork CI tampering check

A workflow or trigger the base branch does not have always counted as
project CI, so a pull request adding a fork-only workflow that its jobs
guard off in apache/airflow was closed for gate tampering although nothing
ran.

Generated-by: Claude Opus 5

* Keep fork CI pull requests sharing a head commit across bases for a maintainer

The green-trust decision is stored as a commit status, which every pull
request with that head commit shares. Pull requests with different bases
have different changed files, so a decision made for one could mark the
other ready without checking it.

Generated-by: Claude Opus 5

* Attribute fork CI runs to the right pull request in the tampering check

Runs from forks carry no pull request, so they were matched by fork branch
alone. Legitimate project CI against a base branch without the gate, made
before a retarget, or a run of another pull request from the same branch,
could close a gated pull request for gate tampering.

Generated-by: Claude Opus 5

* Close fork CI pull requests that change the fork CI gate

A pull request whose CI runs in the contributor's fork could change the gate
that sent it there: its code, its configuration, the gate job or the
monitor. Such a change decides where the pull request's own CI runs, so the
monitor now closes it and explains why. Contributors can still add, change
or remove their own pilot entry, which is how they join and leave the pilot.

Generated-by: Claude Opus 5

* Count only started jobs of fork-only workflows as project CI work

A fork-only workflow run cancelled before a runner took its jobs reports them
completed and cancelled, so it counted as project CI work. It now uses the
same started-job test as ci-amd.yml runs.

Generated-by: Claude Opus 5

* Name the cancelled-run CI passed job in the fork CI tampering docs

The tampering check now accepts the name a cancelled ci-amd.yml run gives
its CI passed job, so the docs list it with the other names of that job.

Generated-by: Claude Opus 5
(cherry picked from commit 8058c88)

---------

Co-authored-by: Dev-iL <6509619+Dev-iL@users.noreply.github.com>
Co-authored-by: Shahar Epstein <60007259+shahar1@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

3 participants