Skip to content

Gate retries: rerun after main changes and treat dependency-download failures as retryable before escalating to ready-for-human #397

Open

Description

@mikeroySoft

What happened

On 2026-10-25, 4 of the 7 ready-for-human items in mikeroySoft/rocm-cli had parked after exhausting retry caps on failures unrelated to the PR's code:

The failures that used up the caps:

  1. A flaky ComfyUI test that was later fixed on main (mikeroySoft/rocm-cli@f2abe50). The parked PRs never re-ran against the fix.
  2. A crates.io HTTP/2 download error in the signatures job (cargo registry fetch, not code).
  3. Stale MANIFEST.md regeneration, and a merge commit missing Signed-off-by. Both are mechanical fixups.
  4. Reviewer runs that crashed (review_exit:1) were counted as unresolved review feedback.

After human triage, none of the four needed code changes: mikeroySoft/rocm-cli#74 and mikeroySoft/rocm-cli#25 merged as-is, mikeroySoft/rocm-cli#66 needed only a merge from main and a rerun, and mikeroySoft/rocm-cli#80 needs only a merge from main plus cargo xtask manifest.

Proposals

  • (a) Re-queue after a fix lands. When main advances past a fix for a known-failing test, re-queue parked items whose only failure was that test.
  • (b) Infra failures don't consume the cap. Classify network and download failures (cargo registry, HTTP/2 framing errors, and similar) as infra, and retry them without consuming the retry cap.
  • (c) Crashed reviews aren't feedback. Don't count a crashed reviewer run (review_exit non-zero with no findings) as unresolved feedback. Retry the review instead.
  • (d) Mechanical fixups before escalating. Before escalating to ready-for-human, run mechanical fixups automatically: cargo xtask manifest (or the repo's equivalent) and adding a sign-off to agent-authored merge commits.

Acceptance

  • An item reaches ready-for-human only when the remaining failure is attributable to the PR's code or needs a real decision.
  • The escalation comment says which failure class consumed each retry.

Related: #383 (control-band alarms on gate/flaky failures and ready-for-human rate), #395 (quarantine PRs for cross-PR flaky tests).

Activity

  1. mikeroysoft-agent commented on Oct 31, 2026

    @mikeroysoft-agent
    Collaborator

    Triage: This is a process/behavior change to the gate's retry and escalation logic containing four loosely-coupled proposals (re-queue after a fix lands, classify infra failures as non-cap-consuming, treat crashed reviews as retryable, and run mechanical fixups before escalation). Each requires design judgment about what counts as infra vs. code failure, how to detect a fix landing on main, and how to classify crashed reviewer runs, rather than a single fully-specified actionable item. It also touches signing policy (adding a Signed-off-by to agent-authored merge commits) and has blast radius beyond this repository, since it changes escalation behavior that affects downstream repos like rocm-cli. This is exactly the kind of design/policy decision reserved for a human.

  2. added and removed
    needs-triageMaintainer needs to evaluate this issue
    on Oct 31, 2026
  3. mikeroySoft commented on Oct 31, 2026

    @mikeroySoft
    OwnerAuthor

    Owner decision (Michael, 2026-10-31): all four proposals are approved as engine work. I split them into child tickets, and this issue stays open as the parent tracker.

    The design calls:

    1. Infra and network failures are matched against a configured list of error patterns. Matching failures don't use up the retry cap.
    2. "A fix landed" means the failing test's file changed on main. Parked items are requeued at that point.
    3. A crashed review is a nonzero exit with no verdict parsed. It's retried without charging a round, and it doesn't count as feedback.
    4. Mechanical fixups are commands set per repo in config, such as cargo xtask manifest, and they run before escalating.
    5. Signed-off-by is added only to the agent's own merge commits.

    Each child ticket has its own acceptance criteria and regression tests. This issue's acceptance still applies across all four: an item reaches ready-for-human only when the remaining failure comes from the PR's code or needs a real decision, and the escalation comment names the failure class behind each retry.

  4. added
    factory-heldNot released to Factory intake; verify dependencies and operator checkpoints before triage
    and removed on Oct 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    factory-heldNot released to Factory intake; verify dependencies and operator checkpoints before triage

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions