Repository navigation
ci: add triage and retry system for CI failures - #38441
sleepytanya wants to merge 7 commits into
Conversation
|
CLA Signature Action: All authors have signed the CLA. You may need to manually re-run the blocking PR check if it doesn't pass in a few minutes. |
PR template — items to address before "Ready for review"Warnings — informational, address before merging:
See docs/readme/ready-for-review.md for the full Definition of Ready for Review. |
536c58b to
2608dd2
Compare
move Metro error to deterministic
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit e4e4bf9. Configure here.
|
Tested
Two skill gaps(?):
|
HowardBraham
left a comment
There was a problem hiding this comment.
This is a combination of a review by my agent that developed the Extension Triage and Retry System, and my own review.
This whole system is extremely, extremely complicated, exploits gaps in GitHub Actions to force it to do things it wasn't designed to do, and has a dozen race conditions.
You need much more thorough tests on the fork that will test not just the simplest paths, but also the complex paths that will trip something up.
| if (hasPr && !hasLabel) | ||
| return { | ||
| key: 'retryable-no-label', | ||
| label: `⏸️ Retryable — add the retry-ci label to PR #${pr} to retry`, |
There was a problem hiding this comment.
I did this too at first, but this is a misleading instruction. The suggestion here is clearer.
| label: `⏸️ Retryable — add the retry-ci label to PR #${pr} to retry`, | |
| label: `⏸️ Retryable — add the retry-ci label to PR #${pr} before the next run to retry`, |
There was a problem hiding this comment.
Where did the contents of this file come from, and how much have you tested it already?
There was a problem hiding this comment.
The initial patterns came from an analysis of recent Mobile CI failures on Oct 27: 50 failed runs for job names, then 100 for job names and error samples. The script we used was removed before the PR, so the results aren't in the repo. The header's "50" is from the first pass.
Tested on the fork (consensys-test/metamask-mobile-tanya) with the review fixes on fork main (0e8c425):
- Push with no PR: classified retryable, not retried (43817900020).
- PR 3 with retry-ci: attempt 1 retried and attempt 2 passed (43818417034). With the stand-in failing every attempt, attempts 1→2 and 2→3 each retried and the label was removed after each one (43818741706, 43818790509).
- PR 3 with no label at classification time: attempt 3 did not retry, and the log said to add retry-ci (43818837744).
- PR 3 at the limit: attempts 4, 5, and 6 all stopped with "reached the retry-ci limit (4)", and the label stayed (43819358400, 43819518387, 43819550758).
Only run on the November 5 classifier: - Deterministic bundle failure with retry-ci: not retried (41964889541, PR 2).
- release/* push: one automatic retry, label not used (41964469952).
Still not run: a PR opened from outside the fork, and a second rerun of the same run id while the first is still running.
For myself - wording nit: past the limit the message reads "attempt 5 of 4".
There was a problem hiding this comment.
This file never runs. Mobile's jest.config.js has no testMatch, and Jest's default pattern doesn't match .mts. Extension needed a jest.config.js testMatch change for exactly this. As it stands, the PR's only test is dead code in CI.
| // | ||
| // Includes Mobile-specific patterns (simulator, emulator, Appium, Metro, etc.) | ||
| // plus common infrastructure patterns from Extension. | ||
| "transientErrorPatterns": [ |
There was a problem hiding this comment.
Pattern ordering makes most of the Mobile transient patterns unreachable.
- Deterministic patterns are checked first, and every regex is compiled with the case-insensitive
iflag. Build failedmatches Gradle'sBUILD FAILED in 3mon every failed Android build. That makesGradle.*timeout,Gradle.*daemon.*crashandCould not resolve.*dependenciesunreachable.Tests:.*\d+ failedandTest Suites:.*failedappear in every failing Jest log, soExceeded timeout of \d+ ms for a testandJest worker encounteredcan never win. Our Mobile discovery found a Jest timeout to be the main transient candidate.TypeError:also matchesTypeError: fetch failed, which is a network error.- These need fixture tests (log text in, classification out) before anyone trusts them.
| // Jobs that are normally deterministic but can fail due to infrastructure | ||
| // issues (OOM, network errors, etc.). Retried only if error matches a | ||
| // transient pattern from the global transientErrorPatterns list. | ||
| "retryableOnTransientError": { |
There was a problem hiding this comment.
Several real jobs match no pattern and fall back to the transient-only default: Report Fixture Validation, Ship JS bundle size check, Run Performance Tests (PR), Delete CI JS deps artifact, merge-unit-and-component-view-tests, Deploy Platform API Docs. Our Mobile plan said to default unmatched jobs to non-retryable until there's data. Either way, list them explicitly.
| # Only populated for pull_request events; empty for push. | ||
| PR_NUMBER_FROM_EVENT: ${{ github.event.workflow_run.pull_requests[0].number }} | ||
| REPO: ${{ github.repository }} | ||
| SENTRY_DSN_PERFORMANCE: ${{ vars.SENTRY_DSN_PERFORMANCE }} |
There was a problem hiding this comment.
The way this is written, I think the Mobile data is going to collide into the Extension dashboard, and it's going to be harder to tell the two apart. Let's discuss options.
There was a problem hiding this comment.
@HowardBraham
You are right!
Mobile could reuse the existing test project for triage logs. The workflow reads vars.SENTRY_DSN_PERFORMANCE, while the test DSN lives in the secret MM_SENTRY_DSN_TEST. Setting the repo variable to that DSN would match the Extension workflow. Pointing the workflow at the secret would be a Mobile-only change.
| fi | ||
|
|
||
| echo "Rerunning failed jobs..." | ||
| gh run rerun "$MAIN_RUN_ID" --failed --repo "$REPO" |
There was a problem hiding this comment.
The PR path doesn't check that the run attempt is still current. The release path does.
- Mobile has a second actor that reruns the same run ID:
rerun-ci-on-skipped-e2e-labels.ymlcancels the run and then callsgh run rerun $RUN_IDwhen E2E labels change. - Triage classifies attempt N. If that other rerun has already finished, triage runs
rerun --failedagainst an attempt it never classified, and spends the label doing it. - Fix: copy the release path's
status / conclusion / run_attempt / head_shacheck into the PR branch. - This is our plan's rule "bind every operation to exact run ID, event, SHA and attempt."
| function checkRetryLabel(prNum: string): boolean { | ||
| if (!prNum) return false; | ||
| try { | ||
| const labels = ghApi(`${repoApi}/issues/${prNum}/labels`); |
There was a problem hiding this comment.
As is, this is not paginated and you'd see only 30 labels
| const labels = ghApi(`${repoApi}/issues/${prNum}/labels`); | |
| const labels = ghApi(`${repoApi}/issues/${prNum}/labels?per_page=100`); |
Smart E2E Test SelectionSelected E2E tags: No E2E tests selected AI Confidence: 95 E2E reasoningExpand to readThe PR adds CI triage and retry tooling: new .mts scripts under .github/scripts, a new workflow (triage-and-retry-system.yml), a retry-config.jsonc rules file, and a Jest transform entry for .mts files. The Jest change applies a TypeScript-only Babel config to .mts files, and no app source uses that extension. No app code, controller, UI, native module, or E2E test infrastructure is modified. The workflow is a separate workflow_run-triggered file and does not change ci.yml or build.yml. None of the E2E smoke flows are affected, so I selected no tags. |
|





Description
Ports the Triage and Retry System from Extension
(#41177, #44956) to Mobile with Phase 1 adaptations.
Key differences from Extension
Phase 1 (stealth mode): A pull request retries only when
retry-ciis present, and that retry consumes the label.release/*pushes still get one automatic retry (attempt 1 → 2).merge_group omitted: Mobile's queue uses ALLGREEN, and nothing defers an "All jobs pass" commit status the way Extension's
ci-status-gate.ymldoes. Retrying merge-queue runs without that deferral would be unsafe.No E2E quality-gate annotation: Mobile has no Jest reporter that emits deterministic E2E annotations, so Appium stays in
alwaysRetryable. Phase 1's label requirement prevents runaway retries.Deterministic before transient: Log patterns are checked in that order.
Quality Gate failed(SonarCloud) andJS bundle exceeds baselineare Mobile-specific additions.Known issues to discuss
retry-budget.test.mtsis updated, but Mobile's Jest config does not compile.mts. CI never runs the test. Extension's config does. Should we copy that Jest change here or as a follow-up?Changelog
CHANGELOG entry: null
Related issues
Fixes: https://consensyssoftware.atlassian.net/browse/MCRM-151?atlOrigin=eyJpIjoiNTFhZWI0YzFlMmUyNDExNGFkYWVlNzFlNWQyODc3YTYiLCJwIjoiaiJ9
Manual testing steps
Test plan:
Phase 1 behavior: PR without retry-ci label shows "add the retry-ci label to PR #X to retry" 🟢
Deterministic patterns (Quality Gate failed) block retry 🟢
Workflow YAML validates 🟢
.mts scripts run on Node 24 (--help and real classifications) 🟢
JS bundle exceeds baseline blocks retry 🟢
Release push gets one automatic retry 🟢
Push to main is observation only (no rerun) 🟢
Fork runs
Live retries ran on
consensys-test/metamask-mobile-tanya. The fork'smaincarries this PR's triage workflow plus a one-jobcistand-in. The stand-in job is namedAppium Smoke Tests, fails attempt 1 withETIMEDOUT talking to the runner, and passes on attempt 2. Other workflows on the fork are disabled.workflow_runruns reportmainas their branch; the failedcirun id in the log is the one under test.Push to main, no retry
ci41962140436, commite1d795688, attempt 1, failure. No rerun.is-retryable=true,No originating PR for event 'push' → will-retry=false,Retry mode: none (automatic limit 1; retry-ci limit 4).The summary line says
Retryable, but the automatic retry budget ended at attempt 1. The run was not retried, which is correct formain, but that sentence reads as if a budget was spent. Worth a wording change.PR without the label
PR #1,
retry-fixtureat4176eaf8d.ci41962633587, attempt 1, failure. No rerun.is-retryable=true,PR #1: retry-ci label absent, retry-mode=none → will-retry=false,Retryable — add the retry-ci label to PR #1 to retry.PR with the label
Same PR, new commit
00be59476, label added before the push.ci41963293969 failed attempt 1, then finished attempt 2 with success.retry-ci label present, retry-mode=label → will-retry=true,Will retry (retry-ci label present; attempt 1 of 4), thenRemoving retry-ci label from PR #1....Release push, one automatic retry
Branch
release/9.9.9-retry-test, commitfcf9b63df.ci41964455569 failed attempt 1, then finished attempt 2 with success.Release branch release/9.9.9-retry-test: retry-mode=automatic → will-retry=true,Will retry automatically (default retry budget: attempt 1 of 2),head still matches failed run fcf9b63…,Rerun triggered.JS bundle baseline blocks retry
PR #2,
bundle-fixtureat296a56006. The stand-in job on this branch is namedJS bundle sizeand exits 1 withJS bundle exceeds baseline. Theretry-cilabel was on the PR beforecifinished.ci41964877081, attempt 1, failure. No rerun.Deterministic: JS bundle exceeds baseline,is-retryable=false,retry-ci label present, retry-mode=none → will-retry=false,Has retry-ci label but non-retryable failures.retry-cilabel.stableuses the same observation path asmain.merge_groupwas not run; the fork has no merge queue, and this PR excludes that event.Triage and Retry System
Run: 41759667183
Classification: ✅ All failures retryable
Retry: ⏸️ Retryable — add the retry-ci label to PR #38564 to retry
Retry mode: none (automatic limit 1; retry-ci limit 4)
Failed jobs: 2
Triage and Retry System
Run: 41768129279
Classification: ❌ Non-retryable failures detected
Retry: ❌ Non-retryable (PR #37252, no retry-ci label)
Retry mode: none (automatic limit 1; retry-ci limit 4)
Failed jobs: 2
Triage and Retry System
Run: 41757054185
Classification: ❌ Non-retryable failures detected
Retry: ❌ Non-retryable (PR #38815, no retry-ci label)
Retry mode: none (automatic limit 1; retry-ci limit 4)
Failed jobs: 27
@lavamoat/git-safe-dependenciestest:depchecklintTriage and Retry System
Run: 41775319673
Classification: ❌ Non-retryable failures detected
Retry: ❌ Non-retryable (PR #38738, no retry-ci label)
Retry mode: none (automatic limit 1; retry-ci limit 4)
Failed jobs: 3
Screenshots/Recordings
NA. Triage runs as a workflow with no UI change; evidence is the classifier output in Manual testing steps.
Before
After
Pre-merge author checklist
Performance checks (if applicable)
trace()for usage andaddTokenfor an exampleFor performance guidelines and tooling, see the Performance Guide.
Pre-merge reviewer checklist
Note
Medium Risk
New workflow uses elevated
actions:write/pull-requests:writeto rerun CI; mis-tuned patterns could retry real failures or skip needed retries, though label gating and deterministic-first rules limit blast radius on PRs.Overview
Introduces a Triage and Retry System that runs after
cifinishes: it classifies failed jobs, publishes a step-summary report (and optional Check Run + Sentry logs), and may triggergh run rerun --failedwhen policy allows.Classification is driven by
retry-config.jsonc(job name categories, blocker cascade, Mobile-specific transient/deterministic log patterns with deterministic checks first). Theclassify-failures.mtsscript pulls failed jobs, annotations, and log tails viagh, then outputsis-retryable,will-retry, and retry-budget fields.Phase 1 (stealth): PR failures retry only when the
retry-cilabel is present (label is removed on use, up to attempt 4);release/*pushes get one automatic retry;main/stablepushes are observe-only.merge_groupis intentionally excluded. The workflow adds stale-SHA checks before rerunning so superseded commits are not retried.Supporting pure helpers (
retry-budget,retry-label,classify-failure-text) ship with unit tests; Jest gains ababel.config.mts-jest.jstransform so tests can import.mtshelpers.Reviewed by Cursor Bugbot for commit d9692e5. Bugbot is set up for automated code reviews on this repo. Configure here.