Skip to content

ci: add triage and retry system for CI failures - #38441

Draft
sleepytanya wants to merge 7 commits into
mainfrom
feat/retry-system-mobile
Draft

sleepytanya wants to merge 7 commits into
mainfrom
feat/retry-system-mobile

Conversation

@sleepytanya

@sleepytanya sleepytanya commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Description

Ports the Triage and Retry System from Extension
(#41177, #44956) to Mobile with Phase 1 adaptations.

Key differences from Extension

  1. Phase 1 (stealth mode): A pull request retries only when retry-ci is present, and that retry consumes the label. release/* pushes still get one automatic retry (attempt 1 → 2).

  2. merge_group omitted: Mobile's queue uses ALLGREEN, and nothing defers an "All jobs pass" commit status the way Extension's ci-status-gate.yml does. Retrying merge-queue runs without that deferral would be unsafe.

  3. No E2E quality-gate annotation: Mobile has no Jest reporter that emits deterministic E2E annotations, so Appium stays in alwaysRetryable. Phase 1's label requirement prevents runaway retries.

  4. Deterministic before transient: Log patterns are checked in that order. Quality Gate failed (SonarCloud) and JS bundle exceeds baseline are Mobile-specific additions.

Known issues to discuss

  • Jest config: retry-budget.test.mts is updated, but Mobile's Jest config does not compile .mts. CI never runs the test. Extension's config does. Should we copy that Jest change here or as a follow-up?

Changelog

CHANGELOG entry: null

Related issues

Fixes: https://consensyssoftware.atlassian.net/browse/MCRM-151?atlOrigin=eyJpIjoiNTFhZWI0YzFlMmUyNDExNGFkYWVlNzFlNWQyODc3YTYiLCJwIjoiaiJ9

Manual testing steps

Test plan:

Phase 1 behavior: PR without retry-ci label shows "add the retry-ci label to PR #X to retry" 🟢
Deterministic patterns (Quality Gate failed) block retry 🟢
Workflow YAML validates 🟢
.mts scripts run on Node 24 (--help and real classifications) 🟢
JS bundle exceeds baseline blocks retry 🟢
Release push gets one automatic retry 🟢
Push to main is observation only (no rerun) 🟢

Fork runs

Live retries ran on consensys-test/metamask-mobile-tanya. The fork's main carries this PR's triage workflow plus a one-job ci stand-in. The stand-in job is named Appium Smoke Tests, fails attempt 1 with ETIMEDOUT talking to the runner, and passes on attempt 2. Other workflows on the fork are disabled. workflow_run runs report main as their branch; the failed ci run id in the log is the one under test.

Push to main, no retry

  • ci 41962140436, commit e1d795688, attempt 1, failure. No rerun.
  • Triage 41962157515: is-retryable=true, No originating PR for event 'push' → will-retry=false, Retry mode: none (automatic limit 1; retry-ci limit 4).

The summary line says Retryable, but the automatic retry budget ended at attempt 1. The run was not retried, which is correct for main, but that sentence reads as if a budget was spent. Worth a wording change.

PR without the label

PR #1, retry-fixture at 4176eaf8d.

  • ci 41962633587, attempt 1, failure. No rerun.
  • Triage 41962647440: is-retryable=true, PR #1: retry-ci label absent, retry-mode=none → will-retry=false, Retryable — add the retry-ci label to PR #1 to retry.

PR with the label

Same PR, new commit 00be59476, label added before the push.

  • ci 41963293969 failed attempt 1, then finished attempt 2 with success.
  • Triage 41963310341: retry-ci label present, retry-mode=label → will-retry=true, Will retry (retry-ci label present; attempt 1 of 4), then Removing retry-ci label from PR #1....
  • PR Inject inpage js #1 now has no labels.

Release push, one automatic retry

Branch release/9.9.9-retry-test, commit fcf9b63df.

  • ci 41964455569 failed attempt 1, then finished attempt 2 with success.
  • Triage 41964469952: Release branch release/9.9.9-retry-test: retry-mode=automatic → will-retry=true, Will retry automatically (default retry budget: attempt 1 of 2), head still matches failed run fcf9b63…, Rerun triggered.

JS bundle baseline blocks retry

PR #2, bundle-fixture at 296a56006. The stand-in job on this branch is named JS bundle size and exits 1 with JS bundle exceeds baseline. The retry-ci label was on the PR before ci finished.

stable uses the same observation path as main. merge_group was not run; the fork has no merge queue, and this PR excludes that event.

Triage and Retry System

Run: 41759667183
Classification: ✅ All failures retryable
Retry: ⏸️ Retryable — add the retry-ci label to PR #38564 to retry
Retry mode: none (automatic limit 1; retry-ci limit 4)
Failed jobs: 2

Job Category Job Retryable Reason
Appium Smoke Tests (Android) / appium-android-smoke (SmokeSeedlessOnboarding,seedless-onboarding, 1, 1) / appium-seedless-onboarding-android-smoke-1 alwaysRetryable ✅ Job is in the always-retryable category
Check all jobs pass optional ❌ Optional job — no retry needed

Triage and Retry System

Run: 41768129279
Classification: ❌ Non-retryable failures detected
Retry: ❌ Non-retryable (PR #37252, no retry-ci label)
Retry mode: none (automatic limit 1; retry-ci limit 4)
Failed jobs: 2

Job Category Job Retryable Reason
SonarCloud quality gate status retryableOnTransientError ❌ Deterministic: Quality Gate failed
Check all jobs pass optional ❌ Optional job — no retry needed

Triage and Retry System

Run: 41757054185
Classification: ❌ Non-retryable failures detected
Retry: ❌ Non-retryable (PR #38815, no retry-ci label)
Retry mode: none (automatic limit 1; retry-ci limit 4)
Failed jobs: 27

Job Category Job Retryable Reason
Compute native build fingerprint retryableOnTransientError ❌ No transient error pattern detected
Prepare CI JS dependencies retryableOnTransientError ❌ Cascade — blocked by Compute native build fingerprint
Dependency audit (advisory) retryableOnTransientError ❌ Cascade — blocked by Compute native build fingerprint
Dedupe retryableOnTransientError ❌ Cascade — blocked by Compute native build fingerprint
Check diff retryableOnTransientError ❌ Cascade — blocked by Compute native build fingerprint
JS bundle size check retryableOnTransientError ❌ Cascade — blocked by Compute native build fingerprint
Run lint (3) retryableOnTransientError ❌ Cascade — blocked by Compute native build fingerprint
Run lint (2) retryableOnTransientError ❌ Cascade — blocked by Compute native build fingerprint
Run @lavamoat/git-safe-dependencies optional ❌ Cascade — blocked by Compute native build fingerprint
Run lint (1) retryableOnTransientError ❌ Cascade — blocked by Compute native build fingerprint
Run test:depcheck retryableOnTransientError ❌ Cascade — blocked by Compute native build fingerprint
Run lint (4) retryableOnTransientError ❌ Cascade — blocked by Compute native build fingerprint
Run lint retryableOnTransientError ❌ Cascade — blocked by Compute native build fingerprint
Component view tests (2) retryableOnTransientError ❌ Cascade — blocked by Compute native build fingerprint
Integration tests (1) retryableOnTransientError ❌ Cascade — blocked by Compute native build fingerprint
Unit tests (6) retryableOnTransientError ❌ Cascade — blocked by Compute native build fingerprint
Unit tests (2) retryableOnTransientError ❌ Cascade — blocked by Compute native build fingerprint
Unit tests (9) retryableOnTransientError ❌ Cascade — blocked by Compute native build fingerprint
Unit tests (3) retryableOnTransientError ❌ Cascade — blocked by Compute native build fingerprint
Unit tests (1) retryableOnTransientError ❌ Cascade — blocked by Compute native build fingerprint
Unit tests (5) retryableOnTransientError ❌ Cascade — blocked by Compute native build fingerprint
Unit tests (4) retryableOnTransientError ❌ Cascade — blocked by Compute native build fingerprint
Unit tests (7) retryableOnTransientError ❌ Cascade — blocked by Compute native build fingerprint
Unit tests (8) retryableOnTransientError ❌ Cascade — blocked by Compute native build fingerprint
Unit tests (10) retryableOnTransientError ❌ Cascade — blocked by Compute native build fingerprint
Build Android APKs / Build Android E2E APKs retryableOnTransientError ❌ Cascade — blocked by Compute native build fingerprint
Check all jobs pass optional ❌ Cascade — blocked by Compute native build fingerprint

Triage and Retry System

Run: 41775319673
Classification: ❌ Non-retryable failures detected
Retry: ❌ Non-retryable (PR #38738, no retry-ci label)
Retry mode: none (automatic limit 1; retry-ci limit 4)
Failed jobs: 3

Job Category Job Retryable Reason
Appium Smoke Tests (Android) / appium-android-smoke (SmokePredictions, predictions, 1, 1) / appium-predictions-android-smoke-1 alwaysRetryable ✅ Job is in the always-retryable category
SonarCloud quality gate status retryableOnTransientError ❌ Deterministic: Quality Gate failed
Check all jobs pass optional ❌ Optional job — no retry needed

Screenshots/Recordings

NA. Triage runs as a workflow with no UI change; evidence is the classifier output in Manual testing steps.

Before

After

Pre-merge author checklist

Performance checks (if applicable)

  • I've tested on Android
    • Ideally on a mid-range device; emulator is acceptable
  • I've tested with a power user scenario
    • Use these power-user SRPs to import wallets with many accounts and tokens
  • I've instrumented key operations with Sentry traces for production performance metrics

For performance guidelines and tooling, see the Performance Guide.

Pre-merge reviewer checklist

  • I've manually tested the PR (e.g. pull and build branch, run the app, test code being changed).
  • I confirm that this PR addresses all acceptance criteria described in the ticket it closes and includes the necessary testing evidence such as recordings and or screenshots.

Note

Medium Risk
New workflow uses elevated actions:write / pull-requests:write to rerun CI; mis-tuned patterns could retry real failures or skip needed retries, though label gating and deterministic-first rules limit blast radius on PRs.

Overview
Introduces a Triage and Retry System that runs after ci finishes: it classifies failed jobs, publishes a step-summary report (and optional Check Run + Sentry logs), and may trigger gh run rerun --failed when policy allows.

Classification is driven by retry-config.jsonc (job name categories, blocker cascade, Mobile-specific transient/deterministic log patterns with deterministic checks first). The classify-failures.mts script pulls failed jobs, annotations, and log tails via gh, then outputs is-retryable, will-retry, and retry-budget fields.

Phase 1 (stealth): PR failures retry only when the retry-ci label is present (label is removed on use, up to attempt 4); release/* pushes get one automatic retry; main/stable pushes are observe-only. merge_group is intentionally excluded. The workflow adds stale-SHA checks before rerunning so superseded commits are not retried.

Supporting pure helpers (retry-budget, retry-label, classify-failure-text) ship with unit tests; Jest gains a babel.config.mts-jest.js transform so tests can import .mts helpers.

Reviewed by Cursor Bugbot for commit d9692e5. Bugbot is set up for automated code reviews on this repo. Configure here.

@sleepytanya
sleepytanya requested a review from a team as a code owner October 28, 2026 22:20
@github-actions

Copy link
Copy Markdown
Contributor

CLA Signature Action: All authors have signed the CLA. You may need to manually re-run the blocking PR check if it doesn't pass in a few minutes.

@metamask-ci metamask-ci Bot added the team-delivery Release delivery team label Oct 28, 2026
@metamask-ci

metamask-ci Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

PR template — items to address before "Ready for review"

Warnings — informational, address before merging:

  • Pre-merge author checklist has unchecked items (e.g. "I've included tests if applicable"). Every box must be consciously checked — see docs/readme/ready-for-review.md.

See docs/readme/ready-for-review.md for the full Definition of Ready for Review.

@sleepytanya
sleepytanya force-pushed the feat/retry-system-mobile branch from 536c58b to 2608dd2 Compare October 28, 2026 22:26

@cursor cursor Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread .github/scripts/classify-failures.mts Resolved
Comment thread .github/rules/retry-config.jsonc Resolved
Comment thread .github/scripts/classify-failures.mts Resolved
  move Metro error to deterministic

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit e4e4bf9. Configure here.

Comment thread .github/scripts/classify-failures.mts Resolved
@sleepytanya

Copy link
Copy Markdown
Contributor Author

@HowardBraham

Tested workflow-fork-testing and ci-run-triage skills while proving this PR's retry paths on the fork.

workflow-fork-testing:
the setup matched what we needed. Per-person fork in consensys-test, Actions enabled before the first push, other workflows disabled, the triage YAML on the fork's main, fixture commits built without touching the working tree, and test PRs opened inside the fork (fork PRs #1 and #2).

ci-run-triage:
new chat was given only triage run 41964889541. It read the skill first, then reported the JS bundle size job as Deterministic: JS bundle exceeds baseline, not retried. The 404 in the screenshot is the attempt-2 lookup, which confirms nothing retried.

Two skill gaps(?):

  • triage run's head branch is main, so filtering by the PR branch misses it. Find the run by the failed ci run id in its log.
  • gh run view --log prints the workflow script, so a grep for retry-ci also matches lines that did not run.
Screenshot 2026-11-04 at 11 24 26 PM

@HowardBraham HowardBraham left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a combination of a review by my agent that developed the Extension Triage and Retry System, and my own review.

This whole system is extremely, extremely complicated, exploits gaps in GitHub Actions to force it to do things it wasn't designed to do, and has a dozen race conditions.

You need much more thorough tests on the fork that will test not just the simplest paths, but also the complex paths that will trip something up.

Comment thread .github/scripts/classify-failures.mts Outdated
if (hasPr && !hasLabel)
return {
key: 'retryable-no-label',
label: `⏸️ Retryable — add the retry-ci label to PR #${pr} to retry`,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did this too at first, but this is a misleading instruction. The suggestion here is clearer.

Suggested change
label: `⏸️ Retryable — add the retry-ci label to PR #${pr} to retry`,
label: `⏸️ Retryable — add the retry-ci label to PR #${pr} before the next run to retry`,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where did the contents of this file come from, and how much have you tested it already?

@sleepytanya sleepytanya Nov 16, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@HowardBraham

The initial patterns came from an analysis of recent Mobile CI failures on Oct 27: 50 failed runs for job names, then 100 for job names and error samples. The script we used was removed before the PR, so the results aren't in the repo. The header's "50" is from the first pass.

Tested on the fork (consensys-test/metamask-mobile-tanya) with the review fixes on fork main (0e8c425):

  • Push with no PR: classified retryable, not retried (43817900020).
  • PR 3 with retry-ci: attempt 1 retried and attempt 2 passed (43818417034). With the stand-in failing every attempt, attempts 1→2 and 2→3 each retried and the label was removed after each one (43818741706, 43818790509).
  • PR 3 with no label at classification time: attempt 3 did not retry, and the log said to add retry-ci (43818837744).
  • PR 3 at the limit: attempts 4, 5, and 6 all stopped with "reached the retry-ci limit (4)", and the label stayed (43819358400, 43819518387, 43819550758).
    Only run on the November 5 classifier:
  • Deterministic bundle failure with retry-ci: not retried (41964889541, PR 2).
  • release/* push: one automatic retry, label not used (41964469952).
    Still not run: a PR opened from outside the fork, and a second rerun of the same run id while the first is still running.

For myself - wording nit: past the limit the message reads "attempt 5 of 4".

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file never runs. Mobile's jest.config.js has no testMatch, and Jest's default pattern doesn't match .mts. Extension needed a jest.config.js testMatch change for exactly this. As it stands, the PR's only test is dead code in CI.

//
// Includes Mobile-specific patterns (simulator, emulator, Appium, Metro, etc.)
// plus common infrastructure patterns from Extension.
"transientErrorPatterns": [

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pattern ordering makes most of the Mobile transient patterns unreachable.

  • Deterministic patterns are checked first, and every regex is compiled with the case-insensitive i flag.
  • Build failed matches Gradle's BUILD FAILED in 3m on every failed Android build. That makes Gradle.*timeout, Gradle.*daemon.*crash and Could not resolve.*dependencies unreachable.
  • Tests:.*\d+ failed and Test Suites:.*failed appear in every failing Jest log, so Exceeded timeout of \d+ ms for a test and Jest worker encountered can never win. Our Mobile discovery found a Jest timeout to be the main transient candidate.
  • TypeError: also matches TypeError: fetch failed, which is a network error.
  • These need fixture tests (log text in, classification out) before anyone trusts them.

// Jobs that are normally deterministic but can fail due to infrastructure
// issues (OOM, network errors, etc.). Retried only if error matches a
// transient pattern from the global transientErrorPatterns list.
"retryableOnTransientError": {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Several real jobs match no pattern and fall back to the transient-only default: Report Fixture Validation, Ship JS bundle size check, Run Performance Tests (PR), Delete CI JS deps artifact, merge-unit-and-component-view-tests, Deploy Platform API Docs. Our Mobile plan said to default unmatched jobs to non-retryable until there's data. Either way, list them explicitly.

# Only populated for pull_request events; empty for push.
PR_NUMBER_FROM_EVENT: ${{ github.event.workflow_run.pull_requests[0].number }}
REPO: ${{ github.repository }}
SENTRY_DSN_PERFORMANCE: ${{ vars.SENTRY_DSN_PERFORMANCE }}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The way this is written, I think the Mobile data is going to collide into the Extension dashboard, and it's going to be harder to tell the two apart. Let's discuss options.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@HowardBraham
You are right!
Mobile could reuse the existing test project for triage logs. The workflow reads vars.SENTRY_DSN_PERFORMANCE, while the test DSN lives in the secret MM_SENTRY_DSN_TEST. Setting the repo variable to that DSN would match the Extension workflow. Pointing the workflow at the secret would be a Mobile-only change.

fi

echo "Rerunning failed jobs..."
gh run rerun "$MAIN_RUN_ID" --failed --repo "$REPO"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR path doesn't check that the run attempt is still current. The release path does.

  • Mobile has a second actor that reruns the same run ID: rerun-ci-on-skipped-e2e-labels.yml cancels the run and then calls gh run rerun $RUN_ID when E2E labels change.
  • Triage classifies attempt N. If that other rerun has already finished, triage runs rerun --failed against an attempt it never classified, and spends the label doing it.
  • Fix: copy the release path's status / conclusion / run_attempt / head_sha check into the PR branch.
  • This is our plan's rule "bind every operation to exact run ID, event, SHA and attempt."

Comment thread .github/scripts/classify-failures.mts Outdated
function checkRetryLabel(prNum: string): boolean {
if (!prNum) return false;
try {
const labels = ghApi(`${repoApi}/issues/${prNum}/labels`);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As is, this is not paginated and you'd see only 30 labels

Suggested change
const labels = ghApi(`${repoApi}/issues/${prNum}/labels`);
const labels = ghApi(`${repoApi}/issues/${prNum}/labels?per_page=100`);

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Smart E2E Test Selection

Selected E2E tags: No E2E tests selected

AI Confidence: 95

E2E reasoning

Expand to read

The PR adds CI triage and retry tooling: new .mts scripts under .github/scripts, a new workflow (triage-and-retry-system.yml), a retry-config.jsonc rules file, and a Jest transform entry for .mts files. The Jest change applies a TypeScript-only Babel config to .mts files, and no app source uses that extension. No app code, controller, UI, native module, or E2E test infrastructure is modified. The workflow is a separate workflow_run-triggered file and does not change ci.yml or build.yml. None of the E2E smoke flows are affected, so I selected no tags.

View run

@sleepytanya
sleepytanya marked this pull request as draft November 13, 2026 18:54
@sonarqubecloud

Copy link
Copy Markdown

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size-XL team-delivery Release delivery team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants