Skip to content

[CI] Surface and fix recurrent iOS Sentry size-analysis upload failures #99083

Closed

Description

@Beamanator

🤖 Posted by Codex, an AI coding agent acting for @Beamanator.

Summary

Upload iOS build to Sentry for size analysis is failing almost continuously, but the deployment workflow remains green and the real sentry-cli error is discarded.

Evidence

Why diagnosis is currently impossible

The composite action captures the CLI output in a command substitution:

OUTPUT=$(npx sentry-cli build upload "$ASSET" ... 2>&1)
echo "$OUTPUT"

GitHub runs Bash with -e. When sentry-cli exits non-zero, the assignment itself exits non-zero, so the step stops before echo "$OUTPUT". Its stderr is therefore captured and lost.

Both the composite upload step and the workflow job use continue-on-error, so the deploy stays successful while GitHub records only the unhelpful exit-code annotation. We can prove that sentry-cli is the failing command, but cannot determine whether the immediate cause is Sentry authentication, an API/quota error, or invalid build metadata from historical logs.

Proposed fix

  1. Preserve and print the CLI output before failing:
set +e
OUTPUT=$(bunx sentry-cli build upload "$ASSET" \
  --org expensify \
  --project app \
  --build-configuration Release \
  --log-level info 2>&1)
EXIT_CODE=$?
set -e
printf '%s\n' "$OUTPUT"

if (( EXIT_CODE != 0 )); then
  echo "::error::Sentry size-analysis upload failed (exit $EXIT_CODE)"
  exit "$EXIT_CODE"
fi
  1. Keep the outer job non-blocking only if that is intentional, but retain the emitted diagnostic in the action log.
  2. Add explicit logging around the release-artifact download so an absent/invalid IPA is distinguishable from a Sentry upload failure.
  3. Re-run a production upload after the observability fix and use the retained sentry-cli output to address the actual service/configuration cause.

Acceptance criteria

  • A failed upload logs the complete sentry-cli diagnostic.
  • The workflow clearly distinguishes artifact-download failures from Sentry-upload failures.
  • The next production run either uploads successfully or provides enough evidence to fix the remaining root cause.

Activity

  1. self-assigned this
    on Aug 20, 2026
  2. Beamanator commented on Aug 20, 2026

    @Beamanator
    ContributorAuthor

    @MelvinBot investigate the issue and proposed fix

  3. MelvinBot commented on Aug 20, 2026

    @MelvinBot
    Contributor

    Issue Analysis

    The diagnosis is correct. The sentry-cli output is lost because of a classic set -e gotcha, not because sentry-cli writes nothing. The proposed fix is the right approach with two corrections (keep npx, not bunx; and item #3 is already largely handled).

    Root cause: GitHub runs shell: bash as bash --noprofile --norc -eo pipefail {0}, so set -e is on. In uploadSentrySizeAnalysis/action.yml:64 the step is:

    OUTPUT=$(npx sentry-cli build upload "$ASSET" ... 2>&1)
    echo "$OUTPUT"

    An assignment whose value comes from a command substitution takes the substitution's exit status, so when sentry-cli exits 1 the assignment exits 1 and set -e aborts the step before echo "$OUTPUT" runs. The captured stderr never prints; all that survives is the Process completed with exit code 1 annotation. continue-on-error: true on both the step and the job then keeps the deploy green, so the failure is invisible except as noise. All of this matches the issue exactly.

    Recommended fix — apply the set +e / capture-exit / print pattern from the proposal, with these adjustments:

    set +e
    OUTPUT=$(npx sentry-cli build upload "$ASSET" \
      --org expensify --project app --build-configuration Release --log-level info 2>&1)
    EXIT_CODE=$?
    set -e
    printf '%s\n' "$OUTPUT"
    
    SENTRY_URL=$(printf '%s' "$OUTPUT" | grep -oE 'https://expensify\.sentry\.io/[^ )"}]+' | head -1)
    if [ "$EXIT_CODE" -ne 0 ]; then
      echo "::warning::${PLATFORM} Sentry size-analysis upload failed (exit $EXIT_CODE) — see log above"
      exit 0
    elif [ -n "$SENTRY_URL" ]; then
      echo "::notice::${PLATFORM} Sentry size analysis: $SENTRY_URL"
      echo "SENTRY_URL=$SENTRY_URL" >> "$GITHUB_OUTPUT"
    fi
    Why these adjustments to the proposal
    1. Keep npx, not bunx. The proposal's snippet silently switches npx → bunx. This composite sets up Node via setupNode (npm), not bun, so bunx would introduce a new failure. Change only the error handling, not the runner.

    2. ::warning:: + exit 0 instead of ::error:: + exit $EXIT_CODE. Both the step and the job already carry continue-on-error: true, so the exit code can't block the deploy either way — but exiting non-zero keeps emitting the red run-level annotation that motivated this issue. Emitting a warning and exiting 0 surfaces the full CLI diagnostic and removes the persistent red noise. If you'd rather keep it visibly failing, exit "$EXIT_CODE" is fine — the diagnostic now prints regardless.

    3. Proposal item Add Expensify eslint #3 (distinguish download vs upload failures) is already handled. The re-promotion path already guards the download with [ -s "$ASSET" ] || { echo "::error::Failed to obtain..."; exit 1; }, and the upload step is gated by if: hashFiles(inputs.asset-name) != ''. So a missing/empty IPA fails at the download step and skips the upload — it can't masquerade as a Sentry failure today. The only small gap is the cherry-pick path (download-artifact with continue-on-error): if that produced no file the upload is silently skipped with no note. An optional else branch on the hashFiles guard logging "no binary found, skipping upload" would close that.

    4. The actual service-level cause still can't be known from code — and that's the point. This fix is observability; it makes the next production run print the real sentry-cli error. Two data points worth noting up front: the issue confirms gh release download succeeds, which needs GH_TOKEN — so the caller's step-level env (including SENTRY_AUTH_TOKEN) is reaching the composite's steps, making a plain auth failure unlikely. Combined with the action's own doc string ("empty if upload failed or quota exhausted"), Sentry size-analysis quota exhaustion is the leading hypothesis for the ~94% rate. Confirm from the retained output once merged.

    Confidence: High on the root cause of the lost output and the fix. The underlying reason sentry-cli itself exits 1 is unverifiable from code alone (by design — this issue exists to surface it).


    Next Steps for Contributor+ team:
    To accept: @MelvinBot implement this to create a draft PR.
    To refine: @MelvinBot <your feedback>
    To reject: Explain why you are rejecting Melvin's proposal.


    view run

  4. Beamanator commented on Aug 24, 2026

    @Beamanator
    ContributorAuthor

    will look into this more this week

  5. Beamanator commented on Sep 6, 2026

    @Beamanator
    ContributorAuthor

    same

  6. Beamanator commented on Sep 23, 2026

    @Beamanator
    ContributorAuthor

    @MelvinBot check if this is still happening and if there's any open conversations about this

  7. 4 remaining items

  8. rlinoz commented on Sep 30, 2026

    @rlinoz
    Contributor

    I think it should be pretty simple if you wanna review it

  9. rlinoz commented on Sep 30, 2026

    @rlinoz
    Contributor

    @MelvinBot verify this is the fix #99083 (comment) and if so implement it.

  10. self-assigned this
    on Sep 30, 2026
  11. MelvinBot commented on Sep 30, 2026

    @MelvinBot
    Contributor

    🤖 Verified: moving the iOS upload to a Mac runner is the right fix. I couldn't open the PR because my token can't push changes to .github/workflows/.

    sentry-cli only includes IPA support when it runs on macOS on Apple Silicon. In @sentry/cli@3.3.0, src/commands/build/upload.rs:38-43 limits the Linux build to APK and AAB. The IPA check at upload.rs:537-540 is left out of Linux builds entirely. iosUploadSentry runs on blacksmith-2vcpu-ubuntu-2404, so it can never succeed. Android works on the same runs, which rules out auth and quota problems. @sentry/cli-darwin is a universal binary, so it runs natively on Apple Silicon.

    Blocker: GitHub rejected the push with refusing to allow a Personal Access Token to create or update workflow .github/workflows/deploy.yml without workflow scope. The change is ready on a local branch. You or someone with workflow push access can apply it with git apply:

    Patch (2 files)
    diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml
    @@ -1151,18 +1151,20 @@ jobs:
       iosUploadSentry:
         name: Upload iOS build to Sentry for size analysis
         needs: [prep, iosBuild, checkDeploymentSuccess]
    -    runs-on: blacksmith-2vcpu-ubuntu-2404
    +    # sentry-cli only supports IPA uploads when running on Apple Silicon macOS
    +    runs-on: blacksmith-6vcpu-macos-latest
         if: >-
           ${{ always()
               && needs.prep.outputs.DEPLOY_ENV == 'production'
               && fromJSON(needs.checkDeploymentSuccess.outputs.IS_RELEASE_READY) }}
         continue-on-error: true
    -    timeout-minutes: 10
    +    timeout-minutes: 20
         outputs:
           SENTRY_URL: ${{ steps.sentry-upload.outputs.SENTRY_URL }}
         steps:
           - name: Checkout
    -        uses: useblacksmith/checkout@0647fdbab2614a5eb86d2971070e325060d91056 # v1.7.0
    +        # Upstream checkout on macOS - Blacksmith's git-mirror cache can't work there (see workflows/README.md)
    +        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
     
           - name: Upload to Sentry for size analysis
             id: sentry-upload
    diff --git a/.github/actions/composite/uploadSentrySizeAnalysis/action.yml b/.github/actions/composite/uploadSentrySizeAnalysis/action.yml
    @@ -61,9 +61,17 @@ runs:
             ASSET: ${{ inputs.asset-name }}
             PLATFORM: ${{ inputs.platform }}
           run: |
    +        # Capture the exit code without aborting so the sentry-cli output is always printed
    +        set +e
             OUTPUT=$(npx sentry-cli build upload "$ASSET" \
               --org expensify --project app --build-configuration Release --log-level info 2>&1)
    +        EXIT_CODE=$?
    +        set -e
             echo "$OUTPUT"
    +        if [ "$EXIT_CODE" -ne 0 ]; then
    +          echo "::error::${PLATFORM} Sentry size-analysis upload failed (exit $EXIT_CODE)"
    +          exit "$EXIT_CODE"
    +        fi
             SENTRY_URL=$(echo "$OUTPUT" | grep -oE 'https://expensify\.sentry\.io/[^ )"}]+' | head -1)
    Why these choices
    • blacksmith-6vcpu-macos-latest: It's the smaller Apple Silicon runner already used in verifyPodfile.yml and syncVersions.yml. The job is a single upload, so it doesn't need the 12-vcpu runner that iosUploadTestflight uses.
    • actions/checkout: useblacksmith/checkout stalls for about 75 seconds on macOS, according to .github/workflows/README.md. Every other macOS job uses upstream checkout.
    • Timeout 10 → 20 min: On macOS, setupNode uses actions/cache instead of sticky disks. A cache miss runs npm ci, which can take longer than 10 minutes on its own.
    • set +e in the composite action: Today, action.yml:64 aborts before echo "$OUTPUT" whenever sentry-cli fails, so the real error is lost. With this change, any remaining iOS-side problem (for example, a dSYM requirement) will show up in the log. The deploy still can't be blocked, because the step and the job both keep continue-on-error.

    Not verified:

    • I couldn't run prettier or a YAML check in this environment. I reviewed the diff by hand.
    • A real upload needs a production deploy to confirm.
    • On re-promotion deploys, the upload uses gh release download. I'm assuming gh is installed on Blacksmith macOS images, as it is on GitHub's macOS images.
  12. Beamanator commented on Sep 30, 2026

    @Beamanator
    ContributorAuthor

    ya i'm down to review :D

  13. added
    ReviewingHas a PR in review
    and removed
    ReviewingHas a PR in review
    on Sep 30, 2026
  14. melvin-bot commented on Oct 5, 2026

    @melvin-bot

    @Beamanator / @rlinoz

    The solution for this issue has been 🚀 deployed to production 🚀 in version 9.5.1-9 and is now subject to a 7-day regression period 📆. Here is the list of pull requests that resolve this issue:

  15. rlinoz commented on Oct 5, 2026

    @rlinoz
    Contributor
    Image
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions