Repository navigation
Description
🤖 Posted by Codex, an AI coding agent acting for @Beamanator.
Summary
Upload iOS build to Sentry for size analysis is failing almost continuously, but the deployment workflow remains green and the real sentry-cli error is discarded.
Evidence
- Linked failure: production deploy run #6003 (2026-08-20).
- The iOS artifact download succeeds:
gh release download "9.4.56-3-staging" --pattern "Expensify.ipa"completed successfully. - The next step,
npx sentry-cli build upload "Expensify.ipa" --org expensify --project app --build-configuration Release --log-level info, exits with code 1. - Among the 33 production deploy runs after this upload moved into deploy (2026-06-26 through 2026-08-20), 31 have the same
Upload to Sentry for size analysis — Process completed with exit code 1annotation (~94%). Only two runs completed without that annotation. - No open App issue matches this failure. Related setup/migration work is closed: [Due for payment 2026-07-03] Set up Sentry release size analysis #82850, [HOLD][No QA] Set up Sentry release size analysis #82937, [No QA] ci: Add Sentry build size analysis #82952, [No QA] Move sentry size analysis upload to deploy step #94039, and [Due for payment 2026-07-02] Bundle Size Analysis links in deploy comments are broken #94382.
Why diagnosis is currently impossible
The composite action captures the CLI output in a command substitution:
OUTPUT=$(npx sentry-cli build upload "$ASSET" ... 2>&1)
echo "$OUTPUT"GitHub runs Bash with -e. When sentry-cli exits non-zero, the assignment itself exits non-zero, so the step stops before echo "$OUTPUT". Its stderr is therefore captured and lost.
Both the composite upload step and the workflow job use continue-on-error, so the deploy stays successful while GitHub records only the unhelpful exit-code annotation. We can prove that sentry-cli is the failing command, but cannot determine whether the immediate cause is Sentry authentication, an API/quota error, or invalid build metadata from historical logs.
Proposed fix
- Preserve and print the CLI output before failing:
set +e
OUTPUT=$(bunx sentry-cli build upload "$ASSET" \
--org expensify \
--project app \
--build-configuration Release \
--log-level info 2>&1)
EXIT_CODE=$?
set -e
printf '%s\n' "$OUTPUT"
if (( EXIT_CODE != 0 )); then
echo "::error::Sentry size-analysis upload failed (exit $EXIT_CODE)"
exit "$EXIT_CODE"
fi- Keep the outer job non-blocking only if that is intentional, but retain the emitted diagnostic in the action log.
- Add explicit logging around the release-artifact download so an absent/invalid IPA is distinguishable from a Sentry upload failure.
- Re-run a production upload after the observability fix and use the retained
sentry-clioutput to address the actual service/configuration cause.
Acceptance criteria
- A failed upload logs the complete
sentry-clidiagnostic. - The workflow clearly distinguishes artifact-download failures from Sentry-upload failures.
- The next production run either uploads successfully or provides enough evidence to fix the remaining root cause.
Activity
Beamanator commentedon Aug 20, 2026
@MelvinBot investigate the issue and proposed fix
MelvinBot commentedon Aug 20, 2026
Issue Analysis
The diagnosis is correct. The
sentry-clioutput is lost because of a classicset -egotcha, not becausesentry-cliwrites nothing. The proposed fix is the right approach with two corrections (keepnpx, notbunx; and item #3 is already largely handled).Root cause: GitHub runs
shell: bashasbash --noprofile --norc -eo pipefail {0}, soset -eis on. InuploadSentrySizeAnalysis/action.yml:64the step is:An assignment whose value comes from a command substitution takes the substitution's exit status, so when
sentry-cliexits 1 the assignment exits 1 andset -eaborts the step beforeecho "$OUTPUT"runs. The captured stderr never prints; all that survives is theProcess completed with exit code 1annotation.continue-on-error: trueon both the step and the job then keeps the deploy green, so the failure is invisible except as noise. All of this matches the issue exactly.Recommended fix — apply the
set +e/ capture-exit / print pattern from the proposal, with these adjustments:Why these adjustments to the proposal
Keep
npx, notbunx. The proposal's snippet silently switchesnpx→bunx. This composite sets up Node viasetupNode(npm), not bun, sobunxwould introduce a new failure. Change only the error handling, not the runner.::warning::+exit 0instead of::error::+exit $EXIT_CODE. Both the step and the job already carrycontinue-on-error: true, so the exit code can't block the deploy either way — but exiting non-zero keeps emitting the red run-level annotation that motivated this issue. Emitting awarningand exiting 0 surfaces the full CLI diagnostic and removes the persistent red noise. If you'd rather keep it visibly failing,exit "$EXIT_CODE"is fine — the diagnostic now prints regardless.Proposal item Add Expensify eslint #3 (distinguish download vs upload failures) is already handled. The re-promotion path already guards the download with
[ -s "$ASSET" ] || { echo "::error::Failed to obtain..."; exit 1; }, and the upload step is gated byif: hashFiles(inputs.asset-name) != ''. So a missing/empty IPA fails at the download step and skips the upload — it can't masquerade as a Sentry failure today. The only small gap is the cherry-pick path (download-artifactwithcontinue-on-error): if that produced no file the upload is silently skipped with no note. An optionalelsebranch on thehashFilesguard logging "no binary found, skipping upload" would close that.The actual service-level cause still can't be known from code — and that's the point. This fix is observability; it makes the next production run print the real
sentry-clierror. Two data points worth noting up front: the issue confirmsgh release downloadsucceeds, which needsGH_TOKEN— so the caller's step-levelenv(includingSENTRY_AUTH_TOKEN) is reaching the composite's steps, making a plain auth failure unlikely. Combined with the action's own doc string ("empty if upload failed or quota exhausted"), Sentry size-analysis quota exhaustion is the leading hypothesis for the ~94% rate. Confirm from the retained output once merged.Confidence: High on the root cause of the lost output and the fix. The underlying reason
sentry-cliitself exits 1 is unverifiable from code alone (by design — this issue exists to surface it).Next Steps for Contributor+ team:
To accept:
@MelvinBot implement thisto create a draft PR.To refine:
@MelvinBot <your feedback>To reject: Explain why you are rejecting Melvin's proposal.
view run
Beamanator commentedon Aug 24, 2026
will look into this more this week
Beamanator commentedon Sep 6, 2026
same
Beamanator commentedon Sep 23, 2026
@MelvinBot check if this is still happening and if there's any open conversations about this
4 remaining items
rlinoz commentedon Sep 30, 2026
I think it should be pretty simple if you wanna review it
rlinoz commentedon Sep 30, 2026
@MelvinBot verify this is the fix #99083 (comment) and if so implement it.
MelvinBot commentedon Sep 30, 2026
🤖 Verified: moving the iOS upload to a Mac runner is the right fix. I couldn't open the PR because my token can't push changes to
.github/workflows/.sentry-clionly includes IPA support when it runs on macOS on Apple Silicon. In@sentry/cli@3.3.0,src/commands/build/upload.rs:38-43limits the Linux build to APK and AAB. The IPA check atupload.rs:537-540is left out of Linux builds entirely.iosUploadSentryruns onblacksmith-2vcpu-ubuntu-2404, so it can never succeed. Android works on the same runs, which rules out auth and quota problems.@sentry/cli-darwinis a universal binary, so it runs natively on Apple Silicon.Blocker: GitHub rejected the push with
refusing to allow a Personal Access Token to create or update workflow .github/workflows/deploy.yml without workflow scope. The change is ready on a local branch. You or someone with workflow push access can apply it withgit apply:Patch (2 files)
Why these choices
blacksmith-6vcpu-macos-latest: It's the smaller Apple Silicon runner already used inverifyPodfile.ymlandsyncVersions.yml. The job is a single upload, so it doesn't need the 12-vcpu runner thatiosUploadTestflightuses.actions/checkout:useblacksmith/checkoutstalls for about 75 seconds on macOS, according to.github/workflows/README.md. Every other macOS job uses upstream checkout.setupNodeusesactions/cacheinstead of sticky disks. A cache miss runsnpm ci, which can take longer than 10 minutes on its own.set +ein the composite action: Today,action.yml:64aborts beforeecho "$OUTPUT"wheneversentry-clifails, so the real error is lost. With this change, any remaining iOS-side problem (for example, a dSYM requirement) will show up in the log. The deploy still can't be blocked, because the step and the job both keepcontinue-on-error.Not verified:
gh release download. I'm assumingghis installed on Blacksmith macOS images, as it is on GitHub's macOS images.Beamanator commentedon Sep 30, 2026
ya i'm down to review :D
melvin-bot commentedon Oct 5, 2026
@Beamanator / @rlinoz
The solution for this issue has been 🚀 deployed to production 🚀 in version 9.5.1-9 and is now subject to a 7-day regression period 📆. Here is the list of pull requests that resolve this issue:
rlinoz commentedon Oct 5, 2026