Repository navigation
Description
Coming from the proposal here
Proposal: Use Adhoc Build Snapshots to Identify Deploy Blockers Faster
Strategy
Expensify operates in a highly competitive market where rapid iteration and consistent daily improvements are critical to maintaining product quality and reducing churn. To sustain this pace, we must minimize engineering interruptions caused by unclear deploy blockers and keep engineers focused on high-ROI roadmap work. Streamlining the deploy process directly improves efficiency and protects free cash flow by reducing wasted engineering time.
Background
With the adoption of Rock, ad hoc builds can now be generated significantly faster than before. Each ad hoc build represents a full, isolated build of the App or Mobile Expensify at a specific point in time.
Deploy blockers occasionally surface on staging without an obvious root cause, especially when the staging deploy checklist contains many merged pull requests. In these cases, engineers must manually debug the issue by locally reverting and rebuilding the app to validate potential fixes.
Problem
When we aim to deploy the App every day, if a deploy blocker appears that requires extensive debugging to identify the offending PR, then assigned engineers are forced into disruptive context-switching and losing focus on critical roadmap work.
Solution
We will use fast ad hoc builds as chronological snapshots of the app state, enabling QA to quickly isolate regressions and identify the responsible PR without engineering intervention.:
- Generate an ad-hoc build snapshot for every merged PR
- Whenever a PR is merged into App or Mobile Expensify, automatically generate an ad hoc build from the latest main that includes that PR.
- Each snapshot represents the exact state of the app immediately after the merge.
- Based on recent testing, build times average ~6 minutes for web, ~5 minutes for iOS, and ~5 minutes for Android when the PR does not include native changes (the majority of PRs).
- At a cost of $0.16 per minute for GitHub CI runners, each build costs approximately $2.50.
- We can exclude PRs that only modify markdown files, GitHub configuration, or help pages.
- We should update the README on main with a link to the latest preview build of main.
- Chronologically order PRs in the staging checklist
- Display merged PRs in the staging checklist in the exact order they were merged.
- This creates a clear, linear timeline of changes leading up to a deploy.
- A linear timeline is a prerequisite for efficiently using binary search to identify which PR introduced a regression.
- Binary-search for the offending PR when a deploy blocker appears
- When a deploy blocker arises, assign a C+ and Applause tester to the deploy blocker
- The tester performs a binary search across the chronological adhoc snapshots to identify the exact PR that introduced the regression (there are cases when the offending PR might be clear, for example it is a new feature so they can skip the search in such case)
- Once identified, the assigned C+ then can create revert PR directly, allowing deploys to get unblocked fast.
This approach removes guesswork, reduces engineering interruptions, and significantly accelerates recovery from deploy blockers. By preserving daily deploy velocity and minimizing context switching, this proposal directly supports faster roadmap execution and more efficient use of internal engineering time.
Issue Owner
Current Issue Owner: @twisterdotcomMetadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsDone
Activity
melvin-bot commentedon Jan 16, 2026
Triggered auto assignment to @twisterdotcom (
NewFeature), see https://stackoverflowteams.com/c/expensify/questions/14418#:~:text=BugZero%20process%20steps%20for%20feature%20requests for more details. Please add this Feature request to a GH project, as outlined in the SO. Remember to draft any necessary HelpDot updates by following the instructions here.melvin-bot commentedon Jan 16, 2026
Triggered auto assignment to Design team member for new feature review - @dubielzyk-expensify (
NewFeature)mountiny commentedon Jan 16, 2026
cc @isagoico as we will need to coordinate on the process
40 remaining items
[-][Tracking] Use Adhoc Build Snapshots to Identify Deploy Blockers Faster[/-][+][Due for payment 2026-02-27] [Tracking] Use Adhoc Build Snapshots to Identify Deploy Blockers Faster[/+]melvin-bot commentedon Feb 20, 2026
Reviewinglabel has been removed, please complete the "BugZero Checklist".melvin-bot commentedon Feb 20, 2026
The solution for this issue has been 🚀 deployed to production 🚀 in version 9.3.23-3 and is now subject to a 7-day regression period 📆. Here is the list of pull requests that resolve this issue:
If no regressions arise, payment will be issued on 2026-02-27. 🎊
For reference, here are some details about the assignees on this issue:
melvin-bot commentedon Feb 20, 2026
BugZero Checklist: The PR adding this new feature has been merged! The following checklist (instructions) will need to be completed before the issue can be closed:
melvin-bot commentedon Mar 5, 2026
@mountiny, @LukasMod, @dubielzyk-expensify Still overdue 6 days?! Let's take care of this!
mountiny commentedon Mar 5, 2026
This has been implemented so I believe we can close this one out