Skip to content

[Due for payment 2026-02-27] [Tracking] Use Adhoc Build Snapshots to Identify Deploy Blockers Faster #79781

Closed
1 of 1 issue completed

Description

@mountiny

Coming from the proposal here

Proposal: Use Adhoc Build Snapshots to Identify Deploy Blockers Faster

Strategy

Expensify operates in a highly competitive market where rapid iteration and consistent daily improvements are critical to maintaining product quality and reducing churn. To sustain this pace, we must minimize engineering interruptions caused by unclear deploy blockers and keep engineers focused on high-ROI roadmap work. Streamlining the deploy process directly improves efficiency and protects free cash flow by reducing wasted engineering time.

Background

With the adoption of Rock, ad hoc builds can now be generated significantly faster than before. Each ad hoc build represents a full, isolated build of the App or Mobile Expensify at a specific point in time.

Deploy blockers occasionally surface on staging without an obvious root cause, especially when the staging deploy checklist contains many merged pull requests. In these cases, engineers must manually debug the issue by locally reverting and rebuilding the app to validate potential fixes.

Problem

When we aim to deploy the App every day, if a deploy blocker appears that requires extensive debugging to identify the offending PR, then assigned engineers are forced into disruptive context-switching and losing focus on critical roadmap work.

Solution

We will use fast ad hoc builds as chronological snapshots of the app state, enabling QA to quickly isolate regressions and identify the responsible PR without engineering intervention.:

  1. Generate an ad-hoc build snapshot for every merged PR
    • Whenever a PR is merged into App or Mobile Expensify, automatically generate an ad hoc build from the latest main that includes that PR.
    • Each snapshot represents the exact state of the app immediately after the merge.
    • Based on recent testing, build times average ~6 minutes for web, ~5 minutes for iOS, and ~5 minutes for Android when the PR does not include native changes (the majority of PRs).
    • At a cost of $0.16 per minute for GitHub CI runners, each build costs approximately $2.50.
    • We can exclude PRs that only modify markdown files, GitHub configuration, or help pages.
    • We should update the README on main with a link to the latest preview build of main.
  2. Chronologically order PRs in the staging checklist
    • Display merged PRs in the staging checklist in the exact order they were merged.
    • This creates a clear, linear timeline of changes leading up to a deploy.
    • A linear timeline is a prerequisite for efficiently using binary search to identify which PR introduced a regression.
  3. Binary-search for the offending PR when a deploy blocker appears
    • When a deploy blocker arises, assign a C+ and Applause tester to the deploy blocker
    • The tester performs a binary search across the chronological adhoc snapshots to identify the exact PR that introduced the regression (there are cases when the offending PR might be clear, for example it is a new feature so they can skip the search in such case)
    • Once identified, the assigned C+ then can create revert PR directly, allowing deploys to get unblocked fast.

This approach removes guesswork, reduces engineering interruptions, and significantly accelerates recovery from deploy blockers. By preserving daily deploy velocity and minimizing context switching, this proposal directly supports faster roadmap execution and more efficient use of internal engineering time.

Issue OwnerCurrent Issue Owner: @twisterdotcom

Activity

  1. melvin-bot commented on Jan 16, 2026

    @melvin-bot

    Triggered auto assignment to @twisterdotcom (NewFeature), see https://stackoverflowteams.com/c/expensify/questions/14418#:~:text=BugZero%20process%20steps%20for%20feature%20requests for more details. Please add this Feature request to a GH project, as outlined in the SO. Remember to draft any necessary HelpDot updates by following the instructions here.

  2. melvin-bot commented on Jan 16, 2026

    @melvin-bot

    Triggered auto assignment to Design team member for new feature review - @dubielzyk-expensify (NewFeature)

  3. mountiny commented on Jan 16, 2026

    @mountiny
    ContributorAuthor

    cc @isagoico as we will need to coordinate on the process

  4. 40 remaining items

  5. changed the title [-][Tracking] Use Adhoc Build Snapshots to Identify Deploy Blockers Faster[/-] [+][Due for payment 2026-02-27] [Tracking] Use Adhoc Build Snapshots to Identify Deploy Blockers Faster[/+] on Feb 20, 2026
  6. melvin-bot commented on Feb 20, 2026

    @melvin-bot

    Reviewing label has been removed, please complete the "BugZero Checklist".

  7. melvin-bot commented on Feb 20, 2026

    @melvin-bot

    The solution for this issue has been 🚀 deployed to production 🚀 in version 9.3.23-3 and is now subject to a 7-day regression period 📆. Here is the list of pull requests that resolve this issue:

    If no regressions arise, payment will be issued on 2026-02-27. 🎊

    For reference, here are some details about the assignees on this issue:

    • @LukasMod does not require payment (Contractor)
  8. melvin-bot commented on Feb 20, 2026

    @melvin-bot

    BugZero Checklist: The PR adding this new feature has been merged! The following checklist (instructions) will need to be completed before the issue can be closed:

    • [@LukasMod] Please propose regression test steps to ensure the new feature will work correctly on production in further releases.
    • [] Link the GH issue for creating/updating the regression test once above steps have been agreed upon.
  9. melvin-bot commented on Mar 5, 2026

    @melvin-bot

    @mountiny, @LukasMod, @dubielzyk-expensify Still overdue 6 days?! Let's take care of this!

  10. mountiny commented on Mar 5, 2026

    @mountiny
    ContributorAuthor

    This has been implemented so I believe we can close this one out

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

Awaiting PaymentAuto-added when associated PR is deployed to productionDailyKSv2NewFeatureSomething to build that is a new item.Task

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions