Repository navigation
Closed
Labels
EngineeringInternalRequires API changes or must be handled by Expensify staffRequires API changes or must be handled by Expensify staffNewFeatureSomething to build that is a new item.Something to build that is a new item.WeeklyKSv2KSv2
Description
Problem: For the most part, the NewDot deploy system served us well over the last couple years. We've successfully completed over 2,000 staging deploys and hundreds of production deploys. :star-struck: However, experience has revealed a few issues with the current system:
- It's complex, which makes it harder to follow and diagnose problems
- It's brittle. It's a complex state machine that doesn't operate well when it gets into an "unexpected state". When it's in an unexpected state, it's difficult to predict what will happen when you take any given action to try and fix it. This has led to us making problems worse, and even doing an accidental production deploy before.
- It's unpredictable. The lack of idempotency in deploy actions means that the same action, taken twice, can end up with different results.
- It's weak. Which is to say, when our QA fails and we do ship bugs to customers, we are limited in our options to fix those, and/or otherwise prevent real users from experiencing those bugs.
- It's slow. The amount of time it takes to handle deploys and push out fixes eats up a lot of deployer time during their week.
- It's fundamentally flawed. There are a few rare cases in which the underlying git logic is flawed, and this can end up in the wrong code being deployed to staging and/or production.
These problems prevent daily deploys, waste deployer time, and result in a higher probability of NewDot users experiencing bugs.
Solution: Let's engage in a holistic rebuild of the NewDot deploy system to address these issues and focus on improving the simplicity, durability, idempotency, speed, and flexibility of the system, and include this project in the scope of #vip-waq-everywhere.
Known Issues
TODO:
- [HOLD] Move No QA PRs to a separate section of the checklist #9905
- Failed CP's can break our CI/CD #6863
- https://github.com/Expensify/Expensify/issues/195693
- Merging code to main can trigger deploy of unrelated systems #9725
- [HOLD] Multiple concurrent deploys are slow #9726
- [HOLD] Implement phased rollouts of the New Expensify mobile app #6306
- Get rid of deprecated set-output commands in GitHub Actions #11877
- https://github.com/Expensify/Expensify/issues/240551
- [$500] [HOLD for payment 2023-12-06] Local testing of GitHub Actions #13604
- [HOLD] Create a production QA checklist #13605
- [HOLD] Race condition between staging and production deploys #14834
- Production deploys can fail if a lower version is pending review #20196
- [Due 2023-11-01] Deprecate altool – use notarytool instead #20763
- https://github.com/Expensify/Expensify/issues/299601
- Race condition can cause big checklists #27123
- https://github.com/Expensify/Expensify/issues/330131
In Progress:
Done:
- [HOLD] Manually CPing a pull request does not add the PR to the checklist #4379
- [$250] Race condition with the CP Staging label #7284
- Failed workflow comments in slack do not have the correct URL #12392
- [HOLD for payment 2023-01-30] Remove at-mentions in deploy comments #14404
- [HOLD for payment 2023-06-21] Cache node_modules to speed up CI/CD #20433
- [HOLD for payment 2023-05-23] Make it so we can run E2E performance tests on any branch #15605
- Investigate why some changes didn't make it to staging as expected #10214
- https://github.com/Expensify/Expensify/issues/280286
- [HOLD for payment 2023-07-06] [HOLD for payment 2023-07-05] Removing CP Staging label #20764
- https://github.com/Expensify/Expensify/issues/293688
- PRs already deployed to production included on staging checklist, PRs incorrectly commented on #23368
Metadata
Metadata
Assignees
Labels
EngineeringInternalRequires API changes or must be handled by Expensify staffRequires API changes or must be handled by Expensify staffNewFeatureSomething to build that is a new item.Something to build that is a new item.WeeklyKSv2KSv2
Activity
[-][Tracking] NewDot Deploy Improvements[/-][+][HOLD][Tracking] NewDot Deploy Improvements[/+]roryabraham commentedon Nov 28, 2022
Putting this on HOLD for WAQ
roryabraham commentedon Dec 14, 2022
I am going to drop this for now with the intention of bringing it back up as a SWM project once they are done with the PR testing project.
roryabraham commentedon Jan 17, 2023
Another potentially valuable addition here would be to deprecate
.github/libsand provide a cleaner way to share code between the main application, tests, and GitHub Actions. Right now we have issues where CJS and ES6 modules don't know how to talk to eachother properly (example)71 remaining items
roryabraham commentedon Jan 17, 2024
I think the last idea is a good one. I also think it's a bit of a problem that we can't retry a build on one platform until the builds for all platforms complete.
roryabraham commentedon Feb 6, 2024
No update right now
roryabraham commentedon Feb 16, 2024
Continued exploration in #22821
roryabraham commentedon Feb 27, 2024
Last week I found and fixed some issues with our pod cache that was making our iOS builds unreliable. I'm hoping this will contribute to significantly reduced build failures and hopefully no need to ever go deleting caches manually (though, if you find you have to do this and it helps, please speak up so we can fix it)
roryabraham commentedon Mar 12, 2024
I think the time has come to close this long-standing tracking issue. There will always be more improvements to make, but overall the deploy system is in a much better place than it used to be stability-wise. Further improvements and new features would likely better be tracked in their own dedicated, focused issues going forward.