Skip to content

[Tracking] NewDot Deploy Improvements #12021

Closed

Description

@roryabraham

Problem: For the most part, the NewDot deploy system served us well over the last couple years. We've successfully completed over 2,000 staging deploys and hundreds of production deploys. :star-struck: However, experience has revealed a few issues with the current system:

  • It's complex, which makes it harder to follow and diagnose problems
  • It's brittle. It's a complex state machine that doesn't operate well when it gets into an "unexpected state". When it's in an unexpected state, it's difficult to predict what will happen when you take any given action to try and fix it. This has led to us making problems worse, and even doing an accidental production deploy before.
  • It's unpredictable. The lack of idempotency in deploy actions means that the same action, taken twice, can end up with different results.
  • It's weak. Which is to say, when our QA fails and we do ship bugs to customers, we are limited in our options to fix those, and/or otherwise prevent real users from experiencing those bugs.
  • It's slow. The amount of time it takes to handle deploys and push out fixes eats up a lot of deployer time during their week.
  • It's fundamentally flawed. There are a few rare cases in which the underlying git logic is flawed, and this can end up in the wrong code being deployed to staging and/or production.

These problems prevent daily deploys, waste deployer time, and result in a higher probability of NewDot users experiencing bugs.

Solution: Let's engage in a holistic rebuild of the NewDot deploy system to address these issues and focus on improving the simplicity, durability, idempotency, speed, and flexibility of the system, and include this project in the scope of #vip-waq-everywhere.


Known Issues

TODO:

In Progress:

Done:

Activity

  1. self-assigned this
    on Oct 19, 2022
  2. added
    InternalRequires API changes or must be handled by Expensify staff
    on Oct 19, 2022
  3. changed the title [-][Tracking] NewDot Deploy Improvements[/-] [+][HOLD][Tracking] NewDot Deploy Improvements[/+] on Nov 28, 2022
  4. roryabraham commented on Nov 28, 2022

    @roryabraham
    ContributorAuthor

    Putting this on HOLD for WAQ

  5. roryabraham commented on Dec 14, 2022

    @roryabraham
    ContributorAuthor

    I am going to drop this for now with the intention of bringing it back up as a SWM project once they are done with the PR testing project.

  6. removed their assignment
    on Dec 14, 2022
  7. roryabraham commented on Jan 17, 2023

    @roryabraham
    ContributorAuthor

    Another potentially valuable addition here would be to deprecate .github/libs and provide a cleaner way to share code between the main application, tests, and GitHub Actions. Right now we have issues where CJS and ES6 modules don't know how to talk to eachother properly (example)

  8. 71 remaining items

  9. roryabraham commented on Jan 17, 2024

    @roryabraham
    ContributorAuthor

    I think the last idea is a good one. I also think it's a bit of a problem that we can't retry a build on one platform until the builds for all platforms complete.

  10. roryabraham commented on Feb 6, 2024

    @roryabraham
    ContributorAuthor

    No update right now

  11. roryabraham commented on Feb 16, 2024

    @roryabraham
    ContributorAuthor

    Continued exploration in #22821

  12. roryabraham commented on Feb 27, 2024

    @roryabraham
    ContributorAuthor

    Last week I found and fixed some issues with our pod cache that was making our iOS builds unreliable. I'm hoping this will contribute to significantly reduced build failures and hopefully no need to ever go deleting caches manually (though, if you find you have to do this and it helps, please speak up so we can fix it)

  13. roryabraham commented on Mar 12, 2024

    @roryabraham
    ContributorAuthor

    I think the time has come to close this long-standing tracking issue. There will always be more improvements to make, but overall the deploy system is in a much better place than it used to be stability-wise. Further improvements and new features would likely better be tracked in their own dedicated, focused issues going forward.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

EngineeringInternalRequires API changes or must be handled by Expensify staffNewFeatureSomething to build that is a new item.WeeklyKSv2

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions