Skip to content

Race condition can cause big checklists #27123

Closed

Description

@roryabraham

Problem

Sometimes we get big checklists like this one that include all the PRs from the previous checklist.

The root cause is a race condition between:

Both are triggered at the same time after updateStaging finishes. If deployStaging finishes (i.e: the tag for the new staging version is pushed to the remote) before the checklist is created, then it works. If not, then we get a huge checklist like this will all the PRs that were deployed in the previous version.

Solution

TBD – probably change the deploy system a bit such that staging deploys are triggered by the creation of a prerelease and production deploys by the creation of a "full" release.

That way we can push the tag as soon as it's created without triggering a deploy

Activity

  1. self-assigned this
    on Sep 11, 2023
  2. roryabraham commented on Sep 20, 2023

    @roryabraham
    ContributorAuthor

    No update right now

  3. roryabraham commented on Sep 29, 2023

    @roryabraham
    ContributorAuthor

    No update

  4. mountiny commented on Oct 3, 2023

    @mountiny
    Contributor

    Looking into this one now

    Noting that the createNewDeployChecklist was moved to the platformDeploy.yml, which feels like the race condition is not happening anymore and the bug might be somewhere else.

    When the updateStaging finished, it force pushes to staging branch and then the deploy.yml runs which pushes the tags and finally we trigger the platformDeploy.yml which would create the checklist but the correct tags should be available

    Looking more into why this is failing

  5. mountiny commented on Oct 3, 2023

    @mountiny
    Contributor

    Created a PR which should hopefully help with this

  6. roryabraham commented on Oct 4, 2023

    @roryabraham
    ContributorAuthor

    Ok, it seems like there's some way the git history can change in production that's different from what we have represented in the CIGitLogicTest.sh, just judging by running this command on staging w/ full git history:

    image

    Neither is correct, because the only version-bump that we expect to be included between 1.3.76-6 and 1.3.77-0 is Update version to 1.3.77-0

  7. 20 remaining items

  8. roryabraham commented on Nov 18, 2023

    @roryabraham
    ContributorAuthor

    This should be fixed by #31458. I hope so 😅

  9. roryabraham commented on Nov 18, 2023

    @roryabraham
    ContributorAuthor
  10. added
    DeployBlockerCashThis issue or pull request should block deployment
    and removed
    DeployBlockerCashThis issue or pull request should block deployment
    on Nov 18, 2023
  11. github-actions commented on Nov 18, 2023

    @github-actions
    Contributor

    👋 Friendly reminder that deploy blockers are time-sensitive ⏱ issues! Check out the open `StagingDeployCash` deploy checklist to see the list of PRs included in this release, then work quickly to do one of the following:

    1. Identify the pull request that introduced this issue and revert it.
    2. Find someone who can quickly fix the issue.
    3. Fix the issue yourself.
  12. melvin-bot commented on Nov 18, 2023

    @melvin-bot

    📣 @github-actions[bot]! 📣
    Hey, it seems we don’t have your contributor details yet! You'll only have to do this once, and this is how we'll hire you on Upwork.
    Please follow these steps:

    1. Make sure you've read and understood the contributing guidelines.
    2. Get the email address used to login to your Expensify account. If you don't already have an Expensify account, create one here. If you have multiple accounts (e.g. one for testing), please use your main account email.
    3. Get the link to your Upwork profile. It's necessary because we only pay via Upwork. You can access it by logging in, and then clicking on your name. It'll look like this. If you don't already have an account, sign up for one here.
    4. Copy the format below and paste it in a comment on this issue. Replace the placeholder text with your actual details.
      Screen Shot 2022-11-16 at 4 42 54 PM
      Format:
    Contributor details
    Your Expensify account email: <REPLACE EMAIL HERE>
    Upwork Profile Link: <REPLACE LINK HERE>
    
  13. roryabraham commented on Nov 18, 2023

    @roryabraham
    ContributorAuthor

    Nice, that worked as expected (well, maybe not the upwork bit, that's weird).

    Anyways, for context I'm marking this as a deploy blocker not because we actually need to block deploy on it, but because I think we should merge #31458 before the next deploy, because it should fix this big checklist issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions