Skip to content

[Due for payment]$125 Flaky Tests - GroupChatNameTests #65357

Closed

Description

@dangrous

Problem

Multiple tests/ui/GroupChatNameTests.tsx tests are flakey on main. Here's a run that failed and then passed the next time: https://github.com/Expensify/App/actions/runs/15983163235/job/45081956645 . There were three fails:

  • ● Tests for group chat name › Should show exact name in header when report name is available with 4 participants
  • ● Tests for group chat name › Should show exact name in header when report name is available with 8 participants
  • ● Tests for group chat name › Should show last message preview in LHN

Solution

Fix the flakiness.

These have been flaky for a while and have been worked on before, but we're still having issues. Let's figure it out!

Previous Issues:

Activity

  1. added
    ExternalAdded to denote the issue can be worked on by a contributor
    Help WantedApply this label when an issue is open to proposals by contributors
    BugSomething is broken. Auto assigns a BugZero manager.
    on Jul 2, 2025
  2. melvin-bot commented on Jul 2, 2025

    @melvin-bot

    Triggered auto assignment to Contributor-plus team member for initial proposal review - @QichenZhu (External)

  3. melvin-bot commented on Jul 2, 2025

    @melvin-bot

    Triggered auto assignment to @Christinadobrzyn (Bug), see https://stackoverflow.com/c/expensify/questions/14418 for more details. Please add this bug to a GH project, as outlined in the SO.

  4. nkdengineer commented on Jul 3, 2025

    @nkdengineer
    Contributor

    Proposal

    Please re-state the problem that we are trying to solve in this issue.

    What is the root cause of that problem?

    The problem is our mock SignInUser API returns an error fetch is not defined so the app doesn't render the sidebar

    Image Image

    return fetch(url, {

    The reason is that the mock was reverted before SignInUser API is called.

    function signInWithTestUser(accountID = 1, login = 'test@user.com', password = 'Password1', authToken = 'asdfqwerty', firstName = 'Test') {
    const originalXhr = HttpUtils.xhr;
    HttpUtils.xhr = jest.fn().mockImplementation(() => {
    const mockedResponse: OnyxResponse = {
    onyxData: [
    {
    onyxMethod: Onyx.METHOD.MERGE,
    key: ONYXKEYS.CREDENTIALS,
    value: {
    login,
    },
    },
    {
    onyxMethod: Onyx.METHOD.MERGE,
    key: ONYXKEYS.ACCOUNT,
    value: {
    validated: true,
    },
    },
    {
    onyxMethod: Onyx.METHOD.MERGE,
    key: ONYXKEYS.PERSONAL_DETAILS_LIST,
    value: {
    [accountID]: buildPersonalDetails(login, accountID, firstName),
    },
    },
    ],
    jsonCode: 200,
    };
    // Return a Promise that resolves with the mocked response
    return Promise.resolve(mockedResponse);
    });
    // Simulate user entering their login and populating the credentials.login
    Session.beginSignIn(login);
    return waitForBatchedUpdates()
    .then(() => {
    HttpUtils.xhr = jest.fn().mockImplementation(() => {
    const mockedResponse: OnyxResponse = {
    onyxData: [
    {
    onyxMethod: Onyx.METHOD.MERGE,
    key: ONYXKEYS.SESSION,
    value: {
    authToken,
    accountID,
    email: login,
    encryptedAuthToken: authToken,
    },
    },
    {
    onyxMethod: Onyx.METHOD.MERGE,
    key: ONYXKEYS.CREDENTIALS,
    value: {
    autoGeneratedLogin: Str.guid('expensify.cash-'),
    autoGeneratedPassword: Str.guid(),
    },
    },
    {
    onyxMethod: Onyx.METHOD.MERGE,
    key: ONYXKEYS.ACCOUNT,
    value: {
    isUsingExpensifyCard: false,
    },
    },
    {
    onyxMethod: Onyx.METHOD.MERGE,
    key: ONYXKEYS.BETAS,
    value: ['all'],
    },
    {
    onyxMethod: Onyx.METHOD.MERGE,
    key: ONYXKEYS.NVP_PRIVATE_PUSH_NOTIFICATION_ID,
    value: 'randomID',
    },
    ],
    jsonCode: 200,
    };
    // Return a Promise that resolves with the mocked response
    return Promise.resolve(mockedResponse);
    });
    Session.signIn(password);
    return waitForBatchedUpdates();
    })
    .then(() => {
    HttpUtils.xhr = originalXhr;
    });
    }

    The problem comes from the previous test. We revert the mock after calling SignInUser API, which causes other APIs on the render App to fail because fetch is not defined. And the retry logic will call the API a few times. Then, when we go to the next test, the SignInUser is added to the queue and it can be called after the mock is reverted since it's the last API in the queue.

    What changes do you think we should make in order to solve the problem?

    We should set the fetch with our mock fetch to ensure this is always available. Use the fetch mock to ensure all APIs are success, then we don't have the problem SignInUser is called after we revert the mock.

    beforeEach(() => {
        jest.clearAllMocks();
    
        global.fetch = TestHelper.getGlobalFetchMock();
        // Unsubscribe to pusher channels
        PusherHelper.teardown();
    
        return Onyx.clear().then(waitForBatchedUpdates);
    });
    

    beforeEach(() => {

    What specific scenarios should we cover in automated tests to prevent reintroducing this issue in the future?

    None

    What alternative solutions did you explore? (Optional)

    Reminder: Please use plain English, be brief and avoid jargon. Feel free to use images, charts or pseudo-code if necessary. Do not post large multi-line diffs or write walls of text. Do not create PRs unless you have been hired for this job.

  5. QichenZhu commented on Jul 3, 2025

    @QichenZhu
    Contributor

    I think the root cause of #59729 isn't clear, and the test is still flaky. So we need to dig deeper to find the real root cause and explore other solutions.

    Image
  6. nkdengineer commented on Jul 3, 2025

    @nkdengineer
    Contributor

    @QichenZhu How can you reproduce it locally?

  7. QichenZhu commented on Jul 3, 2025

    @QichenZhu
    Contributor

    @nkdengineer I ran the command below repeatedly, and it failed on the 4th run:

    TZ=utc NODE_OPTIONS=--experimental-vm-modules npx jest tests/ui/GroupChatNameTests.tsx --no-clear
  8. nkdengineer commented on Jul 3, 2025

    @nkdengineer
    Contributor

    @QichenZhu Thanks, I can reproduce it locally now. Updated proposal

  9. QichenZhu commented on Jul 4, 2025

    @QichenZhu
    Contributor

    Thanks for the proposal, @nkdengineer! It looks promising and seems to fix the flaky tests.

    Could you explain more about the root cause? Without your fix, I see the "fetch is not defined" message even when the test passes. So how could that cause this issue?

  10. nkdengineer commented on Jul 4, 2025

    @nkdengineer
    Contributor

    @QichenZhu Do you mean you see this message when all tests are passed or only some tests are passed.

  11. 29 remaining items

  12. Christinadobrzyn commented on Jul 14, 2025

    @Christinadobrzyn
    Contributor

    PR = #66032 is merged

    @nkdengineer are we waiting on others or can this follow our payment process? Is this $50 too?

  13. nkdengineer commented on Jul 19, 2025

    @nkdengineer
    Contributor

    @nkdengineer are we waiting on others or can this follow our payment process?

    @Christinadobrzyn No we don't need to wait on others issues. It's not related

    Is this $50 too?

    I think the price is the base price $250. Is that right @dangrous ?

  14. dangrous commented on Jul 22, 2025

    @dangrous
    ContributorAuthor

    So it looks like the recent benchmark for flaky tests is $125, let's use that

  15. changed the title [-]Flaky Tests - GroupChatNameTests[/-] [+][Due for payment]Flaky Tests - GroupChatNameTests[/+] on Jul 23, 2025
  16. Christinadobrzyn commented on Jul 23, 2025

    @Christinadobrzyn
    Contributor

    Payment Summary

    Contributor: @nkdengineer paid $125 via Upwork (offer - https://www.upwork.com/nx/wm/offer/108158810)
    Contributor+: @QichenZhu due $125 via Upwork (offer - https://www.upwork.com/nx/wm/offer/108158817)

    Is this correct? @QichenZhu can you please provide your Upwork if the above it true?

  17. QichenZhu commented on Jul 23, 2025

    @QichenZhu
    Contributor

    Here's my upwork profile. Thanks!

  18. nkdengineer commented on Jul 23, 2025

    @nkdengineer
    Contributor

    Contributor: @nkdengineer paid $125 via NewDot

    @Christinadobrzyn, I should be paid via Upwork (my profile is https://www.upwork.com/freelancers/~0134d0b74faef1a5ac). Please help to check again and correct me if I missed something.

  19. changed the title [-][Due for payment]Flaky Tests - GroupChatNameTests[/-] [+][Due for payment]$125 Flaky Tests - GroupChatNameTests[/+] on Jul 23, 2025
  20. Christinadobrzyn commented on Jul 23, 2025

    @Christinadobrzyn
    Contributor

    Thanks @QichenZhu and @nkdengineer - can you please accept me Upwork offer here - #65357 (comment)

    Thanks!

  21. Christinadobrzyn commented on Jul 24, 2025

    @Christinadobrzyn
    Contributor

    Thanks @QichenZhu and @nkdengineer you both have been paid in Upwork - summary here - #65357 (comment)

    Closing!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

BugSomething is broken. Auto assigns a BugZero manager.DailyKSv2ExternalAdded to denote the issue can be worked on by a contributorReviewingHas a PR in review

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions