Skip to content

[$175] Web - Queued ApproveMoneyRequest never reaches the server after a SequentialQueue pause, but the Approve button disappears #105121

Closed

Description

@melvin-bot

Problem

A Copilot approved an expense report on web. The app applied the optimistic approval and hid the Approve button. The ApproveMoneyRequest write never reached the server, and no error came back. The report stayed in Processing on the server. Later, a different Copilot opened the same report, saw Approve, and approved it successfully.

What the client logs show

  1. The SequentialQueue logged "Queue already paused" repeatedly for about 2 minutes. Then "Pause watchdog fired — queue stuck paused with no progress, recovering" fired and flushed 2 persisted requests.
  2. About 6 minutes later, the user opened the report (OpenReport succeeded) and clicked Approve.
  3. The app pushed ApproveMoneyRequest to the SequentialQueue with currentQueueLength: 2, while isOffline was not set. It persisted the request (queue length 3) and then logged "Queue is not running. Flushing the queue."
  4. No "Finished API request" line ever appeared for that request. The server has no record of it.
  5. Other writes in the same session succeeded, including many ApproveMoneyRequest calls on other reports. The network was up.

Expected result

The app sends the queued approval. If the request fails, the app rolls back the optimistic data and shows an error so the user can retry.

Actual result

The request is silently stuck or dropped. The UI shows the report as approved, so the Approve button is gone, but the server never approved it. The user had to hold the report and ask someone else to approve it.

Related

#97331 added the pause watchdog. Here the watchdog fired, but a request queued after it still never went out.

Platform: Web (Chrome, macOS)

Reported by @greg-schroeder via DM with MelvinBot.

Upwork Automation - Do Not Edit

Activity

  1. added
    BugSomething is broken. Auto assigns a BugZero manager.
    on Oct 27, 2026
  2. MelvinBot commented on Oct 27, 2026

    @MelvinBot
    Contributor

    Proposal

    What is the root cause of that problem?

    Most likely, the tab that approved was not the "leader" tab. Only the leader tab sends queued writes. On any other tab, flush() quietly returns early, and the request waits for the leader to send it. In this case the leader was a tab that was dead or frozen, so the request never went out.

    How this fits each log line
    • "Queue already paused" for 2 minutes: a follower tab still pauses on Onyx update gaps. But handleMissingOnyxUpdates returns early on followers, so nothing fetches the missing updates or unpauses. Each Pusher update calls pause() again.
    • Watchdog fired, "flushed 2 requests": the watchdog's escalation is skipped on followers. unpause() then calls flush(false), which returns at the leader check. The "2" is numberOfPersistedRequests on the "Unpausing the queue" log, not a count of requests sent.
    • OpenReport worked, Approve didn't: reads go through MainQueue, which any tab can use. Writes go through SequentialQueue, which only the leader processes.
    • Earlier approvals worked: the tab was leader earlier. Leadership moves whenever another tab mounts or becomes visible (Expensify.tsx:196-221). Nothing calls flush() when a tab becomes leader again.
    • No rollback, no error: the request never failed. It is still sitting in PERSISTED_REQUESTS, so the optimistic data stays.

    Other causes considered

    • Queue paused again after the watchdog. Less likely. Every new pause() re-arms the watchdog, so a 6-minute pause should log more than one "Pause watchdog fired" alert.
    • First queued request hung inside the network middleware. Unlikely. That would leave isSequentialQueueRunning true, but push() logged that the queue was not running.

    Known history: [No QA] Fix leader-election deadlock in ActiveClientManager (SequentialQueue unable to flush) targeted this same deadlock. It was reverted in [CP Staging] Revert "[No QA] Fix leader-election deadlock in ActiveClientManager (SequentialQueue unable to flush)" because several tabs believed they were leader and sent the same requests more than once.

    To confirm from the session logs, find the line logged right after "Queue is not running. Flushing the queue." for ApproveMoneyRequest:

    • "Unable to flush. Client is not the leader." confirms this root cause.
    • "Unable to flush. Queue is paused." points to the re-pause cause instead.
    • Also look for "[OnyxUpdateManager] Pause watchdog escalation skipped — not the leader client" near the watchdog alert.

    What changes do you think we should make in order to solve the problem?

    1. Make leadership exclusive and self-healing. Use the Web Locks API (navigator.locks.request) for the sequential-queue leader on web. The browser releases a lock when its tab dies. Only one tab can hold it, which avoids the duplicate-send problem that got the earlier fix reverted. Other platforms keep the current single-client behavior.
    2. Flush when a tab becomes leader. When isClientTheLeader() changes from false to true, call SequentialQueue.flush(). This drains requests that other tabs left behind.
    3. Stop hiding stuck requests. Add a check that fires when persisted requests exist and the queue is online, not paused, and not running for more than N seconds. It should raise a Log.alert, re-run ActiveClientManager.init(), and flush. Also log the real reason in push() instead of always logging "Flushing the queue."

    What alternative solutions did you explore? (Optional)

    Use a heartbeat with expiry: each tab writes a timestamp to ACTIVE_CLIENTS, and stale entries lose leadership. This works on every platform, but it can briefly produce two leaders. That risk is the reason the earlier fix was reverted.


    Confidence: medium. The code fully explains the log sequence, but I could not see this session's logs to confirm the "Client is not the leader" line.

    Next Steps for Contributor+ team:
    To accept: @MelvinBot implement [this](https://github.com/Expensify/App/issues/105121) to create a draft PR.
    To refine: @MelvinBot <your feedback>
    To reject: Explain why you are rejecting Melvin's proposal.


    view run

  3. added
    ExternalAdded to denote the issue can be worked on by a contributor
    on Oct 27, 2026
  4. melvin-bot commented on Oct 27, 2026

    @melvin-bot
    Author

    Triggered auto assignment to Contributor-plus team member for initial proposal review - @situchan (External)

  5. changed the title [-]Web - Queued ApproveMoneyRequest never reaches the server after a SequentialQueue pause, but the Approve button disappears[/-] [+][$175] Web - Queued ApproveMoneyRequest never reaches the server after a SequentialQueue pause, but the Approve button disappears[/+] on Oct 27, 2026
  6. melvin-bot commented on Oct 27, 2026

    @melvin-bot
    Author
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

BugSomething is broken. Auto assigns a BugZero manager.DailyKSv2ExternalAdded to denote the issue can be worked on by a contributor

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions