Skip to content

[Due for payment 2026-07-21] fix: escalate to an incremental ReconnectApp when the missing-updates fetch stops advancing the client #95486

Closed

Description

@mountiny

Problem

When the app spots a gap in the server's update IDs it fetches the missing range with GetMissingOnyxMessages. Production traces show a failure mode where the server keeps answering that fetch with a 200 that carries no usable updates. The client never advances, so every incoming Pusher update re-detects the same gap and fires the same fetch again, about once per second (traces show 12 to 47 requests in a row). The earlier fix in #93948 only stopped fetches that overlap in time; it cannot stop this loop because each fetch finishes before the next one starts.

Solution

This PR treats one useless answer as proof instead of retrying. A successful fetch must advance the client past the update ID it was fired from, checked via response.lastUpdateID so it does not race the Onyx write. The first 200 that fails this check escalates to a single incremental ReconnectApp from the client's current update ID (deduplicated by the SequentialQueue). Until the client advances, further fetches from the same client state are skipped while the SequentialQueue still resumes each cycle. Both the gap branch and the pending-updates branch of OnyxUpdateManager share this guard, and a Log.alert captures client and response update IDs for monitoring.

PR

#95477

Issue OwnerCurrent Issue Owner: @mallenexpensify

Activity

  1. added
    ReviewingHas a PR in review
    InternalRequires API changes or must be handled by Expensify staff
    BugSomething is broken. Auto assigns a BugZero manager.
    Skip PaymentSkips all the payment process steps for an issue. Use it for fixed-price projects.
    on Jul 7, 2026
  2. MelvinBot commented on Jul 7, 2026

    @MelvinBot
    Contributor

    Issue Analysis

    Root cause (confirmed, high confidence): In the current main code, nothing forces the client's lastUpdateIDAppliedToClient watermark to advance after a GetMissingOnyxMessages (GMOM) fetch returns 200. The watermark only advances when OnyxUpdates.apply sees a response.lastUpdateID that is truthy and strictly greater than the current value. So a 200 that carries no usable updates (empty onyxData, or lastUpdateID missing/<= the fired-from ID) leaves the gap intact. After the fetch settles, finalizeUpdatesAndResumeQueue clears all three guards, so the next Pusher update re-detects the same gap and re-fires GMOM — a self-perpetuating serial loop (~1/sec, matching the reported 12–47 requests in a row).

    The earlier fix in #93948 is correctly characterized as insufficient: it added the in-flight getMissingOnyxUpdatesQueryPromise() check to the gap branch guard, which only coalesces time-overlapping fetches. That promise is null once each fetch finalizes, so it provides zero protection against a serial loop where each fetch completes before the next one begins.

    PR #95477 — assessment: The approach is sound and targets the exact gap above. It treats one non-advancing 200 as proof (rather than retrying): escalateIfFetchStalled checks response.lastUpdateID against the fired-from ID (reading the response directly, so it doesn't race the Onyx write), escalates once to an incremental ReconnectApp deduped by the SequentialQueue, and skips further fetches from the same client state until the client actually advances. Both the gap branch and the pending-updates branch share the guard, and a Log.alert surfaces stalled clients for monitoring.

    Investigation details & evidence

    Trigger path. OnyxUpdateManager subscribes to ONYX_UPDATES_FROM_SERVER and runs handleMissingOnyxUpdates per update. The gap branch fires GMOM whenever a server update implies the client is behind, unless one of three guards short-circuits: areDeferredUpdatesQueued, isFetchingForPendingUpdates, or an in-flight getMissingOnyxUpdatesQueryPromise().

    Why the watermark can stall. GET_MISSING_ONYX_MESSAGES is in requestsToIgnoreLastUpdateID in SaveResponseInOnyx, so the response is applied directly. In OnyxUpdates.apply, didAdvanceLastUpdateID is only true for a real, higher lastUpdateID; otherwise the watermark holds and applyHTTPSOnyxUpdates applies an empty onyxData (no-op).

    Why it loops. After settling, .finally(finalizeUpdatesAndResumeQueue) resolves/recreates the query promise, clears deferred updates, and resets isFetchingForPendingUpdates. With the watermark unchanged, the next Pusher update re-enters the gap branch with every guard clear → same gap, same fetch. validateAndApplyDeferredUpdates's repeated-params abort only protects the recursive in-fetch loop for a single invocation, not across separate Pusher-driven invocations.

    Related: #92541 is the parent performance issue (GMOM/ReconnectApp/OpenApp stampedes) that PR #93948 addressed; this issue is the follow-up serial-loop variant that #93948 could not cover.

    Open question (backend, outside App): The exact server-side trigger — whether GetMissingOnyxMessages genuinely returns 200 with lastUpdateID <= updateIDFrom / empty onyxData for the requested range — lives in Auth/Web-Expensify and can't be confirmed from the App repo. It isn't needed to confirm the client-side loop is real, but the PR's Log.alert (client + response update IDs) is the right hook to feed that separate backend investigation. Worth confirming from production logs that consecutive GMOM requests repeat the same updateIDFrom at ~1/sec.


    This issue is authored by mountiny with PR #95477 already in review, so the fix is in flight — this is a validation of the root cause and approach rather than a new proposal. Reply with @MelvinBot <feedback> to refine.

  3. removed
    Skip PaymentSkips all the payment process steps for an issue. Use it for fixed-price projects.
    on Jul 7, 2026
  4. added and removed
    ReviewingHas a PR in review
    on Jul 7, 2026
  5. 2 remaining items

  6. changed the title [-]fix: escalate to an incremental ReconnectApp when the missing-updates fetch stops advancing the client[/-] [+][Due for payment 2026-07-21] fix: escalate to an incremental ReconnectApp when the missing-updates fetch stops advancing the client[/+] on Jul 14, 2026
  7. melvin-bot commented on Jul 14, 2026

    @melvin-bot

    Reviewing label has been removed, please complete the "BugZero Checklist".

  8. melvin-bot commented on Jul 14, 2026

    @melvin-bot

    The solution for this issue has been 🚀 deployed to production 🚀 in version 9.4.33-8 and is now subject to a 7-day regression period 📆. Here is the list of pull requests that resolve this issue:

    If no regressions arise, payment will be issued on 2026-07-21. 🎊

    For reference, here are some details about the assignees on this issue:

  9. melvin-bot commented on Jul 14, 2026

    @melvin-bot

    @mountiny @adhorodyski The PR fixing this issue has been merged! The following checklist (instructions) will need to be completed before the issue can be closed. Please copy/paste the BugZero Checklist from here into a new comment on this GH and complete it. If you have the K2 extension, you can simply click: [this button]. If no checklist is needed for this issue, you can click: [no checklist button]

  10. melvin-bot commented on Jul 17, 2026

    @melvin-bot

    @adhorodyski Whoops! This issue is 2 days overdue. Let's get this updated quick!

  11. melvin-bot commented on Jul 21, 2026

    @melvin-bot

    Triggered auto assignment to @mallenexpensify (Awaiting Payment)

  12. melvin-bot commented on Jul 21, 2026

    @melvin-bot

    Payment Summary

    Upwork Job

    BugZero Checklist (@mallenexpensify)

    • I have verified the correct assignees and roles are listed above and updated the necessary manual offers
    • I have verified that there are no duplicate or incorrect contracts on Upwork for this job (https://www.upwork.com/ab/applicants//hired)
    • I have verified the PR was not reverted
    • I have applied any discounts due to bugs/regressions introduced by this PR
    • I have paid out the Upwork contracts or cancelled the ones that are incorrect
    • I have verified the payment summary above is correct
  13. mallenexpensify commented on Jul 23, 2026

    @mallenexpensify
    Contributor

    Payment Summary

    Contributor+: @mkhutornyi due $250 via NewDot

    ^ Test case created. Thx

  14. trjExpensify commented on Aug 3, 2026

    @trjExpensify
    Contributor

    approved $250 for @mkhutornyi

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

Awaiting PaymentAuto-added when associated PR is deployed to productionBugSomething is broken. Auto assigns a BugZero manager.DailyKSv2EngineeringInternalRequires API changes or must be handled by Expensify staffPerformance

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions