You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Due for payment 2026-07-21] fix: escalate to an incremental ReconnectApp when the missing-updates fetch stops advancing the client #95486
When the app spots a gap in the server's update IDs it fetches the missing range with GetMissingOnyxMessages. Production traces show a failure mode where the server keeps answering that fetch with a 200 that carries no usable updates. The client never advances, so every incoming Pusher update re-detects the same gap and fires the same fetch again, about once per second (traces show 12 to 47 requests in a row). The earlier fix in #93948 only stopped fetches that overlap in time; it cannot stop this loop because each fetch finishes before the next one starts.
Solution
This PR treats one useless answer as proof instead of retrying. A successful fetch must advance the client past the update ID it was fired from, checked via response.lastUpdateID so it does not race the Onyx write. The first 200 that fails this check escalates to a single incremental ReconnectApp from the client's current update ID (deduplicated by the SequentialQueue). Until the client advances, further fetches from the same client state are skipped while the SequentialQueue still resumes each cycle. Both the gap branch and the pending-updates branch of OnyxUpdateManager share this guard, and a Log.alert captures client and response update IDs for monitoring.
Root cause (confirmed, high confidence): In the current main code, nothing forces the client's lastUpdateIDAppliedToClient watermark to advance after a GetMissingOnyxMessages (GMOM) fetch returns 200. The watermark only advances when OnyxUpdates.apply sees a response.lastUpdateID that is truthy and strictly greater than the current value. So a 200 that carries no usable updates (empty onyxData, or lastUpdateID missing/<= the fired-from ID) leaves the gap intact. After the fetch settles, finalizeUpdatesAndResumeQueue clears all three guards, so the next Pusher update re-detects the same gap and re-fires GMOM — a self-perpetuating serial loop (~1/sec, matching the reported 12–47 requests in a row).
The earlier fix in #93948 is correctly characterized as insufficient: it added the in-flight getMissingOnyxUpdatesQueryPromise() check to the gap branch guard, which only coalesces time-overlapping fetches. That promise is null once each fetch finalizes, so it provides zero protection against a serial loop where each fetch completes before the next one begins.
PR #95477 — assessment: The approach is sound and targets the exact gap above. It treats one non-advancing 200 as proof (rather than retrying): escalateIfFetchStalled checks response.lastUpdateID against the fired-from ID (reading the response directly, so it doesn't race the Onyx write), escalates once to an incremental ReconnectApp deduped by the SequentialQueue, and skips further fetches from the same client state until the client actually advances. Both the gap branch and the pending-updates branch share the guard, and a Log.alert surfaces stalled clients for monitoring.
Investigation details & evidence
Trigger path.OnyxUpdateManager subscribes to ONYX_UPDATES_FROM_SERVER and runs handleMissingOnyxUpdates per update. The gap branch fires GMOM whenever a server update implies the client is behind, unless one of three guards short-circuits: areDeferredUpdatesQueued, isFetchingForPendingUpdates, or an in-flight getMissingOnyxUpdatesQueryPromise().
Why the watermark can stall.GET_MISSING_ONYX_MESSAGES is in requestsToIgnoreLastUpdateID in SaveResponseInOnyx, so the response is applied directly. In OnyxUpdates.apply, didAdvanceLastUpdateID is only true for a real, higher lastUpdateID; otherwise the watermark holds and applyHTTPSOnyxUpdates applies an empty onyxData (no-op).
Why it loops. After settling, .finally(finalizeUpdatesAndResumeQueue) resolves/recreates the query promise, clears deferred updates, and resets isFetchingForPendingUpdates. With the watermark unchanged, the next Pusher update re-enters the gap branch with every guard clear → same gap, same fetch. validateAndApplyDeferredUpdates's repeated-params abort only protects the recursive in-fetch loop for a single invocation, not across separate Pusher-driven invocations.
Related:#92541 is the parent performance issue (GMOM/ReconnectApp/OpenApp stampedes) that PR #93948 addressed; this issue is the follow-up serial-loop variant that #93948 could not cover.
Open question (backend, outside App): The exact server-side trigger — whether GetMissingOnyxMessages genuinely returns 200 with lastUpdateID <= updateIDFrom / empty onyxData for the requested range — lives in Auth/Web-Expensify and can't be confirmed from the App repo. It isn't needed to confirm the client-side loop is real, but the PR's Log.alert (client + response update IDs) is the right hook to feed that separate backend investigation. Worth confirming from production logs that consecutive GMOM requests repeat the same updateIDFrom at ~1/sec.
This issue is authored by mountiny with PR #95477 already in review, so the fix is in flight — this is a validation of the root cause and approach rather than a new proposal. Reply with @MelvinBot <feedback> to refine.
changed the title [-]fix: escalate to an incremental ReconnectApp when the missing-updates fetch stops advancing the client[/-][+][Due for payment 2026-07-21] fix: escalate to an incremental ReconnectApp when the missing-updates fetch stops advancing the client[/+]on Jul 14, 2026
The solution for this issue has been 🚀 deployed to production 🚀 in version 9.4.33-8 and is now subject to a 7-day regression period 📆. Here is the list of pull requests that resolve this issue:
@mountiny@adhorodyski The PR fixing this issue has been merged! The following checklist (instructions) will need to be completed before the issue can be closed. Please copy/paste the BugZero Checklist from here into a new comment on this GH and complete it. If you have the K2 extension, you can simply click: [this button]. If no checklist is needed for this issue, you can click: [no checklist button]
Activity
MelvinBot commentedon Jul 7, 2026
Issue Analysis
Root cause (confirmed, high confidence): In the current
maincode, nothing forces the client'slastUpdateIDAppliedToClientwatermark to advance after aGetMissingOnyxMessages(GMOM) fetch returns200. The watermark only advances whenOnyxUpdates.applysees aresponse.lastUpdateIDthat is truthy and strictly greater than the current value. So a200that carries no usable updates (emptyonyxData, orlastUpdateIDmissing/<=the fired-from ID) leaves the gap intact. After the fetch settles,finalizeUpdatesAndResumeQueueclears all three guards, so the next Pusher update re-detects the same gap and re-fires GMOM — a self-perpetuating serial loop (~1/sec, matching the reported 12–47 requests in a row).The earlier fix in #93948 is correctly characterized as insufficient: it added the in-flight
getMissingOnyxUpdatesQueryPromise()check to the gap branch guard, which only coalesces time-overlapping fetches. That promise isnullonce each fetch finalizes, so it provides zero protection against a serial loop where each fetch completes before the next one begins.PR #95477 — assessment: The approach is sound and targets the exact gap above. It treats one non-advancing
200as proof (rather than retrying):escalateIfFetchStalledchecksresponse.lastUpdateIDagainst the fired-from ID (reading the response directly, so it doesn't race the Onyx write), escalates once to an incrementalReconnectAppdeduped by the SequentialQueue, and skips further fetches from the same client state until the client actually advances. Both the gap branch and the pending-updates branch share the guard, and aLog.alertsurfaces stalled clients for monitoring.Investigation details & evidence
Trigger path.
OnyxUpdateManagersubscribes toONYX_UPDATES_FROM_SERVERand runshandleMissingOnyxUpdatesper update. The gap branch fires GMOM whenever a server update implies the client is behind, unless one of three guards short-circuits:areDeferredUpdatesQueued,isFetchingForPendingUpdates, or an in-flightgetMissingOnyxUpdatesQueryPromise().Why the watermark can stall.
GET_MISSING_ONYX_MESSAGESis inrequestsToIgnoreLastUpdateIDinSaveResponseInOnyx, so the response is applied directly. InOnyxUpdates.apply,didAdvanceLastUpdateIDis only true for a real, higherlastUpdateID; otherwise the watermark holds andapplyHTTPSOnyxUpdatesapplies an emptyonyxData(no-op).Why it loops. After settling,
.finally(finalizeUpdatesAndResumeQueue)resolves/recreates the query promise, clears deferred updates, and resetsisFetchingForPendingUpdates. With the watermark unchanged, the next Pusher update re-enters the gap branch with every guard clear → same gap, same fetch.validateAndApplyDeferredUpdates's repeated-params abort only protects the recursive in-fetch loop for a single invocation, not across separate Pusher-driven invocations.Related: #92541 is the parent performance issue (GMOM/ReconnectApp/OpenApp stampedes) that PR #93948 addressed; this issue is the follow-up serial-loop variant that #93948 could not cover.
Open question (backend, outside App): The exact server-side trigger — whether
GetMissingOnyxMessagesgenuinely returns200withlastUpdateID <= updateIDFrom/ emptyonyxDatafor the requested range — lives in Auth/Web-Expensify and can't be confirmed from the App repo. It isn't needed to confirm the client-side loop is real, but the PR'sLog.alert(client + response update IDs) is the right hook to feed that separate backend investigation. Worth confirming from production logs that consecutive GMOM requests repeat the sameupdateIDFromat ~1/sec.This issue is authored by
mountinywith PR #95477 already in review, so the fix is in flight — this is a validation of the root cause and approach rather than a new proposal. Reply with@MelvinBot <feedback>to refine.2 remaining items
[-]fix: escalate to an incremental ReconnectApp when the missing-updates fetch stops advancing the client[/-][+][Due for payment 2026-07-21] fix: escalate to an incremental ReconnectApp when the missing-updates fetch stops advancing the client[/+]melvin-bot commentedon Jul 14, 2026
Reviewinglabel has been removed, please complete the "BugZero Checklist".melvin-bot commentedon Jul 14, 2026
The solution for this issue has been 🚀 deployed to production 🚀 in version 9.4.33-8 and is now subject to a 7-day regression period 📆. Here is the list of pull requests that resolve this issue:
If no regressions arise, payment will be issued on 2026-07-21. 🎊
For reference, here are some details about the assignees on this issue:
melvin-bot commentedon Jul 14, 2026
@mountiny @adhorodyski The PR fixing this issue has been merged! The following checklist (instructions) will need to be completed before the issue can be closed. Please copy/paste the BugZero Checklist from here into a new comment on this GH and complete it. If you have the K2 extension, you can simply click: [this button]. If no checklist is needed for this issue, you can click: [no checklist button]
melvin-bot commentedon Jul 17, 2026
@adhorodyski Whoops! This issue is 2 days overdue. Let's get this updated quick!
melvin-bot commentedon Jul 21, 2026
Triggered auto assignment to @mallenexpensify (
Awaiting Payment)melvin-bot commentedon Jul 21, 2026
Payment Summary
Upwork Job
BugZero Checklist (@mallenexpensify)
mallenexpensify commentedon Jul 23, 2026
Payment Summary
Contributor+: @mkhutornyi due $250 via NewDot
^ Test case created. Thx
trjExpensify commentedon Aug 3, 2026
approved $250 for @mkhutornyi