Repository navigation
Recover the Pusher apply chain after a rejection, behind a watermark continuity guard - #100594
Conversation
…mark continuity guard A rejected Pusher apply left the shared pusherEventsPromise rejected for the rest of the session, so every later Pusher update failed before its handler ran. Heal the chain in a catch while still handing the rejection to the caller. Healing alone would be data loss, since nothing checks that update IDs are contiguous before advancing the watermark, so also refuse to advance past an update whose apply failed until a catch-up response re-delivers that range. Finally, key recovery off the failure itself in applyOnyxUpdatesReliably: an update with previousUpdateID 0 never reaches the gap check, so a failed apply there would otherwise wait for an incidental gap.
96fedf0 to
0917dbb
Compare
# Conflicts: # tests/unit/OnyxUpdateManagerTest.ts
Inject apply failures with real handlers registered via subscribeToMultiEvent instead of spying on PusherUtils.triggerMultiEventHandler, prove same-tick ordering through merged Onyx state instead of a mock call log, split failure propagation from chain recovery, and drop the SequentialQueue.isPaused assertion.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 94f6a82e5c
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
A full ReconnectApp and OpenApp both re-download the whole state, so they cover a failed update range at least as well as an incremental catch-up. Clearing the latch only on catch-up left the watermark pinned forever on a client whose recovery traffic is full reloads, turning every later update into a false gap.
# Conflicts: # src/libs/actions/OnyxUpdates.ts # src/libs/actions/applyOnyxUpdatesReliably.ts # tests/unit/OnyxUpdatesTest.ts
|
@MelvinBot please review this PR |
|
@nyomanjyotisa Please copy/paste the Reviewer Checklist from here into a new comment on this PR and complete it. If you have the K2 extension, you can simply click: [this button] |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a1f0ca0edb
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
The three-part split is the right shape, and rewriting 🔴 The watermark hold is defeated by
|
A second review pass found four ways the hold added in this PR could be walked over or held too long: - Every catch-up range is computed from getEffectiveLastUpdateID, so a WRITE landing after the failure carried it past the failed range and the recovery fetch skipped the hole. Report only the persisted watermark while a failure is outstanding, which also stops a redelivery of the failed update itself from looking old and being discarded. - Releasing the hold only checked where the recovering response ended. Require proof it covered the range: a catch-up must have started at or below the failure, and any other response must carry a previousUpdateID at or below our watermark, which makes its range contiguous over the hole. - Refuse the failed ID itself, not just IDs above it, so a response that merely ends on it cannot pull the watermark onto the hole. - shouldAdvanceLastUpdateID is captured before the apply is awaited, so a failure whose range a reconnect had already covered was recorded anyway, costing a false gap and a redundant refetch. Re-check the live watermark. Also heal airshipEventsPromise the same way as the Pusher chain: the linked issue asks for both, and one rejected Airship apply otherwise stops every later push for the session. The unpause at the end of the OnyxUpdateManager test was dead code: finalizeUpdatesAndResumeQueue already resumes the queue before the awaited promise settles.
GetMissingOnyxMessages is exclusive of updateIDFrom, so a response whose
updateIDFrom equals the failed update carries only the updates after it and can
never re-deliver the one that failed. Release the latch only when the bound sits
strictly below it.
Add coverage that passes a real data.updateIDFrom: one test pinning the hold at
the boundary, one control pinning the legitimate release one ID lower. The
existing release test passed data: {}, so the bound was never exercised.
The hold was a single number and the catch path took a Math.min, so a second failure at a higher ID was not overwritten, it was never recorded: with 20 and 40 failing the guard read 20, the catch-up that legitimately covered 20 cleared it, and a later response walked the watermark past the still-missing 40 with no gap left to report. Keep the set of unrecovered failures and express the release as an explicit covered range (coveredFrom, lastUpdateID], so a response releases only the failures inside the range it provably re-delivered and every other one stays held. A full reconnect and OpenApp cover from 0, a catch-up covers from its updateIDFrom, a response chained onto the watermark covers from its previousUpdateID, and a response that proves nothing releases nothing. That makes this hole and the off-by-one bound from the last round unrepresentable instead of separately guarded, and leaves the release semantics those rounds converged on. Bound the set at CONST.NETWORK.MAX_TRACKED_FAILED_UPDATE_IDS, alongside the stall guard and watchdog constants it interacts with, by forgetting the highest failures into a remembered band: min and max of what the cap dropped, so a response releases that band only when it covers all of it. Below the cap the watermark advances as far as a response genuinely covers; at the cap it holds until one does, which is what the stall guard's escalation then provokes. failedUpdateCount and hasUntrackedFailedUpdateIDs join the existing failure alert to make the cap observable, and a previousUpdateID of the string "0" no longer counts as covering from 0, which now matters because that value releases everything. Document the hold, its release, the cap and the escalation it leans on under Pausing & Data-Gap Sync, which described both neighbouring mechanisms without it. Add the two-failure sequence, the descending and three-failure orderings, a control that a partial recovery still advances the watermark as far as it is genuinely covered, the straddling case where a catch-up starting above a lower failure must leave that failure held with its own control, the bound with an assertion that fails if it forgets the lowest rather than the highest, and the string zero.
…ous advance The set of tracked failure IDs was the machinery a per-ID release needs. Replace it with the single number this file already had, now named for what it has to hold: highestFailedUpdateID, the highest lastUpdateID whose apply rejected while sitting above the persisted watermark. The release asks one question instead of testing whether each failed ID falls inside this response's range: did the response re-deliver everything from the persisted watermark up? A full reconnect and OpenApp do. A catch-up does when its updateIDFrom is at or below the watermark. Anything else does only on a previousUpdateID above zero and at or below the watermark, so the string "0" still proves nothing. While the hold is set, a response that cannot answer yes has its own onyxData applied but leaves the watermark exactly where it was. That contiguity is what makes a set unnecessary. Every advance during a hold starts at or below the watermark, so the watermark can never step over a lower failure, and one response that reaches the highest failure has necessarily re-delivered the others. It also keeps recovery live without a reconnect: while the hold is set getEffectiveLastUpdateID() reports the persisted watermark, so the catch-up that the held watermark provokes always declares an updateIDFrom that releases it. The catch path takes a maximum where it took a minimum. The direction is load-bearing: a minimum is cleared by the very response that recovers the first failure, and the next unchained response then walks the watermark past the one still missing, which is the bug being fixed. Retains-every-failure covers it from both arrival orders. Removing the per-ID release removes what the bound existed to protect. The Set, MAX_TRACKED_FAILED_UPDATE_IDS, the saturation bound, the drop-the-highest rule, the remembered untracked band and the five helpers are gone, one number cannot grow, and the failure alert carries highestFailedUpdateID in place of the two cap fields. Reword the hold under Pausing & Data-Gap Sync for one number and the contiguity rule. Drop the two tests that pinned the cap and the drop rule, which cannot be tested once they do not exist, and turn the test that pinned a release above the watermark into one that pins the hold there and the release from it. The reviewer's four regression tests pass with no edit to them.
…oc's release list Three corrections from the buddy review of the one-number rewrite. None of them asks for a product-code change; the guard already behaves correctly in every case, and each of these closes a boundary the suite let move silently. The release compares the response's lastUpdateID to the held failure with >=, and the whole suite passed when that was mutated to a strict >. Every existing catch-up test used a range head above the failure, so the equality case went untested — and it is the production shape, since GetMissingOnyxMessages asks for a range ending at the update that triggered the fetch. Pin it: a catch-up whose lastUpdateID is exactly the failed update must clear the hold, not just carry the watermark onto it, which is what the following unchained response proves. A catch-up whose updateIDFrom sits strictly between the watermark and the failure was the retuned test's old value and nothing replaced it, so only the two ends were pinned. A fetch from above the watermark proves nothing about the range between, so that case holds the watermark: pin it at 19 against a watermark of 10 and a failure at 40. Many outstanding failures were released by one contiguous response only in a test that died with the cap. Restore that scenario without the cap: 105 failures, an unchained response refused, one catch-up covering through the highest released, normal flow back. Reword the release list in the doc: exclusivity is no longer why a catch-up is refused, the watermark bound is, and the chained branch requires a previousUpdateID above zero, so a response at a zero watermark is not treated as chained.
…p helper Measured before cutting: of the 44 tests this branch adds to OnyxUpdatesTest, 39 fail with the product code reverted to the merge base, so the suite is testing this change rather than decorating it. Two of the five that pass on merge-base code are the pins that must stay anyway. The four regression tests are byte-identical and untouched, and both mutation pins still bite after this: the faithful minimum still fails three reviewer tests, and the release boundary moving from >= to > still fails its one test. Cut one test. A catch-up starting between the watermark and the failure holds the watermark, which the same file already pins three times over: at the failed ID, below the watermark, and from above the watermark via a staged WRITE. Removing it leaves the mutation that loosens the watermark bound caught by those same three. Its content was an interior value of a comparison, not a case a caller can tell apart. Shrink another. It looped 105 failures because the bound it was written against was 100, and that constant no longer exists; the count was an implementation artifact and the refusal half duplicated the three-failure test beside it. It now uses the same three failures and asserts only what nothing else does: one contiguous response covering through the highest releases every one of them, and the unchained response after it advances, so the hold is genuinely gone rather than overtaken. Replace two hand-built catch-up response literals with the helper the file already has. The two boundary tests the last review round asked for were constructing, inline and identically, what catchUpResponse returns; their assertions are unchanged. Test surface added by the branch: 817 lines, now 786. No product code touched.
mountiny
left a comment
There was a problem hiding this comment.
Thanks, changes look good to me
|
🚧 mountiny has triggered a test Expensify/App build. You can view the workflow run here. |
|
✋ This PR was not deployed to staging yet because QA is ongoing. It will be automatically deployed to staging after the next production release. |
|
🧪🧪 Use the links below to test this adhoc build on Android, iOS, and Web. Happy testing! 🧪🧪
|
|
🚀 Deployed to staging by https://github.com/mountiny in version: 9.4.82-0 🚀
|
|
🤖 No help site changes are required for this PR, so I did not create a draft docs PR. Why: every change is internal client plumbing for how Onyx updates recover after a failed apply. Nothing in the diff adds, removes, or renames a user-facing feature, tab, setting, or button, and no copy strings changed. What I checkedChanged files — three source files, six test files, plus
Evidence that nothing is user-facing:
The user-visible effect is that a client which previously went dark after one rejected update now recovers on the next update. That is a reliability improvement to existing documented behavior, not a change to it. Help site articles describe what the product does, not how the client recovers from an internal failure, so there is nothing to update. Since no docs PR was needed, there is no linked help site PR to review or mark |
|
🚀 Deployed to staging by https://github.com/mountiny in version: 9.4.86-0 🚀
|
|
🤖 No help site changes are required. I did not create a draft PR. This PR changes how the client recovers internally when an Onyx update fails to apply. It adds no feature, setting, button, tab, or user-visible behavior, so nothing in What I checkedChanged files, by category:
The one documentation file touched is a contributor guide, not a help site article. The help site covers what customers do in the product; this PR covers how the client heals a rejected update chain behind the scenes. Help site search: no article under Since there are no help site changes, there is no linked docs PR to review. If you think a customer-facing article should call out something here, reply with |
|
🚀 Deployed to staging by https://github.com/mountiny in version: 9.4.89-0 🚀
|
|
🤖 Re-checked: still no help site changes are required. I did not create a draft PR. Nothing has changed since my previous answer — same merge commit ( The only documentation file it touches is Since there are no help site changes, there is no linked docs PR to review.
|
|
🚀 Deployed to production by https://github.com/luacmartins in version: 9.4.89-0 🚀
Bundle Size Analysis (Sentry): |
Explanation of Change
Three changes:
lastUpdateIDAppliedToClientbehind an update whose apply failed, released only when a response proves it covered that range.Before this, one rejected apply left the shared chain rejected for the session, so every later update failed before running. In production logs for 2026-09-01 to 09-07, about 6 users a day (0.011%, 1 in 9,300) hit it. Of the 43 affected that week, 13 went hours without live updates and 12 needed a reload.
Recovery still waits for the next update, so an idle client stays behind until one arrives or it reconnects.
This replaces the safety net that #99774 pinned in
OnyxUpdatesPusherChainFailureTest.ts: a stalled chain no longer stops a follower update from applying, so that test now asserts the watermark held behind the failed update is what keeps the missing range recoverable, which is the same protection without leaving the client dark.Fixed Issues
$ #100174
PROPOSAL: N/A
Tests
This is a resilience fix for a failure that cannot be triggered from the UI, so the behaviour it restores is covered by unit tests in
tests/unit/OnyxUpdatesTest.tsandtests/unit/OnyxUpdateManagerTest.ts, which run on CI. They cover:doesClientNeedToBeUpdatedstill reports the client as behindOpenAppcovering the failed range lets the watermark advance againSmoke test that the normal path did not regress:
[OnyxUpdateManagerError]lines and no unhandled promise rejections appear in the JS console.Offline tests
No offline-specific behaviour changes. The paths touched only run while connected, when Pusher updates or API responses are being applied. Going offline and back online still triggers the usual reconnect and catch-up flow; this PR only affects what happens when an apply of those updates fails.
QA Steps
PR Author Checklist
### Fixed Issuessection aboveTestssectionOffline stepssectionQA stepssectionAvatar, I verified the components usingAvatarare working as expected)StyleUtils.getBackgroundAndBorderStyle(theme.componentBG))npm run compress-svg)Avataris modified, I verified thatAvataris working as expected in all cases)Designlabel and/or tagged@Expensify/designso the design team can review the changes.mainbranch was merged into this PR after a review, I tested again and verified the outcome was still expected according to theTeststeps.Screenshots/Videos
No UI surface: this changes how Onyx update application recovers from a failure. Nothing visual to capture.
Android: Native
Android: mWeb Chrome
iOS: Native
iOS: mWeb Safari
MacOS: Chrome / Safari