Skip to content

[Due for payment 2026-08-25] [$500] NewDot web tab crashes silently (renderer OOM kill) under high memory / large Onyx dataset #93916

Closed

Description

@melvin-bot

Problem

NewDot's browser tab crashes repeatedly and silently on web. The crash produces no JS error, runs no graceful shutdown, and Sentry captures zero error events. It has been reproduced regularly on staging and on ad-hoc builds while doing normal activity (DMing other users, clicking around inside an expense on the Spend page, etc.).

Root cause (current best understanding)

Renderer process OOM kill. The browser/OS SIGKILLs the renderer process under physical RAM pressure. This happens below jsHeapSizeLimit, so it is completely independent of the V8 heap limit and produces no JS error — which is why nothing reaches Sentry through normal error reporting.

Evidence from crash tracking captured via the instrumentation PRs below:

  • JS heap chronically pinned at ~97–98% usage with a large retained live set (~1.1 GB in one session, ~3.5 GB in a later one). The set is genuinely retained — a GC ran between two snapshots yet couldn't drop below ~1.1 GB.
  • A later crash report: usedJSHeapSizeMB: 3572, totalJSHeapSizeMB: 3589, jsHeapSizeLimitMB: 4192, reportsCount: 1083, domNodes: 2430, visibility: hidden.
  • Earlier memory snapshots before a crash: 98.13% and 97.38% heap usage with only ~23–29 MB free.

The trigger

The crash typically follows a healthy ManualSendMessage (the send itself renders fine in ~100 ms), then dies ~1 s later while draining post-send work. In one ~1.23 s window a single comment generated:

  • 71 Onyx writes (21 of them snapshot_ blobs — the AddComment snapshot fan-out)
  • 27 OnyxDerived recomputes, including reportAttributes ×6 and todos ×6, each iterating all ~89 reports.

With only ~23–29 MB free, that allocation burst is the final straw → OOM kill. Bursts of OnyxDerived recomputation (e.g. 91 computation events at once) were observed correlating with crashes, also triggered by OpenApp.

Proposed action items

  • Investigate the high retained heap — large memory footprint even for a normal number of reports. Find what's keeping ~1.1 GB+ resident.
  • Upgrade the OnyxDerived system to batch/debounce computation when many dependencies change in the same time window (would help many other cases too).
  • Cut the snapshot fan-out — does one comment really need to rewrite ~21 search snapshots?
  • Continue crash instrumentation so renderer OOM kills are observable.

Related work

Reported in Slack.

Issue OwnerCurrent Issue Owner: @mallenexpensify
Issue OwnerCurrent Issue Owner: @mallenexpensify
Upwork Automation - Do Not Edit

Activity

  1. added
    BugSomething is broken. Auto assigns a BugZero manager.
    on Jun 18, 2026
  2. MelvinBot commented on Jun 18, 2026

    @MelvinBot
    Contributor

    Issue Analysis

    Confidence: High for the trigger mechanism, Medium for the dominant retained-heap source (needs a heap snapshot to settle which holds the GBs).

    Root cause confirmed: The crash is a renderer OOM, and the code matches the hypothesis. Two App-side problems amplify every "drain post-send" burst, and a third is purely backend.

    1. OnyxDerived has no batching/debouncing — every dependency write recomputes synchronously (App-fixable ✅)

    The engine wires one Onyx.connectWithoutView per dependency, and each callback immediately recomputes and writes — there is no queue, timer, or microtask coalescing:

    reportAttributes (13 deps) and todos (9 deps) both subscribe to the overlapping report collections (report_, reportActions_, reportMetadata_, transaction_, reportNVP_, policy_). A comment send mutates ~6 of those collections twice (optimistic + server), so each derived key recomputes ~6× — exactly the observed reportAttributes ×6 / todos ×6.

    todos.ts:38-86 always does a full O(all reports) scan with 4 expensive predicates per expense report — no incremental path. reportAttributes has an incremental fast-path but still runs an unconditional all-reports loop every time (reportAttributes.ts:378-396).

    2. The Onyx cache is never bounded by JS-heap pressure (App-fixable ✅)

    evictableKeys includes SNAPSHOT and REPORT_ACTIONS (setup/index.ts:37-43), but eviction in react-native-onyx only fires on a persistent-storage quota error, never on RAM pressure. On web/IndexedDB the disk quota is huge, so it effectively never fires and the in-memory storageMap grows unbounded. Two specific App-side retainers stand out:

    • sortedReportActions (configs/sortedReportActions.ts:40-112) builds a full second copy of every report's actions and is keyed RAM_ONLY_…, so it is never evicted (setup/index.ts:63). This roughly doubles the report-actions footprint — large for accounts with deep chat history even at 89 reports.
    • snapshot_ blobs are denormalized SearchResults copies that accumulate one-per-search with no proactive cleanup, and several screens subscribe to the whole SNAPSHOT collection (e.g. SplitExpensePage, useDeleteTransactions), pinning all of them resident.

    3. Snapshot fan-out (~21 snapshot_ writes per comment) is backend-driven — escalate ⚠️

    The App optimistic path writes only one snapshot (the canned CHAT query) — Report/index.ts:1000-1004. The remaining ~20 arrive in the AddComment server response's onyxData/Pusher. This fan-out cannot be reduced from App — the logic that enumerates which search snapshots get the new report lives in the backend (Web-Expensify/Auth). This needs an internal engineer; the question to answer there is whether AddComment should fan out into every cached snapshot and whether it sends full blobs vs. minimal merges.


    Recommended actions (re-prioritized)

    The single highest-leverage App-side fix is #1 — it cuts both CPU and the allocation burst that is "the final straw," and helps OpenApp and every other multi-write flow:

    1. Batch/debounce OnyxDerived recompute. Coalesce same-tick dependency callbacks into one recompute per derived key (microtask/InteractionManager flush), and/or add an equality check before Onyx.set so unchanged results don't notify subscribers.
    2. Make todos incremental like reportAttributes, scoping work to changed report keys instead of re-scanning all reports every callback.
    3. Bound memory under heap pressure — add a heap-aware eviction trigger (or cap resident snapshot_ count / give snapshots a TTL and clear on navigating away from Search), and reconsider whether sortedReportActions must hold every report's actions in RAM permanently.
    4. Escalate the snapshot fan-out (Add Expensify eslint #3 above) to a backend engineer — not actionable in App.
    5. Keep the crash instrumentation from [TESTING] Add web crash diagnostics to detect abnormal browser session exits #93264 running to observe renderer OOM kills.
    Evidence chain & open questions
    • Trigger: healthy ManualSendMessage → ~6 report collections written twice (optimistic + server) → each fires reportAttributes/todos recompute with no coalescing → full all-reports scans + fresh allocations under ~23–29 MB free → OOM ~1s later. High confidence; the ×6 counts and timing match the code exactly.
    • Retained heap (medium confidence): a Chrome heap-snapshot "retainers" view on the OOM'd tab would settle which dominates — Onyx storageMap snapshots vs. the sortedReportActions RAM-only copy vs. other derived maps. The existing SPAN_ONYX_DERIVED_COMPUTE spans + the [OnyxDerived] dependency X changed, recomputing logs can confirm the 27-recompute count and per-compute duration/collection size in production.
    • Alternative considered: could the burst be a useOnyx re-render storm rather than derived compute? The skipCacheCheck: true writes do force subscriber notifications, so Some initial fixes and code style updates #1's equality-check half also addresses that — they're complementary, not competing.
    • Backend AddComment handler (private): does it skip snapshots that aren't currently mounted/subscribed, and does it send full blobs or merges?

    Next Steps for Contributor+ team:
    To accept: @MelvinBot implement [this](https://github.com/Expensify/App/issues/93916) to create a draft PR (recommend scoping the first PR to action item #1 only — OnyxDerived batching).
    To refine: @MelvinBot <your feedback>
    To reject: Explain why you are rejecting Melvin's proposal.

  3. added
    ExternalAdded to denote the issue can be worked on by a contributor
    on Jun 18, 2026
  4. added
    Help WantedApply this label when an issue is open to proposals by contributors
    on Jun 18, 2026
  5. ahmdshrif commented on Jun 18, 2026

    @ahmdshrif
    Contributor

    Proposal

    Please re-state the problem that we are trying to solve in this issue.

    NewDot's web tab is SIGKILLed by the OS as a renderer OOM — below jsHeapSizeLimit, so there's no JS error and nothing reaches Sentry. It dies ~1s after a normal action (e.g. sending a comment) while draining post-send work.

    What is the root cause of that problem?

    The proximate, frontend-fixable cause is that OnyxDerived has no recompute coalescing — every individual Onyx write synchronously drives a full derived recompute, so a normal write burst becomes an allocation storm.

    1. The driver runs recomputeDerivedValue synchronously inside each Onyx callback (collection, locale, scalar) with no microtask/debounce between writes: index.ts#L64-L143.
    2. todos rescans all reports per recompute with no incremental scoping (todos.ts#L38-L86); reportAttributes already scopes by sourceValues (reportAttributes.ts#L147-L190) — todos is the asymmetric amplifier.

    So one comment's 71 Onyx writes fan into 27 recomputes (todos ×6, reportAttributes ×6), each iterating ~89 reports in 1.23s. With only ~25MB free that transient burst exceeds free RAM → renderer killed below the V8 limit → no JS error. Explains every symptom incl. trigger timing and the missing Sentry event.

    What changes do you think we should make in order to solve the problem?

    Coalesce recomputes at the single chokepoint: in recomputeDerivedValue, accumulate changed sourceValues per key and flush once per microtask (queueMicrotask) instead of per callback. One central change covers all 11 configs and collapses a burst to one recompute per key. Also give todos the sourceValues incremental scoping reportAttributes already has.
    Regression-safe: the flush still settles within the same tick before paint and merges every accumulated sourceValue, so consumers see identical final values — only the count of intermediate recomputes drops.

    What alternative solutions did you explore? (Optional)

    • Cut the snapshot fan-out: fewer writes, but REPORT/REPORT_ACTIONS writes still storm the derived layer — partial mitigation.
    • Bound the retained ~1.1GB heap (cache cap): necessary but separate and not localizable to one PR; the un-coalesced burst still spikes allocations.
    • Debounce only todos: too narrow — the other 10 configs keep recomputing per write.
  6. melvin-bot commented on Jun 18, 2026

    @melvin-bot
    Author

    Triggered auto assignment to Contributor-plus team member for initial proposal review - @eh2077 (External)

  7. changed the title [-]NewDot web tab crashes silently (renderer OOM kill) under high memory / large Onyx dataset[/-] [+][$250] NewDot web tab crashes silently (renderer OOM kill) under high memory / large Onyx dataset[/+] on Jun 18, 2026
  8. melvin-bot commented on Jun 18, 2026

    @melvin-bot
    Author
  9. aswin-s commented on Jun 18, 2026

    @aswin-s
    Contributor

    Proposal

    Please re-state the problem that we are trying to solve in this issue.

    When a user sends a comment (or during OpenApp), a burst of Onyx dependency changes triggers 27+ synchronous, unbatched OnyxDerived recomputes across 11 derived keys (e.g. reportAttributes ×6, todos ×6). Each recompute allocates a new derived value object and immediately writes it to Onyx via setDerivedValue. On a large dataset (~1,083 reports, ~1.1–3.5 GB retained heap), the compounded allocation burst exhausts the remaining JS heap headroom (~23–29 MB free), causing the OS to OOM-kill the browser renderer process with no JS error — which is why Sentry captures nothing. The expected behavior is that rapid, same-tick dependency changes for the same derived key should be coalesced into a single recompute-and-write cycle.

    What is the root cause of that problem?

    In src/libs/actions/OnyxDerived/index.ts, the recomputeDerivedValue closure (line 64) is called directly and synchronously inside each Onyx dependency callback (lines 112, 132, 142). There is no debouncing, batching, or queuing — every Onyx key change in the same burst fires its own compute() + Onyx.set() call immediately. With 11 derived keys each watching multiple overlapping collection dependencies (e.g. REPORT_ACTIONS and REPORT both appear in several configs), a single AddComment write that touches snapshot_, REPORT_ACTIONS, and the report itself can trigger 20+ redundant compute() invocations in rapid succession, each one allocating a full derived-value object over an already-saturated heap.

    What changes do you think we should make in order to solve the problem?

    1. src/libs/actions/OnyxDerived/index.ts — add per-derived-key debounce around recomputeDerivedValue (line 64–99). Replace the direct synchronous call at lines 112, 132, and 142 with a requestAnimationFrame- or setTimeout(0)-based deferred call per derived key, so that all dependency callbacks that fire within the same microtask/event-loop tick are coalesced into one compute() + Onyx.set() per key. The deferred call must use the latest dependencyValues snapshot at the time it fires (already correct since dependencyValues is mutated in place via setDependencyValue before calling recomputeDerivedValue).

    2. src/libs/actions/OnyxDerived/index.ts — track the pending timer per derived key. Introduce a pendingRecomputeTimer variable (scoped inside the for loop per key, alongside areAllConnectionsSet) and cancel any existing timer before scheduling a new one, preventing stale re-queues.

    3. src/libs/actions/OnyxDerived/index.ts — preserve sourceValues coalescing during debounce window. During the debounce window, accumulate all {sourceKey: sourceValue} pairs that arrive rather than overwriting; pass the merged map as context.sourceValues so that incremental-update guards in configs like reportAttributes still receive the full changed-key set and can narrow the recompute scope correctly.

    4. src/libs/actions/OnyxDerived/utils.ts — no change needed. setDerivedValue uses skipCacheCheck: true and is called once per debounced cycle; the fix does not affect this layer.

    5. Separately: audit snapshot_ fan-out in src/libs/actions/Report/index.ts. The issue documents 21 snapshot writes per comment. Investigate whether all active search snapshots need rewriting on every comment, or whether only the snapshots containing the affected report need updating — a targeted invalidation would reduce the Onyx write burst independently of the OnyxDerived batching fix.

    What alternative solutions did you explore? (Optional)

    Increasing the jsHeapSizeLimit via --max-old-space-size is not actionable on web (the flag applies only to Node.js); the OOM is an OS-level renderer kill below V8's limit, so heap-limit changes cannot prevent it — only reducing live allocation pressure can.

  10. 118 remaining items

  11. VickyStash commented on Aug 25, 2026

    @VickyStash
    Contributor

    @mountiny, I think we can move forward with payment for this one

  12. changed the title [-][$250] NewDot web tab crashes silently (renderer OOM kill) under high memory / large Onyx dataset[/-] [+][$500] NewDot web tab crashes silently (renderer OOM kill) under high memory / large Onyx dataset[/+] on Aug 25, 2026
  13. melvin-bot commented on Aug 25, 2026

    @melvin-bot
    Author

    ⚠️ Failed to update price automatically. The BZ team member will need to update the price manually in Upwork.

  14. added
    Awaiting PaymentAuto-added when associated PR is deployed to production
    and removed
    Skip PaymentSkips all the payment process steps for an issue. Use it for fixed-price projects.
    on Aug 25, 2026
  15. changed the title [-][$500] NewDot web tab crashes silently (renderer OOM kill) under high memory / large Onyx dataset[/-] [+][Due for payment 2026-08-25] [$500] NewDot web tab crashes silently (renderer OOM kill) under high memory / large Onyx dataset[/+] on Aug 25, 2026
  16. melvin-bot commented on Aug 25, 2026

    @melvin-bot
    Author

    Triggered auto assignment to @mallenexpensify (Awaiting Payment)

  17. mountiny commented on Aug 25, 2026

    @mountiny
    Contributor

    Sounds good thank you

  18. mallenexpensify commented on Aug 25, 2026

    @mallenexpensify
    Contributor

    Payment Summary

    Contributor+: @eh2077 due $500 via NewDot

  19. laurenreidexpensify commented on Sep 4, 2026

    @laurenreidexpensify
    Contributor

    $500 approved @eh2077 New Expensify

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

Awaiting PaymentAuto-added when associated PR is deployed to productionDailyKSv2

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions