Repository navigation
Description
Problem
NewDot's browser tab crashes repeatedly and silently on web. The crash produces no JS error, runs no graceful shutdown, and Sentry captures zero error events. It has been reproduced regularly on staging and on ad-hoc builds while doing normal activity (DMing other users, clicking around inside an expense on the Spend page, etc.).
Root cause (current best understanding)
Renderer process OOM kill. The browser/OS SIGKILLs the renderer process under physical RAM pressure. This happens below jsHeapSizeLimit, so it is completely independent of the V8 heap limit and produces no JS error — which is why nothing reaches Sentry through normal error reporting.
Evidence from crash tracking captured via the instrumentation PRs below:
- JS heap chronically pinned at ~97–98% usage with a large retained live set (~1.1 GB in one session, ~3.5 GB in a later one). The set is genuinely retained — a GC ran between two snapshots yet couldn't drop below ~1.1 GB.
- A later crash report:
usedJSHeapSizeMB: 3572,totalJSHeapSizeMB: 3589,jsHeapSizeLimitMB: 4192,reportsCount: 1083,domNodes: 2430,visibility: hidden. - Earlier memory snapshots before a crash: 98.13% and 97.38% heap usage with only ~23–29 MB free.
The trigger
The crash typically follows a healthy ManualSendMessage (the send itself renders fine in ~100 ms), then dies ~1 s later while draining post-send work. In one ~1.23 s window a single comment generated:
- 71 Onyx writes (21 of them
snapshot_blobs — the AddComment snapshot fan-out) - 27 OnyxDerived recomputes, including
reportAttributes×6 andtodos×6, each iterating all ~89 reports.
With only ~23–29 MB free, that allocation burst is the final straw → OOM kill. Bursts of OnyxDerived recomputation (e.g. 91 computation events at once) were observed correlating with crashes, also triggered by OpenApp.
Proposed action items
- Investigate the high retained heap — large memory footprint even for a normal number of reports. Find what's keeping ~1.1 GB+ resident.
- Upgrade the OnyxDerived system to batch/debounce computation when many dependencies change in the same time window (would help many other cases too).
- Cut the snapshot fan-out — does one comment really need to rewrite ~21 search snapshots?
- Continue crash instrumentation so renderer OOM kills are observable.
Related work
- [No QA] Add Sentry ReportingObserver integration to capture browser crash/intervention reports #93049 — ReportingObserver / Reporting API instrumentation
- [TESTING] Add web crash diagnostics to detect abnormal browser session exits #93264 — crash + memory tracking PR (this successfully caught a reproduced crash and reported it to Sentry)
- Sentry issue:
7543552932
Reported in Slack.
Issue Owner
Current Issue Owner: @mallenexpensifyIssue Owner
Current Issue Owner: @mallenexpensifyUpwork Automation - Do Not Edit
- Upwork Job URL: https://www.upwork.com/jobs/~022067540022852060482
- Upwork Job ID: 2067540022852060482
- Last Price Increase: 2026-06-18
Metadata
Metadata
Labels
Type
Projects
- StatusShow more project fieldsDone
Activity
MelvinBot commentedon Jun 18, 2026
Issue Analysis
Confidence: High for the trigger mechanism, Medium for the dominant retained-heap source (needs a heap snapshot to settle which holds the GBs).
Root cause confirmed: The crash is a renderer OOM, and the code matches the hypothesis. Two App-side problems amplify every "drain post-send" burst, and a third is purely backend.
1. OnyxDerived has no batching/debouncing — every dependency write recomputes synchronously (App-fixable ✅)
The engine wires one
Onyx.connectWithoutViewper dependency, and each callback immediately recomputes and writes — there is no queue, timer, or microtask coalescing:OnyxDerived/index.ts:64-99—recomputeDerivedValuerunscompute()and writes synchronously on every callback (index.ts:109-112).OnyxDerived/utils.ts:23-26— each recompute writes withskipCacheCheck: true, so there's no deep-equality short-circuit; every write notifies alluseOnyxsubscribers and allocates a fresh object.reportAttributes(13 deps) andtodos(9 deps) both subscribe to the overlapping report collections (report_,reportActions_,reportMetadata_,transaction_,reportNVP_,policy_). A comment send mutates ~6 of those collections twice (optimistic + server), so each derived key recomputes ~6× — exactly the observedreportAttributes ×6/todos ×6.todos.ts:38-86always does a full O(all reports) scan with 4 expensive predicates per expense report — no incremental path.reportAttributeshas an incremental fast-path but still runs an unconditional all-reports loop every time (reportAttributes.ts:378-396).2. The Onyx cache is never bounded by JS-heap pressure (App-fixable ✅)
evictableKeysincludesSNAPSHOTandREPORT_ACTIONS(setup/index.ts:37-43), but eviction inreact-native-onyxonly fires on a persistent-storage quota error, never on RAM pressure. On web/IndexedDB the disk quota is huge, so it effectively never fires and the in-memorystorageMapgrows unbounded. Two specific App-side retainers stand out:sortedReportActions(configs/sortedReportActions.ts:40-112) builds a full second copy of every report's actions and is keyedRAM_ONLY_…, so it is never evicted (setup/index.ts:63). This roughly doubles the report-actions footprint — large for accounts with deep chat history even at 89 reports.snapshot_blobs are denormalizedSearchResultscopies that accumulate one-per-search with no proactive cleanup, and several screens subscribe to the wholeSNAPSHOTcollection (e.g.SplitExpensePage,useDeleteTransactions), pinning all of them resident.3. Snapshot fan-out (~21⚠️
snapshot_writes per comment) is backend-driven — escalateThe App optimistic path writes only one snapshot (the canned CHAT query) —
Report/index.ts:1000-1004. The remaining ~20 arrive in the AddComment server response'sonyxData/Pusher. This fan-out cannot be reduced from App — the logic that enumerates which search snapshots get the new report lives in the backend (Web-Expensify/Auth). This needs an internal engineer; the question to answer there is whether AddComment should fan out into every cached snapshot and whether it sends full blobs vs. minimal merges.Recommended actions (re-prioritized)
The single highest-leverage App-side fix is #1 — it cuts both CPU and the allocation burst that is "the final straw," and helps
OpenAppand every other multi-write flow:InteractionManagerflush), and/or add an equality check beforeOnyx.setso unchanged results don't notify subscribers.todosincremental likereportAttributes, scoping work to changed report keys instead of re-scanning all reports every callback.snapshot_count / give snapshots a TTL and clear on navigating away from Search), and reconsider whethersortedReportActionsmust hold every report's actions in RAM permanently.Evidence chain & open questions
ManualSendMessage→ ~6 report collections written twice (optimistic + server) → each firesreportAttributes/todosrecompute with no coalescing → full all-reports scans + fresh allocations under ~23–29 MB free → OOM ~1s later. High confidence; the×6counts and timing match the code exactly.storageMapsnapshots vs. thesortedReportActionsRAM-only copy vs. other derived maps. The existingSPAN_ONYX_DERIVED_COMPUTEspans + the[OnyxDerived] dependency X changed, recomputinglogs can confirm the 27-recompute count and per-compute duration/collection size in production.useOnyxre-render storm rather than derived compute? TheskipCacheCheck: truewrites do force subscriber notifications, so Some initial fixes and code style updates #1's equality-check half also addresses that — they're complementary, not competing.Next Steps for Contributor+ team:
To accept:
@MelvinBot implement [this](https://github.com/Expensify/App/issues/93916)to create a draft PR (recommend scoping the first PR to action item #1 only — OnyxDerived batching).To refine:
@MelvinBot <your feedback>To reject: Explain why you are rejecting Melvin's proposal.
ahmdshrif commentedon Jun 18, 2026
Proposal
Please re-state the problem that we are trying to solve in this issue.
NewDot's web tab is SIGKILLed by the OS as a renderer OOM — below
jsHeapSizeLimit, so there's no JS error and nothing reaches Sentry. It dies ~1s after a normal action (e.g. sending a comment) while draining post-send work.What is the root cause of that problem?
The proximate, frontend-fixable cause is that OnyxDerived has no recompute coalescing — every individual Onyx write synchronously drives a full derived recompute, so a normal write burst becomes an allocation storm.
recomputeDerivedValuesynchronously inside each Onyx callback (collection, locale, scalar) with no microtask/debounce between writes: index.ts#L64-L143.todosrescans all reports per recompute with no incremental scoping (todos.ts#L38-L86);reportAttributesalready scopes bysourceValues(reportAttributes.ts#L147-L190) —todosis the asymmetric amplifier.So one comment's 71 Onyx writes fan into 27 recomputes (todos ×6, reportAttributes ×6), each iterating ~89 reports in 1.23s. With only ~25MB free that transient burst exceeds free RAM → renderer killed below the V8 limit → no JS error. Explains every symptom incl. trigger timing and the missing Sentry event.
What changes do you think we should make in order to solve the problem?
Coalesce recomputes at the single chokepoint: in
recomputeDerivedValue, accumulate changedsourceValuesper key and flush once per microtask (queueMicrotask) instead of per callback. One central change covers all 11 configs and collapses a burst to one recompute per key. Also givetodosthesourceValuesincremental scopingreportAttributesalready has.Regression-safe: the flush still settles within the same tick before paint and merges every accumulated
sourceValue, so consumers see identical final values — only the count of intermediate recomputes drops.What alternative solutions did you explore? (Optional)
todos: too narrow — the other 10 configs keep recomputing per write.melvin-bot commentedon Jun 18, 2026
Triggered auto assignment to Contributor-plus team member for initial proposal review - @eh2077 (
External)[-]NewDot web tab crashes silently (renderer OOM kill) under high memory / large Onyx dataset[/-][+][$250] NewDot web tab crashes silently (renderer OOM kill) under high memory / large Onyx dataset[/+]melvin-bot commentedon Jun 18, 2026
Job added to Upwork: https://www.upwork.com/jobs/~022067540022852060482
aswin-s commentedon Jun 18, 2026
Proposal
Please re-state the problem that we are trying to solve in this issue.
When a user sends a comment (or during
OpenApp), a burst of Onyx dependency changes triggers 27+ synchronous, unbatchedOnyxDerivedrecomputes across 11 derived keys (e.g.reportAttributes×6,todos×6). Each recompute allocates a new derived value object and immediately writes it to Onyx viasetDerivedValue. On a large dataset (~1,083 reports, ~1.1–3.5 GB retained heap), the compounded allocation burst exhausts the remaining JS heap headroom (~23–29 MB free), causing the OS to OOM-kill the browser renderer process with no JS error — which is why Sentry captures nothing. The expected behavior is that rapid, same-tick dependency changes for the same derived key should be coalesced into a single recompute-and-write cycle.What is the root cause of that problem?
In
src/libs/actions/OnyxDerived/index.ts, therecomputeDerivedValueclosure (line 64) is called directly and synchronously inside each Onyx dependency callback (lines 112, 132, 142). There is no debouncing, batching, or queuing — every Onyx key change in the same burst fires its owncompute()+Onyx.set()call immediately. With 11 derived keys each watching multiple overlapping collection dependencies (e.g.REPORT_ACTIONSandREPORTboth appear in several configs), a single AddComment write that touchessnapshot_,REPORT_ACTIONS, and the report itself can trigger 20+ redundantcompute()invocations in rapid succession, each one allocating a full derived-value object over an already-saturated heap.What changes do you think we should make in order to solve the problem?
src/libs/actions/OnyxDerived/index.ts— add per-derived-key debounce aroundrecomputeDerivedValue(line 64–99). Replace the direct synchronous call at lines 112, 132, and 142 with arequestAnimationFrame- orsetTimeout(0)-based deferred call per derived key, so that all dependency callbacks that fire within the same microtask/event-loop tick are coalesced into onecompute()+Onyx.set()per key. The deferred call must use the latestdependencyValuessnapshot at the time it fires (already correct sincedependencyValuesis mutated in place viasetDependencyValuebefore callingrecomputeDerivedValue).src/libs/actions/OnyxDerived/index.ts— track the pending timer per derived key. Introduce apendingRecomputeTimervariable (scoped inside theforloop per key, alongsideareAllConnectionsSet) and cancel any existing timer before scheduling a new one, preventing stale re-queues.src/libs/actions/OnyxDerived/index.ts— preservesourceValuescoalescing during debounce window. During the debounce window, accumulate all{sourceKey: sourceValue}pairs that arrive rather than overwriting; pass the merged map ascontext.sourceValuesso that incremental-update guards in configs likereportAttributesstill receive the full changed-key set and can narrow the recompute scope correctly.src/libs/actions/OnyxDerived/utils.ts— no change needed.setDerivedValueusesskipCacheCheck: trueand is called once per debounced cycle; the fix does not affect this layer.Separately: audit
snapshot_fan-out insrc/libs/actions/Report/index.ts. The issue documents 21 snapshot writes per comment. Investigate whether all active search snapshots need rewriting on every comment, or whether only the snapshots containing the affected report need updating — a targeted invalidation would reduce the Onyx write burst independently of the OnyxDerived batching fix.What alternative solutions did you explore? (Optional)
Increasing the
jsHeapSizeLimitvia--max-old-space-sizeis not actionable on web (the flag applies only to Node.js); the OOM is an OS-level renderer kill below V8's limit, so heap-limit changes cannot prevent it — only reducing live allocation pressure can.118 remaining items
VickyStash commentedon Aug 25, 2026
@mountiny, I think we can move forward with payment for this one
eh2077 commentedon Aug 25, 2026
@mallenexpensify checklist completed #93916 (comment). I helped review 4 PRs about this issue
I'd like propose a payment of $500 for the completed work, do you agree? cc @mountiny
[-][$250] NewDot web tab crashes silently (renderer OOM kill) under high memory / large Onyx dataset[/-][+][$500] NewDot web tab crashes silently (renderer OOM kill) under high memory / large Onyx dataset[/+]melvin-bot commentedon Aug 25, 2026
[-][$500] NewDot web tab crashes silently (renderer OOM kill) under high memory / large Onyx dataset[/-][+][Due for payment 2026-08-25] [$500] NewDot web tab crashes silently (renderer OOM kill) under high memory / large Onyx dataset[/+]melvin-bot commentedon Aug 25, 2026
Triggered auto assignment to @mallenexpensify (
Awaiting Payment)mountiny commentedon Aug 25, 2026
Sounds good thank you
mallenexpensify commentedon Aug 25, 2026
Payment Summary
Contributor+: @eh2077 due $500 via NewDot
laurenreidexpensify commentedon Sep 4, 2026
$500 approved @eh2077 New Expensify