Repository navigation
zarr: guard against OOM on large recordings (#909) - #62
Merged
Merged
Conversation
Two real, observed failures on the Hallu conversion node (62 GB RAM, ZARR_JOBS=8, so ~6 GB/worker): 1. Split-FIF misrouting: a split recording's bulk lives in split-02.., which carry DIFFERENT stems than split-01, so _recording_size_bytes summed only split-01 -> should_stream returned False -> MNE loaded the WHOLE chain in memory -> OOM. Observed: crossmodal MEG sub-023..028 all 'worker crashed: process pool terminated abruptly' in one batch (an OOM-SIGKILL breaks the ProcessPool and cascades every concurrently-running sibling). Fix: sum the whole split group so large split-FIF routes to the bounded-memory streamer. 2. No memory guard: EDF/BDF/EEGLAB have no streaming reader, so a large one loads at float64 (~6x) and OOM-kills its worker; the crash has no biosigIO .code, so it classifies as INFRA -> the dataset retries every run and burns attempts before terminal 'failed' (observed: on006269 rc=1 across days). Fix: PROJECT each recording's peak RAM (bounded for streaming, size*factor for in-memory) and skip -- BEFORE any load -- anything over a per-recording budget (usable_RAM / --jobs). The skip is a typed RecordingTooLarge whose .code flows through the existing machinery: DETERMINISTIC (no infra retry, checkpoint advances), surfaced in the served index.json with a user-facing reason. Because it raises before the load, the OOM never happens, so no BrokenProcessPool cascade -- no pool restructure needed. Budget scales with --jobs: the 8-way cron skips a large in-memory recording that a manual '--jobs 1' re-run (whole-node budget) converts. All thresholds env-overridable (ZARR_REC_MEM_BUDGET_BYTES, ZARR_INMEM_MEM_FACTOR, ZARR_STREAM_PEAK_BYTES, ZARR_MEM_HEADROOM_FRAC). Validated on the live node: 114 tests green, usable RAM 49 GiB, cron budget 6 GiB. The real fix for large EDF/BDF/SET (a streaming reader so they convert rather than skip) is a follow-up tied to biosigIO / nemar-cli#737.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes two observed production failures on the Hallu zarr conversion node (62 GB RAM,
ZARR_JOBS=8→ ~6 GB/worker). Implements nemarOrg/nemar-cli#909.The failures (from the live logs)
1. Split-FIF misrouting → OOM cascade. A split recording's bulk lives in
split-02.., which carry different stems thansplit-01._recording_size_bytesonly summedsplit-01's same-stem companions → undercount →should_streamreturned False → MNE loaded the whole chain in memory → OOM. Observed:sub-023..028_task-crossmodal_split-01_meg.fifall failedworker crashed: process pool terminated abruptlyin one batch — an OOM-SIGKILL breaks theProcessPoolExecutorand cascades every concurrently-running sibling.2. No memory guard → OOM + retry-burn. EDF/BDF/EEGLAB have no streaming reader, so a large one loads at float64 (~6×) and OOM-kills its worker. The crash carries no biosigIO
.code, so it classifies as infra → the dataset retries every run and burns attempts before terminalfailed(observed:on006269rc=1recurring across days).The fix
size × factorfor in-memory) and skip before any load anything over a per-recording budget =usable_RAM / --jobs. The skip is a typedRecordingTooLargewhose.codeflows through the existing result machinery → deterministic (no infra retry, checkpoint advances), surfaced inindex.jsonwith a user-facing reason. Because it raises before the load, the OOM never happens → noBrokenProcessPoolcascade, no pool restructure.Budget scales with
--jobs: the 8-way cron skips a large in-memory recording that a manual--jobs 1re-run (whole-node budget ≈ 49 GB → ~8 GB EDF) converts. All thresholds env-overridable (ZARR_REC_MEM_BUDGET_BYTES,ZARR_INMEM_MEM_FACTOR,ZARR_STREAM_PEAK_BYTES,ZARR_MEM_HEADROOM_FRAC).Behavior on the cron (validated on the live node)
--jobs 1converts up to ~8 GB)Test plan
usable_ram_bytes=49 GiB, cron budget 6 GiB,--jobs 1budget 49 GiB.Follow-ups
recording_too_large-skipped dataset at--jobs 1so the cron converts medium in-memory recordings without operator action.Deploys automatically (Hallu's
hallu-zarr.shclonesnemarDatasets/.github@main).