Skip to content

zarr: guard against OOM on large recordings (#909) - #62

Merged
neuromechanist merged 1 commit into
mainfrom
fix/zarr-largefile-oom
Jul 7, 2026
Merged

neuromechanist merged 1 commit into
mainfrom
fix/zarr-largefile-oom

Conversation

@neuromechanist

Copy link
Copy Markdown
Member

Fixes two observed production failures on the Hallu zarr conversion node (62 GB RAM, ZARR_JOBS=8 → ~6 GB/worker). Implements nemarOrg/nemar-cli#909.

The failures (from the live logs)

1. Split-FIF misrouting → OOM cascade. A split recording's bulk lives in split-02.., which carry different stems than split-01. _recording_size_bytes only summed split-01's same-stem companions → undercount → should_stream returned False → MNE loaded the whole chain in memory → OOM. Observed: sub-023..028_task-crossmodal_split-01_meg.fif all failed worker crashed: process pool terminated abruptly in one batch — an OOM-SIGKILL breaks the ProcessPoolExecutor and cascades every concurrently-running sibling.

2. No memory guard → OOM + retry-burn. EDF/BDF/EEGLAB have no streaming reader, so a large one loads at float64 (~6×) and OOM-kills its worker. The crash carries no biosigIO .code, so it classifies as infra → the dataset retries every run and burns attempts before terminal failed (observed: on006269 rc=1 recurring across days).

The fix

  • Split-FIF sizing: sum the whole split group so a large chain routes to the bounded-memory streamer (peaks ~4 GB regardless of size).
  • Per-recording memory guard: project peak RAM (bounded for streaming, size × factor for in-memory) and skip before any load anything over a per-recording budget = usable_RAM / --jobs. The skip is a typed RecordingTooLarge whose .code flows through the existing result machinery → deterministic (no infra retry, checkpoint advances), surfaced in index.json with a user-facing reason. Because it raises before the load, the OOM never happens → no BrokenProcessPool cascade, no pool restructure.

Budget scales with --jobs: the 8-way cron skips a large in-memory recording that a manual --jobs 1 re-run (whole-node budget ≈ 49 GB → ~8 GB EDF) converts. All thresholds env-overridable (ZARR_REC_MEM_BUDGET_BYTES, ZARR_INMEM_MEM_FACTOR, ZARR_STREAM_PEAK_BYTES, ZARR_MEM_HEADROOM_FRAC).

Behavior on the cron (validated on the live node)

Recording Before After
Large BrainVision/FIF/CTF (incl. split-FIF) sometimes OOM-cascade streams, converts
EDF/BDF/SET > ~1 GB OOM-cascade or lucky convert clean deterministic skip + visible reason (or --jobs 1 converts up to ~8 GB)
Huge (won't fit even solo) OOM + infra retry-burn clean skip, no retry

Test plan

  • 8 new unittests (split-chain sizing, projection, budget scaling + override, preflight-skip carries the deterministic code, user-facing reason). Full suite 114 green, ruff clean.
  • Validated on the live Hallu node: 114 tests pass in the node's Python; usable_ram_bytes=49 GiB, cron budget 6 GiB, --jobs 1 budget 49 GiB.

Follow-ups

  • The real fix so large EDF/BDF/SET actually convert (not just skip): a streaming reader for these formats — separate issue, tied to biosigIO / nemar-cli#737.
  • Optional: auto-re-enqueue a recording_too_large-skipped dataset at --jobs 1 so the cron converts medium in-memory recordings without operator action.

Deploys automatically (Hallu's hallu-zarr.sh clones nemarDatasets/.github@main).

Two real, observed failures on the Hallu conversion node (62 GB RAM,
ZARR_JOBS=8, so ~6 GB/worker):

1. Split-FIF misrouting: a split recording's bulk lives in split-02.., which
   carry DIFFERENT stems than split-01, so _recording_size_bytes summed only
   split-01 -> should_stream returned False -> MNE loaded the WHOLE chain in
   memory -> OOM. Observed: crossmodal MEG sub-023..028 all 'worker crashed:
   process pool terminated abruptly' in one batch (an OOM-SIGKILL breaks the
   ProcessPool and cascades every concurrently-running sibling). Fix: sum the
   whole split group so large split-FIF routes to the bounded-memory streamer.

2. No memory guard: EDF/BDF/EEGLAB have no streaming reader, so a large one
   loads at float64 (~6x) and OOM-kills its worker; the crash has no biosigIO
   .code, so it classifies as INFRA -> the dataset retries every run and burns
   attempts before terminal 'failed' (observed: on006269 rc=1 across days).
   Fix: PROJECT each recording's peak RAM (bounded for streaming, size*factor
   for in-memory) and skip -- BEFORE any load -- anything over a per-recording
   budget (usable_RAM / --jobs). The skip is a typed RecordingTooLarge whose
   .code flows through the existing machinery: DETERMINISTIC (no infra retry,
   checkpoint advances), surfaced in the served index.json with a user-facing
   reason. Because it raises before the load, the OOM never happens, so no
   BrokenProcessPool cascade -- no pool restructure needed.

Budget scales with --jobs: the 8-way cron skips a large in-memory recording
that a manual '--jobs 1' re-run (whole-node budget) converts. All thresholds
env-overridable (ZARR_REC_MEM_BUDGET_BYTES, ZARR_INMEM_MEM_FACTOR,
ZARR_STREAM_PEAK_BYTES, ZARR_MEM_HEADROOM_FRAC). Validated on the live node:
114 tests green, usable RAM 49 GiB, cron budget 6 GiB.

The real fix for large EDF/BDF/SET (a streaming reader so they convert rather
than skip) is a follow-up tied to biosigIO / nemar-cli#737.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant