Skip to content

Epic: Zarr conversion memory robustness #1108

Closed
5 of 5 issues completed
Closed

Description

@neuromechanist

Zarr conversion on Hallu OOM-kills its pool workers routinely, and a single kill aborts the entire dataset. /mnt/local/zarr-state/.nm-zarr.log records 383 worker crashed events and 96 BrokenProcessPool aborts. The queue currently holds failed=22 (several exhausted at attempts=5), data_failed=43, done=535, pending=154.

The failure

On 2026-08-22, on004998 (MEG FIF) converted 74 of 115 recordings, then one worker was OOM-killed; ProcessPoolExecutor poisoned the pool and the run aborted, losing the remaining 41 recordings, none of which had anything to do with the memory pressure. Thirty-four minutes later on004917 (BrainVision) lost all 24 the same way, with 7 workers dying simultaneously.

Root cause

generate_zarr.py admission control projects each recording's peak RAM as on-disk bytes times INMEM_MEM_FACTOR = 6, and dispatches while the in-flight sum fits MemTotal * 0.8 (~49 GiB on Hallu). on004917's 24 .eeg blobs are 1.18-2.25 GB each, and all but one sit just under the STREAM_MIN_BYTES = 2 GiB streaming threshold, so 23 of 24 took the unbounded in-memory path. Admission then packed exactly 7 of them (45.2 GiB projected, plus 4 GiB for the one streaming recording, against a 49.6 GiB ceiling) and those 7 are precisely the 7 that crashed. The admission arithmetic worked; its input was wrong.

The real in-memory cost of a 66-channel BrainVision recording with ~9.8M samples per channel is roughly: MNE preload at float64 (5.2 GB), plus biosigIO's per-channel self.signals[label] = data DataFrame copy (5.2 GB), plus the block-manager consolidation transient that biosigIO's own PerformanceWarning announces (up to 5.2 GB again), plus the resample copy. That is about 12x on-disk bytes, not 6x. Seven concurrent is ~108 GB on a 62 GB node.

Corroborating: recording_too_large (the #909 guard built to prevent exactly this) has fired zero times in the entire log history, while 383 workers were OOM-killed. It is calibrated so loosely that it is inert.

The bounded path already exists

biosigio.exporters.zarr_stream.stream_to_zarr (1.2.4, installed) is genuinely segmented: pass 1 streams 30-second windows into a channel-major float32 memmap on scratch disk, pass 2 resamples and quantizes one channel at a time. Peak RAM is one read window plus one channel, independent of file size. So the gap is not that segmentation is missing. It is that segmentation is opt-in above a threshold that is too high, the fallback path is unbounded, and EEGLAB .set has no streaming route at all.

Where the code lives

scripts/zarr/ currently sits in nemarDatasets/.github for a reason that no longer holds: the conversion was originally meant to run as GitHub Actions, billed against the Team allowance. It could not sustain bulk or backfill work (a large dataset stalls past the 120-minute cap), so the engine moved to the Hallu cron, but the code never moved back. The backend has already retreated from the Actions path: ZARR_AUTODISPATCH defaults off and the Hallu cron owns conversion. Meanwhile the deployed script on Hallu is a hand-copied file at /data/projects/yahya/nemar/hallu-zarr.sh, and this repo carries a stale 12.8 KB fork of it at scripts/hallu-zarr.sh that runs nowhere. Phase 1 repatriates the engine here and retires the Actions path.

Mitigation already applied

ZARR_STREAM_MIN_BYTES=268435456 was added to the Hallu crontab on 2026-08-22 (backup at /mnt/local/zarr-state/crontab.bak-20260822-stream-threshold), routing .vhdr/.fif/.ds/.mefd at or above 256 MB to the bounded path. It does not cover .set, which has no streaming route to lower a threshold into. Phase 4 makes this permanent in code and retires the override.

Phases

  1. Repatriate the zarr conversion engine to nemar-cli
  2. Contain the blast radius: one OOM costs one recording, not the dataset
  3. Measure and calibrate the RAM projection
  4. Widen the bounded streaming path
  5. Recover the failed backlog

Activity

  1. added
    P1High priority - important features, significant bugs
    featureNew feature or enhancement
    on Aug 22, 2026
  2. neuromechanist commented on Aug 22, 2026

    @neuromechanist
    ContributorAuthor

    Runs alongside epic #1095 ("Zarr availability and hygiene"). The split: #1095 owns what counts as a recording (formats, raw-only scope, the dispatch contract); this epic owns whether conversion survives its own memory use (RAM projection, OOM containment, the bounded streaming path, backlog recovery).

    #1095 was audited on 2026-08-22 and its landed tracks closed (#1096, #1097, #1098, #1102). Its genuine remainder is #1099 (docs), #1100 (the non-OOM half of the whole-dataset failures), #1103 (gate-superset enforcement), and #1106 (events sidecar vs directory recordings). Phase 1 here closes #1103, because moving the converter into this repo is what makes that assertion possible at all. #1095 is driven to closure after this epic.

  3. neuromechanist commented on Aug 22, 2026

    @neuromechanist
    ContributorAuthor

    All five phases merged and delivered to dev via #1119 (merge commit bad7cf4).

    Phase PR Outcome
    1 #1114 Engine repatriated to scripts/zarr/; Actions dispatch retired; #1103 closed
    2 #1115 RLIMIT_DATA backstop; two-pass pool-break isolation
    3 #1116 MemAvailable ceiling; measured peak RSS; per-format factors
    4 #1117 Bounded streaming by default at 256 MiB; .set exclusion evidenced
    5 #1118 requeue for the stranded backlog

    Root cause was two compounding ~2x overestimates: an admission ceiling claiming 50.0 GiB on a node with 23.7 GiB available, and a 6x projection factor against a real ~12x BrainVision cost. Together they packed exactly the seven concurrent recordings that exhausted the box.

    ADRs: 0029 (the conversion engine lives with the CLI, a scoped carve-out from 0020), 0030 (bounded streaming is the default conversion path). ADR 0027 carries a dated addendum for the location and dispatch change.

    The comprehensive epic review found four cross-phase criticals that per-phase review could not see, all fixed before merge: RecordingTooLarge was dead code on the streaming path, an unreadable /proc/meminfo silently collapsed the temporary/permanent verdict split, the "will be retried automatically" message was false for partial-success runs, and the new telemetry fields had no reader in ZarrReadyBody.

    Remaining work is deployment, tracked separately: cut Hallu to a nemar-cli checkout, drop the ZARR_STREAM_MIN_BYTES crontab mitigation, requeue the stranded datasets, and merge nemarDatasets/.github#103.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High priority - important features, significant bugsfeatureNew feature or enhancementzarr

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions