Repository navigation
Description
Zarr conversion on Hallu OOM-kills its pool workers routinely, and a single kill aborts the entire dataset. /mnt/local/zarr-state/.nm-zarr.log records 383 worker crashed events and 96 BrokenProcessPool aborts. The queue currently holds failed=22 (several exhausted at attempts=5), data_failed=43, done=535, pending=154.
The failure
On 2026-08-22, on004998 (MEG FIF) converted 74 of 115 recordings, then one worker was OOM-killed; ProcessPoolExecutor poisoned the pool and the run aborted, losing the remaining 41 recordings, none of which had anything to do with the memory pressure. Thirty-four minutes later on004917 (BrainVision) lost all 24 the same way, with 7 workers dying simultaneously.
Root cause
generate_zarr.py admission control projects each recording's peak RAM as on-disk bytes times INMEM_MEM_FACTOR = 6, and dispatches while the in-flight sum fits MemTotal * 0.8 (~49 GiB on Hallu). on004917's 24 .eeg blobs are 1.18-2.25 GB each, and all but one sit just under the STREAM_MIN_BYTES = 2 GiB streaming threshold, so 23 of 24 took the unbounded in-memory path. Admission then packed exactly 7 of them (45.2 GiB projected, plus 4 GiB for the one streaming recording, against a 49.6 GiB ceiling) and those 7 are precisely the 7 that crashed. The admission arithmetic worked; its input was wrong.
The real in-memory cost of a 66-channel BrainVision recording with ~9.8M samples per channel is roughly: MNE preload at float64 (5.2 GB), plus biosigIO's per-channel self.signals[label] = data DataFrame copy (5.2 GB), plus the block-manager consolidation transient that biosigIO's own PerformanceWarning announces (up to 5.2 GB again), plus the resample copy. That is about 12x on-disk bytes, not 6x. Seven concurrent is ~108 GB on a 62 GB node.
Corroborating: recording_too_large (the #909 guard built to prevent exactly this) has fired zero times in the entire log history, while 383 workers were OOM-killed. It is calibrated so loosely that it is inert.
The bounded path already exists
biosigio.exporters.zarr_stream.stream_to_zarr (1.2.4, installed) is genuinely segmented: pass 1 streams 30-second windows into a channel-major float32 memmap on scratch disk, pass 2 resamples and quantizes one channel at a time. Peak RAM is one read window plus one channel, independent of file size. So the gap is not that segmentation is missing. It is that segmentation is opt-in above a threshold that is too high, the fallback path is unbounded, and EEGLAB .set has no streaming route at all.
Where the code lives
scripts/zarr/ currently sits in nemarDatasets/.github for a reason that no longer holds: the conversion was originally meant to run as GitHub Actions, billed against the Team allowance. It could not sustain bulk or backfill work (a large dataset stalls past the 120-minute cap), so the engine moved to the Hallu cron, but the code never moved back. The backend has already retreated from the Actions path: ZARR_AUTODISPATCH defaults off and the Hallu cron owns conversion. Meanwhile the deployed script on Hallu is a hand-copied file at /data/projects/yahya/nemar/hallu-zarr.sh, and this repo carries a stale 12.8 KB fork of it at scripts/hallu-zarr.sh that runs nowhere. Phase 1 repatriates the engine here and retires the Actions path.
Mitigation already applied
ZARR_STREAM_MIN_BYTES=268435456 was added to the Hallu crontab on 2026-08-22 (backup at /mnt/local/zarr-state/crontab.bak-20260822-stream-threshold), routing .vhdr/.fif/.ds/.mefd at or above 256 MB to the bounded path. It does not cover .set, which has no streaming route to lower a threshold into. Phase 4 makes this permanent in code and retires the override.
Phases
- Repatriate the zarr conversion engine to nemar-cli
- Contain the blast radius: one OOM costs one recording, not the dataset
- Measure and calibrate the RAM projection
- Widen the bounded streaming path
- Recover the failed backlog
Activity
neuromechanist commentedon Aug 22, 2026
Runs alongside epic #1095 ("Zarr availability and hygiene"). The split: #1095 owns what counts as a recording (formats, raw-only scope, the dispatch contract); this epic owns whether conversion survives its own memory use (RAM projection, OOM containment, the bounded streaming path, backlog recovery).
#1095 was audited on 2026-08-22 and its landed tracks closed (#1096, #1097, #1098, #1102). Its genuine remainder is #1099 (docs), #1100 (the non-OOM half of the whole-dataset failures), #1103 (gate-superset enforcement), and #1106 (events sidecar vs directory recordings). Phase 1 here closes #1103, because moving the converter into this repo is what makes that assertion possible at all. #1095 is driven to closure after this epic.
Move patch_duration.py in with the converter it imports
Address PR review findings
Phase 1: Repatriate the zarr conversion engine to nemar-cli (#1114)
Address Phase 2 review findings
Phase 2: Contain the blast radius of an OOM-killed worker (#1115)
Address Phase 3 review findings
Phase 3: Measure and calibrate the RAM projection (#1116)
Do not charge in-memory slack to streamed recordings
Address Phase 4 review findings
Make the streaming projection channel-aware
Phase 4: Make bounded streaming the default conversion path (#1117)
Fix requeue argument parsing and run it before the lock
Forward --dataset to requeue instead of silently ignoring it
Phase 5: Recover the stranded conversion backlog (#1118)
Address comprehensive epic review findings
Address comprehensive test-coverage review
neuromechanist commentedon Aug 22, 2026
All five phases merged and delivered to
devvia #1119 (merge commit bad7cf4).scripts/zarr/; Actions dispatch retired; #1103 closedRLIMIT_DATAbackstop; two-pass pool-break isolationMemAvailableceiling; measured peak RSS; per-format factors.setexclusion evidencedrequeuefor the stranded backlogRoot cause was two compounding ~2x overestimates: an admission ceiling claiming 50.0 GiB on a node with 23.7 GiB available, and a 6x projection factor against a real ~12x BrainVision cost. Together they packed exactly the seven concurrent recordings that exhausted the box.
ADRs: 0029 (the conversion engine lives with the CLI, a scoped carve-out from 0020), 0030 (bounded streaming is the default conversion path). ADR 0027 carries a dated addendum for the location and dispatch change.
The comprehensive epic review found four cross-phase criticals that per-phase review could not see, all fixed before merge:
RecordingTooLargewas dead code on the streaming path, an unreadable/proc/meminfosilently collapsed the temporary/permanent verdict split, the "will be retried automatically" message was false for partial-success runs, and the new telemetry fields had no reader inZarrReadyBody.Remaining work is deployment, tracked separately: cut Hallu to a nemar-cli checkout, drop the
ZARR_STREAM_MIN_BYTEScrontab mitigation, requeue the stranded datasets, and merge nemarDatasets/.github#103.Move the Zarr engine to nemar-cli and retire the Actions path (#103)