You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Make the bounded streaming path the default and close the formats it cannot reach.
Why
stream_to_zarr is genuinely segmented and bounded, but should_stream only routes to it above STREAM_MIN_BYTES = 2 GiB for .vhdr/.fif/.ds/.mefd, while KIT and EDF already stream above 256 MB. on004917's recordings sat at 1.18-2.25 GB, straddling that threshold, so 23 of 24 took the unbounded path. There is no principled reason BrainVision needs a threshold 8x higher than EDF: the driver's own comment notes BrainVision goes through MNE on both paths, so parity holds either way.
EEGLAB .set appears in no STREAM_*_EXTS tuple at all, though stream_to_zarr's docstring lists .set among the formats it reads. A sufficiently large .set is the case where "no machine can handle it" is literally true today: it cannot stream, and the in-memory projection would eventually skip it outright.
Scope
Lower STREAM_MIN_BYTES in code to match the KIT/EDF 256 MB, and retire the ZARR_STREAM_MIN_BYTES crontab override added as mitigation on 2026-08-22.
Verify EEGLAB .set streaming has importer parity with the in-memory path under biosigio >= 1.2.4, then route it. If parity does not hold, record why and leave .set on the guarded in-memory path.
Re-examine whether the in-memory path should survive at all, or exist only below a small-file fast-path threshold.
ADR: bounded streaming is the default conversion path; in-memory is the exception, with the parity requirement that gates each format.
Definition of done
Sub-2-GiB BrainVision converts via the bounded path, producing a store byte-identical to the in-memory path on a fixture.
.set either streams with demonstrated parity, or its exclusion is recorded with evidence.
The crontab carries no ZARR_STREAM_MIN_BYTES override.
STREAM_MIN_BYTES drops from 2 GiB to 256 MiB, making bounded streaming the default path for anything non-trivial (ADR 0030). The streaming projection is channel-aware rather than a flat multiple of file size.
.set is deliberately excluded and the exclusion is evidenced, not assumed: MNE's EEGLAB reader takes the is_embedded branch to _readmat(preload=True) for single-file .set, so there is no bounded read to stream, and MATLAB v7.3 HDF5 .set is refused by MNE outright. EEGLAB datasets therefore keep the in-memory path with the wider hardware ceiling rather than being buried as permanently too large.
Activity
Do not charge in-memory slack to streamed recordings
Address Phase 4 review findings
Make the streaming projection channel-aware
Phase 4: Make bounded streaming the default conversion path (#1117)
neuromechanist commentedon Aug 22, 2026
Done in #1117, delivered to
devvia #1119.STREAM_MIN_BYTESdrops from 2 GiB to 256 MiB, making bounded streaming the default path for anything non-trivial (ADR 0030). The streaming projection is channel-aware rather than a flat multiple of file size..setis deliberately excluded and the exclusion is evidenced, not assumed: MNE's EEGLAB reader takes theis_embeddedbranch to_readmat(preload=True)for single-file.set, so there is no bounded read to stream, and MATLAB v7.3 HDF5.setis refused by MNE outright. EEGLAB datasets therefore keep the in-memory path with the wider hardware ceiling rather than being buried as permanently too large.