Skip to content

Phase 4: Widen the bounded streaming path #1112

Closed

Description

@neuromechanist

Part of epic #1108.

Make the bounded streaming path the default and close the formats it cannot reach.

Why

stream_to_zarr is genuinely segmented and bounded, but should_stream only routes to it above STREAM_MIN_BYTES = 2 GiB for .vhdr/.fif/.ds/.mefd, while KIT and EDF already stream above 256 MB. on004917's recordings sat at 1.18-2.25 GB, straddling that threshold, so 23 of 24 took the unbounded path. There is no principled reason BrainVision needs a threshold 8x higher than EDF: the driver's own comment notes BrainVision goes through MNE on both paths, so parity holds either way.

EEGLAB .set appears in no STREAM_*_EXTS tuple at all, though stream_to_zarr's docstring lists .set among the formats it reads. A sufficiently large .set is the case where "no machine can handle it" is literally true today: it cannot stream, and the in-memory projection would eventually skip it outright.

Scope

  • Lower STREAM_MIN_BYTES in code to match the KIT/EDF 256 MB, and retire the ZARR_STREAM_MIN_BYTES crontab override added as mitigation on 2026-08-22.
  • Verify EEGLAB .set streaming has importer parity with the in-memory path under biosigio >= 1.2.4, then route it. If parity does not hold, record why and leave .set on the guarded in-memory path.
  • Re-examine whether the in-memory path should survive at all, or exist only below a small-file fast-path threshold.
  • ADR: bounded streaming is the default conversion path; in-memory is the exception, with the parity requirement that gates each format.

Definition of done

  • Sub-2-GiB BrainVision converts via the bounded path, producing a store byte-identical to the in-memory path on a fixture.
  • .set either streams with demonstrated parity, or its exclusion is recorded with evidence.
  • The crontab carries no ZARR_STREAM_MIN_BYTES override.

Activity

  1. neuromechanist commented on Aug 22, 2026

    @neuromechanist
    ContributorAuthor

    Done in #1117, delivered to dev via #1119.

    STREAM_MIN_BYTES drops from 2 GiB to 256 MiB, making bounded streaming the default path for anything non-trivial (ADR 0030). The streaming projection is channel-aware rather than a flat multiple of file size.

    .set is deliberately excluded and the exclusion is evidenced, not assumed: MNE's EEGLAB reader takes the is_embedded branch to _readmat(preload=True) for single-file .set, so there is no bounded read to stream, and MATLAB v7.3 HDF5 .set is refused by MNE outright. EEGLAB datasets therefore keep the in-memory path with the wider hardware ceiling rather than being buried as permanently too large.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureNew feature or enhancementzarr

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions