Skip to content

Epic: Zarr availability and hygiene - formats, raw-only scope, whole-dataset failures #1095

Closed
8 of 8 issues completed

Description

@neuromechanist

Expand the Zarr serving copy to formats it silently ignores today,
restrict it to Brain Imaging Data Structure (BIDS) raw recordings,
document the contract,
and diagnose the datasets that publish no index at all.

Five tracks, each independently shippable, spanning four repositories.
They are correlated because they share one root cause surface:
the converter's notion of "what counts as a recording" is both too narrow (formats) and too broad (paths).

Measured evidence (all 200 catalog datasets, live indices plus D1)

Coverage was measured by fetching https://zarr.nemar.org/<id>/zarr/index.json for every dataset
and cross-referencing the zarr_* columns in D1.

Formats silently ignored. PRIMARY_EXTS in generate_zarr.py omits Multiscale Electrophysiology
Format version 3 (MEF3). Dataset on006392 carries one .mefd recording with 194 channels and
reports zarr_status=ready with store_count=0: a conversion that reports success, records no
failure, and produces nothing. This is the worst failure mode because nothing surfaces it.
4D Neuroimaging/BTi, Neurodata Without Borders (NWB), and ITAB match zero datasets today.

Scope too broad. 3,270 of 3,771 reported conversion failures (86.7%) come from derivatives/,
sourcedata/, or code/ paths, for files that were never meant to be viewable recordings
(independent component analysis solutions, epoched and trial-averaged derivatives, raw sourcedata).
Separately, 4,721 live stores (12% of 39,433) were built from those same non-raw paths and are
being served to the viewer. The records pipeline (emit_records.py) already excludes derivatives/;
the Zarr converter never did.

A dispatch gap. The backend's ZARR_DATA_EXTENSIONS omits con, sqd, and kdf, so a push
that changes a KIT/Yokogawa magnetoencephalography (MEG) recording never re-dispatches conversion
and the serving copy goes stale silently. Dataset on007763 has 35 affected .con recordings.

Whole-dataset failures. 11 MEG/intracranial electroencephalography (iEEG) datasets publish no
index because every recording in them fails, roughly 2,300 recordings. The failures are uniform per
dataset rather than scattered, which points at a systematic cause rather than per-file corruption:

Dataset Failing recordings Code
on006502 380 (CTF .ds) file_read_error
on006910 384 file_read_error
on006914 353 corrupt_or_truncated
on006233 347 corrupt_or_truncated
on005545 336 file_read_error
on006720 246 file_read_error
on006012 193 file_read_error
nm000276 40 infra (non-deterministic)
on005931 16 file_read_error
on005670 2 infra (non-deterministic)
on005356 pending, 30 stores stale queue state

on006233 and on006914 both report exactly 347 corrupt_or_truncated, which is unlikely to be
coincidence.

Upstream support already exists

MNE-Python 1.12 ships mne.io.read_raw_mef and maps the .mefd extension to it natively, requiring
the pymef package, and supports preload=False so bounded-memory streaming works.
mne.io.read_raw_bti also exists. No new reader needs to be written for either format.
biosigio currently pins mne>=1.6.

Tracks

  • Track 1, biosigio: MEF3 and 4D/BTi importers, streaming, pymef dependency, mne>=1.12 runtime gate.
  • Track 2, nemarDatasets/.github: generalize CTF .ds directory handling to .mefd and BTi; restrict discovery to BIDS raw. Blocked on Track 1.
  • Track 3, nemar-cli: fix the ZARR_DATA_EXTENSIONS gap, add the new trigger paths, exclude non-raw paths, and record the raw-only scope as an Architecture Decision Record.
  • Track 4, nemar/docs: Zarr contract developer reference, deliberately not promoted in the sidebar.
  • Track 5, investigation: diagnose the 11 whole-dataset failures.

Note that Track 2 and Track 4 land in other repositories, so their pull requests target those
repositories' integration branches rather than an epic branch here. This issue and its sub-issues
are the correlation record.

Activity

  1. neuromechanist commented on Aug 22, 2026

    @neuromechanist
    ContributorAuthor

    Closeout audit (2026-08-22)

    Most of this epic has landed; the sub-issues were never closed. Audited each against merged work and closed the ones that are done.

    Closed as landed:

    Issue Satisfied by Verified
    #1096 Track 1: biosigio MEF3 + BTi biosigio 1.2.4 released pinned as biosigio[zarr,meg,mef3,hdf5]>=1.2.4
    #1097 Track 2: converter dir formats + raw-only .github #98, #100 MEFD_EXT, bti_recordings, EXCLUDED_TREES present on main
    #1098 Track 3: dispatch gate #1101, #1105 con/sqd/kdf in ZARR_DATA_EXTENSIONS on dev; ADR 0027
    #1102 scratch paths in stores .github #98 fix_source_file_attr present

    Still open:

    Relationship to epic #1108

    New epic #1108 "Zarr conversion memory robustness" runs alongside this one. The split:

    #1108 Phase 1 relocates scripts/zarr/ from nemarDatasets/.github into nemar-cli, which turns this epic's remaining code work (#1103, #1106) into single-repo changes with CI. This epic is driven to closure after #1108.

  2. neuromechanist commented on Aug 22, 2026

    @neuromechanist
    ContributorAuthor

    All eight tracks closed.

    Track Issue Outcome
    1 #1096 biosigIO 1.2.4 pinned: MEF3 .mefd and 4D/BTi importers
    2 #1097 Directory formats discovered; BIDS raw-only scope
    3 #1098 ZARR_DATA_EXTENSIONS gap closed; raw-only dispatch
    4 #1099 Contract reference at docs.nemar.org/develop/zarr-contract/
    5 #1100 Diagnosed with evidence for all eleven datasets
    6 #1102 Conversion scratch paths no longer embedded in stores
    7 #1103 Gate/converter superset now enforced by a unit test
    8 #1106 Sidecar edits rebuild CTF, MEF3, and BTi recordings

    ADRs from this epic: 0027 (zarr dispatch is raw-only) and 0028 (MaxShield MEG is SSS-filtered before serving, or declined).

    What Track 5 actually turned out to be

    Worth recording, because the issue's leading hypothesis and my own first answer were both wrong. The hypothesis was that annexed content had never been retrieved; in fact 8 of 11 datasets are data_complete=1 with tens of GB present, and a sampled recording pulls exactly the byte count its annex key names. I then predicted the CTF and MEG failures were covered by the Track 2 and 3 format work. Converting real samples falsified that too.

    Three unrelated causes:

    Both follow-ups are new work rather than epic leftovers, so they are tracked on their own issues instead of hiding inside a diagnosis ticket.

    Related

    Epic #1108 (conversion memory robustness) delivered alongside this and moved the conversion engine into this repo, which is what made #1103 and #1106 single-repo work with CI. Remaining operational steps are in #1122.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions