Repository navigation
Description
Expand the Zarr serving copy to formats it silently ignores today,
restrict it to Brain Imaging Data Structure (BIDS) raw recordings,
document the contract,
and diagnose the datasets that publish no index at all.
Five tracks, each independently shippable, spanning four repositories.
They are correlated because they share one root cause surface:
the converter's notion of "what counts as a recording" is both too narrow (formats) and too broad (paths).
Measured evidence (all 200 catalog datasets, live indices plus D1)
Coverage was measured by fetching https://zarr.nemar.org/<id>/zarr/index.json for every dataset
and cross-referencing the zarr_* columns in D1.
Formats silently ignored. PRIMARY_EXTS in generate_zarr.py omits Multiscale Electrophysiology
Format version 3 (MEF3). Dataset on006392 carries one .mefd recording with 194 channels and
reports zarr_status=ready with store_count=0: a conversion that reports success, records no
failure, and produces nothing. This is the worst failure mode because nothing surfaces it.
4D Neuroimaging/BTi, Neurodata Without Borders (NWB), and ITAB match zero datasets today.
Scope too broad. 3,270 of 3,771 reported conversion failures (86.7%) come from derivatives/,
sourcedata/, or code/ paths, for files that were never meant to be viewable recordings
(independent component analysis solutions, epoched and trial-averaged derivatives, raw sourcedata).
Separately, 4,721 live stores (12% of 39,433) were built from those same non-raw paths and are
being served to the viewer. The records pipeline (emit_records.py) already excludes derivatives/;
the Zarr converter never did.
A dispatch gap. The backend's ZARR_DATA_EXTENSIONS omits con, sqd, and kdf, so a push
that changes a KIT/Yokogawa magnetoencephalography (MEG) recording never re-dispatches conversion
and the serving copy goes stale silently. Dataset on007763 has 35 affected .con recordings.
Whole-dataset failures. 11 MEG/intracranial electroencephalography (iEEG) datasets publish no
index because every recording in them fails, roughly 2,300 recordings. The failures are uniform per
dataset rather than scattered, which points at a systematic cause rather than per-file corruption:
| Dataset | Failing recordings | Code |
|---|---|---|
| on006502 | 380 (CTF .ds) |
file_read_error |
| on006910 | 384 | file_read_error |
| on006914 | 353 | corrupt_or_truncated |
| on006233 | 347 | corrupt_or_truncated |
| on005545 | 336 | file_read_error |
| on006720 | 246 | file_read_error |
| on006012 | 193 | file_read_error |
| nm000276 | 40 | infra (non-deterministic) |
| on005931 | 16 | file_read_error |
| on005670 | 2 | infra (non-deterministic) |
| on005356 | pending, 30 stores | stale queue state |
on006233 and on006914 both report exactly 347 corrupt_or_truncated, which is unlikely to be
coincidence.
Upstream support already exists
MNE-Python 1.12 ships mne.io.read_raw_mef and maps the .mefd extension to it natively, requiring
the pymef package, and supports preload=False so bounded-memory streaming works.
mne.io.read_raw_bti also exists. No new reader needs to be written for either format.
biosigio currently pins mne>=1.6.
Tracks
- Track 1, biosigio: MEF3 and 4D/BTi importers, streaming,
pymefdependency,mne>=1.12runtime gate. - Track 2, nemarDatasets/.github: generalize CTF
.dsdirectory handling to.mefdand BTi; restrict discovery to BIDS raw. Blocked on Track 1. - Track 3, nemar-cli: fix the
ZARR_DATA_EXTENSIONSgap, add the new trigger paths, exclude non-raw paths, and record the raw-only scope as an Architecture Decision Record. - Track 4, nemar/docs: Zarr contract developer reference, deliberately not promoted in the sidebar.
- Track 5, investigation: diagnose the 11 whole-dataset failures.
Note that Track 2 and Track 4 land in other repositories, so their pull requests target those
repositories' integration branches rather than an epic branch here. This issue and its sub-issues
are the correlation record.
Activity
Fix zarr dispatch gate: KIT MEG gap, new formats, raw-only scope (#1101)
Restrict Zarr converter to BIDS raw; fix scratch-path leak (#98)
Add guarded purge tool for already-published non-raw Zarr stores (#99)
neuromechanist commentedon Aug 22, 2026
Closeout audit (2026-08-22)
Most of this epic has landed; the sub-issues were never closed. Audited each against merged work and closed the ones that are done.
Closed as landed:
biosigio[zarr,meg,mef3,hdf5]>=1.2.4.github#98, #100MEFD_EXT,bti_recordings,EXCLUDED_TREESpresent onmaincon/sqd/kdfinZARR_DATA_EXTENSIONSondev; ADR 0027.github#98fix_source_file_attrpresentStill open:
nemar/docs. No evidence of work.BrokenProcessPoolcascade documented in epic Epic: Zarr conversion memory robustness #1108, coveringnm000276,on005670, andon005356. Thefile_read_error/corrupt_or_truncatedclusters are a separate cause this track still owns; see the comment there.generate_zarr.pyinto nemar-cli and closes Nothing enforces the zarr gate stays a superset of the converter's PRIMARY_EXTS #1103 there with a same-repo unit test.main, not fixed by dataset status/list don't authenticate, hiding private datasets #98 or Reorganize S3 bucket to subdirectory layout #100.Relationship to epic #1108
New epic #1108 "Zarr conversion memory robustness" runs alongside this one. The split:
#1108 Phase 1 relocates
scripts/zarr/fromnemarDatasets/.githubinto nemar-cli, which turns this epic's remaining code work (#1103, #1106) into single-repo changes with CI. This epic is driven to closure after #1108.neuromechanist commentedon Aug 22, 2026
All eight tracks closed.
.mefdand 4D/BTi importersZARR_DATA_EXTENSIONSgap closed; raw-only dispatchdocs.nemar.org/develop/zarr-contract/ADRs from this epic: 0027 (zarr dispatch is raw-only) and 0028 (MaxShield MEG is SSS-filtered before serving, or declined).
What Track 5 actually turned out to be
Worth recording, because the issue's leading hypothesis and my own first answer were both wrong. The hypothesis was that annexed content had never been retrieved; in fact 8 of 11 datasets are
data_complete=1with tens of GB present, and a sampled recording pulls exactly the byte count its annex key names. I then predicted the CTF and MEG failures were covered by the Track 2 and 3 format work. Converting real samples falsified that too.Three unrelated causes:
pending, not failing; the counts in the issue body were historical.Both follow-ups are new work rather than epic leftovers, so they are tracked on their own issues instead of hiding inside a diagnosis ticket.
Related
Epic #1108 (conversion memory robustness) delivered alongside this and moved the conversion engine into this repo, which is what made #1103 and #1106 single-repo work with CI. Remaining operational steps are in #1122.
Exclude recorded failures from the backfill sweep
Exclude recorded failures from the backfill sweep
Zarr queue: engine-version stamping + directory-format backfill (#1184)