Skip to content

Track 5: diagnose 11 datasets publishing no Zarr index #1100

Closed

Description

@neuromechanist

Eleven magnetoencephalography (MEG) and intracranial electroencephalography (iEEG) datasets publish no index at all because every recording in them fails, roughly 2,300 recordings. This is the single largest Zarr availability gap, far larger than the format gaps in Tracks 1 to 3.

Dataset Failing Code Deterministic
on006502 380 (CTF .ds) file_read_error yes
on006910 384 file_read_error yes
on006914 353 corrupt_or_truncated (347) + file_read_error (6) yes
on006233 347 corrupt_or_truncated yes
on005545 336 file_read_error no
on006720 246 file_read_error yes
on006012 193 file_read_error yes
nm000276 40 infra no
on005931 16 file_read_error yes
on005670 2 infra no
on005356 pending, 30 stores stale queue state n/a

The failures are uniform per dataset rather than scattered, which points at a systematic cause rather than per-file corruption. on006233 and on006914 both report exactly 347 corrupt_or_truncated, which is unlikely to be coincidence.

Leading hypothesis: the annexed content was never retrieved on the conversion host, so biosigio read a git-annex symlink or an empty placeholder instead of real data, which would surface as exactly these two codes. This is a hypothesis, not a finding, and must be confirmed against evidence on the conversion host before any fix.

Deliverable is a diagnosis with evidence, then a fix or a tracked follow-up. The fix may well be operational (retrieval, re-queue) rather than a code change. Note on006502 is entirely CTF .ds, so if retrieval is the cause it would also explain why a directory format fails wholesale.

Part of epic #1095.

Activity

  1. neuromechanist commented on Aug 20, 2026

    @neuromechanist
    ContributorAuthor

    Diagnosis complete: four distinct root causes, all recoverable

    The leading hypothesis in the issue body was wrong and is refuted.
    The annexed content is not missing:
    data_complete=1 and bytes_present is within 0.1% of file_size for every affected dataset,
    and I verified byte-exactness directly.
    For a failing on006233 recording the downloaded object is 4,211,536 bytes
    with sha256 4a6696...336598,
    both matching the git-annex key exactly.
    The data is intact.

    In every case the signal data is fully readable.
    The rejection comes either from a reader that is stricter than the format's real-world variation,
    or from metadata that is irrelevant to serving a time series.

    1. EDF rejected by pyedflib but read fine by MNE (1,084 recordings)

    biosigio's in-memory EDF path reads via pyedflib. Two separate real-world deviations make it refuse:

    Non-compliant header padding (on006233 347, on006914 353, reported corrupt_or_truncated).
    The "number of data records" field contains b'421\x00', NUL-padded instead of space-padded as the
    European Data Format (EDF) specification requires.
    pyedflib raises OSError: the file is not EDF(+) or BDF(+) compliant (Number of Datarecords).
    MNE reads the same file without complaint: 4 channels, 421,000 samples, 1000 Hz.
    Note these two datasets are NOT duplicate imports as I first suspected; they are distinct OpenNeuro
    accessions (ds006233 "Picture naming", ds006914 "Visual Naming EC") from what appears to be the
    same lab, which is why their subject, session, and task naming coincide.

    Discontinuous EDF+D (on006910 384, reported file_read_error).
    pyedflib raises OSError: The file is discontinuous and cannot be read.
    MNE reads it: 66 channels, 295,410 samples, 1000 Hz. This file's header is clean (zero NUL bytes).

    2. Elekta MaxShield FIF (439 recordings)

    on006012 (193) and on006720 (246), both reported file_read_error.
    mne.io.read_raw_fif refuses with
    ValueError: This file contains raw Internal Active Shielding data... Use allow_maxshield=True if you are sure you want to load the data.
    With allow_maxshield=True both read fine (on006012: 336 channels, 341,000 samples;
    on006720: 328 channels, 127,000 samples, 1000 Hz).

    This one is a judgment call, not just a bug: MNE warns the data may be distorted and that Elekta
    recommends running MaxFilter first. Serving unfiltered Internal Active Shielding data in a viewer
    without saying so would be misleading. See the recommendation below.

    The zarr_queue.py docstring already cites "a MaxShield MEG .fif" as its canonical example of a
    deterministic data failure, so this was known, but it was treated as terminal rather than handled.

    3. CTF .hc fiducial naming dialect (380 recordings)

    on006502, reported file_read_error. Fails in _make_ctf_coord_trans_set with
    RuntimeError: Some of the mandatory HPI device-coordinate info was not there.

    Root cause is in MNE's .hc parser (mne/io/ctf/hc.py). Its _kind_dict matches the lowercase
    literals "nasion", "left ear", "right ear", and the comparison is case-sensitive with no
    aliases. This dataset's .hc writes Nasion, LPA, and RPA, so all three fiducials come back
    unmatched (kind = -1) and the transform build fails.

    Proof: renaming Nasion to nasion, LPA coil to left ear coil, and RPA coil to
    right ear coil in a copy of the .hc makes the identical directory read successfully:
    306 channels, 197,800 samples, 600 Hz. All ten .ds members were verified byte-exact against
    their annex keys first, so this is not a data problem.

    Worth emphasizing: this is a head-coil coordinate transform, used for source localization and sensor
    geometry. It has no bearing on serving a time series. A whole 380-recording dataset is withheld over
    metadata the serving copy never uses.

    4. Side finding: BIDS special files treated as recordings

    on006012 and on006720 both report failures on _acq-crosstalk_meg.fif.
    The Brain Imaging Data Structure reserves _acq-crosstalk_meg.fif and _acq-calibration_meg.dat
    as Elekta calibration files, not recordings. Reading the crosstalk file fails with
    ValueError: Could not find measurement data, correctly, because it contains none.
    These should be excluded by path convention rather than attempted and reported as failures.
    This belongs with the raw-only scope work in #1097 and #1098.

    Recoverable total

    Cause Datasets Recordings
    EDF strict reader (pyedflib) on006233, on006914, on006910 1,084
    MaxShield FIF on006012, on006720 439
    CTF .hc dialect on006502 380
    Total 6 of 11 1,903

    Not yet diagnosed: on005545 (336, non-deterministic), on005931 (16), nm000276 (40, infra),
    on005670 (2, infra), on005356 (stale pending queue row with 30 stores).

    Recommendations

    1. Add an MNE fallback to biosigio's EDF path. Highest value at 1,084 recordings. Important
      constraint: generate_zarr.py documents that MNE rescales EDF to SI volts while pyedflib does
      not, so a fallback must normalize units or the in-memory and streaming paths will disagree on the
      same file. This needs a unit-parity test, not just a try/except.
    2. Decide the MaxShield policy (439 recordings). Either convert with allow_maxshield=True and
      surface an explicit caveat in the index so the viewer can label it as unfiltered Internal Active
      Shielding data, or keep declining but report an honest "requires MaxFilter" reason instead of a
      generic file_read_error. I lean toward converting with the caveat, since the alternative shows
      users nothing at all, but this is a scientific-communication call rather than a technical one.
    3. Handle the CTF .hc dialect. The clean fix is upstream: make MNE's _kind_dict matching
      case-insensitive and add LPA/RPA aliases, which is a small, well-evidenced patch worth
      sending to MNE. In the meantime the converter can normalize a scratch copy of the .hc.
    4. Exclude the BIDS calibration files as part of Track 2: generate_zarr.py directory formats and BIDS-raw-only discovery #1097 and Track 3: fix ZARR_DATA_EXTENSIONS gap and raw-only dispatch #1098.

    Reproduction for all of the above uses public S3 objects
    (https://s3.us-east-2.amazonaws.com/nemar/<dataset>/objects/<annex-key>),
    with annex keys read from the git symlink blobs, so any of this can be re-checked independently.

  2. neuromechanist commented on Aug 20, 2026

    @neuromechanist
    ContributorAuthor

    Update: the failure picture was distorted, and there are two more root causes

    Two corrections to my earlier comment, then two new findings.

    Correction 1: raw coverage is actually 98.86%, not alarming

    My earlier framing leaned on total failure counts, which are dominated by non-raw paths. Recomputed
    ignoring derivatives/, sourcedata/, and code/ entirely, the picture is much better:

    Metric Value
    Raw recordings converted 34,712
    Raw recordings failed 401
    Raw success rate 98.86%
    Datasets with no raw failures at all 150
    Datasets with some raw failures 27
    Datasets showing the viewer nothing real 23

    So the honest statement of the gap is not "thousands of failures", it is "23 datasets show the user
    nothing." That is the number worth fixing.

    Correction 2: my count of all-fail datasets was an UNDERCOUNT

    I previously found 11 by looking for datasets with no published index. That missed a case: a dataset
    whose raw recordings ALL fail can still publish an index if it has derivatives/ stores, so it looks
    healthy from the outside. on005520 is exactly that — 92 stores, every one from
    derivatives/preprocessed/, and 67 of 67 raw BrainVision recordings failing.

    This matters for the raw-only scope change: excluding derivatives/ will not "remove on005520's
    coverage", it will reveal that the dataset has no working raw coverage at all. The derivatives
    stores were masking a total raw failure. Worth stating plainly so the change is not mistaken for a
    regression when that dataset goes from 92 stores to 0.

    I also checked the concern about sourcedata/-only datasets before anything gets deleted: there are
    none. All four datasets with sourcedata/ stores (on005178, on005185, on005207, on005403)
    also have healthy raw coverage, so excluding sourcedata/ costs no dataset its viewer copy.
    on005520 is the single at-risk dataset, and it is at risk from derivatives/, not sourcedata/.

    New root cause 5: BrainVision header pointers not updated after BIDS renaming (~470 recordings)

    The largest remaining cluster. on005520's .vhdr contains:

    DataFile=sub-01_task-resting_eye_closed_eeg.eeg
    MarkerFile=sub-01_task-resting_eye_closed_eeg.vmrk
    

    while the files actually on disk are sub-01_task-restingeyeclosed_eeg.eeg / .vmrk. BIDS forbids
    underscores inside a label value, so the conversion to BIDS renamed the files and never rewrote the
    header's internal pointers. MNE therefore raises FileNotFoundError looking for a file that does not
    exist, which surfaces as file_read_error.

    Proof: rewriting only those two lines to the real sibling filenames makes the identical data read
    cleanly, 67 channels, 417,997 samples, 1000 Hz, values in a sane microvolt range. All three triplet
    members were verified byte-exact against their annex keys first, so the data is intact.

    Confirmed .vhdr file_read_error datasets: on005863 (357), on005520 (67), on007471 (32),
    on007813 (4), and on007406 (10, extension not individually confirmed) — roughly 470 recordings.

    Note this is a genuine defect in the DATASETS, not in our converter: the header really is wrong.
    Two options, and I would do both:

    1. Repair on the scratch copy at conversion time. The pointers are unambiguous, since the correct
      targets are the same-stem siblings sitting next to the header, so the rewrite is deterministic and
      safe. The scratch copy is discarded and BIDS stays the source of truth, so nothing authoritative
      is mutated. This recovers ~470 recordings without waiting on anyone.
    2. Report upstream, because the datasets are genuinely malformed and every other tool reading them
      hits the same wall. Arguably the BIDS validator should flag a .vhdr whose DataFile does not
      resolve.

    Still undiagnosed

    on007509 (70 EEGLAB .set, file_read_error) is a separate cause I have not chased.
    on006033 (2) and on006269/on008082 (non-deterministic, so infra) remain open.
    on005571, on005692, and on005873 are not failures at all: they sit at zarr_status=pending with
    no stores, which looks like stale queue rows rather than anything wrong with the data.

    Revised recoverable total

    Cause Recordings
    European Data Format, strict pyedflib rejection 1,084
    Elekta MaxShield magnetoencephalography 439
    BrainVision header pointers ~470
    CTF .hc fiducial dialect 380
    Total ~2,373

    Every one of these is a reader-strictness or metadata problem on data that is byte-intact and
    readable. None is a data-loss problem.

  3. neuromechanist commented on Aug 20, 2026

    @neuromechanist
    ContributorAuthor

    Final: complete root-cause taxonomy, and a correction to my own headline number

    Correcting myself: catalog-wide raw coverage is 91.11%, not 98.86%

    My 98.86% figure was computed only over datasets that have a published index, which silently
    excluded exactly the worst cases: the 22 datasets where every recording fails publish no index at all,
    so they contributed nothing to that denominator. Including them:

    Recordings
    Raw converted successfully 34,712
    Raw failing in datasets with an index 401
    Raw failing in no-index datasets 2,988
    Total raw recordings 38,101
    Catalog-wide raw coverage 91.11%

    Of the 2,988 in no-index datasets, 2,392 are flagged deterministic (reproducible, therefore
    diagnosable and fixable) and 596 are non-deterministic infrastructure failures. Four further datasets
    (on005356, on005571, on005692, on005873) are not failures at all: they sit at
    zarr_status=pending with no stores, which looks like stale queue rows and is a separate, probably
    cheap, fix.

    So roughly 2,793 recordings are fixable through the six causes below, which would take raw coverage
    from 91.11% to about 98.4%.

    The six root causes

    Every one of these is a reader-strictness or metadata problem on data that is byte-intact. I verified
    byte-exactness against git-annex SHA256E keys before diagnosing each. No data loss anywhere.

    1. European Data Format rejected by pyedflib though MNE reads it (1,084 recordings).
    Two variants. Non-compliant NUL-padded header field: the "number of data records" field contains
    b'421\x00' instead of space padding, so pyedflib raises
    the file is not EDF(+) or BDF(+) compliant (Number of Datarecords) while MNE reads it fine
    (on006233 347, on006914 353). Discontinuous EDF+D: pyedflib raises
    The file is discontinuous and cannot be read; MNE reads 66 channels / 295,410 samples
    (on006910 384). Fix: fall back to MNE when pyedflib refuses. Constraint already documented in
    generate_zarr.py: MNE rescales EDF to SI volts and pyedflib does not, so the fallback needs a
    unit-parity test, not a bare try/except.

    2. Elekta MaxShield / Internal Active Shielding FIF (439 recordings).
    read_raw_fif refuses without allow_maxshield=True; with it, on006012 reads 336 channels and
    on006720 328 channels. Still needs a human decision, because MNE warns the data may be distorted
    and Elekta recommends MaxFilter first: convert with an explicit caveat surfaced in the index, or keep
    declining but report an honest "requires MaxFilter" reason rather than a generic file_read_error.

    3. BrainVision header pointers not updated after BIDS renaming (~470 recordings).
    .vhdr says DataFile=sub-01_task-resting_eye_closed_eeg.eeg while the file on disk is
    sub-01_task-restingeyeclosed_eeg.eeg, because BIDS forbids underscores inside a label value and the
    conversion renamed files without rewriting the header. MNE raises FileNotFoundError. Rewriting only
    those two lines reads 67 channels / 417,997 samples / 1000 Hz. Datasets: on005863 (357),
    on005520 (67), on007471 (32), on007813 (4), on007406 (10), plus 88 more spread across
    on005273, on005274, on005486, on006576 and others. This is a defect in the datasets;
    worth both a deterministic scratch-copy repair (targets are unambiguous same-stem siblings) and an
    upstream report.

    4. CTF .hc fiducial naming dialect (380 recordings).
    on006502. MNE's mne/io/ctf/hc.py _kind_dict matches lowercase "nasion", "left ear",
    "right ear" case-sensitively with no aliases; this dataset writes Nasion, LPA, RPA, so all
    three fiducials go unmatched and _make_ctf_coord_trans_set raises
    Some of the mandatory HPI device-coordinate info was not there. Normalizing the names reads
    306 channels / 197,800 samples / 600 Hz. Note this is a head-coil coordinate transform used for source
    localization; it has no bearing on serving a time series, so a whole 380-recording dataset is withheld
    over metadata the serving copy never touches. Best fixed upstream in MNE (case-insensitive matching
    plus LPA/RPA aliases) — a small, well-evidenced patch.

    5. MATLAB v7.3 EEGLAB .set files (219 recordings) — NEW, and the largest single cluster in the
    indexed datasets.

    MNE's read_raw_eeglab raises NotImplementedError: Please use HDF reader for matlab v7.3 files, e.g. h5py, and biosigio surfaces that as FileReadError. A v7.3 .set is an HDF5 container, not a
    classic MAT file, so scipy.io.loadmat cannot read it either. Affects 7 datasets: on005178,
    on005185, on005343, on006040, on006554, and others.

    I confirmed the data is fully reachable, so this is implementable rather than a dead end. Opening
    on005343's 557 MB sub-S10_task-resting_eeg.set with h5py gives EEG/data as a
    (1236409, 129) float32 array (HDF5 stores it transposed relative to MATLAB), nbchan=129,
    srate=1000, pnts=1236409, trials=1 (continuous), and channel labels resolving through the
    #refs# table to E1...E129. Fix belongs in biosigio as an h5py-based reader for v7.3 .set,
    selected by sniffing the MATLAB 7.3 MAT-file magic. Watch the transpose and the object-reference
    indirection for chanlocs.labels.

    6. BIDS calibration files treated as recordings.
    _acq-crosstalk_meg.fif and _acq-calibration_meg.dat are Elekta calibration files, not recordings.
    Reading the crosstalk file raises ValueError: Could not find measurement data, correctly, because it
    contains none. Already folded into the raw-only scope work in #1097.

    Still open

    on005545 (336, flagged non-deterministic despite the volume, so worth a second look),
    on008082 (173, infra), nm000276 and on006269 (40 each, infra), on006033 (5), on005670 (2),
    and the 4 stale pending rows.

  4. neuromechanist commented on Aug 21, 2026

    @neuromechanist
    ContributorAuthor

    MaxShield policy settled: filter properly when we can, decline honestly when we cannot

    Decision: apply Signal-Space Separation only when both the Elekta fine-calibration and cross-talk
    files are available, and otherwise decline with the same message MNE itself gives, rather than
    serving unfiltered Internal Active Shielding data.

    This turns out to fit the data almost exactly, and it is better than either option I originally put
    forward. We do not need MEGIN's proprietary MaxFilter: mne.preprocessing.maxwell_filter is an
    open-source implementation of the same Signal-Space Separation family, documented as broadly
    comparable to MEGIN MaxFilter 2.2.11.

    Verified on real data

    Ran it against on006012's sub-01_task-POGS_run-06_meg.fif using that dataset's own
    sub-01_acq-calibration_meg.dat and sub-01_acq-crosstalk_meg.fif:

    • Filtering succeeded, 336 channels preserved.
    • info["maxshield"] flips from True to False. This is the strongest available signal: MNE's
      own guardrail, the thing that was refusing to load these files, is satisfied by the result. The
      output is no longer raw Internal Active Shielding data by MNE's own reckoning.
    • Root-mean-square amplitude drops to 62.7% of the unfiltered signal, consistent with interference
      being removed rather than the data being passed through untouched.
    • The Signal-Space Separation parameters are written into the data's own proc_history
      (in_order=8, out_order=3, nfree=72, frame=head, plus both sss_cal and sss_ctc records),
      so the processing is self-documenting and auditable afterwards.

    Temporal Signal-Space Separation (st_duration) also works, and filtering without the calibration
    pair works too, but under this policy we will not use that second path.

    The files we need are already in the datasets

    Every subject in both affected datasets ships both files:

    Dataset Subjects with both files Recordings recoverable Declined
    on006012 20 of 20 158 15
    on006720 23 of 23 207 16
    Total 365 31

    Every single declined recording is sub-emptyroom. Those are empty-room noise reference
    measurements rather than subject data, so declining them is the right outcome on its own merits, not
    a shortfall. No actual subject recording is lost to this policy.

    Correcting my earlier count

    I previously reported 439 recordings for this cause, taken from the zarr_errors column. That figure
    included the 43 calibration and cross-talk files themselves (20 plus 23), which were being attempted
    as recordings and are now excluded as the calibration files they are. The real breakdown is 396
    _meg.fif recordings, of which 365 are recoverable and 31 are empty-room.

    One consequence worth stating plainly

    A Signal-Space Separation filtered store is not a faithful copy of its source recording, unlike
    every other store in the serving copy. It is a processed derivative. That has to be recorded in the
    index so consumers know, and it matters especially for the machine-learning streaming path, where a
    model trained across datasets would otherwise silently mix filtered and unfiltered magnetoencephalography.
    The proc_history provenance makes the processing auditable, but the index needs to say it too.

    This is a decision worth an Architecture Decision Record, since it changes what the serving copy
    guarantees for one class of recording and would be expensive to reverse quietly.

  5. neuromechanist commented on Aug 21, 2026

    @neuromechanist
    ContributorAuthor

    Correction: my whole analysis covered 26% of the catalog

    Every count I reported in this issue was computed over 200 datasets. The catalog has 754.
    GET /datasets caps limit at 200 and paginates by offset, and I read the first page as if it were
    everything without checking total_count, which the response carries and which says 754.

    The root-cause mechanisms below are all still valid, and they were diagnosed from real files, so none
    of that changes. What changes is scale: the fixable volume is roughly four times what I reported.

    Notably, the coverage rate barely moved (91.11% on the sample, 91.81% on the full catalog), so the
    200-dataset sample happened to be representative proportionally. That is exactly what made the error
    easy to miss and is not a defense of it.

    Corrected figures, all 754 datasets

    Reported before Actual
    Raw recordings converted 34,712 138,810
    Raw failures, datasets with an index 401 2,410
    Raw failures, datasets with no index 2,988 9,977
    Total raw recordings 35,113 151,197
    Catalog-wide raw coverage 91.11% 91.81%
    Datasets publishing no index 22 87 (70 failed, 17 merely pending)
    Non-raw stores being served (purge target) 4,721 17,234
    Non-raw phantom failures 3,270 7,335
    Fixable recordings ~2,793 ~11,158

    Of the 9,977 no-index failures, 8,748 are flagged deterministic, so diagnosable and fixable, and
    1,229 are infrastructure failures.

    Raw failure clusters, corrected

    Count Format Code
    866 .vhdr file_read_error
    544 .set file_read_error
    476 .ds file_read_error
    196 .fif corrupt_or_truncated
    103 .edf corrupt_or_truncated
    86 .fif file_read_error
    79 .ds corrupt_or_truncated
    37 .bdf corrupt_or_truncated
    12 .edf channel_count_mismatch

    The BrainVision header cluster is the largest at 866, up from the ~88 I saw in the sample, and the
    EEGLAB v7.3 cluster is 544, up from 219. Both root causes were correctly identified; only the counts
    were low.

    Large datasets I never looked at

    on007523 (579 failures), on007524 (500), on004444 (465), on004447 (418), nm000177 (360),
    nm000173 (300). All flagged deterministic.

    A seventh root cause, already documented and independent of me

    on004444 is the subject of an existing biosigIO issue, neuromechanist/biosigio#109: an EDF file is
    rejected as corrupt when any channel has physical_min == physical_max. That is legal EDF, since a
    reference electrode in a referential montage is identically zero by construction, and one such channel
    discards the whole recording. That issue reports 469 affected recordings with byte-exact evidence,
    and it predates this work. It belongs in the taxonomy as cause 7, and it is the same family as cause 1
    (a strict reader refusing readable data).

    What this changes about priorities

    Nothing about the diagnoses, and nothing about the decisions already taken. It does mean the purge is
    substantially larger than described when it was authorized (17,234 stores, not 4,721), so that is worth
    re-confirming rather than assuming the earlier approval covers a target nearly four times bigger.

  6. neuromechanist commented on Aug 21, 2026

    @neuromechanist
    ContributorAuthor

    The 87 no-index datasets, clustered: five fixes cover 62 of them

    Pulled the recorded failure detail for all 87 and clustered by format and error code. The result is
    unusually clean, and it lands well inside a target of "fewer than 10 to 20 left over."

    Every one of the 87 falls into three buckets

    Bucket Datasets Recordings What it needs
    Covered by a known root cause 62 9,476 Code fixes, five of them, four already approved
    pending with zero errors 17 0 An operational re-queue, no code
    failed, non-deterministic, no failure detail 8 501 An operational re-run, no code

    Nothing in the 87 is unexplained. The 8 in the last bucket all carry deterministic=0, which by design
    means transient infrastructure failure (a crashed worker, an S3 blip) rather than a data problem, so
    they are candidates for a retry rather than a diagnosis: on008082 (173), on004621 (167),
    nm000276 (40), on006269 (40), on004278 (30), on006446 (29), on004598 (20), on005670 (2).

    The 17 pending rows have zero recorded errors and zero stores, which looks like queue state that was
    never advanced rather than anything wrong with the data: nm000225, on004080, on004107,
    on004395, on004917, on004998, on005127, on005356, on005571, on005692, on005873,
    on007720, on007822, on007864, on007865, on007895, on007968. Worth a look at
    zarr_queue.py's reconcile to understand why they are stuck.

    The five fixes, ranked by datasets recovered

    Fix Datasets Recordings Status
    EDF/BDF strict-reader family (includes physical_min == physical_max, biosigio#109) 23 4,332 Not started
    EEGLAB MATLAB v7.3 HDF5 (biosigio#113 / #21) 19 1,638 In progress
    Elekta MaxShield Signal-Space Separation (ADR 0028) 9 2,267 Approved, queued
    BrainVision header pointers 8 704 Approved, queued
    CTF .hc fiducial dialect 3 535 Not started

    Cluster detail, by dominant format and code across the 87: .set/file_read_error 19 datasets (1,638
    recordings), .edf/corrupt_or_truncated 17 (3,147), .fif/file_read_error 9 (2,267),
    .vhdr/file_read_error 8 (704), .edf/file_read_error 3 (736), .bdf/corrupt_or_truncated 3 (449),
    .ds/file_read_error 2 (534), .ds/corrupt_or_truncated 1 (1).

    The 80/20 answer

    The single highest-value remaining item is the EDF/BDF strict-reader family at 23 datasets and 4,332
    recordings
    , and it is the one not yet being worked. It is also already documented with byte-exact
    evidence in biosigio#109, which reports physical_min == physical_max on a reference channel
    discarding an entire otherwise-valid recording. Combining it with the corrupt_or_truncated variants
    already diagnosed here (NUL-padded header field, discontinuous EDF+D) makes one coherent piece of work
    on the EDF read path.

    Doing all five fixes plus the two operational actions would take the no-index count from 87 to
    approximately 0, not merely under 20.

  7. neuromechanist commented on Aug 21, 2026

    @neuromechanist
    ContributorAuthor

    The 17 pending datasets: not a pagination bug, and not one single cause

    I suspected zarr_queue.py's reconcile was only seeing the first 200 datasets, the same trap I fell
    into myself. That is wrong: it paginates correctly. fetch_public_datasets walks offset until
    total_count, and its docstring already documents the 200-per-page cap explicitly. It also documents
    something useful that explains unrelated failures I hit: api.nemar.org sits behind Cloudflare, which
    403s the default Python urllib user-agent as a bot, which is why direct urllib calls fail where curl
    succeeds.

    Confirmed the "no index" classification is real rather than a fetch artifact: sampled these and they
    return a genuine HTTP 404 for index.json, with an empty body, not a transport error.

    The 17 split into at least two different situations:

    Converted once, but serving no index. on004080 has zarr_converted_at = 2026-06-24 and a
    zarr_source_commit, and on005356 has 2026-07-07 plus a recorded zarr_store_count of 30. So a
    conversion ran and a callback landed, yet index.json 404s today and the status never advanced past
    pending. Something produced stores without publishing (or while later losing) the index.

    Never attempted. on007968 and on005873 have no zarr_converted_at, no zarr_checked_at, and
    no zarr_source_commit at all. nm000225 was checked on 2026-06-07 but never converted.

    Distinguishing these properly needs evidence I cannot reach from outside: the queue's own SQLite state
    on the conversion host and the cron logs. The queue tracks its own status separately from D1's
    zarr_status, so a row could be done in the queue while D1 still reads pending, or the reverse,
    and only the host can say which.

    So this bucket is not a code fix and not one root cause. It needs a look at
    /mnt/local/zarr-state and .nm-zarr-cron.log on the conversion host, and the useful first question
    is whether the queue believes these are done while D1 believes they are pending, since that would
    point at the callback rather than the conversion.

    Recording this so the bucket is not mistaken for a single fixable defect. It is 17 datasets and,
    per the recorded counts, 0 failing recordings, so it is the lowest-value of the three buckets by volume
    even though it is the largest by dataset count after the root-cause group.

  8. neuromechanist commented on Aug 22, 2026

    @neuromechanist
    ContributorAuthor

    Root cause found for the infra subset of this track's 11 datasets, from a separate investigation now tracked as epic #1108.

    What it is

    The Hallu converter's pool workers are being OOM-killed, and a single kill poisons the whole pool. /mnt/local/zarr-state/.nm-zarr.log carries 383 worker crashed events and 96 BrokenProcessPool aborts. When one worker dies, ProcessPoolExecutor marks the pool unusable, every remaining recording fails with worker crashed: A process in the process pool was terminated abruptly, and the run returns rc=1 with the untouched tail of the worklist lost.

    Worked example from 2026-08-22: on004998 converted 74 of 115 recordings, then lost the remaining 41 to a single kill. on004917 lost all 24, with 7 workers dying simultaneously.

    The mechanism is the RAM admission projection in generate_zarr.py. projected_peak_bytes estimates the in-memory path at size_bytes * INMEM_MEM_FACTOR with the factor set to 6, and admission dispatches while the in-flight sum fits MemTotal * 0.8 (~49 GiB on Hallu). on004917's 24 BrainVision blobs are 1.18-2.25 GB each and all but one sit just under the STREAM_MIN_BYTES = 2 GiB streaming threshold, so 23 of 24 took the unbounded in-memory path. Admission packed exactly 7 of them (45.2 GiB projected plus 4 GiB for the one streaming recording, against a 49.6 GiB ceiling) and those 7 are precisely the 7 that crashed. The real cost is closer to 12x on-disk bytes, not 6x, so seven concurrent is ~108 GB on a 62 GB node.

    Which of this track's datasets it explains

    • nm000276 (40 failures, recorded here as "infra (non-deterministic)")
    • on005670 (2 failures, "infra")
    • on005356 ("pending, 30 stores, stale queue state") - currently sitting in the queue as failed attempts=5, i.e. retry-exhausted and it will never re-attempt on its own

    Which it does not

    The file_read_error and corrupt_or_truncated clusters remain this track's to explain. In particular on006233 and on006914 both reporting exactly 347 corrupt_or_truncated is still unaccounted for, and an OOM cascade would not produce a uniform typed data code. Keeping this issue open for that half.

    Where the fix lives

    Epic #1108, phases 2-5: contain the blast radius so one OOM costs one recording rather than a dataset, make the projection measurement-based, widen the bounded streaming path, and requeue the stranded backlog (which is where on005356 and the other retry-exhausted datasets get recovered).

    A mitigation is already live: ZARR_STREAM_MIN_BYTES=268435456 was added to the Hallu crontab on 2026-08-22, routing .vhdr/.fif/.ds/.mefd at or above 256 MB to the bounded streaming path.

  9. neuromechanist commented on Aug 22, 2026

    @neuromechanist
    ContributorAuthor

    Diagnosis

    Investigated against the live conversion node and S3 on 2026-08-22. The leading hypothesis in the issue is disproved, and the headline framing is out of date. Nine of the eleven need no fix at all.

    1. The annex-retrieval hypothesis is wrong: the data is present

    data_complete / bytes_present from GET /datasets/{id}:

    Dataset data_complete bytes_present
    on006233 1 18.6 GB
    on006914 1 18.8 GB
    on006502 1 102.8 GB
    on006720 1 146.5 GB
    on006012 1 76.3 GB
    on005545 1 42.9 GB
    on005931 1 0.86 GB
    on006910 (never swept) (unknown)
    nm000276 0 3.25 TB

    Confirmed at the object level rather than trusting the column: pulled sub-007/ses-1/ieeg/sub-007_ses-1_task-picture_ieeg.edf from s3://nemar/on006233/objects/ and got 56,479,648 bytes of real EDF, exactly matching the -s56479648 in its annex key. Not a symlink, not a placeholder.

    2. Ten of the eleven are not failing. They are queued and have never run.

    From the queue DB (zarr-queue.db), all of on006233, on006914, on006502, on006910, on006720, on006012, on005931, on005545, on005670 and nm000276 are:

    status='pending'  attempts=0  converted_version=NULL  enqueued_at=2026-07-01 (nm000276: 2026-07-10)
    

    The per-dataset failure counts in the issue body are historical, recorded before a bulk requeue on 2026-07-21. Nothing has re-attempted these since, so nothing has re-failed. "Every recording fails" is no longer a true statement about current state.

    3. The EDF failures were a false rejection by an older biosigIO, already fixed

    on006233 holds 347 .edf files and reported 347 failures; on006914 holds 353 and reported 347 + 6 = 353. Both were 100% failure, which is what "uniform per dataset rather than scattered" was pointing at. (The "exactly 347 in both" coincidence noted in the issue is just that: on006233 has 347 EDFs in total, while 347 of on006914's 353 carried that particular code.)

    Five files, spread across both datasets and different subjects, read cleanly under the deployed biosigIO 1.2.4 via the same Recording.from_file entry point the converter uses:

    on006233 sub-007  56,479,648 B -> OK
    on006233 sub-067  30,221,856 B -> OK
    on006233 sub-125  42,256,256 B -> OK
    on006914 sub-068  66,115,052 B -> OK
    on006914 sub-127  23,087,584 B -> OK
    

    This matches the pin note in scripts/zarr/requirements.txt: 1.2.4 "recovers three false EDF/BDF rejections." These datasets last ran on a build that had the bug.

    4. The CTF failures are explained by format support, not corruption

    on006502's file census is .acq x380, .cfg x380, .cls x380 — those are CTF .ds internals, i.e. 380 CTF directory recordings. Directory-recording discovery arrived in Track 2 (.github #98/#100) and biosigIO 1.2.3+. The issue already suspected this ("if retrieval is the cause it would also explain why a directory format fails wholesale") — the cause is different, but the prediction that on006502 is one wholesale-format case is right. Same story for the other anat,meg datasets (on006720, on006012).

    5. The queue is healthy and these are days away

    Datasets reaching a terminal state per ISO week:

    2026-W30   79
    2026-W31   88
    2026-W32  195
    2026-W33   50
    2026-W34  180
    

    154 pending at ~100-200/week is roughly a week of backlog. claim_next is FIFO on enqueued_at, so nothing is starved. Worth noting the current head of that queue is on004917 — the BrainVision dataset whose OOM opened epic #1108 — so its next run doubles as a live check of the memory fixes.

    What actually needs action

    Two of the eleven, and neither is a zarr bug:

    Recommendation

    No zarr code change for the other nine. Let the queue drain, then re-measure coverage; the two causes above (biosigIO 1.2.4 EDF recovery, directory-format discovery) are already deployed, so these should convert on their next natural turn. I would keep this issue open until that re-measurement confirms it rather than closing on the prediction.

    Evidence gathered read-only; nothing was requeued or mutated.

  10. neuromechanist commented on Aug 22, 2026

    @neuromechanist
    ContributorAuthor

    Resolved: three causes, not one

    Following up on my earlier comment, which was partly wrong. I had predicted the CTF and MEG failures were explained by directory-format support landing in Tracks 2 and 3. I then actually converted samples from each dataset through the same Recording.from_file -> to_zarr call the converter uses, and that prediction is false. Conversion, not just reading, is what this issue is about, and the eleven datasets split into three unrelated causes.

    Cause 1 — EDF: already fixed, waiting on the queue

    Dataset Recordings Verified
    on006233 347 .edf converts (36.2 MiB store)
    on005545 336 .edf converts (50.2 MiB store)
    on006910 384 .edf converts (50.2 MiB store)
    on006914 353 .edf reads cleanly (2 samples)
    on005931 16 .edf same family, not sampled

    These failed under an older biosigIO that falsely rejected valid EDF/BDF; 1.2.4 recovers exactly that class. Nothing further is needed. All ten of the non-on005356 datasets are status='pending' attempts=0 (enqueued 2026-07-01, never re-attempted), so the counts in the issue body are historical, not current. The queue is FIFO on enqueued_at, draining 100-200 datasets/week with 154 pending, so these publish on their own within about a week.

    Incidental: on005545 and on006910 return the same annex key for their first recording, so they share content.

    Cause 2 — CTF: a real, unfixed reader incompatibility (380 recordings)

    on006502 still fails, with file_read_error: "Some of the mandatory HPI device-coordinate info was not there."

    Root-caused rather than guessed. MNE's _read_hc maps head-position-indicator descriptors through a literal, case-sensitive lookup:

    _kind_dict = {"nasion": ..., "left ear": ..., "right ear": ..., "spare": ...}

    This dataset's .hc files spell them Nasion, LPA, and RPA. Every coil point therefore parses with kind = -1, the LPA/RPA/NAS check in _make_ctf_coord_trans_set finds nothing, and it raises. Confirmed by running MNE's parser directly: 6 points, all kind=-1, 3 in device frame and 3 in head frame, all valid=True. The coordinates are present and correct; only the labels differ.

    Proved by rewriting only those three descriptor labels in a copy and reconverting the otherwise byte-identical recording:

    original -> FAILED  file_read_error (mandatory HPI device-coordinate info)
    labels-only patch -> CONVERTED  groups=['events','meg_250hz','misc_600hz']  60.3 MiB
    

    This is a vendor naming variant MNE does not accept, not corruption and not a NEMAR bug. Filed separately; deliberately not hacked around here, since normalizing acquisition metadata before reading is a correctness-sensitive change that deserves its own decision.

    Cause 3 — MaxShield MEG: already decided, never implemented (439 recordings)

    on006720 (246 .fif) and on006012 (193 .fif) both fail with MNE's Internal Active Shielding refusal:

    This file contains raw Internal Active Shielding data. It may be distorted. Elekta recommends it be run through MaxFilter to produce reliable results.

    This is not a new finding. ADR 0028, accepted 2026-08-20 out of this very epic, names both datasets and decides to apply Signal-Space Separation where the fine-calibration and cross-talk sidecars resolve, and otherwise to decline with a specific reason rather than a generic read error. Its own consequences section projects 365 of 396 recordings becoming viewable, the remaining 31 being sub-emptyroom references.

    The decision was never implemented: there is no maxwell_filter or allow_maxshield anywhere in the converter, only the ADR 0027 calibration-file exclusion. Filed separately.

    The remaining three

    • nm000276 — data_complete=0 against 3.25 TB. A data-completeness problem for the import/recover tooling, not conversion. Zarr cannot succeed until the content is present.
    • on005670 — 2 infra failures, non-deterministic.
    • on005356 — the only genuinely failed row (attempts=5), included in the staged requeue.

    Closing this

    The question this issue asked, why eleven datasets publish no index, is now answered with evidence for every one of them. Closing it, with the two pieces of real work it uncovered tracked on their own issues so neither hides inside a diagnosis ticket.

    All verification was read-only against S3 and ran outside the conversion work directory; the in-flight drain was unaffected.

    Tracked follow-ups: #1125 (CTF .hc coil naming, 380 recordings) and #1126 (implement ADR 0028 for MaxShield, 439 recordings).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureNew feature or enhancement

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions