You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Track 5: diagnose 11 datasets publishing no Zarr index #1100
Eleven magnetoencephalography (MEG) and intracranial electroencephalography (iEEG) datasets publish no index at all because every recording in them fails, roughly 2,300 recordings. This is the single largest Zarr availability gap, far larger than the format gaps in Tracks 1 to 3.
Dataset
Failing
Code
Deterministic
on006502
380 (CTF .ds)
file_read_error
yes
on006910
384
file_read_error
yes
on006914
353
corrupt_or_truncated (347) + file_read_error (6)
yes
on006233
347
corrupt_or_truncated
yes
on005545
336
file_read_error
no
on006720
246
file_read_error
yes
on006012
193
file_read_error
yes
nm000276
40
infra
no
on005931
16
file_read_error
yes
on005670
2
infra
no
on005356
pending, 30 stores
stale queue state
n/a
The failures are uniform per dataset rather than scattered, which points at a systematic cause rather than per-file corruption. on006233 and on006914 both report exactly 347 corrupt_or_truncated, which is unlikely to be coincidence.
Leading hypothesis: the annexed content was never retrieved on the conversion host, so biosigio read a git-annex symlink or an empty placeholder instead of real data, which would surface as exactly these two codes. This is a hypothesis, not a finding, and must be confirmed against evidence on the conversion host before any fix.
Deliverable is a diagnosis with evidence, then a fix or a tracked follow-up. The fix may well be operational (retrieval, re-queue) rather than a code change. Note on006502 is entirely CTF .ds, so if retrieval is the cause it would also explain why a directory format fails wholesale.
Diagnosis complete: four distinct root causes, all recoverable
The leading hypothesis in the issue body was wrong and is refuted.
The annexed content is not missing: data_complete=1 and bytes_present is within 0.1% of file_size for every affected dataset,
and I verified byte-exactness directly.
For a failing on006233 recording the downloaded object is 4,211,536 bytes
with sha256 4a6696...336598,
both matching the git-annex key exactly.
The data is intact.
In every case the signal data is fully readable.
The rejection comes either from a reader that is stricter than the format's real-world variation,
or from metadata that is irrelevant to serving a time series.
1. EDF rejected by pyedflib but read fine by MNE (1,084 recordings)
biosigio's in-memory EDF path reads via pyedflib. Two separate real-world deviations make it refuse:
Non-compliant header padding (on006233 347, on006914 353, reported corrupt_or_truncated).
The "number of data records" field contains b'421\x00', NUL-padded instead of space-padded as the
European Data Format (EDF) specification requires. pyedflib raises OSError: the file is not EDF(+) or BDF(+) compliant (Number of Datarecords).
MNE reads the same file without complaint: 4 channels, 421,000 samples, 1000 Hz.
Note these two datasets are NOT duplicate imports as I first suspected; they are distinct OpenNeuro
accessions (ds006233 "Picture naming", ds006914 "Visual Naming EC") from what appears to be the
same lab, which is why their subject, session, and task naming coincide.
Discontinuous EDF+D (on006910 384, reported file_read_error). pyedflib raises OSError: The file is discontinuous and cannot be read.
MNE reads it: 66 channels, 295,410 samples, 1000 Hz. This file's header is clean (zero NUL bytes).
2. Elekta MaxShield FIF (439 recordings)
on006012 (193) and on006720 (246), both reported file_read_error. mne.io.read_raw_fif refuses with ValueError: This file contains raw Internal Active Shielding data... Use allow_maxshield=True if you are sure you want to load the data.
With allow_maxshield=True both read fine (on006012: 336 channels, 341,000 samples; on006720: 328 channels, 127,000 samples, 1000 Hz).
This one is a judgment call, not just a bug: MNE warns the data may be distorted and that Elekta
recommends running MaxFilter first. Serving unfiltered Internal Active Shielding data in a viewer
without saying so would be misleading. See the recommendation below.
The zarr_queue.py docstring already cites "a MaxShield MEG .fif" as its canonical example of a
deterministic data failure, so this was known, but it was treated as terminal rather than handled.
on006502, reported file_read_error. Fails in _make_ctf_coord_trans_set with RuntimeError: Some of the mandatory HPI device-coordinate info was not there.
Root cause is in MNE's .hc parser (mne/io/ctf/hc.py). Its _kind_dict matches the lowercase
literals "nasion", "left ear", "right ear", and the comparison is case-sensitive with no
aliases. This dataset's .hc writes Nasion, LPA, and RPA, so all three fiducials come back
unmatched (kind = -1) and the transform build fails.
Proof: renaming Nasion to nasion, LPA coil to left ear coil, and RPA coil to right ear coil in a copy of the .hc makes the identical directory read successfully: 306 channels, 197,800 samples, 600 Hz. All ten .ds members were verified byte-exact against
their annex keys first, so this is not a data problem.
Worth emphasizing: this is a head-coil coordinate transform, used for source localization and sensor
geometry. It has no bearing on serving a time series. A whole 380-recording dataset is withheld over
metadata the serving copy never uses.
4. Side finding: BIDS special files treated as recordings
on006012 and on006720 both report failures on _acq-crosstalk_meg.fif.
The Brain Imaging Data Structure reserves _acq-crosstalk_meg.fif and _acq-calibration_meg.dat
as Elekta calibration files, not recordings. Reading the crosstalk file fails with ValueError: Could not find measurement data, correctly, because it contains none.
These should be excluded by path convention rather than attempted and reported as failures.
This belongs with the raw-only scope work in #1097 and #1098.
Add an MNE fallback to biosigio's EDF path. Highest value at 1,084 recordings. Important
constraint: generate_zarr.py documents that MNE rescales EDF to SI volts while pyedflib does
not, so a fallback must normalize units or the in-memory and streaming paths will disagree on the
same file. This needs a unit-parity test, not just a try/except.
Decide the MaxShield policy (439 recordings). Either convert with allow_maxshield=True and
surface an explicit caveat in the index so the viewer can label it as unfiltered Internal Active
Shielding data, or keep declining but report an honest "requires MaxFilter" reason instead of a
generic file_read_error. I lean toward converting with the caveat, since the alternative shows
users nothing at all, but this is a scientific-communication call rather than a technical one.
Handle the CTF .hc dialect. The clean fix is upstream: make MNE's _kind_dict matching
case-insensitive and add LPA/RPA aliases, which is a small, well-evidenced patch worth
sending to MNE. In the meantime the converter can normalize a scratch copy of the .hc.
Reproduction for all of the above uses public S3 objects
(https://s3.us-east-2.amazonaws.com/nemar/<dataset>/objects/<annex-key>),
with annex keys read from the git symlink blobs, so any of this can be re-checked independently.
Update: the failure picture was distorted, and there are two more root causes
Two corrections to my earlier comment, then two new findings.
Correction 1: raw coverage is actually 98.86%, not alarming
My earlier framing leaned on total failure counts, which are dominated by non-raw paths. Recomputed
ignoring derivatives/, sourcedata/, and code/ entirely, the picture is much better:
Metric
Value
Raw recordings converted
34,712
Raw recordings failed
401
Raw success rate
98.86%
Datasets with no raw failures at all
150
Datasets with some raw failures
27
Datasets showing the viewer nothing real
23
So the honest statement of the gap is not "thousands of failures", it is "23 datasets show the user
nothing." That is the number worth fixing.
Correction 2: my count of all-fail datasets was an UNDERCOUNT
I previously found 11 by looking for datasets with no published index. That missed a case: a dataset
whose raw recordings ALL fail can still publish an index if it has derivatives/ stores, so it looks
healthy from the outside. on005520 is exactly that — 92 stores, every one from derivatives/preprocessed/, and 67 of 67 raw BrainVision recordings failing.
This matters for the raw-only scope change: excluding derivatives/ will not "remove on005520's
coverage", it will reveal that the dataset has no working raw coverage at all. The derivatives
stores were masking a total raw failure. Worth stating plainly so the change is not mistaken for a
regression when that dataset goes from 92 stores to 0.
I also checked the concern about sourcedata/-only datasets before anything gets deleted: there are
none. All four datasets with sourcedata/ stores (on005178, on005185, on005207, on005403)
also have healthy raw coverage, so excluding sourcedata/ costs no dataset its viewer copy. on005520 is the single at-risk dataset, and it is at risk from derivatives/, not sourcedata/.
New root cause 5: BrainVision header pointers not updated after BIDS renaming (~470 recordings)
The largest remaining cluster. on005520's .vhdr contains:
while the files actually on disk are sub-01_task-restingeyeclosed_eeg.eeg / .vmrk. BIDS forbids
underscores inside a label value, so the conversion to BIDS renamed the files and never rewrote the
header's internal pointers. MNE therefore raises FileNotFoundError looking for a file that does not
exist, which surfaces as file_read_error.
Proof: rewriting only those two lines to the real sibling filenames makes the identical data read
cleanly, 67 channels, 417,997 samples, 1000 Hz, values in a sane microvolt range. All three triplet
members were verified byte-exact against their annex keys first, so the data is intact.
Note this is a genuine defect in the DATASETS, not in our converter: the header really is wrong.
Two options, and I would do both:
Repair on the scratch copy at conversion time. The pointers are unambiguous, since the correct
targets are the same-stem siblings sitting next to the header, so the rewrite is deterministic and
safe. The scratch copy is discarded and BIDS stays the source of truth, so nothing authoritative
is mutated. This recovers ~470 recordings without waiting on anyone.
Report upstream, because the datasets are genuinely malformed and every other tool reading them
hits the same wall. Arguably the BIDS validator should flag a .vhdr whose DataFile does not
resolve.
Still undiagnosed
on007509 (70 EEGLAB .set, file_read_error) is a separate cause I have not chased. on006033 (2) and on006269/on008082 (non-deterministic, so infra) remain open. on005571, on005692, and on005873 are not failures at all: they sit at zarr_status=pending with
no stores, which looks like stale queue rows rather than anything wrong with the data.
Revised recoverable total
Cause
Recordings
European Data Format, strict pyedflib rejection
1,084
Elekta MaxShield magnetoencephalography
439
BrainVision header pointers
~470
CTF .hc fiducial dialect
380
Total
~2,373
Every one of these is a reader-strictness or metadata problem on data that is byte-intact and
readable. None is a data-loss problem.
Final: complete root-cause taxonomy, and a correction to my own headline number
Correcting myself: catalog-wide raw coverage is 91.11%, not 98.86%
My 98.86% figure was computed only over datasets that have a published index, which silently
excluded exactly the worst cases: the 22 datasets where every recording fails publish no index at all,
so they contributed nothing to that denominator. Including them:
Recordings
Raw converted successfully
34,712
Raw failing in datasets with an index
401
Raw failing in no-index datasets
2,988
Total raw recordings
38,101
Catalog-wide raw coverage
91.11%
Of the 2,988 in no-index datasets, 2,392 are flagged deterministic (reproducible, therefore
diagnosable and fixable) and 596 are non-deterministic infrastructure failures. Four further datasets
(on005356, on005571, on005692, on005873) are not failures at all: they sit at zarr_status=pending with no stores, which looks like stale queue rows and is a separate, probably
cheap, fix.
So roughly 2,793 recordings are fixable through the six causes below, which would take raw coverage
from 91.11% to about 98.4%.
The six root causes
Every one of these is a reader-strictness or metadata problem on data that is byte-intact. I verified
byte-exactness against git-annex SHA256E keys before diagnosing each. No data loss anywhere.
1. European Data Format rejected by pyedflib though MNE reads it (1,084 recordings).
Two variants. Non-compliant NUL-padded header field: the "number of data records" field contains b'421\x00' instead of space padding, so pyedflib raises the file is not EDF(+) or BDF(+) compliant (Number of Datarecords) while MNE reads it fine
(on006233 347, on006914 353). Discontinuous EDF+D: pyedflib raises The file is discontinuous and cannot be read; MNE reads 66 channels / 295,410 samples
(on006910 384). Fix: fall back to MNE when pyedflib refuses. Constraint already documented in generate_zarr.py: MNE rescales EDF to SI volts and pyedflib does not, so the fallback needs a
unit-parity test, not a bare try/except.
2. Elekta MaxShield / Internal Active Shielding FIF (439 recordings). read_raw_fif refuses without allow_maxshield=True; with it, on006012 reads 336 channels and on006720 328 channels. Still needs a human decision, because MNE warns the data may be distorted
and Elekta recommends MaxFilter first: convert with an explicit caveat surfaced in the index, or keep
declining but report an honest "requires MaxFilter" reason rather than a generic file_read_error.
3. BrainVision header pointers not updated after BIDS renaming (~470 recordings). .vhdr says DataFile=sub-01_task-resting_eye_closed_eeg.eeg while the file on disk is sub-01_task-restingeyeclosed_eeg.eeg, because BIDS forbids underscores inside a label value and the
conversion renamed files without rewriting the header. MNE raises FileNotFoundError. Rewriting only
those two lines reads 67 channels / 417,997 samples / 1000 Hz. Datasets: on005863 (357), on005520 (67), on007471 (32), on007813 (4), on007406 (10), plus 88 more spread across on005273, on005274, on005486, on006576 and others. This is a defect in the datasets;
worth both a deterministic scratch-copy repair (targets are unambiguous same-stem siblings) and an
upstream report.
4. CTF .hc fiducial naming dialect (380 recordings). on006502. MNE's mne/io/ctf/hc.py_kind_dict matches lowercase "nasion", "left ear", "right ear" case-sensitively with no aliases; this dataset writes Nasion, LPA, RPA, so all
three fiducials go unmatched and _make_ctf_coord_trans_set raises Some of the mandatory HPI device-coordinate info was not there. Normalizing the names reads
306 channels / 197,800 samples / 600 Hz. Note this is a head-coil coordinate transform used for source
localization; it has no bearing on serving a time series, so a whole 380-recording dataset is withheld
over metadata the serving copy never touches. Best fixed upstream in MNE (case-insensitive matching
plus LPA/RPA aliases) — a small, well-evidenced patch.
5. MATLAB v7.3 EEGLAB .set files (219 recordings) — NEW, and the largest single cluster in the
indexed datasets.
MNE's read_raw_eeglab raises NotImplementedError: Please use HDF reader for matlab v7.3 files, e.g. h5py, and biosigio surfaces that as FileReadError. A v7.3 .set is an HDF5 container, not a
classic MAT file, so scipy.io.loadmat cannot read it either. Affects 7 datasets: on005178, on005185, on005343, on006040, on006554, and others.
I confirmed the data is fully reachable, so this is implementable rather than a dead end. Opening on005343's 557 MB sub-S10_task-resting_eeg.set with h5py gives EEG/data as a (1236409, 129) float32 array (HDF5 stores it transposed relative to MATLAB), nbchan=129, srate=1000, pnts=1236409, trials=1 (continuous), and channel labels resolving through the #refs# table to E1...E129. Fix belongs in biosigio as an h5py-based reader for v7.3 .set,
selected by sniffing the MATLAB 7.3 MAT-file magic. Watch the transpose and the object-reference
indirection for chanlocs.labels.
6. BIDS calibration files treated as recordings. _acq-crosstalk_meg.fif and _acq-calibration_meg.dat are Elekta calibration files, not recordings.
Reading the crosstalk file raises ValueError: Could not find measurement data, correctly, because it
contains none. Already folded into the raw-only scope work in #1097.
Still open
on005545 (336, flagged non-deterministic despite the volume, so worth a second look), on008082 (173, infra), nm000276 and on006269 (40 each, infra), on006033 (5), on005670 (2),
and the 4 stale pending rows.
MaxShield policy settled: filter properly when we can, decline honestly when we cannot
Decision: apply Signal-Space Separation only when both the Elekta fine-calibration and cross-talk
files are available, and otherwise decline with the same message MNE itself gives, rather than
serving unfiltered Internal Active Shielding data.
This turns out to fit the data almost exactly, and it is better than either option I originally put
forward. We do not need MEGIN's proprietary MaxFilter: mne.preprocessing.maxwell_filter is an
open-source implementation of the same Signal-Space Separation family, documented as broadly
comparable to MEGIN MaxFilter 2.2.11.
Verified on real data
Ran it against on006012's sub-01_task-POGS_run-06_meg.fif using that dataset's own sub-01_acq-calibration_meg.dat and sub-01_acq-crosstalk_meg.fif:
Filtering succeeded, 336 channels preserved.
info["maxshield"] flips from True to False. This is the strongest available signal: MNE's
own guardrail, the thing that was refusing to load these files, is satisfied by the result. The
output is no longer raw Internal Active Shielding data by MNE's own reckoning.
Root-mean-square amplitude drops to 62.7% of the unfiltered signal, consistent with interference
being removed rather than the data being passed through untouched.
The Signal-Space Separation parameters are written into the data's own proc_history
(in_order=8, out_order=3, nfree=72, frame=head, plus both sss_cal and sss_ctc records),
so the processing is self-documenting and auditable afterwards.
Temporal Signal-Space Separation (st_duration) also works, and filtering without the calibration
pair works too, but under this policy we will not use that second path.
The files we need are already in the datasets
Every subject in both affected datasets ships both files:
Dataset
Subjects with both files
Recordings recoverable
Declined
on006012
20 of 20
158
15
on006720
23 of 23
207
16
Total
365
31
Every single declined recording is sub-emptyroom. Those are empty-room noise reference
measurements rather than subject data, so declining them is the right outcome on its own merits, not
a shortfall. No actual subject recording is lost to this policy.
Correcting my earlier count
I previously reported 439 recordings for this cause, taken from the zarr_errors column. That figure
included the 43 calibration and cross-talk files themselves (20 plus 23), which were being attempted
as recordings and are now excluded as the calibration files they are. The real breakdown is 396 _meg.fif recordings, of which 365 are recoverable and 31 are empty-room.
One consequence worth stating plainly
A Signal-Space Separation filtered store is not a faithful copy of its source recording, unlike
every other store in the serving copy. It is a processed derivative. That has to be recorded in the
index so consumers know, and it matters especially for the machine-learning streaming path, where a
model trained across datasets would otherwise silently mix filtered and unfiltered magnetoencephalography.
The proc_history provenance makes the processing auditable, but the index needs to say it too.
This is a decision worth an Architecture Decision Record, since it changes what the serving copy
guarantees for one class of recording and would be expensive to reverse quietly.
Correction: my whole analysis covered 26% of the catalog
Every count I reported in this issue was computed over 200 datasets. The catalog has 754. GET /datasets caps limit at 200 and paginates by offset, and I read the first page as if it were
everything without checking total_count, which the response carries and which says 754.
The root-cause mechanisms below are all still valid, and they were diagnosed from real files, so none
of that changes. What changes is scale: the fixable volume is roughly four times what I reported.
Notably, the coverage rate barely moved (91.11% on the sample, 91.81% on the full catalog), so the
200-dataset sample happened to be representative proportionally. That is exactly what made the error
easy to miss and is not a defense of it.
Corrected figures, all 754 datasets
Reported before
Actual
Raw recordings converted
34,712
138,810
Raw failures, datasets with an index
401
2,410
Raw failures, datasets with no index
2,988
9,977
Total raw recordings
35,113
151,197
Catalog-wide raw coverage
91.11%
91.81%
Datasets publishing no index
22
87 (70 failed, 17 merely pending)
Non-raw stores being served (purge target)
4,721
17,234
Non-raw phantom failures
3,270
7,335
Fixable recordings
~2,793
~11,158
Of the 9,977 no-index failures, 8,748 are flagged deterministic, so diagnosable and fixable, and
1,229 are infrastructure failures.
Raw failure clusters, corrected
Count
Format
Code
866
.vhdr
file_read_error
544
.set
file_read_error
476
.ds
file_read_error
196
.fif
corrupt_or_truncated
103
.edf
corrupt_or_truncated
86
.fif
file_read_error
79
.ds
corrupt_or_truncated
37
.bdf
corrupt_or_truncated
12
.edf
channel_count_mismatch
The BrainVision header cluster is the largest at 866, up from the ~88 I saw in the sample, and the
EEGLAB v7.3 cluster is 544, up from 219. Both root causes were correctly identified; only the counts
were low.
A seventh root cause, already documented and independent of me
on004444 is the subject of an existing biosigIO issue, neuromechanist/biosigio#109: an EDF file is
rejected as corrupt when any channel has physical_min == physical_max. That is legal EDF, since a
reference electrode in a referential montage is identically zero by construction, and one such channel
discards the whole recording. That issue reports 469 affected recordings with byte-exact evidence,
and it predates this work. It belongs in the taxonomy as cause 7, and it is the same family as cause 1
(a strict reader refusing readable data).
What this changes about priorities
Nothing about the diagnoses, and nothing about the decisions already taken. It does mean the purge is
substantially larger than described when it was authorized (17,234 stores, not 4,721), so that is worth
re-confirming rather than assuming the earlier approval covers a target nearly four times bigger.
The 87 no-index datasets, clustered: five fixes cover 62 of them
Pulled the recorded failure detail for all 87 and clustered by format and error code. The result is
unusually clean, and it lands well inside a target of "fewer than 10 to 20 left over."
Every one of the 87 falls into three buckets
Bucket
Datasets
Recordings
What it needs
Covered by a known root cause
62
9,476
Code fixes, five of them, four already approved
pending with zero errors
17
0
An operational re-queue, no code
failed, non-deterministic, no failure detail
8
501
An operational re-run, no code
Nothing in the 87 is unexplained. The 8 in the last bucket all carry deterministic=0, which by design
means transient infrastructure failure (a crashed worker, an S3 blip) rather than a data problem, so
they are candidates for a retry rather than a diagnosis: on008082 (173), on004621 (167), nm000276 (40), on006269 (40), on004278 (30), on006446 (29), on004598 (20), on005670 (2).
The 17 pending rows have zero recorded errors and zero stores, which looks like queue state that was
never advanced rather than anything wrong with the data: nm000225, on004080, on004107, on004395, on004917, on004998, on005127, on005356, on005571, on005692, on005873, on007720, on007822, on007864, on007865, on007895, on007968. Worth a look at zarr_queue.py's reconcile to understand why they are stuck.
The five fixes, ranked by datasets recovered
Fix
Datasets
Recordings
Status
EDF/BDF strict-reader family (includes physical_min == physical_max, biosigio#109)
Cluster detail, by dominant format and code across the 87: .set/file_read_error 19 datasets (1,638
recordings), .edf/corrupt_or_truncated 17 (3,147), .fif/file_read_error 9 (2,267), .vhdr/file_read_error 8 (704), .edf/file_read_error 3 (736), .bdf/corrupt_or_truncated 3 (449), .ds/file_read_error 2 (534), .ds/corrupt_or_truncated 1 (1).
The 80/20 answer
The single highest-value remaining item is the EDF/BDF strict-reader family at 23 datasets and 4,332
recordings, and it is the one not yet being worked. It is also already documented with byte-exact
evidence in biosigio#109, which reports physical_min == physical_max on a reference channel
discarding an entire otherwise-valid recording. Combining it with the corrupt_or_truncated variants
already diagnosed here (NUL-padded header field, discontinuous EDF+D) makes one coherent piece of work
on the EDF read path.
Doing all five fixes plus the two operational actions would take the no-index count from 87 to
approximately 0, not merely under 20.
The 17 pending datasets: not a pagination bug, and not one single cause
I suspected zarr_queue.py's reconcile was only seeing the first 200 datasets, the same trap I fell
into myself. That is wrong: it paginates correctly.fetch_public_datasets walks offset until total_count, and its docstring already documents the 200-per-page cap explicitly. It also documents
something useful that explains unrelated failures I hit: api.nemar.org sits behind Cloudflare, which
403s the default Python urllib user-agent as a bot, which is why direct urllib calls fail where curl
succeeds.
Confirmed the "no index" classification is real rather than a fetch artifact: sampled these and they
return a genuine HTTP 404 for index.json, with an empty body, not a transport error.
The 17 split into at least two different situations:
Converted once, but serving no index.on004080 has zarr_converted_at = 2026-06-24 and a zarr_source_commit, and on005356 has 2026-07-07 plus a recorded zarr_store_count of 30. So a
conversion ran and a callback landed, yet index.json 404s today and the status never advanced past pending. Something produced stores without publishing (or while later losing) the index.
Never attempted.on007968 and on005873 have no zarr_converted_at, no zarr_checked_at, and
no zarr_source_commit at all. nm000225 was checked on 2026-06-07 but never converted.
Distinguishing these properly needs evidence I cannot reach from outside: the queue's own SQLite state
on the conversion host and the cron logs. The queue tracks its own status separately from D1's zarr_status, so a row could be done in the queue while D1 still reads pending, or the reverse,
and only the host can say which.
So this bucket is not a code fix and not one root cause. It needs a look at /mnt/local/zarr-state and .nm-zarr-cron.log on the conversion host, and the useful first question
is whether the queue believes these are done while D1 believes they are pending, since that would
point at the callback rather than the conversion.
Recording this so the bucket is not mistaken for a single fixable defect. It is 17 datasets and,
per the recorded counts, 0 failing recordings, so it is the lowest-value of the three buckets by volume
even though it is the largest by dataset count after the root-cause group.
Root cause found for the infra subset of this track's 11 datasets, from a separate investigation now tracked as epic #1108.
What it is
The Hallu converter's pool workers are being OOM-killed, and a single kill poisons the whole pool. /mnt/local/zarr-state/.nm-zarr.log carries 383 worker crashed events and 96 BrokenProcessPool aborts. When one worker dies, ProcessPoolExecutor marks the pool unusable, every remaining recording fails with worker crashed: A process in the process pool was terminated abruptly, and the run returns rc=1 with the untouched tail of the worklist lost.
Worked example from 2026-08-22: on004998 converted 74 of 115 recordings, then lost the remaining 41 to a single kill. on004917 lost all 24, with 7 workers dying simultaneously.
The mechanism is the RAM admission projection in generate_zarr.py. projected_peak_bytes estimates the in-memory path at size_bytes * INMEM_MEM_FACTOR with the factor set to 6, and admission dispatches while the in-flight sum fits MemTotal * 0.8 (~49 GiB on Hallu). on004917's 24 BrainVision blobs are 1.18-2.25 GB each and all but one sit just under the STREAM_MIN_BYTES = 2 GiB streaming threshold, so 23 of 24 took the unbounded in-memory path. Admission packed exactly 7 of them (45.2 GiB projected plus 4 GiB for the one streaming recording, against a 49.6 GiB ceiling) and those 7 are precisely the 7 that crashed. The real cost is closer to 12x on-disk bytes, not 6x, so seven concurrent is ~108 GB on a 62 GB node.
Which of this track's datasets it explains
nm000276 (40 failures, recorded here as "infra (non-deterministic)")
on005670 (2 failures, "infra")
on005356 ("pending, 30 stores, stale queue state") - currently sitting in the queue as failed attempts=5, i.e. retry-exhausted and it will never re-attempt on its own
Which it does not
The file_read_error and corrupt_or_truncated clusters remain this track's to explain. In particular on006233 and on006914 both reporting exactly 347 corrupt_or_truncated is still unaccounted for, and an OOM cascade would not produce a uniform typed data code. Keeping this issue open for that half.
Where the fix lives
Epic #1108, phases 2-5: contain the blast radius so one OOM costs one recording rather than a dataset, make the projection measurement-based, widen the bounded streaming path, and requeue the stranded backlog (which is where on005356 and the other retry-exhausted datasets get recovered).
A mitigation is already live: ZARR_STREAM_MIN_BYTES=268435456 was added to the Hallu crontab on 2026-08-22, routing .vhdr/.fif/.ds/.mefd at or above 256 MB to the bounded streaming path.
Investigated against the live conversion node and S3 on 2026-08-22. The leading hypothesis in the issue is disproved, and the headline framing is out of date. Nine of the eleven need no fix at all.
1. The annex-retrieval hypothesis is wrong: the data is present
data_complete / bytes_present from GET /datasets/{id}:
Dataset
data_complete
bytes_present
on006233
1
18.6 GB
on006914
1
18.8 GB
on006502
1
102.8 GB
on006720
1
146.5 GB
on006012
1
76.3 GB
on005545
1
42.9 GB
on005931
1
0.86 GB
on006910
(never swept)
(unknown)
nm000276
0
3.25 TB
Confirmed at the object level rather than trusting the column: pulled sub-007/ses-1/ieeg/sub-007_ses-1_task-picture_ieeg.edf from s3://nemar/on006233/objects/ and got 56,479,648 bytes of real EDF, exactly matching the -s56479648 in its annex key. Not a symlink, not a placeholder.
2. Ten of the eleven are not failing. They are queued and have never run.
From the queue DB (zarr-queue.db), all of on006233, on006914, on006502, on006910, on006720, on006012, on005931, on005545, on005670 and nm000276 are:
The per-dataset failure counts in the issue body are historical, recorded before a bulk requeue on 2026-07-21. Nothing has re-attempted these since, so nothing has re-failed. "Every recording fails" is no longer a true statement about current state.
3. The EDF failures were a false rejection by an older biosigIO, already fixed
on006233 holds 347 .edf files and reported 347 failures; on006914 holds 353 and reported 347 + 6 = 353. Both were 100% failure, which is what "uniform per dataset rather than scattered" was pointing at. (The "exactly 347 in both" coincidence noted in the issue is just that: on006233 has 347 EDFs in total, while 347 of on006914's 353 carried that particular code.)
Five files, spread across both datasets and different subjects, read cleanly under the deployed biosigIO 1.2.4 via the same Recording.from_file entry point the converter uses:
on006233 sub-007 56,479,648 B -> OK
on006233 sub-067 30,221,856 B -> OK
on006233 sub-125 42,256,256 B -> OK
on006914 sub-068 66,115,052 B -> OK
on006914 sub-127 23,087,584 B -> OK
This matches the pin note in scripts/zarr/requirements.txt: 1.2.4 "recovers three false EDF/BDF rejections." These datasets last ran on a build that had the bug.
4. The CTF failures are explained by format support, not corruption
on006502's file census is .acq x380, .cfg x380, .cls x380 — those are CTF .ds internals, i.e. 380 CTF directory recordings. Directory-recording discovery arrived in Track 2 (.github#98/#100) and biosigIO 1.2.3+. The issue already suspected this ("if retrieval is the cause it would also explain why a directory format fails wholesale") — the cause is different, but the prediction that on006502 is one wholesale-format case is right. Same story for the other anat,meg datasets (on006720, on006012).
154 pending at ~100-200/week is roughly a week of backlog. claim_next is FIFO on enqueued_at, so nothing is starved. Worth noting the current head of that queue is on004917 — the BrainVision dataset whose OOM opened epic #1108 — so its next run doubles as a live check of the memory fixes.
No zarr code change for the other nine. Let the queue drain, then re-measure coverage; the two causes above (biosigIO 1.2.4 EDF recovery, directory-format discovery) are already deployed, so these should convert on their next natural turn. I would keep this issue open until that re-measurement confirms it rather than closing on the prediction.
Evidence gathered read-only; nothing was requeued or mutated.
Following up on my earlier comment, which was partly wrong. I had predicted the CTF and MEG failures were explained by directory-format support landing in Tracks 2 and 3. I then actually converted samples from each dataset through the same Recording.from_file -> to_zarr call the converter uses, and that prediction is false. Conversion, not just reading, is what this issue is about, and the eleven datasets split into three unrelated causes.
Cause 1 — EDF: already fixed, waiting on the queue
Dataset
Recordings
Verified
on006233
347 .edf
converts (36.2 MiB store)
on005545
336 .edf
converts (50.2 MiB store)
on006910
384 .edf
converts (50.2 MiB store)
on006914
353 .edf
reads cleanly (2 samples)
on005931
16 .edf
same family, not sampled
These failed under an older biosigIO that falsely rejected valid EDF/BDF; 1.2.4 recovers exactly that class. Nothing further is needed. All ten of the non-on005356 datasets are status='pending' attempts=0 (enqueued 2026-07-01, never re-attempted), so the counts in the issue body are historical, not current. The queue is FIFO on enqueued_at, draining 100-200 datasets/week with 154 pending, so these publish on their own within about a week.
Incidental: on005545 and on006910 return the same annex key for their first recording, so they share content.
Cause 2 — CTF: a real, unfixed reader incompatibility (380 recordings)
on006502 still fails, with file_read_error: "Some of the mandatory HPI device-coordinate info was not there."
Root-caused rather than guessed. MNE's _read_hc maps head-position-indicator descriptors through a literal, case-sensitive lookup:
This dataset's .hc files spell them Nasion, LPA, and RPA. Every coil point therefore parses with kind = -1, the LPA/RPA/NAS check in _make_ctf_coord_trans_set finds nothing, and it raises. Confirmed by running MNE's parser directly: 6 points, all kind=-1, 3 in device frame and 3 in head frame, all valid=True. The coordinates are present and correct; only the labels differ.
Proved by rewriting only those three descriptor labels in a copy and reconverting the otherwise byte-identical recording:
This is a vendor naming variant MNE does not accept, not corruption and not a NEMAR bug. Filed separately; deliberately not hacked around here, since normalizing acquisition metadata before reading is a correctness-sensitive change that deserves its own decision.
Cause 3 — MaxShield MEG: already decided, never implemented (439 recordings)
on006720 (246 .fif) and on006012 (193 .fif) both fail with MNE's Internal Active Shielding refusal:
This file contains raw Internal Active Shielding data. It may be distorted. Elekta recommends it be run through MaxFilter to produce reliable results.
This is not a new finding. ADR 0028, accepted 2026-08-20 out of this very epic, names both datasets and decides to apply Signal-Space Separation where the fine-calibration and cross-talk sidecars resolve, and otherwise to decline with a specific reason rather than a generic read error. Its own consequences section projects 365 of 396 recordings becoming viewable, the remaining 31 being sub-emptyroom references.
The decision was never implemented: there is no maxwell_filter or allow_maxshield anywhere in the converter, only the ADR 0027 calibration-file exclusion. Filed separately.
The remaining three
nm000276 — data_complete=0 against 3.25 TB. A data-completeness problem for the import/recover tooling, not conversion. Zarr cannot succeed until the content is present.
on005670 — 2 infra failures, non-deterministic.
on005356 — the only genuinely failed row (attempts=5), included in the staged requeue.
Closing this
The question this issue asked, why eleven datasets publish no index, is now answered with evidence for every one of them. Closing it, with the two pieces of real work it uncovered tracked on their own issues so neither hides inside a diagnosis ticket.
All verification was read-only against S3 and ran outside the conversion work directory; the in-flight drain was unaffected.
Tracked follow-ups:#1125 (CTF .hc coil naming, 380 recordings) and #1126 (implement ADR 0028 for MaxShield, 439 recordings).
Activity
neuromechanist commentedon Aug 20, 2026
Diagnosis complete: four distinct root causes, all recoverable
The leading hypothesis in the issue body was wrong and is refuted.
The annexed content is not missing:
data_complete=1andbytes_presentis within 0.1% offile_sizefor every affected dataset,and I verified byte-exactness directly.
For a failing
on006233recording the downloaded object is 4,211,536 byteswith sha256
4a6696...336598,both matching the git-annex key exactly.
The data is intact.
In every case the signal data is fully readable.
The rejection comes either from a reader that is stricter than the format's real-world variation,
or from metadata that is irrelevant to serving a time series.
1. EDF rejected by pyedflib but read fine by MNE (1,084 recordings)
biosigio's in-memory EDF path reads via
pyedflib. Two separate real-world deviations make it refuse:Non-compliant header padding (
on006233347,on006914353, reportedcorrupt_or_truncated).The "number of data records" field contains
b'421\x00', NUL-padded instead of space-padded as theEuropean Data Format (EDF) specification requires.
pyedflibraisesOSError: the file is not EDF(+) or BDF(+) compliant (Number of Datarecords).MNE reads the same file without complaint: 4 channels, 421,000 samples, 1000 Hz.
Note these two datasets are NOT duplicate imports as I first suspected; they are distinct OpenNeuro
accessions (
ds006233"Picture naming",ds006914"Visual Naming EC") from what appears to be thesame lab, which is why their subject, session, and task naming coincide.
Discontinuous EDF+D (
on006910384, reportedfile_read_error).pyedflibraisesOSError: The file is discontinuous and cannot be read.MNE reads it: 66 channels, 295,410 samples, 1000 Hz. This file's header is clean (zero NUL bytes).
2. Elekta MaxShield FIF (439 recordings)
on006012(193) andon006720(246), both reportedfile_read_error.mne.io.read_raw_fifrefuses withValueError: This file contains raw Internal Active Shielding data... Use allow_maxshield=True if you are sure you want to load the data.With
allow_maxshield=Trueboth read fine (on006012: 336 channels, 341,000 samples;on006720: 328 channels, 127,000 samples, 1000 Hz).This one is a judgment call, not just a bug: MNE warns the data may be distorted and that Elekta
recommends running MaxFilter first. Serving unfiltered Internal Active Shielding data in a viewer
without saying so would be misleading. See the recommendation below.
The
zarr_queue.pydocstring already cites "a MaxShield MEG.fif" as its canonical example of adeterministic data failure, so this was known, but it was treated as terminal rather than handled.
3. CTF
.hcfiducial naming dialect (380 recordings)on006502, reportedfile_read_error. Fails in_make_ctf_coord_trans_setwithRuntimeError: Some of the mandatory HPI device-coordinate info was not there.Root cause is in MNE's
.hcparser (mne/io/ctf/hc.py). Its_kind_dictmatches the lowercaseliterals
"nasion","left ear","right ear", and the comparison is case-sensitive with noaliases. This dataset's
.hcwritesNasion,LPA, andRPA, so all three fiducials come backunmatched (
kind = -1) and the transform build fails.Proof: renaming
Nasiontonasion,LPA coiltoleft ear coil, andRPA coiltoright ear coilin a copy of the.hcmakes the identical directory read successfully:306 channels, 197,800 samples, 600 Hz. All ten
.dsmembers were verified byte-exact againsttheir annex keys first, so this is not a data problem.
Worth emphasizing: this is a head-coil coordinate transform, used for source localization and sensor
geometry. It has no bearing on serving a time series. A whole 380-recording dataset is withheld over
metadata the serving copy never uses.
4. Side finding: BIDS special files treated as recordings
on006012andon006720both report failures on_acq-crosstalk_meg.fif.The Brain Imaging Data Structure reserves
_acq-crosstalk_meg.fifand_acq-calibration_meg.datas Elekta calibration files, not recordings. Reading the crosstalk file fails with
ValueError: Could not find measurement data, correctly, because it contains none.These should be excluded by path convention rather than attempted and reported as failures.
This belongs with the raw-only scope work in #1097 and #1098.
Recoverable total
.hcdialectNot yet diagnosed:
on005545(336, non-deterministic),on005931(16),nm000276(40, infra),on005670(2, infra),on005356(stalependingqueue row with 30 stores).Recommendations
constraint:
generate_zarr.pydocuments that MNE rescales EDF to SI volts whilepyedflibdoesnot, so a fallback must normalize units or the in-memory and streaming paths will disagree on the
same file. This needs a unit-parity test, not just a try/except.
allow_maxshield=Trueandsurface an explicit caveat in the index so the viewer can label it as unfiltered Internal Active
Shielding data, or keep declining but report an honest "requires MaxFilter" reason instead of a
generic
file_read_error. I lean toward converting with the caveat, since the alternative showsusers nothing at all, but this is a scientific-communication call rather than a technical one.
.hcdialect. The clean fix is upstream: make MNE's_kind_dictmatchingcase-insensitive and add
LPA/RPAaliases, which is a small, well-evidenced patch worthsending to MNE. In the meantime the converter can normalize a scratch copy of the
.hc.Reproduction for all of the above uses public S3 objects
(
https://s3.us-east-2.amazonaws.com/nemar/<dataset>/objects/<annex-key>),with annex keys read from the git symlink blobs, so any of this can be re-checked independently.
neuromechanist commentedon Aug 20, 2026
Update: the failure picture was distorted, and there are two more root causes
Two corrections to my earlier comment, then two new findings.
Correction 1: raw coverage is actually 98.86%, not alarming
My earlier framing leaned on total failure counts, which are dominated by non-raw paths. Recomputed
ignoring
derivatives/,sourcedata/, andcode/entirely, the picture is much better:So the honest statement of the gap is not "thousands of failures", it is "23 datasets show the user
nothing." That is the number worth fixing.
Correction 2: my count of all-fail datasets was an UNDERCOUNT
I previously found 11 by looking for datasets with no published index. That missed a case: a dataset
whose raw recordings ALL fail can still publish an index if it has
derivatives/stores, so it lookshealthy from the outside.
on005520is exactly that — 92 stores, every one fromderivatives/preprocessed/, and 67 of 67 raw BrainVision recordings failing.This matters for the raw-only scope change: excluding
derivatives/will not "removeon005520'scoverage", it will reveal that the dataset has no working raw coverage at all. The derivatives
stores were masking a total raw failure. Worth stating plainly so the change is not mistaken for a
regression when that dataset goes from 92 stores to 0.
I also checked the concern about
sourcedata/-only datasets before anything gets deleted: there arenone. All four datasets with
sourcedata/stores (on005178,on005185,on005207,on005403)also have healthy raw coverage, so excluding
sourcedata/costs no dataset its viewer copy.on005520is the single at-risk dataset, and it is at risk fromderivatives/, notsourcedata/.New root cause 5: BrainVision header pointers not updated after BIDS renaming (~470 recordings)
The largest remaining cluster.
on005520's.vhdrcontains:while the files actually on disk are
sub-01_task-restingeyeclosed_eeg.eeg/.vmrk. BIDS forbidsunderscores inside a label value, so the conversion to BIDS renamed the files and never rewrote the
header's internal pointers. MNE therefore raises
FileNotFoundErrorlooking for a file that does notexist, which surfaces as
file_read_error.Proof: rewriting only those two lines to the real sibling filenames makes the identical data read
cleanly, 67 channels, 417,997 samples, 1000 Hz, values in a sane microvolt range. All three triplet
members were verified byte-exact against their annex keys first, so the data is intact.
Confirmed
.vhdrfile_read_errordatasets:on005863(357),on005520(67),on007471(32),on007813(4), andon007406(10, extension not individually confirmed) — roughly 470 recordings.Note this is a genuine defect in the DATASETS, not in our converter: the header really is wrong.
Two options, and I would do both:
targets are the same-stem siblings sitting next to the header, so the rewrite is deterministic and
safe. The scratch copy is discarded and BIDS stays the source of truth, so nothing authoritative
is mutated. This recovers ~470 recordings without waiting on anyone.
hits the same wall. Arguably the BIDS validator should flag a
.vhdrwhoseDataFiledoes notresolve.
Still undiagnosed
on007509(70 EEGLAB.set,file_read_error) is a separate cause I have not chased.on006033(2) andon006269/on008082(non-deterministic, so infra) remain open.on005571,on005692, andon005873are not failures at all: they sit atzarr_status=pendingwithno stores, which looks like stale queue rows rather than anything wrong with the data.
Revised recoverable total
pyedflibrejection.hcfiducial dialectEvery one of these is a reader-strictness or metadata problem on data that is byte-intact and
readable. None is a data-loss problem.
neuromechanist commentedon Aug 20, 2026
Final: complete root-cause taxonomy, and a correction to my own headline number
Correcting myself: catalog-wide raw coverage is 91.11%, not 98.86%
My 98.86% figure was computed only over datasets that have a published index, which silently
excluded exactly the worst cases: the 22 datasets where every recording fails publish no index at all,
so they contributed nothing to that denominator. Including them:
Of the 2,988 in no-index datasets, 2,392 are flagged deterministic (reproducible, therefore
diagnosable and fixable) and 596 are non-deterministic infrastructure failures. Four further datasets
(
on005356,on005571,on005692,on005873) are not failures at all: they sit atzarr_status=pendingwith no stores, which looks like stale queue rows and is a separate, probablycheap, fix.
So roughly 2,793 recordings are fixable through the six causes below, which would take raw coverage
from 91.11% to about 98.4%.
The six root causes
Every one of these is a reader-strictness or metadata problem on data that is byte-intact. I verified
byte-exactness against git-annex
SHA256Ekeys before diagnosing each. No data loss anywhere.1. European Data Format rejected by
pyedflibthough MNE reads it (1,084 recordings).Two variants. Non-compliant NUL-padded header field: the "number of data records" field contains
b'421\x00'instead of space padding, sopyedflibraisesthe file is not EDF(+) or BDF(+) compliant (Number of Datarecords)while MNE reads it fine(
on006233347,on006914353). Discontinuous EDF+D:pyedflibraisesThe file is discontinuous and cannot be read; MNE reads 66 channels / 295,410 samples(
on006910384). Fix: fall back to MNE whenpyedflibrefuses. Constraint already documented ingenerate_zarr.py: MNE rescales EDF to SI volts andpyedflibdoes not, so the fallback needs aunit-parity test, not a bare try/except.
2. Elekta MaxShield / Internal Active Shielding FIF (439 recordings).
read_raw_fifrefuses withoutallow_maxshield=True; with it,on006012reads 336 channels andon006720328 channels. Still needs a human decision, because MNE warns the data may be distortedand Elekta recommends MaxFilter first: convert with an explicit caveat surfaced in the index, or keep
declining but report an honest "requires MaxFilter" reason rather than a generic
file_read_error.3. BrainVision header pointers not updated after BIDS renaming (~470 recordings).
.vhdrsaysDataFile=sub-01_task-resting_eye_closed_eeg.eegwhile the file on disk issub-01_task-restingeyeclosed_eeg.eeg, because BIDS forbids underscores inside a label value and theconversion renamed files without rewriting the header. MNE raises
FileNotFoundError. Rewriting onlythose two lines reads 67 channels / 417,997 samples / 1000 Hz. Datasets:
on005863(357),on005520(67),on007471(32),on007813(4),on007406(10), plus 88 more spread acrosson005273,on005274,on005486,on006576and others. This is a defect in the datasets;worth both a deterministic scratch-copy repair (targets are unambiguous same-stem siblings) and an
upstream report.
4. CTF
.hcfiducial naming dialect (380 recordings).on006502. MNE'smne/io/ctf/hc.py_kind_dictmatches lowercase"nasion","left ear","right ear"case-sensitively with no aliases; this dataset writesNasion,LPA,RPA, so allthree fiducials go unmatched and
_make_ctf_coord_trans_setraisesSome of the mandatory HPI device-coordinate info was not there. Normalizing the names reads306 channels / 197,800 samples / 600 Hz. Note this is a head-coil coordinate transform used for source
localization; it has no bearing on serving a time series, so a whole 380-recording dataset is withheld
over metadata the serving copy never touches. Best fixed upstream in MNE (case-insensitive matching
plus
LPA/RPAaliases) — a small, well-evidenced patch.5. MATLAB v7.3 EEGLAB
.setfiles (219 recordings) — NEW, and the largest single cluster in theindexed datasets.
MNE's
read_raw_eeglabraisesNotImplementedError: Please use HDF reader for matlab v7.3 files, e.g. h5py, and biosigio surfaces that asFileReadError. A v7.3.setis an HDF5 container, not aclassic MAT file, so
scipy.io.loadmatcannot read it either. Affects 7 datasets:on005178,on005185,on005343,on006040,on006554, and others.I confirmed the data is fully reachable, so this is implementable rather than a dead end. Opening
on005343's 557 MBsub-S10_task-resting_eeg.setwithh5pygivesEEG/dataas a(1236409, 129) float32array (HDF5 stores it transposed relative to MATLAB),nbchan=129,srate=1000,pnts=1236409,trials=1(continuous), and channel labels resolving through the#refs#table toE1...E129. Fix belongs in biosigio as an h5py-based reader for v7.3.set,selected by sniffing the
MATLAB 7.3 MAT-filemagic. Watch the transpose and the object-referenceindirection for
chanlocs.labels.6. BIDS calibration files treated as recordings.
_acq-crosstalk_meg.fifand_acq-calibration_meg.datare Elekta calibration files, not recordings.Reading the crosstalk file raises
ValueError: Could not find measurement data, correctly, because itcontains none. Already folded into the raw-only scope work in #1097.
Still open
on005545(336, flagged non-deterministic despite the volume, so worth a second look),on008082(173, infra),nm000276andon006269(40 each, infra),on006033(5),on005670(2),and the 4 stale
pendingrows.neuromechanist commentedon Aug 21, 2026
MaxShield policy settled: filter properly when we can, decline honestly when we cannot
Decision: apply Signal-Space Separation only when both the Elekta fine-calibration and cross-talk
files are available, and otherwise decline with the same message MNE itself gives, rather than
serving unfiltered Internal Active Shielding data.
This turns out to fit the data almost exactly, and it is better than either option I originally put
forward. We do not need MEGIN's proprietary MaxFilter:
mne.preprocessing.maxwell_filteris anopen-source implementation of the same Signal-Space Separation family, documented as broadly
comparable to MEGIN MaxFilter 2.2.11.
Verified on real data
Ran it against
on006012'ssub-01_task-POGS_run-06_meg.fifusing that dataset's ownsub-01_acq-calibration_meg.datandsub-01_acq-crosstalk_meg.fif:info["maxshield"]flips fromTruetoFalse. This is the strongest available signal: MNE'sown guardrail, the thing that was refusing to load these files, is satisfied by the result. The
output is no longer raw Internal Active Shielding data by MNE's own reckoning.
being removed rather than the data being passed through untouched.
proc_history(
in_order=8,out_order=3,nfree=72,frame=head, plus bothsss_calandsss_ctcrecords),so the processing is self-documenting and auditable afterwards.
Temporal Signal-Space Separation (
st_duration) also works, and filtering without the calibrationpair works too, but under this policy we will not use that second path.
The files we need are already in the datasets
Every subject in both affected datasets ships both files:
Every single declined recording is
sub-emptyroom. Those are empty-room noise referencemeasurements rather than subject data, so declining them is the right outcome on its own merits, not
a shortfall. No actual subject recording is lost to this policy.
Correcting my earlier count
I previously reported 439 recordings for this cause, taken from the
zarr_errorscolumn. That figureincluded the 43 calibration and cross-talk files themselves (20 plus 23), which were being attempted
as recordings and are now excluded as the calibration files they are. The real breakdown is 396
_meg.fifrecordings, of which 365 are recoverable and 31 are empty-room.One consequence worth stating plainly
A Signal-Space Separation filtered store is not a faithful copy of its source recording, unlike
every other store in the serving copy. It is a processed derivative. That has to be recorded in the
index so consumers know, and it matters especially for the machine-learning streaming path, where a
model trained across datasets would otherwise silently mix filtered and unfiltered magnetoencephalography.
The
proc_historyprovenance makes the processing auditable, but the index needs to say it too.This is a decision worth an Architecture Decision Record, since it changes what the serving copy
guarantees for one class of recording and would be expensive to reverse quietly.
neuromechanist commentedon Aug 21, 2026
Correction: my whole analysis covered 26% of the catalog
Every count I reported in this issue was computed over 200 datasets. The catalog has 754.
GET /datasetscapslimitat 200 and paginates byoffset, and I read the first page as if it wereeverything without checking
total_count, which the response carries and which says 754.The root-cause mechanisms below are all still valid, and they were diagnosed from real files, so none
of that changes. What changes is scale: the fixable volume is roughly four times what I reported.
Notably, the coverage rate barely moved (91.11% on the sample, 91.81% on the full catalog), so the
200-dataset sample happened to be representative proportionally. That is exactly what made the error
easy to miss and is not a defense of it.
Corrected figures, all 754 datasets
pending)Of the 9,977 no-index failures, 8,748 are flagged deterministic, so diagnosable and fixable, and
1,229 are infrastructure failures.
Raw failure clusters, corrected
.vhdr.set.ds.fif.edf.fif.ds.bdf.edfThe BrainVision header cluster is the largest at 866, up from the ~88 I saw in the sample, and the
EEGLAB v7.3 cluster is 544, up from 219. Both root causes were correctly identified; only the counts
were low.
Large datasets I never looked at
on007523(579 failures),on007524(500),on004444(465),on004447(418),nm000177(360),nm000173(300). All flagged deterministic.A seventh root cause, already documented and independent of me
on004444is the subject of an existing biosigIO issue, neuromechanist/biosigio#109: an EDF file isrejected as corrupt when any channel has
physical_min == physical_max. That is legal EDF, since areference electrode in a referential montage is identically zero by construction, and one such channel
discards the whole recording. That issue reports 469 affected recordings with byte-exact evidence,
and it predates this work. It belongs in the taxonomy as cause 7, and it is the same family as cause 1
(a strict reader refusing readable data).
What this changes about priorities
Nothing about the diagnoses, and nothing about the decisions already taken. It does mean the purge is
substantially larger than described when it was authorized (17,234 stores, not 4,721), so that is worth
re-confirming rather than assuming the earlier approval covers a target nearly four times bigger.
neuromechanist commentedon Aug 21, 2026
The 87 no-index datasets, clustered: five fixes cover 62 of them
Pulled the recorded failure detail for all 87 and clustered by format and error code. The result is
unusually clean, and it lands well inside a target of "fewer than 10 to 20 left over."
Every one of the 87 falls into three buckets
pendingwith zero errorsfailed, non-deterministic, no failure detailNothing in the 87 is unexplained. The 8 in the last bucket all carry
deterministic=0, which by designmeans transient infrastructure failure (a crashed worker, an S3 blip) rather than a data problem, so
they are candidates for a retry rather than a diagnosis:
on008082(173),on004621(167),nm000276(40),on006269(40),on004278(30),on006446(29),on004598(20),on005670(2).The 17
pendingrows have zero recorded errors and zero stores, which looks like queue state that wasnever advanced rather than anything wrong with the data:
nm000225,on004080,on004107,on004395,on004917,on004998,on005127,on005356,on005571,on005692,on005873,on007720,on007822,on007864,on007865,on007895,on007968. Worth a look atzarr_queue.py'sreconcileto understand why they are stuck.The five fixes, ranked by datasets recovered
physical_min == physical_max, biosigio#109).hcfiducial dialectCluster detail, by dominant format and code across the 87:
.set/file_read_error 19 datasets (1,638recordings),
.edf/corrupt_or_truncated 17 (3,147),.fif/file_read_error 9 (2,267),.vhdr/file_read_error 8 (704),.edf/file_read_error 3 (736),.bdf/corrupt_or_truncated 3 (449),.ds/file_read_error 2 (534),.ds/corrupt_or_truncated 1 (1).The 80/20 answer
The single highest-value remaining item is the EDF/BDF strict-reader family at 23 datasets and 4,332
recordings, and it is the one not yet being worked. It is also already documented with byte-exact
evidence in biosigio#109, which reports
physical_min == physical_maxon a reference channeldiscarding an entire otherwise-valid recording. Combining it with the
corrupt_or_truncatedvariantsalready diagnosed here (NUL-padded header field, discontinuous EDF+D) makes one coherent piece of work
on the EDF read path.
Doing all five fixes plus the two operational actions would take the no-index count from 87 to
approximately 0, not merely under 20.
neuromechanist commentedon Aug 21, 2026
The 17
pendingdatasets: not a pagination bug, and not one single causeI suspected
zarr_queue.py'sreconcilewas only seeing the first 200 datasets, the same trap I fellinto myself. That is wrong: it paginates correctly.
fetch_public_datasetswalksoffsetuntiltotal_count, and its docstring already documents the 200-per-page cap explicitly. It also documentssomething useful that explains unrelated failures I hit:
api.nemar.orgsits behind Cloudflare, which403s the default Python urllib user-agent as a bot, which is why direct urllib calls fail where curl
succeeds.
Confirmed the "no index" classification is real rather than a fetch artifact: sampled these and they
return a genuine HTTP 404 for
index.json, with an empty body, not a transport error.The 17 split into at least two different situations:
Converted once, but serving no index.
on004080haszarr_converted_at = 2026-06-24and azarr_source_commit, andon005356has2026-07-07plus a recordedzarr_store_countof 30. So aconversion ran and a callback landed, yet
index.json404s today and the status never advanced pastpending. Something produced stores without publishing (or while later losing) the index.Never attempted.
on007968andon005873have nozarr_converted_at, nozarr_checked_at, andno
zarr_source_commitat all.nm000225was checked on 2026-06-07 but never converted.Distinguishing these properly needs evidence I cannot reach from outside: the queue's own SQLite state
on the conversion host and the cron logs. The queue tracks its own
statusseparately from D1'szarr_status, so a row could bedonein the queue while D1 still readspending, or the reverse,and only the host can say which.
So this bucket is not a code fix and not one root cause. It needs a look at
/mnt/local/zarr-stateand.nm-zarr-cron.logon the conversion host, and the useful first questionis whether the queue believes these are
donewhile D1 believes they arepending, since that wouldpoint at the callback rather than the conversion.
Recording this so the bucket is not mistaken for a single fixable defect. It is 17 datasets and,
per the recorded counts, 0 failing recordings, so it is the lowest-value of the three buckets by volume
even though it is the largest by dataset count after the root-cause group.
neuromechanist commentedon Aug 22, 2026
Root cause found for the infra subset of this track's 11 datasets, from a separate investigation now tracked as epic #1108.
What it is
The Hallu converter's pool workers are being OOM-killed, and a single kill poisons the whole pool.
/mnt/local/zarr-state/.nm-zarr.logcarries 383worker crashedevents and 96BrokenProcessPoolaborts. When one worker dies,ProcessPoolExecutormarks the pool unusable, every remaining recording fails withworker crashed: A process in the process pool was terminated abruptly, and the run returns rc=1 with the untouched tail of the worklist lost.Worked example from 2026-08-22:
on004998converted 74 of 115 recordings, then lost the remaining 41 to a single kill.on004917lost all 24, with 7 workers dying simultaneously.The mechanism is the RAM admission projection in
generate_zarr.py.projected_peak_bytesestimates the in-memory path atsize_bytes * INMEM_MEM_FACTORwith the factor set to 6, and admission dispatches while the in-flight sum fitsMemTotal * 0.8(~49 GiB on Hallu).on004917's 24 BrainVision blobs are 1.18-2.25 GB each and all but one sit just under theSTREAM_MIN_BYTES = 2 GiBstreaming threshold, so 23 of 24 took the unbounded in-memory path. Admission packed exactly 7 of them (45.2 GiB projected plus 4 GiB for the one streaming recording, against a 49.6 GiB ceiling) and those 7 are precisely the 7 that crashed. The real cost is closer to 12x on-disk bytes, not 6x, so seven concurrent is ~108 GB on a 62 GB node.Which of this track's datasets it explains
failed attempts=5, i.e. retry-exhausted and it will never re-attempt on its ownWhich it does not
The
file_read_errorandcorrupt_or_truncatedclusters remain this track's to explain. In particularon006233andon006914both reporting exactly 347corrupt_or_truncatedis still unaccounted for, and an OOM cascade would not produce a uniform typed data code. Keeping this issue open for that half.Where the fix lives
Epic #1108, phases 2-5: contain the blast radius so one OOM costs one recording rather than a dataset, make the projection measurement-based, widen the bounded streaming path, and requeue the stranded backlog (which is where
on005356and the other retry-exhausted datasets get recovered).A mitigation is already live:
ZARR_STREAM_MIN_BYTES=268435456was added to the Hallu crontab on 2026-08-22, routing.vhdr/.fif/.ds/.mefdat or above 256 MB to the bounded streaming path.neuromechanist commentedon Aug 22, 2026
Diagnosis
Investigated against the live conversion node and S3 on 2026-08-22. The leading hypothesis in the issue is disproved, and the headline framing is out of date. Nine of the eleven need no fix at all.
1. The annex-retrieval hypothesis is wrong: the data is present
data_complete/bytes_presentfromGET /datasets/{id}:Confirmed at the object level rather than trusting the column: pulled
sub-007/ses-1/ieeg/sub-007_ses-1_task-picture_ieeg.edffroms3://nemar/on006233/objects/and got 56,479,648 bytes of real EDF, exactly matching the-s56479648in its annex key. Not a symlink, not a placeholder.2. Ten of the eleven are not failing. They are queued and have never run.
From the queue DB (
zarr-queue.db), all of on006233, on006914, on006502, on006910, on006720, on006012, on005931, on005545, on005670 and nm000276 are:The per-dataset failure counts in the issue body are historical, recorded before a bulk requeue on 2026-07-21. Nothing has re-attempted these since, so nothing has re-failed. "Every recording fails" is no longer a true statement about current state.
3. The EDF failures were a false rejection by an older biosigIO, already fixed
on006233 holds 347
.edffiles and reported 347 failures; on006914 holds 353 and reported 347 + 6 = 353. Both were 100% failure, which is what "uniform per dataset rather than scattered" was pointing at. (The "exactly 347 in both" coincidence noted in the issue is just that: on006233 has 347 EDFs in total, while 347 of on006914's 353 carried that particular code.)Five files, spread across both datasets and different subjects, read cleanly under the deployed biosigIO 1.2.4 via the same
Recording.from_fileentry point the converter uses:This matches the pin note in
scripts/zarr/requirements.txt: 1.2.4 "recovers three false EDF/BDF rejections." These datasets last ran on a build that had the bug.4. The CTF failures are explained by format support, not corruption
on006502's file census is
.acqx380,.cfgx380,.clsx380 — those are CTF.dsinternals, i.e. 380 CTF directory recordings. Directory-recording discovery arrived in Track 2 (.github#98/#100) and biosigIO 1.2.3+. The issue already suspected this ("if retrieval is the cause it would also explain why a directory format fails wholesale") — the cause is different, but the prediction that on006502 is one wholesale-format case is right. Same story for the otheranat,megdatasets (on006720, on006012).5. The queue is healthy and these are days away
Datasets reaching a terminal state per ISO week:
154 pending at ~100-200/week is roughly a week of backlog.
claim_nextis FIFO onenqueued_at, so nothing is starved. Worth noting the current head of that queue is on004917 — the BrainVision dataset whose OOM opened epic #1108 — so its next run doubles as a live check of the memory fixes.What actually needs action
Two of the eleven, and neither is a zarr bug:
data_complete=0against 3.25 TB. A genuine data-completeness problem belonging to the import/recover tooling (nemar admin recover, epic Incident: OpenNeuro auto-import publishes 0-byte content (56 datasets, ~680GB missing) #967), not to conversion. Zarr cannot succeed here until the content is present.failed(attempts=5). It is in the 65-dataset requeue set staged in Finish the Hallu zarr cutover once the on007808 drain releases the lock #1122.Recommendation
No zarr code change for the other nine. Let the queue drain, then re-measure coverage; the two causes above (biosigIO 1.2.4 EDF recovery, directory-format discovery) are already deployed, so these should convert on their next natural turn. I would keep this issue open until that re-measurement confirms it rather than closing on the prediction.
Evidence gathered read-only; nothing was requeued or mutated.
neuromechanist commentedon Aug 22, 2026
Resolved: three causes, not one
Following up on my earlier comment, which was partly wrong. I had predicted the CTF and MEG failures were explained by directory-format support landing in Tracks 2 and 3. I then actually converted samples from each dataset through the same
Recording.from_file->to_zarrcall the converter uses, and that prediction is false. Conversion, not just reading, is what this issue is about, and the eleven datasets split into three unrelated causes.Cause 1 — EDF: already fixed, waiting on the queue
.edf.edf.edf.edf.edfThese failed under an older biosigIO that falsely rejected valid EDF/BDF; 1.2.4 recovers exactly that class. Nothing further is needed. All ten of the non-
on005356datasets arestatus='pending' attempts=0(enqueued 2026-07-01, never re-attempted), so the counts in the issue body are historical, not current. The queue is FIFO onenqueued_at, draining 100-200 datasets/week with 154 pending, so these publish on their own within about a week.Incidental:
on005545andon006910return the same annex key for their first recording, so they share content.Cause 2 — CTF: a real, unfixed reader incompatibility (380 recordings)
on006502still fails, withfile_read_error: "Some of the mandatory HPI device-coordinate info was not there."Root-caused rather than guessed. MNE's
_read_hcmaps head-position-indicator descriptors through a literal, case-sensitive lookup:This dataset's
.hcfiles spell themNasion,LPA, andRPA. Every coil point therefore parses withkind = -1, the LPA/RPA/NAS check in_make_ctf_coord_trans_setfinds nothing, and it raises. Confirmed by running MNE's parser directly: 6 points, allkind=-1, 3 in device frame and 3 in head frame, allvalid=True. The coordinates are present and correct; only the labels differ.Proved by rewriting only those three descriptor labels in a copy and reconverting the otherwise byte-identical recording:
This is a vendor naming variant MNE does not accept, not corruption and not a NEMAR bug. Filed separately; deliberately not hacked around here, since normalizing acquisition metadata before reading is a correctness-sensitive change that deserves its own decision.
Cause 3 — MaxShield MEG: already decided, never implemented (439 recordings)
on006720(246.fif) andon006012(193.fif) both fail with MNE's Internal Active Shielding refusal:This is not a new finding. ADR 0028, accepted 2026-08-20 out of this very epic, names both datasets and decides to apply Signal-Space Separation where the fine-calibration and cross-talk sidecars resolve, and otherwise to decline with a specific reason rather than a generic read error. Its own consequences section projects 365 of 396 recordings becoming viewable, the remaining 31 being
sub-emptyroomreferences.The decision was never implemented: there is no
maxwell_filterorallow_maxshieldanywhere in the converter, only the ADR 0027 calibration-file exclusion. Filed separately.The remaining three
data_complete=0against 3.25 TB. A data-completeness problem for the import/recover tooling, not conversion. Zarr cannot succeed until the content is present.failedrow (attempts=5), included in the staged requeue.Closing this
The question this issue asked, why eleven datasets publish no index, is now answered with evidence for every one of them. Closing it, with the two pieces of real work it uncovered tracked on their own issues so neither hides inside a diagnosis ticket.
All verification was read-only against S3 and ran outside the conversion work directory; the in-flight drain was unaffected.
Tracked follow-ups: #1125 (CTF
.hccoil naming, 380 recordings) and #1126 (implement ADR 0028 for MaxShield, 439 recordings).