You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
zarr: restrict discovery to raw BIDS recordings; skip derivatives/, sourcedata/, and eye-tracker EDF #88
As a result the converter walks into derivatives/ and sourcedata/ and attempts to convert files that are not raw continuous recordings and never can be. These then land in the failure log and inflate the failure counts, masking real conversion bugs.
Impact
Of 7,959 unique failing recordings in the current Hallu backfill log, 2,016 (about 25%) should never have been attempted:
Scope
Unique failures
Extensions
derivatives/
1,991
.fif 1,017, .set 546, .edf 428
sourcedata/
25
.edf 22, .bdf 3
Within derivatives/, 172 are MNE epoched exports (*-epo.edf, *-epo.fif). Epoched data is not a continuous raw recording, so it is correctly unreadable by a raw reader; we simply should not be asking.
Eye-tracker EDF must be skipped
sourcedata/eyetracking/*.edf files (for example S22_Occ.edf, S15_Occ.edf) are EyeLink / SR Research eye-tracker files. That is a proprietary binary format which merely shares the .edf extension with European Data Format; it will never parse as EEG EDF, and converting it is meaningless.
Excluding sourcedata/ alone is not sufficient. Under the BIDS eye-tracking specification, eye-tracking data is also valid inside the BIDS tree proper, as the eyetrack datatype:
sub-01/eyetrack/sub-01_task-x_eyetrack.edf
So the exclusion needs to key on datatype and suffix, not only on the sourcedata/ prefix.
Note: I confirmed the sourcedata/eyetracking/ cases from the failure log; I have not yet audited the corpus for in-tree eyetrack/ datatype files, so please treat that as a defensive requirement rather than a measured count.
Proposed change
Skip any path under derivatives/ and sourcedata/ during discovery
Skip the eyetrack BIDS datatype directory and the _eyetrack BIDS suffix, regardless of location, so EyeLink .edf is never handed to the EDF reader
Skip epoched exports (*-epo.*), which are not continuous raw data
Log skipped-by-scope files distinctly from conversion failures, so the failure log measures real problems only
Why it matters beyond noise
Failure counts are currently used to judge conversion health. With a quarter of failures being files we chose to attempt and cannot ever convert, genuine regressions are hard to see. Related: nemarOrg/nemar-cli#909 (surface explicit downstream skip vs failure reasons).
Verification of the eye-tracker point, now that I could re-check the corpus.
All 22 eye-tracker failures currently sit under sourcedata/eyetracking/ (S1_Occ.edf through S27_Occ.edf), so a sourcedata/ prefix exclusion does catch every case present today. Searching the whole failure log for the BIDS _eyetrack suffix returns 0 matches.
The _eyetrack guard is still worth adding, because nothing in discovery would stop it. bids_suffix_modality() only knows four suffixes:
and it returns None for anything else. That None is used only to pick the sampling-rate cap, never to filter, so a file like sub-01/eyetrack/sub-01_task-x_eyetrack.edf would still be selected by is_primary() on its .edf extension and handed to the EDF reader.
Suggested precedence: filter on path scope and BIDS datatype/suffix before the extension check, rather than relying on extension alone and letting the reader reject it.
Activity
neuromechanist commentedon Aug 1, 2026
Verification of the eye-tracker point, now that I could re-check the corpus.
All 22 eye-tracker failures currently sit under
sourcedata/eyetracking/(S1_Occ.edfthroughS27_Occ.edf), so asourcedata/prefix exclusion does catch every case present today. Searching the whole failure log for the BIDS_eyetracksuffix returns 0 matches.The
_eyetrackguard is still worth adding, because nothing in discovery would stop it.bids_suffix_modality()only knows four suffixes:and it returns
Nonefor anything else. ThatNoneis used only to pick the sampling-rate cap, never to filter, so a file likesub-01/eyetrack/sub-01_task-x_eyetrack.edfwould still be selected byis_primary()on its.edfextension and handed to the EDF reader.Suggested precedence: filter on path scope and BIDS datatype/suffix before the extension check, rather than relying on extension alone and letting the reader reject it.