Skip to content

zarr: restrict discovery to raw BIDS recordings; skip derivatives/, sourcedata/, and eye-tracker EDF #88

Open

Description

@neuromechanist

Summary

scripts/zarr/generate_zarr.py discovers recordings purely by file extension, with no path or BIDS-datatype filter:

PRIMARY_EXTS = (".set", ".edf", ".bdf", ".vhdr", ".fif", ".con", ".sqd", ".kdf")
CTF_DS_EXT   = ".ds"

As a result the converter walks into derivatives/ and sourcedata/ and attempts to convert files that are not raw continuous recordings and never can be. These then land in the failure log and inflate the failure counts, masking real conversion bugs.

Impact

Of 7,959 unique failing recordings in the current Hallu backfill log, 2,016 (about 25%) should never have been attempted:

Scope Unique failures Extensions
derivatives/ 1,991 .fif 1,017, .set 546, .edf 428
sourcedata/ 25 .edf 22, .bdf 3

Within derivatives/, 172 are MNE epoched exports (*-epo.edf, *-epo.fif). Epoched data is not a continuous raw recording, so it is correctly unreadable by a raw reader; we simply should not be asking.

Eye-tracker EDF must be skipped

sourcedata/eyetracking/*.edf files (for example S22_Occ.edf, S15_Occ.edf) are EyeLink / SR Research eye-tracker files. That is a proprietary binary format which merely shares the .edf extension with European Data Format; it will never parse as EEG EDF, and converting it is meaningless.

Excluding sourcedata/ alone is not sufficient. Under the BIDS eye-tracking specification, eye-tracking data is also valid inside the BIDS tree proper, as the eyetrack datatype:

sub-01/eyetrack/sub-01_task-x_eyetrack.edf

So the exclusion needs to key on datatype and suffix, not only on the sourcedata/ prefix.

Note: I confirmed the sourcedata/eyetracking/ cases from the failure log; I have not yet audited the corpus for in-tree eyetrack/ datatype files, so please treat that as a defensive requirement rather than a measured count.

Proposed change

  • Skip any path under derivatives/ and sourcedata/ during discovery
  • Skip the eyetrack BIDS datatype directory and the _eyetrack BIDS suffix, regardless of location, so EyeLink .edf is never handed to the EDF reader
  • Skip epoched exports (*-epo.*), which are not continuous raw data
  • Log skipped-by-scope files distinctly from conversion failures, so the failure log measures real problems only

Why it matters beyond noise

Failure counts are currently used to judge conversion health. With a quarter of failures being files we chose to attempt and cannot ever convert, genuine regressions are hard to see. Related: nemarOrg/nemar-cli#909 (surface explicit downstream skip vs failure reasons).

Activity

  1. neuromechanist commented on Aug 1, 2026

    @neuromechanist
    MemberAuthor

    Verification of the eye-tracker point, now that I could re-check the corpus.

    All 22 eye-tracker failures currently sit under sourcedata/eyetracking/ (S1_Occ.edf through S27_Occ.edf), so a sourcedata/ prefix exclusion does catch every case present today. Searching the whole failure log for the BIDS _eyetrack suffix returns 0 matches.

    The _eyetrack guard is still worth adding, because nothing in discovery would stop it. bids_suffix_modality() only knows four suffixes:

    _SUFFIX_MODALITY = {"eeg": "EEG", "meg": "MEG", "ieeg": "IEEG", "emg": "EMG"}

    and it returns None for anything else. That None is used only to pick the sampling-rate cap, never to filter, so a file like sub-01/eyetrack/sub-01_task-x_eyetrack.edf would still be selected by is_primary() on its .edf extension and handed to the EDF reader.

    Suggested precedence: filter on path scope and BIDS datatype/suffix before the extension check, rather than relying on extension alone and letting the reader reject it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions