Skip to content

Zarr converter walks the whole repository, not the BIDS raw tree #1058

Open

Description

@neuromechanist

Summary

The Zarr serving-copy converter appears to walk every readable signal file in a dataset repository, including derivatives/ and sourcedata/. It should convert the BIDS raw tree only, and derived data, if converted at all, should be namespaced and explicitly marked.

This produces four distinct problems: a large share of the catalog's reported conversion failures are artifacts of converting things that cannot be converted; store_count overstates available recordings by up to 2x; the index is a silent provenance hazard for downstream clients; and at least one dataset has its entire Zarr copy built from sourcedata/ with the wrong modality.

Evidence

Dataset store_count Raw (sub-*) derivatives/ sourcedata/ Failures
on007763 (MEG) 105 41 64 0 41, all under derivatives/
on004718 (EEG) 96 48 48 0 6, all raw .set, file_read_error
on004362 (EDF) 3052 0 0 3052 none reported
nm000329 (EEG) 112 112 0 0 0

Reproduce for any dataset:

curl -s https://nemar.s3.us-east-2.amazonaws.com/<id>/zarr/index.json \
  | jq -r '.stores[].path' | cut -d/ -f1 | sort | uniq -c

1. Derivatives are the source of the failures. Every one of on007763's 41 failures is under derivatives/. The dominant code is not_continuous, thrown on trial-averaged _ave.fif files:

{"path":"derivatives/sub-01/meg/sub-01_task-BCCWJreading_meg_0.1-40-ica_ave.fif","code":"not_continuous"}
{"path":"derivatives/sub-02/meg/sub-02_task-BCCWJreading_meg_0.1-40-ica_ave.fif","code":"not_continuous"}

Those are evoked responses. There is no continuous signal to convert. Restricting the walk takes on007763 from 41 failures to zero. Worth checking how much of the catalog-wide failure count this explains before doing anything else.

2. store_count overstates available recordings. on004718 is exactly doubled. Any coverage or capacity figure computed from store_count is inflated by an unknown per-dataset factor, including the zarr_store_count column (migration 0035) and the observability dashboard.

3. Provenance hazard for downstream clients. This is the one with consequences outside NEMAR.

sub-01/meg/sub-01_task-BCCWJreading_meg.con                            <- raw
derivatives/sub-01/meg/sub-01_task-BCCWJreading_meg_0.1-40_ica_raw.fif  <- bandpass 0.1-40 Hz, ICA cleaned

Both appear in the same stores array, same format, same attributes. Nothing marks the second as derived, and nothing records what pipeline produced it. A client that matches on subject and task rather than exact path, or that simply iterates stores, will train on someone else's preprocessing and report it as raw NEMAR data. That error is invisible in the results.

The store-level note attribute says "Derived serving copy ... BIDS source remains authoritative", but that refers to the Zarr-vs-BIDS relationship, not to whether the BIDS source was itself a derivative. The two senses of "derived" collide.

4. sourcedata/ conversion, plus modality misdetection. on004362 has 3052 stores, all under sourcedata/rawdata/, and nothing from the BIDS raw tree. In BIDS, sourcedata/ is explicitly the pre-conversion material. The same dataset also mistypes modality:

{
  "path": "sourcedata/rawdata/S001/S001R01.edf",
  "modalities": ["misc"],
  "groups": [{"name":"misc_160hz","modality":"MISC","rate":160.0,"n_channels":64,"n_samples":9760,"duration_s":61.0}]
}

64-channel EEG typed as MISC. A client filtering on modality == "EEG" sees an empty dataset. Plausibly the same root cause: if modality detection keys on BIDS path structure (sub-*/eeg/), anything under sourcedata/ fails to classify. Fixing scope may fix typing for free.

Proposed fix

  1. Walk sub-*/[ses-*/]<datatype>/ only.
  2. Skip sourcedata/ unconditionally.
  3. Skip derivatives/ by default.
  4. Skip non-continuous files rather than reporting them as failures. A file skipped by policy should not appear in failures, or should carry a distinct skipped_by_policy code excluded from failure metrics.

If derived data is wanted later, and there is a reasonable case since some derivatives are the most usable version of a dataset, do it explicitly: a separate <id>/zarr/derivatives-index.json, a derived: true field, and pipeline provenance from the derivatives/<pipeline>/dataset_description.json GeneratedBy block.

Acceptance criteria

  • on007763 index contains 41 stores and 0 failures.
  • on004718 index contains 48 stores.
  • on004362 either converts its BIDS raw tree or reports zero convertible recordings, and does not expose sourcedata/.
  • Catalog-wide failure count re-measured after re-conversion, with the delta recorded.
  • on004362-style EEG typed as EEG, not MISC, or a separate issue opened if the cause is unrelated to scope.

Activity

  1. neuromechanist commented on Aug 22, 2026

    @neuromechanist
    ContributorAuthor

    Audit 2026-08-22: the code fix has landed; the published data has not caught up.

    Discovery now excludes the trees this issue is about. Verified directly:

    derivatives/prep/sub-01/eeg/sub-01_task-x_eeg.set   excluded=True  primary=False
    sourcedata/sub-01/eeg/sub-01_task-x_eeg.edf        excluded=True  primary=False
    code/x.set                                          excluded=True  primary=False
    sub-01/eeg/sub-01_task-x_eeg.set                   excluded=False primary=True
    

    That is ADR 0027 (raw-only discovery), delivered through epic #1095 Track 2. is_excluded_from_discovery is applied at discovery, in compute_worklist, and in compute_clean_orphans (so an already-published derivative store is not mistaken for an orphan and deleted).

    But this issue's own evidence still reproduces on the live index. on004362, cited here as 3052 stores all built from sourcedata/:

    store_count = 3052,  1526 of them under sourcedata/ | derivatives/ | code/
    

    The converter would not produce those today. They are stores published before the fix, and index.json carries them forward until the dataset is re-converted. The hourly cron runs --clean, so this self-heals the next time each affected dataset comes up in the queue; it is not a code defect any more.

    So what remains here is a re-conversion, not a change. Re-queueing the affected datasets is the action, and it belongs with the other queue operations in #1122 rather than being tracked as an open converter bug.

    Leaving this open only until the affected datasets have re-converted and their indexes are clean. Worth re-measuring on007763, on004718 and on004362 from the table above at that point, since they are the documented cases.

  2. neuromechanist commented on Aug 23, 2026

    @neuromechanist
    ContributorAuthor

    Status after v0.9.9 shipped ADR 0027 (raw-only discovery) and ADR 0023 (--clean reconciles rather than wipes).

    The code fix is live on Hallu. Re-checked all four datasets from the table above via the public index:

    Dataset stores non-raw state
    on004362 1526 0 fixed — was 3052, all sourcedata/; reconverted today at 16:21 UTC
    on007763 35 0 derivatives gone (failure_count still reports 41; worth a separate look)
    nm000329 112 0 was already clean
    on004718 96 51 derivatives/ not yet reconverted

    Keeping this open. Discovery is raw-only now, but a dataset only sheds its previously-published non-raw stores when it is actually reconverted, so on004718 and any other dataset still carrying them keep serving them until the backlog reaches them. That is the archive-wide purge AGENTS.md flags as a separate, explicitly authorized operation.

    Closing condition: no dataset in the catalog has a store under derivatives/, sourcedata/, or code/. The reproduce command in the issue body still works as the check.

    Worth noting on007763 reporting 35 stores against 41 failures — the derivatives are gone, so those failures are now something else and should not be assumed to be the old not_continuous ones.

  3. neuromechanist commented on Sep 2, 2026

    @neuromechanist
    ContributorAuthor

    Purge of the pre-ADR-0027 non-raw stores is executing on Hallu as of 2026-09-02 19:51 UTC (scripts/zarr/purge_non_raw_stores.py --all --bucket nemar --execute, audit record at /mnt/local/zarr-state/purge-nonraw-20260902T195139Z.json). Dry run on on004718 first: 51 derivatives/ stores targeted, 0 anomalies, 0 drift. Targets come only from each dataset's own index.json entries under derivatives/, sourcedata/ or code/, every delete is re-asserted inside <id>/zarr/, and the IAM profile can delete only under */zarr/*, so raw objects/ cannot be touched. The closing condition here (no dataset serving a store under those trees) is re-checked with the reproduce command in the issue body once the run finishes; nemar-cli phase 7 (#1204) additionally makes the converter drop any such carried-over entry from the index on every run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions