Repository navigation
Description
Summary
The Zarr serving-copy converter appears to walk every readable signal file in a dataset repository, including derivatives/ and sourcedata/. It should convert the BIDS raw tree only, and derived data, if converted at all, should be namespaced and explicitly marked.
This produces four distinct problems: a large share of the catalog's reported conversion failures are artifacts of converting things that cannot be converted; store_count overstates available recordings by up to 2x; the index is a silent provenance hazard for downstream clients; and at least one dataset has its entire Zarr copy built from sourcedata/ with the wrong modality.
Evidence
| Dataset | store_count |
Raw (sub-*) |
derivatives/ |
sourcedata/ |
Failures |
|---|---|---|---|---|---|
on007763 (MEG) |
105 | 41 | 64 | 0 | 41, all under derivatives/ |
on004718 (EEG) |
96 | 48 | 48 | 0 | 6, all raw .set, file_read_error |
on004362 (EDF) |
3052 | 0 | 0 | 3052 | none reported |
nm000329 (EEG) |
112 | 112 | 0 | 0 | 0 |
Reproduce for any dataset:
curl -s https://nemar.s3.us-east-2.amazonaws.com/<id>/zarr/index.json \
| jq -r '.stores[].path' | cut -d/ -f1 | sort | uniq -c1. Derivatives are the source of the failures. Every one of on007763's 41 failures is under derivatives/. The dominant code is not_continuous, thrown on trial-averaged _ave.fif files:
{"path":"derivatives/sub-01/meg/sub-01_task-BCCWJreading_meg_0.1-40-ica_ave.fif","code":"not_continuous"}
{"path":"derivatives/sub-02/meg/sub-02_task-BCCWJreading_meg_0.1-40-ica_ave.fif","code":"not_continuous"}Those are evoked responses. There is no continuous signal to convert. Restricting the walk takes on007763 from 41 failures to zero. Worth checking how much of the catalog-wide failure count this explains before doing anything else.
2. store_count overstates available recordings. on004718 is exactly doubled. Any coverage or capacity figure computed from store_count is inflated by an unknown per-dataset factor, including the zarr_store_count column (migration 0035) and the observability dashboard.
3. Provenance hazard for downstream clients. This is the one with consequences outside NEMAR.
sub-01/meg/sub-01_task-BCCWJreading_meg.con <- raw
derivatives/sub-01/meg/sub-01_task-BCCWJreading_meg_0.1-40_ica_raw.fif <- bandpass 0.1-40 Hz, ICA cleaned
Both appear in the same stores array, same format, same attributes. Nothing marks the second as derived, and nothing records what pipeline produced it. A client that matches on subject and task rather than exact path, or that simply iterates stores, will train on someone else's preprocessing and report it as raw NEMAR data. That error is invisible in the results.
The store-level note attribute says "Derived serving copy ... BIDS source remains authoritative", but that refers to the Zarr-vs-BIDS relationship, not to whether the BIDS source was itself a derivative. The two senses of "derived" collide.
4. sourcedata/ conversion, plus modality misdetection. on004362 has 3052 stores, all under sourcedata/rawdata/, and nothing from the BIDS raw tree. In BIDS, sourcedata/ is explicitly the pre-conversion material. The same dataset also mistypes modality:
{
"path": "sourcedata/rawdata/S001/S001R01.edf",
"modalities": ["misc"],
"groups": [{"name":"misc_160hz","modality":"MISC","rate":160.0,"n_channels":64,"n_samples":9760,"duration_s":61.0}]
}64-channel EEG typed as MISC. A client filtering on modality == "EEG" sees an empty dataset. Plausibly the same root cause: if modality detection keys on BIDS path structure (sub-*/eeg/), anything under sourcedata/ fails to classify. Fixing scope may fix typing for free.
Proposed fix
- Walk
sub-*/[ses-*/]<datatype>/only. - Skip
sourcedata/unconditionally. - Skip
derivatives/by default. - Skip non-continuous files rather than reporting them as failures. A file skipped by policy should not appear in
failures, or should carry a distinctskipped_by_policycode excluded from failure metrics.
If derived data is wanted later, and there is a reasonable case since some derivatives are the most usable version of a dataset, do it explicitly: a separate <id>/zarr/derivatives-index.json, a derived: true field, and pipeline provenance from the derivatives/<pipeline>/dataset_description.json GeneratedBy block.
Acceptance criteria
-
on007763index contains 41 stores and 0 failures. -
on004718index contains 48 stores. -
on004362either converts its BIDS raw tree or reports zero convertible recordings, and does not exposesourcedata/. - Catalog-wide failure count re-measured after re-conversion, with the delta recorded.
-
on004362-style EEG typed asEEG, notMISC, or a separate issue opened if the cause is unrelated to scope.
Activity
neuromechanist commentedon Aug 22, 2026
Audit 2026-08-22: the code fix has landed; the published data has not caught up.
Discovery now excludes the trees this issue is about. Verified directly:
That is ADR 0027 (raw-only discovery), delivered through epic #1095 Track 2.
is_excluded_from_discoveryis applied at discovery, incompute_worklist, and incompute_clean_orphans(so an already-published derivative store is not mistaken for an orphan and deleted).But this issue's own evidence still reproduces on the live index.
on004362, cited here as 3052 stores all built fromsourcedata/:The converter would not produce those today. They are stores published before the fix, and
index.jsoncarries them forward until the dataset is re-converted. The hourly cron runs--clean, so this self-heals the next time each affected dataset comes up in the queue; it is not a code defect any more.So what remains here is a re-conversion, not a change. Re-queueing the affected datasets is the action, and it belongs with the other queue operations in #1122 rather than being tracked as an open converter bug.
Leaving this open only until the affected datasets have re-converted and their indexes are clean. Worth re-measuring
on007763,on004718andon004362from the table above at that point, since they are the documented cases.neuromechanist commentedon Aug 23, 2026
Status after v0.9.9 shipped ADR 0027 (raw-only discovery) and ADR 0023 (
--cleanreconciles rather than wipes).The code fix is live on Hallu. Re-checked all four datasets from the table above via the public index:
on004362sourcedata/; reconverted today at 16:21 UTCon007763failure_countstill reports 41; worth a separate look)nm000329on004718derivatives/Keeping this open. Discovery is raw-only now, but a dataset only sheds its previously-published non-raw stores when it is actually reconverted, so
on004718and any other dataset still carrying them keep serving them until the backlog reaches them. That is the archive-wide purge AGENTS.md flags as a separate, explicitly authorized operation.Closing condition: no dataset in the catalog has a store under
derivatives/,sourcedata/, orcode/. The reproduce command in the issue body still works as the check.Worth noting
on007763reporting 35 stores against 41 failures — the derivatives are gone, so those failures are now something else and should not be assumed to be the oldnot_continuousones.neuromechanist commentedon Sep 2, 2026
Purge of the pre-ADR-0027 non-raw stores is executing on Hallu as of 2026-09-02 19:51 UTC (
scripts/zarr/purge_non_raw_stores.py --all --bucket nemar --execute, audit record at/mnt/local/zarr-state/purge-nonraw-20260902T195139Z.json). Dry run on on004718 first: 51derivatives/stores targeted, 0 anomalies, 0 drift. Targets come only from each dataset's own index.json entries under derivatives/, sourcedata/ or code/, every delete is re-asserted inside<id>/zarr/, and the IAM profile can delete only under*/zarr/*, so rawobjects/cannot be touched. The closing condition here (no dataset serving a store under those trees) is re-checked with the reproduce command in the issue body once the run finishes; nemar-cli phase 7 (#1204) additionally makes the converter drop any such carried-over entry from the index on every run.