Repository navigation
Description
ADR 0028 ("MaxShield MEG is Signal-Space Separation filtered before serving, or declined") was accepted on 2026-08-20 and has never been implemented. on006012 and on006720 still publish no Zarr index.
Confirmed while resolving #1100 by converting a real recording from each through the same Recording.from_file -> to_zarr call the converter uses:
on006720 sub-155_task-rest_meg.fif 164M -> FAILED file_read_error
on006012 (first task .fif) 439M -> FAILED file_read_error
Both with MNE's refusal:
This file contains raw Internal Active Shielding data. It may be distorted. Elekta recommends it be run through MaxFilter to produce reliable results.
The converter has no maxwell_filter and no allow_maxshield; the only MaxShield-adjacent code is the ADR 0027 exclusion that keeps _acq-crosstalk_meg.fif and _acq-calibration_meg.dat from being discovered as recordings.
What the ADR already decided
Read .context/decisions/0028-*.md for the reasoning; the decision itself:
- Apply Signal-Space Separation via
mne.preprocessing.maxwell_filterand serve the filtered result, only when both the fine-calibration and cross-talk files resolve for that recording. - When either is missing, decline deterministically and surface MNE's own explanation rather than a generic read error.
- Unfiltered Internal Active Shielding data is never served, and SSS is never applied without the site-specific calibration that makes it trustworthy.
Its consequences section projects 365 of 396 recordings becoming viewable, the 31 that stay declined all being sub-emptyroom references rather than subject data. Every subject in both datasets ships both sidecars (20 of 20 in on006012, 23 of 23 in on006720).
Work implied
- Resolve the fine-calibration and cross-talk sidecars for a recording through BIDS inheritance, before conversion starts.
- Read with
allow_maxshield=Trueand applymaxwell_filterwith those inputs. - Disclose the filtering in the index. The ADR is explicit that this matters: every other store is the source signal quantized and resampled, nothing more, while an SSS store is a processed derivative. A model training across datasets would otherwise silently mix filtered and unfiltered MEG. MNE writes the parameters into
proc_history, but the ADR states auditability inside the data is not the same as disclosure and the index has to say it too. - Give the declined case a specific failure code instead of
file_read_error, so a user can distinguish an unreadable file from a policy decision. - Account for the cost: SSS is a real computation rather than a read, so these recordings get materially more expensive to convert. Worth checking against the memory admission work from epic Epic: Zarr conversion memory robustness #1108 before turning it on for a 396-recording backlog. (An earlier draft of this issue said 439; that figure is wrong and ADR 0028 says why. It came from the
zarr_errorscolumn and counted the 43 calibration and cross-talk files themselves, which were being attempted as recordings before ADR 0027 excluded them. The correct figures are 396 affected, 365 recoverable, 31 declined.)
Note on scope
This is the only known MaxShield data in the catalog today, but the handling is general and any future MEGIN dataset acquired with Internal Active Shielding will need it.
Blocks the last two datasets from #1100 that are not already explained.
Activity
neuromechanist commentedon Aug 22, 2026
Implemented in #1127, merged to
devas e8e4572. Public docs in nemarOrg/docs#13.ADR 0028 is now built. Recordings carrying raw Internal Active Shielding are corrected with
mne.preprocessing.maxwell_filterusing the dataset's own fine-calibration and cross-talk pair, and served with the correction disclosed in both the store root attributes and the index entry. Where the pair does not resolve they are declined withmaxshield_uncalibratedinstead of an opaquefile_read_error.Verified on the conversion node through the shipped functions:
apply_ssspeaks at 2.95 GiB on a 716 MiB recording,info["maxshield"]is clear on the output, and the resolver reproduces this ADR's coverage projection exactly (207 of 223 inon006720, 158 of 173 inon006012, every decline undersub-emptyroom).Two things review caught that are worth recording:
is_maxshield_fifswallowed every exception silently. Each one it can swallow is a FIF whose header would not read, and returning False routes a genuinely shielded recording back to the pre-ADR behaviour with no trace of why. It warns now.maxwell_filterfailure other than a missing pair was uncoded and would have retried the filter forever at about a minute a time. A calibration pair that is present but does not fit the recording is as permanent as a missing one, so it gets the same terminal verdict;MemoryErrorstill propagates as retryable.And one I caught before commit: the memory factor was initially 12.2x, measured from a pipeline production does not run. The filtered copy streams, so conversion adds nothing to the filter's peak. Re-measured across both datasets and the size range (5.0x, 5.0x, 4.4x, 4.2x at 160, 164, 438, 716 MiB) and set to 6, which clears every point.
The 365-recording backfill has not run. At roughly 5 minutes per recording that is on the order of 30 hours of node time; it joins the deployment queue in #1122.