Skip to content

Implement ADR 0028: SSS-filter MaxShield MEG or decline it specifically #1126

Closed

Description

@neuromechanist

ADR 0028 ("MaxShield MEG is Signal-Space Separation filtered before serving, or declined") was accepted on 2026-08-20 and has never been implemented. on006012 and on006720 still publish no Zarr index.

Confirmed while resolving #1100 by converting a real recording from each through the same Recording.from_file -> to_zarr call the converter uses:

on006720  sub-155_task-rest_meg.fif  164M  -> FAILED  file_read_error
on006012  (first task .fif)          439M  -> FAILED  file_read_error

Both with MNE's refusal:

This file contains raw Internal Active Shielding data. It may be distorted. Elekta recommends it be run through MaxFilter to produce reliable results.

The converter has no maxwell_filter and no allow_maxshield; the only MaxShield-adjacent code is the ADR 0027 exclusion that keeps _acq-crosstalk_meg.fif and _acq-calibration_meg.dat from being discovered as recordings.

What the ADR already decided

Read .context/decisions/0028-*.md for the reasoning; the decision itself:

  • Apply Signal-Space Separation via mne.preprocessing.maxwell_filter and serve the filtered result, only when both the fine-calibration and cross-talk files resolve for that recording.
  • When either is missing, decline deterministically and surface MNE's own explanation rather than a generic read error.
  • Unfiltered Internal Active Shielding data is never served, and SSS is never applied without the site-specific calibration that makes it trustworthy.

Its consequences section projects 365 of 396 recordings becoming viewable, the 31 that stay declined all being sub-emptyroom references rather than subject data. Every subject in both datasets ships both sidecars (20 of 20 in on006012, 23 of 23 in on006720).

Work implied

  1. Resolve the fine-calibration and cross-talk sidecars for a recording through BIDS inheritance, before conversion starts.
  2. Read with allow_maxshield=True and apply maxwell_filter with those inputs.
  3. Disclose the filtering in the index. The ADR is explicit that this matters: every other store is the source signal quantized and resampled, nothing more, while an SSS store is a processed derivative. A model training across datasets would otherwise silently mix filtered and unfiltered MEG. MNE writes the parameters into proc_history, but the ADR states auditability inside the data is not the same as disclosure and the index has to say it too.
  4. Give the declined case a specific failure code instead of file_read_error, so a user can distinguish an unreadable file from a policy decision.
  5. Account for the cost: SSS is a real computation rather than a read, so these recordings get materially more expensive to convert. Worth checking against the memory admission work from epic Epic: Zarr conversion memory robustness #1108 before turning it on for a 396-recording backlog. (An earlier draft of this issue said 439; that figure is wrong and ADR 0028 says why. It came from the zarr_errors column and counted the 43 calibration and cross-talk files themselves, which were being attempted as recordings before ADR 0027 excluded them. The correct figures are 396 affected, 365 recoverable, 31 declined.)

Note on scope

This is the only known MaxShield data in the catalog today, but the handling is general and any future MEGIN dataset acquired with Internal Active Shielding will need it.

Blocks the last two datasets from #1100 that are not already explained.

Activity

  1. neuromechanist commented on Aug 22, 2026

    @neuromechanist
    ContributorAuthor

    Implemented in #1127, merged to dev as e8e4572. Public docs in nemarOrg/docs#13.

    ADR 0028 is now built. Recordings carrying raw Internal Active Shielding are corrected with mne.preprocessing.maxwell_filter using the dataset's own fine-calibration and cross-talk pair, and served with the correction disclosed in both the store root attributes and the index entry. Where the pair does not resolve they are declined with maxshield_uncalibrated instead of an opaque file_read_error.

    Verified on the conversion node through the shipped functions: apply_sss peaks at 2.95 GiB on a 716 MiB recording, info["maxshield"] is clear on the output, and the resolver reproduces this ADR's coverage projection exactly (207 of 223 in on006720, 158 of 173 in on006012, every decline under sub-emptyroom).

    Two things review caught that are worth recording:

    • is_maxshield_fif swallowed every exception silently. Each one it can swallow is a FIF whose header would not read, and returning False routes a genuinely shielded recording back to the pre-ADR behaviour with no trace of why. It warns now.
    • A maxwell_filter failure other than a missing pair was uncoded and would have retried the filter forever at about a minute a time. A calibration pair that is present but does not fit the recording is as permanent as a missing one, so it gets the same terminal verdict; MemoryError still propagates as retryable.

    And one I caught before commit: the memory factor was initially 12.2x, measured from a pipeline production does not run. The filtered copy streams, so conversion adds nothing to the filter's peak. Re-measured across both datasets and the size range (5.0x, 5.0x, 4.4x, 4.2x at 160, 164, 438, 716 MiB) and set to 6, which clears every point.

    The 365-recording backfill has not run. At roughly 5 minutes per recording that is on the order of 30 hours of node time; it joins the deployment queue in #1122.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions