Skip to content

SSS-filter MaxShield MEG, or decline it specifically - #1127

Merged
neuromechanist merged 2 commits into
devfrom
feature/issue-1126-maxshield-sss
Aug 22, 2026
Merged

neuromechanist merged 2 commits into
devfrom
feature/issue-1126-maxshield-sss

Conversation

@neuromechanist

Copy link
Copy Markdown
Contributor

Summary

Implements ADR 0028, accepted 2026-08-20 and never built. on006012 and on006720 publish no Zarr index at all: MNE refuses raw Internal Active Shielding data, so every one of their 396 MEG recordings fails as an opaque file_read_error.

Recordings carrying it are now corrected with mne.preprocessing.maxwell_filter using the dataset's own fine-calibration and cross-talk files, and served. Where that pair does not resolve the recording is declined with a new deterministic code rather than a generic read error.

Expected outcome, matching the ADR: 365 of 396 recordings become viewable; the 31 that stay declined are all sub-emptyroom references.

Verified on the conversion node, through these functions

Not a stand-in pipeline — the shipped is_maxshield_fif, apply_sss and convert_recording, against a real 716 MiB on006720 recording with that dataset's own sidecars:

Stage Time Peak
is_maxshield_fif (header only) 1.5 s 0.14 GiB
apply_sss 57.5 s 2.95 GiB
convert_recording (streams) 260 s no increase

info["maxshield"] is clear on the output — MNE's own guardrail, the check that refused the file, is satisfied by the result. Store root carries the sss attribute; groups are meg_250hz, misc_1000hz.

Three things worth review attention

The calibration pair needed its own resolver. Every existing sidecar resolver requires the candidate's entities to be a subset of the recording's. sub-01_acq-calibration_meg.dat carries acq=calibration while sub-01_task-rest_meg.fif has no acq at all, so that rule rejects the very file it should find. acq names these sidecars rather than scoping them, so it is exempt here. Run over the real repository trees the resolver reproduces the ADR's projection exactly: 207 of 223 and 158 of 173, every decline under sub-emptyroom.

Admission cannot detect MaxShield. It projects from git-annex pointers without downloading, and the BIDS sidecar carries no marker (SoftwareFilters there is CTF-style SpatialCompensation, unrelated). It uses the one signal available from head_files — a resolvable calibration pair — and the worker does the real detection. Over-reserving costs concurrency; under-reserving costs the node.

MAXSHIELD_MEM_FACTOR is load-bearing, not tuning. apply_worker_mem_limit sizes RLIMIT_DATA from the projection before the worker starts, so an under-projection kills the filter mid-run rather than merely over-scheduling. The pre-existing 4 GiB streaming floor clears this dataset's largest recording by under 5%, which is coincidence: that floor was sized for conversion, and filtering is a separate phase peaking before it.

I first measured a 12.2x factor and nearly shipped it. That number came from forcing the in-memory path; the filtered copy actually streams, so conversion adds nothing to the filter's peak and the real figure is 4.2x. Corrected before commit.

Test plan

  • 14 new tests: pair resolution (both, half, neither, inherited from an ancestor, bare form at dataset root, nearest-wins, another subject, a sibling session), the acq exemption pinned as a regression, calibration files staying excluded from discovery while remaining conversion inputs, the decline being deterministic and outside RETRYABLE_CODES, and the projection covering the filter phase without summing the two phases
  • Removing the acq exemption fails 6 of 10 resolution tests
  • Full suite 292 green; ADR index test green; shellcheck clean
  • End-to-end on the node, outside WORK_DIR, with no S3 writes and the in-flight 1.5 TB drain unaffected

Not in this PR

Converting the backlog. At roughly 5 minutes per recording measured, 365 recordings is on the order of 30 hours of node time — a queue operation to schedule once this ships, tracked with the other deployment steps in #1122.

Public docs updated separately in nemarOrg/docs (new failure code plus the derivative disclosure ADR 0028 requires consumers to be able to see).

Closes #1126. Part of epic #1095's follow-up.

Implements ADR 0028, accepted 2026-08-20 and never built. on006012 and
on006720 publish no Zarr index because MNE refuses raw Internal Active
Shielding data, and all 396 recordings fail as file_read_error.

Recordings carrying it are now corrected with mne.preprocessing.
maxwell_filter using the dataset's own fine-calibration and cross-talk
files, and served. Where that pair does not resolve the recording is
declined with a new deterministic code, maxshield_uncalibrated, instead
of an opaque read error: ADR 0028 rejects uncalibrated filtering because
it is a weaker correction a consumer could not distinguish from a good
one.

The calibration pair needs its own resolver. Every existing sidecar
resolver requires the candidate's entities to be a subset of the
recording's, and sub-01_acq-calibration_meg.dat carries acq=calibration
while the recording has no acq at all, so that rule rejects the very file
it should find. acq names these sidecars rather than scoping them, so it
is exempt here. Run over the real repository trees the resolver
reproduces the ADR's projection exactly: 207 of 223 and 158 of 173, every
decline under sub-emptyroom.

Filtering is disclosed in the store root attributes and the index entry.
ADR 0028 requires this: the result is a processed derivative unlike every
other store, and a model training across datasets would otherwise
silently mix filtered and unfiltered MEG. proc_history does not survive
into the store, so it is written explicitly.

Admission cannot detect MaxShield: it projects from annex pointers
without downloading, and the BIDS sidecar carries no marker. It uses the
one signal available from head_files, a resolvable calibration pair, and
the worker does the real detection.

MAXSHIELD_MEM_FACTOR is measured, not reasoned: 2.95 GiB peak on a
716 MiB recording, 4.2x, end to end through these functions on the
conversion node. Load-bearing, since apply_worker_mem_limit sizes
RLIMIT_DATA from the projection before the worker starts.

An earlier draft of this work used a 12.2x factor taken from a pipeline
production does not run. The filtered copy streams, so conversion adds
nothing to the filter's peak.
@github-actions

github-actions Bot commented Aug 22, 2026 •

Copy link
Copy Markdown

📦 Package Preview

Your package is ready to test!

bun install -g nemar-cli@PR1127

Version: 0.9.9-PR1127.0453203
Commit: 0453203


🤖 Updated on each push.

Four findings from review of #1127.

is_maxshield_fif swallowed every exception with no diagnostic. Every one
it can swallow is a .fif whose header would not read, and returning False
routes a genuinely shielded recording back down the pre-ADR-0028 path,
where MNE refuses it and the operator sees an opaque file_read_error with
no hint the correction was skipped. Now warns and says what to expect.

maxwell_filter failures other than a missing pair were uncoded, so they
retried the filter forever at about a minute a time. A calibration pair
that is present but does not fit the recording is as permanent as a
missing one, so it now gets the same terminal verdict. MemoryError still
propagates, since that one really is transient.

Local mode resolved the pair without checking it exists, while the remote
branch declines cleanly when a tracked file cannot be materialised. A
working tree can hold an annex pointer whose content was never fetched.

The memory factor rested on one recording, which is exactly how the
BrainVision factor was wrong by 2x. Measured across both datasets and the
size range: 5.0x, 5.0x, 4.4x, 4.2x for 160, 164, 438 and 716 MiB. The
ratio falls with size, so the worst case sits where the absolute number
is trivial. 6 clears every point and clears the largest by 43%. Pinned by
a test, along with the reserve at the end of the admission chain rather
than the projection alone, since that is where the slack suppression
lives.
@neuromechanist
neuromechanist merged commit e8e4572 into dev Aug 22, 2026
12 checks passed
@neuromechanist
neuromechanist deleted the feature/issue-1126-maxshield-sss branch August 22, 2026 18:46

This branch was previously deployed

1 inactive deployment
npm — 04532033 Deployed Aug 22, 2026 by neuromechanist via publish #2046
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant