Repository navigation
SSS-filter MaxShield MEG, or decline it specifically - #1127
Merged
Merged
Conversation
Implements ADR 0028, accepted 2026-08-20 and never built. on006012 and on006720 publish no Zarr index because MNE refuses raw Internal Active Shielding data, and all 396 recordings fail as file_read_error. Recordings carrying it are now corrected with mne.preprocessing. maxwell_filter using the dataset's own fine-calibration and cross-talk files, and served. Where that pair does not resolve the recording is declined with a new deterministic code, maxshield_uncalibrated, instead of an opaque read error: ADR 0028 rejects uncalibrated filtering because it is a weaker correction a consumer could not distinguish from a good one. The calibration pair needs its own resolver. Every existing sidecar resolver requires the candidate's entities to be a subset of the recording's, and sub-01_acq-calibration_meg.dat carries acq=calibration while the recording has no acq at all, so that rule rejects the very file it should find. acq names these sidecars rather than scoping them, so it is exempt here. Run over the real repository trees the resolver reproduces the ADR's projection exactly: 207 of 223 and 158 of 173, every decline under sub-emptyroom. Filtering is disclosed in the store root attributes and the index entry. ADR 0028 requires this: the result is a processed derivative unlike every other store, and a model training across datasets would otherwise silently mix filtered and unfiltered MEG. proc_history does not survive into the store, so it is written explicitly. Admission cannot detect MaxShield: it projects from annex pointers without downloading, and the BIDS sidecar carries no marker. It uses the one signal available from head_files, a resolvable calibration pair, and the worker does the real detection. MAXSHIELD_MEM_FACTOR is measured, not reasoned: 2.95 GiB peak on a 716 MiB recording, 4.2x, end to end through these functions on the conversion node. Load-bearing, since apply_worker_mem_limit sizes RLIMIT_DATA from the projection before the worker starts. An earlier draft of this work used a 12.2x factor taken from a pipeline production does not run. The filtered copy streams, so conversion adds nothing to the filter's peak.
📦 Package PreviewYour package is ready to test! bun install -g nemar-cli@PR1127Version: 🤖 Updated on each push. |
Four findings from review of #1127. is_maxshield_fif swallowed every exception with no diagnostic. Every one it can swallow is a .fif whose header would not read, and returning False routes a genuinely shielded recording back down the pre-ADR-0028 path, where MNE refuses it and the operator sees an opaque file_read_error with no hint the correction was skipped. Now warns and says what to expect. maxwell_filter failures other than a missing pair were uncoded, so they retried the filter forever at about a minute a time. A calibration pair that is present but does not fit the recording is as permanent as a missing one, so it now gets the same terminal verdict. MemoryError still propagates, since that one really is transient. Local mode resolved the pair without checking it exists, while the remote branch declines cleanly when a tracked file cannot be materialised. A working tree can hold an annex pointer whose content was never fetched. The memory factor rested on one recording, which is exactly how the BrainVision factor was wrong by 2x. Measured across both datasets and the size range: 5.0x, 5.0x, 4.4x, 4.2x for 160, 164, 438 and 716 MiB. The ratio falls with size, so the worst case sits where the absolute number is trivial. 6 clears every point and clears the largest by 43%. Pinned by a test, along with the reserve at the end of the admission chain rather than the projection alone, since that is where the slack suppression lives.
This was referenced Aug 22, 2026
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements ADR 0028, accepted 2026-08-20 and never built.
on006012andon006720publish no Zarr index at all: MNE refuses raw Internal Active Shielding data, so every one of their 396 MEG recordings fails as an opaquefile_read_error.Recordings carrying it are now corrected with
mne.preprocessing.maxwell_filterusing the dataset's own fine-calibration and cross-talk files, and served. Where that pair does not resolve the recording is declined with a new deterministic code rather than a generic read error.Expected outcome, matching the ADR: 365 of 396 recordings become viewable; the 31 that stay declined are all
sub-emptyroomreferences.Verified on the conversion node, through these functions
Not a stand-in pipeline — the shipped
is_maxshield_fif,apply_sssandconvert_recording, against a real 716 MiBon006720recording with that dataset's own sidecars:is_maxshield_fif(header only)apply_sssconvert_recording(streams)info["maxshield"]is clear on the output — MNE's own guardrail, the check that refused the file, is satisfied by the result. Store root carries thesssattribute; groups aremeg_250hz,misc_1000hz.Three things worth review attention
The calibration pair needed its own resolver. Every existing sidecar resolver requires the candidate's entities to be a subset of the recording's.
sub-01_acq-calibration_meg.datcarriesacq=calibrationwhilesub-01_task-rest_meg.fifhas noacqat all, so that rule rejects the very file it should find.acqnames these sidecars rather than scoping them, so it is exempt here. Run over the real repository trees the resolver reproduces the ADR's projection exactly: 207 of 223 and 158 of 173, every decline undersub-emptyroom.Admission cannot detect MaxShield. It projects from git-annex pointers without downloading, and the BIDS sidecar carries no marker (
SoftwareFiltersthere is CTF-styleSpatialCompensation, unrelated). It uses the one signal available fromhead_files— a resolvable calibration pair — and the worker does the real detection. Over-reserving costs concurrency; under-reserving costs the node.MAXSHIELD_MEM_FACTORis load-bearing, not tuning.apply_worker_mem_limitsizesRLIMIT_DATAfrom the projection before the worker starts, so an under-projection kills the filter mid-run rather than merely over-scheduling. The pre-existing 4 GiB streaming floor clears this dataset's largest recording by under 5%, which is coincidence: that floor was sized for conversion, and filtering is a separate phase peaking before it.I first measured a 12.2x factor and nearly shipped it. That number came from forcing the in-memory path; the filtered copy actually streams, so conversion adds nothing to the filter's peak and the real figure is 4.2x. Corrected before commit.
Test plan
acqexemption pinned as a regression, calibration files staying excluded from discovery while remaining conversion inputs, the decline being deterministic and outsideRETRYABLE_CODES, and the projection covering the filter phase without summing the two phasesacqexemption fails 6 of 10 resolution testsWORK_DIR, with no S3 writes and the in-flight 1.5 TB drain unaffectedNot in this PR
Converting the backlog. At roughly 5 minutes per recording measured, 365 recordings is on the order of 30 hours of node time — a queue operation to schedule once this ships, tracked with the other deployment steps in #1122.
Public docs updated separately in nemarOrg/docs (new failure code plus the derivative disclosure ADR 0028 requires consumers to be able to see).
Closes #1126. Part of epic #1095's follow-up.