Skip to content

CTF recordings fail when the .hc file names coils Nasion/LPA/RPA #1125

Closed

Description

@neuromechanist

on006502 publishes no Zarr index: all 380 CTF .ds recordings fail with file_read_error, reason "Some of the mandatory HPI device-coordinate info was not there."

Found while resolving #1100. The data is intact and the head-position-indicator coordinates are present; only the labels naming them differ from what MNE accepts.

Root cause

MNE's _read_hc (mne/io/ctf/hc.py) classifies each coil point by substring-matching the descriptor line against a literal, case-sensitive table:

_kind_dict = {
    "nasion": CTF.CTFV_COIL_NAS,
    "left ear": CTF.CTFV_COIL_LPA,
    "right ear": CTF.CTFV_COIL_RPA,
    "spare": CTF.CTFV_COIL_SPARE,
}

This dataset's .hc files use different vocabulary:

measured Nasion coil position relative to head (cm):
measured LPA coil position relative to head (cm):
measured RPA coil position relative to head (cm):

Nasion fails on case; LPA and RPA are not in the table at all. Every point falls through to kind = -1, so _make_ctf_coord_trans_set (mne/io/ctf/trans.py) finds no LPA, RPA, or NAS and raises.

Verified by running MNE's own parser against a recording from this dataset:

points parsed: 6
  kind=-1  frame=CTF_DEVICE  valid=True     (x3)
  kind=-1  frame=CTF_HEAD    valid=True     (x3)

Both coordinate frames are present and every point is valid. The information MNE says is missing is right there under a different name.

Confirmed by patching only the labels

Copied one recording, rewrote only the three descriptor labels in its .hc (Nasion -> nasion, LPA -> left ear, RPA -> right ear), changed no coordinates and no other file, and reconverted through the same Recording.from_file -> to_zarr call the converter uses:

original           -> FAILED     file_read_error (mandatory HPI device-coordinate info)
labels-only patch  -> CONVERTED  groups=['events','meg_250hz','misc_600hz']  60.3 MiB

Scope

380 recordings in on006502, the whole dataset. No other Track 5 dataset is CTF, so nothing else is currently known to be affected, but any CTF dataset written with this naming will hit it.

Options

  1. Upstream in MNE: make _kind_dict matching case-insensitive and add lpa/rpa as aliases. Correct place for it, slowest path, and it would need an MNE release before it reaches us.
  2. In biosigIO: normalize the descriptors when opening a CTF directory, before handing off to MNE. Keeps NEMAR out of it and fixes every biosigIO consumer.
  3. In the converter: rewrite the .hc in the per-recording scratch copy before reading.

Option 3 is the fastest and the one I would be most careful about: it means rewriting acquisition metadata before reading it. It is contained (the scratch copy is discarded after conversion, the archived file is never touched) but it sets a precedent for the converter silently correcting source data, which is worth an explicit decision rather than a quiet patch. Options 1 or 2 are cleaner if the wait is acceptable.

Recommend deciding between these before implementing.

Activity

  1. neuromechanist commented on Aug 22, 2026

    @neuromechanist
    ContributorAuthor

    Decision: fix this in biosigIO, not upstream in MNE and not by normalizing in the converter.

    That settles the three options in the description. Reasoning, for the record:

    • Upstream MNE is the most correct place but the slowest: it needs a PR, a review cycle and a release before it reaches us, and 380 recordings wait on someone else's schedule.
    • Normalizing the .hc in the converter's scratch copy would work and is contained, but it means the converter silently rewrites acquisition metadata before reading it. That is a precedent worth not setting for a vendor naming variant.
    • biosigIO already owns the MEG import path (importers/meg.py calls read_raw_fif / the CTF reader directly), so normalizing the coil descriptors there fixes every biosigIO consumer at once and keeps NEMAR out of the business of editing source data.

    Scope for the biosigIO change: MNE's _kind_dict matches the literal lowercase strings nasion, left ear, right ear. Accept the Nasion / LPA / RPA spelling this dataset uses, case-insensitively, when opening a CTF .ds. The coordinates are already correct and in both frames; only the labels differ, which is why rewriting the three descriptor lines and nothing else made the byte-identical recording convert.

    Tracking this issue against that work rather than closing it: it stays open until a biosigIO release carries the fix, the pin here is bumped, and on006502 converts.

  2. neuromechanist commented on Aug 22, 2026

    @neuromechanist
    ContributorAuthor

    Filed upstream: mne-tools/mne-python#14190. There was no existing MNE issue for this; I searched their tracker for the error text, _read_hc, and the CTF/HPI phrasings before opening one.

    This does not change the plan here. We fix it in biosigIO regardless and do not wait on MNE.

    What the upstream report adds

    Two things I did not have when writing this issue.

    A public reproducer that needs no NEMAR access. on006502 is OpenNeuro ds006502 (CC0), and the failure is raised before the .meg4 is ever opened, so a single recording's metadata is enough:

    aws s3 cp --no-sign-request --recursive --exclude "*.meg4" \
      s3://openneuro.org/ds006502/sub-01/ses-meg1/meg/sub-01_ses-meg1_task-rest_run-1_meg.ds/ \
      sub-01_ses-meg1_task-rest_run-1_meg.ds/
    

    3.2 MB, then mne.io.read_raw_ctf(...) raises. Worth keeping for our own regression test, since it means we can cover this without a MEG-sized fixture.

    CTF's own reader is more permissive than MNE, which is the argument for fixing it upstream rather than normalising on our side. readHc inside readCTFds.m (bundled in FieldTrip at external/ctf/readCTFds.m) does not match coil names against a vocabulary at all; it reads the descriptor positionally and stores whatever name is there. MEGCoilLabel = strvcat('nasion','left ear','right ear') is declared at line 528 and never used. The same reader carries an explicit workaround for a stadard typo emitted by some Acq releases, so descriptor variation is clearly expected on the CTF side.

    Provenance, in case it recurs: NIMH MEG Core Facility, DSQ-2010, Acq 5.2.1-linux-20060809, each dataset later Rewritten by writeCTFds per its .hist. writeCTFds's writeHc writes ds.hc.names verbatim, so the rewrite preserves the spelling rather than introducing it. Which upstream step produced Nasion/LPA/RPA is still unknown. Sampled sub-01, sub-03, sub-15, sub-20: all the same.

    Correction to my root-cause note above

    I described this as a case-sensitivity problem. Case is only half of it. Nasion misses nasion on case, but LPA and RPA are a different vocabulary entirely and would miss left ear / right ear at any casing. A biosigIO fix that only lowercases will not work.

    The mapping that does work, verified against both the variant file and a canonically spelled one (ds000247), is to lowercase the descriptor and add the aliases:

    "lpa": CTF.CTFV_COIL_LPA,
    "rpa": CTF.CTFV_COIL_RPA,

    With that, the unmodified .hc parses to [NAS, LPA, RPA, NAS, LPA, RPA] across both coordinate frames and reading proceeds to the data stage; a canonically spelled .hc parses exactly as before.

    Where the fix goes

    Still biosigIO, not here, and not a .hc rewrite in the converter. If MNE takes the upstream patch, biosigIO's workaround becomes a version-gated no-op rather than something we have to unpick.

  3. neuromechanist commented on Aug 22, 2026

    @neuromechanist
    ContributorAuthor

    Upstream PR is open: mne-tools/mne-python#14191, fixing mne-tools/mne-python#14190.

    Two changes to mne/io/ctf/hc.py: match the .hc descriptor line case-insensitively, and add lpa/rpa as aliases for left ear/right ear. The test copies MNE's CTF testing dataset, rewrites only the three descriptor words, and asserts the resulting dev_head_t and every digitization point match a read of the untouched original; it is parametrized over Nasion/LPA/RPA and a mixed-case spelling. Both parametrizations fail on main with exactly the error we see on on006502 and pass with the change, and the rest of test_ctf.py is unaffected (same 5 pre-existing failures before and after, all from testing-data files I did not download locally).

    We are not waiting on this. The biosigIO fix still goes in; if MNE merges, it becomes a version-gated no-op.

  4. neuromechanist commented on Sep 3, 2026

    @neuromechanist
    ContributorAuthor

    Closing. Fixed upstream in biosigio 1.2.7 (pinned in scripts/zarr/requirements.txt); zarr.nemar.org/on006502/zarr/index.json shows no coil-naming failures. on006502's remaining failures are memory-class and are being re-converted by the engine-3 wave under the v0.9.14 memory fixes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions