Skip to content

Nothing enforces the zarr gate stays a superset of the converter's PRIMARY_EXTS #1103

Closed

Description

@neuromechanist

ZARR_DATA_EXTENSIONS in backend/src/routes/webhooks/github.ts must stay a superset of
PRIMARY_EXTS in generate_zarr.py (nemarDatasets/.github). Nothing enforces that but a doc
comment, and it has already drifted once: .con, .sqd, and .kdf were missing, so a push
changing a KIT/Yokogawa/RICOH magnetoencephalography recording never re-dispatched conversion and the
serving copy went stale silently. Dataset on007763 has 35 affected .con recordings. Fixed in
#1101, but the next divergence will be just as invisible.

The two constants live in different repositories and different languages, which is exactly why a
comment is not enough.

Why this is worth mechanizing

The failure is silent in both directions:

  • Gate too NARROW (what happened): pushes to a real recording format never trigger conversion.
    Nothing errors; the serving copy just quietly stops tracking the dataset.
  • Gate too WIDE: pushes dispatch jobs the converter then ignores. Wasted runs, no signal.

Neither shows up in tests, logs, or the failure index, because the whole symptom is an event that
never happens.

Suggested approaches

  1. A periodic continuous-integration check in this repo that fetches generate_zarr.py from
    nemarDatasets/.github, extracts PRIMARY_EXTS (it is a plain tuple literal, so a small regex or
    an ast parse is enough), and asserts the gate is a superset. Run it on a schedule rather than per
    pull request, since it depends on another repository's state and should not make unrelated pull
    requests red. This is the option I would pick.
  2. Publish the format list from one side as a small shared data file (for example a committed JSON
    list) that both the gate and the converter read, removing the duplication instead of checking it.
    Cleaner in principle, but it couples deploys across two repositories.
  3. Leave it as a comment and rely on review. This is the status quo, and the status quo already failed.

Worth noting the same class of cross-repo coupling exists elsewhere in the Zarr pipeline, so whatever
mechanism is chosen may be reusable: run-enrichment.yml hardcodes a production callback URL and
run-generate-archive.yml hardcodes the production bucket, both of which break against staging.

Raised by the comment/Architecture Decision Record review on #1101. Part of epic #1095.

Activity

  1. neuromechanist commented on Aug 21, 2026

    @neuromechanist
    ContributorAuthor

    Rescope: the check as originally proposed would not have caught either real violation

    This issue's Approach 1, diffing the gate's extension set against PRIMARY_EXTS extracted from
    generate_zarr.py, is worth having and would have caught the original KIT .con/.sqd/.kdf gap that
    prompted it. But both violations found since are invisible to it, because neither is an
    extension-set problem:

    1. The _meg narrowing (fixed in Widen the BTi dispatch gate so it stays a superset of the converter #1105). isBtiMember required the parent segment to end in
      _meg while the converter's bti_recordings imposes no naming constraint. A directory/content
      membership rule, not an extension.
    2. The config/hs_file gap (also Widen the BTi dispatch gate so it stays a superset of the converter #1105). The converter rebuilds a BTi recording on any touched
      member file, including config alone, and has tests asserting exactly that. The gate matched only
      c,rf*. Again a membership rule.

    Worth noting how both were found: the KIT gap surfaced from an unrelated catalog sweep, and the BTi one
    because an implementing agent flagged it as an unverified risk in a report. Neither was caught by
    review of the change that introduced it. In the _meg case the reviewer who suggested the narrowing
    could not have caught it, because the converter-side BTi rule did not exist yet on main at the time
    and lived only in an unmerged sibling-repo pull request. That is the structural problem: the ground
    truth for this invariant regularly lives in another repository, sometimes unmerged.

    Better scope: a shared behavioural corpus, not a constant diff

    Check the two implementations against the same inputs rather than comparing their source constants.

    Commit a list of representative paths covering every mechanism, not just extensions:

    • extension primaries and companions (.set, .fdt, .edf, .bdf, .vhdr, .vmrk, .eeg, .fif, .con, .sqd, .kdf)
    • CTF .ds members, MEF3 .mefd members
    • 4D/BTi members: c,rfDC, a filtered c,rfDC,fn50,o, config, hs_file, and an unrelated file in the same directory
    • .datalad/config, the false-positive that constrains the whole BTi rule
    • excluded-tree cases under derivatives/, sourcedata/, code/, plus segment-boundary near misses (mycode/, derivatives_old/, sourcedatafoo/)
    • BIDS calibration files (_acq-crosstalk_meg.fif, _acq-calibration_meg.dat)
    • _events.tsv sidecars, top-level and under an excluded tree

    Run every path through both sides: isZarrTriggerPath in TypeScript, and a small Python classifier
    composed from the converter's own is_primary, dir_recording_of, bti_recordings, and
    is_excluded_from_discovery. Assert the gate returns true for every path the converter would treat as
    belonging to a buildable recording. The reverse direction is only a wasted workflow run, so report it
    rather than failing on it.

    Run it on a schedule rather than per pull request, since it depends on another repository's state and
    should not redden unrelated work. That part of the original scope was right.

    This single mechanism would have caught all three known gaps: the KIT extensions, the _meg
    narrowing, and the config/hs_file omission.

    One deliberate exception to encode

    The gate cannot evaluate the converter's sibling-config rule, because it sees one changed path at a
    time rather than the tree (verified: its only caller invokes it inside for (const p of touched)). So a
    config/hs_file-only change inside a BTi directory not named ..._meg/ will not re-dispatch. That is
    accepted and documented in the code. The corpus should record it as a known, allowed divergence rather
    than as a failure, so the check does not go permanently red on something we chose.

    Credit for this scoping to the review of #1105.

  2. neuromechanist commented on Aug 22, 2026

    @neuromechanist
    ContributorAuthor

    Closed by #1114 (epic #1108 Phase 1), delivered to dev via #1119.

    This was unenforceable while the gate and the converter lived in different repos. With scripts/zarr/generate_zarr.py in this repo it is a same-repo unit test: test/zarr-gate-superset.unit.test.ts parses PRIMARY_EXTS, STREAM_EXTS, and DIR_RECORDING_EXTS out of the converter and asserts ZARR_DATA_EXTENSIONS covers them, so the two can no longer drift silently.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions