You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Nothing enforces the zarr gate stays a superset of the converter's PRIMARY_EXTS #1103
ZARR_DATA_EXTENSIONS in backend/src/routes/webhooks/github.ts must stay a superset of PRIMARY_EXTS in generate_zarr.py (nemarDatasets/.github). Nothing enforces that but a doc
comment, and it has already drifted once: .con, .sqd, and .kdf were missing, so a push
changing a KIT/Yokogawa/RICOH magnetoencephalography recording never re-dispatched conversion and the
serving copy went stale silently. Dataset on007763 has 35 affected .con recordings. Fixed in #1101, but the next divergence will be just as invisible.
The two constants live in different repositories and different languages, which is exactly why a
comment is not enough.
Why this is worth mechanizing
The failure is silent in both directions:
Gate too NARROW (what happened): pushes to a real recording format never trigger conversion.
Nothing errors; the serving copy just quietly stops tracking the dataset.
Gate too WIDE: pushes dispatch jobs the converter then ignores. Wasted runs, no signal.
Neither shows up in tests, logs, or the failure index, because the whole symptom is an event that
never happens.
Suggested approaches
A periodic continuous-integration check in this repo that fetches generate_zarr.py from nemarDatasets/.github, extracts PRIMARY_EXTS (it is a plain tuple literal, so a small regex or
an ast parse is enough), and asserts the gate is a superset. Run it on a schedule rather than per
pull request, since it depends on another repository's state and should not make unrelated pull
requests red. This is the option I would pick.
Publish the format list from one side as a small shared data file (for example a committed JSON
list) that both the gate and the converter read, removing the duplication instead of checking it.
Cleaner in principle, but it couples deploys across two repositories.
Leave it as a comment and rely on review. This is the status quo, and the status quo already failed.
Worth noting the same class of cross-repo coupling exists elsewhere in the Zarr pipeline, so whatever
mechanism is chosen may be reusable: run-enrichment.yml hardcodes a production callback URL and run-generate-archive.yml hardcodes the production bucket, both of which break against staging.
Raised by the comment/Architecture Decision Record review on #1101. Part of epic #1095.
Rescope: the check as originally proposed would not have caught either real violation
This issue's Approach 1, diffing the gate's extension set against PRIMARY_EXTS extracted from generate_zarr.py, is worth having and would have caught the original KIT .con/.sqd/.kdf gap that
prompted it. But both violations found since are invisible to it, because neither is an
extension-set problem:
Worth noting how both were found: the KIT gap surfaced from an unrelated catalog sweep, and the BTi one
because an implementing agent flagged it as an unverified risk in a report. Neither was caught by
review of the change that introduced it. In the _meg case the reviewer who suggested the narrowing
could not have caught it, because the converter-side BTi rule did not exist yet on main at the time
and lived only in an unmerged sibling-repo pull request. That is the structural problem: the ground
truth for this invariant regularly lives in another repository, sometimes unmerged.
Better scope: a shared behavioural corpus, not a constant diff
Check the two implementations against the same inputs rather than comparing their source constants.
Commit a list of representative paths covering every mechanism, not just extensions:
_events.tsv sidecars, top-level and under an excluded tree
Run every path through both sides: isZarrTriggerPath in TypeScript, and a small Python classifier
composed from the converter's own is_primary, dir_recording_of, bti_recordings, and is_excluded_from_discovery. Assert the gate returns true for every path the converter would treat as
belonging to a buildable recording. The reverse direction is only a wasted workflow run, so report it
rather than failing on it.
Run it on a schedule rather than per pull request, since it depends on another repository's state and
should not redden unrelated work. That part of the original scope was right.
This single mechanism would have caught all three known gaps: the KIT extensions, the _meg
narrowing, and the config/hs_file omission.
One deliberate exception to encode
The gate cannot evaluate the converter's sibling-config rule, because it sees one changed path at a
time rather than the tree (verified: its only caller invokes it inside for (const p of touched)). So a config/hs_file-only change inside a BTi directory not named ..._meg/ will not re-dispatch. That is
accepted and documented in the code. The corpus should record it as a known, allowed divergence rather
than as a failure, so the check does not go permanently red on something we chose.
Closed by #1114 (epic #1108 Phase 1), delivered to dev via #1119.
This was unenforceable while the gate and the converter lived in different repos. With scripts/zarr/generate_zarr.py in this repo it is a same-repo unit test: test/zarr-gate-superset.unit.test.ts parses PRIMARY_EXTS, STREAM_EXTS, and DIR_RECORDING_EXTS out of the converter and asserts ZARR_DATA_EXTENSIONS covers them, so the two can no longer drift silently.
Activity
Widen the BTi dispatch gate so it stays a superset of the converter (#…
neuromechanist commentedon Aug 21, 2026
Rescope: the check as originally proposed would not have caught either real violation
This issue's Approach 1, diffing the gate's extension set against
PRIMARY_EXTSextracted fromgenerate_zarr.py, is worth having and would have caught the original KIT.con/.sqd/.kdfgap thatprompted it. But both violations found since are invisible to it, because neither is an
extension-set problem:
_megnarrowing (fixed in Widen the BTi dispatch gate so it stays a superset of the converter #1105).isBtiMemberrequired the parent segment to end in_megwhile the converter'sbti_recordingsimposes no naming constraint. A directory/contentmembership rule, not an extension.
config/hs_filegap (also Widen the BTi dispatch gate so it stays a superset of the converter #1105). The converter rebuilds a BTi recording on any touchedmember file, including
configalone, and has tests asserting exactly that. The gate matched onlyc,rf*. Again a membership rule.Worth noting how both were found: the KIT gap surfaced from an unrelated catalog sweep, and the BTi one
because an implementing agent flagged it as an unverified risk in a report. Neither was caught by
review of the change that introduced it. In the
_megcase the reviewer who suggested the narrowingcould not have caught it, because the converter-side BTi rule did not exist yet on
mainat the timeand lived only in an unmerged sibling-repo pull request. That is the structural problem: the ground
truth for this invariant regularly lives in another repository, sometimes unmerged.
Better scope: a shared behavioural corpus, not a constant diff
Check the two implementations against the same inputs rather than comparing their source constants.
Commit a list of representative paths covering every mechanism, not just extensions:
.set,.fdt,.edf,.bdf,.vhdr,.vmrk,.eeg,.fif,.con,.sqd,.kdf).dsmembers, MEF3.mefdmembersc,rfDC, a filteredc,rfDC,fn50,o,config,hs_file, and an unrelated file in the same directory.datalad/config, the false-positive that constrains the whole BTi rulederivatives/,sourcedata/,code/, plus segment-boundary near misses (mycode/,derivatives_old/,sourcedatafoo/)_acq-crosstalk_meg.fif,_acq-calibration_meg.dat)_events.tsvsidecars, top-level and under an excluded treeRun every path through both sides:
isZarrTriggerPathin TypeScript, and a small Python classifiercomposed from the converter's own
is_primary,dir_recording_of,bti_recordings, andis_excluded_from_discovery. Assert the gate returns true for every path the converter would treat asbelonging to a buildable recording. The reverse direction is only a wasted workflow run, so report it
rather than failing on it.
Run it on a schedule rather than per pull request, since it depends on another repository's state and
should not redden unrelated work. That part of the original scope was right.
This single mechanism would have caught all three known gaps: the KIT extensions, the
_megnarrowing, and the
config/hs_fileomission.One deliberate exception to encode
The gate cannot evaluate the converter's sibling-
configrule, because it sees one changed path at atime rather than the tree (verified: its only caller invokes it inside
for (const p of touched)). So aconfig/hs_file-only change inside a BTi directory not named..._meg/will not re-dispatch. That isaccepted and documented in the code. The corpus should record it as a known, allowed divergence rather
than as a failure, so the check does not go permanently red on something we chose.
Credit for this scoping to the review of #1105.
Phase 1: Repatriate the zarr conversion engine to nemar-cli (#1114)
neuromechanist commentedon Aug 22, 2026
Closed by #1114 (epic #1108 Phase 1), delivered to
devvia #1119.This was unenforceable while the gate and the converter lived in different repos. With
scripts/zarr/generate_zarr.pyin this repo it is a same-repo unit test:test/zarr-gate-superset.unit.test.tsparsesPRIMARY_EXTS,STREAM_EXTS, andDIR_RECORDING_EXTSout of the converter and assertsZARR_DATA_EXTENSIONScovers them, so the two can no longer drift silently.