You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Events sidecar edits do not rebuild directory-format recordings (CTF, MEF3, BTi) #1106
An edit to a _events.tsv sidecar does not rebuild its recording in incremental mode when that
recording is a directory format. This affects all three: CTF .ds, MEF3 .mefd, and 4D/BTi.
Verified by execution against compute_worklist in scripts/zarr/generate_zarr.py
(nemarDatasets/.github):
compute_worklist(head, [("M", ".../sub-01_task-x_events.tsv")], full=False)
# -> convert: [] remove: [] for CTF .ds, MEF3 .mefd, and BTi alike
Cause
compute_worklist builds its by_dir map only from heads, the file-extension primaries. It is never
populated from dirrec_dirs or bti_dirs. So when affected_primaries routes an events sidecar to its
sibling recording, a directory-based recording is never found as a match and nothing is queued.
Why it matters
The converter embeds events into each store, so the store keeps serving the previous events after the
sidecar is corrected. There is no error and no failure entry; the index still reports the store as
present and current. A curator fixing event codes or onsets for a CTF, MEF3, or BTi recording would see
the commit land and reasonably assume the serving copy followed.
It is not permanent: a --full or --clean run rebuilds everything, and the hourly production cron
invokes --clean unconditionally, so in practice the stale window closes on the next version bump
rather than persisting indefinitely. That materially lowers the severity but does not make the
incremental path correct.
Scope note
This is pre-existing, not a regression. It has always been true for CTF .ds on main, and nemarDatasets/.github#100 simply extended the same gap to the two new directory formats by adding them
to a mechanism that already had it. Confirmed identical behaviour on unmodified main for CTF. The
author of #100 was right not to widen scope, and the reviewer flagged it as informational rather than
blocking.
Filing it because it now affects three formats instead of one, and because "the cron papers over it"
is a reason to deprioritize rather than a reason to leave it undocumented.
Fix sketch
Populate the sidecar-routing map from the directory-recording sets as well as from the extension
primaries, so an _events.tsv resolves to a directory recording sitting in the same directory. Any fix
wants a test per format, since the three kinds are discovered by different mechanisms (extension-keyed
for .ds/.mefd, content-keyed for BTi) and a fix that works for one may miss another.
dirrec_dirs and bti_dirs are never added, so when affected_primaries resolves an _events.tsv against primaries_by_dir, a directory-format recording is not among the candidates and nothing is queued. Confirmed for CTF .ds, MEF3 .mefd, and 4D/BTi alike.
Staying open. Note that epic #1108 Phase 1 (#1109) moves generate_zarr.py into nemar-cli, so the fix becomes a single-repo change with CI once that lands.
compute_worklist keyed by_dir on file heads only, so affected_primaries received an empty bucket for a CTF .ds, MEF3 .mefd, or 4D/BTi recording and an _events.tsv edit beside one queued nothing. Reproduced across all three formats before fixing (each returned []), and all three now rebuild.
The --clean cron papering over it on the next full rebuild is exactly why it stayed invisible: the incremental path was wrong without ever being visibly wrong.
Eleven tests, seven of which fail without the one-line change; the rest are negative guards against over-matching, plus a session/run/acq case since CTF and MEF3 are in practice the multi-session, multi-run formats.
Activity
neuromechanist commentedon Aug 22, 2026
Re-verified on current
.githubmainduring the epic #1095 closeout audit: still present, not fixed by #100 or #98.compute_worklistbuilds itsby_dirmap fromheadsonly:dirrec_dirsandbti_dirsare never added, so whenaffected_primariesresolves an_events.tsvagainstprimaries_by_dir, a directory-format recording is not among the candidates and nothing is queued. Confirmed for CTF.ds, MEF3.mefd, and 4D/BTi alike.Staying open. Note that epic #1108 Phase 1 (#1109) moves
generate_zarr.pyinto nemar-cli, so the fix becomes a single-repo change with CI once that lands.Rebuild directory recordings on sidecar edits (#1123)
neuromechanist commentedon Aug 22, 2026
Fixed in #1123, merged to
devas 1580f72.compute_worklistkeyedby_diron file heads only, soaffected_primariesreceived an empty bucket for a CTF.ds, MEF3.mefd, or 4D/BTi recording and an_events.tsvedit beside one queued nothing. Reproduced across all three formats before fixing (each returned[]), and all three now rebuild.The
--cleancron papering over it on the next full rebuild is exactly why it stayed invisible: the incremental path was wrong without ever being visibly wrong.Eleven tests, seven of which fail without the one-line change; the rest are negative guards against over-matching, plus a session/run/acq case since CTF and MEF3 are in practice the multi-session, multi-run formats.