Skip to content

Events sidecar edits do not rebuild directory-format recordings (CTF, MEF3, BTi) #1106

Closed

Description

@neuromechanist

An edit to a _events.tsv sidecar does not rebuild its recording in incremental mode when that
recording is a directory format. This affects all three: CTF .ds, MEF3 .mefd, and 4D/BTi.

Verified by execution against compute_worklist in scripts/zarr/generate_zarr.py
(nemarDatasets/.github):

compute_worklist(head, [("M", ".../sub-01_task-x_events.tsv")], full=False)
# -> convert: []   remove: []      for CTF .ds, MEF3 .mefd, and BTi alike

Cause

compute_worklist builds its by_dir map only from heads, the file-extension primaries. It is never
populated from dirrec_dirs or bti_dirs. So when affected_primaries routes an events sidecar to its
sibling recording, a directory-based recording is never found as a match and nothing is queued.

Why it matters

The converter embeds events into each store, so the store keeps serving the previous events after the
sidecar is corrected. There is no error and no failure entry; the index still reports the store as
present and current. A curator fixing event codes or onsets for a CTF, MEF3, or BTi recording would see
the commit land and reasonably assume the serving copy followed.

It is not permanent: a --full or --clean run rebuilds everything, and the hourly production cron
invokes --clean unconditionally, so in practice the stale window closes on the next version bump
rather than persisting indefinitely. That materially lowers the severity but does not make the
incremental path correct.

Scope note

This is pre-existing, not a regression. It has always been true for CTF .ds on main, and
nemarDatasets/.github#100 simply extended the same gap to the two new directory formats by adding them
to a mechanism that already had it. Confirmed identical behaviour on unmodified main for CTF. The
author of #100 was right not to widen scope, and the reviewer flagged it as informational rather than
blocking.

Filing it because it now affects three formats instead of one, and because "the cron papers over it"
is a reason to deprioritize rather than a reason to leave it undocumented.

Fix sketch

Populate the sidecar-routing map from the directory-recording sets as well as from the extension
primaries, so an _events.tsv resolves to a directory recording sitting in the same directory. Any fix
wants a test per format, since the three kinds are discovered by different mechanisms (extension-keyed
for .ds/.mefd, content-keyed for BTi) and a fix that works for one may miss another.

Found during review of nemarDatasets/.github#100. Part of epic #1095.

Activity

  1. neuromechanist commented on Aug 22, 2026

    @neuromechanist
    ContributorAuthor

    Re-verified on current .github main during the epic #1095 closeout audit: still present, not fixed by #100 or #98.

    compute_worklist builds its by_dir map from heads only:

    by_dir: dict[str, list[str]] = {}
    for p in heads:
        by_dir.setdefault(os.path.dirname(p), []).append(p)

    dirrec_dirs and bti_dirs are never added, so when affected_primaries resolves an _events.tsv against primaries_by_dir, a directory-format recording is not among the candidates and nothing is queued. Confirmed for CTF .ds, MEF3 .mefd, and 4D/BTi alike.

    Staying open. Note that epic #1108 Phase 1 (#1109) moves generate_zarr.py into nemar-cli, so the fix becomes a single-repo change with CI once that lands.

  2. added a commit that references this issue on Aug 22, 2026
    1580f72
  3. neuromechanist commented on Aug 22, 2026

    @neuromechanist
    ContributorAuthor

    Fixed in #1123, merged to dev as 1580f72.

    compute_worklist keyed by_dir on file heads only, so affected_primaries received an empty bucket for a CTF .ds, MEF3 .mefd, or 4D/BTi recording and an _events.tsv edit beside one queued nothing. Reproduced across all three formats before fixing (each returned []), and all three now rebuild.

    The --clean cron papering over it on the next full rebuild is exactly why it stayed invisible: the incremental path was wrong without ever being visibly wrong.

    Eleven tests, seven of which fail without the one-line change; the rest are negative guards against over-matching, plus a session/run/acq case since CTF and MEF3 are in practice the multi-session, multi-run formats.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions