Skip to content

Publish <id>/zarr/events.parquet with converter-computed sample indices #1060

Closed

Description

@neuromechanist

Depends on: batch with N1 and N2

Summary

There are no events in the Zarr stores. I probed for an annotations node and got 404. Every client that wants to epoch must go back to the BIDS tree, fetch events.tsv per recording, parse TSV, and align onsets to sample indices in the resampled signal. Publish one columnar file per dataset instead.

Why this is the highest-value item on the list

The alignment is only correct if you compute it. The converter is the only party that knows the exact resampling relationship between source rate and level-0 rate, including polyphase filter delay. Every client currently recomputes round(onset * rate) and quietly gets a sub-sample offset wrong wherever the source rate is not an integer multiple of the target, which is common: nm000329 resamples 1000 Hz to 250 Hz, on004362 is at 160 Hz, on007763 has groups at both 200 Hz and 250 Hz within one store. Publishing the sample index makes alignment correct by construction and removes an entire class of silent error from every downstream tool at once.

One fetch plans the whole job. A 100-subject dataset with 500 events each is 50k rows, roughly 0.5 to 2 MB dictionary-encoded. A client fetches one object and can compute how many epochs per condition, which shards contain them, and what the event density is, with zero signal bytes read.

It makes shard planning trivial. shard_id = sample_index // shard_samples is one column operation.

Schema

store_path    string (dict)     joins to index.json stores[].zarr
subject       string (dict)
session       string (dict, nullable)
task          string (dict)
run           string (dict, nullable)
onset_s       float64           as in events.tsv
duration_s    float32
sample_index  int64             onset in LEVEL-0 SAMPLES, post-resample
group_name    string (dict)     which zarr group, e.g. "eeg_250hz"
trial_type    string (dict)
value         string (dict, nullable)
hed           string (dict, nullable)
...           pass through remaining events.tsv columns

hed matters for N8 and for agent readability generally: NEMAR already tracks has_hed and hed_version in the datasets table, and HED is what turns an opaque trial_type into something a machine can reason about.

Acceptance criteria

  • <id>/zarr/events.parquet present for every dataset with events.
  • sample_index verified against a known-rate-change dataset (nm000329, 1000 Hz to 250 Hz) to within one sample.
  • Schema documented alongside the index schema from N2.
  • n_events and trial_types in index.json derived from the same source.

Activity

  1. neuromechanist commented on Sep 3, 2026

    @neuromechanist
    ContributorAuthor

    Shipped in v0.9.12 (epic #1181, merged to dev via #1206 and released via #1209 on 2026-09-03). Landed through #1205 (/zarr/events.parquet with converter-computed sample_index = floor(onset_s * rate + 0.5), one row per event and channel group). Closing by hand: the phase PRs merged into the epic branch and the epic into dev, neither of which is the default branch, so the Closes references never fired.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions