Repository navigation
Description
Epic #1095 (directory-format support: MEF3, CTF .ds, 4D/BTi; raw-vs-derived scope) closed 2026-08-22. Its cleanup, plus the serving-side work it exposed, is what's left. This epic organizes that, per Yahya's request (2026-09-01) to reorganize the zarr backlog rather than let it sprawl across unlinked issues.
Sub-issues
Backfill and correctness (finishing what #1095 started)
- Zarr backfill: directory-format datasets converted before epic #1095 are stranded at store_count 0 #1172 — Zarr backfill: directory-format datasets converted before epic Epic: Zarr availability and hygiene - formats, raw-only scope, whole-dataset failures #1095 are stranded at
store_count 0. Needs the one-off requeue plus an engine/format-version stamp so future discovery-rule widenings don't strand the back catalog again. - Zarr fidelity: 12 datasets serve one-channel copies of multichannel EEG; verification gate + quarantine + re-conversion #1068 — Zarr fidelity: 12+ datasets serve one-channel copies of multichannel EEG. Converter-side fix and gate are in; the confirmed-affected datasets have not all been re-converted, and two (
nm000182,nm000183) re-converted and are still wrong — a live, not-yet-diagnosed second instance of the bug.
Serving contract roadmap (the N-series referenced in #1065 and #1178)
- Zarr converter walks the whole repository, not the BIDS raw tree #1058 — Zarr converter walks the whole repository, not the BIDS raw tree (scope: exclude
derivatives//sourcedata/). - Zarr index format v3: declare the data plane, the source tree, and a real schema #1059 — Zarr index format v3: declare the data plane, the source tree, and a real schema.
- Publish
<id>/zarr/events.parquetwith converter-computed sample indices #1060 — Publish<id>/zarr/events.parquetwith converter-computed sample indices. - Make zarr.nemar.org the contract: proxy the index, redirect the bytes #1061 — Make zarr.nemar.org the contract: proxy the index, redirect the bytes.
- Expose Zarr state in the public catalog, and publish a top-level zarr catalog #1062 — Expose Zarr state in the public catalog, and publish a top-level zarr catalog (the
has_zarrflag Epic: NEMAR MCP server #1065'ssearch_datasetsneeds). - Structured provenance in store attributes, and a written store contract #1064 — Structured provenance in store attributes, and a written store contract (depends on Zarr index format v3: declare the data plane, the source tree, and a real schema #1059).
Transfer efficiency
- Zarr serving: transfer-efficiency recommendations from viewer audit #1178 — Zarr serving: transfer-efficiency recommendations from viewer audit (producer chunk shape, edge-cache Range caching,
n_view_levelsin attrs, metadata TTLs, index.json weight). - zarr index.json is 95% of Worker egress (1.22 GB vs 54 MB of chunk data) #1035 — zarr index.json is 95% of Worker egress. Overlaps Zarr serving: transfer-efficiency recommendations from viewer audit #1178 item 5 (index.json weight) and the untokened-metadata half of item 4 (TTL); keep both open but converge the fix so it isn't done twice — Zarr serving: transfer-efficiency recommendations from viewer audit #1178's payload-trim and zarr index.json is 95% of Worker egress (1.22 GB vs 54 MB of chunk data) #1035's caching angle are complementary, not competing.
Staging
- Zarr conversion pipeline for the staging data plane (zarr-test.nemar.org) #1180 — Zarr conversion pipeline for the staging data plane (zarr-test.nemar.org). New; see body for what backs zarr-test today and why it can't currently exercise a real re-conversion.
Strategic connection to the NEMAR MCP server (#1065)
#1178's serving-efficiency work is not only a cost/latency fix for the browser viewer — it is a prerequisite for #1065. An MCP (or any serverless/agent) consumer of the zarr data plane needs exactly the properties #1178 describes: bounded request counts per read (so a tool call doesn't fan out into hundreds of Range requests), cacheable/deterministic byte ranges (so repeated agent queries hit the edge instead of S3), and a declared view pyramid (so a client doesn't have to probe for it). #1065's read_window/render_overview tools inherit #1178's chunk geometry directly — see the chunk-geometry comment #1065 already carries (nm000103, 4 s inner shards) predicting exactly the request-count problem #1178 measures. Fixing #1178 on the producer/edge side makes the MCP tools cheap by construction rather than by client-side workaround. Cross-referenced on #1178 directly.
Not restructuring #1065 itself — it already tracks its own phasing — but noting the dependency here since it changes how #1178 should be prioritized: the browser-viewer case and the future MCP case both need the same fix.
Explicitly out of scope for this epic
Left off deliberately, still open, still real, but tactical/ops rather than "what does zarr serving look like next":
- Finish the Hallu zarr cutover once the on007808 drain releases the lock #1122 (finishing the Hallu cutover from epic Epic: Zarr conversion memory robustness #1108 — an ops runbook, not a serving-contract item)
- Scratch sweep is guarded by a lock that does not cover the directory it deletes #1121, Zarr: MaxShield probe failure has no failure code #1139, Zarr queue: no retry around the catalog fetch #1140, Zarr queue: claim_next docstring overstates atomicity #1142 (zarr queue robustness bugs — filed and explicitly deferred out of v0.9.9's release review, PR Address the v0.9.9 release review #1143)
- zarr: streaming reader for EDF/BDF/EEGLAB (convert large in-memory recordings, not skip) #944 (streaming reader for large in-memory formats — converter throughput, not serving)
- CTF recordings fail when the .hc file names coils Nasion/LPA/RPA #1125 (CTF
.hccoil-label bug — upstream/converter fix, tracked on its own; biosigIO 1.2.5 landed part of the fix buton006502still shows 264/361 failures live, so it isn't closeable yet either)
Activity
<id>/zarr/events.parquetwith converter-computed sample indices #1060Address PR #1199 five-reviewer follow-up
Phase 1: edge cache for zarr data plane (#1199)
feat: redirect zarr bytes for non-browser clients
docs: document chunk-redirect in access-metrics detail contract
Phase 6: redirect zarr bytes for non-browser clients (#1202)
neuromechanist commentedon Sep 3, 2026
Epic merged into dev via PR #1206 (regular merge, a8054de). Phases 1-10 and 12 landed; phase 11 (closeout ops: release to main, engine bump ack on Hallu, staging cron flip to dev, docs#15) is tracked in the release that follows.
docs(mcp): cite by content, not plan decision numbers (group C)
Phase 3: MCP recording tools (list_recordings, get_events, render_ove…