You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Common schema for per-dataset process reports (publishing, Zarr, data quality, preprocessing) on neuroschema core plus aux extensions #1198
Later effort, filed at Yahya's request 2026-09-02 while organising epic #1181. Not scheduled; recorded so the Zarr index schema work in #1059 is designed as the first instance of it rather than a one-off.
Problem
Every process NEMAR runs on a dataset produces its own ad-hoc report shape: the Zarr converter's index.json (nemar-zarr-index, format_version 1, undocumented; #1059), the archive callback body, the availability report (ADR 0005), the recording-stats sweep, the HED sweep, the data-integrity sweep, and soon data-quality and preprocessing outputs from the pipeline. Each invents its own field names for the same ideas (which commit it ran on, when, what it found per recording, what failed and why, what is pending), each is consumed by a different reader (website, observability dashboard, MCP #1065, CLI), and none can be validated. The datasets table is deliberately counts-only (ADR 0034, #1190), so the per-recording detail has to live in these documents, which makes their schema the actual contract.
Proposal
One schema language for process reports, built on neuroschema (https://github.com/nemarOrg/neuroschema): the frozen core supplies dataset identity, provenance and the dataset/record two-level document model; a NEMAR extension namespace supplies a processReport aux schema with a common envelope (process, engine_version, source_commit, started_utc, finished_utc, discovered_count, succeeded_count, failed_count, pending_count, per-record entries with path, status, code, reason, detail, attempts) and per-process aux schemas for the payload (Zarr store geometry and groups, archive completeness, quality metrics, preprocessing provenance). Reports are validated against the schema at publish time and served with format and format_version, and a stability policy says which fields are frozen.
Steps when scheduled
Agree the envelope in neuroschema (issue there; link back here) and publish it under schema/extensions/.
Activity
neuromechanist commentedon Sep 2, 2026
First instance: #1197 and #1059 (Zarr index v3) under epic #1181.