Skip to content

Common schema for per-dataset process reports (publishing, Zarr, data quality, preprocessing) on neuroschema core plus aux extensions #1198

Open

Description

@neuromechanist

Later effort, filed at Yahya's request 2026-09-02 while organising epic #1181. Not scheduled; recorded so the Zarr index schema work in #1059 is designed as the first instance of it rather than a one-off.

Problem

Every process NEMAR runs on a dataset produces its own ad-hoc report shape: the Zarr converter's index.json (nemar-zarr-index, format_version 1, undocumented; #1059), the archive callback body, the availability report (ADR 0005), the recording-stats sweep, the HED sweep, the data-integrity sweep, and soon data-quality and preprocessing outputs from the pipeline. Each invents its own field names for the same ideas (which commit it ran on, when, what it found per recording, what failed and why, what is pending), each is consumed by a different reader (website, observability dashboard, MCP #1065, CLI), and none can be validated. The datasets table is deliberately counts-only (ADR 0034, #1190), so the per-recording detail has to live in these documents, which makes their schema the actual contract.

Proposal

One schema language for process reports, built on neuroschema (https://github.com/nemarOrg/neuroschema): the frozen core supplies dataset identity, provenance and the dataset/record two-level document model; a NEMAR extension namespace supplies a processReport aux schema with a common envelope (process, engine_version, source_commit, started_utc, finished_utc, discovered_count, succeeded_count, failed_count, pending_count, per-record entries with path, status, code, reason, detail, attempts) and per-process aux schemas for the payload (Zarr store geometry and groups, archive completeness, quality metrics, preprocessing provenance). Reports are validated against the schema at publish time and served with format and format_version, and a stability policy says which fields are frozen.

Steps when scheduled

  • Agree the envelope in neuroschema (issue there; link back here) and publish it under schema/extensions/.
  • Make the Zarr index v3 (Zarr index format v3: declare the data plane, the source tree, and a real schema #1059) conform to the envelope, with its store fields as the first aux payload; the coverage fields from the per-dataset coverage issue are envelope fields.
  • Migrate the archive and availability reports to the envelope when they are next touched.
  • Define the data-quality and preprocessing aux schemas with the pipeline (sccn/NEMAR-pipeline) before those outputs are published.
  • Validation in the publish paths (converter, callbacks) and a schema page on docs.nemar.org.

Activity

  1. neuromechanist commented on Sep 2, 2026

    @neuromechanist
    ContributorAuthor

    First instance: #1197 and #1059 (Zarr index v3) under epic #1181.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions