Skip to content

Expose Zarr state in the public catalog, and publish a top-level zarr catalog #1062

Closed

Description

@neuromechanist

Depends on: nothing

Summary

api.nemar.org/datasets exposes no Zarr field. None of the four SELECT lists in backend/src/routes/datasets/catalog.ts touch one, and the single zarr match in the file is a comment at line 31. Yet the datasets table carries eleven columns: zarr_status, zarr_converted_at, zarr_store_count, zarr_index_etag, zarr_source_commit (0035), zarr_checked_at (0038), and zarr_errors, zarr_failure_count, zarr_deterministic, zarr_data_failures, zarr_failed_at (0046).

A client that wants to decide stream-versus-download must currently probe <id>/zarr/index.json once per dataset. The failure columns in particular are exactly what that decision needs.

Proposal

  1. Add the zarr columns to the catalog SELECT lists and the response shape, at minimum zarr_status, zarr_store_count, zarr_converted_at, zarr_source_commit, zarr_errors.
  2. Add a has_zarr=true filter alongside the existing has_doi / has_hed filters.
  3. Publish s3://nemar/zarr-catalog.json: one entry per streamable dataset with id, DOI, license, modalities, participants, tasks, store count, total duration, has_hed. This is the front door for anyone, human or agent, who cannot ListBucket and does not already know a dataset id.

Acceptance criteria

  • GET /datasets?has_zarr=true returns only datasets with a Zarr copy.
  • Zarr fields present in the dataset object.
  • zarr-catalog.json published and refreshed by the same job that runs the zarr sweep.

Activity

  1. neuromechanist commented on Aug 11, 2026

    @neuromechanist
    ContributorAuthor

    Wording update from #1071: anonymous ListBucket is no longer fully denied — aws s3 ls --no-sign-request s3://nemar/ now lists the bucket top level (prefix must be empty, delimiter /). Listing inside any prefix is still denied, so index.json / the catalog remain the only way to enumerate a dataset's contents; the discovery story here is unchanged, but any text claiming ListBucket is denied outright should say top-level-only.

  2. neuromechanist commented on Sep 3, 2026

    @neuromechanist
    ContributorAuthor

    Shipped in v0.9.12 (epic #1181, merged to dev via #1206 and released via #1209 on 2026-09-03). Landed through #1201 (has_zarr and the zarr fields on the public catalog, the failures summary contract, zarr-catalog.json). Closing by hand: the phase PRs merged into the epic branch and the epic into dev, neither of which is the default branch, so the Closes references never fired.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions