You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Expose Zarr state in the public catalog, and publish a top-level zarr catalog #1062
api.nemar.org/datasets exposes no Zarr field. None of the four SELECT lists in backend/src/routes/datasets/catalog.ts touch one, and the single zarr match in the file is a comment at line 31. Yet the datasets table carries eleven columns: zarr_status, zarr_converted_at, zarr_store_count, zarr_index_etag, zarr_source_commit (0035), zarr_checked_at (0038), and zarr_errors, zarr_failure_count, zarr_deterministic, zarr_data_failures, zarr_failed_at (0046).
A client that wants to decide stream-versus-download must currently probe <id>/zarr/index.json once per dataset. The failure columns in particular are exactly what that decision needs.
Proposal
Add the zarr columns to the catalog SELECT lists and the response shape, at minimum zarr_status, zarr_store_count, zarr_converted_at, zarr_source_commit, zarr_errors.
Add a has_zarr=true filter alongside the existing has_doi / has_hed filters.
Publish s3://nemar/zarr-catalog.json: one entry per streamable dataset with id, DOI, license, modalities, participants, tasks, store count, total duration, has_hed. This is the front door for anyone, human or agent, who cannot ListBucket and does not already know a dataset id.
Acceptance criteria
GET /datasets?has_zarr=true returns only datasets with a Zarr copy.
Zarr fields present in the dataset object.
zarr-catalog.json published and refreshed by the same job that runs the zarr sweep.
Wording update from #1071: anonymous ListBucket is no longer fully denied — aws s3 ls --no-sign-request s3://nemar/ now lists the bucket top level (prefix must be empty, delimiter /). Listing inside any prefix is still denied, so index.json / the catalog remain the only way to enumerate a dataset's contents; the discovery story here is unchanged, but any text claiming ListBucket is denied outright should say top-level-only.
Shipped in v0.9.12 (epic #1181, merged to dev via #1206 and released via #1209 on 2026-09-03). Landed through #1201 (has_zarr and the zarr fields on the public catalog, the failures summary contract, zarr-catalog.json). Closing by hand: the phase PRs merged into the epic branch and the epic into dev, neither of which is the default branch, so the Closes references never fired.
Activity
neuromechanist commentedon Aug 11, 2026
Wording update from #1071: anonymous ListBucket is no longer fully denied —
aws s3 ls --no-sign-request s3://nemar/now lists the bucket top level (prefix must be empty, delimiter /). Listing inside any prefix is still denied, so index.json / the catalog remain the only way to enumerate a dataset's contents; the discovery story here is unchanged, but any text claiming ListBucket is denied outright should say top-level-only.neuromechanist commentedon Sep 3, 2026
Shipped in v0.9.12 (epic #1181, merged to dev via #1206 and released via #1209 on 2026-09-03). Landed through #1201 (has_zarr and the zarr fields on the public catalog, the failures summary contract, zarr-catalog.json). Closing by hand: the phase PRs merged into the epic branch and the epic into dev, neither of which is the default branch, so the Closes references never fired.