Skip to content

Phase 2: zarr state in the catalog - #1201

Merged
neuromechanist merged 26 commits into
feature/issue-1181-epic-zarr-servingfrom
feature/issue-1062-phase2-catalog-zarr
Sep 2, 2026
Merged

neuromechanist merged 26 commits into
feature/issue-1181-epic-zarr-servingfrom
feature/issue-1062-phase2-catalog-zarr

Conversation

@neuromechanist

@neuromechanist neuromechanist commented Sep 2, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Add has_zarr=true/has_zarr=1 filter to GET /datasets, GET /datasets/search, and the nemar dataset list/search CLI (--has-zarr), following the exact has_hed idiom (AND d.zarr_status = 'ready' AND COALESCE(d.zarr_store_count, 0) > 0). Backed by the existing idx_datasets_zarr_status index; no new migration needed.
  • Add the seven existing-but-unexposed zarr columns (zarr_store_count, zarr_converted_at, zarr_source_commit, zarr_errors, zarr_failure_count, zarr_deterministic, zarr_failed_at) to the catalog list/detail projections and the shared wire contract. No new D1 columns (ADR 0034).
  • Add a derived (not stored) zarr_index_url on list and detail rows: the absolute <zarr-cache-host>/<id>/zarr/index.json URL when zarr_status='ready', else null. The contract now asserts this pairing with a .superRefine(): zarr_index_url set while zarr_status isn't 'ready' fails validation.
  • Parse zarr_data_failures server-side into the bounded {count, detail_ref, compacted_by?} object on GET /datasets/:id instead of forwarding the raw stored string (Heads-up: zarr_data_failures changes from an array to a {count, detail_ref} summary #1191); defensive against a legacy array shape, and against unparseable/non-object input (logged at console.error with the dataset id and reason, never a 500).
  • Add backend/src/services/zarr-catalog.ts: buildZarrCatalog (pure) and publishZarrCatalog(env), which queries D1 for public/active/ready/store>0 datasets and PUTs zarr-catalog.json to the env's own S3 bucket -- the top-level discovery document for a client with no dataset id to start from. buildZarrCatalog re-applies the ready/store>0 eligibility test itself (not just trusted from the SQL WHERE clause), so a future caller handing it an unfiltered row set still can't publish an ineligible entry.
  • Wire publishZarrCatalog into the daily cron (both prod and dev, tracked by a named DEV_CRON_ALLOWLIST constant in index.ts -- it only touches the env's own D1 and S3 bucket, no email/GitHub/DOI, so it's safe on the non-prod allowlist) and into a new POST /admin/zarr-catalog/publish (+ nemar admin zarr-catalog publish) for on-demand refresh.
  • Add GET /catalog.json to backend/src/routes/zarr-data.ts (inside createZarrDataRoutes(deps), using the phase 1 dependency-injected fetch/cache/safeCachePut), proxying and edge-caching (1h TTL) the published zarr-catalog.json via a signed GET, above the root / handler. A 403 from S3 answers 503 (ZarrCatalogForbiddenError, distinct from the 404 "not yet published" case); any other fetch failure answers 502. No D1 query on this route.

Related

Review response (four-reviewer pass on this PR)

Code review found nothing blocking; three other reviewers found real gaps, all addressed in follow-up commits:

  • parseZarrDataFailures now logs (console.error, dataset id + reason) and returns null for genuinely unparseable input (bad JSON, or a parsed value that's neither object nor array), instead of silently swallowing it -- still never 500s.
  • buildZarrCatalog defends its own ready/store>0 eligibility postcondition instead of only trusting the SQL WHERE clause.
  • The contract asserts zarr_index_url == null || zarr_status === 'ready' via .superRefine() on both catalogItemSchema and datasetDetailSchema.
  • endpointUrl moved off the shared PresignedUrlOptions (s3.ts) onto a ZarrCatalogS3Options local to zarr-catalog.ts, so the ~30 other S3 call sites don't carry a dead test-only field.
  • fetchZarrCatalogObject's 403 case now logs at console.error (not warn) with the bucket/key and throws a distinct ZarrCatalogForbiddenError instead of returning null like a genuine 404 absence -- the GET /catalog.json route (now on the merged phase-1 harness) answers 503 for it and 404 only for a true absence.
  • backend/test/cron-sweep-wiring.test.ts now pins publishZarrCatalog's wiring explicitly (called, wrapped in ctx.waitUntil, chained with .catch, outside the prod-only block) since it isn't sweep-shaped and falls outside that file's regex-based discovery by design; publishZarrCatalog also asserts S3_BUCKET is configured before attempting a PUT.
  • New backend/test/zarr-catalog-publish-route.test.ts covers the admin route directly: unauthenticated (401), non-admin (403), a thrown config error (500 carrying the message), and a real success round-trip against a local Bun.serve() receiver (registerZarrCatalogRoutes gained an optional DI seam for the publish function, mirroring runRecordingStatsSweep's fetchIndex injection -- omitted, so identical, in both production call sites).
  • deriveZarrIndexUrl gained unit tests (non-default base, trailing slash) that caught and fixed a latent double-slash bug when a caller passes an already-slashed base; combined-filter route tests (has_zarr=1&has_hed=1, has_zarr=1&modality=eeg) prove the clauses AND together; the route now documents that has_zarr=false/=yes is silently ignored (same as has_hed).
  • The hasZarr doc comment no longer promises a specific future redefinition ("phase 8 will tighten this"); phase 8 will add a separate has_zarr_verified filter instead, so today's meaning is stable.

IAM: no grant needed -- both nemar-worker-prod and nemar-worker-dev already hold s3:PutObject on their whole bucket (confirmed by the lead).

Deviations from the brief (with reasons)

  • No new migration. The brief said to add a partial index "if none covers zarr_status" -- idx_datasets_zarr_status already exists (migration 0035, recreated in 0071's rebuild), so the condition for adding one was never met.
  • GET /datasets/facets is not one of the has_zarr entry points. The brief listed it alongside list/search/CLI as "already accepting has_hed," but that endpoint takes no query parameters at all today (it returns the whole vocabulary unconditionally). has_hed isn't threaded into it either, so there was nothing to mirror. has_zarr was threaded into every place has_hed actually does flow: the list route, the search route (including its exact-id/FTS/semantic tiers via buildDatasetFilterClauses), hasActiveFilters in dataset-search.ts, and the CLI.
  • modalities/tasks reuse. The brief called this "the modalities/tasks JSON parsing the catalog route already does." The catalog route doesn't parse these into arrays anywhere (it serves them as raw comma-joined strings); the actual reusable logic is splitCsv in services/data-router.ts (used by the data.nemar.org neuroschema builder), which I exported and reused rather than re-implementing.
  • zarr_index_url on list rows, not just detail. The brief's wording was ambiguous about scope; I added it to both list and detail responses since the "Why" section's own motivating example (a client deciding stream-vs-download across many datasets) is a list use case, and the derivation is free (no extra query).

Test plan

  • cd backend && bun test -- full green.
  • Root bun test -- green for everything this PR touches; remaining failures are pre-existing, live-backend/test/.env.test-dependent integration tests and unrelated subprocess-timeout flakiness under load (verified unaffected by this change).
  • Biome clean, bunx tsc --noEmit clean in both backend/ and root.
  • Key test files: backend/test/catalog-has-zarr.test.ts, backend/test/zarr-catalog.test.ts, backend/test/zarr-catalog-publish-route.test.ts, backend/test/cron-sweep-wiring.test.ts, backend/test/zarr-catalog-route.test.ts (phase-1 harness sibling to zarr-data-cache.test.ts), test/contract-schemas.test.ts, test/has-zarr-cli-flag.test.ts.

Closes #1062
Closes #1191
Part of epic #1181

@github-actions

github-actions Bot commented Sep 2, 2026 •

Copy link
Copy Markdown

📦 Package Preview

Your package is ready to test!

bun install -g nemar-cli@PR1201

Version: 0.9.12-PR1201.9681a51
Commit: 9681a51


🤖 Updated on each push.

@neuromechanist
neuromechanist force-pushed the feature/issue-1062-phase2-catalog-zarr branch from cb015ee to 9681a51 Compare September 2, 2026 17:32

This branch was previously deployed

1 inactive deployment
npm — 9681a51d Deployed Sep 2, 2026 by neuromechanist via publish #2176
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant