Repository navigation
Phase 2: zarr state in the catalog - #1201
Merged
neuromechanist merged 26 commits intoSep 2, 2026
Merged
neuromechanist merged 26 commits into
neuromechanist merged 26 commits into
Conversation
📦 Package PreviewYour package is ready to test! bun install -g nemar-cli@PR1201Version: 🤖 Updated on each push. |
neuromechanist
force-pushed
the
feature/issue-1062-phase2-catalog-zarr
branch
from
September 2, 2026 17:32
cb015ee to
9681a51
Compare
neuromechanist
merged commit Sep 2, 2026
8f9f76a
into
feature/issue-1181-epic-zarr-serving
12 checks passed
This was referenced Sep 2, 2026
Closed
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
has_zarr=true/has_zarr=1filter toGET /datasets,GET /datasets/search, and thenemar dataset list/searchCLI (--has-zarr), following the exacthas_hedidiom (AND d.zarr_status = 'ready' AND COALESCE(d.zarr_store_count, 0) > 0). Backed by the existingidx_datasets_zarr_statusindex; no new migration needed.zarr_store_count,zarr_converted_at,zarr_source_commit,zarr_errors,zarr_failure_count,zarr_deterministic,zarr_failed_at) to the catalog list/detail projections and the shared wire contract. No new D1 columns (ADR 0034).zarr_index_urlon list and detail rows: the absolute<zarr-cache-host>/<id>/zarr/index.jsonURL whenzarr_status='ready', elsenull. The contract now asserts this pairing with a.superRefine():zarr_index_urlset whilezarr_statusisn't'ready'fails validation.zarr_data_failuresserver-side into the bounded{count, detail_ref, compacted_by?}object onGET /datasets/:idinstead of forwarding the raw stored string (Heads-up: zarr_data_failures changes from an array to a {count, detail_ref} summary #1191); defensive against a legacy array shape, and against unparseable/non-object input (logged atconsole.errorwith the dataset id and reason, never a 500).backend/src/services/zarr-catalog.ts:buildZarrCatalog(pure) andpublishZarrCatalog(env), which queries D1 for public/active/ready/store>0 datasets and PUTszarr-catalog.jsonto the env's own S3 bucket -- the top-level discovery document for a client with no dataset id to start from.buildZarrCatalogre-applies the ready/store>0 eligibility test itself (not just trusted from the SQL WHERE clause), so a future caller handing it an unfiltered row set still can't publish an ineligible entry.publishZarrCataloginto the daily cron (both prod and dev, tracked by a namedDEV_CRON_ALLOWLISTconstant in index.ts -- it only touches the env's own D1 and S3 bucket, no email/GitHub/DOI, so it's safe on the non-prod allowlist) and into a newPOST /admin/zarr-catalog/publish(+nemar admin zarr-catalog publish) for on-demand refresh.GET /catalog.jsontobackend/src/routes/zarr-data.ts(insidecreateZarrDataRoutes(deps), using the phase 1 dependency-injectedfetch/cache/safeCachePut), proxying and edge-caching (1h TTL) the publishedzarr-catalog.jsonvia a signed GET, above the root/handler. A 403 from S3 answers 503 (ZarrCatalogForbiddenError, distinct from the 404 "not yet published" case); any other fetch failure answers 502. No D1 query on this route.Related
has_zarr") is exactly what this PR does. The per-recordingdiscovered_count/pending_retry/detailwork it also asks for is converter/index-format territory (phase 7), out of scope here.Review response (four-reviewer pass on this PR)
Code review found nothing blocking; three other reviewers found real gaps, all addressed in follow-up commits:
parseZarrDataFailuresnow logs (console.error, dataset id + reason) and returnsnullfor genuinely unparseable input (bad JSON, or a parsed value that's neither object nor array), instead of silently swallowing it -- still never 500s.buildZarrCatalogdefends its own ready/store>0 eligibility postcondition instead of only trusting the SQL WHERE clause.zarr_index_url == null || zarr_status === 'ready'via.superRefine()on bothcatalogItemSchemaanddatasetDetailSchema.endpointUrlmoved off the sharedPresignedUrlOptions(s3.ts) onto aZarrCatalogS3Optionslocal to zarr-catalog.ts, so the ~30 other S3 call sites don't carry a dead test-only field.fetchZarrCatalogObject's 403 case now logs atconsole.error(notwarn) with the bucket/key and throws a distinctZarrCatalogForbiddenErrorinstead of returningnulllike a genuine 404 absence -- theGET /catalog.jsonroute (now on the merged phase-1 harness) answers 503 for it and 404 only for a true absence.backend/test/cron-sweep-wiring.test.tsnow pinspublishZarrCatalog's wiring explicitly (called, wrapped inctx.waitUntil, chained with.catch, outside the prod-only block) since it isn't sweep-shaped and falls outside that file's regex-based discovery by design;publishZarrCatalogalso assertsS3_BUCKETis configured before attempting a PUT.backend/test/zarr-catalog-publish-route.test.tscovers the admin route directly: unauthenticated (401), non-admin (403), a thrown config error (500 carrying the message), and a real success round-trip against a localBun.serve()receiver (registerZarrCatalogRoutesgained an optional DI seam for the publish function, mirroringrunRecordingStatsSweep'sfetchIndexinjection -- omitted, so identical, in both production call sites).deriveZarrIndexUrlgained unit tests (non-default base, trailing slash) that caught and fixed a latent double-slash bug when a caller passes an already-slashed base; combined-filter route tests (has_zarr=1&has_hed=1,has_zarr=1&modality=eeg) prove the clauses AND together; the route now documents thathas_zarr=false/=yesis silently ignored (same ashas_hed).hasZarrdoc comment no longer promises a specific future redefinition ("phase 8 will tighten this"); phase 8 will add a separatehas_zarr_verifiedfilter instead, so today's meaning is stable.IAM: no grant needed -- both
nemar-worker-prodandnemar-worker-devalready holds3:PutObjecton their whole bucket (confirmed by the lead).Deviations from the brief (with reasons)
zarr_status" --idx_datasets_zarr_statusalready exists (migration 0035, recreated in 0071's rebuild), so the condition for adding one was never met.GET /datasets/facetsis not one of thehas_zarrentry points. The brief listed it alongside list/search/CLI as "already accepting has_hed," but that endpoint takes no query parameters at all today (it returns the whole vocabulary unconditionally).has_hedisn't threaded into it either, so there was nothing to mirror.has_zarrwas threaded into every placehas_hedactually does flow: the list route, the search route (including its exact-id/FTS/semantic tiers viabuildDatasetFilterClauses),hasActiveFiltersindataset-search.ts, and the CLI.splitCsvinservices/data-router.ts(used by thedata.nemar.orgneuroschema builder), which I exported and reused rather than re-implementing.zarr_index_urlon list rows, not just detail. The brief's wording was ambiguous about scope; I added it to both list and detail responses since the "Why" section's own motivating example (a client deciding stream-vs-download across many datasets) is a list use case, and the derivation is free (no extra query).Test plan
cd backend && bun test-- full green.bun test-- green for everything this PR touches; remaining failures are pre-existing, live-backend/test/.env.test-dependent integration tests and unrelated subprocess-timeout flakiness under load (verified unaffected by this change).bunx tsc --noEmitclean in bothbackend/and root.backend/test/catalog-has-zarr.test.ts,backend/test/zarr-catalog.test.ts,backend/test/zarr-catalog-publish-route.test.ts,backend/test/cron-sweep-wiring.test.ts,backend/test/zarr-catalog-route.test.ts(phase-1 harness sibling to zarr-data-cache.test.ts),test/contract-schemas.test.ts,test/has-zarr-cli-flag.test.ts.Closes #1062Closes #1191Part of epic #1181