Repository navigation
Description
zarr-test.nemar.org exists and is wired end to end (Cloudflare Workers Custom Domain on the nemar-api-dev worker, backend/wrangler-sccn.toml [env.dev], ZARR_HOSTNAME="zarr-test.nemar.org", S3_BUCKET="nemar-dev"), but nothing keeps it populated with freshly-converted Zarr stores. Filing this so "staging is a full test platform for the viewer and the rest" (Yahya, 2026-09-01) includes the Zarr data plane, not just the API/website halves.
What backs zarr-test.nemar.org today (read from this checkout, backend/wrangler-sccn.toml + backend/src/routes/zarr-data.ts)
- Separate bucket, not a view onto prod.
[env.dev.vars] S3_BUCKET = "nemar-dev", fully distinct from prod'snemarbucket (separate Object Lock/versioning, separate IAM principalnemar-worker-devwith read-only prod access and no prod write path).zarr-data.ts'sserve()builds the S3 URL fromenv.S3_BUCKET, so which bucket a request reads is entirely a function of which worker environment answers it — dev vs prod are already cleanly separated at the code level. - Gate is per-environment D1.
isPublicDataset()checksvisibility = 'public'againstenv.DB, which for the dev worker isnemar-db-dev(database_id 017d8f4e-...), a partial mirror of prod (per this repo's own.context/plan-923-test-staging.mdand the website's AGENTS.md staging notes: ~722 datasets, ~600 users). A dataset's zarr rows there are independent of its prod rows. - The only thing that has ever populated
nemar-dev/*/zarr/*is a one-time cross-bucket copy, not conversion..context/plan-923-test-staging.mdPhase 5 documents the "exemplar fleet" (scripts/exemplar-fleet.json, ~7 smallxx0999NNdatasets, one per modality + one zarr-ready + one archive-ready): itscopystep doesbatchServerSideCopyofs3://nemar/<src>/zarr/*tos3://nemar-dev/<xx>/zarr/*when--include-derivedis passed, i.e. it clones whatever the production converter already built for the source dataset. There is no step anywhere that runs the Zarr converter againstnemar-devand produces a store that did not already exist in prod. - The actual conversion engine is one Hallu cron, pointed at prod only.
scripts/zarr/hallu-zarr.sh's single crontab entry (.context/systems-inventory.md§3.3,30 * * * *) runs with no environment overrides, so it takes every default:API_BASE=https://api.nemar.org,S3_BUCKET=nemar,ZARR_AWS_PROFILE=nemar-zarr(an IAM profile scoped tos3:Get/Put/Deleteonnemar/*/zarr/*only — it cannot write tonemar-deveven if pointed there),ZARR_STATE_DIRunder/mnt/local/zarr-state(its SQLite queue, venv, and driver clone). There is no second instance, no second state directory, and no scoped credential fornemar-dev. - The plumbing to point the engine elsewhere already exists, unused. Every one of the above is environment-overridable today, with zero code changes needed to try it:
hallu-zarr.sh:API_BASE,S3_BUCKET,ZARR_CALLBACK_URL(${API_BASE}/webhooks/zarr-readyby default),ZARR_STATE_DIR/ZARR_WORK_DIR/ZARR_QUEUE_DB/ZARR_LOCK_FILE,ZARR_AWS_PROFILE,ZARR_DRIVER_REF.scripts/zarr/zarr_queue.py reconcile --api-base <url>already takes an explicit catalog base (fetch_public_datasets(api_base)); it is not hardcoded to prod.scripts/zarr/generate_zarr.py --bucket <name>already takes an explicit target bucket (default"nemar", not baked in).- By contrast,
backend/src/services/github/dispatch.ts'striggerArchiveGeneration/triggerManifestGeneration(the GitHub-Actions-dispatched central workflows for archive/manifest) already carrys3_bucket/callback_base_urlin theirclient_payload, added under epic Epic: Staging environment (test.nemar.org) + exemplar test-dataset fleet #923 specifically so those two pipelines follow the caller's environment. Zarr never got the equivalent treatment, but only because its production path moved off Actions entirely onto the Hallu cron (ADR 0029) before epic Epic: Staging environment (test.nemar.org) + exemplar test-dataset fleet #923 landed — there is notriggerZarrGenerationindispatch.tsto extend in the first place, and the Actions-basedrun-generate-zarr.ymlis manual-recovery-only today (headed for removal pernemarDatasets/.github#103, tracked in Finish the Hallu zarr cutover once the on007808 drain releases the lock #1122).
- Reconcile's own scoping already does the dataset selection for free.
qpy reconcile --api-base https://api-test.nemar.orgwould enqueue exactly the datasetsnemar-db-dev's catalog marks public — today that is the exemplar fleet plus any dev-range (xx09*) uploads. Nothing further is needed to keep the target set small; it falls out of pointingreconcileat the test catalog rather than the prod one.
Why this matters now
#1178 (transfer-efficiency recommendations: view-level chunk shape, edge-cache Range caching, n_view_levels in attrs, metadata TTLs, index.json weight) and #1059/#1062/#1064 (index format v3, catalog exposure, provenance) all require a re-conversion to observe their effect. Today the only way to verify any of that before it reaches all ~40k prod stores is to convert on prod and check prod — there is no lower-stakes place to run a changed generate_zarr.py against real BIDS recordings and watch zarr-test.nemar.org serve the result. The exemplar-fleet copy mechanism cannot exercise this at all, since it only ever copies bytes prod already produced.
Proposed minimal design
Given the plumbing above is already in place, this does not need new converter code — it needs to actually be run and given credentials:
- Scoped IAM profile for
nemar-dev. Anemar-zarr-dev-equivalent principal withs3:Get/Put/Deleteonnemar-dev/*/zarr/*+ListBucket, mirroringnemar-zarr's scope but on the dev bucket. Without this,ZARR_AWS_PROFILEhas nothing valid to point at. - A second, independently-stated Hallu invocation, not a code fork of
hallu-zarr.sh: a--testconvenience flag (or documented env-var recipe, either is fine) that setsAPI_BASE=https://api-test.nemar.org,S3_BUCKET=nemar-dev,ZARR_AWS_PROFILE=nemar-zarr-dev, a distinctZARR_STATE_DIR(so its SQLite queue/lock/venv/driver-clone never collide with the prod instance's), andZARR_DRIVER_REF=dev(so it tracks the same unreleased driver code the rest of thedevbranch's staging story tracks). BecauseSTATE_DIRalready derivesLOCK_FILE/QUEUE_DB, this alone gives the test run its ownflock— it can run on a separate, much less frequent cron tick (e.g. nightly, or on demand) without contending with the prod backfill for Hallu's cores. - Confirm the webhook round-trip.
ZARR_CALLBACK_URLwould default tohttps://api-test.nemar.org/webhooks/zarr-ready; verify the dev worker'sNEMAR_WEBHOOK_TOKENsecret is set (wrangler secret put NEMAR_WEBHOOK_TOKEN --env dev) so the callback authenticates and updatesnemar-db-dev'szarr_statuscolumns, the same way prod's callback updatesnemar-db. - Leave dataset scope to the existing catalog gate. Don't build a new allowlist mechanism — pointing
reconcileatapi-test.nemar.orgalready limits conversion to whatever is public innemar-db-dev(the exemplar fleet today). If a specific format or size case needs coverage the fleet doesn't have, extendscripts/exemplar-fleet.jsonrather than inventing a second selection path. - Decide whether the exemplar-fleet
copystep should keep cross-bucket-copying prod'szarr/*at all once this exists, or switch to relying on the test conversion for fleet datasets that have a prod Zarr copy. Not blocking — the copy path is harmless to keep as a fast bootstrap for freshly-created exemplars while the conversion pipeline is still new.
Acceptance criteria
- A scoped
nemar-dev-only AWS IAM profile exists for Zarr writes. -
hallu-zarr.sh(or a thin wrapper) can be invoked againstapi-test.nemar.org/nemar-devwithout touching prod state (verified: prod queue DB/lock untouched, prod bucket untouched, e.g. via CloudTrail). - At least one dataset converts through the test path end to end: queued from
nemar-db-dev, converted, pushed tonemar-dev, webhook callback updatesnemar-db-dev, andzarr-test.nemar.org/<id>/zarr/index.jsonserves the result. - Documented in
.context/systems-inventory.mdalongside the existing §3.3 Hallu cron entry, and in the website's AGENTS.md staging section if it changes what staging can now exercise. -
nemarOrg/website#212-style caveat added if needed: note explicitly what staging can now cover for the Zarr viewer that it could not before (real re-conversions, not just copied bytes).
Relationship to other open work
Feeds #1178 (needs a re-conversion to verify the view-chunk-shape and provenance-attrs recommendations before a prod-wide rebuild) and the #1059/#1062/#1064 index-format roadmap (same — format changes need a place to prove out against real recordings before touching all ~40k prod stores). Not blocking any of them; this is the harness they should land in, not itself a fix to a specific serving behavior.
Activity
neuromechanist commentedon Sep 3, 2026
The staging pipeline shipped in epic #1181 (phase 3, PR #1200) and the cron on Hallu now tracks dev. The nightly run had never enqueued anything because reconcile rejected the xx0999NN exemplar ids; fixed by #1211 (reconcile --accept-exemplars, passed by --test). The first run that converts the fleet is verified on zarr-test.nemar.org once the dev worker's webhook token is placed in the test state dir.