Skip to content

Zarr conversion pipeline for the staging data plane (zarr-test.nemar.org) #1180

Closed

Description

@neuromechanist

zarr-test.nemar.org exists and is wired end to end (Cloudflare Workers Custom Domain on the nemar-api-dev worker, backend/wrangler-sccn.toml [env.dev], ZARR_HOSTNAME="zarr-test.nemar.org", S3_BUCKET="nemar-dev"), but nothing keeps it populated with freshly-converted Zarr stores. Filing this so "staging is a full test platform for the viewer and the rest" (Yahya, 2026-09-01) includes the Zarr data plane, not just the API/website halves.

What backs zarr-test.nemar.org today (read from this checkout, backend/wrangler-sccn.toml + backend/src/routes/zarr-data.ts)

  • Separate bucket, not a view onto prod. [env.dev.vars] S3_BUCKET = "nemar-dev", fully distinct from prod's nemar bucket (separate Object Lock/versioning, separate IAM principal nemar-worker-dev with read-only prod access and no prod write path). zarr-data.ts's serve() builds the S3 URL from env.S3_BUCKET, so which bucket a request reads is entirely a function of which worker environment answers it — dev vs prod are already cleanly separated at the code level.
  • Gate is per-environment D1. isPublicDataset() checks visibility = 'public' against env.DB, which for the dev worker is nemar-db-dev (database_id 017d8f4e-...), a partial mirror of prod (per this repo's own .context/plan-923-test-staging.md and the website's AGENTS.md staging notes: ~722 datasets, ~600 users). A dataset's zarr rows there are independent of its prod rows.
  • The only thing that has ever populated nemar-dev/*/zarr/* is a one-time cross-bucket copy, not conversion. .context/plan-923-test-staging.md Phase 5 documents the "exemplar fleet" (scripts/exemplar-fleet.json, ~7 small xx0999NN datasets, one per modality + one zarr-ready + one archive-ready): its copy step does batchServerSideCopy of s3://nemar/<src>/zarr/* to s3://nemar-dev/<xx>/zarr/* when --include-derived is passed, i.e. it clones whatever the production converter already built for the source dataset. There is no step anywhere that runs the Zarr converter against nemar-dev and produces a store that did not already exist in prod.
  • The actual conversion engine is one Hallu cron, pointed at prod only. scripts/zarr/hallu-zarr.sh's single crontab entry (.context/systems-inventory.md §3.3, 30 * * * *) runs with no environment overrides, so it takes every default: API_BASE=https://api.nemar.org, S3_BUCKET=nemar, ZARR_AWS_PROFILE=nemar-zarr (an IAM profile scoped to s3:Get/Put/Delete on nemar/*/zarr/* only — it cannot write to nemar-dev even if pointed there), ZARR_STATE_DIR under /mnt/local/zarr-state (its SQLite queue, venv, and driver clone). There is no second instance, no second state directory, and no scoped credential for nemar-dev.
  • The plumbing to point the engine elsewhere already exists, unused. Every one of the above is environment-overridable today, with zero code changes needed to try it:
    • hallu-zarr.sh: API_BASE, S3_BUCKET, ZARR_CALLBACK_URL (${API_BASE}/webhooks/zarr-ready by default), ZARR_STATE_DIR/ZARR_WORK_DIR/ZARR_QUEUE_DB/ZARR_LOCK_FILE, ZARR_AWS_PROFILE, ZARR_DRIVER_REF.
    • scripts/zarr/zarr_queue.py reconcile --api-base <url> already takes an explicit catalog base (fetch_public_datasets(api_base)); it is not hardcoded to prod.
    • scripts/zarr/generate_zarr.py --bucket <name> already takes an explicit target bucket (default "nemar", not baked in).
    • By contrast, backend/src/services/github/dispatch.ts's triggerArchiveGeneration/triggerManifestGeneration (the GitHub-Actions-dispatched central workflows for archive/manifest) already carry s3_bucket/callback_base_url in their client_payload, added under epic Epic: Staging environment (test.nemar.org) + exemplar test-dataset fleet #923 specifically so those two pipelines follow the caller's environment. Zarr never got the equivalent treatment, but only because its production path moved off Actions entirely onto the Hallu cron (ADR 0029) before epic Epic: Staging environment (test.nemar.org) + exemplar test-dataset fleet #923 landed — there is no triggerZarrGeneration in dispatch.ts to extend in the first place, and the Actions-based run-generate-zarr.yml is manual-recovery-only today (headed for removal per nemarDatasets/.github#103, tracked in Finish the Hallu zarr cutover once the on007808 drain releases the lock #1122).
  • Reconcile's own scoping already does the dataset selection for free. qpy reconcile --api-base https://api-test.nemar.org would enqueue exactly the datasets nemar-db-dev's catalog marks public — today that is the exemplar fleet plus any dev-range (xx09*) uploads. Nothing further is needed to keep the target set small; it falls out of pointing reconcile at the test catalog rather than the prod one.

Why this matters now

#1178 (transfer-efficiency recommendations: view-level chunk shape, edge-cache Range caching, n_view_levels in attrs, metadata TTLs, index.json weight) and #1059/#1062/#1064 (index format v3, catalog exposure, provenance) all require a re-conversion to observe their effect. Today the only way to verify any of that before it reaches all ~40k prod stores is to convert on prod and check prod — there is no lower-stakes place to run a changed generate_zarr.py against real BIDS recordings and watch zarr-test.nemar.org serve the result. The exemplar-fleet copy mechanism cannot exercise this at all, since it only ever copies bytes prod already produced.

Proposed minimal design

Given the plumbing above is already in place, this does not need new converter code — it needs to actually be run and given credentials:

  1. Scoped IAM profile for nemar-dev. A nemar-zarr-dev-equivalent principal with s3:Get/Put/Delete on nemar-dev/*/zarr/* + ListBucket, mirroring nemar-zarr's scope but on the dev bucket. Without this, ZARR_AWS_PROFILE has nothing valid to point at.
  2. A second, independently-stated Hallu invocation, not a code fork of hallu-zarr.sh: a --test convenience flag (or documented env-var recipe, either is fine) that sets API_BASE=https://api-test.nemar.org, S3_BUCKET=nemar-dev, ZARR_AWS_PROFILE=nemar-zarr-dev, a distinct ZARR_STATE_DIR (so its SQLite queue/lock/venv/driver-clone never collide with the prod instance's), and ZARR_DRIVER_REF=dev (so it tracks the same unreleased driver code the rest of the dev branch's staging story tracks). Because STATE_DIR already derives LOCK_FILE/QUEUE_DB, this alone gives the test run its own flock — it can run on a separate, much less frequent cron tick (e.g. nightly, or on demand) without contending with the prod backfill for Hallu's cores.
  3. Confirm the webhook round-trip. ZARR_CALLBACK_URL would default to https://api-test.nemar.org/webhooks/zarr-ready; verify the dev worker's NEMAR_WEBHOOK_TOKEN secret is set (wrangler secret put NEMAR_WEBHOOK_TOKEN --env dev) so the callback authenticates and updates nemar-db-dev's zarr_status columns, the same way prod's callback updates nemar-db.
  4. Leave dataset scope to the existing catalog gate. Don't build a new allowlist mechanism — pointing reconcile at api-test.nemar.org already limits conversion to whatever is public in nemar-db-dev (the exemplar fleet today). If a specific format or size case needs coverage the fleet doesn't have, extend scripts/exemplar-fleet.json rather than inventing a second selection path.
  5. Decide whether the exemplar-fleet copy step should keep cross-bucket-copying prod's zarr/* at all once this exists, or switch to relying on the test conversion for fleet datasets that have a prod Zarr copy. Not blocking — the copy path is harmless to keep as a fast bootstrap for freshly-created exemplars while the conversion pipeline is still new.

Acceptance criteria

  • A scoped nemar-dev-only AWS IAM profile exists for Zarr writes.
  • hallu-zarr.sh (or a thin wrapper) can be invoked against api-test.nemar.org/nemar-dev without touching prod state (verified: prod queue DB/lock untouched, prod bucket untouched, e.g. via CloudTrail).
  • At least one dataset converts through the test path end to end: queued from nemar-db-dev, converted, pushed to nemar-dev, webhook callback updates nemar-db-dev, and zarr-test.nemar.org/<id>/zarr/index.json serves the result.
  • Documented in .context/systems-inventory.md alongside the existing §3.3 Hallu cron entry, and in the website's AGENTS.md staging section if it changes what staging can now exercise.
  • nemarOrg/website#212-style caveat added if needed: note explicitly what staging can now cover for the Zarr viewer that it could not before (real re-conversions, not just copied bytes).

Relationship to other open work

Feeds #1178 (needs a re-conversion to verify the view-chunk-shape and provenance-attrs recommendations before a prod-wide rebuild) and the #1059/#1062/#1064 index-format roadmap (same — format changes need a place to prove out against real recordings before touching all ~40k prod stores). Not blocking any of them; this is the harness they should land in, not itself a fix to a specific serving behavior.

Activity

  1. neuromechanist commented on Sep 3, 2026

    @neuromechanist
    ContributorAuthor

    The staging pipeline shipped in epic #1181 (phase 3, PR #1200) and the cron on Hallu now tracks dev. The nightly run had never enqueued anything because reconcile rejected the xx0999NN exemplar ids; fixed by #1211 (reconcile --accept-exemplars, passed by --test). The first run that converts the fleet is verified on zarr-test.nemar.org once the dev worker's webhook token is placed in the test state dir.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureNew feature or enhancementzarr

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions