Repository navigation
Phase 3: staging zarr conversion pipeline - #1200
Merged
neuromechanist merged 16 commits intoSep 2, 2026
Merged
neuromechanist merged 16 commits into
neuromechanist merged 16 commits into
Conversation
📦 Package PreviewYour package is ready to test! bun install -g nemar-cli@PR1200Version: 🤖 Updated on each push. |
neuromechanist
merged commit Sep 2, 2026
4057cc4
into
feature/issue-1181-epic-zarr-serving
12 checks passed
neuromechanist
deleted the
feature/issue-1180-phase3-staging-pipeline
branch
September 2, 2026 17:04
This was referenced Sep 2, 2026
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
--testand--print-configtoscripts/zarr/hallu-zarr.sh, giving the Zarr converter a second, independently-stated instance that converts real BIDS recordings againstnemar-db-dev/nemar-dev/zarr-test.nemar.orginstead of prod.--testis a pre-pass over"$@"that exports test defaults (API_BASE,TEST_API_URL,S3_BUCKET,ZARR_AWS_PROFILE,ZARR_STATE_DIR,ZARR_WORK_DIR,ZARR_DRIVER_REF,ZARR_JOBS=4) only for variables still unset, before the existing Config block resolves, and recordsTEST_PREPASS_SEENas the raw, can't-be-swallowed record of "--testwas in argv".--testrefuses to run ifS3_BUCKET=nemar,API_BASE/TEST_API_URLname the prod hostapi.nemar.org(case-insensitive, scheme/port/path/trailing-slash agnostic —api-test.nemar.orgis still allowed),AWS_PROFILE=nemar-zarr, orSTATE_DIR/WORK_DIRnormalize (trailing/doubled slashes collapsed,realpath-resolved when the dir exists) to the prod defaults — a stale prod value exported alongside--teststops the run instead of silently pointing at prod. The guard block keys offTEST_PREPASS_SEEN, not the arg parser'sTEST_MODE, so it cannot be suppressed by a parsing quirk in some other flag.--dataset/--limitnow refuse a value starting with--(same guard--requeuealready had), closing a hole where--dataset --test/--limit --testswallowed the--testtoken as the flag's own value.--test) run explicitly unsets an ambientTEST_API_URLleft over from an earlier--testsession, logging one line when it does, so a real prod conversion can never depend on stale shell state.--print-configprints every resolved config value (secrets only as present/absent), including the operation flags (ONLY_DATASET,LIMIT,REQUEUE,BACKFILL_DIR_FORMATS,PREVIEW_ENGINE_BUMP,EXECUTE), and exits 0 before touching the filesystem, network, or lock, in every documented flag order/combination.convert_dataset()'s metadata-clone step shells out to the separatenemarCLI binary, whose API-base resolution (src/lib/api/client.tsgetApiUrl()) is independent of$API_BASE— it readsTEST_API_URLfirst, then a stored account config, then defaults toapi.nemar.org. This was missed by issue Zarr conversion pipeline for the staging data plane (zarr-test.nemar.org) #1180's own env-var inventory (scoped to this script and the Python driver/queue) and surfaced asDataset not foundon the first live--test --datasetattempt, because the CLI was looking the dataset up in prod. Fixed by having--testalso exportTEST_API_URL— the same existing hook AGENTS.md's staging-auth section already documents, not a new mechanism.scripts/zarr/test_hallu_zarr_config.py(29 tests, real subprocess execution of the script, no mocks — up from 10 after a three-reviewer pass reproduced and closed four guard-bypasses live), wired into CI'szarr-python-testjob..context/systems-inventory.md§3.3 corrected to match the live crontab ground truth (prod cron invokes thenemar-cliclone's own copy ofhallu-zarr.sh, not a hand-placed standalone file; hand-placement is bootstrap-only) and §3.4 (paths, credentials table, bootstrap/run-by-hand recipes, cron line, what is/isn't shared with prod, live verification results);.context/validated_workflows.md§9 (bootstrap-and-run recipe with the exact commands that worked and the gotchas hit, including a note that later runs must invoke the clone's copy, not the bootstrap copy); ADR 0029 gets a dated amendment note (decision text left as originally written) pointing at the corrected description; one-sentence pointer added toAGENTS.md's Zarr paragraph.zarr_queue.py/generate_zarr.pythemselves —reconcile --api-baseandgenerate_zarr.py --bucketalready thread everything through as issue Zarr conversion pipeline for the staging data plane (zarr-test.nemar.org) #1180 said. The exemplar-fleet copy path is untouched — it's the copy phase ofnemar admin exemplar create --include-derived, not a standalonecopysubcommand.Live verification (2026-09-02)
Ran on Hallu, bootstrapped from this branch (
ZARR_DRIVER_REF=feature/issue-1180-phase3-staging-pipeline), against datasetxx099905([TEST COPY] BNCI 2014-004 Motor Imagery dataset, an exemplar-fleet dataset public in dev, never converted there before):api-test.nemar.org, all 5 raw.bdfrecordings streamed froms3://nemar-dev/xx099905/objects/, converted, and pushed tos3://nemar-dev/xx099905/zarr/.https://zarr-test.nemar.org/xx099905/zarr/index.jsonserved the fresh result immediately:store_count: 5,updated_utc: 2026-09-02T16:08:31Z,source_commitmatching the clone's HEAD.zarr-queue.dbmtime and.nm-zarr.lockunchanged before/after;aws s3api list-objects-v2 --bucket nemar --prefix xx099905/zarr/(my own admin credentials) returned zero objects;api.nemar.org/datasets/xx099905404s (the id exists only in dev)..zarr-secrets.envexists yet at/mnt/local/zarr-state-test/, the script logged the non-fatal warning, andapi-test.nemar.org/datasets/xx099905'szarr_status/zarr_store_count/zarr_converted_atstayednull— S3/index.json (what the viewer actually reads) advanced regardless./mnt/local/zarr-state/, never--execute'd any requeue against prod.Review round 2 (2026-09-02)
Three reviewers ran the script directly and reproduced four live guard bypasses; all four are fixed, with regression tests, plus the doc/wording follow-ups they flagged:
--dataset --test/--limit --testswallowed--testas the flag's value, leavingTEST_MODEunset and the guard block silent even though the pre-pass had already applied test defaults over an ambient prod value. Fixed both ways: the value parser now refuses a---prefixed value, and the guard block was moved offTEST_MODEonto the pre-pass's ownTEST_PREPASS_SEEN.STATE_DIR/WORK_DIRguards were exact-string comparisons; a trailing or doubled slash slipped past. Now normalized (slash-collapsed,realpath-resolved when the directory exists) on both sides, andWORK_DIRis guarded at all for the first time.API_BASE/TEST_API_URLguards were case-sensitive substring matches;https://API.NEMAR.ORGslipped past. Now a case-insensitive, host-only comparison (scheme/port/path/trailing-slash stripped) that still allowsapi-test.nemar.org.TEST_API_URLleft exported from an earlier--testsession would leak into a later plain prod run'snemarCLI call. A plain run now explicitly unsets it, logging once when it does.hallu-zarr.sh, hand-placement is bootstrap-only), ADR 0029 dated amendment, "copy --include-derived" reworded to "the copy phase ofnemar admin exemplar create --include-derived" (no standalonecopysubcommand exists), the.nm-zarr.logtable row reworded to note only the wrapper's own lines carry the[test]prefix, the AGENTS.md pointer reworded ("converts intonemar-dev's S3 zarr prefix, served atzarr-test.nemar.org"), the IAM credentials row dated, and a new gotcha in validated_workflows.md §9 about invoking the clone's copy (not the bootstrap copy) on later runs.All of it re-verified live against the real
--testguard rails on the actual script (not just the pytest suite) before pushing.What remains for ops
.context/systems-inventory.md§3.4):15 3 * * * /mnt/local/zarr-state-test/nemar-cli/scripts/zarr/hallu-zarr.sh --test >> /mnt/local/zarr-state-test/.nm-zarr-cron.log 2>&1..zarr-secrets.env(devNEMAR_WEBHOOK_TOKEN) at/mnt/local/zarr-state-test/so the zarr-ready callback stops being skipped and D1zarr_statusstarts advancing on the test instance.nemarOrg/website(not edited here): the website'sAGENTS.mdstaging section should now note thatzarr-test.nemar.orgcan carry real re-conversions, not just server-side-copied bytes from the exemplar fleet.Test plan
bash -n scripts/zarr/hallu-zarr.shshellcheck --severity=warning scripts/zarr/hallu-zarr.shcd scripts/zarr && uv run --with pytest pytest test_hallu_zarr_config.py -v(29 passed)python3 scripts/zarr/test_zarr_queue.py,test_generate_zarr.py,test_purge_non_raw_stores.py,test_patch_power_line_frequency.pyall still green (no regressions)nemar-dev/api-test.nemar.org/zarr-test.nemar.org(see above)Closes #1180
Part of epic #1181