Repository navigation
Description
Part of epic #1108.
Move the Zarr conversion engine from nemarDatasets/.github into this repo, where it can actually be developed and tested, and retire the GitHub Actions path it was originally shaped for.
Why
scripts/zarr/ lives in nemarDatasets/.github because the conversion was first designed to run as Actions, billed against the Team allowance (run-generate-zarr.yml: "Lives at nemarDatasets/.github so Actions minutes bill against the Team allowance"). Actions could not sustain bulk or backfill conversion; a large dataset stalls past the 120-minute cap. The engine moved to the Hallu cron and the backend retreated from the Actions path (ZARR_AUTODISPATCH defaults off, "the Hallu cron is the conversion engine"), but the code never moved back.
The result is three drifting copies of hallu-zarr.sh: the deployed hand-copy at /data/projects/yahya/nemar/hallu-zarr.sh on Hallu, the nemarDatasets/.github original it is byte-identical to (md5 cd7421da955e890ad81cf698e04d6091), and a stale 12.8 KB fork in this repo at scripts/hallu-zarr.sh that runs nowhere.
Scope
Move to this repo (scripts/zarr/), preserving history where practical:
generate_zarr.py,zarr_queue.py,patch_power_line_frequency.py,purge_non_raw_stores.pytest_generate_zarr.py,test_patch_power_line_frequency.py,test_purge_non_raw_stores.py,test_zarr_queue.pyrequirements.txt,hallu-zarr.sh
Then:
- Delete this repo's stale
scripts/hallu-zarr.shfork. - Add a pytest job to CI for
scripts/zarr/(uv-managed, per repo convention). These tests have never run in this repo's CI. - Repoint
hallu-zarr.sh'ssetup()at anemarOrg/nemar-cliclone instead ofnemarDatasets/.github, and make the deployed copy on Hallu a git checkout rather than a hand-copied file, sogit pullis the deployment mechanism. - Delete
run-generate-zarr.ymlandtest-generate-zarr.ymlfromnemarDatasets/.github, along withscripts/zarr/. - Remove the now-dead
triggerZarrGenerationdispatcher, theZARR_AUTODISPATCHbinding, and the webhook branch that consults it. Keep/webhooks/zarr-ready; the Hallu cron still POSTs it. - ADR recording where the conversion engine lives and why the Actions path was retired.
Manual per-dataset recovery becomes hallu-zarr.sh --dataset <id>, which has no 120-minute cap and is strictly better than the workflow_dispatch button it replaces.
Cutover
Cut over between cron ticks: wait for the flock to free, repoint DRIVER_REPO, verify with a full --dataset nm099999 run, then let the hourly cron resume. The queue DB lives in STATE_DIR and is untouched by the move.
Definition of done
- Hallu runs the engine from a
nemar-clicheckout and a fresh--dataset nm099999conversion completes end to end. scripts/zarr/tests run in this repo's CI and pass.nemarDatasets/.githubcontains no zarr code or workflows.- No
hallu-zarr.shcopy exists outside a git checkout.
Activity
Move patch_duration.py in with the converter it imports
Address PR review findings
Phase 1: Repatriate the zarr conversion engine to nemar-cli (#1114)
neuromechanist commentedon Aug 22, 2026
Done in #1114, merged to the epic branch and delivered to
devvia #1119 (merge commit bad7cf4).The conversion engine now lives at
scripts/zarr/in this repo. The GitHub Actions dispatch path is retired (triggerZarrGeneration, thezarrDecisionwebhook branch, andZARR_AUTODISPATCHare gone);/webhooks/zarr-readystays, since the Hallu cron POSTs it. ADR 0029 records the carve-out from ADR 0020.Read zarr driver secrets from outside the checkout (#1120)
Address the v0.9.9 release review (#1143)
Phase 4: dev owns its reserved fixtures (#1445)
Move the Zarr engine to nemar-cli and retire the Actions path (#103)