Repository navigation
Move the Zarr engine to nemar-cli and retire the Actions path - #103
Merged
Merged
Conversation
The converter was designed to run as Actions billed against the Team allowance, but Actions cannot finish a large dataset inside the 120-minute cap. The engine moved to the SDSC Hallu cron and ZARR_AUTODISPATCH was defaulted off; the code never followed, so every change has been a cross-repo PR. Removes scripts/zarr/, run-generate-zarr.yml and test-generate-zarr.yml. They now live in nemarOrg/nemar-cli (ADR 0029), which is where the Hallu cron pulls the driver from as of nemar-cli#1109. patch_duration.py and its test move too: patch_duration.py imports generate_zarr (_run, materialize_recording), so it belongs with the module it depends on. generate-records.yml now checks out nemar-cli and runs it from there. That checkout is also the single source of the biosigIO pin, so this repo deliberately keeps no requirements file. emit_records.py is untouched; it only mentions generate_zarr in comments and duplicates the helpers it needs on purpose. DO NOT MERGE until nemar-cli main carries scripts/zarr/ and scripts/patch_duration.py, and Hallu has been cut over. Until then the deployed cron still clones this repo for its driver, and generate-records.yml checks out nemar-cli@main. Refs nemarOrg/nemar-cli#1109, nemarOrg/nemar-cli#1108
This was referenced Aug 22, 2026
neuromechanist
marked this pull request as ready for review
September 29, 2026 00:42
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Removes the Zarr conversion engine from this repo. It now lives in
nemarOrg/nemar-cliunderscripts/zarr/— see nemarOrg/nemar-cli#1114 and ADR 0029.The engine was here because conversion was originally designed to run as GitHub Actions billed against the Team allowance (
run-generate-zarr.yml: "Lives at nemarDatasets/.github so Actions minutes bill against the Team allowance"). Actions could not sustain bulk or backfill conversion — a large dataset stalls past the 120-minute cap — so the engine moved to the hourly SDSC Hallu cron and the Worker defaultedZARR_AUTODISPATCHoff. Only the code never followed, which made every converter change a cross-repo PR (five in three days during nemarOrg/nemar-cli#1095 alone) and left the deployed script on Hallu a hand-copied file rather than a checkout.Removed
scripts/zarr/(converter, queue, Hallu driver, the two satellite scripts, requirements, four test files).github/workflows/run-generate-zarr.yml— the dispatch path, already off by default and unreachable from the Worker.github/workflows/test-generate-zarr.yml— its tests run in nemar-cli CI now.github/workflows/test-patch-duration.yml— same, moved with its subjectMoved, not just deleted
scripts/patch_duration.py(+ its test) goes to nemar-cli too. It doessys.path.insert(..., "zarr"); import generate_zarrand calls_runandmaterialize_recording, so deletingscripts/zarr/without moving it would have brokengenerate-records.yml's Tier-2 duration backfill — a live publish-path workflow.generate-records.ymltherefore gains anactions/checkoutofnemarOrg/nemar-cli(public, no token) and runs the script from there. That checkout is also the single source of the biosigIO pin, so this repo deliberately keeps no requirements file: there is still exactly one pin, innemar-cli/scripts/zarr/requirements.txt.emit_records.pyis untouched. It only mentionsgenerate_zarrin comments and duplicates the helpers it needs on purpose; verified it imports nothing but stdlib.What stays here
Every workflow that genuinely runs as Actions against dataset repos — manifest, records, archive, BIDS validation, DOI, prescreen, onboarding — plus
emit_records.py,emit_manifest.py,backfill_records.py. ADR 0020 continues to govern those unchanged; ADR 0029 is a scoped carve-out for code that does not run in Actions at all, not a supersede.Manual recovery
workflow_dispatchonrun-generate-zarr.ymlis replaced byhallu-zarr.sh --dataset <id>on Hallu, which has no 120-minute cap and can actually finish a large dataset.DO NOT MERGE YET
Hard gate, in order:
main—generate-records.ymlhere checks outnemar-cli@main, soscripts/zarr/andscripts/patch_duration.pymust exist there first.hallu-zarr.sh --dataset nm099999.Merging before both is a production break in two places: the hourly cron still clones this repo for its driver, and the Tier-2 duration step would check out a
mainwithout the script.Refs nemarOrg/nemar-cli#1109, nemarOrg/nemar-cli#1108