Skip to content

Move the Zarr engine to nemar-cli and retire the Actions path - #103

Merged
neuromechanist merged 1 commit into
mainfrom
chore/retire-zarr-engine
Sep 29, 2026
Merged

neuromechanist merged 1 commit into
mainfrom
chore/retire-zarr-engine

Conversation

@neuromechanist

Copy link
Copy Markdown
Member

Summary

Removes the Zarr conversion engine from this repo. It now lives in nemarOrg/nemar-cli under scripts/zarr/ — see nemarOrg/nemar-cli#1114 and ADR 0029.

The engine was here because conversion was originally designed to run as GitHub Actions billed against the Team allowance (run-generate-zarr.yml: "Lives at nemarDatasets/.github so Actions minutes bill against the Team allowance"). Actions could not sustain bulk or backfill conversion — a large dataset stalls past the 120-minute cap — so the engine moved to the hourly SDSC Hallu cron and the Worker defaulted ZARR_AUTODISPATCH off. Only the code never followed, which made every converter change a cross-repo PR (five in three days during nemarOrg/nemar-cli#1095 alone) and left the deployed script on Hallu a hand-copied file rather than a checkout.

Removed

  • scripts/zarr/ (converter, queue, Hallu driver, the two satellite scripts, requirements, four test files)
  • .github/workflows/run-generate-zarr.yml — the dispatch path, already off by default and unreachable from the Worker
  • .github/workflows/test-generate-zarr.yml — its tests run in nemar-cli CI now
  • .github/workflows/test-patch-duration.yml — same, moved with its subject

Moved, not just deleted

scripts/patch_duration.py (+ its test) goes to nemar-cli too. It does sys.path.insert(..., "zarr"); import generate_zarr and calls _run and materialize_recording, so deleting scripts/zarr/ without moving it would have broken generate-records.yml's Tier-2 duration backfill — a live publish-path workflow.

generate-records.yml therefore gains an actions/checkout of nemarOrg/nemar-cli (public, no token) and runs the script from there. That checkout is also the single source of the biosigIO pin, so this repo deliberately keeps no requirements file: there is still exactly one pin, in nemar-cli/scripts/zarr/requirements.txt.

emit_records.py is untouched. It only mentions generate_zarr in comments and duplicates the helpers it needs on purpose; verified it imports nothing but stdlib.

What stays here

Every workflow that genuinely runs as Actions against dataset repos — manifest, records, archive, BIDS validation, DOI, prescreen, onboarding — plus emit_records.py, emit_manifest.py, backfill_records.py. ADR 0020 continues to govern those unchanged; ADR 0029 is a scoped carve-out for code that does not run in Actions at all, not a supersede.

Manual recovery

workflow_dispatch on run-generate-zarr.yml is replaced by hallu-zarr.sh --dataset <id> on Hallu, which has no 120-minute cap and can actually finish a large dataset.

DO NOT MERGE YET

Hard gate, in order:

  1. Phase 1: Repatriate the zarr conversion engine to nemar-cli nemarOrg/nemar-cli#1114 merges, and the epic reaches nemar-cli main — generate-records.yml here checks out nemar-cli@main, so scripts/zarr/ and scripts/patch_duration.py must exist there first.
  2. Hallu is cut over to a nemar-cli checkout and verified with hallu-zarr.sh --dataset nm099999.

Merging before both is a production break in two places: the hourly cron still clones this repo for its driver, and the Tier-2 duration step would check out a main without the script.

Refs nemarOrg/nemar-cli#1109, nemarOrg/nemar-cli#1108

The converter was designed to run as Actions billed against the Team
allowance, but Actions cannot finish a large dataset inside the
120-minute cap. The engine moved to the SDSC Hallu cron and
ZARR_AUTODISPATCH was defaulted off; the code never followed, so every
change has been a cross-repo PR.

Removes scripts/zarr/, run-generate-zarr.yml and test-generate-zarr.yml.
They now live in nemarOrg/nemar-cli (ADR 0029), which is where the
Hallu cron pulls the driver from as of nemar-cli#1109.

patch_duration.py and its test move too: patch_duration.py imports
generate_zarr (_run, materialize_recording), so it belongs with the
module it depends on. generate-records.yml now checks out nemar-cli
and runs it from there. That checkout is also the single source of the
biosigIO pin, so this repo deliberately keeps no requirements file.

emit_records.py is untouched; it only mentions generate_zarr in
comments and duplicates the helpers it needs on purpose.

DO NOT MERGE until nemar-cli main carries scripts/zarr/ and
scripts/patch_duration.py, and Hallu has been cut over. Until then the
deployed cron still clones this repo for its driver, and
generate-records.yml checks out nemar-cli@main.

Refs nemarOrg/nemar-cli#1109, nemarOrg/nemar-cli#1108
@neuromechanist
neuromechanist marked this pull request as ready for review September 29, 2026 00:42
@neuromechanist
neuromechanist merged commit bdcbe20 into main Sep 29, 2026
2 checks passed
@neuromechanist
neuromechanist deleted the chore/retire-zarr-engine branch September 29, 2026 00:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant