Skip to content

Phase 1: Repatriate the zarr conversion engine to nemar-cli #1109

Closed

Description

@neuromechanist

Part of epic #1108.

Move the Zarr conversion engine from nemarDatasets/.github into this repo, where it can actually be developed and tested, and retire the GitHub Actions path it was originally shaped for.

Why

scripts/zarr/ lives in nemarDatasets/.github because the conversion was first designed to run as Actions, billed against the Team allowance (run-generate-zarr.yml: "Lives at nemarDatasets/.github so Actions minutes bill against the Team allowance"). Actions could not sustain bulk or backfill conversion; a large dataset stalls past the 120-minute cap. The engine moved to the Hallu cron and the backend retreated from the Actions path (ZARR_AUTODISPATCH defaults off, "the Hallu cron is the conversion engine"), but the code never moved back.

The result is three drifting copies of hallu-zarr.sh: the deployed hand-copy at /data/projects/yahya/nemar/hallu-zarr.sh on Hallu, the nemarDatasets/.github original it is byte-identical to (md5 cd7421da955e890ad81cf698e04d6091), and a stale 12.8 KB fork in this repo at scripts/hallu-zarr.sh that runs nowhere.

Scope

Move to this repo (scripts/zarr/), preserving history where practical:

  • generate_zarr.py, zarr_queue.py, patch_power_line_frequency.py, purge_non_raw_stores.py
  • test_generate_zarr.py, test_patch_power_line_frequency.py, test_purge_non_raw_stores.py, test_zarr_queue.py
  • requirements.txt, hallu-zarr.sh

Then:

  • Delete this repo's stale scripts/hallu-zarr.sh fork.
  • Add a pytest job to CI for scripts/zarr/ (uv-managed, per repo convention). These tests have never run in this repo's CI.
  • Repoint hallu-zarr.sh's setup() at a nemarOrg/nemar-cli clone instead of nemarDatasets/.github, and make the deployed copy on Hallu a git checkout rather than a hand-copied file, so git pull is the deployment mechanism.
  • Delete run-generate-zarr.yml and test-generate-zarr.yml from nemarDatasets/.github, along with scripts/zarr/.
  • Remove the now-dead triggerZarrGeneration dispatcher, the ZARR_AUTODISPATCH binding, and the webhook branch that consults it. Keep /webhooks/zarr-ready; the Hallu cron still POSTs it.
  • ADR recording where the conversion engine lives and why the Actions path was retired.

Manual per-dataset recovery becomes hallu-zarr.sh --dataset <id>, which has no 120-minute cap and is strictly better than the workflow_dispatch button it replaces.

Cutover

Cut over between cron ticks: wait for the flock to free, repoint DRIVER_REPO, verify with a full --dataset nm099999 run, then let the hourly cron resume. The queue DB lives in STATE_DIR and is untouched by the move.

Definition of done

  • Hallu runs the engine from a nemar-cli checkout and a fresh --dataset nm099999 conversion completes end to end.
  • scripts/zarr/ tests run in this repo's CI and pass.
  • nemarDatasets/.github contains no zarr code or workflows.
  • No hallu-zarr.sh copy exists outside a git checkout.

Activity

  1. neuromechanist commented on Aug 22, 2026

    @neuromechanist
    ContributorAuthor

    Done in #1114, merged to the epic branch and delivered to dev via #1119 (merge commit bad7cf4).

    The conversion engine now lives at scripts/zarr/ in this repo. The GitHub Actions dispatch path is retired (triggerZarrGeneration, the zarrDecision webhook branch, and ZARR_AUTODISPATCH are gone); /webhooks/zarr-ready stays, since the Hallu cron POSTs it. ADR 0029 records the carve-out from ADR 0020.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureNew feature or enhancementzarr

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions