Skip to content

feat(zarr): queue-driven Hallu cron (SQLite engine) - #692

Merged
neuromechanist merged 6 commits into
devfrom
feature/zarr-cron-queue
Jun 3, 2026
Merged

neuromechanist merged 6 commits into
devfrom
feature/zarr-cron-queue

Conversation

@neuromechanist

Copy link
Copy Markdown
Contributor

Makes the Hallu zarr cron drain a queue gracefully (a full backfill is long, and bursts of due datasets can exceed an hour): a per-dataset manifest ({version, converted_utc}) lets it skip datasets already at their latest published version (no re-download), so progress is steady across hourly runs and resumable after a kill. The flock keeps a long run draining across ticks; --limit paces a single run; per-dataset rm -rf bounds disk to one dataset at a time. --force ignores the manifest. Part of #684.

Handles a long backfill + bursts of due datasets nicely: a per-dataset
manifest (version + converted_utc) lets the cron skip datasets already at their
latest published version (no re-download), so progress is steady across hourly
runs and resumable after a kill. The flock keeps a long run draining across
ticks; --limit paces a single run; per-dataset rm -rf bounds disk to one
dataset at a time. --force ignores the manifest.
Replaces the JSON-manifest skip with the persistent zarr_queue.py engine
(.github PR #23): each run reconciles (enqueue pending + recover stale
inprogress) then drains (claim -> download -> convert -> push -> rm -rf ->
done/fail-with-backoff) until empty or --limit. AWS_PROFILE=nemar-zarr.
--stats prints queue status. Crash-safe + resumable across restarts/ticks.
@neuromechanist neuromechanist changed the title feat(zarr): queue-graceful Hallu cron (manifest skip + resume) feat(zarr): queue-driven Hallu cron (SQLite engine) Jun 3, 2026
@github-actions

github-actions Bot commented Jun 3, 2026 •

Copy link
Copy Markdown

📦 Package Preview

Your package is ready to test!

bun install -g nemar-cli@PR692

Version: 0.8.46-PR692.156c9d8
Commit: 156c9d8


🤖 Updated on each push.

Clone metadata only then let the driver stream each recording's annex blob
from S3 objects/ just-in-time (the CI's non-local path), instead of downloading
the whole 18 GB upfront via --local. Starts converting immediately, never holds
the full dataset, ~40% faster. IAM profile gains s3:GetObject on objects/.

This branch was previously deployed

1 inactive deployment
npm — 156c9d89 Deployed Jun 3, 2026 by neuromechanist via publish #1296
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant