Skip to content

Stop the zarr drain when the tracked ref moves - #1132

Merged
neuromechanist merged 1 commit into
devfrom
feature/issue-1129-redeploy-on-ref-move
Aug 23, 2026
Merged

neuromechanist merged 1 commit into
devfrom
feature/issue-1129-redeploy-on-ref-move

Conversation

@neuromechanist

Copy link
Copy Markdown
Contributor

Summary

setup() resets the Hallu clone to origin/$ZARR_DRIVER_REF on every run, which is what makes the Python driver self-deploying (ADR 0029). That property silently does not hold during a backfill, because a run does not end until the queue is empty: the drain claims dataset after dataset inside one invocation while holding the flock, and every hourly tick meanwhile logs another hallu-zarr instance holds the lock; exiting without ever reaching setup().

Closes #1129.

This is not hypothetical

Measured on Hallu, roughly two days after the cutover to the nemar-cli clone:

cron entry     -> /mnt/local/zarr-state/nemar-cli/scripts/zarr/hallu-zarr.sh  (correct)
clone HEAD     -> 9a8ce8c, behind origin/dev
running script -> /data/projects/yahya/nemar/hallu-zarr.sh   (a third, older path)
running driver -> nemarDatasets/.github@0447ab3, i.e. the pre-ADR-0029 location

That run had been holding the lock for 1 day 23 hours. The driver it was executing had ADR 0027 discovery, the #1068 gate and streaming, but no apply_worker_mem_limit, no rss_trusted, and no MaxShield -- so two days of conversions ran without the entire epic #1108 memory work, including #1110, where an OOM-killed worker takes the whole dataset down and burns its retry budget. That is the exact failure epic #1108 was written to remove.

The existing drift guard could not catch this. It compares the running script against $DRIVER_REPO/scripts/zarr/$(basename $0), and a stale run's DRIVER_REPO is the same stale clone it was launched from: self-consistent, and two deploys behind. It answers "is this script stale relative to the repo it was launched from", not "has that repo moved since".

Change

ref_moved() compares git ls-remote origin $DRIVER_REF against the clone's HEAD, checked between datasets in the drain loop. On a move the run logs and breaks, and the next :30 tick redeploys.

One ls-remote per dataset is noise against per-dataset conversion times measured in minutes, and the queue persists across runs, so stopping costs nothing but the tick.

Failing safe was the design constraint. A network blip must never halt hours of backfill, so an unresolvable ref or an unreachable remote returns "not moved" and the drain continues; the worst case is that the redeploy waits for the next dataset boundary.

Test plan

There is no harness for this script, so ref_moved() was exercised directly against a real local git remote, all four paths:

Case Result
clone at remote HEAD keeps draining
remote advanced by one commit stops
DRIVER_REF does not resolve keeps draining
remote URL unreachable keeps draining

bash -n clean; shellcheck -S warning clean on the whole script.

AGENTS.md said the driver redeploys "every run", which was the assumption that hid this; it now says what actually happens.

This branch was previously deployed

1 inactive deployment
npm — f0475e0f Deployed Aug 23, 2026 by neuromechanist via publish #2053
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant