Skip to content

Long drains freeze the deployed Zarr driver: setup() never re-runs #1129

Closed

Description

@neuromechanist

setup() resets the Hallu clone to origin/$ZARR_DRIVER_REF on every run, which is what makes the Python driver self-deploying (AGENTS.md, ADR 0029). That property does not hold during a backfill, because a run does not end until the queue is empty.

The drain loop claims dataset after dataset inside a single invocation while holding the flock. Every hourly tick meanwhile logs another hallu-zarr instance holds the lock; exiting and returns without ever reaching setup(). So the deployed driver is frozen at whatever was current when the long run started, for as long as the backfill lasts -- which right now is weeks, not hours.

Observed

Measured on Hallu 2026-08-22, roughly a day after the cutover to the nemar-cli clone:

cron entry     -> /mnt/local/zarr-state/nemar-cli/scripts/zarr/hallu-zarr.sh  (correct)
clone HEAD     -> 9a8ce8c (#1124), 3 commits behind origin/dev
running driver -> /mnt/local/zarr-state/dotgithub/scripts/zarr/generate_zarr.py
                  i.e. nemarDatasets/.github@0447ab3, dated Aug 21

Feature markers in the driver actually executing:

Marker Present
is_excluded_from_discovery (ADR 0027) yes
ChannelCountMismatch (#1068) yes
stream_to_zarr (ADR 0030) yes
apply_worker_mem_limit (epic #1108) no
rss_trusted (#1124) no
maxwell_filter / MaxShieldUncalibrated (#1126) no

So every dataset converted since the cutover ran without the memory-robustness work from epic #1108 -- including #1110, where an OOM-killed worker takes the whole dataset down and burns its retry budget, which is the exact failure that produced the backlog that epic existed to fix.

The stores it produces are correct; what it lacks is the failure handling and the MaxShield path. But the window is unbounded, which is the actual problem.

Why the drift guard did not catch it

The guard compares the running script against $DRIVER_REPO/scripts/zarr/$(basename $0). The running instance started before the cutover with DRIVER_REPO pointing at the .github clone, so its own copy matched its own repo and the check passed. It answers "is this script stale relative to the repo it was launched from", not "is that repo still the one the cron would use".

Suggested fix

Re-check the tracked ref between datasets in the drain loop and exit cleanly when it has moved, letting the next tick redeploy:

remote_head="$(git -C "$DRIVER_REPO" ls-remote origin "$DRIVER_REF" | cut -f1)"
if [[ -n "$remote_head" && "$remote_head" != "$(git -C "$DRIVER_REPO" rev-parse HEAD)" ]]; then
  log "origin/$DRIVER_REF moved; stopping after this dataset so the next run redeploys"
  break
fi

One ls-remote per dataset is negligible against per-dataset conversion times measured in minutes, and the queue already persists across runs, so breaking out costs nothing. A failed ls-remote should be non-fatal (keep draining), since a network blip must not stop a backfill.

Alternatives considered: a default --limit so runs are always bounded (simple, but re-pays clone and reconcile cost far more often), or an operator stop-file (manual, does not fix the general case).

Until this lands, deploying a driver change during a backfill requires killing the run by hand.

Activity

  1. neuromechanist commented on Aug 23, 2026

    @neuromechanist
    ContributorAuthor

    Fixed in #1132, merged to dev as 3bc0629.

    ref_moved() compares git ls-remote origin $DRIVER_REF against the clone's HEAD between datasets in the drain loop; on a move the run logs and breaks, and the next :30 tick redeploys. Fails safe by design: an unresolvable ref or an unreachable remote returns "not moved" and the drain continues, because a network blip must never halt hours of backfill.

    Exercised against a real local git remote across all four paths (in sync, advanced, bad ref, unreachable remote). shellcheck -S warning clean.

    AGENTS.md said the driver redeploys "every run", which is the assumption that hid this; it now says what actually happens during a backfill.

    Note this does not rescue the currently running instance, which predates the fix and still holds the lock. That one still needs a manual stop (tracked on #1122).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions