Repository navigation
Description
setup() resets the Hallu clone to origin/$ZARR_DRIVER_REF on every run, which is what makes the Python driver self-deploying (AGENTS.md, ADR 0029). That property does not hold during a backfill, because a run does not end until the queue is empty.
The drain loop claims dataset after dataset inside a single invocation while holding the flock. Every hourly tick meanwhile logs another hallu-zarr instance holds the lock; exiting and returns without ever reaching setup(). So the deployed driver is frozen at whatever was current when the long run started, for as long as the backfill lasts -- which right now is weeks, not hours.
Observed
Measured on Hallu 2026-08-22, roughly a day after the cutover to the nemar-cli clone:
cron entry -> /mnt/local/zarr-state/nemar-cli/scripts/zarr/hallu-zarr.sh (correct)
clone HEAD -> 9a8ce8c (#1124), 3 commits behind origin/dev
running driver -> /mnt/local/zarr-state/dotgithub/scripts/zarr/generate_zarr.py
i.e. nemarDatasets/.github@0447ab3, dated Aug 21
Feature markers in the driver actually executing:
| Marker | Present |
|---|---|
is_excluded_from_discovery (ADR 0027) |
yes |
ChannelCountMismatch (#1068) |
yes |
stream_to_zarr (ADR 0030) |
yes |
apply_worker_mem_limit (epic #1108) |
no |
rss_trusted (#1124) |
no |
maxwell_filter / MaxShieldUncalibrated (#1126) |
no |
So every dataset converted since the cutover ran without the memory-robustness work from epic #1108 -- including #1110, where an OOM-killed worker takes the whole dataset down and burns its retry budget, which is the exact failure that produced the backlog that epic existed to fix.
The stores it produces are correct; what it lacks is the failure handling and the MaxShield path. But the window is unbounded, which is the actual problem.
Why the drift guard did not catch it
The guard compares the running script against $DRIVER_REPO/scripts/zarr/$(basename $0). The running instance started before the cutover with DRIVER_REPO pointing at the .github clone, so its own copy matched its own repo and the check passed. It answers "is this script stale relative to the repo it was launched from", not "is that repo still the one the cron would use".
Suggested fix
Re-check the tracked ref between datasets in the drain loop and exit cleanly when it has moved, letting the next tick redeploy:
remote_head="$(git -C "$DRIVER_REPO" ls-remote origin "$DRIVER_REF" | cut -f1)"
if [[ -n "$remote_head" && "$remote_head" != "$(git -C "$DRIVER_REPO" rev-parse HEAD)" ]]; then
log "origin/$DRIVER_REF moved; stopping after this dataset so the next run redeploys"
break
fiOne ls-remote per dataset is negligible against per-dataset conversion times measured in minutes, and the queue already persists across runs, so breaking out costs nothing. A failed ls-remote should be non-fatal (keep draining), since a network blip must not stop a backfill.
Alternatives considered: a default --limit so runs are always bounded (simple, but re-pays clone and reconcile cost far more often), or an operator stop-file (manual, does not fix the general case).
Until this lands, deploying a driver change during a backfill requires killing the run by hand.
Activity
neuromechanist commentedon Aug 23, 2026
Fixed in #1132, merged to
devas3bc0629.ref_moved()comparesgit ls-remote origin $DRIVER_REFagainst the clone's HEAD between datasets in the drain loop; on a move the run logs and breaks, and the next:30tick redeploys. Fails safe by design: an unresolvable ref or an unreachable remote returns "not moved" and the drain continues, because a network blip must never halt hours of backfill.Exercised against a real local git remote across all four paths (in sync, advanced, bad ref, unreachable remote).
shellcheck -S warningclean.AGENTS.md said the driver redeploys "every run", which is the assumption that hid this; it now says what actually happens during a backfill.
Note this does not rescue the currently running instance, which predates the fix and still holds the lock. That one still needs a manual stop (tracked on #1122).