Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -315,6 +315,10 @@ Environments and pre-release checks: [`.context/release-safety-playbook.md`](.co
hand-placed copy git never touches. Ship it with `scp` + atomic `mv`; the script
logs `DRIFT:` when the deployed copy differs from the repo copy. Manual recovery
for one dataset is `hallu-zarr.sh --dataset <id>`.
"Every run" is load-bearing and used to be a lie during a backfill: a run holds
the lock until the queue empties, so `setup()` never re-ran and the node sat two
deploys behind for two days (#1129). The drain now re-checks `origin/$DRIVER_REF`
between datasets and stops when it moves, so the next tick redeploys.
Conversion streams by default above 256 MiB, so peak RAM is a read window
plus one channel rather than the whole recording (ADR 0030). EEGLAB `.set`
is the one format that cannot: MNE refuses v7.3 files that biosigIO reads,
Expand Down
35 changes: 34 additions & 1 deletion scripts/zarr/hallu-zarr.sh
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,10 @@
# resample/compress.
# flock keeps a single long backfill draining across hourly cron ticks
# (a later tick finds the lock held and exits). As long as the box is on,
# the cron fires and the queue resumes exactly where it left off.
# the cron fires and the queue resumes exactly where it left off. The
# drain also stops itself at a dataset boundary when origin/$DRIVER_REF
# moves, so a merged driver change reaches the node on the next tick
# instead of waiting out the backfill (#1129).
#
# Usage:
# ./hallu-zarr.sh # reconcile + drain the queue
Expand Down Expand Up @@ -450,6 +453,32 @@ if [[ -f "$REPO_SELF" && "$SELF" != "$REPO_SELF" ]] && ! cmp -s "$SELF" "$REPO_S
err "DRIFT: $SELF differs from $REPO_SELF (origin/${DRIVER_REF}). Changes merged to this script are NOT live; deploy it."
fi

# The drift guard above answers "is this script stale relative to the repo it was
# launched from". It CANNOT answer "has that repo moved since this run started",
# and during a backfill that is the question that matters: a run does not end
# until the queue is empty, so setup() never re-runs and the deployed driver is
# frozen for the whole drain. Measured once, and it is not a small window --
# Hallu ran a driver two deploys behind for two days, missing the entire epic
# #1108 memory work, because the run holding the lock had started before the
# cutover (#1129).
#
# So re-check the tracked ref BETWEEN datasets and stop cleanly when it moves.
# One ls-remote per dataset is noise against per-dataset conversion times
# measured in minutes, and the queue persists across runs, so stopping costs
# nothing but the next cron tick.
#
# A failed ls-remote must NEVER stop a backfill: a network blip is not a reason
# to halt hours of work, and the worst case of ignoring it is that the redeploy
# waits for the next dataset boundary.
ref_moved() {
local remote_head local_head
remote_head="$(git -C "$DRIVER_REPO" ls-remote origin "$DRIVER_REF" 2>/dev/null | cut -f1)"
[[ -z "$remote_head" ]] && return 1
local_head="$(git -C "$DRIVER_REPO" rev-parse HEAD 2>/dev/null)"
[[ -z "$local_head" ]] && return 1
[[ "$remote_head" != "$local_head" ]]
}

if [[ -n "$STATS_ONLY" ]]; then
qpy stats
exit 0
Expand Down Expand Up @@ -488,6 +517,10 @@ while :; do
log "reached --limit $LIMIT; stopping (queue persists; next run continues)"
break
fi
if ref_moved; then
log "origin/${DRIVER_REF} moved; stopping so the next run redeploys (queue persists)"
break
fi
done

log "run complete: processed $n dataset(s); $(qpy stats | head -1)"
Loading