You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Scratch sweep is guarded by a lock that does not cover the directory it deletes #1121
Running hallu-zarr.sh --stats on Hallu with ZARR_STATE_DIR pointed at a throwaway directory (to avoid disturbing the in-flight on007808 drain) deleted that drain's live scratch tree:
[2026-08-22 16:25:52 UTC] swept 1 orphaned scratch entries from a previous run (~40630 MiB reclaimed)
rm: cannot remove '/mnt/local/zarr-scratch/tmpttqpq8m0/stores/.../eeg_250hz/view/3/c/0/0': Directory not empty
aws: [ERROR]: The user-provided path /mnt/local/zarr-scratch/tmpttqpq8m0/stores/.../run-01_eeg.zarr does not exist.
::warning::[1374/1974] conversion failed for sub-01/ses-20240624/.../run-01_eeg.edf: aws s3 sync failed after 4 attempts
Cost was one recording out of 1974; the drain recovered by itself and continued. It could have been far worse with different timing, since the sweep had already reclaimed ~40 GB of in-flight state.
The bug
sweep_orphaned_scratch (scripts/zarr/hallu-zarr.sh:412) states its own safety argument at lines 408-411:
Sweeping is safe HERE and only here: we hold the single-instance lock and have not started the driver, so nothing owns anything under WORK_DIR and every tmp* entry is by definition from a dead run.
That invariant depends on the lock covering WORK_DIR. It does not. The lock is ${STATE_DIR}/.nm-zarr.lock and WORK_DIR is set independently by ZARR_WORK_DIR, so the two can be decoupled by configuration. Override ZARR_STATE_DIR alone and the new process takes an uncontended lock in its private state dir, then sweeps the sharedWORK_DIR out from under a running drain. find "$WORK_DIR" -maxdepth 1 -name 'tmp*' cannot distinguish a dead run's leftovers from a live run's active temp dir; "by definition from a dead run" holds only while the lock and the swept directory refer to the same deployment.
Nothing warns. The sweep reports its reclaim as a success.
Suggested fix
Make the guard cover the resource actually being mutated, rather than a sibling one. Options, roughly in order of preference:
Put the single-instance lock in WORK_DIR (or take a second lock there). The lock then means what the comment claims.
Refuse to sweep when exactly one of ZARR_STATE_DIR/ZARR_WORK_DIR is overridden, i.e. when the two no longer belong to the same deployment. Cheap, and turns a silent deletion into a loud refusal.
Skip tmp* entries that are demonstrably live: recent mtime, or an open file handle. Weaker on its own, but it also defends against a manual driver invocation that never took the lock at all.
Option 1 or 2, plus 3 as belt-and-braces, would close it properly. Whichever is chosen, the comment at lines 408-411 needs to stop asserting an invariant that configuration can break.
Related
Found while staging the epic #1108 Hallu cutover. Not a regression from that epic; the sweep predates it (#1068).
Activity
Zarr: scratch-sweep guard, probe code, claim (#1221)
neuromechanist commentedon Sep 3, 2026
Fixed in #1221 (dev commit 1867c27); ships with the next dev to main release.