Skip to content

Scratch sweep is guarded by a lock that does not cover the directory it deletes #1121

Closed

Description

@neuromechanist

What happened

Running hallu-zarr.sh --stats on Hallu with ZARR_STATE_DIR pointed at a throwaway directory (to avoid disturbing the in-flight on007808 drain) deleted that drain's live scratch tree:

[2026-08-22 16:25:52 UTC] swept 1 orphaned scratch entries from a previous run (~40630 MiB reclaimed)
rm: cannot remove '/mnt/local/zarr-scratch/tmpttqpq8m0/stores/.../eeg_250hz/view/3/c/0/0': Directory not empty
aws: [ERROR]: The user-provided path /mnt/local/zarr-scratch/tmpttqpq8m0/stores/.../run-01_eeg.zarr does not exist.
::warning::[1374/1974] conversion failed for sub-01/ses-20240624/.../run-01_eeg.edf: aws s3 sync failed after 4 attempts

Cost was one recording out of 1974; the drain recovered by itself and continued. It could have been far worse with different timing, since the sweep had already reclaimed ~40 GB of in-flight state.

The bug

sweep_orphaned_scratch (scripts/zarr/hallu-zarr.sh:412) states its own safety argument at lines 408-411:

Sweeping is safe HERE and only here: we hold the single-instance lock and have not started the driver, so nothing owns anything under WORK_DIR and every tmp* entry is by definition from a dead run.

That invariant depends on the lock covering WORK_DIR. It does not. The lock is ${STATE_DIR}/.nm-zarr.lock and WORK_DIR is set independently by ZARR_WORK_DIR, so the two can be decoupled by configuration. Override ZARR_STATE_DIR alone and the new process takes an uncontended lock in its private state dir, then sweeps the shared WORK_DIR out from under a running drain. find "$WORK_DIR" -maxdepth 1 -name 'tmp*' cannot distinguish a dead run's leftovers from a live run's active temp dir; "by definition from a dead run" holds only while the lock and the swept directory refer to the same deployment.

Nothing warns. The sweep reports its reclaim as a success.

Suggested fix

Make the guard cover the resource actually being mutated, rather than a sibling one. Options, roughly in order of preference:

  1. Put the single-instance lock in WORK_DIR (or take a second lock there). The lock then means what the comment claims.
  2. Refuse to sweep when exactly one of ZARR_STATE_DIR/ZARR_WORK_DIR is overridden, i.e. when the two no longer belong to the same deployment. Cheap, and turns a silent deletion into a loud refusal.
  3. Skip tmp* entries that are demonstrably live: recent mtime, or an open file handle. Weaker on its own, but it also defends against a manual driver invocation that never took the lock at all.

Option 1 or 2, plus 3 as belt-and-braces, would close it properly. Whichever is chosen, the comment at lines 408-411 needs to stop asserting an invariant that configuration can break.

Related

Found while staging the epic #1108 Hallu cutover. Not a regression from that epic; the sweep predates it (#1068).

Activity

  1. added a commit that references this issue on Sep 3, 2026
    1867c27
  2. neuromechanist commented on Sep 3, 2026

    @neuromechanist
    ContributorAuthor

    Fixed in #1221 (dev commit 1867c27); ships with the next dev to main release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions