Skip to content

🐛 fix(async): honor cancellation and per-thread gates - #844

Merged
gaborbernat merged 11 commits into
mainfrom
fix/async-cancel-threadlocal
Oct 17, 2026
Merged

gaborbernat merged 11 commits into
mainfrom
fix/async-cancel-threadlocal

Conversation

@gaborbernat

@gaborbernat gaborbernat commented Oct 16, 2026 •

Copy link
Copy Markdown
Member

Two async locks ignored their callers while waiting. The async read-write wrappers ignored cancellation while another process held the lock. A thread-local AsyncFileLock shared by threads with their own event loops deadlocked on its first contention. ⏳ This PR lets a cancelled acquire stop its backend wait and gives each thread-local lock context its own transition gate (the queue that orders one lock's acquire and release calls). It builds on #838, which records the wrappers' sync-lock hold under a task owner.

Cancellation waited for the other process to let go

AsyncReadWriteLock and AsyncSoftReadWriteLock handed the whole contended wait to one executor call. Since #652, copied to the soft wrapper in #686, a CancelledError made _run_acquire drain that call before rolling it back. Nothing told the call to stop. For SQLite it was one busy_timeout wait of about 23 days at timeout=-1; for the soft lock an open-ended poll loop. So asyncio.timeout() or task.cancel(), which the how-to recommends for bounding the wait, took effect only when the holder released. asyncio.run() needed several Ctrl-C presses to exit. With a finite lock timeout the drained call ended in Timeout, which replaced the cancellation, so a background task cancelled at shutdown kept retrying.

sequenceDiagram
    participant T as task
    participant W as executor worker
    participant S as SQLite / lock files
    participant P as another process
    P->>S: holds the write lock
    T->>W: acquire (one call)
    W->>S: wait
    T-->>T: asyncio.timeout(1) fires: CancelledError
    rect rgb(255, 214, 214)
        Note over T,S: before: drain the call. It ends when P releases, then Timeout or CancelledError
    end
    rect rgb(216, 243, 220)
        Note over T,S: after: set the abandon flag. The worker gives up at its next slice, the task sees CancelledError
        T->>W: abandoned.set()
        W->>S: busy_timeout slice ends, database is locked
        W-->>T: Timeout → re-raised as the cancellation
    end
Loading

The wrapper now passes an abandon flag into the sync-lock acquire and sets it on cancellation. ReadWriteLock waits for SQLite in bounded busy-timeout slices on the same connection, retrying BEGIN while SQLite reports "database is locked". A read's deferred BEGIN succeeds before its SELECT takes the shared lock, so the retry rolls that back first. _acquire_thread_lock() takes the lock's in-process lock in the same slices. SoftReadWriteLock checks the flag between polls, so its existing rollback withdraws the published claim. A wait that gave up this way ends in a Timeout the wrapper re-raises as the caller's cancellation. The wrapper still rolls back an acquire that won the race meanwhile. Slicing inside the sync lock keeps one connection open for the wait; retrying the whole acquire from the wrapper would reopen it ten times a second. Synchronous callers keep the single uninterrupted wait.

#840 splits a threaded process's wait into 50 ms attempts for the same _acquire. Whichever of the two lands second has to pass holder and cancel_check into each attempt and ask cancel_check() between attempts, because a 50 ms attempt is shorter than the 100 ms slice and does not install the check on its own. The retry loop in _acquire therefore also checks for cancellation after each attempt that times out, and gives up when the caller has abandoned the acquire.

A thread-local async lock deadlocked between its threads

BaseAsyncFileLock created one transition gate per instance in #652. thread_local=True gives each thread its own lock state through threading.local. A thread whose own context held nothing took the gate and kept it for its whole poll loop. The thread that held the lock then queued on the same gate inside release(). With the default timeout=-1 neither thread ever returned and the lock file stayed held. With a finite timeout every wait ended in Timeout, however early the holder finished. Any other process holding the lock file while both threads called acquire() set the order up by itself. The sync BaseFileLock skips its gate for thread-local instances; the async side did not.

sequenceDiagram
    participant A as thread 1 (holds the lock)
    participant G as gate
    participant B as thread 2 (own context, unlocked)
    rect rgb(255, 214, 214)
        Note over A,B: before: one gate per instance
        B->>G: acquire(): take the gate, poll while A holds
        A->>G: release(): queue behind B
        Note over A,B: each waits for the other
    end
    rect rgb(216, 243, 220)
        Note over A,B: after: one gate per context
        B->>B: acquire(): own gate, poll
        A->>A: release(): own gate, unlock at once
        B->>B: next poll takes the lock
    end
Loading

The gate moved into AsyncFileLockContext. AsyncThreadLocalFileContext already delegates every dataclass field to the calling thread's context, so a thread-local lock gets one gate per thread and a shared lock keeps one per instance. Skipping the gate for thread-local instances, as the sync lock does, would not do: several tasks of one thread still share that thread's context and need the serialization.

Behavior changes

A cancelled AsyncReadWriteLock or AsyncSoftReadWriteLock acquire that was waiting on another process now ends with the caller's CancelledError within about 0.1 s (one poll interval for the soft lock). It used to end in Timeout, or in a cancellation delayed until the holder released. An async writer waiting for readers now holds SQLite's PENDING lock only within a slice and gives it up between slices. A reader in another process can therefore enter between two slices where it used to wait behind the writer.

@gaborbernat
gaborbernat force-pushed the fix/async-soft-rw-ownership branch from 009d6a0 to 5cfca83 Compare October 16, 2026 02:09
@gaborbernat
gaborbernat force-pushed the fix/async-cancel-threadlocal branch from ee91b85 to 113ccbb Compare October 16, 2026 02:09
@gaborbernat
gaborbernat force-pushed the fix/async-soft-rw-ownership branch from 5cfca83 to 815ccec Compare October 16, 2026 02:36
@gaborbernat
gaborbernat force-pushed the fix/async-cancel-threadlocal branch 2 times, most recently from 5904f33 to f6a1290 Compare October 16, 2026 03:05
@gaborbernat
gaborbernat force-pushed the fix/async-soft-rw-ownership branch from 1b1f466 to e3815b7 Compare October 16, 2026 04:52
@gaborbernat
gaborbernat force-pushed the fix/async-cancel-threadlocal branch from f6a1290 to bdf41da Compare October 16, 2026 04:52
@gaborbernat
gaborbernat force-pushed the fix/async-soft-rw-ownership branch from e3815b7 to da8e556 Compare October 16, 2026 05:04
@gaborbernat
gaborbernat force-pushed the fix/async-cancel-threadlocal branch from bdf41da to a6568ca Compare October 16, 2026 05:04
@gaborbernat
gaborbernat force-pushed the fix/async-soft-rw-ownership branch from da8e556 to 5c82a91 Compare October 16, 2026 05:52
@gaborbernat
gaborbernat force-pushed the fix/async-cancel-threadlocal branch from a6568ca to cc7e5af Compare October 16, 2026 05:52
@gaborbernat
gaborbernat force-pushed the fix/async-soft-rw-ownership branch from 5c82a91 to 4434e7e Compare October 16, 2026 06:18
@gaborbernat
gaborbernat force-pushed the fix/async-cancel-threadlocal branch from cc7e5af to ad1c1d7 Compare October 16, 2026 06:18
@gaborbernat
gaborbernat force-pushed the fix/async-soft-rw-ownership branch from 4434e7e to c2026e3 Compare October 16, 2026 08:21
@gaborbernat
gaborbernat force-pushed the fix/async-cancel-threadlocal branch 2 times, most recently from 17c8bf7 to b7346cb Compare October 16, 2026 08:23
@gaborbernat
gaborbernat force-pushed the fix/async-soft-rw-ownership branch from c2026e3 to 8ad2b6c Compare October 16, 2026 08:34
@gaborbernat
gaborbernat force-pushed the fix/async-cancel-threadlocal branch from b7346cb to c6de040 Compare October 16, 2026 08:34
@gaborbernat
gaborbernat force-pushed the fix/async-soft-rw-ownership branch from 8ad2b6c to e538301 Compare October 16, 2026 13:43
@gaborbernat
gaborbernat force-pushed the fix/async-cancel-threadlocal branch from c6de040 to 242352d Compare October 16, 2026 13:43
@gaborbernat
gaborbernat force-pushed the fix/async-soft-rw-ownership branch from e538301 to bfd299c Compare October 16, 2026 13:46
@gaborbernat
gaborbernat force-pushed the fix/async-cancel-threadlocal branch from 242352d to 1ad3357 Compare October 16, 2026 13:46
@gaborbernat
gaborbernat force-pushed the fix/async-soft-rw-ownership branch from bfd299c to 76f9e66 Compare October 16, 2026 17:01
@gaborbernat
gaborbernat force-pushed the fix/async-cancel-threadlocal branch from 1ad3357 to 893d6db Compare October 16, 2026 17:02
@gaborbernat
gaborbernat force-pushed the fix/async-soft-rw-ownership branch from 76f9e66 to 1650d3e Compare October 16, 2026 19:57
@gaborbernat
gaborbernat force-pushed the fix/async-cancel-threadlocal branch from 893d6db to c12e211 Compare October 16, 2026 19:57
@gaborbernat
gaborbernat force-pushed the fix/async-soft-rw-ownership branch from 1650d3e to a322d41 Compare October 16, 2026 20:24
@gaborbernat
gaborbernat force-pushed the fix/async-cancel-threadlocal branch 2 times, most recently from 8a4336a to 86e81b4 Compare October 16, 2026 20:47
@gaborbernat
gaborbernat force-pushed the fix/async-soft-rw-ownership branch from a322d41 to 5bee786 Compare October 17, 2026 03:08
@gaborbernat
gaborbernat force-pushed the fix/async-cancel-threadlocal branch from 86e81b4 to c7f5d55 Compare October 17, 2026 03:08
@gaborbernat
gaborbernat force-pushed the fix/async-soft-rw-ownership branch from 5bee786 to 013e940 Compare October 17, 2026 09:02
Base automatically changed from fix/async-soft-rw-ownership to main October 17, 2026 09:05
AsyncReadWriteLock and AsyncSoftReadWriteLock handed the whole contended
wait to one executor call and, when the caller was cancelled, waited that
call out before rolling it back. While another process held the lock, an
asyncio.timeout() or task.cancel() took effect only once that process let
go (with the default timeout=-1, up to SQLite's 23-day busy cap), and with
a finite lock timeout the cancellation was replaced by filelock's Timeout,
so a task cancelled at shutdown kept retrying.

The wrapper now tells the running backend call to stop. ReadWriteLock waits
for SQLite and for its in-process lock in short slices on the same
connection and gives up between them; SoftReadWriteLock checks between
polls. A wait that gave up this way ends in the caller's cancellation.
BaseAsyncFileLock created one transition gate per instance, while a
thread_local=True lock keeps separate lock state per thread. A thread whose
own context was unlocked took the gate for its whole poll loop, and the
thread holding the lock then queued on that same gate in release(), so each
waited for the other: with the default timeout both hung and the lock file
stayed held; with a finite timeout the waiter always ended in Timeout.

The gate now lives in the lock context, so a thread-local lock gets one per
thread and a shared lock keeps one per instance.
One sentence per idea in the sliced-wait, soft rollback and per-context
gate comments, and a changelog entry in the same style.
executescript commits a pending transaction under SQLite's legacy
transaction control, so the retry succeeded with or without the rollback
and the test could not tell. Record whether the retried statement starts
inside a transaction and expect it not to.
With other threads alive, a blocking acquire waits in 50 ms slices so a
fork can interleave. The cancel check only runs inside a single wait
longer than 100 ms, so no slice ever reached it and a canceled acquire
kept retrying until its timeout.

The retry loop now asks the cancel check after each timed-out slice and
gives up when it is true. The read-retry test mocks every connection
from the second on, so the in-place rollback path is still the one
exercised.
@gaborbernat
gaborbernat force-pushed the fix/async-cancel-threadlocal branch from c7f5d55 to fbc5893 Compare October 17, 2026 09:29
@gaborbernat
gaborbernat merged commit 86c6d5f into main Oct 17, 2026
46 checks passed
@gaborbernat
gaborbernat deleted the fix/async-cancel-threadlocal branch October 17, 2026 09:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant