Repository navigation
🐛 fix(async): honor cancellation and per-thread gates - #844
Merged
Merged
Conversation
gaborbernat
force-pushed
the
fix/async-soft-rw-ownership
branch
from
October 16, 2026 02:09
009d6a0 to
5cfca83
Compare
gaborbernat
force-pushed
the
fix/async-cancel-threadlocal
branch
from
October 16, 2026 02:09
ee91b85 to
113ccbb
Compare
gaborbernat
force-pushed
the
fix/async-soft-rw-ownership
branch
from
October 16, 2026 02:36
5cfca83 to
815ccec
Compare
gaborbernat
force-pushed
the
fix/async-cancel-threadlocal
branch
2 times, most recently
from
October 16, 2026 03:05
5904f33 to
f6a1290
Compare
gaborbernat
force-pushed
the
fix/async-soft-rw-ownership
branch
from
October 16, 2026 04:52
1b1f466 to
e3815b7
Compare
gaborbernat
force-pushed
the
fix/async-cancel-threadlocal
branch
from
October 16, 2026 04:52
f6a1290 to
bdf41da
Compare
gaborbernat
force-pushed
the
fix/async-soft-rw-ownership
branch
from
October 16, 2026 05:04
e3815b7 to
da8e556
Compare
gaborbernat
force-pushed
the
fix/async-cancel-threadlocal
branch
from
October 16, 2026 05:04
bdf41da to
a6568ca
Compare
gaborbernat
force-pushed
the
fix/async-soft-rw-ownership
branch
from
October 16, 2026 05:52
da8e556 to
5c82a91
Compare
gaborbernat
force-pushed
the
fix/async-cancel-threadlocal
branch
from
October 16, 2026 05:52
a6568ca to
cc7e5af
Compare
gaborbernat
force-pushed
the
fix/async-soft-rw-ownership
branch
from
October 16, 2026 06:18
5c82a91 to
4434e7e
Compare
gaborbernat
force-pushed
the
fix/async-cancel-threadlocal
branch
from
October 16, 2026 06:18
cc7e5af to
ad1c1d7
Compare
gaborbernat
force-pushed
the
fix/async-soft-rw-ownership
branch
from
October 16, 2026 08:21
4434e7e to
c2026e3
Compare
gaborbernat
force-pushed
the
fix/async-cancel-threadlocal
branch
2 times, most recently
from
October 16, 2026 08:23
17c8bf7 to
b7346cb
Compare
gaborbernat
force-pushed
the
fix/async-soft-rw-ownership
branch
from
October 16, 2026 08:34
c2026e3 to
8ad2b6c
Compare
gaborbernat
force-pushed
the
fix/async-cancel-threadlocal
branch
from
October 16, 2026 08:34
b7346cb to
c6de040
Compare
gaborbernat
force-pushed
the
fix/async-soft-rw-ownership
branch
from
October 16, 2026 13:43
8ad2b6c to
e538301
Compare
gaborbernat
force-pushed
the
fix/async-cancel-threadlocal
branch
from
October 16, 2026 13:43
c6de040 to
242352d
Compare
gaborbernat
force-pushed
the
fix/async-soft-rw-ownership
branch
from
October 16, 2026 13:46
e538301 to
bfd299c
Compare
gaborbernat
force-pushed
the
fix/async-cancel-threadlocal
branch
from
October 16, 2026 13:46
242352d to
1ad3357
Compare
gaborbernat
force-pushed
the
fix/async-soft-rw-ownership
branch
from
October 16, 2026 17:01
bfd299c to
76f9e66
Compare
gaborbernat
force-pushed
the
fix/async-cancel-threadlocal
branch
from
October 16, 2026 17:02
1ad3357 to
893d6db
Compare
gaborbernat
force-pushed
the
fix/async-soft-rw-ownership
branch
from
October 16, 2026 19:57
76f9e66 to
1650d3e
Compare
gaborbernat
force-pushed
the
fix/async-cancel-threadlocal
branch
from
October 16, 2026 19:57
893d6db to
c12e211
Compare
gaborbernat
force-pushed
the
fix/async-soft-rw-ownership
branch
from
October 16, 2026 20:24
1650d3e to
a322d41
Compare
gaborbernat
force-pushed
the
fix/async-cancel-threadlocal
branch
2 times, most recently
from
October 16, 2026 20:47
8a4336a to
86e81b4
Compare
gaborbernat
force-pushed
the
fix/async-soft-rw-ownership
branch
from
October 17, 2026 03:08
a322d41 to
5bee786
Compare
gaborbernat
force-pushed
the
fix/async-cancel-threadlocal
branch
from
October 17, 2026 03:08
86e81b4 to
c7f5d55
Compare
gaborbernat
force-pushed
the
fix/async-soft-rw-ownership
branch
from
October 17, 2026 09:02
5bee786 to
013e940
Compare
AsyncReadWriteLock and AsyncSoftReadWriteLock handed the whole contended wait to one executor call and, when the caller was cancelled, waited that call out before rolling it back. While another process held the lock, an asyncio.timeout() or task.cancel() took effect only once that process let go (with the default timeout=-1, up to SQLite's 23-day busy cap), and with a finite lock timeout the cancellation was replaced by filelock's Timeout, so a task cancelled at shutdown kept retrying. The wrapper now tells the running backend call to stop. ReadWriteLock waits for SQLite and for its in-process lock in short slices on the same connection and gives up between them; SoftReadWriteLock checks between polls. A wait that gave up this way ends in the caller's cancellation.
BaseAsyncFileLock created one transition gate per instance, while a thread_local=True lock keeps separate lock state per thread. A thread whose own context was unlocked took the gate for its whole poll loop, and the thread holding the lock then queued on that same gate in release(), so each waited for the other: with the default timeout both hung and the lock file stayed held; with a finite timeout the waiter always ended in Timeout. The gate now lives in the lock context, so a thread-local lock gets one per thread and a shared lock keeps one per instance.
One sentence per idea in the sliced-wait, soft rollback and per-context gate comments, and a changelog entry in the same style.
executescript commits a pending transaction under SQLite's legacy transaction control, so the retry succeeded with or without the rollback and the test could not tell. Record whether the retried statement starts inside a transaction and expect it not to.
With other threads alive, a blocking acquire waits in 50 ms slices so a fork can interleave. The cancel check only runs inside a single wait longer than 100 ms, so no slice ever reached it and a canceled acquire kept retrying until its timeout. The retry loop now asks the cancel check after each timed-out slice and gives up when it is true. The read-retry test mocks every connection from the second on, so the in-place rollback path is still the one exercised.
gaborbernat
force-pushed
the
fix/async-cancel-threadlocal
branch
from
October 17, 2026 09:29
c7f5d55 to
fbc5893
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two async locks ignored their callers while waiting. The async read-write wrappers ignored cancellation while another process held the lock. A thread-local
AsyncFileLockshared by threads with their own event loops deadlocked on its first contention. ⏳ This PR lets a cancelled acquire stop its backend wait and gives each thread-local lock context its own transition gate (the queue that orders one lock's acquire and release calls). It builds on #838, which records the wrappers' sync-lock hold under a task owner.Cancellation waited for the other process to let go
AsyncReadWriteLockandAsyncSoftReadWriteLockhanded the whole contended wait to one executor call. Since #652, copied to the soft wrapper in #686, aCancelledErrormade_run_acquiredrain that call before rolling it back. Nothing told the call to stop. For SQLite it was onebusy_timeoutwait of about 23 days attimeout=-1; for the soft lock an open-ended poll loop. Soasyncio.timeout()ortask.cancel(), which the how-to recommends for bounding the wait, took effect only when the holder released.asyncio.run()needed several Ctrl-C presses to exit. With a finite locktimeoutthe drained call ended inTimeout, which replaced the cancellation, so a background task cancelled at shutdown kept retrying.sequenceDiagram participant T as task participant W as executor worker participant S as SQLite / lock files participant P as another process P->>S: holds the write lock T->>W: acquire (one call) W->>S: wait T-->>T: asyncio.timeout(1) fires: CancelledError rect rgb(255, 214, 214) Note over T,S: before: drain the call. It ends when P releases, then Timeout or CancelledError end rect rgb(216, 243, 220) Note over T,S: after: set the abandon flag. The worker gives up at its next slice, the task sees CancelledError T->>W: abandoned.set() W->>S: busy_timeout slice ends, database is locked W-->>T: Timeout → re-raised as the cancellation endThe wrapper now passes an abandon flag into the sync-lock acquire and sets it on cancellation.
ReadWriteLockwaits for SQLite in bounded busy-timeout slices on the same connection, retryingBEGINwhile SQLite reports "database is locked". A read's deferredBEGINsucceeds before itsSELECTtakes the shared lock, so the retry rolls that back first._acquire_thread_lock()takes the lock's in-process lock in the same slices.SoftReadWriteLockchecks the flag between polls, so its existing rollback withdraws the published claim. A wait that gave up this way ends in aTimeoutthe wrapper re-raises as the caller's cancellation. The wrapper still rolls back an acquire that won the race meanwhile. Slicing inside the sync lock keeps one connection open for the wait; retrying the whole acquire from the wrapper would reopen it ten times a second. Synchronous callers keep the single uninterrupted wait.#840 splits a threaded process's wait into 50 ms attempts for the same
_acquire. Whichever of the two lands second has to passholderandcancel_checkinto each attempt and askcancel_check()between attempts, because a 50 ms attempt is shorter than the 100 ms slice and does not install the check on its own. The retry loop in_acquiretherefore also checks for cancellation after each attempt that times out, and gives up when the caller has abandoned the acquire.A thread-local async lock deadlocked between its threads
BaseAsyncFileLockcreated one transition gate per instance in #652.thread_local=Truegives each thread its own lock state throughthreading.local. A thread whose own context held nothing took the gate and kept it for its whole poll loop. The thread that held the lock then queued on the same gate insiderelease(). With the defaulttimeout=-1neither thread ever returned and the lock file stayed held. With a finite timeout every wait ended inTimeout, however early the holder finished. Any other process holding the lock file while both threads calledacquire()set the order up by itself. The syncBaseFileLockskips its gate for thread-local instances; the async side did not.sequenceDiagram participant A as thread 1 (holds the lock) participant G as gate participant B as thread 2 (own context, unlocked) rect rgb(255, 214, 214) Note over A,B: before: one gate per instance B->>G: acquire(): take the gate, poll while A holds A->>G: release(): queue behind B Note over A,B: each waits for the other end rect rgb(216, 243, 220) Note over A,B: after: one gate per context B->>B: acquire(): own gate, poll A->>A: release(): own gate, unlock at once B->>B: next poll takes the lock endThe gate moved into
AsyncFileLockContext.AsyncThreadLocalFileContextalready delegates every dataclass field to the calling thread's context, so a thread-local lock gets one gate per thread and a shared lock keeps one per instance. Skipping the gate for thread-local instances, as the sync lock does, would not do: several tasks of one thread still share that thread's context and need the serialization.Behavior changes
A cancelled
AsyncReadWriteLockorAsyncSoftReadWriteLockacquire that was waiting on another process now ends with the caller'sCancelledErrorwithin about 0.1 s (one poll interval for the soft lock). It used to end inTimeout, or in a cancellation delayed until the holder released. An async writer waiting for readers now holds SQLite'sPENDINGlock only within a slice and gives it up between slices. A reader in another process can therefore enter between two slices where it used to wait behind the writer.