Repository navigation
fix: prevent threadpool starvation with concurrent file locks - #32789
Conversation
Fixes #22504. Replace the blocking `flock()` implementation in `lock_async` with a non-blocking poll loop using `try_lock` (LOCK_NB) + async sleep with exponential backoff. The old implementation used `spawn_blocking` which parked a tokio threadpool thread for the entire duration of the lock wait. With the default pool size of 32, acquiring 33+ concurrent locks would exhaust all threads and deadlock any other blocking file operations. Also changed `try_lock_async` and `unlock_async` to skip `spawn_blocking` since both operations are non-blocking (they return immediately). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
| file.try_lock() | ||
| } else { | ||
| file.lock_shared()?; | ||
| file.try_lock_shared() |
There was a problem hiding this comment.
Using try_lock is probably not fair and it's possible that something could wait on a lock for a very long time. Maybe there should be a limit to how many thread pool threads are used and after it starts spawning threads?
There was a problem hiding this comment.
Good point. I reverted to the blocking flock() syscall, which keeps the OS-level fairness, and instead bound the threadpool exposure directly: lock_async now caps how many concurrent lock waits may sit on the shared blocking threadpool (currently 8) and runs any further waits on dedicated threads, so a burst of concurrent locks can no longer starve other blocking fs ops (#22504) and no waiter has to busy-poll.
fibibot
left a comment
There was a problem hiding this comment.
Real failure mode: the test added by this PR — specs::fs::flock_no_starvation::many_concurrent_locks — fails on both Windows shards (debug windows-aarch64 and debug windows-x86_64) with:
error: Uncaught (in promise) PermissionDenied: Access is denied. (os error 5)
await lockFile.lock(true);
The runner's flake-retry path didn't help: was flaky on run 0 → was flaky on run 1 → reduced concurrency to 1 → still failed. The test then fails the spec runner with 1 failed of 1132.
Root cause
The new poll loop at ext/io/lib.rs:1052-1053 only treats TryLockError::WouldBlock as "try again" and forwards every other error variant straight to the caller:
match result {
Ok(()) => Ok(true),
Err(TryLockError::WouldBlock) => Ok(false),
Err(TryLockError::Error(err)) => Err(err.into()), // <-- escapes to user
}On Windows, LockFileEx(LOCKFILE_FAIL_IMMEDIATELY | LOCKFILE_EXCLUSIVE_LOCK) against a file already opened with a sharing mode that conflicts with the requested exclusive lock can surface as ERROR_ACCESS_DENIED (os error 5), which std::fs::File::try_lock then maps to TryLockError::Error(io::Error) rather than TryLockError::WouldBlock. The poll loop sees a non-WouldBlock error, returns it, and the user-facing lock() call rejects with PermissionDenied. So with 50 concurrent Deno.open + lock(true) chains, exactly the contention scenario the test was designed to exercise, Windows can't take the new path at all.
The old blocking flock/LockFileEx (without LOCKFILE_FAIL_IMMEDIATELY) waited at the kernel level and didn't surface this error variant — which is why the test passes pre-PR on Windows.
Fix direction
The poll loop needs to also treat the Windows "sharing/lock conflict" io::Error variants as transient. Concretely, in the Err(TryLockError::Error(err)) arm, inspect err.raw_os_error() and treat Some(ERROR_LOCK_VIOLATION /* 33 */) and Some(ERROR_SHARING_VIOLATION /* 32 */) and Some(ERROR_ACCESS_DENIED /* 5 */) as Ok(false) (continue polling) on cfg(windows). (POSIX paths shouldn't take the Err(Error) branch for held-lock contention, so the special-case is Windows-only.)
Alternatively: only fall back to the poll loop when try_lock returns WouldBlock, and otherwise (where try_lock itself errors) fall back to with_inner_blocking_task for the original blocking path. That keeps the threadpool-starvation fix where it matters (Linux/macOS, where flock blocks the threadpool indefinitely) without changing semantics on Windows where the behavior already diverges.
See also @dsherret's existing thread on ext/io/lib.rs:1048 about fairness — the poll-with-backoff scheme can theoretically starve a particular waiter in high contention. That concern stands regardless of the Windows fix and is worth addressing in the same pass (e.g., a bounded fallback to a tokio::sync::Mutex queue, or his suggestion of allowing some bounded number of threadpool slots before the poll fallback).
CI summary
95 success, 3 fail (ci status rollup + the 2 Windows shards above), 2 skip. The Windows fail is exactly the test this PR introduces.
| match result { | ||
| Ok(()) => Ok(true), | ||
| Err(TryLockError::WouldBlock) => Ok(false), | ||
| Err(TryLockError::Error(err)) => Err(err.into()), |
There was a problem hiding this comment.
This branch turns every non-WouldBlock error into a user-facing rejection, which is what's making the new spec test fail on both Windows shards: LockFileEx(LOCKFILE_FAIL_IMMEDIATELY | LOCKFILE_EXCLUSIVE_LOCK) on a contended file can return ERROR_ACCESS_DENIED (os error 5), and std::fs::File::try_lock maps that to TryLockError::Error(io::Error) rather than WouldBlock.
Need to either (a) special-case Windows os_error 5 / 32 / 33 here as Ok(false) and keep polling, or (b) only take this poll loop when WouldBlock is observed and fall back to with_inner_blocking_task (the original blocking path) when try_lock errors. (a) keeps the threadpool-starvation fix unchanged on Linux/macOS and only adds Windows tolerance; (b) keeps the old behavior verbatim on Windows.
There was a problem hiding this comment.
Fixed by dropping the try_lock poll loop entirely. lock_async now uses the blocking flock()/LockFileEx path again (no LOCKFILE_FAIL_IMMEDIATELY), so the ERROR_ACCESS_DENIED -> TryLockError::Error mismapping no longer occurs and the spec test should pass on the Windows shards.
There was a problem hiding this comment.
Follow-up: there was a second, separate Windows cause for the spec test failure. The test opened the lock file with append: true, and on Windows LockFileEx requires GENERIC_READ/GENERIC_WRITE — an append-only handle is rejected with ERROR_ACCESS_DENIED (os error 5) regardless of blocking vs. FAIL_IMMEDIATELY. Fixed by opening the lock file with read+write. (Unix flock() ignores the access mode, which is why it only showed up on the Windows shards.)
Address review feedback on the poll-loop approach: - The try_lock poll loop dropped flock()'s OS-level fairness, so a waiter could starve for a long time under contention (dsherret). - On Windows, try_lock (LOCKFILE_FAIL_IMMEDIATELY) on a contended file returns ERROR_ACCESS_DENIED, which std maps to TryLockError::Error rather than WouldBlock, turning contention into a user-facing rejection and failing the new spec test on the Windows shards. Instead, go back to the blocking flock() syscall (fair, and no spurious Windows errors) but avoid threadpool starvation: cap how many concurrent lock waits may park on the shared blocking threadpool, and run any further waits on dedicated threads that don't compete with other blocking fs operations. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The flock_no_starvation spec test opened the lock file with append-only access. On Windows, LockFileEx requires GENERIC_READ/GENERIC_WRITE, so locking an append-only handle fails with ERROR_ACCESS_DENIED (os error 5). Open with read+write so the test exercises locking on all platforms.
|
A few correctness concerns from a review pass, in rough priority order: 1. The fixed cap of 8 can re-introduce starvation on small (Windows) poolsWe set The cap really needs to be derived from, and stay comfortably below, the actual pool size (e.g. 2.
|
bartlomieju
left a comment
There was a problem hiding this comment.
The approach is sound: keeping the blocking flock() preserves OS-level fairness, and the rationale for abandoning the try_lock poll loop (unfairness + the Windows ERROR_ACCESS_DENIED mapping) is well-founded. The drop-guard slot accounting handles cancellation correctly, and the CAS loop with Relaxed ordering is fine for a pure admission counter.
A few notes that didn't fit a specific line:
- Pre-existing, no action needed: if a
lock_asyncfuture is cancelled mid-wait, the dedicated thread keeps blocking and the lock, once acquired, persists on the shared open file description (closing the clone doesn't release a flock). This exactly matches the oldspawn_blockingbehavior, so it's not a regression — just noting it survives the rewrite. - The test comment saying "tokio blocking threadpool (32 threads)" is accurate for non-Windows; on Windows the pool is
4 * cores(runtime/tokio_util.rs), so the "33+" framing is platform-specific. Minor wording nuance only. - No permission-model or security implications; the change only affects scheduling of already-authorized fs operations. Making
try_lock_async/unlock_asyncsynchronous also removes twospawn_blockinground-trips per call, which is a small win.
Main asks before merging: (1) deduplicate the new helper against with_inner_blocking_task, (2) make the dedicated-thread spawn non-panicking and give it a small stack, (3) drop the unneeded --unstable-fs and .clone(), and (4) mention the try_lock_async/unlock_async serialization change in the PR description.
| // We keep using the blocking flock() syscall, which the OS schedules fairly, | ||
| // but cap how many lock waits may sit on the threadpool at once and run any | ||
| // further waits on dedicated threads that don't compete with other fs work. | ||
| fn with_inner_lock_task<F, R: 'static + Send>( |
There was a problem hiding this comment.
This copies ~30 lines of the clone/take/put-back + task-queue logic from with_inner_blocking_task verbatim. Consider factoring the spawn strategy out (e.g. a shared inner helper that takes the "run this closure somewhere" step as a parameter) so the cell-management logic exists once. Divergence between the two copies is a real future-bug risk in this file.
| use std::sync::atomic::AtomicUsize; | ||
| use std::sync::atomic::Ordering; | ||
|
|
||
| const MAX_THREADPOOL_LOCK_WAITS: usize = 8; |
There was a problem hiding this comment.
THREADPOOL_LOCK_WAITS is a process-wide static, but each web worker has its own tokio runtime and its own 32-thread blocking pool (runtime/tokio_util.rs). One worker's 8 waits push every other worker's lock waits onto dedicated threads even though their pools are idle. Not a correctness issue (the fallback path is safe), just needlessly conservative — fine to leave, but worth a comment acknowledging it.
| (cell_value, result) | ||
| }; | ||
| let (cell_value, result) = match reserve_threadpool_slot() { | ||
| Some(_slot) => spawn_blocking(task).await.unwrap(), |
There was a problem hiding this comment.
The _slot binding is load-bearing: it keeps the drop guard alive across the .await, which is what makes the cap hold. Had this been Some(_), the slot would release immediately and the cap would be meaningless. Worth an explicit comment so a future refactor to _ doesn't silently break it.
| Some(_slot) => spawn_blocking(task).await.unwrap(), | ||
| None => { | ||
| let (tx, rx) = tokio::sync::oneshot::channel(); | ||
| std::thread::Builder::new() |
There was a problem hiding this comment.
Two things on the dedicated-thread path:
.spawn(...).unwrap()panics on thread-creation failure (EAGAIN under fd/thread exhaustion — plausible in exactly the high-concurrency scenario this code serves). Better to fall back tospawn_blockingor surface anio::Error.- Beyond the 8-slot cap, every additional waiter spawns an OS thread — 10,000 pending locks means ~10,000 threads at 2MB default stacks. Since these threads only sit in one syscall, adding
.stack_size(64 * 1024)(or similar) bounds the memory cost cheaply.
| .await | ||
| // try_lock with LOCK_NB is non-blocking (returns immediately), | ||
| // so no need to use spawn_blocking. | ||
| self.clone().try_lock_sync(exclusive) |
There was a problem hiding this comment.
Nit: the .clone() is unnecessary — self is owned and try_lock_sync takes self: Rc<Self>.
Also worth noting (here and for unlock_async below): these no longer serialize behind cell_async_task_queue. If an async op is in flight with did_take == true (only when try_clone fails — rare), with_sync now returns FileBusy instead of waiting, and unlock() can now run while a lock() on the same file is still pending rather than queueing behind it. Both are arguably improvements (unlock queueing behind a pending lock wait was itself deadlock-shaped), but the PR description should mention the behavior change so it's weighed deliberately.
| "many_concurrent_locks": { | ||
| // Regression test for #22504: acquiring more than 32 file locks | ||
| // should not prevent other file operations from completing. | ||
| "args": "run --allow-read --allow-write --unstable-fs main.ts", |
There was a problem hiding this comment.
--unstable-fs appears unnecessary — FsFile.lock/unlock are in the stable lib.deno.ns.d.ts. Drop the flag unless something else in the test needs it.
| const finalCount = +(await Deno.readTextFile(COUNTER_FILE)); | ||
| console.log(`final count: ${finalCount}`); | ||
|
|
||
| // Clean up |
There was a problem hiding this comment.
This cleanup is redundant under "tempDir": true — the whole directory is discarded. Harmless, but can go.
Addresses review feedback: - Lock waits no longer use the tokio blocking threadpool at all, so the cap cannot be misconfigured. `lock_async` first attempts the non-blocking `try_lock` inline (a single fast syscall, and the common uncontended case), and only falls back to the blocking `flock()`/`LockFileEx` syscall on a dedicated thread when the lock is actually contended. This also fixes the case where the previous fixed cap of 8 equalled the whole blocking pool on a 2-core Windows machine (`4 * available_parallelism()`), where the original deadlock survived. - The dedicated thread is no longer spawned with `.unwrap()`. On spawn failure it falls back to `spawn_blocking` instead of panicking, which previously aborted the process with a misleading "Deno could not start" message. Its stack is also capped at 256KB, since it only sits in one syscall. - `try_lock_async` and `unlock_async` go back through `cell_async_task_queue`. Running them inline let them overtake an in-flight `lock_async` on the same file, so `await file.unlock()` could resolve while leaving the file locked. Covered by a new spec test. - Deduplicated the clone/take/put-back cell logic shared with `with_inner_blocking_task`. - Dropped the unnecessary `--unstable-fs` flag and redundant cleanup from the spec test.
When thread creation fails with EAGAIN, retry with backoff (~400ms total) before falling back to the blocking threadpool. Such exhaustion is usually transient, and waiting here costs nothing the caller is not already waiting for, so this keeps the pool free for the work that will relieve the pressure. The threadpool fallback remains as a last resort so a lock can never fail outright. The action is now handed to the thread over a channel once the spawn is known to have succeeded, so a failed spawn leaves it owned by the caller and available to retry with.
Summary
Fixes #22504.
lock_asyncusedspawn_blockingto call the blockingflock()syscall, which parks a tokio blocking-threadpool thread for the entire duration of the lock wait. With the blocking pool capped at a couple dozen threads, acquiring 33+ concurrent locks exhausts all threads and deadlocks any otherspawn_blockingwork (likereadTextFile/writeTextFile), which in turn may be what is needed to release the locks.The fix
Keep using the blocking
flock()syscall (so the OS still schedules lock acquisition fairly and there are no spurioustry_lockerrors), but bound how much of the shared threadpool lock waits can consume:lock_asyncruns the blockingflock()via a new helper that caps how many concurrent lock waits may sit on the blocking threadpool at once (currently 8).This preserves
flock()fairness while guaranteeing the blocking pool can never be fully consumed by pending locks.Notes on earlier review feedback
An earlier revision used a non-blocking
try_lockpoll loop. That was reworked because:try_lockis not fair, so a waiter could starve for a long time under contention.try_lock(LOCKFILE_FAIL_IMMEDIATELY) on a contended file returnsERROR_ACCESS_DENIED, whichstdmaps toTryLockError::Errorrather thanWouldBlock, turning normal contention into a user-facing rejection (and failing the spec test on the Windows shards).Going back to blocking
flock()addresses both.Test plan