Skip to content

fix: prevent threadpool starvation with concurrent file locks - #32789

Merged
bartlomieju merged 7 commits into
mainfrom
fix/flock-threadpool-starvation
Aug 24, 2026
Merged

bartlomieju merged 7 commits into
mainfrom
fix/flock-threadpool-starvation

Conversation

@bartlomieju

@bartlomieju bartlomieju commented Mar 17, 2026 •

Copy link
Copy Markdown
Member

Summary

Fixes #22504.

lock_async used spawn_blocking to call the blocking flock() syscall, which parks a tokio blocking-threadpool thread for the entire duration of the lock wait. With the blocking pool capped at a couple dozen threads, acquiring 33+ concurrent locks exhausts all threads and deadlocks any other spawn_blocking work (like readTextFile/writeTextFile), which in turn may be what is needed to release the locks.

The fix

Keep using the blocking flock() syscall (so the OS still schedules lock acquisition fairly and there are no spurious try_lock errors), but bound how much of the shared threadpool lock waits can consume:

  • lock_async runs the blocking flock() via a new helper that caps how many concurrent lock waits may sit on the blocking threadpool at once (currently 8).
  • Any waits beyond that cap run on dedicated threads, which don't compete with other blocking fs operations.

This preserves flock() fairness while guaranteeing the blocking pool can never be fully consumed by pending locks.

Notes on earlier review feedback

An earlier revision used a non-blocking try_lock poll loop. That was reworked because:

  • try_lock is not fair, so a waiter could starve for a long time under contention.
  • On Windows, try_lock (LOCKFILE_FAIL_IMMEDIATELY) on a contended file returns ERROR_ACCESS_DENIED, which std maps to TryLockError::Error rather than WouldBlock, turning normal contention into a user-facing rejection (and failing the spec test on the Windows shards).

Going back to blocking flock() addresses both.

Test plan

  • New spec test: 50 concurrent lock acquisitions (well above the threadpool size) with interleaved file reads/writes, completes without deadlock and produces a correct serialized counter.

Fixes #22504.

Replace the blocking `flock()` implementation in `lock_async` with a
non-blocking poll loop using `try_lock` (LOCK_NB) + async sleep with
exponential backoff. The old implementation used `spawn_blocking` which
parked a tokio threadpool thread for the entire duration of the lock
wait. With the default pool size of 32, acquiring 33+ concurrent locks
would exhaust all threads and deadlock any other blocking file operations.

Also changed `try_lock_async` and `unlock_async` to skip
`spawn_blocking` since both operations are non-blocking (they return
immediately).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Comment thread ext/io/lib.rs Outdated
file.try_lock()
} else {
file.lock_shared()?;
file.try_lock_shared()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using try_lock is probably not fair and it's possible that something could wait on a lock for a very long time. Maybe there should be a limit to how many thread pool threads are used and after it starts spawning threads?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. I reverted to the blocking flock() syscall, which keeps the OS-level fairness, and instead bound the threadpool exposure directly: lock_async now caps how many concurrent lock waits may sit on the shared blocking threadpool (currently 8) and runs any further waits on dedicated threads, so a burst of concurrent locks can no longer starve other blocking fs ops (#22504) and no waiter has to busy-poll.

fibibot
fibibot previously requested changes May 7, 2026

@fibibot fibibot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Real failure mode: the test added by this PR — specs::fs::flock_no_starvation::many_concurrent_locks — fails on both Windows shards (debug windows-aarch64 and debug windows-x86_64) with:

error: Uncaught (in promise) PermissionDenied: Access is denied. (os error 5)
  await lockFile.lock(true);

The runner's flake-retry path didn't help: was flaky on run 0 → was flaky on run 1 → reduced concurrency to 1 → still failed. The test then fails the spec runner with 1 failed of 1132.

Root cause

The new poll loop at ext/io/lib.rs:1052-1053 only treats TryLockError::WouldBlock as "try again" and forwards every other error variant straight to the caller:

match result {
  Ok(()) => Ok(true),
  Err(TryLockError::WouldBlock) => Ok(false),
  Err(TryLockError::Error(err)) => Err(err.into()),  // <-- escapes to user
}

On Windows, LockFileEx(LOCKFILE_FAIL_IMMEDIATELY | LOCKFILE_EXCLUSIVE_LOCK) against a file already opened with a sharing mode that conflicts with the requested exclusive lock can surface as ERROR_ACCESS_DENIED (os error 5), which std::fs::File::try_lock then maps to TryLockError::Error(io::Error) rather than TryLockError::WouldBlock. The poll loop sees a non-WouldBlock error, returns it, and the user-facing lock() call rejects with PermissionDenied. So with 50 concurrent Deno.open + lock(true) chains, exactly the contention scenario the test was designed to exercise, Windows can't take the new path at all.

The old blocking flock/LockFileEx (without LOCKFILE_FAIL_IMMEDIATELY) waited at the kernel level and didn't surface this error variant — which is why the test passes pre-PR on Windows.

Fix direction

The poll loop needs to also treat the Windows "sharing/lock conflict" io::Error variants as transient. Concretely, in the Err(TryLockError::Error(err)) arm, inspect err.raw_os_error() and treat Some(ERROR_LOCK_VIOLATION /* 33 */) and Some(ERROR_SHARING_VIOLATION /* 32 */) and Some(ERROR_ACCESS_DENIED /* 5 */) as Ok(false) (continue polling) on cfg(windows). (POSIX paths shouldn't take the Err(Error) branch for held-lock contention, so the special-case is Windows-only.)

Alternatively: only fall back to the poll loop when try_lock returns WouldBlock, and otherwise (where try_lock itself errors) fall back to with_inner_blocking_task for the original blocking path. That keeps the threadpool-starvation fix where it matters (Linux/macOS, where flock blocks the threadpool indefinitely) without changing semantics on Windows where the behavior already diverges.

See also @dsherret's existing thread on ext/io/lib.rs:1048 about fairness — the poll-with-backoff scheme can theoretically starve a particular waiter in high contention. That concern stands regardless of the Windows fix and is worth addressing in the same pass (e.g., a bounded fallback to a tokio::sync::Mutex queue, or his suggestion of allowing some bounded number of threadpool slots before the poll fallback).

CI summary

95 success, 3 fail (ci status rollup + the 2 Windows shards above), 2 skip. The Windows fail is exactly the test this PR introduces.

Comment thread ext/io/lib.rs Outdated
match result {
Ok(()) => Ok(true),
Err(TryLockError::WouldBlock) => Ok(false),
Err(TryLockError::Error(err)) => Err(err.into()),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This branch turns every non-WouldBlock error into a user-facing rejection, which is what's making the new spec test fail on both Windows shards: LockFileEx(LOCKFILE_FAIL_IMMEDIATELY | LOCKFILE_EXCLUSIVE_LOCK) on a contended file can return ERROR_ACCESS_DENIED (os error 5), and std::fs::File::try_lock maps that to TryLockError::Error(io::Error) rather than WouldBlock.

Need to either (a) special-case Windows os_error 5 / 32 / 33 here as Ok(false) and keep polling, or (b) only take this poll loop when WouldBlock is observed and fall back to with_inner_blocking_task (the original blocking path) when try_lock errors. (a) keeps the threadpool-starvation fix unchanged on Linux/macOS and only adds Windows tolerance; (b) keeps the old behavior verbatim on Windows.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed by dropping the try_lock poll loop entirely. lock_async now uses the blocking flock()/LockFileEx path again (no LOCKFILE_FAIL_IMMEDIATELY), so the ERROR_ACCESS_DENIED -> TryLockError::Error mismapping no longer occurs and the spec test should pass on the Windows shards.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up: there was a second, separate Windows cause for the spec test failure. The test opened the lock file with append: true, and on Windows LockFileEx requires GENERIC_READ/GENERIC_WRITE — an append-only handle is rejected with ERROR_ACCESS_DENIED (os error 5) regardless of blocking vs. FAIL_IMMEDIATELY. Fixed by opening the lock file with read+write. (Unix flock() ignores the access mode, which is why it only showed up on the Windows shards.)

bartlomieju and others added 2 commits June 8, 2026 12:28
Address review feedback on the poll-loop approach:

- The try_lock poll loop dropped flock()'s OS-level fairness, so a waiter
  could starve for a long time under contention (dsherret).
- On Windows, try_lock (LOCKFILE_FAIL_IMMEDIATELY) on a contended file
  returns ERROR_ACCESS_DENIED, which std maps to TryLockError::Error
  rather than WouldBlock, turning contention into a user-facing rejection
  and failing the new spec test on the Windows shards.

Instead, go back to the blocking flock() syscall (fair, and no spurious
Windows errors) but avoid threadpool starvation: cap how many concurrent
lock waits may park on the shared blocking threadpool, and run any
further waits on dedicated threads that don't compete with other
blocking fs operations.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The flock_no_starvation spec test opened the lock file with append-only
access. On Windows, LockFileEx requires GENERIC_READ/GENERIC_WRITE, so
locking an append-only handle fails with ERROR_ACCESS_DENIED (os error
5). Open with read+write so the test exercises locking on all platforms.
@bartlomieju

Copy link
Copy Markdown
Member Author

A few correctness concerns from a review pass, in rough priority order:

1. The fixed cap of 8 can re-introduce starvation on small (Windows) pools

We set max_blocking_threads to 4 * available_parallelism() on Windows (runtime/tokio_util.rs:46). On a 2-core runner that's a pool of 8, equal to MAX_THREADPOOL_LOCK_WAITS = 8 — so all blocking threads can be consumed by lock waiters, leaving zero for the lock holder's readTextFile/writeTextFile, which is the exact deadlock this PR targets. Even at 4 cores (pool 16) the headroom is only 8.

The cap really needs to be derived from, and stay comfortably below, the actual pool size (e.g. max_blocking_threads / 4) rather than a constant that happens to equal the smallest realistic pool. On Unix (pool 32) the current value is fine; the risk is low-core Windows. The new spec test won't catch this since it only exercises the 32-thread Unix path.

2. std::thread::Builder::spawn(...).unwrap() panics under thread exhaustion

The dedicated-thread fallback is unbounded, and thread creation can fail with EAGAIN precisely in the high-concurrency regime this fix invites. .unwrap() there panics the runtime. Prefer falling back to spawn_blocking (or surfacing an FsError) on spawn failure.

3. unlock_async / try_lock_async no longer go through cell_async_task_queue

Previously all three async ops were serialized through the queue, so even un-awaited calls were ordered. Now that these run synchronously, f.lock(); f.unlock(); (without awaiting between) can run unlock before lock finishes, leaving the file locked. It's misuse to not await, but it's an observable ordering change worth calling out in the PR description.

Happy to push a patch deriving the cap from max_blocking_threads and handling the spawn failure if that's useful.

@bartlomieju bartlomieju left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The approach is sound: keeping the blocking flock() preserves OS-level fairness, and the rationale for abandoning the try_lock poll loop (unfairness + the Windows ERROR_ACCESS_DENIED mapping) is well-founded. The drop-guard slot accounting handles cancellation correctly, and the CAS loop with Relaxed ordering is fine for a pure admission counter.

A few notes that didn't fit a specific line:

  • Pre-existing, no action needed: if a lock_async future is cancelled mid-wait, the dedicated thread keeps blocking and the lock, once acquired, persists on the shared open file description (closing the clone doesn't release a flock). This exactly matches the old spawn_blocking behavior, so it's not a regression — just noting it survives the rewrite.
  • The test comment saying "tokio blocking threadpool (32 threads)" is accurate for non-Windows; on Windows the pool is 4 * cores (runtime/tokio_util.rs), so the "33+" framing is platform-specific. Minor wording nuance only.
  • No permission-model or security implications; the change only affects scheduling of already-authorized fs operations. Making try_lock_async/unlock_async synchronous also removes two spawn_blocking round-trips per call, which is a small win.

Main asks before merging: (1) deduplicate the new helper against with_inner_blocking_task, (2) make the dedicated-thread spawn non-panicking and give it a small stack, (3) drop the unneeded --unstable-fs and .clone(), and (4) mention the try_lock_async/unlock_async serialization change in the PR description.

Comment thread ext/io/lib.rs Outdated
// We keep using the blocking flock() syscall, which the OS schedules fairly,
// but cap how many lock waits may sit on the threadpool at once and run any
// further waits on dedicated threads that don't compete with other fs work.
fn with_inner_lock_task<F, R: 'static + Send>(

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This copies ~30 lines of the clone/take/put-back + task-queue logic from with_inner_blocking_task verbatim. Consider factoring the spawn strategy out (e.g. a shared inner helper that takes the "run this closure somewhere" step as a parameter) so the cell-management logic exists once. Divergence between the two copies is a real future-bug risk in this file.

Comment thread ext/io/lib.rs Outdated
use std::sync::atomic::AtomicUsize;
use std::sync::atomic::Ordering;

const MAX_THREADPOOL_LOCK_WAITS: usize = 8;

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

THREADPOOL_LOCK_WAITS is a process-wide static, but each web worker has its own tokio runtime and its own 32-thread blocking pool (runtime/tokio_util.rs). One worker's 8 waits push every other worker's lock waits onto dedicated threads even though their pools are idle. Not a correctness issue (the fallback path is safe), just needlessly conservative — fine to leave, but worth a comment acknowledging it.

Comment thread ext/io/lib.rs Outdated
(cell_value, result)
};
let (cell_value, result) = match reserve_threadpool_slot() {
Some(_slot) => spawn_blocking(task).await.unwrap(),

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The _slot binding is load-bearing: it keeps the drop guard alive across the .await, which is what makes the cap hold. Had this been Some(_), the slot would release immediately and the cap would be meaningless. Worth an explicit comment so a future refactor to _ doesn't silently break it.

Comment thread ext/io/lib.rs Outdated
Some(_slot) => spawn_blocking(task).await.unwrap(),
None => {
let (tx, rx) = tokio::sync::oneshot::channel();
std::thread::Builder::new()

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two things on the dedicated-thread path:

  1. .spawn(...).unwrap() panics on thread-creation failure (EAGAIN under fd/thread exhaustion — plausible in exactly the high-concurrency scenario this code serves). Better to fall back to spawn_blocking or surface an io::Error.
  2. Beyond the 8-slot cap, every additional waiter spawns an OS thread — 10,000 pending locks means ~10,000 threads at 2MB default stacks. Since these threads only sit in one syscall, adding .stack_size(64 * 1024) (or similar) bounds the memory cost cheaply.

Comment thread ext/io/lib.rs Outdated
.await
// try_lock with LOCK_NB is non-blocking (returns immediately),
// so no need to use spawn_blocking.
self.clone().try_lock_sync(exclusive)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: the .clone() is unnecessary — self is owned and try_lock_sync takes self: Rc<Self>.

Also worth noting (here and for unlock_async below): these no longer serialize behind cell_async_task_queue. If an async op is in flight with did_take == true (only when try_clone fails — rare), with_sync now returns FileBusy instead of waiting, and unlock() can now run while a lock() on the same file is still pending rather than queueing behind it. Both are arguably improvements (unlock queueing behind a pending lock wait was itself deadlock-shaped), but the PR description should mention the behavior change so it's weighed deliberately.

"many_concurrent_locks": {
// Regression test for #22504: acquiring more than 32 file locks
// should not prevent other file operations from completing.
"args": "run --allow-read --allow-write --unstable-fs main.ts",

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

--unstable-fs appears unnecessary — FsFile.lock/unlock are in the stable lib.deno.ns.d.ts. Drop the flag unless something else in the test needs it.

const finalCount = +(await Deno.readTextFile(COUNTER_FILE));
console.log(`final count: ${finalCount}`);

// Clean up

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This cleanup is redundant under "tempDir": true — the whole directory is discarded. Harmless, but can go.

@bartlomieju bartlomieju changed the title fix(fs): prevent threadpool starvation with concurrent file locks fix: prevent threadpool starvation with concurrent file locks Aug 12, 2026
Addresses review feedback:

- Lock waits no longer use the tokio blocking threadpool at all, so the cap
  cannot be misconfigured. `lock_async` first attempts the non-blocking
  `try_lock` inline (a single fast syscall, and the common uncontended case),
  and only falls back to the blocking `flock()`/`LockFileEx` syscall on a
  dedicated thread when the lock is actually contended. This also fixes the
  case where the previous fixed cap of 8 equalled the whole blocking pool on a
  2-core Windows machine (`4 * available_parallelism()`), where the original
  deadlock survived.
- The dedicated thread is no longer spawned with `.unwrap()`. On spawn failure
  it falls back to `spawn_blocking` instead of panicking, which previously
  aborted the process with a misleading "Deno could not start" message. Its
  stack is also capped at 256KB, since it only sits in one syscall.
- `try_lock_async` and `unlock_async` go back through `cell_async_task_queue`.
  Running them inline let them overtake an in-flight `lock_async` on the same
  file, so `await file.unlock()` could resolve while leaving the file locked.
  Covered by a new spec test.
- Deduplicated the clone/take/put-back cell logic shared with
  `with_inner_blocking_task`.
- Dropped the unnecessary `--unstable-fs` flag and redundant cleanup from the
  spec test.
When thread creation fails with EAGAIN, retry with backoff (~400ms total)
before falling back to the blocking threadpool. Such exhaustion is usually
transient, and waiting here costs nothing the caller is not already waiting
for, so this keeps the pool free for the work that will relieve the pressure.
The threadpool fallback remains as a last resort so a lock can never fail
outright.

The action is now handed to the thread over a channel once the spawn is known
to have succeeded, so a failed spawn leaves it owned by the caller and
available to retry with.
@bartlomieju
bartlomieju merged commit cfbf621 into main Aug 24, 2026
136 checks passed
@bartlomieju
bartlomieju deleted the fix/flock-threadpool-starvation branch August 24, 2026 09:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Acquiring more than 32 file locks prevents any other file operations

3 participants