Skip to content

fix(server): spawns no longer stall a large server on Linux - #101

Merged
alexcstark merged 1 commit into
mainfrom
stark/spawn-helper
Nov 4, 2026
Merged

alexcstark merged 1 commit into
mainfrom
stark/spawn-helper

Conversation

@alexcstark

Copy link
Copy Markdown
Owner

On Linux, Node starts children with fork() (libuv 1.52 still does; posix_spawn is Apple-only), which copies the parent's page tables on the main thread. The cost grows with the server's memory. A server that had run 6 days and grown to 6.8 GB blocked the main thread 64–223 ms per spawn. With about 2 git spawns a second, that was 22% of wall-clock time, and HTTP/WebSocket handling froze for 2–11 s: image uploads sat at 0%, and requests took up to 11.5 s.

Evidence (lax6e, 2026-11-04)

  • perf: 73% of samples were the main thread in the kernel, mainly copy_present_pte (30%) and zap_pte_range (28%), reached through uv_spawn → _Fork → clone → dup_mmap.
  • bpftrace: 222 forks in 120 s (218 git), each blocking 64–223 ms, 26.7 s in total.
  • Spawn cost vs parent memory (Node on lax6e): 46 MB → 0.7 ms, 1.1 GB → 25 ms, 3.3 GB → 68 ms.

Fix

  • Children start in a small helper process (t3 __spawn-helper, a fast path in bin.ts that loads only Node built-ins), launched once at boot while the server is small.
  • The server sends spawn requests over IPC (advanced serialization). The helper relays each child's output back, and stdin writes the other way.
    • Bytes are relayed rather than pipe handles passed: a passed pipe loses whatever the helper already read, and its hand-over races a fast exit. The new concurrency test caught exactly that.
  • @effect/platform-node-shared gets a small patch that exposes the spawn call (layerWithSpawn), so Effect's ChildProcessSpawner and everything on it (ProcessRunner, git, provider CLIs) is reused unchanged. server.ts swaps it in, on Linux only.
  • Children get the server's current env/cwd, not the helper's startup snapshot.
  • The helper's heap is capped (NODE_OPTIONS), because V8 collects lazily and an uncapped helper drifts upward.
  • If the helper dies, its in-flight children are ended and reported, not left hanging. If it can't launch, spawns fall back to running directly.

Proof

  • SpawnHelper.test.ts (6, it.live, through the real bin.ts fast path):
    • children run under the helper (their parent PID isn't the server's);
    • output, stderr and exit codes come back, stdin writes through, and timeouts kill the child;
    • 40 concurrent children plus a 1 MB output all arrive intact;
    • a dead helper ends its children and a new helper starts;
    • a helper that can't launch falls back to direct spawns.
  • Each of these was made to fail against an injected defect: spawning locally, dropping the child on exit, skipping the in-flight cleanup, and removing the launch fallback. processRunner.test.ts 19/19.
  • Linux, real linux-x64 single executable from this branch, parent holding 3.1 GB: main-thread time per spawn 63.12 ms direct → 0.06 ms through the helper; worst event-loop stall 82 ms → 11 ms. The helper held 136 MB of anonymous memory (1.6 MB of page tables) and plateaued at about 213 MB RSS over 4,000 spawns.
  • tsc-rs passes for apps/server; vp fmt --check and vp lint are clean on the changed files. The lockfile is checked with pnpm install --frozen-lockfile.

Not covered: spawns that bypass ChildProcessSpawner still fork the server. That's the Claude SDK's CLI launch, node-pty for opening terminals, and some CLI-only commands, together about 4 of 222 measured spawns. macOS and Windows behave as before. Related: #99 (Cursor agent-store memory growth) and #100 (shared in-flight git reads).

🤖 Generated with Claude Code

https://claude.ai/code/session_01StkPGjC6GMoMkcwbiPzuUv

On Linux, Node spawns children with fork(), which copies the parent's page
tables on the main thread. The cost grows with the server's memory (about
21 ms per GB, measured), so a long-running server that had grown to 6.8 GB
spent 64-223 ms blocked per spawn. With ~2 git spawns a second, that was
22% of wall-clock time and froze HTTP and WebSocket handling for 2-11 s,
which is why uploads and requests stalled.

Children now start in a small helper process launched at boot. The server
sends spawn requests over IPC and the helper relays each child's output back,
so spawn cost no longer depends on the server's size. A small patch to
@effect/platform-node-shared exposes the spawn call so the existing
ChildProcessSpawner is reused unchanged. The helper's heap is capped so it
stays small, a dead helper ends its in-flight children instead of hanging
callers, and a helper that cannot launch falls back to spawning directly.
macOS (posix_spawn) and Windows are unchanged.

Claude-Session: https://claude.ai/code/session_01StkPGjC6GMoMkcwbiPzuUv
@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:L labels Nov 4, 2026
@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

ℹ️ The exact PR base did not have a successful artifact. Baseline uses the latest successful main measurement shown below.

Provider Metric Main baseline This PR Impact PR ceiling
Codex Total thread wire 5.0 KiB 5.0 KiB 0 B (0.0%) 6.8 KiB ✅
Codex Thread snapshot wire 3.8 KiB 3.8 KiB 0 B (0.0%) 4.9 KiB ✅
Codex Live turn WebSocket wire 1.2 KiB 1.2 KiB 0 B (0.0%) 2.0 KiB ✅
Codex Live turn WebSocket decoded 20.9 KiB 20.9 KiB 0 B (0.0%) 29.3 KiB ✅
Codex Live turn messages 2 2 0 (0.0%) 8 ✅
Claude Total thread wire 5.0 KiB 5.0 KiB 0 B (0.0%) 6.8 KiB ✅
Claude Thread snapshot wire 3.8 KiB 3.8 KiB 0 B (0.0%) 4.9 KiB ✅
Claude Live turn WebSocket wire 1.2 KiB 1.2 KiB 0 B (0.0%) 2.0 KiB ✅
Claude Live turn WebSocket decoded 21.2 KiB 21.2 KiB 0 B (0.0%) 29.3 KiB ✅
Claude Live turn messages 2 2 0 (0.0%) 8 ✅

Baseline: 99c96b0 · PR result: 54f74c8 · Source CI: failure

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

  • Codex decoded thread snapshot: 108.5 KiB
  • Claude decoded thread snapshot: 108.8 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

@alexcstark
alexcstark merged commit 606b4ff into main Nov 4, 2026
36 of 42 checks passed
@alexcstark
alexcstark deleted the stark/spawn-helper branch November 4, 2026 18:38
alexcstark added a commit that referenced this pull request Nov 4, 2026
On Linux, Node spawns children with fork(), which copies the parent's page
tables on the main thread. The cost grows with the server's memory (about
21 ms per GB, measured), so a long-running server that had grown to 6.8 GB
spent 64-223 ms blocked per spawn. With ~2 git spawns a second, that was
22% of wall-clock time and froze HTTP and WebSocket handling for 2-11 s,
which is why uploads and requests stalled.

Children now start in a small helper process launched at boot. The server
sends spawn requests over IPC and the helper relays each child's output back,
so spawn cost no longer depends on the server's size. A small patch to
@effect/platform-node-shared exposes the spawn call so the existing
ChildProcessSpawner is reused unchanged. The helper's heap is capped so it
stays small, a dead helper ends its in-flight children instead of hanging
callers, and a helper that cannot launch falls back to spawning directly.
macOS (posix_spawn) and Windows are unchanged.

Claude-Session: https://claude.ai/code/session_01StkPGjC6GMoMkcwbiPzuUv
alexcstark added a commit that referenced this pull request Nov 4, 2026
On Linux, Node spawns children with fork(), which copies the parent's page
tables on the main thread. The cost grows with the server's memory (about
21 ms per GB, measured), so a long-running server that had grown to 6.8 GB
spent 64-223 ms blocked per spawn. With ~2 git spawns a second, that was
22% of wall-clock time and froze HTTP and WebSocket handling for 2-11 s,
which is why uploads and requests stalled.

Children now start in a small helper process launched at boot. The server
sends spawn requests over IPC and the helper relays each child's output back,
so spawn cost no longer depends on the server's size. A small patch to
@effect/platform-node-shared exposes the spawn call so the existing
ChildProcessSpawner is reused unchanged. The helper's heap is capped so it
stays small, a dead helper ends its in-flight children instead of hanging
callers, and a helper that cannot launch falls back to spawning directly.
macOS (posix_spawn) and Windows are unchanged.

Claude-Session: https://claude.ai/code/session_01StkPGjC6GMoMkcwbiPzuUv
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:L vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant