Skip to content

feat(server): admit new turns only while the host has headroom - #20

Merged
jappyjan merged 11 commits into
feat/multi-repo-projectsfrom
feat/host-capacity
Nov 8, 2026
Merged

jappyjan merged 11 commits into
feat/multi-repo-projectsfrom
feat/host-capacity

Conversation

@jappyjan

@jappyjan jappyjan commented Nov 8, 2026 •

Copy link
Copy Markdown
Owner

Many parallel chats, scheduled runs, delegated tasks and pipeline steps can overload one machine. This adds server-side turn admission per environment, with two opt-in controls (both off by default): Pace new turns by host load (hold turns at severe memory pressure, one per 30 s while elevated or CPU is busy) and Limit running turns (1–64). Waiting turns stay queued with an admissionWait reason and start oldest first through a cross-thread pump. Running turns are never paused. Steer, interrupt and stop are never gated. "Start now" bypasses admission; MCP has no bypass. A turn blocked in t3_thread_wait, or in delegate_task/task_status with mode: "wait", frees its slot.

Stacked on feat/multi-repo-projects. The first commit is the port of pingdotgg#27253 (memory pressure), kept unchanged so it can be dropped once upstream merges it. Docs: docs/user/host-capacity.md, docs/internals/host-admission.md.

Known: ServerEnvironment.test.ts (macOS path), Playwright browser tests (no Chromium), and an OrchestratorMcpService model-manifest test ("gpt-5.5 not advertised") fail locally and are outside this change.

QA findings fixed in this PR

  • The sidebar rows showed "Working" (pulsing) for turns waiting on admission; only the command palette showed "Queued". Rows now show a static "Queued" and re-render only on their own wait.
  • A run left waiting when the server restarted never started. The first pump now runs after recovery.
  • Cancelling a waiting run kept admissionWait; it is now cleared.
  • The sidebar pill popover grew without bound; it now scrolls and truncates long titles.
  • Fixed two typecheck errors in the host admission tests.

Review findings fixed in this PR

Seven parallel reviews (correctness, requirements, architecture, security, performance, tests, UX) plus a re-review of the fixes.

  • Pump hot loop: each denial inside the pump requested another pump, so the server re-read the database continuously while any turn waited (about 3,000 passes a second in a test, 6 after the fix).
  • Restart and shutdown: the pump could start waiting turns while startup or shutdown recovery was reconciling, and recovery then cancelled or held them. The pump now stays off until recovery finishes and from the start of shutdown.
  • Oldest first: a thread's own next turn could take a freed slot from an older turn on another thread, and a busy thread could starve the rest. decide now counts older waiting threads, judging each by its oldest waiting turn. A thread held for another reason drops its wait marker so it cannot block newer turns.
  • Same-thread order: a second message on a thread with a waiting turn could start first. It now queues behind it.
  • Lost or stuck turns: a server message on a held queue got held forever instead of waiting for a slot; a forked thread's first message lost its fork context when it waited; a launch requeued at release ran after follow-ups sent while it was prepared.
  • Deadlock: delegate_task with mode: "wait" kept the parent's slot, so at the cap the parent and its child blocked each other until the wait timed out.
  • Performance: the snapshot queries scanned every run (now use the recovery index); a host CPU sample ran under the global admission lock.
  • Clients: web and mobile disagreed on "Queued" (now one shared check from the thread's own data, also used by pipeline status); settings showed for servers that cannot store them and wrote one host's limit to every selected environment; mobile offered a Start now the server refuses; the counts line went stale with both controls off; copy was duplicated (now in client-runtime); settings search entries added.
  • Tests: new cases for oldest-first by request time, concurrent sends for the last slot, same-thread order, held queues, launch order at release, a turn left waiting across a restart, and a thread judged by its oldest turn. The replay harness now always builds host admission, so existing orchestration tests cover the unlimited path. Each new case fails without its fix.

Open

  • task_status describes workState as working, waiting_for_children or result_available; it does not mention the new queued value. Changing that text needs the mcp-tool-use evals, and this machine has no API keys.
  • A delegated child task's record is created "running" before its turn starts, so its timeline row reads running while the child waits (its own thread shows Queued and MCP reports queued). This predates host capacity.
  • Follow-ups: keep hostAdmission out of the server config map so live counts don't recompute the sidebar; check the cap before loading a thread in the pump; trigger the pump on unarchive.
  • The client fixes after QA were checked by tests, lint and typecheck, not in a running client.

QA (before the review fixes): verified on web with the Replay provider (cap, Start now, Remove, oldest first, scheduled task, restart, light/dark, 375 px) and on an iOS simulator. Severe memory pressure and the memory warning were covered by unit tests only.

🤖 Generated with Claude Code

@github-actions github-actions Bot added size:XXL vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. labels Nov 8, 2026
juliusmarminge and others added 7 commits November 8, 2026 12:42
…use the most

Agents starved of memory freeze silently: in the incident the kernel throttled
the agents slice for hours and nothing told the user why every turn hung.

The server now reads memory pressure (Linux PSI for the host and the agents
slice, macOS's pressure level) and derives none/elevated/severe with
hysteresis. Clients that opt in get it on the config stream. While pressure is
severe, web shows a sidebar warning and mobile a banner on the thread list,
both naming the heaviest agents with links to their threads and Force stop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A server-side admission service decides when a queued or new run may start:
an optional hard cap on running turns and an optional pacing mode driven by
the host's memory pressure and CPU. Waiting runs stay queued with an
`admissionWait` reason and start oldest first through a cross-thread pump.
Running turns are never paused; steer and interrupt are never gated, and
"Start now" bypasses admission. A turn blocked in `t3_thread_wait` frees its
slot. Clients show the reason, live counts, a Queued status, and the two
settings; MCP results carry the wait reason.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The sidebar list resolves status with resolveSidebarThreadStatus, which mapped a queued run to Working, so only the command palette showed the static Queued label. Rows now read a per-thread wait flag, so they re-render only when their own wait starts or ends.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ait on cancel

Runs left waiting by the previous process had no trigger to start them, so they stayed queued with free capacity. The first pump now runs after recovery, because pumping earlier raced the reconcile that cancels unstarted runs. Cancelling a waiting run also clears admissionWait. Fixes two typecheck errors in the host admission tests.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The replay harness exported its own settings service for the host capacity
tests, so harnesses that provide their own settings (thread launch, MCP) read
the hidden copy and missed per-project provider changes. It now exports one
only when host capacity is requested.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@jappyjan
jappyjan marked this pull request as ready for review November 8, 2026 12:22
jappyjan and others added 4 commits November 8, 2026 13:29
…ttings follow the host

Review found web and mobile deciding "Queued" differently (web needed the
opt-in live list, mobile used the thread's own runtime), copy duplicated
across clients, settings shown for servers that cannot store them, and a
mobile Start now the server would refuse.

- One shared check (`threadRuntimeAwaitsTurn`) from the thread's own runtime,
  used by web rows, the command palette, mobile rows and pipeline status
  helpers, so pipeline parents no longer show Working for a queued step.
- Wait reasons, summaries and the default limit live in client-runtime and
  contracts.
- Host capacity settings appear only for one environment whose server reports
  the capability; the live counts show only while a control is on; search
  entries added.
- `canStartNow` per queued run, shared by web and mobile.
- Mobile labels a not-yet-started delegated task Queued; the sidebar pill
  truncates, names the environment and states the actual reason.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…serves the oldest turn first

Review found the admission pump re-requesting itself forever while any turn
waited (about 3,000 passes a second in a test), the pump starting turns that
startup or shutdown recovery then cancelled, and several ways for a newer turn
to overtake or stall older ones.

- A denial no longer requests another pump; run creation now triggers one.
- The pump stays off until startup recovery finishes and from the start of
  shutdown; reconcile events never trigger it.
- `decide` counts older waiting threads against the free slots, judging a
  thread by its oldest waiting turn. A thread held back for another reason
  (usage limit, mode ceiling, held queue) loses its wait marker so it cannot
  block newer turns.
- A message on a thread with a waiting turn queues behind it; a run queued only
  by admission no longer inherits a held queue's hold; fork transfers skip
  admission like merge-back; a requeued launch keeps its place ahead of
  follow-ups.
- `delegate_task` / `task_status` waits free the caller's slot like
  `t3_thread_wait`.
- The snapshot queries use the recovery index; the slot reservation is
  registered under the gate; CPU and pressure are read before it; the state
  dedupe resets when the last listener leaves.
- The replay harness always builds host admission, so existing orchestration
  tests run the unlimited path the server uses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The replay harness's settings export follows the option's rename to
hostAdmission, so harnesses that provide their own settings keep reading them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jappyjan
jappyjan force-pushed the feat/multi-repo-projects branch from 60f70df to df10fa6 Compare November 8, 2026 12:44
@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown

Thread transfer impact

⚠️ The latest CI run did not produce a thread transfer result for b1d9dde.

This comment will update automatically after the next completed run.

@jappyjan
jappyjan merged commit 7cffdf1 into feat/multi-repo-projects Nov 8, 2026
21 of 36 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XXL vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants