Repository navigation
Add a bench for main-process lag under agent load (#1661) - #1683
Conversation
Runs N fake stream-json claude agents through the real runner, Store and IPC against a /tmp copy of the store, and reports main event-loop delay, main CPU and renderer-timed threads.list round-trip per level. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f073b7fb3f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const id = bigThreads.length | ||
| ? bigThreads[i].id | ||
| : (await js(`window.coder.threads.create(${JSON.stringify({ projectId: project.id, title: `bench ${n}/${i}` })})`)).id; |
There was a problem hiding this comment.
When BENCH_BIG_THREADS is unset and the copied store has defaultProvider set to Codex, Grok, or another provider, threads.create inherits that provider while only CODER_CLAUDE_BIN has been redirected. The benchmark can therefore launch a real installed CLI—potentially consuming paid usage—and measure a completely different adapter instead of the advertised fake-Claude workload. Pass provider: "claude" when creating every benchmark thread.
Useful? React with 👍 / 👎.
| threadIds.push(id); | ||
| await js(`window.coder.runs.start(${JSON.stringify({ threadId: id, prompt: "bench" })})`); |
There was a problem hiding this comment.
With a normal copied store, maxConcurrentRuns defaults to 4, and runs.start returns a queued result for starts beyond that cap. Consequently the default 5-, 10-, and 20-agent rows run only four streaming processes during the sample, so the reported scaling data cannot support conclusions about those agent counts. Clear the cap in the cloned store or verify that all requested thread IDs are active before starting each measurement.
Useful? React with 👍 / 👎.
Adds
scripts/perf/agent-churn-bench.js. It is the measurement for step 1 of #1661, which closed #1661: main stays responsive under agent load, so runner and Store stay in the main process.The bench uses the same wiring as
electron/smoke.js: real Store, runner,registerIpc, and a sandboxed window onpreload.js. It does not bootmain.js. It works on a/tmpclone of a store copy and refuses any other path. It starts N fake claude CLIs that stream stream-json partials throughrunner-provider-claude.js, then reports, for each agent count:threads.listround-trip, timed in the rendererBENCH_BIG_THREADS=1streams into the largest transcripts in the clone (the worst case). It first moves those threads onto a scratch git repo, so no run touches a real project.Results against the live store (1952 threads): p99 loop lag stays at 4–6 ms from 0 to 20 agents. The full tables are on #1661. This is a manual tool like
store-stream-bench.js; no test or CI job runs it.🤖 Generated with Claude Code