Cost a builder a wrong conclusion today, and nearly a wrong PR body.
builder-air-106 ran the full packages/codev suite at 19:49 UTC and got 78 failures across 16 files — update/adopt, protocol-drift-audit, spawn-*, session-manager, consult lanes. It A/B'd with its own changes reverted, got the identical count and files, and reasonably concluded they were pre-existing.
They were not. At that same moment I was running the full suite on main from the workspace root, against the same ~/.agent-farm, the same Tower on 4100, and the same port range — and had just restarted Tower and removed 17 worktrees.
Measured afterwards:
| run |
result |
main, me alone |
6009 passed, 0 failed |
| builder worktree, me alone |
6013 passed, 0 failed |
| builder worktree, both at once |
78 failed / 16 files |
protocol-drift-audit.test.ts alone, in the builder worktree, during |
20 passed |
Why it matters more than a flake
The failure is stable and reproducible-looking — same count, same files, across an A/B. That is exactly the shape that reads as a real pre-existing breakage rather than interference, so the honest conclusion from inside one worktree is the wrong one. A builder cannot see that another agent is running a suite.
And the workspace is designed for concurrency: several builders plus architects, all able to run porch check (which runs the suite) at any time. This is not a rare collision.
Where the isolation stops
cluesmith#1515 isolated the agent-farm dir for spawned test Towers (createIsolatedAgentFarmDir, CODEV_AGENT_FARM_DIR). What is still shared:
- the test process's own
~/.agent-farm — vitest-e2e-setup.ts calls ensureLocalKey() against the real one on every patched fetch
- fixed ports (
TEST_TOWER_PORT = 14300 in tower-api.e2e.test.ts and friends) — two runs pick the same one
- the developer's live Tower on 4100, which some paths reach
Options
- Per-run
CODEV_AGENT_FARM_DIR for the whole vitest process, not only spawned children
- Port allocation from a per-run base (there is already a
findAvailablePort helper; the fixed constants bypass it)
- A coarse lockfile so a second full-suite run waits rather than interleaving — crude, but it makes the failure mode "slow" instead of "wrong"
Related: #6 was the same shared-~/.agent-farm root cause in a different disguise (the local-key race), and fixing it did not address the rest of the sharing.
Cost a builder a wrong conclusion today, and nearly a wrong PR body.
builder-air-106ran the fullpackages/codevsuite at 19:49 UTC and got 78 failures across 16 files —update/adopt,protocol-drift-audit,spawn-*,session-manager, consult lanes. It A/B'd with its own changes reverted, got the identical count and files, and reasonably concluded they were pre-existing.They were not. At that same moment I was running the full suite on
mainfrom the workspace root, against the same~/.agent-farm, the same Tower on 4100, and the same port range — and had just restarted Tower and removed 17 worktrees.Measured afterwards:
main, me aloneprotocol-drift-audit.test.tsalone, in the builder worktree, duringWhy it matters more than a flake
The failure is stable and reproducible-looking — same count, same files, across an A/B. That is exactly the shape that reads as a real pre-existing breakage rather than interference, so the honest conclusion from inside one worktree is the wrong one. A builder cannot see that another agent is running a suite.
And the workspace is designed for concurrency: several builders plus architects, all able to run
porch check(which runs the suite) at any time. This is not a rare collision.Where the isolation stops
cluesmith#1515 isolated the agent-farm dir for spawned test Towers (
createIsolatedAgentFarmDir,CODEV_AGENT_FARM_DIR). What is still shared:~/.agent-farm—vitest-e2e-setup.tscallsensureLocalKey()against the real one on every patched fetchTEST_TOWER_PORT = 14300intower-api.e2e.test.tsand friends) — two runs pick the same oneOptions
CODEV_AGENT_FARM_DIRfor the whole vitest process, not only spawned childrenfindAvailablePorthelper; the fixed constants bypass it)Related: #6 was the same shared-
~/.agent-farmroot cause in a different disguise (thelocal-keyrace), and fixing it did not address the rest of the sharing.