Summary
test_sim appears to ignore a per-call projectPath and use the session default instead. When the two differ, it reported SUCCEEDED having run zero tests.
What happened
I had several agents working concurrently in separate git worktrees of one repo. Session defaults were set once, pointing projectPath at the main checkout:
session_set_defaults
projectPath: <repo>/src/PinPoint.xcodeproj
scheme: PinPoint
simulatorName: iPhone 17 Pro
An agent working in <repo>/.claude/worktrees/agent-XXXX/ called test_sim with its own worktree's projectPath and an -only-testing filter for a suite that exists only in that worktree. The call returned SUCCEEDED with 0 tests passed and 0 failed.
The suite it was asked to run did not exist in the project it actually built, so there was nothing to run and nothing to fail. The agent's only clue was the zero count; the status field said success.
Why this is worth fixing above its size
A test tool that returns SUCCEEDED without running the requested tests is the worst available failure mode — it is indistinguishable from a pass at the point where a caller decides whether work is done. In an agent workflow nothing reads the count; it reads the status. Two of my agents would have reported tickets complete on the strength of it. Both worked around it by driving xcodebuild directly with their own -derivedDataPath, which is the right answer but only because they noticed.
Suggested
Either honour a per-call projectPath over the session default, or fail loudly when one is passed and cannot be honoured. Failing that, a zero-test run should not report as SUCCEEDED — an -only-testing filter matching nothing is an error, not a pass.
Environment
Multiple concurrent git worktrees of one project, one shared MCP session, iOS simulator destination. Repointing the shared session defaults was not an option, since a concurrent agent was relying on them.
Summary
test_simappears to ignore a per-callprojectPathand use the session default instead. When the two differ, it reportedSUCCEEDEDhaving run zero tests.What happened
I had several agents working concurrently in separate git worktrees of one repo. Session defaults were set once, pointing
projectPathat the main checkout:An agent working in
<repo>/.claude/worktrees/agent-XXXX/calledtest_simwith its own worktree'sprojectPathand an-only-testingfilter for a suite that exists only in that worktree. The call returnedSUCCEEDEDwith 0 tests passed and 0 failed.The suite it was asked to run did not exist in the project it actually built, so there was nothing to run and nothing to fail. The agent's only clue was the zero count; the status field said success.
Why this is worth fixing above its size
A test tool that returns
SUCCEEDEDwithout running the requested tests is the worst available failure mode — it is indistinguishable from a pass at the point where a caller decides whether work is done. In an agent workflow nothing reads the count; it reads the status. Two of my agents would have reported tickets complete on the strength of it. Both worked around it by drivingxcodebuilddirectly with their own-derivedDataPath, which is the right answer but only because they noticed.Suggested
Either honour a per-call
projectPathover the session default, or fail loudly when one is passed and cannot be honoured. Failing that, a zero-test run should not report asSUCCEEDED— an-only-testingfilter matching nothing is an error, not a pass.Environment
Multiple concurrent git worktrees of one project, one shared MCP session, iOS simulator destination. Repointing the shared session defaults was not an option, since a concurrent agent was relying on them.