[feat] Limit each benchmark trial's time, tokens and turns #1400

Parent: #1137

Goal

A benchmark trial has no limit of its own. #1180's guide says so: "A trial has no timeout of its own. A tool that never returns runs until the suite's timeout, and every trial after it settles not_run." One hung tool turns a whole run into not_run, and a tool that loops spends tokens no table shows until the end.

Workflows already have the limits a trial needs: timeout, budget: { tokens } and max_turns (docs/guide/workflow-advanced.md, "Requirements and limits"). A trial is a prompt step, so let the suite set them for each trial.

Example

# benchmarks/fix-auth/suite.yaml
schemaVersion: v1alpha1
name: fix-auth
repo: https://github.com/acme/auth-service.git
defaults: { tool: claude }
arms:
  - name: claude
  - name: codex
    tool: codex
repetitions: 3
timeout: 45m
trial:
  timeout: 8m
  budget: { tokens: 300000 }
  max_turns: 60

| Arm | Tool | … | Passed | Failed | Error | … | Stopped by a limit | Deviations | … |
| claude | claude | … | 4 | 2 | 0 | … | 1 | 0 | … |
| codex | codex | … | 3 | 3 | 0 | … | 2 | 3 | … |

Scope

• trial in suite.yaml, with any of timeout, budget and max_turns, for every trial's prompt step. plan writes them onto that step with on_limit: end_turn, a key any workflow prompt step can use, on which budget is allowed:

  - id: prompt
    timeout: 8m
    budget: { tokens: 300000 }
    max_turns: 60
    on_limit: end_turn      # a limit ends the agent's turn; the step still succeeds, with limit: <kind>

  Under end_turn the limits count from the tool's launch, so preparing the clone and setup never count against the turn. No key means no limit, as today.

• The check still decides. When a limit stops the prompt, teamai runs the trial's check on what the agent left, after restoring protect as usual. The trial is passed or failed by that check, and its record entry gains limit: timeout | budget | max_turns. A limit is part of the task, like the time a person has, so a slower tool that does not finish fails the task; it is not an error of the harness. A task judged by successCriteria is judged on what the agent left, in the same way.

• The table gains "Stopped by a limit", counted per arm, so a pass rate shows how many trials ran out.

• An unenforced limit is a deviation. The local driver passes max_turns to headless Claude only, and other tools report it in unenforced. A driver that does not declare can.end_turn gets the released on_limit: fail lease instead, and its pass lists on_limit in unenforced. A trial whose report lists a suite limit, or on_limit, in unenforced counts as a deviation, as a trial on another tool or model does today, so the table never compares a capped arm with an uncapped one silently.

• The suite's timeout stays the cap of the whole run. A trial.timeout that is not shorter than it makes the suite invalid (trial.timeout must be shorter than timeout, which bounds the whole run): teamai review --branch fails it, teamai pull warns, and teamai benchmark run refuses it before a run is created, as for any other invalid suite.

• budget follows the step rule: a tool that reports no tokens cannot be held to a budget, and its trial ends as that rule says (budget_unmeasured).

• The plan frozen at start includes trial, so a pull during the run changes no limit.

Out of scope

Limits per task, a soft limit that stops the agent and counts the trial as a success, and limits on setup or on the check, which stay under the suite's timeout.

• Arms that run a workflow ([feat] Compare team asset sets in benchmarks #1181). timeout and budget are one allowance for the whole workflow, spent by the measured duration and usage of its prompt turns. Each inner prompt is limited to what is left. Once it is spent, the workflow's remaining steps are skipped and the check runs. max_turns is listed in unenforced for such an arm, so each of its trials counts as a deviation.

Acceptance criteria

1. With trial.timeout: 1m and a fake tool that sleeps 5 minutes, that trial's tool is stopped after 1 minute, the check runs, the trial is failed with limit: timeout, and the next trial runs.
2. A fake tool that writes the fix and then hangs is stopped at the limit, and its trial is passed with limit: timeout.
3. A trial whose tool passes budget.tokens is stopped, its check runs, and its entry has limit: budget.
4. With max_turns set, under the local driver, the codex arm's trials count as deviations and list max_turns in unenforced; the claude arm's do not.
5. trial.timeout not shorter than the suite's timeout fails teamai review --branch with the file and line, and teamai benchmark run refuses the suite before a run is created.
6. The table counts "Stopped by a limit" per arm, and each such record entry carries limit.
7. A suite without trial runs and reports as before.

Test plan

• Unit tests for parsing trial, the checks against the suite's timeout, and the deviation rule.
• E2E with a fixture project and fake tools for criteria 1 to 4, in the sandbox [feat] Run benchmark suites as workflows #1180 requires.
• One real-CLI run with Claude for the PR body.

Definition of done

• docs/guide/benchmarks.md and docs/guide/zh-CN/benchmarks.md document trial, "Stopped by a limit", limit on the record entry, and how the limits apply to a workflow arm.
• The workflow guide pages, English and Chinese, document on_limit: end_turn on a prompt step, budget under it, and can.end_turn for external drivers.
• The tests in Test plan pass, including the e2e for criteria 1 to 4.
• The PR's real-CLI record includes one Claude trial stopped by trial.timeout whose check still ran.

Watch out for

• The trial's prompt step must stop its tool's process group, not only the step, so the check never runs while the tool still writes.
• Document trial in docs/guide/benchmarks.md and its zh-CN page.

Dependencies

• Blocked by: [feat] Run benchmark suites as workflows #1180
• Unblocks: none