diff --git a/CHANGELOG.md b/CHANGELOG.md index 2e78fd67d..f2e32a6dd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,6 +21,7 @@ All notable changes to this project will be documented in this file. See [standa - teamai ships default workflows a team gets without writing them: `teamai-share-learnings` shares what a session taught as team learnings, one or several after one approval, each recorded as an outcome of the session, then runs `teamai-share-session`, which publishes the session's transcript and scores it. pull writes no `teamai-share-learnings` skill while recall is off or the source is read-only, since its learning step runs `share`, and `teamai recall enable`/`disable` add or remove that skill at once. A team file of the same name replaces a default; for one name the project's team file wins, then the user scope's, then teamai's default. A team `teamai-share-session` keeps the five inputs `teamai-share-learnings` passes it, with their types: with other inputs, or one of another type, pull warns and writes no `teamai-share-learnings` skill, `teamai-share-learnings` refuses to start with `invalid_inputs` before any learning is shared, and the friction reminder goes back to `teamai skill get share`. `teamai list skills -v` labels the skills pull generates for workflows `[workflow]`. `teamai list workflows` shows each workflow's source (`team` or `default`), `workflow start --describe` reports it as `source`, a run records it as `origin` with the teamai version of a default, and the dashboard's Workflow Runs page shows it in a Definition column, with the learnings a run shared beside its pull requests in an Outcomes column. A run of a default continued on a machine whose teamai ships another version of it refuses with `definition_mismatch` (for [#1388](https://github.com/Tencent/teamai-cli/issues/1388)). - `workflow start --describe teamai-share-learnings --driver skill`, and the same for `teamai-share-session`, proposes a `value` and a `why` for each input, for the calling session: the session and its tool, publishing when the attended publish policy is `ask`, the scorers that sample the session when attended scoring is `ask`, and whether teamai's own judge or the session's agent judges. A team override gets values for the inputs that keep their name and type (for [#1388](https://github.com/Tencent/teamai-cli/issues/1388)). - The dashboard's Overview has a **Member settings** panel: for the selected workspace, each member setting some level sets, its value, the level and file that set it, and what the levels below say, as `teamai doctor` lists them, plus the keys of a settings-only `~/.teamai/config.yaml` that nothing reads. The Session sharing panel names the setting behind the daily judging limit (for [#1459](https://github.com/Tencent/teamai-cli/issues/1459)). +- `teamai benchmark run ` runs a benchmark suite from `benchmarks//`: each task, a starting commit with a check or success criteria, goes to every arm (a tool, model, team revision or team workflow) a few times, each trial in its own clean clone on a machine declared a sandbox. The run prints a pass-rate table and publishes its record for the dashboard's Benchmarks page. `try` checks one task before it joins a suite, `plan` and `report` print the plan and the table, and `--fail-on-regression` fails a CI job when an arm's pass rate dropped by more than chance against the last comparable published run. It runs as `teamai-benchmark`, a default workflow the team can replace (for [#1180](https://github.com/Tencent/teamai-cli/issues/1180)). ### Changed diff --git a/README.ja.md b/README.ja.md index 41ef0e1d3..8df66c734 100644 --- a/README.ja.md +++ b/README.ja.md @@ -105,7 +105,7 @@ Git を基盤に、3 層の能力を構築します： - **Team Execution** — すべてのエージェントをチームのやり方で動かす：skills、rules、culture、agents、hooks、MCP、workflows、env、models。 - **Team Context**（beta）— すべてのエージェントにチームを理解させる：learnings、コード知識グラフ、teamwiki。 -- **Team Improvement**（beta）— 一回ごとの実行をチームの力として蓄積する：usage、sessions、scorers。 +- **Team Improvement**（beta）— 一回ごとの実行をチームの力として蓄積する：usage、sessions、scorers、benchmarks。 @@ -113,35 +113,35 @@ Git を基盤に、3 層の能力を構築します： - + - + - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + +
Agent Team Execution Team Context (beta)Team Improvement (beta)Team Improvement (beta)
skillsrulescultureagentshooksmcpworkflowsenvmodelsusagesessionsscorersusagesessionsscorersbenchmarks
Claude Code✓✓✓✓✓✓✓✓✓✓✓✓✓
Codex✓✓¹✓✓✓✓✓✓✓✓✓✓✓
Cursor✓✓✓✓✓✓✓✓—✓✓✓✓
CodeBuddy✓✓✓✓✓✓✓✓✓✓✓✓✓
WorkBuddy✓✓✓✓✓✓✓✓✓✓✓✓✓
GitHub Copilot CLI✓✓✓✓✓✓✓✓—✓✓✓✓³
OpenCode✓✓¹✓✓✓✓✓✓✓✓✓✓✓
Pi Coding Agent✓✓¹✓—✓✓✓✓²✓✓✓—✓
Oh My Pi✓✓✓✓✓✓✓✓✓✓✓—✓
OpenClaw✓✓¹✓—✓—✓✓—✓——✓³
Hermes✓✓¹✓—✓—✓✓—✓——✓³
DeepSeek Harness✓✓¹✓—✓—✓✓²—✓✓—✓³
Qoder✓✓✓✓✓✓✓✓—✓✓✓✓
Qoder CN✓✓✓✓✓✓✓✓—✓✓✓✓
Kiro✓✓✓✓✓✓✓✓—✓✓✓✓³
Trae✓✓✓✓—✓✓✓—✓——✓³
Trae CN✓✓✓✓—✓✓✓—✓——✓³
ZCode✓✓¹✓✓✓✓✓✓²—✓✓✓✓³
JoyCode✓✓¹✓✓✓—✓✓—✓✓✓✓
Muse Code✓✓¹✓✓✓✓✓✓✓✓✓✓✓³
Antigravity✓✓✓✓✓✓✓✓✓✓✓✓✓
Claude Code✓✓✓✓✓✓✓✓✓✓✓✓✓✓
Codex✓✓¹✓✓✓✓✓✓✓✓✓✓✓✓
Cursor✓✓✓✓✓✓✓✓—✓✓✓✓—
CodeBuddy✓✓✓✓✓✓✓✓✓✓✓✓✓✓
WorkBuddy✓✓✓✓✓✓✓✓✓✓✓✓✓✓
GitHub Copilot CLI✓✓✓✓✓✓✓✓—✓✓✓✓³—
OpenCode✓✓¹✓✓✓✓✓✓✓✓✓✓✓✓
Pi Coding Agent✓✓¹✓—✓✓✓✓²✓✓✓—✓—
Oh My Pi✓✓✓✓✓✓✓✓✓✓✓—✓—
OpenClaw✓✓¹✓—✓—✓✓—✓——✓³—
Hermes✓✓¹✓—✓—✓✓—✓——✓³—
DeepSeek Harness✓✓¹✓—✓—✓✓²—✓✓—✓³—
Qoder✓✓✓✓✓✓✓✓—✓✓✓✓—
Qoder CN✓✓✓✓✓✓✓✓—✓✓✓✓—
Kiro✓✓✓✓✓✓✓✓—✓✓✓✓³—
Trae✓✓✓✓—✓✓✓—✓——✓³—
Trae CN✓✓✓✓—✓✓✓—✓——✓³—
ZCode✓✓¹✓✓✓✓✓✓²—✓✓✓✓³—
JoyCode✓✓¹✓✓✓—✓✓—✓✓✓✓—
Muse Code✓✓¹✓✓✓✓✓✓✓✓✓✓✓³—
Antigravity✓✓✓✓✓✓✓✓✓✓✓✓✓✓
diff --git a/README.ko.md b/README.ko.md index ec0930d0b..07906ea98 100644 --- a/README.ko.md +++ b/README.ko.md @@ -105,7 +105,7 @@ Git을 기반으로 세 층의 역량을 구축합니다: - **Team Execution** — 모든 에이전트가 팀의 방식대로 일하게 합니다: skills, rules, culture, agents, hooks, MCP, workflows, env, models. - **Team Context** (beta) — 모든 에이전트가 팀을 이해하게 합니다: learnings, 코드베이스 그래프, teamwiki. -- **Team Improvement** (beta) — 매 실행이 팀의 역량으로 쌓이게 합니다: usage, sessions, scorers. +- **Team Improvement** (beta) — 매 실행이 팀의 역량으로 쌓이게 합니다: usage, sessions, scorers, benchmarks. @@ -113,35 +113,35 @@ Git을 기반으로 세 층의 역량을 구축합니다: - + - + - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + +
Agent Team Execution Team Context (beta)Team Improvement (beta)Team Improvement (beta)
skillsrulescultureagentshooksmcpworkflowsenvmodelsusagesessionsscorersusagesessionsscorersbenchmarks
Claude Code✓✓✓✓✓✓✓✓✓✓✓✓✓
Codex✓✓¹✓✓✓✓✓✓✓✓✓✓✓
Cursor✓✓✓✓✓✓✓✓—✓✓✓✓
CodeBuddy✓✓✓✓✓✓✓✓✓✓✓✓✓
WorkBuddy✓✓✓✓✓✓✓✓✓✓✓✓✓
GitHub Copilot CLI✓✓✓✓✓✓✓✓—✓✓✓✓³
OpenCode✓✓¹✓✓✓✓✓✓✓✓✓✓✓
Pi Coding Agent✓✓¹✓—✓✓✓✓²✓✓✓—✓
Oh My Pi✓✓✓✓✓✓✓✓✓✓✓—✓
OpenClaw✓✓¹✓—✓—✓✓—✓——✓³
Hermes✓✓¹✓—✓—✓✓—✓——✓³
DeepSeek Harness✓✓¹✓—✓—✓✓²—✓✓—✓³
Qoder✓✓✓✓✓✓✓✓—✓✓✓✓
Qoder CN✓✓✓✓✓✓✓✓—✓✓✓✓
Kiro✓✓✓✓✓✓✓✓—✓✓✓✓³
Trae✓✓✓✓—✓✓✓—✓——✓³
Trae CN✓✓✓✓—✓✓✓—✓——✓³
ZCode✓✓¹✓✓✓✓✓✓²—✓✓✓✓³
JoyCode✓✓¹✓✓✓—✓✓—✓✓✓✓
Muse Code✓✓¹✓✓✓✓✓✓✓✓✓✓✓³
Antigravity✓✓✓✓✓✓✓✓✓✓✓✓✓
Claude Code✓✓✓✓✓✓✓✓✓✓✓✓✓✓
Codex✓✓¹✓✓✓✓✓✓✓✓✓✓✓✓
Cursor✓✓✓✓✓✓✓✓—✓✓✓✓—
CodeBuddy✓✓✓✓✓✓✓✓✓✓✓✓✓✓
WorkBuddy✓✓✓✓✓✓✓✓✓✓✓✓✓✓
GitHub Copilot CLI✓✓✓✓✓✓✓✓—✓✓✓✓³—
OpenCode✓✓¹✓✓✓✓✓✓✓✓✓✓✓✓
Pi Coding Agent✓✓¹✓—✓✓✓✓²✓✓✓—✓—
Oh My Pi✓✓✓✓✓✓✓✓✓✓✓—✓—
OpenClaw✓✓¹✓—✓—✓✓—✓——✓³—
Hermes✓✓¹✓—✓—✓✓—✓——✓³—
DeepSeek Harness✓✓¹✓—✓—✓✓²—✓✓—✓³—
Qoder✓✓✓✓✓✓✓✓—✓✓✓✓—
Qoder CN✓✓✓✓✓✓✓✓—✓✓✓✓—
Kiro✓✓✓✓✓✓✓✓—✓✓✓✓³—
Trae✓✓✓✓—✓✓✓—✓——✓³—
Trae CN✓✓✓✓—✓✓✓—✓——✓³—
ZCode✓✓¹✓✓✓✓✓✓²—✓✓✓✓³—
JoyCode✓✓¹✓✓✓—✓✓—✓✓✓✓—
Muse Code✓✓¹✓✓✓✓✓✓✓✓✓✓✓³—
Antigravity✓✓✓✓✓✓✓✓✓✓✓✓✓✓
diff --git a/README.md b/README.md index b313d333e..f8ba7b9d8 100644 --- a/README.md +++ b/README.md @@ -105,7 +105,7 @@ Three layers of capability, built on Git: - **Team Execution** — make every agent work the team's way: skills, rules, culture, agents, hooks, MCP, workflows, env, models. - **Team Context** (beta) — make every agent understand the team: learnings, codebase graph, teamwiki. -- **Team Improvement** (beta) — make every execution improve the team: usage, sessions, scorers. +- **Team Improvement** (beta) — make every execution improve the team: usage, sessions, scorers, benchmarks. @@ -113,35 +113,35 @@ Three layers of capability, built on Git: - + - + - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + +
Agent Team Execution Team Context (beta)Team Improvement (beta)Team Improvement (beta)
skillsrulescultureagentshooksmcpworkflowsenvmodelsusagesessionsscorersusagesessionsscorersbenchmarks
Claude Code✓✓✓✓✓✓✓✓✓✓✓✓✓
Codex✓✓¹✓✓✓✓✓✓✓✓✓✓✓
Cursor✓✓✓✓✓✓✓✓—✓✓✓✓
CodeBuddy✓✓✓✓✓✓✓✓✓✓✓✓✓
WorkBuddy✓✓✓✓✓✓✓✓✓✓✓✓✓
GitHub Copilot CLI✓✓✓✓✓✓✓✓—✓✓✓✓³
OpenCode✓✓¹✓✓✓✓✓✓✓✓✓✓✓
Pi Coding Agent✓✓¹✓—✓✓✓✓²✓✓✓—✓
Oh My Pi✓✓✓✓✓✓✓✓✓✓✓—✓
OpenClaw✓✓¹✓—✓—✓✓—✓——✓³
Hermes✓✓¹✓—✓—✓✓—✓——✓³
DeepSeek Harness✓✓¹✓—✓—✓✓²—✓✓—✓³
Qoder✓✓✓✓✓✓✓✓—✓✓✓✓
Qoder CN✓✓✓✓✓✓✓✓—✓✓✓✓
Kiro✓✓✓✓✓✓✓✓—✓✓✓✓³
Trae✓✓✓✓—✓✓✓—✓——✓³
Trae CN✓✓✓✓—✓✓✓—✓——✓³
ZCode✓✓¹✓✓✓✓✓✓²—✓✓✓✓³
JoyCode✓✓¹✓✓✓—✓✓—✓✓✓✓
Muse Code✓✓¹✓✓✓✓✓✓✓✓✓✓✓³
Antigravity✓✓✓✓✓✓✓✓✓✓✓✓✓
Claude Code✓✓✓✓✓✓✓✓✓✓✓✓✓✓
Codex✓✓¹✓✓✓✓✓✓✓✓✓✓✓✓
Cursor✓✓✓✓✓✓✓✓—✓✓✓✓—
CodeBuddy✓✓✓✓✓✓✓✓✓✓✓✓✓✓
WorkBuddy✓✓✓✓✓✓✓✓✓✓✓✓✓✓
GitHub Copilot CLI✓✓✓✓✓✓✓✓—✓✓✓✓³—
OpenCode✓✓¹✓✓✓✓✓✓✓✓✓✓✓✓
Pi Coding Agent✓✓¹✓—✓✓✓✓²✓✓✓—✓—
Oh My Pi✓✓✓✓✓✓✓✓✓✓✓—✓—
OpenClaw✓✓¹✓—✓—✓✓—✓——✓³—
Hermes✓✓¹✓—✓—✓✓—✓——✓³—
DeepSeek Harness✓✓¹✓—✓—✓✓²—✓✓—✓³—
Qoder✓✓✓✓✓✓✓✓—✓✓✓✓—
Qoder CN✓✓✓✓✓✓✓✓—✓✓✓✓—
Kiro✓✓✓✓✓✓✓✓—✓✓✓✓³—
Trae✓✓✓✓—✓✓✓—✓——✓³—
Trae CN✓✓✓✓—✓✓✓—✓——✓³—
ZCode✓✓¹✓✓✓✓✓✓²—✓✓✓✓³—
JoyCode✓✓¹✓✓✓—✓✓—✓✓✓✓—
Muse Code✓✓¹✓✓✓✓✓✓✓✓✓✓✓³—
Antigravity✓✓✓✓✓✓✓✓✓✓✓✓✓✓
diff --git a/README.th.md b/README.th.md index 759f98f0e..85338aa2c 100644 --- a/README.th.md +++ b/README.th.md @@ -105,7 +105,7 @@ teamai init https://github.com/your-org/your-repo --scope user - **Team Execution** — ทำให้ทุก Agent ทำงานตามแนวทางของทีม: skills, rules, culture, agents, hooks, MCP, workflows, env, models - **Team Context** (beta) — ทำให้ทุก Agent เข้าใจทีม: learnings, กราฟความรู้โค้ดเบส, teamwiki -- **Team Improvement** (beta) — ทำให้ทุกการทำงานยกระดับทีม: usage, sessions, scorers +- **Team Improvement** (beta) — ทำให้ทุกการทำงานยกระดับทีม: usage, sessions, scorers, benchmarks @@ -113,35 +113,35 @@ teamai init https://github.com/your-org/your-repo --scope user - + - + - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + +
Agent Team Execution Team Context (beta)Team Improvement (beta)Team Improvement (beta)
skillsrulescultureagentshooksmcpworkflowsenvmodelsusagesessionsscorersusagesessionsscorersbenchmarks
Claude Code✓✓✓✓✓✓✓✓✓✓✓✓✓
Codex✓✓¹✓✓✓✓✓✓✓✓✓✓✓
Cursor✓✓✓✓✓✓✓✓—✓✓✓✓
CodeBuddy✓✓✓✓✓✓✓✓✓✓✓✓✓
WorkBuddy✓✓✓✓✓✓✓✓✓✓✓✓✓
GitHub Copilot CLI✓✓✓✓✓✓✓✓—✓✓✓✓³
OpenCode✓✓¹✓✓✓✓✓✓✓✓✓✓✓
Pi Coding Agent✓✓¹✓—✓✓✓✓²✓✓✓—✓
Oh My Pi✓✓✓✓✓✓✓✓✓✓✓—✓
OpenClaw✓✓¹✓—✓—✓✓—✓——✓³
Hermes✓✓¹✓—✓—✓✓—✓——✓³
DeepSeek Harness✓✓¹✓—✓—✓✓²—✓✓—✓³
Qoder✓✓✓✓✓✓✓✓—✓✓✓✓
Qoder CN✓✓✓✓✓✓✓✓—✓✓✓✓
Kiro✓✓✓✓✓✓✓✓—✓✓✓✓³
Trae✓✓✓✓—✓✓✓—✓——✓³
Trae CN✓✓✓✓—✓✓✓—✓——✓³
ZCode✓✓¹✓✓✓✓✓✓²—✓✓✓✓³
JoyCode✓✓¹✓✓✓—✓✓—✓✓✓✓
Muse Code✓✓¹✓✓✓✓✓✓✓✓✓✓✓³
Antigravity✓✓✓✓✓✓✓✓✓✓✓✓✓
Claude Code✓✓✓✓✓✓✓✓✓✓✓✓✓✓
Codex✓✓¹✓✓✓✓✓✓✓✓✓✓✓✓
Cursor✓✓✓✓✓✓✓✓—✓✓✓✓—
CodeBuddy✓✓✓✓✓✓✓✓✓✓✓✓✓✓
WorkBuddy✓✓✓✓✓✓✓✓✓✓✓✓✓✓
GitHub Copilot CLI✓✓✓✓✓✓✓✓—✓✓✓✓³—
OpenCode✓✓¹✓✓✓✓✓✓✓✓✓✓✓✓
Pi Coding Agent✓✓¹✓—✓✓✓✓²✓✓✓—✓—
Oh My Pi✓✓✓✓✓✓✓✓✓✓✓—✓—
OpenClaw✓✓¹✓—✓—✓✓—✓——✓³—
Hermes✓✓¹✓—✓—✓✓—✓——✓³—
DeepSeek Harness✓✓¹✓—✓—✓✓²—✓✓—✓³—
Qoder✓✓✓✓✓✓✓✓—✓✓✓✓—
Qoder CN✓✓✓✓✓✓✓✓—✓✓✓✓—
Kiro✓✓✓✓✓✓✓✓—✓✓✓✓³—
Trae✓✓✓✓—✓✓✓—✓——✓³—
Trae CN✓✓✓✓—✓✓✓—✓——✓³—
ZCode✓✓¹✓✓✓✓✓✓²—✓✓✓✓³—
JoyCode✓✓¹✓✓✓—✓✓—✓✓✓✓—
Muse Code✓✓¹✓✓✓✓✓✓✓✓✓✓✓³—
Antigravity✓✓✓✓✓✓✓✓✓✓✓✓✓✓
diff --git a/README.zh-CN.md b/README.zh-CN.md index 896f55fa7..9819f6d13 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -111,7 +111,7 @@ teamai init https://github.com/your-org/your-repo --scope user - **Team Execution** —— 让每个 Agent 按团队的方式工作：skills、rules、culture、agents、hooks、MCP、workflows、env、models。 - **Team Context**（beta）—— 让每个 Agent 理解整个团队：learnings、代码知识图谱、teamwiki。 -- **Team Improvement**（beta）—— 让每一次执行都沉淀为团队能力：usage、sessions、scorers。 +- **Team Improvement**（beta）—— 让每一次执行都沉淀为团队能力：usage、sessions、scorers、benchmarks。 @@ -119,35 +119,35 @@ teamai init https://github.com/your-org/your-repo --scope user - + - + - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + +
Agent Team Execution Team Context (beta)Team Improvement (beta)Team Improvement (beta)
skillsrulescultureagentshooksmcpworkflowsenvmodelsusagesessionsscorersusagesessionsscorersbenchmarks
Claude Code✓✓✓✓✓✓✓✓✓✓✓✓✓
Codex✓✓¹✓✓✓✓✓✓✓✓✓✓✓
Cursor✓✓✓✓✓✓✓✓—✓✓✓✓
CodeBuddy✓✓✓✓✓✓✓✓✓✓✓✓✓
WorkBuddy✓✓✓✓✓✓✓✓✓✓✓✓✓
GitHub Copilot CLI✓✓✓✓✓✓✓✓—✓✓✓✓³
OpenCode✓✓¹✓✓✓✓✓✓✓✓✓✓✓
Pi Coding Agent✓✓¹✓—✓✓✓✓²✓✓✓—✓
Oh My Pi✓✓✓✓✓✓✓✓✓✓✓—✓
OpenClaw✓✓¹✓—✓—✓✓—✓——✓³
Hermes✓✓¹✓—✓—✓✓—✓——✓³
DeepSeek Harness✓✓¹✓—✓—✓✓²—✓✓—✓³
Qoder✓✓✓✓✓✓✓✓—✓✓✓✓
Qoder CN✓✓✓✓✓✓✓✓—✓✓✓✓
Kiro✓✓✓✓✓✓✓✓—✓✓✓✓³
Trae✓✓✓✓—✓✓✓—✓——✓³
Trae CN✓✓✓✓—✓✓✓—✓——✓³
ZCode✓✓¹✓✓✓✓✓✓²—✓✓✓✓³
JoyCode✓✓¹✓✓✓—✓✓—✓✓✓✓
Muse Code✓✓¹✓✓✓✓✓✓✓✓✓✓✓³
Antigravity✓✓✓✓✓✓✓✓✓✓✓✓✓
Claude Code✓✓✓✓✓✓✓✓✓✓✓✓✓✓
Codex✓✓¹✓✓✓✓✓✓✓✓✓✓✓✓
Cursor✓✓✓✓✓✓✓✓—✓✓✓✓—
CodeBuddy✓✓✓✓✓✓✓✓✓✓✓✓✓✓
WorkBuddy✓✓✓✓✓✓✓✓✓✓✓✓✓✓
GitHub Copilot CLI✓✓✓✓✓✓✓✓—✓✓✓✓³—
OpenCode✓✓¹✓✓✓✓✓✓✓✓✓✓✓✓
Pi Coding Agent✓✓¹✓—✓✓✓✓²✓✓✓—✓—
Oh My Pi✓✓✓✓✓✓✓✓✓✓✓—✓—
OpenClaw✓✓¹✓—✓—✓✓—✓——✓³—
Hermes✓✓¹✓—✓—✓✓—✓——✓³—
DeepSeek Harness✓✓¹✓—✓—✓✓²—✓✓—✓³—
Qoder✓✓✓✓✓✓✓✓—✓✓✓✓—
Qoder CN✓✓✓✓✓✓✓✓—✓✓✓✓—
Kiro✓✓✓✓✓✓✓✓—✓✓✓✓³—
Trae✓✓✓✓—✓✓✓—✓——✓³—
Trae CN✓✓✓✓—✓✓✓—✓——✓³—
ZCode✓✓¹✓✓✓✓✓✓²—✓✓✓✓³—
JoyCode✓✓¹✓✓✓—✓✓—✓✓✓✓—
Muse Code✓✓¹✓✓✓✓✓✓✓✓✓✓✓³—
Antigravity✓✓✓✓✓✓✓✓✓✓✓✓✓✓
diff --git a/docs/dev/workflow-engine.md b/docs/dev/workflow-engine.md index 144f2abfe..e9c5063fa 100644 --- a/docs/dev/workflow-engine.md +++ b/docs/dev/workflow-engine.md @@ -6,12 +6,12 @@ Each rule below spans more than one file. The rule names the code that owns it, ## Rules that span files -- **A run ends only in `settleRun`.** The shared `finishRun` path calls it when settling reaches the top workflow's result, before it hands any lease back or out, and on a call that takes no lease too. `reportAttempt` first reconciles earlier reports, so an end cancels a later report before it can advance the revision. It records an accepted outcome and never ends the run. After a failed report, the next `next` skips the dependents and ends the run. Respond, cancel and retry in `run-controls.ts`, and next in `engine.ts`, share that path. Whole-run cancel settles directly as `cancelled`, without reconciling another round. To run code when a run ends, hook into `finishRun`. Do network I/O only after `withRun` releases the run lock. That path queues the end record, a local write, under the lock, and saves whether it did in the run's `endRecordQueued` field. When `updateRun` then fails to save the run file, the queue is one save ahead of the run file, and the next `next` settles the run again and overwrites the queued `.end.json`. A later `next` reports a record state only for a run whose field is true (`runRecordState` in `records.ts`). +- **A run ends only in `settleRun`.** The shared `finishRun` path calls it when settling reaches the top workflow's result, before it hands any lease back or out, and on a call that takes no lease too. `reportAttempt` first reconciles earlier reports, so an end cancels a later report before it can advance the revision. It records an accepted outcome and never ends the run. After a failed report, the next `next` skips the dependents and ends the run. Respond, cancel and retry in `run-controls.ts`, and next in `engine.ts`, share that path. Whole-run cancel settles directly as `cancelled`, without reconciling another round. A run whose workflow declares `finally` does not end there: after any judge wait, `finishRun` freezes the verdicts, then calls `enterFinalizing`, which keeps the run `running`, holds the outcome in `finalizing` with `until` as the run's deadline, and adds the `finally` instances; `reconcileRun` then settles `finallyScope` instead of `topScope`, and `settleRun` ends the run with the held outcome, whatever the `finally` steps did. To run code when a run ends, hook into `finishRun`. Do network I/O only after `withRun` releases the run lock. That path queues the end record, a local write, under the lock, and saves whether it did in the run's `endRecordQueued` field. When `updateRun` then fails to save the run file, the queue is one save ahead of the run file, and the next `next` settles the run again and overwrites the queued `.end.json`. A later `next` reports a record state only for a run whose field is true (`runRecordState` in `records.ts`). - **Reconcile the latest report before applying another or handing out a lease.** `reconcileRun`, shared by `leaseNext` and `reportAttempt`, settles every driverless transition to a fixpoint at the run's current revision, through `settleScope` and `settleItems`, including workflow starts, fan-out expansion, render failures and bound exhaustion. `reportAttempt` advances the revision only after it, so only the latest revision can be unreconciled. A reached end applies before a loop decision. Only after reconciliation does `leaseNext` hand back an omitted open lease or issue the first ready leaf in declaration order that this driver can take. A running workflow occupies its parent's position in that order. Settling returns every ready leaf in that order (`ready`), and `leaseNext` walks it. A ready item does not count against `max_parallel` until it is leased, and one `next` leases at most one step, so a held-back item leaves its room to the next one without exceeding the cap. `canTakeLease: false` still reconciles driverless transitions. Every changed run is saved, including a `wait`. - **A report is checked against the revision of its own lease.** `reportAttempt` and `retryStep` increment `revision`; skips, `if`, `end`, and the creation of child steps inside `next` never do. A lease keeps the run revision it was handed out at, and `reportAttempt` compares a report with that, so the report of a parallel lease does not make it stale. An attempt that is not an open lease keeps the older answers: `stale_revision` when its revision is not the run's, otherwise `wrong_attempt`. -- **An end or break stops what is still open under its workflow.** `stopWhere` cancels every leased step under the workflow that reached the end, its sub-workflows included, and keeps the attempt on the step and in `cancelledAttempts`. It cancels running workflow steps there and skips pending and waiting ones. The walk takes a path predicate and failure. End and break construct their own end_reached failure; manual cancel and retry supply theirs. Step descendants use the same `isUnder` predicate as `archiveUnder` in `loop.ts`. A cancelled delivering step keeps its agent settlement and its original `settledAt` for explicit recovery, records and scoring. Automatic delivery and workflow-item branch summaries exclude cancelled children. Cancel records the ids this walk changed before reconciliation can settle or skip other steps. `reportAttempt` answers a cancelled attempt `lease_cancelled` before it checks the driver; `cancelledAttempts` is never trimmed. Every `next` answer lists in `cancelled` the attempts the driver says it runs that are no longer its open leases, so the driver learns of a cancellation from any answer, `done` included. +- **An end or break stops what is still open under its workflow.** `stopWhere` cancels every leased step under the workflow that reached the end, its sub-workflows included, and keeps the attempt on the step and in `cancelledAttempts`. It cancels running workflow steps there and skips pending and waiting ones; while the run's `finally` steps run, where only a whole-run cancel or the deadline reaches it, it cancels those too. The walk takes a path predicate and failure. End and break construct their own end_reached failure; manual cancel and retry supply theirs. Step descendants use the same `isUnder` predicate as `archiveUnder` in `loop.ts`. A cancelled delivering step keeps its agent settlement and its original `settledAt` for explicit recovery, records and scoring. Automatic delivery and workflow-item branch summaries exclude cancelled children. Cancel records the ids this walk changed before reconciliation can settle or skip other steps. `reportAttempt` answers a cancelled attempt `lease_cancelled` before it checks the driver; `cancelledAttempts` is never trimmed. Every `next` answer lists in `cancelled` the attempts the driver says it runs that are no longer its open leases, so the driver learns of a cancellation from any answer, `done` included. - **The local driver overlaps steps that do not share a checkout.** `runLocalDriver` runs up to `workflows.max_parallel` steps (local config, default 1) and sends the ones it has started as `running`. A command with no workspace, and a prompt in the current checkout, share that checkout, so only one of them runs there at a time. A prompt earlier in the queue for that checkout blocks a later command there. The driver starts the prompt when the checkout is free and fewer than `max_parallel` steps are in flight; commands in other worktrees keep running. A prompt with its own worktree does not take the checkout. A headless prompt takes one of the slots, and the driver keeps leasing steps for the others while it runs. An interactive prompt uses the terminal, starts only when no command is running, and no step is leased while it runs. On `cancelled`, it drops a held lease, and it stops a running command with SIGTERM, then SIGKILL after 5 s while any captured descendant or process in a detached group remains, even after its command exited. Attached commands snapshot their descendant tree and start times before stopping, then probe and send SIGKILL only to descendants whose start times still match. If the identity snapshot fails, only the shell is stopped; detached commands use their process group. It does not send a stopping command as `running` again, so `next` does not cancel it twice. Dying commands still take a slot, and a new attempt of their step waits until they are gone. The signal policy follows `max_parallel`. At 1, a command runs in teamai's process group, as `env exec` runs one (`src/env-exec.ts`): the terminal's Ctrl-C reaches it, and it can read `/dev/tty`. A signal `env exec` would pass on (SIGTERM or SIGHUP, and SIGINT or SIGQUIT when the terminal does not reach the command) makes the driver stop the command, wait for it, and exit with that signal. Above 1, each command runs detached in a process group of its own, recorded in the attempt's `group-.json` as setup's is, with no terminal, and SIGINT, SIGQUIT, SIGTERM or SIGHUP makes the driver stop every group, wait for them, and exit with that signal. Windows uses `taskkill /T /F` in place of the group kill. Staging runs inside the attempt at every `max_parallel` (`stageThenStart`), so the driver keeps polling while it stages, and a cancel aborts staging. A step whose attempt was cancelled, or whose lease expired, while it staged never starts. `runPlaceSetup` in `place/place.ts` runs setup (`setup.ts`) after the git phase, only in a worktree prepare just created, with setup's own keys, and removes the worktree when setup fails. It records `setting-up` before setup starts, and a prepare that finds that phase gives the place up as a failed setup does, so a second setup is never checked against what one cut short left there. Before it stages an attempt the engine hands back, the driver stops every group a killed driver's setup or command left there (`stopRecordedGroup`), each recorded in the attempt's own `group-.json`. -- **A workspace is a git contract on the lease. TeamAI owns the workspace contract and its git; the host owns the place. TeamAI reads the delivered commit and opens the PR outside the run lock.** `place/` prepares and finishes every place: the local driver prepares through `place.ts` in process, and `next --prepare` through `protocol.ts` for a driver that runs the protocol from the CLI; every driver's report finishes and releases its place through `managedReport` in `protocol.ts`. `preparePlace` records each place in `attempts//place.json` by phase (`opening`, `preparing`, `staged`, `setting-up`, `ready`, `finished`, `abandoned`), with the reader's snapshot in `before.json` and the branch's agent files in `overlay/` beside it, so whichever process reports the attempt finishes it. It records `opening` before git touches the copy, with whether the copy existed and, once it is open, a writer's range start, taken before any merge, so a prepare killed in git is finished by the next prepare of the attempt, which still runs setup in a copy the killed one created and keeps its merges in the step's `changes`. It records `preparing` before it touches the agent files, and copies them aside to `overlay.partial/`, renamed to `overlay/` once complete, so a prepare killed during the overlay is finished by the next prepare of the attempt, or put back by `abandonPlace`. `next --prepare` leases and prepares under the run lock, and a prepare that fails settles the attempt `failed` with its own code. `report` refuses caller `changes` and `read` for a prepared attempt, finishes its place before it reports, and leaves the place as prepare found it (`abandonPlace`) when the attempt is `released` or the run answers `lease_cancelled`. `planWorkspace` in `workspace.ts` chooses the mode. An explicit `workspace` wins. A command that omits it is read-only, even inside a copy; when it exits 0 and changed the checkout, the driver fails it `read_only_dirty`. That includes a commit that leaves the worktree clean, a switch to another branch or a detach at the same commit, a change that only stages an existing edit, another staged hunk that leaves the status mixed, a permission change on a file that was already dirty, an edit to a quoted non-ASCII path, a symlink whose target changes while the file it names stays the same, and an edit to the destination of a staged rename, and a change inside a submodule that was already dirty. An interactive prompt that sets `workspace: read-only` is checked the same way. An interactive prompt that does not may change the checkout, and those edits stay. Conflict paths, including agent files, are read without Git's quoting, so a non-ASCII name is still checked for leftover markers. A prompt that omits it continues the nearest enclosing `new` or `continue` workflow, and a `for_each` ancestor makes that `new`, so items do not share a worktree. `leaseNext` puts the contract on the lease for `new` and `continue`, and for a read-only step inside a copy (`mode: 'read-only'`, `merge: []`). That reader's branch is the one its declared `needs` wrote or read (`changes` or `read`, through items and children, but not `after` items or inherited needs), or the enclosing copy's when they touched none; needs on several branches fail `workspace_base`. It needs `continue` in `can.workspace` and the copy's owner, as `continue` does. A read-only step outside any copy gets no contract and runs in the active checkout. `preparePlace` stages it there as a reader, and `finishPlace` checks it; a second prepare or finish returns the first one's result. `place/reader.ts` stages a reader in a copy without merges, takes its snapshot over the base overlay, and afterwards restores the overlay; when the step changed the copy or left its branch, it keeps the state on `refs/teamai/kept/` (`git add -A`, then `write-tree` and `commit-tree` on HEAD; not the stash list, which every worktree shares), and pushes each submodule commit that tree records and the read commit does not into the checkout's git dir as `refs/teamai/kept-modules//` (at any depth, without the user's push hooks or `push.recurseSubmodules`), since a gitlink does not keep its commit reachable once the copy is removed, then runs `git checkout -f `, `git reset --hard` to the commit it read, `git clean -fd`, and `git submodule update --checkout --recursive` on the submodules it changed, so a later writer never commits a reader's edits. It resets nothing when a repository among the changed paths has uncommitted or untracked files, or a nested submodule commit its parent does not record, which the keep cannot hold. The copy counts as put back only when HEAD is `refs/heads/` at that commit and `git status --porcelain --untracked-files=all --ignore-submodules=none` prints the same lines it printed when the reader was staged; ignored files are left in place. Otherwise the reader fails `workspace_base`. `preparePlace` writes a `teamai-unrestored` marker, with that status, in the worktree's git dir before the reader's overlay, and the driver removes it once the copy is put back or left untouched, so a driver killed mid-reader still leaves it. `openWorktree` in `place/created.ts` refuses later leases in a marked copy until the copy is back, when it removes the marker. A status line does not show content, so a marker whose status was not empty stays until the person who puts the copy back deletes it. `ensureWorktree` also refuses an existing worktree whose HEAD is not on its branch or a directory that is not a worktree, and prunes and recreates one whose directory was deleted. Parallel steps run git in one repository, and git writes a new worktree's entry without a lock, so a command that reads every worktree (`fetch`, `branch`, `worktree add`) beside it dies on the entry half written (`failed to read .git/worktrees//commondir`). `git` in `worktree.ts` runs such a command again, and `ensureWorktree` makes the branch before `worktree add`: `worktree add -b` makes it before it reads the worktrees, and would find it made when it ran again. The reader reports `read` (`branch`, `base`, `commit`) and never `changes`. `report` returns `invalid_report` for `read` on any other lease, and for `changes` or a success without `read` on a reader's lease. `run.copies[].commit` and fan-out `branches` follow `changes` only. `cleanup` comes from the lease's contract, so a reader that was the copy's last user removes it too. `leaseNext` holds a step back for this driver, and tries the next ready leaf, when the driver's `can.workspace` lacks that mode, or when a `continue` copy belongs to another driver (`run.copies`). It answers `wait`, with every held step's reason joined by `; `, only when every ready leaf is held back. Driver capabilities stay out of `reconcileRun`, which `reportAttempt` and the run controls also call. It fails `workspace_base` when `continue` has no enclosing `new`, or when the run has no base. `workflow start` in a git checkout records the base (commit, and the branch unless HEAD is detached); otherwise the first `next` that sends `base` sets it, and a later `base` never changes it. The local driver fails a lease whose `workspace` it cannot read with `workspace_base` instead of running it in the active checkout. The branch is `teamai///p` (`branchName`); a `[n]` in the path becomes `-n`. The first branch in `needs` order is where the step starts, and the rest are merged in that order. An item's `after` branches come first. `place/` (`created.ts`, `overlay.ts`, `worktree.ts`) adds the worktree under `.teamai/workflow-worktrees/` of the checkout the run started from (`run.checkout`), or of the driver's own checkout when it does not share that checkout's git directory, such as a separate clone (`placeRoot`), points `core.hooksPath` at an empty directory, and replaces `AGENT_PATHS` (`AGENTS.md`, `CLAUDE.md`, `.agents`, `.githooks`, and each default `toolPaths` entry's project directory, project MCP file and Copilot `.github` paths) with the run's base commit. An agent path under a symlinked parent fails `base_config` before anything is copied or removed, since both would follow the link. It then gives the worktree the inline `pull` a new worktree gets (`prepareCheckout`), which delivers the team's skills, rules, agents and MCP and records that checkout's team revision. The overlay is taken after that delivery, so those files are not the step's commit. A path the step edits is committed. A step that left HEAD off the leased branch fails `workspace_base` before anything is committed, reported as `changes`, or pushed, and the worktree stays as it left it. A path it leaves untouched is put back from the branch. A merge conflict stops the merge: a command fails `merge_conflict`, and a prompt is told the branches that remain. Snapshots and commits read submodules with `--ignore-submodules=none`, and after `git add -A`, which skips a submodule whose `ignore` is `all`, each submodule is staged by name, so `submodule..ignore` and `diff.ignoreSubmodules` hide no change from a reader's check, its keep, or a writer's commit. The driver commits as `TeamAI ` and reports `changes` (`branch`, `base`, `from`, `to`, `commits`). It pushes when `origin` exists and this step made commits, whether the step succeeded or failed, and also when a later attempt adds no commit but this branch has commits `origin` does not have, including a first push that never landed and an origin with no tracking refs yet. A failed push fails a step that succeeded with `push_failed`, and the worktree stays. `report` sets `cleanup` only when the step succeeded and no step that can still run uses that branch, judged on the run as the next `next` settles it: steps a loop runs again count, and a workflow or `for_each` step that has not started counts by the branch `plannedBranch` gives, the one around it unless it opens its own; only then does `releasePlace` run `git worktree remove --force`. It does not delete the branch, and it never resets, cleans, or switches the active checkout. A command that starts its own sandbox, such as `docker`, stays read-only. What it writes does not feed `new`, `branches`, or a later step. `session: continue` and `session.from` are step reads. The engine puts `{ tool, id }` on the lease, with `earlier`, the sessions that one continued, newest first, and never copies a session. The driver resumes Claude and Codex by that id, copies another tool through `AgentSession.copy`, restores a copy from the team session archive when neither works, and tries each `earlier` session the same way before it starts fresh. OpenCode has no resume flag, so it starts fresh. A driver that continued the lease's session or an earlier one, by that id or in a copy, echoes the one it continued as `resumed` in its report. One that started a new session leaves `resumed` out. `reportAttempt` answers `invalid_report` for a `resumed` on a step that resumes no session, and for one that no pass of the run reported and no link continued (`knowsSession`). When the reported `tool` and `session` differ from `resumed`, `reportAttempt` appends `{ from: resumed, to }` to `run.sessionContinuations`. `sessionOf` follows those links from the session a step would resume, taking the latest link from each session, so a later step resumes the copy that holds the last turns, whichever machine or tool made it. The snapshot publishes the links, so `resumableSessions` on this machine and the claim on the next one both see them. A step that resumed by the same id adds no link, so a run on one machine with one tool leases the same sessions as without links. +- **A workspace is a git contract on the lease. TeamAI owns the workspace contract and its git; the host owns the place. TeamAI reads the delivered commit and opens the PR outside the run lock.** `place/` prepares and finishes every place: the local driver prepares through `place.ts` in process, and `next --prepare` through `protocol.ts` for a driver that runs the protocol from the CLI; every driver's report finishes and releases its place through `managedReport` in `protocol.ts`. `preparePlace` records each place in `attempts//place.json` by phase (`opening`, `preparing`, `staged`, `setting-up`, `ready`, `finished`, `abandoned`), with the reader's snapshot in `before.json` and the branch's agent files in `overlay/` beside it, so whichever process reports the attempt finishes it. It records `opening` before git touches the copy, with whether the copy existed and, once it is open, a writer's range start, taken before any merge, so a prepare killed in git is finished by the next prepare of the attempt, which still runs setup in a copy the killed one created and keeps its merges in the step's `changes`. It records `preparing` before it touches the agent files, and copies them aside to `overlay.partial/`, renamed to `overlay/` once complete, so a prepare killed during the overlay is finished by the next prepare of the attempt, or put back by `abandonPlace`. `next --prepare` leases and prepares under the run lock, and a prepare that fails settles the attempt `failed` with its own code. `report` refuses caller `changes` and `read` for a prepared attempt, finishes its place before it reports, and leaves the place as prepare found it (`abandonPlace`) when the attempt is `released` or the run answers `lease_cancelled`. `planWorkspace` in `workspace.ts` chooses the mode. An explicit `workspace` wins. A command that omits it is read-only, even inside a copy; when it exits 0 and changed the checkout, the driver fails it `read_only_dirty`. That includes a commit that leaves the worktree clean, a switch to another branch or a detach at the same commit, a change that only stages an existing edit, another staged hunk that leaves the status mixed, a permission change on a file that was already dirty, an edit to a quoted non-ASCII path, a symlink whose target changes while the file it names stays the same, and an edit to the destination of a staged rename, and a change inside a submodule that was already dirty. An interactive prompt that sets `workspace: read-only` is checked the same way. An interactive prompt that does not may change the checkout, and those edits stay. Conflict paths, including agent files, are read without Git's quoting, so a non-ASCII name is still checked for leftover markers. A prompt that omits it continues the nearest enclosing `new` or `continue` workflow, and a `for_each` ancestor makes that `new`, so items do not share a worktree. `leaseNext` puts the contract on the lease for `new` and `continue`, and for a read-only step inside a copy (`mode: 'read-only'`, `merge: []`). That reader's branch is the one its declared `needs` wrote or read (`changes` or `read`, through items and children, but not `after` items or inherited needs), or the enclosing copy's when they touched none; needs on several branches fail `workspace_base`. It needs `continue` in `can.workspace` and the copy's owner, as `continue` does. A read-only step outside any copy gets no contract and runs in the active checkout. A step whose `workspace` is `clone: { commit, repo?, team }` opens a clone (`place/clone.ts`), which needs `clone` in `can.workspace` and `next --prepare`: `preparePlace` makes a fresh clone of that commit outside the checkout, with its own `.git`, no remotes and hooks off, and installs the team under the clone's own HOME. The steps inside it that `continue` run there, their commits stay in the clone, and `sweepClones` removes it once nothing in it is still to settle. A run whose definition opens a clone, or runs a `from` step, is isolated (`isIsolated`): its records say `isolated: true`, and it is not published, claimed, retried, or counted in member stats. `preparePlace` stages it there as a reader, and `finishPlace` checks it; a second prepare or finish returns the first one's result. `place/reader.ts` stages a reader in a copy without merges, takes its snapshot over the base overlay, and afterwards restores the overlay; when the step changed the copy or left its branch, it keeps the state on `refs/teamai/kept/` (`git add -A`, then `write-tree` and `commit-tree` on HEAD; not the stash list, which every worktree shares), and pushes each submodule commit that tree records and the read commit does not into the checkout's git dir as `refs/teamai/kept-modules//` (at any depth, without the user's push hooks or `push.recurseSubmodules`), since a gitlink does not keep its commit reachable once the copy is removed, then runs `git checkout -f `, `git reset --hard` to the commit it read, `git clean -fd`, and `git submodule update --checkout --recursive` on the submodules it changed, so a later writer never commits a reader's edits. It resets nothing when a repository among the changed paths has uncommitted or untracked files, or a nested submodule commit its parent does not record, which the keep cannot hold. The copy counts as put back only when HEAD is `refs/heads/` at that commit and `git status --porcelain --untracked-files=all --ignore-submodules=none` prints the same lines it printed when the reader was staged; ignored files are left in place. Otherwise the reader fails `workspace_base`. `preparePlace` writes a `teamai-unrestored` marker, with that status, in the worktree's git dir before the reader's overlay, and the driver removes it once the copy is put back or left untouched, so a driver killed mid-reader still leaves it. `openWorktree` in `place/created.ts` refuses later leases in a marked copy until the copy is back, when it removes the marker. A status line does not show content, so a marker whose status was not empty stays until the person who puts the copy back deletes it. `ensureWorktree` also refuses an existing worktree whose HEAD is not on its branch or a directory that is not a worktree, and prunes and recreates one whose directory was deleted. Parallel steps run git in one repository, and git writes a new worktree's entry without a lock, so a command that reads every worktree (`fetch`, `branch`, `worktree add`) beside it dies on the entry half written (`failed to read .git/worktrees//commondir`). `git` in `worktree.ts` runs such a command again, and `ensureWorktree` makes the branch before `worktree add`: `worktree add -b` makes it before it reads the worktrees, and would find it made when it ran again. The reader reports `read` (`branch`, `base`, `commit`) and never `changes`. `report` returns `invalid_report` for `read` on any other lease, and for `changes` or a success without `read` on a reader's lease. `run.copies[].commit` and fan-out `branches` follow `changes` only. `cleanup` comes from the lease's contract, so a reader that was the copy's last user removes it too. `leaseNext` holds a step back for this driver, and tries the next ready leaf, when the driver's `can.workspace` lacks that mode, or when a `continue` copy belongs to another driver (`run.copies`). It answers `wait`, with every held step's reason joined by `; `, only when every ready leaf is held back. Driver capabilities stay out of `reconcileRun`, which `reportAttempt` and the run controls also call. It fails `workspace_base` when `continue` has no enclosing `new`, or when the run has no base. `workflow start` in a git checkout records the base (commit, and the branch unless HEAD is detached); otherwise the first `next` that sends `base` sets it, and a later `base` never changes it. The local driver fails a lease whose `workspace` it cannot read with `workspace_base` instead of running it in the active checkout. The branch is `teamai///p` (`branchName`); a `[n]` in the path becomes `-n`. The first branch in `needs` order is where the step starts, and the rest are merged in that order. An item's `after` branches come first. `place/` (`created.ts`, `overlay.ts`, `worktree.ts`) adds the worktree under `.teamai/workflow-worktrees/` of the checkout the run started from (`run.checkout`), or of the driver's own checkout when it does not share that checkout's git directory, such as a separate clone (`placeRoot`), points `core.hooksPath` at an empty directory, and replaces `AGENT_PATHS` (`AGENTS.md`, `CLAUDE.md`, `.agents`, `.githooks`, and each default `toolPaths` entry's project directory, project MCP file and Copilot `.github` paths) with the run's base commit. An agent path under a symlinked parent fails `base_config` before anything is copied or removed, since both would follow the link. It then gives the worktree the inline `pull` a new worktree gets (`prepareCheckout`), which delivers the team's skills, rules, agents and MCP and records that checkout's team revision. The overlay is taken after that delivery, so those files are not the step's commit. A path the step edits is committed. A step that left HEAD off the leased branch fails `workspace_base` before anything is committed, reported as `changes`, or pushed, and the worktree stays as it left it. A path it leaves untouched is put back from the branch. A merge conflict stops the merge: a command fails `merge_conflict`, and a prompt is told the branches that remain. Snapshots and commits read submodules with `--ignore-submodules=none`, and after `git add -A`, which skips a submodule whose `ignore` is `all`, each submodule is staged by name, so `submodule..ignore` and `diff.ignoreSubmodules` hide no change from a reader's check, its keep, or a writer's commit. The driver commits as `TeamAI ` and reports `changes` (`branch`, `base`, `from`, `to`, `commits`). It pushes when `origin` exists and this step made commits, whether the step succeeded or failed, and also when a later attempt adds no commit but this branch has commits `origin` does not have, including a first push that never landed and an origin with no tracking refs yet. A failed push fails a step that succeeded with `push_failed`, and the worktree stays. `report` sets `cleanup` only when the step succeeded and no step that can still run uses that branch, judged on the run as the next `next` settles it: steps a loop runs again count, and a workflow or `for_each` step that has not started counts by the branch `plannedBranch` gives, the one around it unless it opens its own; only then does `releasePlace` run `git worktree remove --force`. It does not delete the branch, and it never resets, cleans, or switches the active checkout. A command that starts its own sandbox, such as `docker`, stays read-only. What it writes does not feed `new`, `branches`, or a later step. `session: continue` and `session.from` are step reads. The engine puts `{ tool, id }` on the lease, with `earlier`, the sessions that one continued, newest first, and never copies a session. The driver resumes Claude and Codex by that id, copies another tool through `AgentSession.copy`, restores a copy from the team session archive when neither works, and tries each `earlier` session the same way before it starts fresh. OpenCode has no resume flag, so it starts fresh. A driver that continued the lease's session or an earlier one, by that id or in a copy, echoes the one it continued as `resumed` in its report. One that started a new session leaves `resumed` out. `reportAttempt` answers `invalid_report` for a `resumed` on a step that resumes no session, and for one that no pass of the run reported and no link continued (`knowsSession`). When the reported `tool` and `session` differ from `resumed`, `reportAttempt` appends `{ from: resumed, to }` to `run.sessionContinuations`. `sessionOf` follows those links from the session a step would resume, taking the latest link from each session, so a later step resumes the copy that holds the last turns, whichever machine or tool made it. The snapshot publishes the links, so `resumableSessions` on this machine and the claim on the next one both see them. A step that resumed by the same id adds no link, so a run on one machine with one tool leases the same sessions as without links. - **A driver pushes the commits it reports, and fetches the commits it is leased.** Every driver keeps these two git rules, so a run another driver or another machine continues finds each commit on `origin`. Whenever `origin` exists, a driver pushes the commits it reports in `changes` before it reports them, failed steps included; `finishWriter` in `place/writer.ts` pushes for every outcome. A push that fails still reports the commits. A step that succeeded fails `push_failed`. A step that failed keeps its own failure, and the push sentence is added to its message: a retry must fix the step first, and a `push_failed` in its place would hide that cause from the person and from the run record. Before it starts from, merges, or applies a lease commit (`start`, each `merge[]` entry, `configCommit`), a driver checks it with `git cat-file -e ^{commit}`. For a missing one it fetches `+refs/heads/teamai//*:refs/remotes/origin/teamai//*`, and checks again, since a refspec fetch that matches no branch exits 0. A fetch by commit id is not used: a server can refuse it, and a fetched commit no ref holds can be collected later. A commit still missing fails `workspace_base` naming the commit and its branch, before anything merges or runs, so it is never a file-less `merge_conflict` or a prompt told to resolve one. `openWorktree` in `place/created.ts` does this through `fetchMissingCommits`, for the local driver and for `next --prepare`, and `adoptedMismatch` in `place/adopted.ts` for a place the host made. `workflowGitConformanceScenarios` in `conformance/git.ts` checks both rules against a bare origin, and `workflow-git-conformance.test.ts` runs them against the local driver. - **`next` delivers pull requests, outside the run lock.** A `deliver: pr` step whose agent succeeded becomes `delivering` (`startDelivery`), holding `delivery` (`branch`, `head`, and a kept `reported` URL). It is neither `stopped()` nor `settled()`, so dependents wait, and `startedItems` does not count it, since `max_parallel` bounds agents. Every `leaseNext` calls `drainDeliveries` after an optional claim and before its locked body, whichever driver called it. The drain takes `deliveryLockPath` once; while another process of this machine holds it, the call delivers nothing and `next` answers `wait.delivering`, which the local driver polls. Each round picks a job under the run lock (`nextDelivery`, then `unresolvedCreate` in `delivery-ledger.ts`), runs `deliverPr` or `findDelivered` in `pr-delivery.ts` with no lock held, and applies the outcome under the run lock with `completeDelivery` or `resolveCreate`, which read the immutable job rather than the step. `deliverPr` requires origin's tip of the branch (`ls-remote`) to be the reported head or the latest `copies[branch].commit` a report recorded, and a recorded later tip to contain the reported head (`merge-base --is-ancestor`), else `branch_rewritten`. A tip past the recorded commit that still contains it is `branch_ahead`. `completeDelivery` keeps the step `delivering` while a step that writes the branch is leased (`branchWriterLeased` in `engine.ts`), since that step may push before it reports, or when `copies[branch].commit` is no longer the job's `recordedHead`, since a report recorded a commit while the drain read the branch. The drain does not pick it again in that call. Otherwise no report of that tip can come, and the step fails at once with the `branch_ahead` message, which says the tip holds the checked changes and later ones no report recorded. The same two conditions hold back a pull request: the step stays `delivering`, and one the job found or was told of gets no receipt, while one it opened does. The `branch_moved` and `branch_missing` remedies name the recorded commit. `deliverPr` fetches and diffs the selected tip against the run base on the run's checkout, running every git call that reaches origin within `DELIVERY_TIMEOUT_MS` and with `NON_INTERACTIVE_GIT_ENV`, and rejects protected paths (`protectedPaths` in `worktree.ts`: `AGENT_PATHS` plus `DELIVERY_ONLY_PATHS`) before it accepts a reported URL, looks up or creates. When a recorded later tip changes protected paths, it also diffs the step's own head: a clean own head is `later_protected_path`, a `delivery_failed` that names the later commit, so a retry delivers again instead of running the agent. Its remedies are the ones the run accepts: put origin back at the step's own head, or retry the workflow step whose copy the branch is (the job's `openedBy`). The job of a workflow step carries `childWork` (`childWorkOff` in `workspace.ts`): the head of each step inside it that succeeded on another branch that no step inside it delivers. Each must be an ancestor of the tip, else `child_work_left_out`, since the pull request would leave that work out. A workflow step whose branch holds nothing beyond the run base opens no pull request, so `finishStep` fails it with the same message when such work exists (`workLeftUndelivered`). It looks the branch up before the create and again after it, and reads origin's tip again between that first lookup and the gate. Only a lookup's URL is ever recorded. A post-create lookup that fails, or that finds none after a create that answered, fails delivery with `create_unconfirmed` and keeps the unresolved entry for a later lookup. It treats a found URL on another host than the origin's pull requests as a fault. Its gate, `markCreating`, takes the run lock right before the create: it reconciles, refuses unless the same pass of the same commit still delivers, answers `busy` when `copies[branch].commit` is no longer the job's `recordedHead` or while a step that writes the branch is leased (`branchWriterLeased`), and lists the request in `run.creating`. `nextDelivery` passes over a step while a step that writes its branch is leased (`branchWriterLeased`), so no git or provider call runs for it, and no reported URL or found request is accepted, until that step's commit can be checked. The gate's check covers a lease granted while the delivery runs. Timeouts (`expire` in `limits.ts`) and budgets settle in `reconcileRun`, which both `nextDelivery` and this gate apply, so no delivery is looked up, opened or accepted once a limit stops the step or the run. A new condition that stops a run belongs there too. A run lock it cannot take is `busy` too, as is a tip that moved during the lookup, and neither is a refusal: nothing is listed or opened, the step stays `delivering`, and the drain stops for that call, so the next one checks the branch again. A new receipt on a run that already ended queues the end record under the run lock, and it publishes after the drain releases the delivery lock. In `reportAttempt`, the lookup that vets a reported `changes.pr_url` (`reportedPrMismatch`) is the only delivery I/O, and it runs before the run lock. On the origin's host, a lookup that fails or answers an unsafe URL drops the reported URL. No provider code merges; `teamai/no-provider-merge` enforces this in `src/providers/**` and delivery. - **A pull request is reused by provider lookup, never by a stored receipt.** `delivery-schema.ts` owns the schemas of delivery receipts and unresolved creates, and `CommitIdSchema`. It imports only `zod` and `pr-url.ts`, so `snapshot.ts` and the run store load no delivery code or provider, which `teamai/workflow-boundaries` checks through the runtime dependency graph. `delivery-ledger.ts` owns their pure updates. The engine passes `reconcileRun` into the ledger and keeps the drain and create gate. Three fields keep what delivery did. `run.deliveries` is an append-only receipt list, deduplicated by URL, that only `recordReceipt` writes, for `completeDelivery` and `resolveCreate`. `deliveryTarget` in `workspace.ts` reads which branches delivered, `buildEndRecord` in `records.ts` reads it as `prs` in list order, and the snapshot publishes it. `copies[].from` is the branch a copy grew from: `rememberCopy` writes it once, at the copy's first lease, from the contract `planWorkspace` ordered (`after`, then `needs`, then inherited needs), before the copy has a commit. `deliveryTarget` walks it to the nearest branch with a receipt, else `run.base.ref`, and the snapshot publishes it and refuses one outside `teamai//`. `run.creating` lists each request TeamAI asked a host to open and holds no receipt for. `markCreating` appends it, `completeDelivery` removes it with the receipt of a delivered outcome and keeps it on a failed one, and `resolveCreate` applies a lookup of it: found appends the receipt and removes it, unsafe removes it immediately, and none or failed removes it after its second none or third failed answer. Each entry keeps total, none and failed lookup counts in `lookups`, set to zero by the gate each time it opens, including for a request listed before, and advanced only by `resolveCreate`. `unresolvedCreate` picks each at most once per drain, never in the drain that created or failed it, after the delivering steps, in an ended run too, and `findDelivered` reads only the origin and `findOpen`. It touches no step, so a retry, cancel or archive cannot lose the request, and a lookup cannot settle or fail a step. A delivering step whose request is listed still looks up before it creates again. The snapshot publishes the list, so a claimer looks the requests up. @@ -32,12 +32,14 @@ Each rule below spans more than one file. The rule names the code that owns it, - **`run.steps` is replaced, never changed in place.** Its type is `readonly`, so `addItems`, `insertChildren` and `archiveUnder` add or remove instances by assigning a new array. An instance's `id` is `readonly` too, so an instance cannot move to another scope's group. `statesIn` in `scope.ts` keeps the instances of every scope per array, in a `WeakMap`, so settling a scope reads its own instances without scanning every step. A new array starts a new index, except that `insertChildren`, which adds the instances of a workflow a step starts, carries the old index over. - **A pull check that changes what a file means carries a `code`.** `dropsKeptVersion` in `cache.ts` removes the cached copy when a diagnostic has a `code`. An older definition then cannot keep running with the new meaning. A diagnostic without a `code` keeps the last valid version, but only after `checkReferences` checks that version again against what the workflows it runs settle to in the same sync. - **The workflow cache holds only what one sync settled.** `checkReferences` checks each file's version against the versions the other files settle to, and `dropUnsettled` removes every other cache entry. `startRun` relies on this: it reads the workflows a run uses from the cache and does not check them against each other again. A new way for a file to keep or lose its version goes through `checkReferences`. +- **`src/workflow/` never imports `src/benchmark/`.** The engine and its drivers know only generic primitives: clone places, `workflow: { from }`, `finally`, `on_limit: end_turn`, `requires.strict` and `sampling: all`. A benchmark is the default workflow `teamai-benchmark`, whose steps run the `teamai benchmark` verbs as commands. Rule 6 in `src/__tests__/workflow-module-boundaries.test.ts` fails any import from `src/workflow/` into `src/benchmark/`. +- **What a run runs is admitted once, in `admission.ts`.** `admitWorkflow` takes in a cached workflow for `startRun`, and the output of a step whose `output` has a `workflow` field when that step reports. It parses such a bundle as workflow files, its named workflows passed as the files a step may run; pins the team scorers its steps list at the bundle's `teamRevision` (`withTeamRevision`), else from the cache, and its generated scorers, whose names hold a colon, as the bundle gives them; and refuses a secret value and a delivery it cannot make. `reportAttempt` admits before it takes the run lock, and on a refusal fails the producer `invalid_output`. It keeps the whole output in `run.admitted` under the producer's path, hashed, never masked or replaced. A `from` step runs it: `stepWorkflow` reads what a step's `workflow` names, and `workflowRunBy` what it runs in a run; every reader goes through them. - **A teamai default is resolved, never cached, and a run records which definition it ran.** `loadCachedWorkflow` and `listCachedWorkflows` in `cache.ts` return the cached team file of a name or alias first, else the default the package ships in `workflows/` (`DefaultWorkflows` in `src/default-workflows.ts`). Every default's name starts with `teamai-`, and its generated skill keeps that name (`workflow-skills.ts`), so prune and uninstall look at `workflow-*` and `teamai-*` directories, by the marker only. A sync never writes a default, and `checkReferences` lets a team file run one. `resolveStart` picks the scope a start runs in: the project's team file, then the user scope's, then the default in the project's scope; `readyStartConfig` and the friction reminder's `teamai-share-learnings` offer go through it. `startRun` writes `run.origin` (`originOf`: `team`, or `default` with the teamai version); the records, the snapshot and the dashboard read it, and `sourceOfRun` gives describe its `source`. A claim of a default run (`definitionsOf` in `handoff.ts`) uses only the default this install ships, and refuses with `definition_mismatch` when its bytes differ from the ones the run ran. - **The items of a `for_each` step settle like a workflow.** When settling reaches a `for_each` step, `advanceInstance` evaluates its list and `addItems` puts an instance per item right after the step; the step is `running` until every item settled. `settleRunning` settles a running `for_each` step through `settleItems`, the way it settles a workflow step through `settleScope`. An item waits for the items its `after` names and is skipped when one of them stopped (`afterStanding`). It starts only while fewer than `max_parallel` items are leased or running (`startedItems`, counted once per scan). A failed item cancels nothing; the step fails once every item settled (`itemsSettled`). The rules that decide the items are in `fan-out.ts`, the settling in `engine.ts`. Pull rejects `for_each` with `loop`, `end` or `break` (`loopProblems`). - **`item` is bound per instance, in the engine and in pull alike.** `contextOf` gives an item its own value and any other instance its workflow's (`Scope.item`), which `childScope` passes into the inline workflow an item runs. A `for_each` step's own `if` and `for_each` therefore read the enclosing item, never their own. Pull checks the same binding: `ReadSite.item` marks the templates of a `for_each` step, and `ReadScope.item` a workflow that runs per item. A template reads `item` only where it is bound: `parseTemplate` takes the binding, which pull decides from the step and the engine from whether the context holds an item, so elsewhere `{{ item }}` stays text in both. A new place that reads `item` must decide its binding in both. - **Step ids are paths.** A step inside a sub-workflow has an id such as `ship.check.classify`, an item of a `for_each` step one such as `build[2]`, and a step inside the workflow an item runs one such as `build[2].verify`. `parseStepPath` in `step-path.ts` reads them all; don't split ids by hand. A child step exists in the run only after the engine reaches it. To find a step's definition, use `stepDefinitions` in `scope.ts`, which reads the definitions alone, so it also finds the step of an archived pass. `locateStep` in `engine.ts` also gives the step's scope, and needs the instances along the path. Don't look a step up by top-level id. -- **What a step requires is decided once per lease, in `negotiate`.** `offerTerms` in `engine.ts` builds the offer for the ready loop and the hand-back alike, and `negotiate` in `requirements.ts` answers grant, hold or fail. `ENFORCEMENT` classifies every requirement as strict, which holds the step for a driver whose `can` declares it (`wait.held`), or best effort, which grants it and records the gap in `unenforced` on the lease, the step and the record. A copy is used once an accepted report shows its step ran there: a success, an `exitCode`, `changes` or `read` (`run.copies[].used`); a staging or setup failure leaves it unused. Every lease on a copy in a workflow that declares `setup` carries it, so `can.setup` holds the lease, and the driver decides whether setup runs: when it creates the copy, or finds its setup unfinished (the marker `openWorktree` in `place/created.ts` leaves). A held hand-back cancels the attempt and releases the step; on an unused copy the driver also gives the copy up, so a capable driver can take the step, unless the driver still holds another lease in it, while a used copy keeps its owner, whose work lives there. A granted hand-back keeps its driver as the copy's owner (`rememberCopy`). The member's `workflows.allow` holds before any capability. Env values never enter the run: the lease carries key names in `envScope` (and `setup.envScope`), and the driver resolves them. Deadlines and budgets settle in `reconcileRun(run, now)`, which runs `expire` (`limits.ts`) first; the run deadline and the run's budget count from `run.limitsFrom`, which a retry sets when it reopens the run or finds it ended, or else from `createdAt` with nothing spent; a workflow step's budget counts from the `limitsFrom` its `start` records (`startLimits`); a `for_each` step spends its budget over all its items together (`itemsSpent` in `settleItems`), and each item keeps only the budget of the workflow file it runs. An attempt the engine stops goes through `stopAttempt`, which records it in `run.stoppedAttempts` with its step, driver and stop time, whatever its step becomes later; a late report from that driver adds its tokens once (`countLateTokens`), and a budget whose `limitsFrom` started at or after the stop takes them into its baseline, so it does not pay for them. Known secret values are masked at the three exits, `updateRun`, `enqueueRecord` and `writeEnvelope`, each of which takes the secret set as a required argument. The set holds the value of each key `env/secrets.yaml` declares, wherever this machine resolves it from, and each environment value whose name looks secret; masking follows the team's declarations, not what a run requires. A command resolves it once and passes it on: `startRun`, `withRun` and the run controls require it, so what the envelope masks and what the run file masks agree; the local driver also masks an attempt's `result.json` before it reports, as text when it is not JSON, and every attempt's result file when it stops driving the run, so an attempt it never reported keeps no secret either. +- **What a step requires is decided once per lease, in `negotiate`.** `offerTerms` in `engine.ts` builds the offer for the ready loop and the hand-back alike, and `negotiate` in `requirements.ts` answers grant, hold or fail. `ENFORCEMENT` classifies every requirement as strict, which holds the step for a driver whose `can` declares it (`wait.held`), or best effort, which grants it and records the gap in `unenforced` on the lease, the step and the record. `requires.strict` makes a level's `isolation` and `network` strict and a floor (`mergeRequirements`): an inner level below it is raised to it. A strict `isolation: sandbox` also needs `can.env: true`, and its lease withholds `SANDBOX_WITHHELD_ENV` from setup, the step and the clone's git, whatever `requires.env` names. A local start with a step the local driver could never take for a strict requirement is refused `requirement_unmet` before the run exists (`createRun`). A prompt with `permissions: unattended` is admitted only under a strict `isolation: sandbox` floor, as the scopes merge requirements (`unattendedUnsandboxed` in `admission.ts`): a cached start from its root, refused `unattended_unsandboxed`, and a bundle under the requirements of each `from` step that runs it (`fromRequirements`), failing `invalid_output`. It is held for `can.unattended`, a strict requirement, and its lease carries `permissions`; each step adapter in `tool-launch/adapters.ts` declares its tool's unattended arguments or `'none'`. A copy is used once an accepted report shows its step ran there: a success, an `exitCode`, `changes` or `read` (`run.copies[].used`); a staging or setup failure leaves it unused. Every lease on a copy in a workflow that declares `setup` carries it, so `can.setup` holds the lease, and the driver decides whether setup runs: when it creates the copy, or finds its setup unfinished (the marker `openWorktree` in `place/created.ts` leaves). A held hand-back cancels the attempt and releases the step; on an unused copy the driver also gives the copy up, so a capable driver can take the step, unless the driver still holds another lease in it, while a used copy keeps its owner, whose work lives there. A granted hand-back keeps its driver as the copy's owner (`rememberCopy`). The member's `workflows.allow` holds before any capability. Env values never enter the run: the lease carries key names in `envScope` (and `setup.envScope`), and the driver resolves them. Deadlines and budgets settle in `reconcileRun(run, now)`, which runs `expire` (`limits.ts`) first; the run deadline and the run's budget count from `run.limitsFrom`, which a retry sets when it reopens the run or finds it ended, or else from `createdAt` with nothing spent; a workflow step's budget counts from the `limitsFrom` its `start` records (`startLimits`); a `for_each` step spends its budget over all its items together (`itemsSpent` in `settleItems`), and each item keeps only the budget of the workflow file it runs. Under `on_limit: end_turn` a step's own `timeout` is no deadline (`ownDeadline`) and a workflow step's own `budget` no scope budget (`ownBudget` in `scope.ts`): a prompt's `timeout`, `max_turns` and `budget` travel as lease `limits` (`turnLimits` in `limits.ts`), only to a driver with `can.end_turn`, which must report `durationMs` and names the one it stopped at in `limit`; any other driver gets the `on_limit: fail` lease and the pass records `unenforced: on_limit`. A hand-back decides that form again for the driver's current `can` (`turnTerms` in `engine.ts`). The step's own timeout counts from the first lease (`lease.leasedAt`), so a restart never extends it: the fail lease keeps that deadline, and an end_turn lease's `limits.timeoutMs` is what is left of it. A turn with none left times out through `expire`, as any lease past its deadline does. A workflow step's `timeout` and `budget` are the allowance its scope carries (`scope.allowances`): each inner prompt's lease `limits` is what is left after the prompt passes inside its current pass (`allowanceLeft`), a `limit` reported at that remainder marks it `allowanceSpent` (`spendAllowances`), and `settleScope` then skips what is still pending under it. An attempt the engine stops goes through `stopAttempt`, which records it in `run.stoppedAttempts` with its step, driver and stop time, whatever its step becomes later; a late report from that driver adds its tokens once (`countLateTokens`), and a budget whose `limitsFrom` started at or after the stop takes them into its baseline, so it does not pay for them. Known secret values are masked at the three exits, `updateRun`, `enqueueRecord` and `writeEnvelope`, each of which takes the secret set as a required argument. The set holds the value of each key `env/secrets.yaml` declares, wherever this machine resolves it from, and each environment value whose name looks secret; masking follows the team's declarations, not what a run requires. A command resolves it once and passes it on: `startRun`, `withRun` and the run controls require it, so what the envelope masks and what the run file masks agree; the local driver also masks an attempt's `result.json` before it reports, as text when it is not JSON, and every attempt's result file when it stops driving the run, so an attempt it never reported keeps no secret either. - **A HITL request waits before any action or end.** `settleScope` checks loop exhaustion first, renders a HITL request message, and moves an unanswered step to `waiting`. Waiting is neither stopped nor settled. An answer moves it to pending without a revision bump. Approve no skips, choose succeeds, approve yes continues. An end-only step with HITL reaches its end only after yes. The shared `templatedTexts` makes HITL reads participate in readiness. Only `driver.can.hitl` declares HITL. `parseNextPayload` reads `driver.can` into `NextOptions.can`; stored `DriverIdentity` and leases contain only kind, machine and workspace. Busy polls never answer waiting HITL requests through the no-HITL rules. Only a driver declaring HITL receives a waiting request; a busy poll without HITL receives wait. An interactive step is a requirement, not a gate: `negotiate` holds it for a driver without HITL, on new leases and hand-backs alike. The JSON payload parser reads `autoApprove` only from a driver declaring HITL, which sends it when the person chose auto-approve for the run; `answerOptionalGates` then answers optional approvals and choices with a default, recorded `auto_approve`, and never reads `workflows.auto_approve`. `respond --json` uses the same identity schema as next, reports unread fields through ignored, and records a driver UI's declared person answer with channel driver. The dashboard answers through the same `respondToHitl` with channel dashboard, always sending the pass it showed (`answerFromDashboard` in `src/dashboard/workflow-runs.ts`). `respondProvenance` in `run-controls.ts` decides `agent` from where the answer came in and the process that sent it: a terminal without a TTY, `respond --driver skill`, and any process with an `AGENT_SESSION_ENV` variable give `agent: true`, which a required approval refuses with `agent_in_path`. A process with `TEAMAI_WORKFLOW_ATTEMPT` answers nothing: `respond` refuses it with `nested_workflow` first (below). Answer driver identity stays local; records publish only choice, channel and agent. - **An agent session starts or continues a workflow only as its `invoke` lets it.** `start` in `protocol.ts` checks the definition's `invoke`, absent meaning `deny`, against the caller's agent evidence (`agentEvidence` in `attempt-marker.ts`), for every driver, before the run exists: `TEAMAI_WORKFLOW_ATTEMPT` answers `nested_workflow` (below), and with an `AGENT_SESSION_ENV` variable, or from a `--driver skill` call, which only a coding agent makes (`skillContext`), `deny` answers `invoke_denied`, and `ask` answers `confirm_needed` unless the start passed `--confirmed`, the agent's word that the person agreed, which the CLI cannot verify, and `unattended_denied` in an unattended run (`InvokeCaller.mode`), `--confirmed` or not, with describe's R17 wording (`unattendedReason`). A process without evidence, such as the person's terminal or a CI job, starts any workflow; `CI` exempts nothing. `workflow resume`, with either driver, `next --driver skill --takeover`, and a `next` payload's `claim` make the same check through `resumeRefusal`, of the definition the run carries (this machine's run, else the one on teamai-reports), before anything is claimed or changed; `--driver skill` counts there as for a start, and the session whose skill.json here names it as the run's owner is not checked again (`drivesRun`), as its start or takeover passed. A claim answers `confirm_needed` with the `resume --claim --confirmed` literal, since a payload cannot carry the person's yes. - **No step starts or drives a run.** Every driver gives the processes it starts for an attempt `TEAMAI_WORKFLOW_ATTEMPT=/`: the local driver its commands, setup (`runPlaceSetup`) and tools, the skill driver its runners, and through them their commands and setup. `workflow-commands.ts` answers `nested_workflow` before it reads any run (`attempt-marker.ts`): to every run-changing subcommand (`start`, `next`, `report`, `respond`, `retry`, `cancel`, `resume`, `publish`) from a process with the marker, except `report --driver skill --attempt ` whose marker names exactly `/`, the runner of that attempt; and to `start` from a workflow's place, a directory under `.teamai/workflow-worktrees` or a checkout whose HEAD is on a `teamai//…` branch, the only trace a place the host made keeps. `start` and `respond` in `protocol.ts` refuse the marker too, so a client in process answers as the CLI does. Only `start` reads the place, since a step's child and a `--json` driver report the attempt from it. A step command that reports its own attempt passes as the runner and gets `held`, since the runner holds the attempt's `runner.lock`. @@ -45,7 +47,7 @@ Each rule below spans more than one file. The rule names the code that owns it, - **The local driver keeps tools and questions interruptible.** A single question joins command and prompt exits in the driver's race. An interactive tool owns the terminal alone: no question opens while it runs, and starting one aborts and closes an open question. A poll closes the question when its own step is leased or a different step or pass waits; an unrelated lease leaves it open. It polls next every second while asking, every two seconds while tools run, and aborts a replaced HITL request or cancelled prompt. `src/tool-launch/process-stop.ts` shares process stopping between commands and adapters. Headless tools detach and stop with SIGTERM; interactive tools stay attached and stop with their captured descendant tree. Foreground interactive signals reuse env-exec's terminal detection and signal policy: when the terminal reaches the child, teamai ignores SIGINT and SIGQUIT, while SIGTERM and SIGHUP are forwarded and stop the driver. Interactive adapters inherit stdio and accept an optional result file. Repository edits are kept for interactive steps; only headless prompts run the read-only check. A nonzero interactive exit without a result reports its exit status. `leaseNext` records the identity of the latest driver that called next in `RunState.driver`, without capabilities. Resume uses this machine's open local lease workspace, then run.driver.workspace for a local driver on this machine, then cwd. Controls pass run.driver to finishRun so end records retain the driver kind. - **One local driver owns a run on this machine.** `takeRunLock` in `store.ts` takes `driverLockPath` with `acquireLock`, waiting at most the caller's `waitMs`, and releases it in finally; the `RunLock` it passes is the proof a caller holds it. It answers `held` with the body's value or `busy`, so no answer of a body reads as a busy lock; `withRunLock` turns `busy` into the `driver_running` error for a body that answers an envelope. `withDriverLock` in `local-driver.ts` takes it without waiting. `runLocalDriver` drives inside it, and `workflow resume` also claims inside it, so a claim never lands under a driver already running the run. A live holder returns `driver_running`, including its pid, before anything is claimed or driven; the existing lock machinery reclaims dead owners. The driver lock spans tool execution and questions, while `withRun` holds a different run lock only during one engine operation. Neither lock's scope is extended by the other. Every HITL decision without a HITL driver logs once when the engine changes the step. HITL replies carry one literal `respond` command template, which drivers display unchanged. -- **Judge work never comes from `workflow next`.** `scoring.ts` computes open judge work on every `teamai score --next` from facts already on disk: the run files, the scorers `startRun` pinned in `run.scorers`, the `installId` that `reportAttempt` stamps on each pass it settles, the `within` stamp on archived passes, and the labels, attempts and on-demand requests (`teamai score --run`) under `/scores/`. Nothing else is queued. The engine has no scoring code path, and scoring never writes a run file, so a run's counts, tokens, failure and `endedAt` cannot move and it settles without waiting. A judge lease settles only through `teamai score`: `--lease --submit`, or `--lease ` and `--run`, which run the local judge and settle through the same submit path. `workflow report` answers `judge_lease`. A driver never lists a judge ref in `workflow next`'s `running`, which holds step attempts only, and `leaseNext` answers `judge_lease` before it changes anything when one does: judge refs go in `score --next`'s `running`, which renews them. +- **Judge work never comes from `workflow next`, and only `sampling: all` scorers hold a run open.** `scoring.ts` computes open judge work on every `teamai score --next` from facts already on disk: the run files, the scorers `startRun` pinned in `run.scorers`, the `installId` that `reportAttempt` stamps on each pass it settles, the `within` stamp on archived passes, and the labels, attempts and on-demand requests (`teamai score --run`) under `/scores/`. Nothing else is queued. Scoring never writes a run file, so a run's counts, tokens and failure cannot move. A run settles without waiting unless a step's scorers set `sampling: all` (a benchmark trial's prompt, or any workflow step). Every pass of such a step is judged whatever its scorers sample, and a run that reaches a `succeeded` end gets `wait` with `judging`, the count of those entries still open or leased (`verdictEntries` in `scoring-entries.ts`, which imports neither the engine nor the local driver), until they settle. A timeout, a limit or a cancel ends the run as before. There is one judge barrier per run, in this order: the steps settle, the judges run, the verdicts freeze, the `finally` steps run, the run finishes. `finishRun` owns that order for both finish paths: it freezes every such entry into `run.verdicts`, labelled or unscored (its judge failure code, `expired` or `run_ended`), and removes the evidence collected for those passes, before it enters the `finally` phase, so nothing a `finally` step does changes a verdict. After that (`verdictsFrozen`) the entry is not offered, and a submit for it answers `entry_closed` and releases its lease. The evidence of a step that runs in a clone is staged into `scores/evidence/subjects/` before the clone can go, and that is all its judges read, so no judge entry of such a step opens before its evidence exists: a prompt's by `stagePassEvidence` in `place/place.ts`, which `settleManaged` calls once the place is finished and before the engine accepts the report, and which removes it again when the engine does not accept a success; a workflow step's by `stageSettledEvidence` in `place/clone.ts`, which `withRun` in `engine.ts` calls for each step the operation settled `success` before it saves the run, as such a step settles in a later `next`, once its last step does, so neither its judge entry nor a dependent is visible before its evidence. `sweepClones` removes a clone once its run is not running or no step in it is still to settle, with the evidence staged for each pass there that did not succeed, since a report killed before the engine answered leaves evidence no verdict freezes. The local driver judges its own run's entries with `judgeRunEntries` while `next` waits, and submits a failed judge at once. A judge lease settles only through `teamai score`: `--lease --submit`, or `--lease ` and `--run`, which run the local judge and settle through the same submit path. `workflow report` answers `judge_lease`. A driver never lists a judge ref in `workflow next`'s `running`, which holds step attempts only, and `leaseNext` answers `judge_lease` before it changes anything when one does: judge refs go in `score --next`'s `running`, which renews them. ## Tests @@ -76,7 +78,7 @@ Test parsing through `parseWorkflow` and `syncWorkflowDefinitions`. Test the eng | R17 | none | describe | run mode `unattended`, R1, R16 and R23 do not apply, and `attendedReason` names what needs a person: `invoke: ask`, an interactive step, a required approval, or a choice without a default | none | none | describe `ask.kind: denied` with that reason, no literal | | R18 | none | describe | run mode `unattended`, and R1, R16, R23, R17 and R27 do not apply | none | none | describe `ask.kind: start` with the start of R3's recommended option under `--auto-approve`; `{inputs}` kept and `inputs` listed when the workflow has inputs | | R27 | none | describe | the session names no tool, the definition has a step whose model or judging needs one (`needsTool`), and R1, R16, R2, R23 and R17 do not apply | none | none | describe `ask.kind: tool` with the reason, the describe literal with `--tools default='{tool}'` for the agent to fill, and the ids of the tools teamai knows | -| R8 | none | start `--driver skill`, or R19's input | the definition has a local route; for R19's input, the run's own definition, once R20 and R21 do not apply, or, in run mode `unattended`, as the run stands before the claim | none | none | error `needs_local_driver` with the reason, next null; for R19's input it names the person's `teamai workflow resume ` literal, and the run stays readied here, or, unattended, as it was: nothing is claimed | +| R8 | none | start `--driver skill`, or R19's input | the definition has a local route or a benchmark trial; for R19's input, the run's own definition, once R20 and R21 do not apply, or, in run mode `unattended`, as the run stands before the claim | none | none | error `needs_local_driver` with the reason, next null; for R19's input it names the person's `teamai workflow resume ` literal, and the run stays readied here, or, unattended, as it was: nothing is claimed | | R25 | none | start `--driver skill` | run mode `unattended`, R8 does not apply, and `attendedReason` names what needs a person, as R17's | none | none | error `unattended_denied` with that reason, next null: no run is created | | R9 | none | start `--driver skill` | `checkStartAnswers` answers `confirm-needed`, and R8 and R25 do not apply | none | none | error `confirm_needed`, next: the describe literal | | R10 | none | start `--driver skill` | `checkStartAnswers` answers `ok` | `WorkflowClient.start`; write skill.json: identity `{skill, machine, run checkout}`, invocation, options (`--parallel`, the HITL mode, the caller's run mode), host `pending` when the definition has a prompt step or the session judges the run, with whether it has a prompt step, owner := the caller's session, working in the tool its `--tools default=` names, else the one its host names; judging on with the session tool when describe judges here, or `sessionJudge` with it when describe judges by the session, and the start has no `--no-judge` | host when it has a prompt step or the session judges it, else drive | the turn of a tick | @@ -139,7 +141,7 @@ Test parsing through `parseWorkflow` and `syncWorkflowDefinitions`. Test the eng | Row | State | Input | Guard | Effects | Next | Action | |---|---|---|---|---|---|---| -| C1 | preparing | runner: prepared | job `command` | run the command outside the lock with `{...stepEnvironment(env, envScope).env, ...lease.env, TEAMAI_WORKFLOW_ATTEMPT}` (`process.env` for a lease without `envScope`), output masked into `output.log` | running | | +| C1 | preparing | runner: prepared | job `command` | run the command outside the lock with `{...stepEnvironment(env, envScope).env, ...lease.env, TEAMAI_WORKFLOW_ATTEMPT}` (`process.env` for a lease without `envScope`), the running teamai first on PATH, output masked into `output.log` | running | | | C2 | preparing | runner: prepared | job `prompt` | `pickChildModel`; `writeChildBrief` | offered | `dispatch` in each foreground turn until acked | | C3 | preparing | runner: prepared | job `prompt`; `pickChildModel` fails `model_unavailable` | write-ahead `failed` with that failure | reporting | | | C4 | preparing | runner: place busy, behind b | | `behind` := b | queued | | diff --git a/docs/guide/benchmarks.md b/docs/guide/benchmarks.md new file mode 100644 index 000000000..4ddd0bec0 --- /dev/null +++ b/docs/guide/benchmarks.md @@ -0,0 +1,551 @@ +# Benchmarks + +> Part of the [TeamAI CLI Usage Guide](../usage-guide.md). + +```text +teamai benchmark run fix-auth --sandbox +``` + +A benchmark suite compares tools or models on your team's own tasks. One run gives every task to every arm, a few times each, checks each result, and prints a table. Trials run only on a machine declared a sandbox, such as a CI runner or a container. + +A benchmark run is a run of the workflow `teamai-benchmark`, which teamai ships and a team can replace. See [The benchmark workflow](#the-benchmark-workflow). + +## Write a suite + +A suite is a directory in the team repo: `benchmarks//suite.yaml`, and one file per task in `tasks/`. + +```yaml +# benchmarks/fix-auth/suite.yaml +schemaVersion: v1alpha1 +name: fix-auth +description: Fix bugs in the auth service. +repo: https://github.com/acme/auth-service.git +defaults: + tool: claude +arms: + - name: claude + - name: codex + tool: codex + - name: strong + model: strong +repetitions: 3 +timeout: 45m +setup: + command: npm ci +``` + +| Key | Value | +| --- | --- | +| `repo` | The project the tasks start from: an https, ssh or git URL without credentials. | +| `defaults` | The arm the others are compared with: `tool`, and optionally `model`, `team` and `workflow`. Not a workflow's `defaults`. | +| `arms` | 1 to 6. Each changes at most one `defaults` key, and at most one changes none. | +| `matrix` | Instead of `arms`: lists of values that teamai crosses into arms. See [Cross tools with team assets](#cross-tools-with-team-assets). | +| `team` | The team assets an arm runs with. See [Arms on other team assets](#arms-on-other-team-assets). | +| `workflow` | A team workflow each trial runs on the task, instead of one prompt. See [Arms that run a workflow](#arms-that-run-a-workflow). | +| `model` | A model alias from `models/aliases.yaml`, resolved for each arm's tool when the run starts, or a model name. | +| `repetitions` | 1 to 10, 3 by default. | +| `parallel` | 1 to 32 lanes that baselines and trials run in, 1 by default. See [Parallel lanes](#parallel-lanes). | +| `timeout` | Required: the whole run's. | +| `trial` | Optional limits of every trial's prompt. See [Trial limits](#trial-limits). | +| `setup` | Optional, a [workflow's `setup`](workflow-advanced.md#requirements-and-limits). Runs in every fresh clone before the tool or the check. | +| `scorers` | Optional [scorers](scorers.md) that judge every trial. A name with no valid `scorers/.md` fails `teamai review --branch`, `teamai pull` warns, and start refuses the suite. | + +### Arms on other team assets + +An arm runs with the team revision this checkout pulled unless it sets `team`. Measure what the team's assets add, a branch before you merge it, or one skill: + +```yaml +defaults: + tool: claude +arms: + - name: pulled # the revision you pulled + - name: bare + team: none # no team assets at all + - name: next + team: next # the team repo's branch next + - name: no-auth-tokens + team: + without: [skills/auth-tokens] # the pulled revision, without this skill +``` + +| `team` | The arm runs with | +| --- | --- | +| `pulled`, or no `team` | The team revision this checkout pulled. | +| `none` | No team assets. Start refuses it on a machine whose user-scope teamai install delivers to the arm's tool, since its trials would read those assets in your home. | +| A branch, tag or commit | That revision of the team repo, resolved when the run starts. Name a branch as the team repo names it, `next`, not `origin/next`. | +| `without: [, ...]` | The pulled revision, without these files and directories of the team repo. See [Leave team assets out](#leave-team-assets-out). | + +`skillChanges` in the record lists, for each arm on another revision or `without` paths, the team skills it adds, removes or changes against the pulled revision, and the results show how often each arm loaded them. See [Results](#results). + +### Leave team assets out + +`without` takes paths in the team repo, as git lists them: a skill, a rule, an agent, a `claudemd/` file, a doc, a whole namespace directory, or any other file, such as `hooks/hooks.yaml`: + +```yaml +arms: + - name: pulled + - name: no-release + team: + without: [skills/release, skills/ops/release] + - name: no-ops + team: + without: [skills/ops, rules/ops] +``` + +The run plans every arm first, as pull would deliver to you, with your roles, projects, tags and excluded skills. It refuses an arm, and starts no run, when: + +| `without` lists | Because | +| --- | --- | +| A path the pulled revision does not have, or `teamai.yaml` | There is nothing to leave out, or no trial could install the team. | +| A path inside another path it lists | List only the outer one. | +| A path under `skills/`, `rules/`, `agents/`, `claudemd/` or `docs/` that pull delivers the same with or without | Leaving it out changes nothing. The message says why: a namespace your roles or projects do not receive, a shared skill they receive only through a tag you do not subscribe to, a skill you exclude, a rule tagged only with tags you do not subscribe to. | +| A shared path a namespace you receive delivers in its place | The namespace copy still reaches the trial: the message says `list skills/ops/release too`. | + +A path anywhere else only has to exist. A `without` path removes a whole file: to leave out one entry inside a shared file, such as one MCP server, hook, env key or model alias, push a branch of the team repo without it and name the branch: + +```yaml +arms: + - name: pulled + - name: no-github-mcp + team: no-github-mcp # a team repo branch whose mcp/mcp.yaml drops the github server +``` + +Benchmarks do not run in single-repo mode, where the project's own commit decides the team assets: the run refuses every arm with `team_ref_blocked`. + +### Arms that run a workflow + +An arm with `workflow` runs a team workflow on each task instead of one prompt. The workflow takes the task's prompt as its input `task`: + +```yaml +defaults: + tool: claude +arms: + - name: prompt + - name: plan-then-fix + workflow: plan-then-fix # workflows/plan-then-fix.yaml, with inputs: { task: string } +``` + +The run reads the workflow, and the workflows it runs, at the arm's team revision, through the namespaces you receive like any team workflow, and runs every prompt step in it on the arm's tool, model and effort, unattended as the trial's own prompt would be. It refuses the arm (`arm_workflow_unrunnable`) when the workflow takes no input `task`, requires another input, or has `finally` steps, when a step of it delivers, asks a person, opens a workspace other than `continue`, or runs `workflow: { from }`, or when a prompt of it runs an `agent` and the arm sets no `model`, as the agent's own model could replace the tool's default (set `model` in `defaults`: an arm with `workflow` changes no other key), and with `workflow_namespace_conflict` when two namespaces you receive both have it. The trial's check, or its judge, decides on what the workflow left in the clone. Its duration is the sum of its prompt turns, its tokens and cost the sum of its prompts'. [Trial limits](#trial-limits) `timeout` and `budget` are what its prompts may spend together; `max_turns` is not enforced, and is listed in `unenforced`. + +### Cross tools with team assets + +`matrix` makes one arm for each combination of its values, so every two arms that differ in one value compare that value alone. Here, how much the team's assets help each tool, and which tool does better with them: + +```yaml +defaults: + tool: claude +matrix: + tool: [claude, codex] + team: [pulled, none] +``` + +| Arm | Tool | Team | +| --- | --- | --- | +| `claude-pulled` | claude | pulled | +| `claude-none` | claude | none | +| `codex-pulled` | codex | pulled | +| `codex-none` | codex | none | + +A matrix varies `tool`, `model`, `team` or `workflow`, at least one of them with two values or more, and makes 2 to 12 arms. A key it leaves out comes from `defaults`. Each arm's name joins its values for the keys that vary: a ref is used as it is, with a character other than a letter, digit, `-` or `_` written `-`, and `without: [skills/auth-tokens]` is `without-skills-auth-tokens`. A suite has `arms` or `matrix`, not both. + +A task starts from a full commit of `repo` and is decided by exactly one of `check` and `successCriteria`: + +```yaml +# benchmarks/fix-auth/tasks/expired-token.yaml +schemaVersion: v1alpha1 +title: Reject expired refresh tokens +prompt: | + Refresh tokens past their expiry are still accepted. Fix it. +startingCommit: 3f2c9a7d1e0b4c6a8f5d2e1b0c9a8f7e6d5c4b3a +check: + run: npx vitest run test/auth/refresh.test.ts + protect: ['test/**', 'vitest.config.ts', 'package.json', 'package-lock.json'] + add: [test/auth/refresh.test.ts] +source: { platform: claude-code, sessionId: 7c1e2b9a, fix: 9a1d4e7c2b8f0a3d6e5c1b9f8a7d2e4c6b0a1f3e } +``` + +```yaml +# benchmarks/fix-auth/tasks/login-copy.yaml +schemaVersion: v1alpha1 +title: Clearer login error +prompt: Make the login error say which field is wrong. +startingCommit: 3f2c9a7d1e0b4c6a8f5d2e1b0c9a8f7e6d5c4b3a +successCriteria: The error names the wrong field and still reveals nothing about which accounts exist. +``` + +| Key | Value | +| --- | --- | +| `startingCommit` | The full commit of `repo` the task starts from. | +| `check` | `run`, a command that exits 0 once the task is done, and `protect`, the paths the agent must not supply. | +| `check.add` | Optional: files written before the check that the agent never has: a file as `source.fix` has it, such as the fix's test, or `{ from, to }`, a file or directory of the suite directory. A directory adds every file under it except its dot entries, such as `.DS_Store`; name a dotfile in `from` to add it. See [Tests the fix brought](#tests-the-fix-brought) and [Tests in the suite directory](#tests-in-the-suite-directory). | +| `successCriteria` | Instead of `check`: a judge decides from these. | +| `source` | Optional: the session the task came from, `platform` and `sessionId`, and `fix`, the full commit that fixed it there, which [`benchmark try`](#try-a-task) checks. | + +`teamai pull` and `teamai review --branch` check every file and report each problem with its file and line. A suite in `benchmarks///` goes only to members of that role or project, as [Roles and projects](distribution.md#roles-and-projects) describes. + +### What `protect` covers + +A trial passes when the check exits 0 after the agent's work. Before the check, teamai puts every `protect` path back as the starting commit has it: + +```text +the agent left the check sees +src/auth/refresh.ts edited edited +test/auth/session.test.ts edited as 3f2c9a7 has it +test/helpers/skip.js added deleted +vitest.config.ts edited as 3f2c9a7 has it +``` + +So `protect` lists everything the check reads that the agent must not supply: the tests, the test config, the scripts and helpers they load, and the package manifest and lockfile. + +| Do not protect | Why | +| --- | --- | +| A path `setup` writes into, such as a build directory | The restore deletes what setup wrote there. | +| Files git converts on checkout: Git LFS files, files with an `eol=crlf` attribute | Each protected file must hold exactly the bytes the starting commit stores, so the trial ends in `error`. | +| A path in or under a submodule | teamai cannot put a submodule's files back, so the trial ends in `error`. | + +A task whose check reads things you cannot list belongs in `successCriteria`. + +### Tests the fix brought + +When the fix brought the test that proves it, list that test file in `check.add`, also when the fix only added a case to a file the starting commit has. The check reads it as `source.fix` has it; the agent never does: + +```text +step test/auth/refresh.test.ts +the agent's clone absent +trial check protect restored, then written from 9a1d4e7 +baseline at 3f2c9a7 written from 9a1d4e7: the check must fail +try at 3f2c9a7 written from 9a1d4e7: the check must fail +try at 9a1d4e7 protect restored, then written from 9a1d4e7: the check must pass +``` + +| `teamai pull` and `teamai review --branch` refuse | Because | +| --- | --- | +| `check.add` without `source.fix` | The files come from `source.fix`. | +| A `check.add` path no `protect` entry matches | The agent could supply it. | +| A glob in `check.add` | Each entry names one file. | + +### Tests in the suite directory + +A test the task's commits do not have, such as one you wrote for the task, can live in the suite directory. A `{ from, to }` entry of `check.add` writes it where the check reads it, at the same steps as a fix's test; the agent never has it: + +```text +benchmarks/fix-auth/ + suite.yaml + tasks/expired-token.yaml + check-files/expired-token/refresh.test.ts +``` + +```yaml +check: + run: npx vitest run test/auth/refresh.test.ts + protect: ['test/**', 'vitest.config.ts', 'package.json', 'package-lock.json'] + add: + - from: check-files/expired-token/refresh.test.ts + to: test/auth/refresh.test.ts +``` + +`from` is a file or a directory of the suite directory; a directory writes every file under it below `to`. Each file is written as bytes, with its executable bit. `teamai pull` keeps the files with the suite, and a run writes them as they were when it started, so a pull during the run changes no check. Editing one changes the task's `sourceHash`. The entry needs no `source.fix`. + +| `teamai pull` and `teamai review --branch` refuse | Because | +| --- | --- | +| A `from` that names nothing in the suite directory, leaves it, or is a symlink | Only the suite directory's own files are read. | +| A `to` with `..` or a glob | It names one path in the clone. | +| A `to`, or a file under it, that no `protect` entry matches | The agent could supply it. | +| Two entries that write one path | Each path is written once. | + +### Baselines + +Before a task's first trial, teamai runs setup, writes the task's `check.add` files and runs the check once, on a fresh clone of the starting commit: + +| The check at the starting commit | The task | +| --- | --- | +| fails | Valid: its trials run. | +| passes | `invalid: check passes without changes`, no trials. | +| cannot run | `check could not run`, no trials. | + +| A check cannot run when | It is read as | +| --- | --- | +| it exits 127 | `not_found` | +| it exits 126 | `not_executable` | +| it exits 129 to 192, outside Windows | `signal`: a shell reports a killed command as 128 plus the signal. Do not use these as your check's own failure code. | +| it runs past the timeout | `timeout` | + +After a prompt, a check that cannot run gives the trial `error`, not `failed`. + +### Trial limits + +`trial` limits every trial's prompt. Reaching a limit ends the agent's turn, and teamai runs the trial's check, or its judges, on what the agent left, so a tool that runs out of time fails the task rather than the harness: + +```yaml +timeout: 45m +trial: + timeout: 8m + budget: { tokens: 300000 } + max_turns: 60 +``` + +| Key | Limits | On the local driver | +| --- | --- | --- | +| `timeout` | The time from the tool's launch. Shorter than the suite's `timeout`, which still ends the whole run. | Stops the tool and every process it started. | +| `budget.tokens` | Input plus output tokens, as a [workflow budget](workflow-advanced.md#requirements-and-limits) counts them. | Counts them from the tool's output as it runs, and stops the tool once they pass the budget, so a trial can go over by its last model call. Codex states its tokens only as it ends, and CodeBuddy's are not counted as it runs: both run to the end, and list `budget` in `unenforced`. A tool that reports no tokens ends its trial `error`, unless another limit stopped it first: then its trial lists `budget` in `unenforced`. | +| `max_turns` | Agent turns, as the tool counts them. | Passes it to Claude. Other tools list `max_turns` in `unenforced`. | + +A trial a limit stopped carries `limit` in the record, and counts under "Stopped by a limit". A trial that lists a suite limit in `unenforced` counts as a deviation, so a capped arm is never compared with an uncapped one silently. A driver that cannot end a turn at all runs the trial with the limits as a hard deadline and lists `on_limit` in `unenforced`. A run keeps the limits it started with. + +## Try a task + +Try a new task file before it joins the suite, in a checkout of the suite's `repo`, with the team repo checked out beside it: + +```console +$ teamai benchmark try ../team/benchmarks/fix-auth/tasks/expired-token.yaml +ℹ Task expired-token: trying the check at 3f2c9a7 +ℹ Task expired-token: trying the check at the fix 9a1d4e7, with the protected paths as 3f2c9a7 has them +{"data":{"try":{"task":"expired-token","message":"Task \"expired-token\": the check fails at 3f2c9a7 and passes at the fix 9a1d4e7, with the protected paths as 3f2c9a7 has them."}}} +``` + +```text +task at startingCommit at source.fix (protect restored) exit +pod-startup check fails check passes 0 +old-bug check passes not tried 1: "check passes without changes at 3f2c9a7" +bad-fix check fails check fails 1: "check fails at the fix 9a1d4e7" +test-in-fix check fails check fails (fix's test restored) 1: "The fix changes test/startup.test.ts, which protect puts back as 3f2c9a7 has them: list a test the fix brought in check.add" +changed-test check fails check passes (check.add written) 0 +add-missing not tried not tried 1: "test/missing.test.ts is not a file at the fix 9a1d4e7" +no-fix check fails not tried 0: "the check was not tried on a fix" +criteria-only no check no check 0: "there is no check to try" +missing-tool could not run not tried 1: names the command and 3f2c9a7 +setup-error setup failed not tried 1: names setup and 3f2c9a7 +``` + +`benchmark try` reads the task file and the `suite.yaml` two directories up from it as they are written, and validates both as `teamai pull` does, with the files its `check.add` entries name from that suite directory. It needs no `teamai init`. A commit the checkout lacks is fetched by SHA from the suite's `repo`, as a run does. Each commit gets a fresh clone, the suite's `setup`, the environment a trial gets (see [Run a suite](#run-a-suite)), and the suite's `timeout`. It creates no run and no record. The check runs the project's code on your machine, so try only tasks you trust. + +## Run a suite + +```console +$ teamai benchmark run fix-auth --sandbox +ℹ Run r38302f8e6a18c377 started. If it stops before it ends, continue it with: teamai workflow resume r38302f8e6a18c377 --sandbox +``` + +`benchmark run` plans the suite first, so a suite that cannot run leaves no run. It then starts the workflow `teamai-benchmark` with the suite's `timeout` and drives it with the local driver. + +Each trial prompt runs with [`permissions: unattended`](workflow-advanced.md#unattended-prompts), because the shipped workflow runs the trials in a strict sandbox: the tool edits files and runs commands without asking, whatever this machine's settings for it say. + +| Flag or setting | Effect | +| --- | --- | +| `--sandbox` | Declares this machine a sandbox for this process. Only on a machine you can throw away. | +| `workflows.sandbox: true` in config | Declares it for every process, as [Requirements and limits](workflow-advanced.md#requirements-and-limits) describes. | +| neither | The run is refused at start with `requirement_unmet`: the trials require a strict sandbox. | +| `--member-aliases` | Your own `~/.teamai/models/aliases.yaml` wins over the team's `models/aliases.yaml`, and the table shows `member` as the source of each model your file decided. | +| `--fail-on-regression` | Also exit 1 when an arm's pass rate dropped. See [Fail on a regression](#fail-on-a-regression). | + +| `benchmark run` exits | When | +| --- | --- | +| 0 | The run succeeded and its report was written, whatever the trials' results. | +| 1 | The suite or the start was refused, the run timed out or was cancelled, the report failed or was cancelled, or `--fail-on-regression` found a drop. | + +A record still waiting for publication never changes the exit code. + +| A trial gets | | +| --- | --- | +| A fresh clone of the starting commit | With no remotes. Your checkout is never changed. | +| The team assets and the suite | Of the team revision this checkout last pulled, which must be pushed, or of the arm's `team`. Frozen at start: a pull during the run changes nothing it runs. | +| An environment | Without `GITHUB_TOKEN`, `GH_TOKEN`, `GITLAB_TOKEN`, `GL_TOKEN`, `CI_JOB_TOKEN`, `SSH_AUTH_SOCK`, `GIT_ASKPASS` and `SSH_ASKPASS`: the trials step is a [strict sandbox](workflow-advanced.md#requirements-and-limits), which keeps them from every process of a trial. | +| Your home directory | As it is: teamai cannot remove its logins, MCP servers and plugins. | +| The team repo on this machine | As it is: your team checkout, and the copy of the team revision beside each trial's clone that its team assets are installed from, hold `benchmarks/`, check files included. An agent that searches the machine can find them. | + +The local driver runs each trial as fresh clone, team assets, setup, the arm's tool on the prompt, then the check. + +| | | +| --- | --- | +| Tools | It runs `claude`, `codex` and `opencode`. Install and log in every tool the arms use. An arm whose tool it cannot find gets its trials `not_run`, and the other arms still run. | +| Lanes | One trial at a time, unless the suite sets [`parallel`](#parallel-lanes). | +| Timeout | The suite's `timeout` is the whole run's. Setup, the tool and the check all stop at it: the trial they ran for counts as `error`, and the trials still open settle `not_run`. Keep it below your CI job's limit, or the job is killed before it prints the table. | +| A hung trial | Without [`trial.timeout`](#trial-limits), a tool that never returns runs until the suite's timeout, and every trial after it settles `not_run`. | +| Judges | A task with `successCriteria`, and the suite's scorers, are judged before the run ends, on the install that ran the trials. Start refuses a suite that needs a judge on a machine that cannot judge. | +| Control commands | A trial's steps exist only once the run's `plan` step reported. Until then, `teamai workflow cancel ` on a trial step answers `step_not_found`. A benchmark run is [isolated](workflow-advanced.md#isolated-runs): `workflow retry`, `publish` and `--claim` refuse it, `publish` and `--claim` with `isolated_run`, and retry names `teamai benchmark run `. | +| Stopped | `teamai workflow resume --sandbox` continues it. | + +A start that cannot run refuses with a message that says what to do, and makes no run. When your user-scope teamai install also reaches the team, or the team is served over HTTP, the run cannot pin the team a trial gets, and refuses with `team_ref_blocked`, saying what to change. + +### Parallel lanes + +```yaml +parallel: 4 +``` + +Baselines and trials run in up to `parallel` lanes, one after another in each lane. The local driver runs at most `workflows.max_parallel` of them at once, 1 unless your config sets it, so set both. Trials in parallel share the machine's CPU, memory and network, and the tools' rate limits: durations, and sometimes results, differ from a serial run. Compare runs made with the same `parallel`. + +## The benchmark workflow + +`benchmark run` starts the workflow `teamai-benchmark`. teamai ships this one: + +```yaml +# workflows/teamai-benchmark.yaml, shipped in the npm package +schemaVersion: v1alpha1 +name: teamai-benchmark +description: Run a benchmark suite. Every arm works the same tasks in fresh clones, judged by checks it never sees. +invoke: deny +inputs: { suite: string, aliases: string } +steps: + - id: plan + command: teamai benchmark plan "$SUITE" --aliases "$ALIASES" + env: { SUITE: "{{ inputs.suite }}", ALIASES: "{{ inputs.aliases }}" } + output: { definition: workflow, plan: json } + - id: trials + needs: [plan] + workflow: { from: plan } + requires: { isolation: sandbox, strict: true } +finally: + - id: report + command: teamai benchmark report + output: { text: string, table: json } +outputs: { text: steps.report.outputs.text, table: steps.report.outputs.table } +``` + +| Command | Runs as | Does | +| --- | --- | --- | +| `teamai benchmark plan ` | The `plan` step | Freezes the team revision you pulled, fetches every commit a trial reads, and prints the trials as a workflow, with one clone per baseline and trial. | +| `teamai benchmark check --baseline\|--trial` | A step in each clone | Writes `check.add`, puts `protect` back for a trial, runs the task's check, and prints its result. | +| `teamai benchmark report` | The `finally` step | Classifies the trials, writes the record, and prints the results. | + +`trials` runs what `plan` printed: see [Run a workflow a step printed](workflow-advanced.md#run-a-workflow-a-step-printed). `report` runs after the trials whatever happened to them, as [finally steps](workflow-advanced.md#finally-steps) do. `benchmark run --member-aliases` passes `aliases` as `member`, else as `team`. + +To change it, push your own `workflows/teamai-benchmark.yaml`, or `workflows//teamai-benchmark.yaml` for members of one role or project. It replaces teamai's for whoever receives it, as any [default workflow](workflow.md#override-a-default) does, and `benchmark run` says so: + +```text +ℹ Running the team's workflows/ops/teamai-benchmark.yaml, which replaces teamai's benchmark workflow. +``` + +Yours must take input `suite`, and input `aliases` to run with `--member-aliases`; `benchmark run` refuses it otherwise. Keep `plan`'s step id and `report` as a `finally` step. Dropping `strict: true` lets trials run without the credentials being kept from them, and with each tool's own permission settings rather than unattended: a tool that may not edit fails its trials, and the report opens with a line that says so. When two namespaces you receive both have a `teamai-benchmark.yaml`, `benchmark run` refuses with `workflow_namespace_conflict`, naming both files, and never falls back to teamai's. + +## Results + +When the run ends, `benchmark run` prints the results on stderr. The JSON envelope on stdout carries the same text in `data.done.outputs.text`, and the numbers in `data.done.outputs.table`. For this suite with a third task, `old-bug`, whose check already passes: + +```text +| Arm | Tool | Model (source) | Planned | Passed | Failed | Error | Unscored | Not run | Stopped by a limit | Deviations | Not verified | Pass rate | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| claude | claude | tool default (team) | 9 | 4 | 2 | 0 | 0 | 3 | 0 | 0 | 0 | 44% (4/9) | +| codex | codex | tool default (team) | 9 | 3 | 3 | 0 | 0 | 3 | 0 | 0 | 0 | 33% (3/9) | +| strong | claude | claude-opus-5-5 high (team) | 9 | 6 | 0 | 0 | 0 | 3 | 0 | 0 | 0 | 67% (6/9) | + +| Arm | Median tokens (input + output) | Median duration | Cost | +| --- | --- | --- | --- | +| claude | 41200 (6) | 84.0 s (6) | 1.82 USD (6) | +| codex | 38900 (6) | 97.5 s (6) | unknown (0) | +| strong | 52300 (6) | 121.0 s (6) | 4.1 USD (6) | + +| Task | claude | codex | strong | +| --- | --- | --- | --- | +| expired-token | 33% (1/3) | 33% (1/3) | 100% (3/3) | +| login-copy | 100% (3/3) | 67% (2/3) | 100% (3/3) | +| old-bug | 0% (0/3) | 0% (0/3) | 0% (0/3) | + +| Task | Baseline | +| --- | --- | +| old-bug | invalid: check passes without changes | + +These results hold for these tasks, this run and this machine. +``` + +| Table | Shows | +| --- | --- | +| Arms | Each arm's trials by result, and how many a [trial limit](#trial-limits) stopped. A deviation is a trial whose prompt ran on another tool, team revision or model than its arm's, or without a trial limit its driver could not enforce. A trial is not verified when its record lacks one of those. When an arm runs on other team assets than the pulled revision, a Team column shows each arm's, such as `pulled 3f9c2ab`, `next 81d0e4c`, `3f9c2ab without skills/auth-tokens` or `none`. | +| Usage | Each arm's medians and cost total, with how many trials each used. Tokens and cost appear only as the tool stated them, so a tool that states none shows `unknown (0)`. | +| Tasks | Each task's pass rate on each arm. | +| Baselines | The tasks that did not run, and why. | +| Skills | Only when an arm changes team skills: each skill it adds, removes or changes, and how many of the arm's trials loaded it, beside the arm that changes nothing. | + +```text +| Arm | Skill | Change | Loaded in this arm | Loaded in the defaults arm | +| --- | --- | --- | --- | --- | +| no-auth-tokens | auth-tokens | removed | 0/3 | 2/3, 1 unknown | +``` + +A load is counted only where teamai could see it: `unknown` is a trial whose tool reports no skill loads, never a trial that did not load the skill. + +The results are not the run's status: a run whose trials all failed still succeeds. The run fails when it times out or is cancelled. + +`report` writes the run's benchmark record into the run's directory on this machine. With [usage reporting](reporting.md) on, it also publishes it to `benchmarks/runs/.json` on the team repo's `teamai-reports` branch, so the job's token also needs push access to the team repo. A publication that fails waits in the queue and is retried on the next `teamai pull`; until then the text ends with `record: pending`. [Run records](workflow.md#run-records) lists the record's fields and gives the `jq` command that computes each arm's pass rate from the record alone. With it off, the results are still in the envelope. The dashboard's [Benchmarks page](dashboard.md#benchmarks) draws the published runs and this machine's over time. + +## Fail on a regression + +```console +$ teamai benchmark run fix-auth --sandbox --fail-on-regression +Arm "claude" passed 4/9 paired trials, against 8/9 in run r1c0d9e2f4b6a8c7d: a drop of 44 points, with a 95% interval of -73 to -16 points. +``` + +`--fail-on-regression` compares this run with the newest earlier run of the suite that succeeded and is published on `teamai-reports` with the same tasks and arms. An arm is the same when its name, tool, model, effort, workflow and team treatment are, whatever revision it pulled. Trials pair by task and repetition, as the dashboard's Compare pairs them, and the run exits 1 when an arm's pass rate dropped by more than chance. Records still waiting on this machine are not compared. With no such run, it prints a notice and leaves the exit code alone. + +## Run in CI + +A GitHub Actions job: + +```yaml +on: workflow_dispatch +jobs: + benchmark: + runs-on: ubuntu-latest + timeout-minutes: 60 + env: + TEAM_REPO: ${{ vars.TEAM_REPO }} + GH_TOKEN: ${{ secrets.TEAM_TOKEN }} + ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} + steps: + - uses: actions/checkout@v5 + with: { fetch-depth: 0 } + - uses: actions/setup-node@v4 + with: { node-version: 22 } + - run: npm install -g @anthropic-ai/claude-code + - run: npx teamai-cli init "$TEAM_REPO" --scope workspace --agent claude + - run: npx teamai-cli pull + - run: npx teamai-cli benchmark run fix-auth --sandbox > done.json + - if: always() + run: jq -r '.data.done.outputs.text // empty' done.json >> "$GITHUB_STEP_SUMMARY" +``` + +- `TEAM_TOKEN` reads the team repo, and pushes to it when usage reporting is on. +- `--agent claude` sets teamai up for Claude Code, which a fresh runner has never run. +- The checkout of the suite's `repo` has its full history, so every task's starting commit is there. +- The last step puts the results in the job's summary, also when the run failed. + +A GitLab CI job: + +```yaml +benchmark: + image: node:22 + timeout: 1h + when: manual + script: + - npm install -g @anthropic-ai/claude-code + - npx teamai-cli init "$TEAM_REPO" --scope workspace --agent claude + - npx teamai-cli pull + - npx teamai-cli benchmark run fix-auth --sandbox > done.json + artifacts: + when: always + paths: [done.json] +``` + +Set `TEAM_REPO`, `GITLAB_TOKEN` (reads the team repo) and `ANTHROPIC_API_KEY` as masked CI/CD variables. The job log shows the results, and `done.json` keeps them. For another tool, install it and set its key the same way: each arm's tool must be on the runner. + +## Your own driver + +A benchmark run is an ordinary workflow run, so any driver runs it through `workflow next --prepare` and `workflow report`, as [Clone workspaces in your own driver](workflow-advanced.md#clone-workspaces-in-your-own-driver) describes: + +```text +teamai workflow start teamai-benchmark --inputs '{"suite":"fix-auth","aliases":"team"}' --timeout 45m +teamai workflow next --prepare --json '{"driver":{..., "can":{"workspace":["read-only","new","continue","clone"],"isolation":"sandbox","env":true,"setup":true,"tools":["claude"],"unattended":true,"end_turn":true}}}' + -> a command lease (plan, check, report): run lease.command in lease.cwd, or in your checkout without one, with TEAMAI_WORKFLOW_ATTEMPT=/ + -> a prompt lease: launch lease.tool in lease.cwd on lease.prompt, with lease.model, in its unattended mode for lease.permissions, held to lease.limits +teamai workflow report --json '{..., "tool":"claude", "durationMs":84000, "cost":{"amount":0.31,"unit":"USD"}}' +``` + +| | | +| --- | --- | +| `driver.can` | On every `next`: `clone` in `workspace`, `isolation: "sandbox"`, `env: true`, `setup: true`, the arms' tools in `tools`, `unattended: true`, and `end_turn: true` when it ends a turn at the lease's limits. A trial is held for a driver that lacks one of them, and `wait.held[].missing` names it. | +| `--prepare` | teamai makes each clone on this machine, installs its team assets and runs setup, then answers the lease with `cwd`. | +| Every process | Runs without the names in `lease.envScope.withhold`, with `lease.env`, and with `TEAMAI_WORKFLOW_ATTEMPT=/`, so a `teamai` call from it changes no run. A command step prints its outputs as JSON on stdout; report them as `outputs`. | +| `report` | For a prompt: `tool`, `durationMs` measured around the tool launch, and `cost` only when the tool stated one. | +| Limits | Hold the tool to `lease.limits`. When one stopped it, report `outcome: "success"` with `limit`: `timeout`, `budget` or `max_turns`, and list in `unenforced` a `budget` or `max_turns` you could not hold it to. | +| Timeout | `benchmark run` passes the suite's `timeout` as `--timeout`; a direct `workflow start teamai-benchmark` must pass it too, or `plan` fails. | +| Judges | While the suite's scorers still judge, `next` answers `wait` with `judging`: run `teamai score --next` on the install that ran the trials. | diff --git a/docs/guide/dashboard.md b/docs/guide/dashboard.md index ae2f7fd77..338f7f635 100644 --- a/docs/guide/dashboard.md +++ b/docs/guide/dashboard.md @@ -9,7 +9,7 @@ teamai dashboard teamai dashboard --port 8080 ``` -Open `http://localhost:3721`. The sidebar is Overview, then three groups: Execution (Sessions, Tokens & Cost, Workflow Runs, Scorers), Context (Knowledge Base, Resources, Distribution), and Improvement (Improvement Center). Sessions and trends are from this machine; the Knowledge Base and the Improvement Center read the team repo. +Open `http://localhost:3721`. The sidebar is Overview, then three groups: Execution (Sessions, Tokens & Cost, Workflow Runs, Scorers, Benchmarks), Context (Knowledge Base, Resources, Distribution), and Improvement (Improvement Center). Sessions and trends are from this machine; the Knowledge Base and the Improvement Center read the team repo. ## Improvement Center @@ -61,7 +61,7 @@ The Workflow Runs page has three lists for the selected workspace. **Local runs* On the install that offered a run, the next step is `teamai workflow resume ` instead, which takes it back there. A local run that is offered, or that another install now holds, says so under its status. **A runner needs** (**Their machine needs** on a run that needs a person) lists the `requires.env` keys and prompt steps of the steps left: set those up on the runner, as [its job](workflow.md#take-offered-runs-from-ci) does. **Not on this machine** is what this machine lacks of them, as `teamai workflow resume` lists it as `cannot run here`. The status line counts the offered runs and the ones that need a person. The page claims nothing: the command runs a driver on the machine that takes the run. -The page shows what a run record publishes, plus each step's live status and the options of a waiting HITL request. Inputs, `for_each` items, outputs, prompts, HITL messages, and local paths stay in the run file. Read a HITL question where the run's driver asks it. A run's details show what each pass delivered: the commits on its branch, the pull request delivered for it, and the learnings `teamai contribute` recorded inside its attempt. +The page shows what a run record publishes, plus each step's live status and the options of a waiting HITL request. Inputs, `for_each` items, outputs, prompts, HITL messages, and local paths stay in the run file. Read a HITL question where the run's driver asks it. A run's details show what each pass delivered: the commits on its branch, the pull request delivered for it, and the learnings `teamai contribute` recorded inside its attempt. The details of a [benchmark](benchmarks.md) run that ended show its results, usage and task × arm tables, as the CLI prints them. A waiting approve or choose request can be answered from the run's details. The answer goes through the same `respond` operation as the CLI, with channel `dashboard` and `agent: false`, so it can answer a required approval. A dashboard started inside an agent session records `agent: true`, as an answer from that shell would, and cannot answer a required approval. It carries the pass the page showed: if that request was answered or replaced in the meantime, the page shows the refusal and reloads. Another website cannot send an answer. A request without an `Origin` header, such as a plain `curl`, is refused too. This marks an answer as sent from a browser; it does not stop a local script that sets the header. @@ -108,6 +108,45 @@ Labels from benchmark runs are left out. The page reads labels and recorded conf Labels from sessions and workflow runs show how the tools did on the work that came along. To compare tools, models or team assets on the same tasks, run a benchmark. +## Benchmarks + +The Benchmarks page shows every ended [benchmark](benchmarks.md) run of the selected workspace: the runs on this machine and the run records on `teamai-reports`, over 30, 90 or 365 days. Pick a suite in **Suites**. Each question below names the panel that answers it, with what that panel showed for a team that ran `fix-auth` every week and `migrate-db` on three Claude models: + +| Question | Panel | What it showed | +|---|---|---| +| Should the team switch its auth fixes to claude? | **Leading arm** | `claude 67% · 95% 57%–75% · not separated from codex`. Codex has 56% (47%–65%). The intervals overlap, so these 7 runs do not tell the two apart. Add repetitions or tasks before you switch. | +| Did the auth skill we merged on October 7 help? | **Pass rate per run** | A dotted line, `Team assets changed · 62432de`, marks the first run with the skill. Compare each arm's points on either side. The **Runs** table marks that run's team revision `changed`. | +| Which model is worth paying for on `migrate-db`? | **Pass rate against cost**, on **Cost** | sonnet passes 73% for 0.31 USD a trial, opus 78% for 0.89 USD, and their intervals overlap. The dashed line joins the arms nothing beats on both pass rate and cost: haiku, sonnet, opus. | +| Which task needs fixing, in the code or in the suite? | **Tasks by arm** | `refresh-race` has 7/7 under **Unstable runs**: in every run, some arm both passed and failed it, so one trial says little. `session-fixation` is 0% to 19% on every arm: read its check before blaming the tools. | +| Why did the September 23 run drop to 27%? | **Tasks per run** and **Runs** | `login-copy` is a hatched cell in that run's column, and **Runs** shows `1 not measured` under its pass rate: the task's check passed without changes in that run, so its [baseline](benchmarks.md#baselines) was invalid, and its trials count as planned but not passed. **Tasks by arm** marks it `not measured in 1/7`. Open the run from **Runs** to read its trials. | + +**Last run** shows the newest run's trials by result for each arm, with its usage, deviations and the tasks it did not measure. **Open the run** opens its details on the Workflow Runs page. + +The panels above are the **Arms** tab. Two more tabs answer what changed between arms, and between team revisions. Here, for `onboarding-docs`, a [matrix](benchmarks.md#cross-tools-with-team-assets) of claude and codex with the pulled team assets and with none, and for `fix-auth` again: + +| Question | Tab and panel | What it showed | +|---|---|---| +| How much do our team assets help on onboarding docs? | **Compare**, by **Team assets** | `pulled team against no team assets +41 pp · 95% +16 pp to +59 pp` on claude, over 27 paired trials, and `−60%` cost per passed trial. | +| Do they help codex as much? | **Difference per pair of arms** | codex gains +15 pp, but its interval crosses zero: these runs do not tell its two arms apart. | +| Which task do the assets fix? | **Each task** | `oncall-runbook` goes from 1/9 to 7/9 with the assets; `first-pr-guide` barely moves. | +| Did the auth-tokens skill help claude on `fix-auth`? | **Team changes**, on `claude · claude-opus-5-5` | 67%, 62%, 70% over three team revisions, in commit order, none told apart from the one before it. Click `67ee2ce · Add the auth-tokens skill and a security reviewer`: it changed three files, and `session-fixation` went from 1/12 to 3/12. | +| Did a tool update move the numbers? | **Arms**, **Pass rate per run** | A dashed `Tool updated · claude 2.4.0` line marks the first run on the new version. | + +**Compare** lists each two arms of the suite's current version that differ only in their team assets, tool, model or workflow, and pairs their trials by run, task and repetition. **Team changes** follows one tool, model and workflow across the team revisions it ran on with the pulled assets or a ref, and names each revision from this machine's team clone, with the files outside `benchmarks/` it changed since the one before it. + +How the numbers are counted: + +- A pass rate is passed trials over planned trials, as the run's own table prints it. +- The pooled numbers, the task tables and the leader count only the runs of the suite's current version: the suite files of its newest run. **Pass rate per run** also draws the older runs, after a dashed `Suite changed` line. +- An arm is its name, tool, model, effort and team assets. When one of them changed, the older runs are a separate line, marked `earlier configuration`. +- **Pass rate by arm** also counts the arm's deviations and trials not verified over those runs, as each run's table counts them. A deviation is a trial whose prompt ran on another tool, model or team revision than its arm's. +- The interval is a 95% Wilson score interval. Two arms whose intervals overlap are not told apart by these runs. +- A difference in **Compare** is over paired trials, with a 95% interval for paired proportions (Newcombe). One whose interval crosses zero is `not separated`. +- Usage is the median over the trials whose tool reported it. Cost is per trial. +- A run on this machine that was also published shows once, from this machine. + +The page shows what a run record publishes. It never shows a prompt, a diff, or a transcript. + ## Resources The Resources page lists every rule in the team repo for the selected workspace: its name, namespace, description, and whether `pull` delivers it there. Root rules reach everyone; namespaced rules follow the active knowledge namespaces, and a root rule is marked replaced when an active namespace rule of the same name stands for it. Rules left out by tag subscriptions are marked too. diff --git a/docs/guide/distribution.md b/docs/guide/distribution.md index f7eb386fe..ea9857b4f 100644 --- a/docs/guide/distribution.md +++ b/docs/guide/distribution.md @@ -101,7 +101,7 @@ sharing: .teamai/mcp/mcp.yaml ``` -`teamai push` sends every distributed resource type — skills, rules, agents, docs, env, hooks, mcp, models, claudemd, learnings, wiki, workflows, prompts, and scorers — with `--project` and `--role` namespace placement. +`teamai push` sends every distributed resource type — skills, rules, agents, docs, env, hooks, mcp, models, claudemd, learnings, wiki, workflows, prompts, scorers, and benchmarks — with `--project` and `--role` namespace placement. ### Git permissions @@ -155,6 +155,8 @@ A project is the service they are working on. After this directory is set to tha `knowledge` covers rules and shared instructions (`claudemd/`). A project can also declare learnings, env, hooks, mcp, models, docs, and wiki on its own. +Benchmark suites follow `resources.benchmarks`. `benchmarks//` goes to everyone. Once a role or project lists ``, `benchmarks///` goes only to members who have `` active, and `benchmarks//` holds suites, not a `suite.yaml` of its own. + ```yaml # manifest/roles.yaml version: 1 diff --git a/docs/guide/scorers.md b/docs/guide/scorers.md index ea0e45fac..ff039337a 100644 --- a/docs/guide/scorers.md +++ b/docs/guide/scorers.md @@ -92,7 +92,7 @@ $ teamai score --lease s5d1e07c2a9b4a/a91c0d7e0 --json '{"driver":{"kind":"local {"data":{"label":{"scorer":"review-quality","label":"good","score":1,"passed":true,…},"published":true}} ``` -`samplingRate` picks the succeeded passes to judge, the same way on every machine. `workflow next` never hands out judge work. `--lease ` judges here with the tool the lease names, in a [confined profile](#isolation), and settles the lease like any driver's answer. A failed judge leaves the lease open. +`samplingRate` picks the succeeded passes to judge, the same way on every machine. `workflow next` never hands out judge work. `--lease ` judges here with the tool the lease names, in a [confined profile](#isolation), and settles the lease like any driver's answer. A failed judge leaves the lease open. A step whose `scorers` sets `sampling: all` is the exception: its scorers judge every succeeded pass, and the run waits for them before its `finally` steps and its end, while `workflow next` answers `wait` with `judging`, the judge entries still open. The local driver judges them itself and settles a failed judge at once; another driver's are judged with `teamai score --next` on the install that ran them. A [benchmark](benchmarks.md) judges its trials this way. The label records `used.skills`, the team skills the pass or its subagents loaded, found as for [a session](#score-a-session) from what the checkout held as teamai delivered it when the pass started. A `workflow:` step's label lists every skill its child passes recorded, and has no `used` when none of them recorded one. @@ -319,7 +319,7 @@ Session judging: attended auto, set by scorers/scoring.yaml sessions; unattended | What is judged | Set by | |---|---| -| A workflow step's passes | The step's `scorers`, sampled by each scorer's `samplingRate`. `sessions` plays no part. | +| A workflow step's passes | The step's `scorers`, sampled by each scorer's `samplingRate`, or every pass under `sampling: all`. `sessions` plays no part. | | A captured session: yours, or one a `CI` or `TEAMAI_MODE` job ran that teamai did not launch | `sessions` for the session's run mode, for every scorer with `sessions: true`, sampled by its `samplingRate`. | | A session teamai launched, such as a workflow step or a judge | Nothing. It writes no capture, and its step is judged as a pass. | | A session of a tool teamai reads no sessions from, such as Pi | Nothing. `teamai doctor` notes each such tool the team enables, in each mode whose publish or scoring policy is not `off`. With the defaults, that is both modes. | @@ -350,7 +350,7 @@ Every unattended driver ends with `teamai score --drain`. The local driver runs 1. In unattended mode, every session this data home captured as unattended counts as finished, the calling session included. Attended sessions keep the six-hour rule. 2. It judges every open judge entry this machine can judge: sampled step passes, and sampled sessions the policies allow, within `daily_limit`. While another judge pass runs on this machine, such as the background judge, it judges nothing and says so. Run the drain again once that pass ends. 3. It publishes every pending label, as `score --push` does, and the workflow run records a run could not publish, as `teamai pull` does. -4. With `publish: auto` for the current mode, it archives on `teamai-reports`, scrubbed, the finished captured sessions of that mode and the step sessions of the runs driven here that are not archived yet. A step session it cannot archive from here, because teamai reads no sessions from its tool or this machine cannot read it, is listed once as skipped with the reason, and never tried again. The other step sessions of that run are still archived. +4. With `publish: auto` for the current mode, it archives on `teamai-reports`, scrubbed, the finished captured sessions of that mode and the step sessions of the runs driven here that are not archived yet. A step session it cannot archive from here, because teamai reads no sessions from its tool or this machine cannot read it, is listed once as skipped with the reason, and never tried again. The other step sessions of that run are still archived. The trial sessions of a [benchmark](benchmarks.md) run stay on the machine that ran it. 5. It reports usage to `stats/.yaml`, as `teamai pull` does, so the counts of an ephemeral runner reach the team. 6. It prints what is left: entries whose scorer pins a model no judge tool here runs (`judge_model_unavailable`), or whose isolation this machine does not meet, and every open entry when no judge tool is installed here. Those wait for another driver's `score --next`. @@ -430,4 +430,4 @@ A `best-effort` tool cannot judge for a scorer that requires `verified`, nor can | `/sessions/captures/` | What each session started as, written by the session-start hook. Removed a day after the session's file is gone. | | `scores//.jsonl` on `teamai-reports` | Published labels, the newest per scorer, rubric, subject and judge kind. Labels wait in `/scores/pending/` until a push lands, and stay there while usage reporting is off. | -Labels never enter the run or its record. Workflow Runs shows each pass's labels, and what judging cost apart from the run's own tokens. For a run on this machine that counts every judge attempt; for a run a member published, only the labelled ones. The dashboard's [Scorers page](dashboard.md#scorers) shows each scorer's pass rate over time and by configuration. +Labels never enter a workflow run or its record. A benchmark run freezes each trial's verdict when it ends, and its record carries the trial's result, not the label. Workflow Runs shows each pass's labels, and what judging cost apart from the run's own tokens. For a run on this machine that counts every judge attempt; for a run a member published, only the labelled ones. The dashboard's [Scorers page](dashboard.md#scorers) shows each scorer's pass rate over time and by configuration. diff --git a/docs/guide/tool-support.md b/docs/guide/tool-support.md index 50219e623..dbcfcc035 100644 --- a/docs/guide/tool-support.md +++ b/docs/guide/tool-support.md @@ -74,33 +74,33 @@ Paths are relative to the project root. `cwd only`: the tool reads it only from User scope, from #1221's matrix; a fix merged since is marked `t`, with a note. `✓` works. `◐` works by default, and a condition breaks it while doctor still reports it delivered. `✗` broken in the default setup. `+` the tool supports it and teamai does not build it. `—` the tool has no such feature. `?` unknown. The letter after the mark is the evidence, as above. -| Tool | Env | Workflows (local driver) | Recall | Stats | Tokens & cost | AI backend | Votes judge | Session judge | HTTP agent hooks | -| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | -| Claude Code | ✓t | ✓v | ✓v | ✓v | ✓v | ✓v | ✓v | ✓t1 | ✓v | -| Codex CLI | ✓t | ✓t | ✓v | ✓t2 | ✓v | ✓v | ✓t3 | ✓t1 | ✓t4 | -| Codex CLI Internal | ? | ✓t | ? | ? | ✓t | ✓t | ? | ? | ? | -| Claude Code Internal | ? | ✓t | ? | ? | ✓t | ✓t | ? | ? | ? | -| TClaude | ? | ✓t | ? | ? | ✓t | ✓t | ? | ? | ? | -| TCodex | ? | ✓t | ? | ? | ✓t | ✓t | ? | ? | ? | -| Cursor | ? | +s5 | ✗v6 | ✓v7 | +d8 | ✓d | +s | ✓t1 | +t9 | -| Copilot | ? | +s | ✓v | ✓t2 | ✓t10 | ✓d | ? | ✓t1 | —t | -| JoyCode | ? | ? | ◐s | ✓t2 | ✓t | ? | ? | ✓t1 | ? | -| Qoder | ? | ? | ◐s | ✓s | ◐t11 | ✓d | ? | ✓s12 | ✓s | -| Qoder CN | ? | ? | ◐s | ✓s | ◐t11 | ? | ? | ✓s12 | ✓t13 | -| Kiro | ? | ? | ◐d | ? | ? | ✓d | ? | ✓d14 | ✓d15 | -| Trae | ? | ? | ? | ? | ? | ? | ? | ✓s16 | ? | -| Trae CN | ? | ? | ? | ? | ? | ? | ? | ✓s16 | ? | -| ZCode | ✓t | +s | ✓s | ✓s | ? | +s | ✓t3 | ✓d17 | +t18 | -| Oh My Pi | ✓s | +s | ✓v | ✓t2 | ✓t | ✓d | ? | ✓s19 | ✓s20 | -| Pi Coding Agent | ✓s | +v | ✓v | ✓t2 | ✓t | ✓s | ? | ◐s21 | ✓s22 | -| CodeBuddy | ? | ✓d | ✓s | ✓v | ✓t | ✓d | +v | ✓t1 | ✓s | -| OpenClaw | ◐d | +d | ✓s | +s | +s23 | ✓d | —t | ◐s24 | ✓s25 | -| Hermes | ◐s | +s | ✓v | +s | +s23 | ✓s | —t | ✓s26 | ✓s27 | -| DeepSeek Harness | ✗s | +s | ✓s | ✓t2 | +s28 | +s | ? | ◐s29 | —t | -| WorkBuddy | ? | ✓t | ✓t | ? | ✓t | ✓t | ? | ✓t30 | ? | -| OpenCode | ? | ✓t | ✓v | ✓t2 | ✓v | ✓v | ? | ✓t1 | ✓v31 | -| Muse Code | ✓t | +v | ✓t | ✓v | ✓t | ✓v | ✓t3 | ✓t1 | ✓t32 | -| Antigravity | ✓v33 | ✓v | ✓v | ✓v | ◐v34 | ✓v | ✗v35 | ✓t36 | ✓t37 | +| Tool | Env | Workflows (local driver) | Unattended prompts | Recall | Stats | Tokens & cost | AI backend | Votes judge | Session judge | HTTP agent hooks | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| Claude Code | ✓t | ✓v | ✓v | ✓v | ✓v | ✓v | ✓v | ✓v | ✓t1 | ✓v | +| Codex CLI | ✓t | ✓t | ✓d | ✓v | ✓t2 | ✓v | ✓v | ✓t3 | ✓t1 | ✓t4 | +| Codex CLI Internal | ? | ✓t | ✓t | ? | ? | ✓t | ✓t | ? | ? | ? | +| Claude Code Internal | ? | ✓t | ✓t | ? | ? | ✓t | ✓t | ? | ? | ? | +| TClaude | ? | ✓t | ✓t | ? | ? | ✓t | ✓t | ? | ? | ? | +| TCodex | ? | ✓t | ✓t | ? | ? | ✓t | ✓t | ? | ? | ? | +| Cursor | ? | +s5 | ? | ✗v6 | ✓v7 | +d8 | ✓d | +s | ✓t1 | +t9 | +| Copilot | ? | +s | ? | ✓v | ✓t2 | ✓t10 | ✓d | ? | ✓t1 | —t | +| JoyCode | ? | ? | ? | ◐s | ✓t2 | ✓t | ? | ? | ✓t1 | ? | +| Qoder | ? | ? | ? | ◐s | ✓s | ◐t11 | ✓d | ? | ✓s12 | ✓s | +| Qoder CN | ? | ? | ? | ◐s | ✓s | ◐t11 | ? | ? | ✓s12 | ✓t13 | +| Kiro | ? | ? | ? | ◐d | ? | ? | ✓d | ? | ✓d14 | ✓d15 | +| Trae | ? | ? | ? | ? | ? | ? | ? | ? | ✓s16 | ? | +| Trae CN | ? | ? | ? | ? | ? | ? | ? | ? | ✓s16 | ? | +| ZCode | ✓t | +s | ? | ✓s | ✓s | ? | +s | ✓t3 | ✓d17 | +t18 | +| Oh My Pi | ✓s | +s | ? | ✓v | ✓t2 | ✓t | ✓d | ? | ✓s19 | ✓s20 | +| Pi Coding Agent | ✓s | +v | ? | ✓v | ✓t2 | ✓t | ✓s | ? | ◐s21 | ✓s22 | +| CodeBuddy | ? | ✓d | ✓d | ✓s | ✓v | ✓t | ✓d | +v | ✓t1 | ✓s | +| OpenClaw | ◐d | +d | ? | ✓s | +s | +s23 | ✓d | —t | ◐s24 | ✓s25 | +| Hermes | ◐s | +s | ? | ✓v | +s | +s23 | ✓s | —t | ✓s26 | ✓s27 | +| DeepSeek Harness | ✗s | +s | ? | ✓s | ✓t2 | +s28 | +s | ? | ◐s29 | —t | +| WorkBuddy | ? | ✓t | ✓t | ✓t | ? | ✓t | ✓t | ? | ✓t30 | ? | +| OpenCode | ? | ✓t | ✓d | ✓v | ✓t2 | ✓v | ✓v | ? | ✓t1 | ✓v31 | +| Muse Code | ✓t | +v | ? | ✓t | ✓v | ✓t | ✓v | ✓t3 | ✓t1 | ✓t32 | +| Antigravity | ✓v33 | ✓v | ✓d | ✓v | ✓v | ◐v34 | ✓v | ✗v35 | ✓t36 | ✓t37 | Notes: diff --git a/docs/guide/workflow-advanced.md b/docs/guide/workflow-advanced.md index 07fd47b7c..ce888793f 100644 --- a/docs/guide/workflow-advanced.md +++ b/docs/guide/workflow-advanced.md @@ -439,6 +439,78 @@ Three runs of `loop-wait-for-ci` with `--driver local` end like this: A workflow with `loop` cannot contain a step with `loop`. Put one of the two loops in a sub-workflow. +## Finally steps + +```yaml +schemaVersion: v1alpha1 +name: nightly-e2e +description: Run the end-to-end suite, then always summarize it. +timeout: 1h +steps: + - id: e2e + command: npm run e2e +finally: + - id: summary + command: node scripts/e2e-summary.js + output: { text: string } +outputs: { summary: steps.summary.outputs.text } +``` + +`finally` steps run once the run's steps have ended, however they ended: succeeded, failed, timed out or cancelled. They are commands, read-only, with no workspace of their own, and carry only `id`, `needs`, `if`, `command`, `env`, `output` and `timeout`. The run stays `running` while they run, for at most 10 minutes, then ends with the outcome its steps reached: a failed `finally` step never changes it. `done.finally` lists each one that failed or was cancelled, with its failure. `workflow cancel` during them ends them and the run. A workflow with `finally` runs only as a run of its own, never as another workflow's step. + +`done.outputs` holds the workflow's top-level `outputs`, resolved after `finally`, so they may read `finally` steps. An output whose step failed is left out. + +```text +teamai workflow start nightly-e2e --driver local --timeout 30m +``` + +`--timeout` bounds the run to the shorter of it and the file's `timeout`, with `--driver local` or `--driver skill`. Past it, the run fails `timeout`, and its `finally` steps still run. + +## Run a workflow a step printed + +```yaml +schemaVersion: v1alpha1 +name: shard-tests +description: Plan one test job per changed package, then run them. +steps: + - id: plan + command: node scripts/plan-shards.js + output: { definition: workflow, notes: json } + - id: shards + needs: [plan] + workflow: { from: plan } +``` + +`plan` prints its outputs as JSON, a workflow included: + +```text +{"definition":{"definition":{"schemaVersion":"v1alpha1","name":"shards","description":"One job per package.","steps":[{"id":"api","command":"npm test -w api"},{"id":"web","command":"npm test -w web"}]}},"notes":{"packages":2}} +``` + +| Output type | Holds | +| --- | --- | +| `workflow` | A bundle: `definition`, the workflow to run, and optionally `workflows`, the named workflows it runs, `scorers`, generated scorer pins, and `teamRevision`, the team commit whose scorers it uses. | +| `json` | Any JSON value. Expressions cannot read into it; top-level `outputs` can pass it whole. | + +`workflow: { from: }` runs the bundle of a step it `needs`. When that step reports, teamai checks the bundle as `workflow start` checks a file, pins it, and keeps it for the run. A bundle it cannot take, one that holds a secret value, a `from` step, a `prompt_file` step or `finally` steps, fails the producing step `invalid_output` with the reason. The producing step runs once, outside any loop or `for_each`. A run with a `from` step is [isolated](#isolated-runs). + +## Clone workspaces + +```yaml + - id: try-fix + workspace: + clone: + commit: 3f2c9a7d1e0b4c6a8f5d2e1b0c9a8f7e6d5c4b3a + team: { treatment: none } + prompt: Fix the failing test. +``` + +A step with `workspace: { clone: { commit, repo?, team } }` runs in a fresh clone of `commit` outside the checkout: its own git data, no remotes, hooks off. `team` names the team assets installed for it, under a home of its own: `{ treatment: none }`, or `{ revision, selection, treatment }` with `treatment` `pulled`, a ref, or `{ without: [, ...] }`. `selection` holds `primaryRole`, `additionalRoles`, `projects`, `subscribedTags` and `excludedSkills` as your config names them, and `enabledAgents`, the tools the assets install for: by default the tool of the step that opens the clone, else your enabled agents. teamai makes the clone offline, from the run's checkout or the commit cache; `repo` only names the cache, never a URL. A workflow step that opens a clone owns it for every `continue` step under it. Nothing in a clone is pushed or delivered. [Benchmarks](benchmarks.md) write these steps for you. + +### Isolated runs + +A run is isolated when one of its steps opens a clone or runs `workflow: { from }`. It runs only on the machine that started it: `workflow publish` and `--claim` refuse it with `isolated_run`, and `workflow retry` refuses it and names the command that starts a new run. Its record carries `isolated: true`, still reaches `teamai-reports` with usage reporting on, and counts in no workflow total. + ## Expression reference An `if` is an expression. The rules for which steps it can read, and for what counts as true, are in [Workflows](workflow.md). @@ -580,7 +652,7 @@ Pass the whole list, as `explain` does in `fan-build`, when the prompt needs eve ## Requirements and limits -A step says what it needs in `requires`. `defaults.requires` in a workflow file applies to each step in it. A step's own keys replace those, and `env` lists add up. The workflow file can also set `setup`, `timeout`, and `budget`, and a prompt step can set `max_turns`. +A step says what it needs in `requires`. `defaults.requires` in a workflow file applies to each step in it. A step's own keys replace those, and `env` lists add up. The workflow file can also set `setup`, `timeout`, and `budget`, and a prompt step can set `max_turns`, `on_limit` and `permissions`. ```yaml schemaVersion: v1alpha1 @@ -610,10 +682,13 @@ steps: | `requires.env` | Gives the step these [team env](sharing.md#env-environment-variables) keys. Every other team env key, and every key another step of the run requires, is kept from it. Only key names enter the run. A key with no value on this machine fails the step `missing_env` before it runs, and the message names what gives it one: see [A required key with no value here](#a-required-key-with-no-value-here). A step `env:` entry that names a team secret or a required key fails `env_conflict`. | The step waits. | | `requires.network` | `false` asks that the step get no network. | The step runs, and its `unenforced` lists `network`. The local driver cannot deny the network. | | `requires.isolation` | `none`, `worktree`, or `sandbox`. A step with a workspace has at least `worktree`. | The step runs, and its `unenforced` lists `isolation`. | +| `requires.strict` | `true` makes `isolation` and `network` at this level a floor: every step under it gets at least them, whatever an inner level asks. A strict `isolation: sandbox` also keeps `GITHUB_TOKEN`, `GH_TOKEN`, `GITLAB_TOKEN`, `GL_TOKEN`, `CI_JOB_TOKEN`, `SSH_AUTH_SOCK`, `GIT_ASKPASS` and `SSH_ASKPASS` from every process of the step, and pull refuses a step under it whose `requires.env` names one. With `--driver local`, a start the local driver could never run is refused `requirement_unmet`. | The step waits. A strict sandbox also needs `can.env: true`. | | `setup` | The driver runs `command` in each workspace it creates, before the step. Setup gets its `requires.env` keys; a step gets them only if it requires them too. Setup fails the step without running it, and the workspace is removed, when it exits non-zero (`setup_failed`), changes a file git would commit (`setup_dirty`), or leaves the value of one of its keys in a file it wrote, ignored files included (`setup_secret`). The step's `timeout` bounds setup. | Steps with a workspace wait. | | `timeout` | `500ms`, `30s`, `10m`, or `2h`, on a step or the workflow file. A step past it is stopped and ends with outcome `timeout`. On a workflow step, it stops every step inside. On the workflow file, the run ends `failed` with `timeout`. A retry starts the timeout again. | Not a capability. Every driver gets `expiresAt` on the lease. | -| `budget` | `{ tokens: N }` on the workflow file or a workflow step. Input and output tokens of every step under it count from when it starts: the run's from its start, a workflow step's from each time it starts, such as a loop's next pass. A retry starts the budget again. Once they pass `N`, its open steps stop and it fails `budget_exceeded`. | A headless prompt under a budget waits. Its success report without tokens fails `budget_unmeasured`. An interactive prompt runs, since a person drives it; when its report has no tokens, its `unenforced` lists `budget`. | +| `budget` | `{ tokens: N }` on the workflow file, a workflow step, or a prompt step with `on_limit: end_turn`. Input and output tokens of every step under it count from when it starts: the run's from its start, a workflow step's from each time it starts, such as a loop's next pass. A retry starts the budget again. Once they pass `N`, its open steps stop and it fails `budget_exceeded`. | A headless prompt under a budget waits. Its success report without tokens fails `budget_unmeasured`. An interactive prompt runs, since a person drives it; when its report has no tokens, its `unenforced` lists `budget`. | | `max_turns` | Caps the agent turns of a prompt step. The local driver passes it to headless Claude only. | The step runs, and its `unenforced` lists `max_turns`. | +| `permissions` | `unattended` on a prompt step: its tool edits files and runs commands without asking. Only under a strict `isolation: sandbox`, set on the step, a step around it, `defaults`, or the step that runs its workflow: a start without one is refused `unattended_unsandboxed`, and a [workflow a step printed](#run-a-workflow-a-step-printed) without one fails that step `invalid_output`. See [Unattended prompts](#unattended-prompts). | The step waits. A run whose only driver lacks `can.unattended` fails it `tool_unavailable`. | +| `on_limit` | `end_turn` on a prompt step with `timeout`, `max_turns` or `budget`: reaching one, counted from the tool's launch, ends the agent's turn instead of failing the step. The step succeeds with `limit` set and keeps its work, so the steps after it still run. On a workflow step with `timeout` or `budget`, those are what its prompts may spend together, one turn at a time: once spent, its pending steps are skipped. `fail`, the default, fails the step at the limit. A step handed back keeps the deadline of its first lease. | Without `can.end_turn`, the step runs under `fail`, and its `unenforced` lists `on_limit`. | `unenforced` is on the lease, the step, and the run record. Each `interactive: true` step also needs a driver with HITL, as [HITL](#hitl-human-in-the-loop) describes. @@ -629,7 +704,7 @@ workflows: ``` - `allow.network: false` holds each step that requires `network: true`. `allow.env` holds each step whose `requires.env`, or whose setup's, names a key it does not list. `wait.held[].missing` names `policy.network` or `policy.env`. A bad `allow` in `~/.teamai/config.yaml` stops workflow commands until it is fixed, so a typo never lifts a restriction. -- `sandbox: true` says this host contains what it runs, such as a CI VM or a container, so the local driver declares `sandbox` isolation. +- `sandbox: true` says this host contains what it runs, such as a CI VM or a container, so the local driver declares `sandbox` isolation. A step with [`permissions: unattended`](#unattended-prompts) then runs its tool without asking, in whatever workspace the step uses, your own checkout included. `teamai workflow start` and `resume` take `--sandbox` to declare it for one process. - `max_parallel` also caps how many leases of one run every driver on this machine holds at once. `allow.network` filters declared step requests. Setup declares only environment keys in v1 and receives no network restriction from that setting. To guarantee that setup and steps cannot reach the network, use a driver and host that enforce it for every process. @@ -641,15 +716,45 @@ A driver other than `local` declares what it can do in `driver.can` on each `nex | `driver.can` key | The driver | |---|---| | `hitl` | Lets a person answer HITL requests and work in interactive steps. | -| `workspace` | Takes leases in these workspace modes: `read-only`, `new`, `continue`. | +| `workspace` | Takes leases in these workspace modes: `read-only`, `new`, `continue`, `clone`. A clone step is leased only through [`next --prepare`](#clone-workspaces-in-your-own-driver). | | `env` | `true`: resolves `lease.envScope.provide` itself, removes `lease.envScope.withhold`, and keeps the values from what runs the step. `"commands"`: does that for command steps and setup only, so a prompt step with `requires.env` waits. A driver whose agent runs the step leaves it out, or sends `"commands"` when it starts the commands itself. teamai before 0.28.0 answers `"commands"` with `invalid_json`. | | `setup` | Runs `lease.setup.command`, with `lease.setup.envScope`, in each workspace it creates and in one whose setup never finished, and fails the step with the codes above. Every lease with a workspace carries `setup` when the workflow declares it. | | `tokens` | Reports the tokens of every successful headless prompt step. | | `max_turns` | Passes `lease.maxTurns` to the tools it can cap, and reports `"unenforced":["max_turns"]` for the others. | | `network` | Can deny a step the network. | | `isolation` | Keeps every step at this level: `none`, `worktree`, or `sandbox`. | +| `tools` | Launches a prompt step that names its `tool` on these tools, such as `["claude"]`. Without it, no such step. | +| `end_turn` | Ends a prompt's turn at `lease.limits` and reports the `limit` it stopped at, with `durationMs`. | +| `unattended` | Launches a prompt leased with `permissions: "unattended"` in its tool's own mode for editing and running commands without asking, and fails the step rather than launch a tool that has none. | + +An omitted key means the driver cannot. Isolation is best effort, unless `strict: true` makes it a floor. `next` lists any other key in `ignored`. The skill driver of a [coding session](workflow.md#run-from-a-coding-session) declares `hitl`, every `workspace` mode, `env: "commands"`, and `setup`. It never declares `tokens`, `max_turns`, `network`, or `unattended`, and its `isolation` is `none`: a coding session never bypasses its person's permissions. -An omitted key means the driver cannot. `next` lists any other key in `ignored`. The skill driver of a [coding session](workflow.md#run-from-a-coding-session) declares `hitl`, every `workspace` mode, `env: "commands"`, and `setup`. It never declares `tokens`, `max_turns`, or `network`, and its `isolation` is `none`. +### Unattended prompts + +A tool started headless asks before it edits a file or runs a command, and with no one there to answer, the step does nothing. `permissions: unattended` lets the prompt's tool go ahead, so give it only where the machine is one you can throw away: + +```yaml +schemaVersion: v1alpha1 +name: unattended-fix +description: Fix a failing test on a disposable machine. +steps: + - id: work + prompt: Fix the failing test. + permissions: unattended + requires: { isolation: sandbox, strict: true } +``` + +The local driver then launches the tool in its own mode for that: + +| Tool | Launched with | +|---|---| +| Claude Code, and the tools on its flags | `--permission-mode bypassPermissions` | +| CodeBuddy, WorkBuddy | `--permission-mode bypassPermissions` | +| Codex CLI, and the tools on its flags | `--dangerously-bypass-approvals-and-sandbox` | +| OpenCode | `--auto`, which still keeps what a permission rule denies | +| agy | `--dangerously-skip-permissions` | + +A tool teamai knows no such mode of fails the step `tool_unavailable`, and is never launched without it. The `Unattended prompts` column of [Tool support](tool-support.md) says which tools have one, and on what evidence. The lease carries `permissions: "unattended"`, so a driver of your own sees it; only one with `can.unattended` takes the step. ### A required key with no value here @@ -929,6 +1034,22 @@ To prepare an attempt the driver already leased, run `next --prepare --att What TeamAI records to prepare an attempt stays on this machine: paths and file fingerprints, with no prompt, env value, or graph. +## Clone workspaces in your own driver + +```text +teamai workflow next --prepare --json '{"driver":{"kind":"script","machine":"runner","workspace":"/work","can":{"workspace":["read-only","new","continue","clone"],"isolation":"sandbox","env":true,"setup":true,"tools":["claude"],"end_turn":true}},"running":[]}' +``` + +With `--prepare`, teamai makes a [clone step](#clone-workspaces)'s clone on this machine, installs its team assets and runs setup, then answers the lease with `cwd`, the clone. A call can take as long as setup takes. A clone step is held from a driver that does not list `clone` in `workspace` or does not ask with `--prepare`, and `wait.held[].missing` names `workspace.clone` or `prepare`. + +For each lease the driver: + +1. Runs every process with `lease.env` and `TEAMAI_WORKFLOW_ATTEMPT=/`, so a `teamai` call from it changes no run, and without the names in `lease.envScope.withhold`. It runs a command lease, such as `teamai benchmark check`, in `cwd`, or in its own workspace for a lease without one, and reports the JSON it prints as `outputs`. +2. Launches a prompt lease's `lease.tool` in `cwd` on `lease.prompt`, with `lease.model` as given, and holds it to `lease.limits` from the tool's launch. +3. Reports a prompt with `tool`, the `model`, `session` and `tokens` the tool reported, `durationMs` measured around the tool launch only, and `cost` (`{"amount":0.12,"unit":"USD"}`) only when the tool stated one. When a limit stopped the tool, the report has `outcome: "success"` and `limit`: `timeout`, `budget` or `max_turns`; a `budget` or `max_turns` the driver could not hold the tool to goes in `unenforced`. A report whose `limit` the lease does not set answers `invalid_report`, as does one with `changes` or `read`, which teamai fills from the clone. + +`src/__tests__/e2e/benchmark-driver-contract.test.ts` checks this contract against a driver that runs a [benchmark](benchmarks.md#your-own-driver). + ## Judge work in your own driver ```text diff --git a/docs/guide/workflow.md b/docs/guide/workflow.md index 7210e12ad..e0b5c96c8 100644 --- a/docs/guide/workflow.md +++ b/docs/guide/workflow.md @@ -24,7 +24,7 @@ steps: `schemaVersion` must be `v1alpha1`. `name` must match the file name. `description` is required. Each step has a unique `id` and exactly one action: `command`, `prompt`, `prompt_file`, or `workflow`. A step can omit the action when it has `end`, `approve`, or `choose`. `agent` is optional on prompt steps. `output` uses the existing shorthand: `string`, `number`, `boolean`, a literal union, an object, or an array with one item type. TeamAI validates successful command stdout or a prompt tool's `output` against it. -Local command steps read stdout and stderr as UTF-8. Non-ASCII JSON string values are preserved in stored outputs and when later steps read them. +Local command steps read stdout and stderr as UTF-8. Non-ASCII JSON string values are preserved in stored outputs and when later steps read them. A command that runs `teamai` runs the teamai that drives the step. A prompt step can set `model`: an alias from `models/aliases.yaml` or a model name, without spaces or templates. `defaults: { model: fast }` at the top of a file sets one for its prompt steps. A prompt step runs on the first of these that is set: @@ -53,6 +53,8 @@ steps: scorers: { names: [review-quality], batch: false } ``` +`scorers: { names: [review-quality], sampling: all }` judges every succeeded pass, whatever each scorer's `samplingRate`, and the run waits for those judges before its [`finally` steps](workflow-advanced.md#finally-steps) and its end. See [A run and its judge](scorers.md#a-run-and-its-judge). + Every step after the first lists the steps it waits for in `needs`. `needs: []` starts another branch with no dependency. Each `next` leases the first step in the file whose needs are done. `--driver local` runs one step at a time unless `workflows.max_parallel` is above 1. A step is skipped when a step it needs failed, or when every step it needs was skipped. A skipped step never runs. A failure does not stop other branches. When nothing is left to run, the run ends `failed` if any step failed, otherwise `succeeded`. A missing `needs`, a `needs` id that is not a step, or a cycle makes the file invalid. A missing `needs` also leaves the workflow with nothing to run, even when an older version was valid. A step can carry an `if` expression. When it is false, the step is skipped, and that does not fail the run. An expression reads `steps..outcome` (`success`, `failed`, or `skipped`) and `steps..outputs.`, only for steps it needs, directly or through their needs. It has `==`, `!=`, `<`, `<=`, `>`, `>=`, `&&`, `||`, `!`, parentheses, and the literals `'text'`, numbers, `true`, `false`, `null`, and `[]`. A path can index an array, as in `steps.review.outputs.files[0]`. The functions `length`, `any`, `all`, and `filter` read arrays. A missing value reads as `null`. `false`, `null`, `0`, and `''` are false; everything else is true. `==` is strict and compares whole values: `'1' == 1` and `'Build' == 'build'` are false. @@ -166,6 +168,21 @@ teamai workflow start build --driver local teamai workflow start build --driver local --tools default=agy # every step on Antigravity CLI ``` +A prompt step can name its own tool, and an exact model: + +```yaml +schemaVersion: v1alpha1 +name: second-opinion +description: Review the change on another tool, on an exact model. +steps: + - id: review + prompt: Review the change. + tool: codex + model: { name: gpt-5.5, effort: high } +``` + +`tool:` wins over `--tools`, `workflows.tools`, `TEAMAI_AI_TOOL` and `ai.tool`. A driver is leased the step only when it lists the tool in `driver.can.tools`. With `--driver local`, a step whose tool cannot be launched here fails `tool_unavailable`. `model: { name, effort }` runs exactly that model and effort, never resolved through `models/aliases.yaml`; a string `model` stays an alias or a model name. + `--driver local` runs a step with no `workspace` in the current checkout, unless an enclosing step opened a workspace. A command there is read-only: if it exits 0 and changed repository files, their permissions, a symlink target, the destination of a staged rename, a file inside a dirty submodule, the index, or HEAD, the step fails and leaves that change in place. This includes a change inside an existing untracked directory; leaving that directory unchanged does not fail the step. A step with `workspace: new` changes files on its own branch. See [Workspaces](#workspaces). Steps that share the current checkout run one at a time. Steps in different worktrees run together up to [`workflows.max_parallel`](workflow-advanced.md#run-local-commands-in-parallel). Each attempt uses `result.json` under the scope's TeamAI data directory, outside the checkout. The driver calls the same `next` and `report` operations shown below: ```text @@ -440,7 +457,7 @@ Any claim takes the offer away: `--next`, `resume ` on a machine without th With [usage reporting](reporting.md) on, every run publishes a privacy-preserving record to the team's `teamai-reports` branch: once when it starts and once when it ends. With it off, nothing is published and the envelopes carry no `record`. Records live in `workflows/runs/.json`, keyed by run id, so a retried publication updates the same record instead of adding a copy. -A record carries only: the run id, workflow name, its `alias` when its definition has one, `sourceHash`, optional `teamRevision`, `origin` (the team's definition, or a [default](#default-workflows) and the teamai version that shipped it), phase (`start` or `end`), driver kind, the run mode it started in (`runMode`), status, the end the run reached (when it reached one), the failure code of a run that failed without reaching an end, when it is a code teamai sets, such as `loop_exhausted` for a loop that ran out or `tool_unavailable` when the local driver cannot find the tool (never its message), start and end times, duration, per-step entries (step path such as `check.classify`, agent, tool, model, outcome, tokens, the pass's `runMode` when its driver sent one on `next`, and what the pass delivered (`outcomes`: the commit shas on its branch, the pull request delivered for it, and the ids of the learnings `teamai contribute` recorded in its attempt) — a mixed-agent run records each step's own fields; `tool` is kept only when it is one of the local driver's tools, and `model` is the name the driver reported, kept only when it is shaped like a model name), summary step counts (succeeded, failed, skipped, cancelled), the cumulative settled-attempt count, the pull requests the run delivered (`prs`), and the publishing username. `sourceHash` identifies the exact workflow and prompt bytes that ran. `teamRevision` is present only when one remote-reachable team commit stores those exact bytes for the parent and every child workflow; local or unpushed edits and mixed cached revisions leave it out. Every entry has its `pass` number, and a step that a loop ran more than once has one entry per pass. An item of a `for_each` counts the times that item ran, so its `pass` can be lower than its step's when an earlier pass had fewer items. A cancelled step's entry has the outcome `cancelled`, and a skipped step's has none. The counts cover each step once, as the run ended. Steps of a sub-workflow that never started are not counted. Prompts, command text, input values, `env` values, outputs, diffs, secret values, local workspace paths, session ids, and hostnames have no field to land in — the record is built from an allowlist, not filtered from the run state. +A record carries only: the run id, workflow name, its `alias` when its definition has one, `sourceHash`, optional `teamRevision`, `origin` (the team's definition, or a [default](#default-workflows) and the teamai version that shipped it), phase (`start` or `end`), driver kind, the run mode it started in (`runMode`), status, the end the run reached (when it reached one), the failure code of a run that failed without reaching an end, when it is a code teamai sets, such as `loop_exhausted` for a loop that ran out or `tool_unavailable` when the local driver cannot find the tool (never its message), start and end times, duration, per-step entries (step path such as `check.classify`, agent, tool, model, outcome, tokens, the pass's `runMode` when its driver sent one on `next`, and what the pass delivered (`outcomes`: the commit shas on its branch, the pull request delivered for it, and the ids of the learnings `teamai contribute` recorded in its attempt) — a mixed-agent run records each step's own fields; `tool` is kept only when it is one of the local driver's tools, and `model` is the name the driver reported, kept only when it is shaped like a model name), summary step counts (succeeded, failed, skipped, cancelled), the cumulative settled-attempt count, the pull requests the run delivered (`prs`), `isolated: true` for an [isolated run](workflow-advanced.md#isolated-runs), and the publishing username. `sourceHash` identifies the exact workflow and prompt bytes that ran. `teamRevision` is present only when one remote-reachable team commit stores those exact bytes for the parent and every child workflow; local or unpushed edits and mixed cached revisions leave it out. Every entry has its `pass` number, and a step that a loop ran more than once has one entry per pass. An item of a `for_each` counts the times that item ran, so its `pass` can be lower than its step's when an earlier pass had fewer items. A cancelled step's entry has the outcome `cancelled`, and a skipped step's has none. The counts cover each step once, as the run ended. Steps of a sub-workflow that never started are not counted. Prompts, command text, input values, `env` values, outputs, diffs, secret values, local workspace paths, session ids, and hostnames have no field to land in — the record is built from an allowlist, not filtered from the run state. An entry's `agent` is what that pass ran as, as observed. It holds the step's `agent:` definition and how the tool got it (`main` when the tool reads its own agent file, `prompt` when the instructions are prepended), the tool and its version, the model and effort the launch asked for and the model the tool reported, the team commit the checkout last pulled and whether a delivered file was edited since, the project commit, the TeamAI version, and the driver kind. A value nobody observed is left out. `fingerprint` hashes the definition, tool, model, effort and team commit, so two passes with the same fingerprint ran with the same recorded parts. It is left out when the model or the team commit is unknown, or a delivered file was edited. A pass nothing observed, and a record from an older TeamAI, carry only the definition's name. While a step runs, the tool it starts, and any session that tool starts in the same checkout, skips the session-start pull when the checkout's last pull delivered completely to that tool, so the step keeps the team commit it records. @@ -450,6 +467,14 @@ An entry's `agent` is what that pass ran as, as observed. It holds the step's `a jq -r 'select((.prs // []) | index("")) | .run' workflows/runs/*.json ``` +A [benchmark run](benchmarks.md) also writes a benchmark record, `benchmarks/runs/.json`, so the comparison can be rebuilt after the suite has changed. Its `report` step writes it into the run's directory on this machine, and publishes it with usage reporting on, as the run's own records go: queued, and retried on the next `teamai pull` when publication fails. It holds the suite's name and `sourceHash`, the repetitions, `limits`, the suite's [trial limits](benchmarks.md#trial-limits) when it set them, the baseline team revision, each scorer that judges its trials with its rubric `sourceHash` (`scorers`: the suite's, and `teamai:correctness:` for each task judged by success criteria), each task's id, starting commit, `sourceHash` and `baseline`, each arm's name, tool, team (`pulled`, `none`, a ref or `without` paths, with its revision), `workflow` (name and `sourceHash`) for an arm that runs one, and the model and effort its model alias resolved to for its tool at that revision, with their `source`: `team`, or `member` under `benchmark run --member-aliases`. `skillChanges` lists the team skills each arm adds, removes or changes. It has one entry per planned trial: task, arm, repetition, order, the trial's step, its `result`, `limit` when a trial limit ended its prompt, `unenforced`, the trial limits its driver did not enforce, the prompt's `durationMs` (around the tool launch only), `tokens` and `cost` (`{ amount, unit }`), only as the tool stated them, each with `complete: false` when a prompt of a workflow arm stated none, `used`, each team skill the trial loaded (`loaded`, or `unknown` when a prompt that did not load it went unobserved), and `scoringCost` (`{ amount, unit, complete }`), what this install's judges of the trial stated they cost, failed ones included, with `complete: false` when one stated none. It also holds `schemaVersion`, `run`, the run's `outcome` before its `finally` steps, `teamaiVersion`, `startedAt`, `finishedAt`, `username`, and `workflow`, the `teamai-benchmark` workflow that ran (`name`, `origin` and `sourceHash`). `baseline` is what the task's check did on a fresh clone of its starting commit, before any trial: `valid` when it failed there, as a task expects, `invalid` when it passed, `could_not_run` when it could not run, `not_run` when no driver ran it, and `not_needed` for a task judged by success criteria. Only the trials of a `valid` task run. A result is `passed` or `failed` when the task's check ran to a verdict, `error` when the trial or its check could not finish, `passed` or `failed` for a task judged by success criteria by the correctness label its judge gave, `unscored` when that judge failed or the run ended first, and `not_run` for a trial that never started. The suite's scorers judge every trial, whatever their `samplingRate`, before `report` runs. A trial result never becomes the run's status or a step outcome, and a benchmark run counts in no workflow total. + +A benchmark run's plan is frozen when it starts: editing the suite and pulling during the run changes nothing it runs. A trial cancelled before a driver leased it is `not_run`, and one cancelled after is `error`. To compute each arm's pass rate from the benchmark record alone, as the table prints it, run this in a checkout of the team repo's `teamai-reports` branch: + +```text +jq -r '. as $b | $b.arms[] | .name as $arm | [$b.trials[] | select(.arm == $arm)] | (map(select(.result == "passed")) | length) as $passed | "\($arm) \($passed / length * 100 | round)% (\($passed)/\(length))"' benchmarks/runs/.json +``` + The operation that settles the run queues its end record under the run lock and publishes it after releasing that lock. Retry updates the same record path when the run ends again. Answer entries publish choice, channel, and agent only; notes, answer timestamps, and answer driver identities stay local. A publication that fails — no remote, a busy reports lock, a rejected push — is queued in this scope's data directory and retried on the next `teamai pull`. The `start` and `next` envelopes then carry `record: pending` instead of `record: published`. The run itself is unaffected: a machine that dies after starting a run leaves it visible as started but not ended. Run ids are `r` plus 16 hex digits. Runs created before this scheme used 8 hex digits and still resolve. @@ -532,7 +557,7 @@ After the start, every reply is a turn: the actions for the agent, the steps tha `teamai skill get workflow` prints the instructions the agent follows. `teamai workflow --help` lists every option. -The skill driver declares `env: "commands"` (see [Requirements and limits](workflow-advanced.md#requirements-and-limits)). A command step and setup get their `requires.env` keys as with the local driver. A prompt step with `requires.env` waits, because TeamAI scopes team env only in the processes it starts. The driver never declares `tokens`, `max_turns`, or `network`, since no host reports a child's tokens or turns to TeamAI. A headless prompt step under a `budget` waits, so a run-level `budget` holds every headless prompt step of the run. `max_turns` and `requires.network: false` are listed in `unenforced`. A worktree is not a sandbox: `requires.isolation: sandbox` is listed in `unenforced` too. +The skill driver declares `env: "commands"` (see [Requirements and limits](workflow-advanced.md#requirements-and-limits)). A command step and setup get their `requires.env` keys as with the local driver. A prompt step with `requires.env` waits, because TeamAI scopes team env only in the processes it starts. The driver never declares `tokens`, `max_turns`, or `network`, since no host reports a child's tokens or turns to TeamAI, nor `unattended`, so a prompt step with `permissions: unattended` waits. A headless prompt step under a `budget` waits, so a run-level `budget` holds every headless prompt step of the run. `max_turns` and `requires.network: false` are listed in `unenforced`. A worktree is not a sandbox: `requires.isolation: sandbox` is listed in `unenforced` too. When a step that waits for env or a budget works in a workspace that an earlier step of the run opens, describe sends the whole run to the local driver, for the person to start in their own terminal. Otherwise the session runs what it can. Once only waiting steps are left, it hands the run to the person, who continues it with `teamai workflow resume ` in their own terminal. The local driver runs its prompt steps with the headless tool that `workflows.tools` in local config names, or that `--tools` picks. When none of `TEAMAI_AI_TOOL`, `workflows.tools.default` and either `ai.tool` names a tool, the local start and the `resume` command the session gives carry `--tools default=`. @@ -568,12 +593,13 @@ When a step lists [scorers](scorers.md) and the confined judge of the session's ## Default workflows -teamai ships two workflows a team gets without writing them. Both start with `teamai-`, so you can tell them from the team's: +teamai ships workflows a team gets without writing them: | Workflow | What it does | Its skill appears | |---|---|---| | `teamai-share-learnings` | shares what the session taught as team learnings, one or several, then runs `teamai-share-session` | while [recall](knowledge.md) is on and the source is writable, since its first step runs `teamai skill get share`; a team copy of it too | | `teamai-share-session` | publishes the session's transcript, scrubbed, and scores it with the team's scorers | always | +| `teamai-benchmark` | runs a [benchmark suite](benchmarks.md#the-benchmark-workflow) through `teamai benchmark run` | always | A team file with the same name replaces a default: @@ -600,6 +626,8 @@ For one name, the first runnable version wins: the project's team file, then the ⚠ [workflows] workflows/teamai-share-session.yaml has no runnable version, so teamai's default teamai-share-session stays in effect ``` +The exception is a name that two namespaces you receive both define: it has no runnable version, and a start of it is refused with `workflow_namespace_conflict`, naming both files. + `teamai-share-learnings` runs `teamai-share-session` with its five inputs, so a team `teamai-share-session`, or a team workflow whose alias is `teamai-share-session`, declares the same five with the same types, even the ones it ignores: ```yaml diff --git a/docs/guide/zh-CN/benchmarks.md b/docs/guide/zh-CN/benchmarks.md new file mode 100644 index 000000000..2c1991bec --- /dev/null +++ b/docs/guide/zh-CN/benchmarks.md @@ -0,0 +1,551 @@ +# Benchmark + +> 本文是 [TeamAI CLI 使用指南](../../usage-guide.zh-CN.md) 的一部分。 + +```text +teamai benchmark run fix-auth --sandbox +``` + +benchmark 套件用团队自己的任务比较工具或模型。一次运行把每个任务交给每个 arm 各做几次，检查每个结果，然后打印表格。trial 只在声明为 sandbox 的机器上运行，例如 CI runner 或容器。 + +一次 benchmark 运行就是工作流 `teamai-benchmark` 的一次运行，这个工作流由 teamai 提供，团队可以替换它。见 [benchmark 工作流](#benchmark-工作流)。 + +## 编写套件 + +套件是团队仓库里的一个目录：`benchmarks//suite.yaml`，以及 `tasks/` 里每个任务一个文件。 + +```yaml +# benchmarks/fix-auth/suite.yaml +schemaVersion: v1alpha1 +name: fix-auth +description: Fix bugs in the auth service. +repo: https://github.com/acme/auth-service.git +defaults: + tool: claude +arms: + - name: claude + - name: codex + tool: codex + - name: strong + model: strong +repetitions: 3 +timeout: 45m +setup: + command: npm ci +``` + +| 键 | 值 | +| --- | --- | +| `repo` | 任务所基于的项目：不带凭据的 https、ssh 或 git URL。 | +| `defaults` | 其他 arm 用来对比的那个 arm：`tool`，以及可选的 `model`、`team` 和 `workflow`。它不是工作流的 `defaults`。 | +| `arms` | 1 到 6 个。每个最多改变一个 `defaults` 键，最多只有一个什么都不改。 | +| `matrix` | 代替 `arms`：teamai 把这些值列表交叉组合成 arm。见[用 matrix 交叉工具和团队资产](#用-matrix-交叉工具和团队资产)。 | +| `team` | arm 运行时使用的团队资产。见[使用其他团队资产的 arm](#使用其他团队资产的-arm)。 | +| `workflow` | 每个 trial 在任务上运行的团队工作流，代替单个 prompt。见[运行工作流的 arm](#运行工作流的-arm)。 | +| `model` | `models/aliases.yaml` 里的模型别名（运行启动时按每个 arm 的工具解析），或一个模型名。 | +| `repetitions` | 1 到 10，默认 3。 | +| `parallel` | 运行基线和 trial 的通道数，1 到 32，默认 1。见[并行通道](#并行通道)。 | +| `timeout` | 必填：整个运行的超时。 | +| `trial` | 可选：每个 trial 的 prompt 的限制。见 [trial 限制](#trial-限制)。 | +| `setup` | 可选，即[工作流的 `setup`](workflow-advanced.md#要求和限制)。在每个全新克隆中、工具或 check 之前运行。 | +| `scorers` | 可选，评判每个 trial 的[评分器](scorers.md)。没有有效 `scorers/.md` 的名字会让 `teamai review --branch` 失败，`teamai pull` 会警告，start 会拒绝该套件。 | + +### 使用其他团队资产的 arm + +arm 默认使用本检出目录 pull 到的团队修订，除非它设置了 `team`。可以用它衡量团队资产带来了多少提升、合并前的某个分支，或某一个 skill： + +```yaml +defaults: + tool: claude +arms: + - name: pulled # 你 pull 到的修订 + - name: bare + team: none # 完全不带团队资产 + - name: next + team: next # 团队仓库的 next 分支 + - name: no-auth-tokens + team: + without: [skills/auth-tokens] # pull 到的修订，去掉这个 skill +``` + +| `team` | arm 运行时使用 | +| --- | --- | +| `pulled`，或不写 `team` | 本检出目录 pull 到的团队修订。 | +| `none` | 不带团队资产。如果本机 user 范围的 teamai 安装会把资产交付给该 arm 的工具，start 会拒绝它，因为它的 trial 会读到你 home 目录里的这些资产。 | +| 分支、tag 或提交 | 团队仓库的该修订，在运行启动时解析。分支按团队仓库里的名字写，例如 `next`，而不是 `origin/next`。 | +| `without: [, ...]` | pull 到的修订，去掉团队仓库中的这些文件和目录。见[去掉团队资产](#去掉团队资产)。 | + +记录中的 `skillChanges` 为每个使用其他修订或 `without` 路径的 arm 列出它相对 pull 到的修订新增、去掉或改变的团队 skill，结果会显示每个 arm 加载它们的频率。见[结果](#结果)。 + +### 去掉团队资产 + +`without` 接受团队仓库中的路径，写法与 git 列出的一致：一个 skill、一条 rule、一个 agent、一个 `claudemd/` 文件、一篇文档、整个命名空间目录，或任何其他文件，例如 `hooks/hooks.yaml`： + +```yaml +arms: + - name: pulled + - name: no-release + team: + without: [skills/release, skills/ops/release] + - name: no-ops + team: + without: [skills/ops, rules/ops] +``` + +运行先规划每个 arm，按 pull 交付给你的方式，带上你的角色、项目、tag 和排除的 skill。以下情况它会拒绝该 arm，并且不启动运行： + +| `without` 列出 | 原因 | +| --- | --- | +| pull 到的修订中没有的路径，或 `teamai.yaml` | 没有可去掉的内容，或者没有 trial 能安装团队。 | +| 位于它列出的另一个路径之内的路径 | 只列出外层那个。 | +| `skills/`、`rules/`、`agents/`、`claudemd/` 或 `docs/` 下、无论去不去掉 pull 都交付得一样的路径 | 去掉它什么都不改变。消息会说明原因：你的角色或项目不接收的命名空间、只通过你没有订阅的 tag 才接收的共享 skill、你排除的 skill、只带有你没有订阅的 tag 的 rule。 | +| 你接收的某个命名空间会用自己的副本代替的共享路径 | 命名空间的副本仍会到达 trial：消息会说 `list skills/ops/release too`。 | + +其他位置的路径只需要存在。`without` 路径会去掉整个文件：要去掉共享文件中的一个条目，例如一个 MCP 服务器、hook、env 键或模型别名，推送一个不含它的团队仓库分支，并指定这个分支： + +```yaml +arms: + - name: pulled + - name: no-github-mcp + team: no-github-mcp # 团队仓库的一个分支，其 mcp/mcp.yaml 去掉了 github 服务器 +``` + +benchmark 不在单仓库模式下运行，因为那里由项目自己的提交决定团队资产：运行会以 `team_ref_blocked` 拒绝每个 arm。 + +### 运行工作流的 arm + +带 `workflow` 的 arm 在每个任务上运行一个团队工作流，而不是单个 prompt。工作流把任务的 prompt 作为输入 `task`： + +```yaml +defaults: + tool: claude +arms: + - name: prompt + - name: plan-then-fix + workflow: plan-then-fix # workflows/plan-then-fix.yaml，带 inputs: { task: string } +``` + +运行在 arm 的团队修订上读取该工作流及其运行的工作流，像任何团队工作流一样经过你接收的命名空间，并在 arm 的工具、模型和 effort 上运行其中每个 prompt 步骤，像 trial 自己的 prompt 一样无人值守。工作流没有输入 `task`、还要求其他输入，或有 `finally` 步骤时，它的某个步骤会交付、询问某个人、打开 `continue` 以外的工作区，或运行 `workflow: { from }` 时，或者它的某个 prompt 运行 `agent` 而 arm 没有设置 `model`（agent 自己的模型可能取代工具的默认模型；请在 `defaults` 中设置 `model`：带 `workflow` 的 arm 不能再改其他键）时，运行拒绝该 arm（`arm_workflow_unrunnable`）；你接收的两个命名空间都有该工作流时，以 `workflow_namespace_conflict` 拒绝。trial 的 check 或评判根据工作流留在克隆中的内容判定。它的时长是各个 prompt 轮次之和，token 和成本是各个 prompt 之和。[trial 限制](#trial-限制)的 `timeout` 和 `budget` 是它的 prompt 加起来可以花费的量；`max_turns` 不执行，会列在 `unenforced` 中。 + +### 用 matrix 交叉工具和团队资产 + +`matrix` 为其值的每种组合生成一个 arm，因此任意两个只差一个值的 arm 只比较这一个值。下面这个例子同时回答：团队资产对每个工具帮助多大，以及带上资产后哪个工具更好： + +```yaml +defaults: + tool: claude +matrix: + tool: [claude, codex] + team: [pulled, none] +``` + +| Arm | 工具 | 团队 | +| --- | --- | --- | +| `claude-pulled` | claude | pulled | +| `claude-none` | claude | none | +| `codex-pulled` | codex | pulled | +| `codex-none` | codex | none | + +matrix 可以变化 `tool`、`model`、`team` 或 `workflow`，其中至少一个要有两个或更多值，并生成 2 到 12 个 arm。它没写的键取自 `defaults`。每个 arm 的名字由各个变化键的值拼接而成：ref 原样使用，字母、数字、`-`、`_` 以外的字符写成 `-`，`without: [skills/auth-tokens]` 写成 `without-skills-auth-tokens`。一个套件只能有 `arms` 或 `matrix` 之一。 + +任务从 `repo` 的一个完整提交开始，并且恰好由 `check` 和 `successCriteria` 之一判定： + +```yaml +# benchmarks/fix-auth/tasks/expired-token.yaml +schemaVersion: v1alpha1 +title: Reject expired refresh tokens +prompt: | + Refresh tokens past their expiry are still accepted. Fix it. +startingCommit: 3f2c9a7d1e0b4c6a8f5d2e1b0c9a8f7e6d5c4b3a +check: + run: npx vitest run test/auth/refresh.test.ts + protect: ['test/**', 'vitest.config.ts', 'package.json', 'package-lock.json'] + add: [test/auth/refresh.test.ts] +source: { platform: claude-code, sessionId: 7c1e2b9a, fix: 9a1d4e7c2b8f0a3d6e5c1b9f8a7d2e4c6b0a1f3e } +``` + +```yaml +# benchmarks/fix-auth/tasks/login-copy.yaml +schemaVersion: v1alpha1 +title: Clearer login error +prompt: Make the login error say which field is wrong. +startingCommit: 3f2c9a7d1e0b4c6a8f5d2e1b0c9a8f7e6d5c4b3a +successCriteria: The error names the wrong field and still reveals nothing about which accounts exist. +``` + +| 键 | 值 | +| --- | --- | +| `startingCommit` | 任务开始时 `repo` 的完整提交。 | +| `check` | `run`：任务完成后以 0 退出的命令；`protect`：agent 不能提供的路径。 | +| `check.add` | 可选：check 之前写入、agent 永远拿不到的文件：按 `source.fix` 中的样子写入的文件，例如修复的测试；或者 `{ from, to }`，套件目录中的一个文件或目录。目录会加入其下的所有文件，但不包括以点开头的条目，例如 `.DS_Store`；要加入点文件，在 `from` 中直接指明它。见[修复带来的测试](#修复带来的测试)和[套件目录中的测试](#套件目录中的测试)。 | +| `successCriteria` | 代替 `check`：由评判器按这些标准判定。 | +| `source` | 可选：任务来自的会话，`platform` 和 `sessionId`，以及 `fix`，即在那里修复它的完整提交，[`benchmark try`](#试运行任务) 会检查它。 | + +`teamai pull` 和 `teamai review --branch` 检查每个文件，并带文件和行号报告每个问题。`benchmarks///` 下的套件只发给该角色或项目的成员，见[角色和项目](distribution.md#角色和项目)。 + +### `protect` 要覆盖什么 + +agent 完成工作后 check 以 0 退出，trial 就通过。check 之前，teamai 把每个 `protect` 路径恢复成起始提交里的样子： + +```text +the agent left the check sees +src/auth/refresh.ts edited edited +test/auth/session.test.ts edited as 3f2c9a7 has it +test/helpers/skip.js added deleted +vitest.config.ts edited as 3f2c9a7 has it +``` + +所以 `protect` 要列出 check 读取、而 agent 不能提供的一切：测试、测试配置、它们加载的脚本和辅助文件，以及包清单和锁文件。 + +| 不要保护 | 原因 | +| --- | --- | +| `setup` 写入的路径，例如构建目录 | 恢复会删除 setup 写在那里的内容。 | +| git 检出时会转换的文件：Git LFS 文件、带 `eol=crlf` 属性的文件 | 每个受保护文件都必须与起始提交存储的字节完全一致，所以 trial 的结果是 `error`。 | +| 子模块内部或其下的路径 | teamai 无法恢复子模块里的文件，所以 trial 的结果是 `error`。 | + +check 读取的内容无法一一列出的任务，应改用 `successCriteria`。 + +### 修复带来的测试 + +如果证明修复的测试是修复本身带来的，把这个测试文件列进 `check.add`；修复只是在起始提交已有的文件里加了一个用例时也一样。check 读到的是 `source.fix` 中的版本，agent 永远看不到： + +```text +step test/auth/refresh.test.ts +the agent's clone absent +trial check protect restored, then written from 9a1d4e7 +baseline at 3f2c9a7 written from 9a1d4e7: the check must fail +try at 3f2c9a7 written from 9a1d4e7: the check must fail +try at 9a1d4e7 protect restored, then written from 9a1d4e7: the check must pass +``` + +| `teamai pull` 和 `teamai review --branch` 拒绝 | 原因 | +| --- | --- | +| 没有 `source.fix` 的 `check.add` | 这些文件来自 `source.fix`。 | +| 没有任何 `protect` 条目匹配的 `check.add` 路径 | agent 可以自己提供它。 | +| `check.add` 里的通配符 | 每一项都只指明一个文件。 | + +### 套件目录中的测试 + +任务的提交里没有的测试，例如你专门为任务写的测试，可以放在套件目录中。`check.add` 的 `{ from, to }` 条目把它写到 check 读取的位置，写入的时机和修复的测试相同；agent 永远拿不到它： + +```text +benchmarks/fix-auth/ + suite.yaml + tasks/expired-token.yaml + check-files/expired-token/refresh.test.ts +``` + +```yaml +check: + run: npx vitest run test/auth/refresh.test.ts + protect: ['test/**', 'vitest.config.ts', 'package.json', 'package-lock.json'] + add: + - from: check-files/expired-token/refresh.test.ts + to: test/auth/refresh.test.ts +``` + +`from` 是套件目录中的一个文件或目录；目录会把其下的每个文件写到 `to` 之下。每个文件按字节写入，保留可执行位。`teamai pull` 把这些文件和套件一起保存，运行写入的是它启动时的版本，所以运行期间 pull 不会改变任何 check。修改其中一个文件会改变任务的 `sourceHash`。这种条目不需要 `source.fix`。 + +| `teamai pull` 和 `teamai review --branch` 拒绝 | 原因 | +| --- | --- | +| `from` 在套件目录中找不到、指向套件目录之外，或者是符号链接 | 只读取套件目录自己的文件。 | +| 带 `..` 或通配符的 `to` | 它只指明克隆中的一个路径。 | +| 没有任何 `protect` 条目匹配的 `to` 或其下的文件 | agent 可以自己提供它。 | +| 两个条目写同一个路径 | 每个路径只写一次。 | + +### 基线 + +在任务的第一个 trial 之前，teamai 在起始提交的全新克隆上运行 setup，写入任务的 `check.add` 文件，并运行一次 check： + +| check 在起始提交上 | 任务 | +| --- | --- | +| 失败 | 有效：它的 trial 会运行。 | +| 通过 | `invalid: check passes without changes`，不运行 trial。 | +| 无法运行 | `check could not run`，不运行 trial。 | + +| check 无法运行的情况 | 记为 | +| --- | --- | +| 以 127 退出 | `not_found` | +| 以 126 退出 | `not_executable` | +| 在 Windows 以外以 129 到 192 退出 | `signal`：shell 把被信号杀死的命令报告为 128 加上信号编号。不要把它们用作 check 自己的失败退出码。 | +| 运行超过超时时间 | `timeout` | + +prompt 之后无法运行的 check 让 trial 得到 `error`，而不是 `failed`。 + +### trial 限制 + +`trial` 限制每个 trial 的 prompt。达到限制会结束智能体的这一轮，teamai 随后在智能体留下的结果上运行 trial 的 check 或评判，所以用完时间的工具算作任务失败，而不是测试框架出错： + +```yaml +timeout: 45m +trial: + timeout: 8m + budget: { tokens: 300000 } + max_turns: 60 +``` + +| 键 | 限制 | 在本地驱动上 | +| --- | --- | --- | +| `timeout` | 从工具启动起的时间。要短于套件的 `timeout`，后者仍然结束整个运行。 | 停止工具及其启动的每个进程。 | +| `budget.tokens` | 输入加输出 token，按 [工作流预算](workflow-advanced.md#要求和限制)的方式计算。 | 在工具运行时从其输出中计数，超过预算后停止工具，所以 trial 可能多出最后一次模型调用。Codex 只在结束时给出 token，CodeBuddy 的 token 不会在运行时计数：两者都会运行到结束，并在 `unenforced` 中列出 `budget`。不报告 token 的工具会让其 trial 以 `error` 结束，除非另一个限制先停止了它：这时该 trial 会在 `unenforced` 中列出 `budget`。 | +| `max_turns` | 智能体轮数，按工具自己的计数。 | 传给 Claude。其他工具在 `unenforced` 中列出 `max_turns`。 | + +被限制停止的 trial 在记录中带 `limit`，并计入 “Stopped by a limit”。在 `unenforced` 中列出套件限制的 trial 算作偏差，所以有上限的 arm 不会被悄悄地与没有上限的 arm 比较。完全无法结束一轮的驱动会把这些限制当作硬性截止时间来运行 trial，并在 `unenforced` 中列出 `on_limit`。运行保持它启动时的限制。 + +## 试运行任务 + +新任务文件加入套件之前，可以先试运行。在套件 `repo` 的检出目录中运行，团队仓库检出在旁边： + +```console +$ teamai benchmark try ../team/benchmarks/fix-auth/tasks/expired-token.yaml +ℹ Task expired-token: trying the check at 3f2c9a7 +ℹ Task expired-token: trying the check at the fix 9a1d4e7, with the protected paths as 3f2c9a7 has them +{"data":{"try":{"task":"expired-token","message":"Task \"expired-token\": the check fails at 3f2c9a7 and passes at the fix 9a1d4e7, with the protected paths as 3f2c9a7 has them."}}} +``` + +```text +task at startingCommit at source.fix (protect restored) exit +pod-startup check fails check passes 0 +old-bug check passes not tried 1: "check passes without changes at 3f2c9a7" +bad-fix check fails check fails 1: "check fails at the fix 9a1d4e7" +test-in-fix check fails check fails (fix's test restored) 1: "The fix changes test/startup.test.ts, which protect puts back as 3f2c9a7 has them: list a test the fix brought in check.add" +changed-test check fails check passes (check.add written) 0 +add-missing not tried not tried 1: "test/missing.test.ts is not a file at the fix 9a1d4e7" +no-fix check fails not tried 0: "the check was not tried on a fix" +criteria-only no check no check 0: "there is no check to try" +missing-tool could not run not tried 1: names the command and 3f2c9a7 +setup-error setup failed not tried 1: names setup and 3f2c9a7 +``` + +`benchmark try` 按原样读取任务文件及其上两级目录中的 `suite.yaml`，并像 `teamai pull` 一样校验两者（连同其 `check.add` 条目从该套件目录指明的文件）。它不需要 `teamai init`。检出目录缺少的提交会像运行时一样，按 SHA 从套件的 `repo` 获取。每个提交都有一个全新克隆、套件的 `setup`、trial 得到的环境（见[运行套件](#运行套件)），以及套件的 `timeout`。它不创建运行，也不写记录。check 会在你的机器上运行项目代码，所以只试运行你信任的任务。 + +## 运行套件 + +```console +$ teamai benchmark run fix-auth --sandbox +ℹ Run r38302f8e6a18c377 started. If it stops before it ends, continue it with: teamai workflow resume r38302f8e6a18c377 --sandbox +``` + +`benchmark run` 先规划套件，所以无法运行的套件不会留下运行。然后它用套件的 `timeout` 启动工作流 `teamai-benchmark`，并用本地驱动驱动它。 + +每个 trial 的 prompt 都以 [`permissions: unattended`](workflow-advanced.md#无人值守的-prompt) 运行，因为 teamai 自带的工作流在严格的 sandbox 中运行 trial：无论这台机器上该工具的设置如何，工具都无需询问就编辑文件和运行命令。 + +| 选项或配置 | 效果 | +| --- | --- | +| `--sandbox` | 只为当前这个进程把本机声明为 sandbox。只在可以丢弃的机器上使用。 | +| 配置里的 `workflows.sandbox: true` | 为每个进程声明它，见[要求和限制](workflow-advanced.md#要求和限制)。 | +| 两者都没有 | 运行在启动时以 `requirement_unmet` 被拒绝：trial 需要严格 sandbox。 | +| `--member-aliases` | 你自己的 `~/.teamai/models/aliases.yaml` 优先于团队的 `models/aliases.yaml`，表格会把由你的文件决定的模型的来源显示为 `member`。 | +| `--fail-on-regression` | 某个 arm 的通过率下降时也以 1 退出。见[回归时失败](#回归时失败)。 | + +| `benchmark run` 退出码 | 何时 | +| --- | --- | +| 0 | 运行成功并写出了报告，无论 trial 的结果如何。 | +| 1 | 套件或启动被拒绝、运行超时或被取消、报告失败或被取消，或者 `--fail-on-regression` 发现了下降。 | + +仍在等待发布的记录从不改变退出码。 + +| trial 得到 | | +| --- | --- | +| 起始提交的全新克隆 | 没有远程仓库。你的检出目录从不被改动。 | +| 团队资产和套件 | 来自本检出目录上次 pull 到的团队修订（该修订必须已推送），或来自 arm 的 `team`。启动时冻结：运行期间 pull 不会改变它运行的任何内容。 | +| 环境 | 去掉 `GITHUB_TOKEN`、`GH_TOKEN`、`GITLAB_TOKEN`、`GL_TOKEN`、`CI_JOB_TOKEN`、`SSH_AUTH_SOCK`、`GIT_ASKPASS` 和 `SSH_ASKPASS`：trials 步骤是[严格 sandbox](workflow-advanced.md#要求和限制)，它让 trial 的每个进程都拿不到这些变量。 | +| 你的主目录 | 保持原样：teamai 去不掉其中的登录信息、MCP 服务器和插件。 | +| 本机上的团队仓库 | 保持原样：你的团队检出目录，以及每个 trial 克隆旁边用来安装其团队资产的那份团队修订副本，都包含 `benchmarks/`，check 文件也在其中。在本机上搜索的 agent 能找到它们。 | + +本地驱动按这个顺序运行每个 trial：全新克隆、团队资产、setup、用 prompt 启动 arm 的工具，然后运行 check。 + +| | | +| --- | --- | +| 工具 | 它能运行 `claude`、`codex` 和 `opencode`。安装 arm 用到的每个工具并登录。找不到工具的 arm，它的 trial 为 `not_run`，其他 arm 照常运行。 | +| 通道 | 一次一个 trial，除非套件设置了 [`parallel`](#并行通道)。 | +| 超时 | 套件的 `timeout` 是整个运行的超时。setup、工具和 check 到时都会停止：它们所属的 trial 记为 `error`，仍未完成的 trial 结算为 `not_run`。让它低于 CI 任务的时限，否则任务会在打印表格之前被杀掉。 | +| 卡住的 trial | 没有 [`trial.timeout`](#trial-限制) 时，一个永不返回的工具会一直运行到套件超时，它之后的每个 trial 结算为 `not_run`。 | +| 评判 | 带 `successCriteria` 的任务和套件的评分器在运行结束前评判，评判在运行了 trial 的那个安装上进行。机器无法评判时，需要评判的套件在启动时被拒绝。 | +| 控制命令 | trial 的步骤要等运行的 `plan` 步骤报告后才存在。在此之前，对 trial 步骤执行 `teamai workflow cancel ` 会返回 `step_not_found`。benchmark 运行是[隔离的](workflow-advanced.md#隔离运行)：`workflow retry`、`publish` 和 `--claim` 拒绝它，`publish` 和 `--claim` 返回 `isolated_run`，retry 会给出 `teamai benchmark run `。 | +| 停下 | 用 `teamai workflow resume --sandbox` 继续。 | + +无法运行的启动会被拒绝，消息说明该怎么做，并且不会创建运行。你的 user 范围 teamai 安装也能到达该团队，或团队通过 HTTP 提供时，运行无法固定 trial 得到的团队，会以 `team_ref_blocked` 拒绝，并说明要改什么。 + +### 并行通道 + +```yaml +parallel: 4 +``` + +基线和 trial 最多在 `parallel` 个通道中运行，每个通道内一个接一个。本地驱动同时最多运行 `workflows.max_parallel` 个，除非你的配置设置了它，否则为 1，所以两者都要设置。并行的 trial 共享机器的 CPU、内存和网络，以及工具的速率限制：时长、有时还有结果，都会与串行运行不同。只比较 `parallel` 相同的运行。 + +## benchmark 工作流 + +`benchmark run` 启动工作流 `teamai-benchmark`。teamai 提供的是这个： + +```yaml +# workflows/teamai-benchmark.yaml，随 npm 包发布 +schemaVersion: v1alpha1 +name: teamai-benchmark +description: Run a benchmark suite. Every arm works the same tasks in fresh clones, judged by checks it never sees. +invoke: deny +inputs: { suite: string, aliases: string } +steps: + - id: plan + command: teamai benchmark plan "$SUITE" --aliases "$ALIASES" + env: { SUITE: "{{ inputs.suite }}", ALIASES: "{{ inputs.aliases }}" } + output: { definition: workflow, plan: json } + - id: trials + needs: [plan] + workflow: { from: plan } + requires: { isolation: sandbox, strict: true } +finally: + - id: report + command: teamai benchmark report + output: { text: string, table: json } +outputs: { text: steps.report.outputs.text, table: steps.report.outputs.table } +``` + +| 命令 | 作为 | 作用 | +| --- | --- | --- | +| `teamai benchmark plan ` | `plan` 步骤 | 冻结你 pull 到的团队修订，获取 trial 读取的每个提交，并把 trial 打印成一个工作流，每个基线和 trial 一个克隆。 | +| `teamai benchmark check --baseline\|--trial` | 每个克隆中的一个步骤 | 写入 `check.add`，对 trial 恢复 `protect`，运行任务的 check，并打印其结果。 | +| `teamai benchmark report` | `finally` 步骤 | 给 trial 分类，写出记录，并打印结果。 | + +`trials` 运行 `plan` 打印的内容：见[运行步骤打印的工作流](workflow-advanced.md#运行步骤打印的工作流)。`report` 在 trial 之后运行，无论它们发生了什么，就像 [finally 步骤](workflow-advanced.md#finally-步骤)那样。`benchmark run --member-aliases` 把 `aliases` 传为 `member`，否则传为 `team`。 + +要修改它，推送你自己的 `workflows/teamai-benchmark.yaml`，或者为某个角色或项目的成员推送 `workflows//teamai-benchmark.yaml`。它会为接收它的人替换 teamai 的版本，就像任何[默认工作流](workflow.md#覆盖默认工作流)一样，`benchmark run` 会说明这一点： + +```text +ℹ Running the team's workflows/ops/teamai-benchmark.yaml, which replaces teamai's benchmark workflow. +``` + +你的版本必须接受输入 `suite`，要用 `--member-aliases` 运行还必须接受输入 `aliases`；否则 `benchmark run` 会拒绝它。保留 `plan` 的步骤 id，并把 `report` 保留为 `finally` 步骤。去掉 `strict: true` 会让 trial 在不扣留凭据的情况下运行，并且使用各工具自己的权限设置，而不是无人值守：不能编辑的工具会让它的 trial 失败，报告的第一行会说明这一点。你接收的两个命名空间都有 `teamai-benchmark.yaml` 时，`benchmark run` 以 `workflow_namespace_conflict` 拒绝，并指出两个文件，绝不退回 teamai 自带的版本。 + +## 结果 + +运行结束时，`benchmark run` 把结果打印到 stderr。stdout 上的 JSON 返回在 `data.done.outputs.text` 中带同样的文本，在 `data.done.outputs.table` 中带数字。对这个套件，再加一个 check 本来就通过的第三个任务 `old-bug`： + +```text +| Arm | Tool | Model (source) | Planned | Passed | Failed | Error | Unscored | Not run | Stopped by a limit | Deviations | Not verified | Pass rate | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| claude | claude | tool default (team) | 9 | 4 | 2 | 0 | 0 | 3 | 0 | 0 | 0 | 44% (4/9) | +| codex | codex | tool default (team) | 9 | 3 | 3 | 0 | 0 | 3 | 0 | 0 | 0 | 33% (3/9) | +| strong | claude | claude-opus-5-5 high (team) | 9 | 6 | 0 | 0 | 0 | 3 | 0 | 0 | 0 | 67% (6/9) | + +| Arm | Median tokens (input + output) | Median duration | Cost | +| --- | --- | --- | --- | +| claude | 41200 (6) | 84.0 s (6) | 1.82 USD (6) | +| codex | 38900 (6) | 97.5 s (6) | unknown (0) | +| strong | 52300 (6) | 121.0 s (6) | 4.1 USD (6) | + +| Task | claude | codex | strong | +| --- | --- | --- | --- | +| expired-token | 33% (1/3) | 33% (1/3) | 100% (3/3) | +| login-copy | 100% (3/3) | 67% (2/3) | 100% (3/3) | +| old-bug | 0% (0/3) | 0% (0/3) | 0% (0/3) | + +| Task | Baseline | +| --- | --- | +| old-bug | invalid: check passes without changes | + +These results hold for these tasks, this run and this machine. +``` + +| 表格 | 内容 | +| --- | --- | +| Arm | 按结果统计每个 arm 的 trial，以及被 [trial 限制](#trial-限制)停止的数量。偏差是 prompt 运行时的工具、团队修订或模型与其 arm 不同的 trial，或者其 driver 无法执行某个 trial 限制的 trial。记录缺少其中某个值的 trial 为未验证。有 arm 使用 pull 到的修订以外的团队资产时，Team 列显示每个 arm 的团队资产，例如 `pulled 3f9c2ab`、`next 81d0e4c`、`3f9c2ab without skills/auth-tokens` 或 `none`。 | +| 用量 | 每个 arm 的中位数和成本合计，并注明各自用了多少个 trial。token 和成本只按工具自己声明的显示，所以不声明的工具显示 `unknown (0)`。 | +| 任务 | 每个任务在每个 arm 上的通过率。 | +| 基线 | 没有运行的任务及原因。 | +| Skill | 仅当某个 arm 改变团队 skill 时出现：它新增、去掉或改变的每个 skill，以及该 arm 的 trial 中有多少加载了它，与什么都不改变的 arm 并列。 | + +```text +| Arm | Skill | Change | Loaded in this arm | Loaded in the defaults arm | +| --- | --- | --- | --- | --- | +| no-auth-tokens | auth-tokens | removed | 0/3 | 2/3, 1 unknown | +``` + +只有 teamai 能看到的加载才计数：`unknown` 是其工具不报告 skill 加载的 trial，而不是没有加载该 skill 的 trial。 + +结果不是运行的状态：所有 trial 都失败的运行仍然成功。运行只在超时或被取消时失败。 + +`report` 把运行的 benchmark 记录写到本机上该运行的目录中。[使用统计上报](reporting.md)开启时，它还会把记录发布到团队仓库 `teamai-reports` 分支的 `benchmarks/runs/.json`，所以任务的 token 还需要团队仓库的推送权限。发布失败的记录会在队列中等待，并在下一次 `teamai pull` 时重试；在此之前，文本以 `record: pending` 结尾。[运行记录](workflow.md#运行记录)列出记录的字段，并给出只从记录算出每个 arm 通过率的 `jq` 命令。关闭时，结果仍在返回中。Dashboard 的[基准测试页面](dashboard.md#基准测试benchmarks)按时间画出已发布的运行和本机的运行。 + +## 回归时失败 + +```console +$ teamai benchmark run fix-auth --sandbox --fail-on-regression +Arm "claude" passed 4/9 paired trials, against 8/9 in run r1c0d9e2f4b6a8c7d: a drop of 44 points, with a 95% interval of -73 to -16 points. +``` + +`--fail-on-regression` 把这次运行与该套件最新的一次更早的运行比较，那次运行必须已成功、已发布在 `teamai-reports` 上，并且任务和 arm 相同。名字、工具、模型、effort、工作流和团队处理方式都相同的 arm 就是同一个 arm，无论它 pull 到的是哪个修订。trial 按任务和重复次数配对，就像 Dashboard 的 Compare 那样配对，某个 arm 的通过率下降超出偶然范围时，运行以 1 退出。仍在本机等待的记录不参与比较。没有这样的运行时，它打印一条提示，不改变退出码。 + +## 在 CI 中运行 + +GitHub Actions 任务： + +```yaml +on: workflow_dispatch +jobs: + benchmark: + runs-on: ubuntu-latest + timeout-minutes: 60 + env: + TEAM_REPO: ${{ vars.TEAM_REPO }} + GH_TOKEN: ${{ secrets.TEAM_TOKEN }} + ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} + steps: + - uses: actions/checkout@v5 + with: { fetch-depth: 0 } + - uses: actions/setup-node@v4 + with: { node-version: 22 } + - run: npm install -g @anthropic-ai/claude-code + - run: npx teamai-cli init "$TEAM_REPO" --scope workspace --agent claude + - run: npx teamai-cli pull + - run: npx teamai-cli benchmark run fix-auth --sandbox > done.json + - if: always() + run: jq -r '.data.done.outputs.text // empty' done.json >> "$GITHUB_STEP_SUMMARY" +``` + +- `TEAM_TOKEN` 读取团队仓库，并在使用统计上报开启时推送到团队仓库。 +- `--agent claude` 为 Claude Code 设置 teamai，全新的 runner 还从未运行过它。 +- 套件 `repo` 的检出带完整历史，所以每个任务的起始提交都在。 +- 最后一步把结果写进任务摘要，运行失败时也会写。 + +GitLab CI 任务： + +```yaml +benchmark: + image: node:22 + timeout: 1h + when: manual + script: + - npm install -g @anthropic-ai/claude-code + - npx teamai-cli init "$TEAM_REPO" --scope workspace --agent claude + - npx teamai-cli pull + - npx teamai-cli benchmark run fix-auth --sandbox > done.json + artifacts: + when: always + paths: [done.json] +``` + +把 `TEAM_REPO`、`GITLAB_TOKEN`（读取团队仓库）和 `ANTHROPIC_API_KEY` 设为 masked CI/CD 变量。任务日志会显示结果，`done.json` 会保存结果。换成别的工具时，同样安装它并设置它的密钥：每个 arm 的工具都必须在 runner 上。 + +## 在自己的驱动里运行 + +benchmark 运行是一次普通的工作流运行，所以任何驱动都通过 `workflow next --prepare` 和 `workflow report` 运行它，见[在自己的驱动里使用克隆工作区](workflow-advanced.md#在自己的驱动里使用克隆工作区)： + +```text +teamai workflow start teamai-benchmark --inputs '{"suite":"fix-auth","aliases":"team"}' --timeout 45m +teamai workflow next --prepare --json '{"driver":{..., "can":{"workspace":["read-only","new","continue","clone"],"isolation":"sandbox","env":true,"setup":true,"tools":["claude"],"unattended":true,"end_turn":true}}}' + -> a command lease (plan, check, report): run lease.command in lease.cwd, or in your checkout without one, with TEAMAI_WORKFLOW_ATTEMPT=/ + -> a prompt lease: launch lease.tool in lease.cwd on lease.prompt, with lease.model, in its unattended mode for lease.permissions, held to lease.limits +teamai workflow report --json '{..., "tool":"claude", "durationMs":84000, "cost":{"amount":0.31,"unit":"USD"}}' +``` + +| | | +| --- | --- | +| `driver.can` | 每次 `next` 都带：`workspace` 中的 `clone`、`isolation: "sandbox"`、`env: true`、`setup: true`、`tools` 中列出各 arm 的工具、`unattended: true`，以及在租约的限制处结束一轮时带 `end_turn: true`。驱动缺少其中任何一项时，trial 会被扣住，`wait.held[].missing` 会指明缺的是哪项。 | +| `--prepare` | teamai 在本机创建每个克隆、安装其团队资产并运行 setup，然后返回带 `cwd` 的租约。 | +| 每个进程 | 运行时不带 `lease.envScope.withhold` 中的名字，带 `lease.env` 和 `TEAMAI_WORKFLOW_ATTEMPT=/`，所以从中调用的 `teamai` 不会改变任何运行。命令步骤把它的输出以 JSON 打印到 stdout；把它们作为 `outputs` 报告。 | +| `report` | 对 prompt：`tool`、只围绕工具启动测得的 `durationMs`，以及仅在工具给出时才带的 `cost`。 | +| 限制 | 让工具遵守 `lease.limits`。某个限制停止了工具时，报告 `outcome: "success"` 并带 `limit`：`timeout`、`budget` 或 `max_turns`；无法让工具遵守的 `budget` 或 `max_turns` 列在 `unenforced` 中。 | +| 超时 | `benchmark run` 把套件的 `timeout` 作为 `--timeout` 传入；直接执行 `workflow start teamai-benchmark` 时也必须传入它，否则 `plan` 会失败。 | +| 评判 | 套件的评分器仍在评判时，`next` 返回带 `judging` 的 `wait`：在运行了 trial 的那个安装上运行 `teamai score --next`。 | diff --git a/docs/guide/zh-CN/dashboard.md b/docs/guide/zh-CN/dashboard.md index 7b091c62c..2fff4be47 100644 --- a/docs/guide/zh-CN/dashboard.md +++ b/docs/guide/zh-CN/dashboard.md @@ -9,7 +9,7 @@ teamai dashboard teamai dashboard --port 8080 ``` -浏览器打开 `http://localhost:3721`。侧栏是总览，下面分三组：执行（会话、Token 与费用、工作流运行、评分器）、上下文（知识库、资源、分发）、改进（改进中心）。会话和趋势是这台机器上的；知识库和改进中心读团队仓库。 +浏览器打开 `http://localhost:3721`。侧栏是总览，下面分三组：执行（会话、Token 与费用、工作流运行、评分器、基准测试）、上下文（知识库、资源、分发）、改进（改进中心）。会话和趋势是这台机器上的；知识库和改进中心读团队仓库。 ## 改进中心 @@ -61,7 +61,7 @@ teamai dashboard --port 8080 在交出运行的那个安装上，下一步改为 `teamai workflow resume `，在那里把它收回。待接手的本机运行，或现在由其他安装持有的本机运行，会在状态下方注明。**运行器需要**（需要人工的运行上为**其机器需要**）列出剩余步骤的 `requires.env` 键和提示词步骤：照[其作业](workflow.md#从-ci-接手交出的运行)那样在运行器上配好。**本机缺少**是本机缺的那部分，与 `teamai workflow resume` 列为 `cannot run here` 的一致。状态栏统计待接手的运行和需要人工的运行。页面本身不认领任何运行：命令在接手运行的机器上启动驱动。 -页面显示运行记录会发布的内容，以及每个步骤的实时状态和等待中 HITL 请求的选项。输入、`for_each` 条目、输出、提示词、HITL 消息和本地路径都留在运行文件里。HITL 问题请在运行的驱动提问处阅读。运行详情显示每个 pass 的产出：它分支上的 commit、为它交付的拉取请求，以及 `teamai contribute` 在它的 attempt 中记录的经验。 +页面显示运行记录会发布的内容，以及每个步骤的实时状态和等待中 HITL 请求的选项。输入、`for_each` 条目、输出、提示词、HITL 消息和本地路径都留在运行文件里。HITL 问题请在运行的驱动提问处阅读。运行详情显示每个 pass 的产出：它分支上的 commit、为它交付的拉取请求，以及 `teamai contribute` 在它的 attempt 中记录的经验。已结束的[基准测试](benchmarks.md)运行的详情显示它的结果、用量和任务 × arm 表格，与 CLI 打印的一致。 等待中的 approve 或 choose 请求可以在运行详情里回答。回答和 CLI 走同一个 `respond` 操作，渠道为 `dashboard`，`agent: false`，因此可以回答必需审批。在 agent 会话中启动的仪表盘记录为 `agent: true`，与从该 shell 回答相同，不能回答必需审批。回答带上页面显示的 pass：如果该请求期间已被回答或替换，页面会显示拒绝原因并刷新。其他网站无法提交回答。不带 `Origin` 请求头的请求（如普通 `curl`）也会被拒绝。这只表明回答来自浏览器，并不能阻止自行设置该请求头的本地脚本。 @@ -108,6 +108,45 @@ partial 未通过 alice good 通过 codex / gpt-5.1-codex 10/2 来自会话和工作流运行的标签反映的是工具在实际遇到的工作上的表现。要在相同任务上比较工具、模型或团队资产，请运行基准测试。 +## 基准测试（Benchmarks） + +基准测试页面显示所选工作区每一次已结束的[基准测试](benchmarks.md)运行：本机上的运行，以及 `teamai-reports` 上的运行记录，时间窗口为 30、90 或 365 天。在 **套件** 中选择一个套件。下面每个问题都指出回答它的面板，以及该面板在一个团队里显示的内容：这个团队每周运行 `fix-auth`，并用三个 Claude 模型运行 `migrate-db`： + +| 问题 | 面板 | 显示的内容 | +|---|---|---| +| 团队的 auth 修复要不要改用 claude？ | **领先的 arm** | `claude 67% · 95% 57%–75% · 未能区分于 codex`。codex 是 56%（47%–65%）。两个区间重叠，所以这 7 次运行区分不了两者。改用之前先增加重复次数或任务。 | +| 我们 10 月 7 日合入的 auth 技能有用吗？ | **每次运行的通过率** | 一条点线 `团队资产已改动 · 62432de` 标出第一次带这个技能的运行。比较每个 arm 在线两侧的点。**运行** 表把这次运行的团队修订标为 `已改动`。 | +| `migrate-db` 上哪个模型值得花钱？ | **通过率与成本**，选 **成本** | sonnet 以每个 trial 0.31 USD 通过 73%，opus 以 0.89 USD 通过 78%，两者区间重叠。虚线连接通过率和成本都无人胜过的 arm：haiku、sonnet、opus。 | +| 哪个任务需要修，是代码还是套件？ | **任务 × arm** | `refresh-race` 的 **不稳定的运行** 是 7/7：每次运行中都有 arm 对它既通过又失败，所以单个 trial 说明不了什么。`session-fixation` 在每个 arm 上都是 0% 到 19%：先读它的检查，再怪工具。 | +| 9 月 23 日那次运行为什么降到 27%？ | **每次运行的任务** 和 **运行** | `login-copy` 在那次运行一列中是斜线格，**运行** 在它的通过率下显示 `1 未测量`：那次这个任务的检查不改代码就通过了，所以它的[基线](benchmarks.md#基线)无效，它的 trial 计为计划但未通过。**任务 × arm** 把它标为 `未测量 1/7`。从 **运行** 打开这次运行，查看它的 trial。 | + +**最近一次运行** 显示最新一次运行中每个 arm 按结果统计的 trial，以及用量、偏差和它没有测量的任务。**打开这次运行** 会在工作流运行页面打开它的详情。 + +以上面板属于 **Arm** 标签页。另外两个标签页回答 arm 之间、以及团队修订之间变了什么。下面是 `onboarding-docs`（claude 和 codex 分别带 pull 到的团队资产和不带资产的 [matrix](benchmarks.md#用-matrix-交叉工具和团队资产)），以及前面的 `fix-auth`： + +| 问题 | 标签页和面板 | 显示了什么 | +|---|---|---| +| 团队资产对 onboarding 文档帮助有多大？ | **对比**，按 **团队资产** | claude 上 `已拉取的团队版本 对比 无团队资产 +41 pp · 95% +16 pp 至 +59 pp`，共 27 个配对 trial，每个通过 trial 的成本 `−60%`。 | +| 对 codex 帮助一样大吗？ | **每对 arm 的差异** | codex 提升 +15 pp，但区间跨过零：这些运行区分不了它的两个 arm。 | +| 资产修好了哪个任务？ | **每个任务** | `oncall-runbook` 带资产后从 1/9 到 7/9；`first-pr-guide` 几乎没变。 | +| auth-tokens skill 对 `fix-auth` 上的 claude 有帮助吗？ | **团队变更**，选 `claude · claude-opus-5-5` | 按提交顺序的三个团队版本上是 67%、62%、70%，没有一个与前一个可区分。点击 `67ee2ce · Add the auth-tokens skill and a security reviewer`：它改了三个文件，`session-fixation` 从 1/12 到 3/12。 | +| 工具更新影响数字了吗？ | **Arm**，**每次运行的通过率** | 一条虚线 `工具已更新 · claude 2.4.0` 标出新版本上的第一次运行。 | + +**对比** 列出套件当前版本中只在团队资产、工具、模型或工作流上不同的每两个 arm，并按运行、任务和重复次数配对它们的 trial。**团队变更** 跟踪同一工具、模型和工作流在它用 pull 到的资产或某个 ref 运行过的各团队版本上的表现，用本机的团队仓库克隆为每个版本命名，并列出它相对前一个版本改动的 `benchmarks/` 之外的文件。 + +数字的计算方式： + +- 通过率是通过的 trial 数除以计划的 trial 数，与运行自己的表一致。 +- 合并统计的数字、任务表和领先者只统计套件当前版本的运行：即最新一次运行的套件文件。**每次运行的通过率** 也会画出更早的运行，位于一条 `套件已改动` 虚线之后。 +- 一个 arm 由它的名称、工具、模型、effort 和团队资产确定。其中一项改变时，更早的运行是另一条线，标为 `较早的配置`。 +- **各 arm 的通过率** 还会按每次运行自己的表的算法，统计这些运行中该 arm 的偏差和未验证的 trial。偏差是指 prompt 运行时用的工具、模型或团队修订与它的 arm 不同的 trial。 +- 区间是 95% Wilson score 区间。两个 arm 的区间重叠时，这些运行区分不了它们。 +- **对比** 中的差异基于配对 trial，区间是配对比例的 95% 区间（Newcombe）。区间跨过零时标为 `无法区分`。 +- 用量是工具报告了用量的 trial 的中位数。成本按每个 trial 计。 +- 本机上的运行若也已发布，只显示一次，取本机的副本。 + +页面只显示运行记录发布的内容，从不显示 prompt、diff 或对话记录。 + ## 资源（Resources） 资源页列出团队仓库里的每条规则在当前工作区的情况：名称、命名空间、描述，以及 `pull` 是否会把它交付到这里。根规则人人都收到；命名空间规则跟随活跃的知识命名空间；同名根规则被活跃命名空间规则取代时会标注「已取代」；被标签订阅排除的规则也会标注。 diff --git a/docs/guide/zh-CN/distribution.md b/docs/guide/zh-CN/distribution.md index 7351a6a34..182a5e7ad 100644 --- a/docs/guide/zh-CN/distribution.md +++ b/docs/guide/zh-CN/distribution.md @@ -101,7 +101,7 @@ sharing: .teamai/mcp/mcp.yaml ``` -`teamai push` 发送所有分布式资源类型——skills、rules、agents、docs、env、hooks、mcp、models、claudemd、learnings、wiki、workflows、prompts、scorers——并按 `--project` 和 `--role` 做命名空间放置。 +`teamai push` 发送所有分布式资源类型——skills、rules、agents、docs、env、hooks、mcp、models、claudemd、learnings、wiki、workflows、prompts、scorers、benchmarks——并按 `--project` 和 `--role` 做命名空间放置。 ### Git 权限 @@ -155,6 +155,8 @@ scripts: `knowledge` 包含 rules 和共享指令（`claudemd/`）。项目还可以单独声明 learnings、env、hooks、mcp、models、docs、wiki。 +基准测试套件按 `resources.benchmarks` 分发。`benchmarks//` 发给所有人。某个角色或项目列出 `` 后，`benchmarks///` 只发给激活了 `` 的成员，`benchmarks//` 下放套件，不能直接放 `suite.yaml`。 + ```yaml # manifest/roles.yaml version: 1 diff --git a/docs/guide/zh-CN/scorers.md b/docs/guide/zh-CN/scorers.md index e070ff71b..9af29b87d 100644 --- a/docs/guide/zh-CN/scorers.md +++ b/docs/guide/zh-CN/scorers.md @@ -92,7 +92,7 @@ $ teamai score --lease s5d1e07c2a9b4a/a91c0d7e0 --json '{"driver":{"kind":"local {"data":{"label":{"scorer":"review-quality","label":"good","score":1,"passed":true,…},"published":true}} ``` -`samplingRate` 决定评判哪些成功的执行轮次，在每台机器上结果相同。`workflow next` 从不分发评审任务。`--lease ` 在本机用租约选定的工具、以[受限配置](#隔离)评判，并像任意 driver 的答案一样结算租约。评审失败时租约保持打开。 +`samplingRate` 决定评判哪些成功的执行轮次，在每台机器上结果相同。`workflow next` 从不分发评审任务。`--lease ` 在本机用租约选定的工具、以[受限配置](#隔离)评判，并像任意 driver 的答案一样结算租约。评审失败时租约保持打开。`scorers` 设置了 `sampling: all` 的步骤是例外：它的评分器评判每个成功的执行轮次，运行会等它们完成后才运行 `finally` 步骤并结束，在此期间 `workflow next` 返回带 `judging`（仍未结算的评判条目）的 `wait`。本地驱动自己评判它们，并立即结算失败的评审；其他驱动的轮次在运行它们的安装上用 `teamai score --next` 评判。[基准测试](benchmarks.md)就是这样评判它的 trial。 标签记录 `used.skills`，即该轮次或其子 agent 加载过的团队技能。判定方式与[会话](#为会话评分)相同，依据该轮次开始时检出目录中由 teamai 交付并未经修改的技能。`workflow:` 步骤的标签列出其子轮次记录的全部技能。若子轮次都未记录，标签没有 `used`。 @@ -319,7 +319,7 @@ Session judging: attended auto, set by scorers/scoring.yaml sessions; unattended | 评判对象 | 由谁决定 | |---|---| -| 工作流步骤的 pass | 步骤的 `scorers`，按每个评分器的 `samplingRate` 抽样。与 `sessions` 无关。 | +| 工作流步骤的 pass | 步骤的 `scorers`，按每个评分器的 `samplingRate` 抽样，或在 `sampling: all` 下评判每个 pass。与 `sessions` 无关。 | | 已捕获的会话：你的会话，或 `CI`、`TEAMAI_MODE` 任务中不是由 teamai 启动的会话 | 该会话运行模式对应的 `sessions`，由每个设置了 `sessions: true` 的评分器按其 `samplingRate` 抽样评判。 | | teamai 启动的会话，例如工作流步骤或评审 | 不评判。它不写捕获记录，它的步骤按 pass 评判。 | | teamai 读不到会话的工具（例如 Pi）的会话 | 不评判。对发布或评判策略不是 `off` 的每种模式，`teamai doctor` 会列出团队启用的每个这类工具。按默认值，两种模式都会列出。 | @@ -350,7 +350,7 @@ $ teamai score --drain 1. 在无人值守模式下，本数据目录记录为无人值守的每个已捕获会话都算已结束，包括调用它的会话。有人值守的会话仍按六小时规则。 2. 评判本机能评判的每个未完成的评审项：抽样的步骤 pass，以及策略允许的抽样会话，受 `daily_limit` 限制。本机有另一个评审过程（例如后台评审）正在运行时，它不评判任何评审项，并会说明这一点。等那个过程结束后再运行一次 drain。 3. 像 `score --push` 一样发布每个待发布的标签，并像 `teamai pull` 一样发布运行当时没能发布的工作流运行记录。 -4. 当前模式的 `publish` 为 `auto` 时，把该模式下已结束、尚未归档的已捕获会话，以及在这里驱动的运行中尚未归档的步骤会话，脱敏后归档到 `teamai-reports`。无法从这里归档的步骤会话（teamai 读不到其工具的会话，或本机读不到这个会话）只列出一次已跳过及原因，之后不再重试。该运行的其他步骤会话照常归档。 +4. 当前模式的 `publish` 为 `auto` 时，把该模式下已结束、尚未归档的已捕获会话，以及在这里驱动的运行中尚未归档的步骤会话，脱敏后归档到 `teamai-reports`。无法从这里归档的步骤会话（teamai 读不到其工具的会话，或本机读不到这个会话）只列出一次已跳过及原因，之后不再重试。该运行的其他步骤会话照常归档。[基准测试](benchmarks.md)运行的 trial 会话留在运行它的机器上。 5. 像 `teamai pull` 一样把用量上报到 `stats/.yaml`，让临时 runner 的计数到达团队。 6. 输出剩下的工作：评分器固定的模型本机没有评审工具能运行（`judge_model_unavailable`），或本机达不到其隔离要求的评审项；本机没有安装评审工具时，是所有未完成的评审项。它们等待其他 driver 的 `score --next`。 @@ -430,4 +430,4 @@ Judge tools here: claude 2.1.308 (Claude Code) is verified, codebuddy 2.3.0 is b | `/sessions/captures/` | 每个会话启动时的运行配置，由会话启动 hook 写入。会话文件消失一天后删除。 | | `teamai-reports` 上的 `scores//.jsonl` | 已发布的标签，每个评分器、评分标准、评分对象和评审类型只保留最新一条。推送成功前标签留在 `/scores/pending/`；关闭用量上报时一直留在那里。 | -标签从不进入运行或它的记录。工作流运行页面显示每一轮的标签，以及评审开销，与运行本身的 token 分开统计。本机的运行统计每一次评审尝试；成员发布的运行只统计产生了标签的评审。仪表盘的[评分器页面](dashboard.md#评分器scorers)按时间和配置显示每个评分器的通过率。 +标签从不进入工作流运行或它的记录。基准测试运行在结束时冻结每个 trial 的评判结果，其记录携带 trial 的结果，而不是标签。工作流运行页面显示每一轮的标签，以及评审开销，与运行本身的 token 分开统计。本机的运行统计每一次评审尝试；成员发布的运行只统计产生了标签的评审。仪表盘的[评分器页面](dashboard.md#评分器scorers)按时间和配置显示每个评分器的通过率。 diff --git a/docs/guide/zh-CN/tool-support.md b/docs/guide/zh-CN/tool-support.md index 1357d28fa..df7229830 100644 --- a/docs/guide/zh-CN/tool-support.md +++ b/docs/guide/zh-CN/tool-support.md @@ -74,33 +74,33 @@ teamai 为每个工具把每类资源写到哪里，以及工具读取哪里， 用户范围，取自 #1221 的矩阵；之后合入的修复标为 `t` 并附注。`✓` 可用。`◐` 默认可用，但某个条件会让它失效，而 doctor 仍报告已送达。`✗` 默认配置下不可用。`+` 工具支持，teamai 尚未实现。`—` 工具没有此功能。`?` 未知。标记后的字母是依据，含义同上。 -| 工具 | 环境变量 | 工作流（本地驱动） | Recall | 统计 | Token 与费用 | AI 后端 | 投票评审 | 会话评审 | HTTP agent hooks | -| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | -| Claude Code | ✓t | ✓v | ✓v | ✓v | ✓v | ✓v | ✓v | ✓t1 | ✓v | -| Codex CLI | ✓t | ✓t | ✓v | ✓t2 | ✓v | ✓v | ✓t3 | ✓t1 | ✓t4 | -| Codex CLI Internal | ? | ✓t | ? | ? | ✓t | ✓t | ? | ? | ? | -| Claude Code Internal | ? | ✓t | ? | ? | ✓t | ✓t | ? | ? | ? | -| TClaude | ? | ✓t | ? | ? | ✓t | ✓t | ? | ? | ? | -| TCodex | ? | ✓t | ? | ? | ✓t | ✓t | ? | ? | ? | -| Cursor | ? | +s5 | ✗v6 | ✓v7 | +d8 | ✓d | +s | ✓t1 | +t9 | -| Copilot | ? | +s | ✓v | ✓t2 | ✓t10 | ✓d | ? | ✓t1 | —t | -| JoyCode | ? | ? | ◐s | ✓t2 | ✓t | ? | ? | ✓t1 | ? | -| Qoder | ? | ? | ◐s | ✓s | ◐t11 | ✓d | ? | ✓s12 | ✓s | -| Qoder CN | ? | ? | ◐s | ✓s | ◐t11 | ? | ? | ✓s12 | ✓t13 | -| Kiro | ? | ? | ◐d | ? | ? | ✓d | ? | ✓d14 | ✓d15 | -| Trae | ? | ? | ? | ? | ? | ? | ? | ✓s16 | ? | -| Trae CN | ? | ? | ? | ? | ? | ? | ? | ✓s16 | ? | -| ZCode | ✓t | +s | ✓s | ✓s | ? | +s | ✓t3 | ✓d17 | +t18 | -| Oh My Pi | ✓s | +s | ✓v | ✓t2 | ✓t | ✓d | ? | ✓s19 | ✓s20 | -| Pi Coding Agent | ✓s | +v | ✓v | ✓t2 | ✓t | ✓s | ? | ◐s21 | ✓s22 | -| CodeBuddy | ? | ✓d | ✓s | ✓v | ✓t | ✓d | +v | ✓t1 | ✓s | -| OpenClaw | ◐d | +d | ✓s | +s | +s23 | ✓d | —t | ◐s24 | ✓s25 | -| Hermes | ◐s | +s | ✓v | +s | +s23 | ✓s | —t | ✓s26 | ✓s27 | -| DeepSeek Harness | ✗s | +s | ✓s | ✓t2 | +s28 | +s | ? | ◐s29 | —t | -| WorkBuddy | ? | ✓t | ✓t | ? | ✓t | ✓t | ? | ✓t30 | ? | -| OpenCode | ? | ✓t | ✓v | ✓t2 | ✓v | ✓v | ? | ✓t1 | ✓v31 | -| Muse Code | ✓t | +v | ✓t | ✓v | ✓t | ✓v | ✓t3 | ✓t1 | ✓t32 | -| Antigravity | ✓v33 | ✓v | ✓v | ✓v | ◐v34 | ✓v | ✗v35 | ✓t36 | ✓t37 | +| 工具 | 环境变量 | 工作流（本地驱动） | 无人值守提示 | Recall | 统计 | Token 与费用 | AI 后端 | 投票评审 | 会话评审 | HTTP agent hooks | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| Claude Code | ✓t | ✓v | ✓v | ✓v | ✓v | ✓v | ✓v | ✓v | ✓t1 | ✓v | +| Codex CLI | ✓t | ✓t | ✓d | ✓v | ✓t2 | ✓v | ✓v | ✓t3 | ✓t1 | ✓t4 | +| Codex CLI Internal | ? | ✓t | ✓t | ? | ? | ✓t | ✓t | ? | ? | ? | +| Claude Code Internal | ? | ✓t | ✓t | ? | ? | ✓t | ✓t | ? | ? | ? | +| TClaude | ? | ✓t | ✓t | ? | ? | ✓t | ✓t | ? | ? | ? | +| TCodex | ? | ✓t | ✓t | ? | ? | ✓t | ✓t | ? | ? | ? | +| Cursor | ? | +s5 | ? | ✗v6 | ✓v7 | +d8 | ✓d | +s | ✓t1 | +t9 | +| Copilot | ? | +s | ? | ✓v | ✓t2 | ✓t10 | ✓d | ? | ✓t1 | —t | +| JoyCode | ? | ? | ? | ◐s | ✓t2 | ✓t | ? | ? | ✓t1 | ? | +| Qoder | ? | ? | ? | ◐s | ✓s | ◐t11 | ✓d | ? | ✓s12 | ✓s | +| Qoder CN | ? | ? | ? | ◐s | ✓s | ◐t11 | ? | ? | ✓s12 | ✓t13 | +| Kiro | ? | ? | ? | ◐d | ? | ? | ✓d | ? | ✓d14 | ✓d15 | +| Trae | ? | ? | ? | ? | ? | ? | ? | ? | ✓s16 | ? | +| Trae CN | ? | ? | ? | ? | ? | ? | ? | ? | ✓s16 | ? | +| ZCode | ✓t | +s | ? | ✓s | ✓s | ? | +s | ✓t3 | ✓d17 | +t18 | +| Oh My Pi | ✓s | +s | ? | ✓v | ✓t2 | ✓t | ✓d | ? | ✓s19 | ✓s20 | +| Pi Coding Agent | ✓s | +v | ? | ✓v | ✓t2 | ✓t | ✓s | ? | ◐s21 | ✓s22 | +| CodeBuddy | ? | ✓d | ✓d | ✓s | ✓v | ✓t | ✓d | +v | ✓t1 | ✓s | +| OpenClaw | ◐d | +d | ? | ✓s | +s | +s23 | ✓d | —t | ◐s24 | ✓s25 | +| Hermes | ◐s | +s | ? | ✓v | +s | +s23 | ✓s | —t | ✓s26 | ✓s27 | +| DeepSeek Harness | ✗s | +s | ? | ✓s | ✓t2 | +s28 | +s | ? | ◐s29 | —t | +| WorkBuddy | ? | ✓t | ✓t | ✓t | ? | ✓t | ✓t | ? | ✓t30 | ? | +| OpenCode | ? | ✓t | ✓d | ✓v | ✓t2 | ✓v | ✓v | ? | ✓t1 | ✓v31 | +| Muse Code | ✓t | +v | ? | ✓t | ✓v | ✓t | ✓v | ✓t3 | ✓t1 | ✓t32 | +| Antigravity | ✓v33 | ✓v | ✓d | ✓v | ✓v | ◐v34 | ✓v | ✗v35 | ✓t36 | ✓t37 | 注： diff --git a/docs/guide/zh-CN/workflow-advanced.md b/docs/guide/zh-CN/workflow-advanced.md index 19bd17460..2d8df046a 100644 --- a/docs/guide/zh-CN/workflow-advanced.md +++ b/docs/guide/zh-CN/workflow-advanced.md @@ -439,6 +439,78 @@ steps: 带 `loop` 的工作流不能包含带 `loop` 的步骤。请把其中一个循环放进子工作流。 +## finally 步骤 + +```yaml +schemaVersion: v1alpha1 +name: nightly-e2e +description: Run the end-to-end suite, then always summarize it. +timeout: 1h +steps: + - id: e2e + command: npm run e2e +finally: + - id: summary + command: node scripts/e2e-summary.js + output: { text: string } +outputs: { summary: steps.summary.outputs.text } +``` + +`finally` 步骤在运行的步骤都结束后运行，无论它们如何结束：成功、失败、超时或被取消。它们是命令，只读，没有自己的工作区，只带 `id`、`needs`、`if`、`command`、`env`、`output` 和 `timeout`。它们运行期间，运行保持 `running`，最多 10 分钟，然后以其步骤达到的结果结束：失败的 `finally` 步骤从不改变它。`done.finally` 列出每个失败或被取消的 `finally` 步骤及其失败原因。在它们运行期间执行 `workflow cancel` 会结束它们和这次运行。带 `finally` 的工作流只能作为独立的运行运行，从不作为另一个工作流的步骤。 + +`done.outputs` 带工作流的顶层 `outputs`，在 `finally` 之后解析，所以它们可以读取 `finally` 步骤。所属步骤失败的输出会被省略。 + +```text +teamai workflow start nightly-e2e --driver local --timeout 30m +``` + +使用 `--driver local` 或 `--driver skill` 时，`--timeout` 把运行限制在它和文件的 `timeout` 中较短的那个。超过后运行以 `timeout` 失败，它的 `finally` 步骤仍会运行。 + +## 运行步骤打印的工作流 + +```yaml +schemaVersion: v1alpha1 +name: shard-tests +description: Plan one test job per changed package, then run them. +steps: + - id: plan + command: node scripts/plan-shards.js + output: { definition: workflow, notes: json } + - id: shards + needs: [plan] + workflow: { from: plan } +``` + +`plan` 把它的输出以 JSON 打印出来，其中包括一个工作流： + +```text +{"definition":{"definition":{"schemaVersion":"v1alpha1","name":"shards","description":"One job per package.","steps":[{"id":"api","command":"npm test -w api"},{"id":"web","command":"npm test -w web"}]}},"notes":{"packages":2}} +``` + +| 输出类型 | 包含 | +| --- | --- | +| `workflow` | 一个 bundle：`definition`，即要运行的工作流；以及可选的 `workflows`，它运行的具名工作流；`scorers`，生成的评分器固定版本；`teamRevision`，它使用其评分器的团队提交。 | +| `json` | 任意 JSON 值。表达式无法读取其内部；顶层 `outputs` 可以整个传递它。 | + +`workflow: { from: }` 运行它 `needs` 的某个步骤的 bundle。该步骤报告时，teamai 像 `workflow start` 检查文件一样检查这个 bundle，固定它，并在运行期间保留它。无法接受的 bundle，例如包含秘密值、`from` 步骤、`prompt_file` 步骤或 `finally` 步骤的 bundle，会让产生它的步骤以 `invalid_output` 失败并给出原因。产生 bundle 的步骤只运行一次，不在任何循环或 `for_each` 中。带 `from` 步骤的运行是[隔离的](#隔离运行)。 + +## 克隆工作区 + +```yaml + - id: try-fix + workspace: + clone: + commit: 3f2c9a7d1e0b4c6a8f5d2e1b0c9a8f7e6d5c4b3a + team: { treatment: none } + prompt: Fix the failing test. +``` + +带 `workspace: { clone: { commit, repo?, team } }` 的步骤在检出目录之外、`commit` 的全新克隆中运行：有自己的 git 数据，没有远程仓库，hook 关闭。`team` 指明为它安装的团队资产，安装在它自己的 home 下：`{ treatment: none }`，或 `{ revision, selection, treatment }`，其中 `treatment` 为 `pulled`、一个 ref 或 `{ without: [, ...] }`。`selection` 包含与你的配置同名的 `primaryRole`、`additionalRoles`、`projects`、`subscribedTags` 和 `excludedSkills`，以及 `enabledAgents`，即资产安装到的工具：默认为打开克隆的步骤的工具，否则为你启用的 agent。teamai 离线创建克隆，来源是运行的检出目录或提交缓存；`repo` 只指明缓存，从不是 URL。打开克隆的 workflow 步骤拥有它，供其下每个 `continue` 步骤使用。克隆中的任何内容都不会被推送或交付。[Benchmark](benchmarks.md) 会为你写出这些步骤。 + +### 隔离运行 + +某个步骤打开克隆或运行 `workflow: { from }` 时，运行就是隔离的。它只在启动它的机器上运行：`workflow publish` 和 `--claim` 以 `isolated_run` 拒绝它，`workflow retry` 拒绝它并给出启动新运行的命令。它的记录带 `isolated: true`，使用统计上报开启时仍会到达 `teamai-reports`，并且不计入任何工作流总量。 + ## 表达式参考 `if` 是一个表达式。它能读取哪些步骤、哪些值为真，见 [工作流](workflow.md)。 @@ -580,7 +652,7 @@ steps: ## 要求和限制 -步骤在 `requires` 中声明它需要什么。工作流文件中的 `defaults.requires` 作用于其中每个步骤。步骤自己的键会替换这些键，`env` 列表则会合并。工作流文件还可以设置 `setup`、`timeout` 和 `budget`，prompt 步骤可以设置 `max_turns`。 +步骤在 `requires` 中声明它需要什么。工作流文件中的 `defaults.requires` 作用于其中每个步骤。步骤自己的键会替换这些键，`env` 列表则会合并。工作流文件还可以设置 `setup`、`timeout` 和 `budget`，prompt 步骤可以设置 `max_turns`、`on_limit` 和 `permissions`。 ```yaml schemaVersion: v1alpha1 @@ -610,10 +682,13 @@ steps: | `requires.env` | 把这些[团队 env](sharing.md#env环境变量) 键交给步骤。其他团队 env 键，以及运行中其他步骤要求的键，都不会交给它。运行中只记录键名。某个键在本机没有值时，步骤在运行前以 `missing_env` 失败，消息写明怎样给它一个值：见[必需的键在本机没有值](#必需的键在本机没有值)。步骤 `env:` 中的项与团队 secret 或被要求的键同名时，以 `env_conflict` 失败。 | 步骤等待。 | | `requires.network` | `false` 要求步骤不能访问网络。 | 步骤照常运行，其 `unenforced` 列出 `network`。本地驱动无法禁止网络。 | | `requires.isolation` | `none`、`worktree` 或 `sandbox`。有工作区的步骤至少是 `worktree`。 | 步骤照常运行，其 `unenforced` 列出 `isolation`。 | +| `requires.strict` | `true` 让这一级的 `isolation` 和 `network` 成为下限：其下每个步骤至少得到它们，无论内层要求什么。严格的 `isolation: sandbox` 还会让步骤的每个进程都拿不到 `GITHUB_TOKEN`、`GH_TOKEN`、`GITLAB_TOKEN`、`GL_TOKEN`、`CI_JOB_TOKEN`、`SSH_AUTH_SOCK`、`GIT_ASKPASS` 和 `SSH_ASKPASS`，其下某个步骤的 `requires.env` 指定了其中之一时，pull 会拒绝它。使用 `--driver local` 时，本地驱动永远无法运行的启动会以 `requirement_unmet` 被拒绝。 | 步骤等待。严格的 sandbox 还需要 `can.env: true`。 | | `setup` | 驱动在它创建的每个工作区中、在步骤之前运行 `command`。setup 获得它的 `requires.env` 键；步骤只有自己也要求时才会获得。setup 以非零退出（`setup_failed`）、改动 git 会提交的文件（`setup_dirty`），或把自己某个键的值留在它写入的文件中（包括被忽略的文件，`setup_secret`）时，步骤不运行即失败，工作区被删除。步骤的 `timeout` 也约束 setup。 | 带工作区的步骤等待。 | | `timeout` | `500ms`、`30s`、`10m` 或 `2h`，写在步骤或工作流文件上。超时的步骤被停止，结果为 `timeout`。写在 workflow 步骤上时，会停止其中每个步骤。写在工作流文件上时，运行以 `failed` 结束，失败码为 `timeout`。重试会重新开始计时。 | 不是能力。每个驱动都会在租约上拿到 `expiresAt`。 | -| `budget` | `{ tokens: N }`，写在工作流文件或 workflow 步骤上。其下每个步骤的输入和输出 token 从它开始时计入：运行从开始时算起，workflow 步骤从每次开始时算起，例如循环的下一轮。重试会重新开始计算预算。超过 `N` 时，其中未结束的步骤停止，并以 `budget_exceeded` 失败。 | 预算内的非交互 prompt 步骤等待，它没有 token 的成功报告以 `budget_unmeasured` 失败。交互式 prompt 由人操作，照常运行；它的报告没有 token 时，其 `unenforced` 列出 `budget`。 | +| `budget` | `{ tokens: N }`，写在工作流文件、workflow 步骤或带 `on_limit: end_turn` 的 prompt 步骤上。其下每个步骤的输入和输出 token 从它开始时计入：运行从开始时算起，workflow 步骤从每次开始时算起，例如循环的下一轮。重试会重新开始计算预算。超过 `N` 时，其中未结束的步骤停止，并以 `budget_exceeded` 失败。 | 预算内的非交互 prompt 步骤等待，它没有 token 的成功报告以 `budget_unmeasured` 失败。交互式 prompt 由人操作，照常运行；它的报告没有 token 时，其 `unenforced` 列出 `budget`。 | | `max_turns` | 限制 prompt 步骤的智能体轮数。本地驱动只把它传给无头 Claude。 | 步骤照常运行，其 `unenforced` 列出 `max_turns`。 | +| `permissions` | prompt 步骤上的 `unattended`：它的工具无需询问就编辑文件和运行命令。只能在严格的 `isolation: sandbox` 之下使用，可以写在步骤本身、外层步骤、`defaults`，或运行其工作流的步骤上：没有它时，启动以 `unattended_unsandboxed` 被拒绝；[步骤打印的工作流](#运行步骤打印的工作流)没有它时，该步骤以 `invalid_output` 失败。见[无人值守的 prompt](#无人值守的-prompt)。 | 步骤等待。运行唯一的驱动没有 `can.unattended` 时，该步骤以 `tool_unavailable` 失败。 | +| `on_limit` | 在带 `timeout`、`max_turns` 或 `budget` 的 prompt 步骤上为 `end_turn`：从工具启动算起达到其中之一时，结束智能体的这一轮，而不是让步骤失败。步骤以设置了 `limit` 的状态成功并保留它的工作，所以它之后的步骤仍会运行。在带 `timeout` 或 `budget` 的 workflow 步骤上，这些是其中的 prompt 加起来可以花费的量，每次只运行一个 prompt：用完后，其未运行的步骤被跳过。`fail` 是默认值，在达到限制时让步骤失败。交还的步骤保持其第一次租约的截止时间。 | 没有 `can.end_turn` 时，步骤按 `fail` 运行，其 `unenforced` 列出 `on_limit`。 | `unenforced` 出现在租约、步骤和运行记录中。每个 `interactive: true` 步骤还需要支持 HITL 的驱动，见 [HITL](#hitl人在回路中)。 @@ -629,7 +704,7 @@ workflows: ``` - `allow.network: false` 会暂缓每个要求 `network: true` 的步骤。`allow.env` 会暂缓 `requires.env`（或其 setup 的 `requires.env`）中含有未列出键的步骤。`wait.held[].missing` 指出 `policy.network` 或 `policy.env`。`~/.teamai/config.yaml` 中错误的 `allow` 会让工作流命令停止运行，直到修正为止，因此拼写错误不会解除限制。 -- `sandbox: true` 表示本机会隔离它运行的内容，例如 CI 虚拟机或容器，因此本地驱动声明 `sandbox` 隔离。 +- `sandbox: true` 表示本机会隔离它运行的内容，例如 CI 虚拟机或容器，因此本地驱动声明 `sandbox` 隔离。此时设置 [`permissions: unattended`](#无人值守的-prompt) 的步骤运行其工具时不再询问，无论该步骤使用哪个工作区，包括你自己的 checkout。`teamai workflow start` 和 `resume` 接受 `--sandbox`，只为当前进程声明。 - `max_parallel` 还限制本机上所有驱动同时持有同一运行的租约数。 `allow.network` 筛选步骤显式声明的网络请求。v1 的 setup 仅声明环境变量键，不会从该设置继承网络限制。如需保证 setup 和步骤都无法联网，请使用能约束每个进程的驱动和运行环境。 @@ -641,15 +716,45 @@ workflows: | `driver.can` 键 | 驱动 | |---|---| | `hitl` | 让人回答 HITL 请求并在交互步骤中工作。 | -| `workspace` | 接受这些工作区模式的租约：`read-only`、`new`、`continue`。 | +| `workspace` | 接受这些工作区模式的租约：`read-only`、`new`、`continue`、`clone`。克隆步骤只通过 [`next --prepare`](#在自己的驱动里使用克隆工作区) 租出。 | | `env` | `true`：自己解析 `lease.envScope.provide`，移除 `lease.envScope.withhold`，并且不把值交给运行步骤的程序。`"commands"`：只对命令步骤和 setup 这样做，所以带 `requires.env` 的提示词步骤会等待。由智能体运行步骤的驱动不要声明它；如果命令由驱动自己启动，可以发送 `"commands"`。0.28.0 之前的 teamai 对 `"commands"` 返回 `invalid_json`。 | | `setup` | 在它创建的每个工作区以及 setup 未完成的工作区中，用 `lease.setup.envScope` 运行 `lease.setup.command`，并以上述失败码让步骤失败。工作流声明了 `setup` 时，每个带工作区的租约都携带 `setup`。 | | `tokens` | 报告每个成功的非交互 prompt 步骤的 token。 | | `max_turns` | 把 `lease.maxTurns` 传给它能限制的工具，对其他工具报告 `"unenforced":["max_turns"]`。 | | `network` | 能禁止步骤访问网络。 | | `isolation` | 以此级别运行每个步骤：`none`、`worktree` 或 `sandbox`。 | +| `tools` | 在这些工具上启动指定了 `tool` 的 prompt 步骤，例如 `["claude"]`。没有它，就不会拿到这样的步骤。 | +| `end_turn` | 在 `lease.limits` 处结束 prompt 的这一轮，并报告它停在的 `limit` 和 `durationMs`。 | +| `unattended` | 对租约带 `permissions: "unattended"` 的 prompt，以其工具自己的无需询问即可编辑和运行命令的模式启动它；工具没有这种模式时让步骤失败，而不是照常启动。 | + +省略的键表示驱动做不到。隔离是尽力而为，除非 `strict: true` 让它成为下限。`next` 会在 `ignored` 中列出其他键。[编程会话](workflow.md#在编程会话中运行)的 skill 驱动声明 `hitl`、所有 `workspace` 模式、`env: "commands"` 和 `setup`。它从不声明 `tokens`、`max_turns`、`network` 或 `unattended`，其 `isolation` 为 `none`：编程会话绝不绕过使用者自己的权限。 -省略的键表示驱动做不到。`next` 会在 `ignored` 中列出其他键。[编程会话](workflow.md#在编程会话中运行)的 skill 驱动声明 `hitl`、所有 `workspace` 模式、`env: "commands"` 和 `setup`。它从不声明 `tokens`、`max_turns` 或 `network`，其 `isolation` 为 `none`。 +### 无人值守的 prompt + +以无头方式启动的工具在编辑文件或运行命令前会先询问，没有人回答时，步骤什么也做不了。`permissions: unattended` 让 prompt 的工具直接执行，所以只在可以丢弃的机器上使用： + +```yaml +schemaVersion: v1alpha1 +name: unattended-fix +description: Fix a failing test on a disposable machine. +steps: + - id: work + prompt: Fix the failing test. + permissions: unattended + requires: { isolation: sandbox, strict: true } +``` + +本地驱动随后以工具自己的这种模式启动它： + +| 工具 | 启动参数 | +|---|---| +| Claude Code，以及使用其参数的工具 | `--permission-mode bypassPermissions` | +| CodeBuddy、WorkBuddy | `--permission-mode bypassPermissions` | +| Codex CLI，以及使用其参数的工具 | `--dangerously-bypass-approvals-and-sandbox` | +| OpenCode | `--auto`，权限规则拒绝的操作仍然被拒绝 | +| agy | `--dangerously-skip-permissions` | + +对于 teamai 不知道有这种模式的工具，步骤以 `tool_unavailable` 失败，工具绝不会在没有该模式的情况下启动。[工具支持](tool-support.md)的 `Unattended prompts` 列说明哪些工具有这种模式，以及依据。租约带 `permissions: "unattended"`，所以你自己的驱动能看到它；只有声明了 `can.unattended` 的驱动才会拿到该步骤。 ### 必需的键在本机没有值 @@ -929,6 +1034,22 @@ teamai workflow next --json '{"claim":true,"driver":{"kind":"script","mach TeamAI 为准备 attempt 记录的内容只保存在本机：路径和文件指纹，不含提示词、env 值或图。 +## 在自己的驱动里使用克隆工作区 + +```text +teamai workflow next --prepare --json '{"driver":{"kind":"script","machine":"runner","workspace":"/work","can":{"workspace":["read-only","new","continue","clone"],"isolation":"sandbox","env":true,"setup":true,"tools":["claude"],"end_turn":true}},"running":[]}' +``` + +带上 `--prepare` 时，teamai 在本机为[克隆步骤](#克隆工作区)创建克隆、安装其团队资产并运行 setup，然后返回带 `cwd`（即该克隆）的租约。一次调用可能要等 setup 运行完。驱动未在 `workspace` 中列出 `clone`，或没有用 `--prepare` 请求时，克隆步骤会被暂缓，`wait.held[].missing` 列出 `workspace.clone` 或 `prepare`。 + +对每个租约，驱动： + +1. 运行每个进程时带上 `lease.env` 和 `TEAMAI_WORKFLOW_ATTEMPT=/`，这样从中发出的 `teamai` 调用不会改变任何运行，并去掉 `lease.envScope.withhold` 里的名字。它在 `cwd` 中运行命令租约（例如 `teamai benchmark check`），没有 `cwd` 的租约则在它自己的工作区中运行，并把命令打印的 JSON 作为 `outputs` 报告。 +2. 在 `cwd` 中以 `lease.prompt` 启动 prompt 租约的 `lease.tool`，按原样使用 `lease.model`，并从工具启动起让它遵守 `lease.limits`。 +3. 报告 prompt 时带上 `tool`、工具报告的 `model`、`session` 和 `tokens`、只围绕工具启动测得的 `durationMs`，以及仅在工具给出时才带的 `cost`（`{"amount":0.12,"unit":"USD"}`）。某个限制停止了工具时，报告带 `outcome: "success"` 和 `limit`：`timeout`、`budget` 或 `max_turns`；驱动无法让工具遵守的 `budget` 或 `max_turns` 列在 `unenforced` 中。`limit` 是租约没有设置的限制时，报告返回 `invalid_report`；带有 `changes` 或 `read`（由 teamai 从克隆中填写）的报告也一样。 + +`src/__tests__/e2e/benchmark-driver-contract.test.ts` 用这份约定检查运行 [benchmark](benchmarks.md#在自己的驱动里运行) 的驱动。 + ## 在自己的驱动里评审 ```text diff --git a/docs/guide/zh-CN/workflow.md b/docs/guide/zh-CN/workflow.md index 816e8bba7..486c2f951 100644 --- a/docs/guide/zh-CN/workflow.md +++ b/docs/guide/zh-CN/workflow.md @@ -24,7 +24,7 @@ steps: `schemaVersion` 必须是 `v1alpha1`。`name` 必须和文件名一致。`description` 必填。每个步骤有唯一的 `id`，并且只能使用 `command`、`prompt`、`prompt_file`、`workflow` 中的一种动作。带有 `end`、`approve` 或 `choose` 的步骤可以省略动作。提示词步骤可选填 `agent`。`output` 沿用已有简写：`string`、`number`、`boolean`、字面量联合、对象，或只含一种元素类型的数组。TeamAI 会用它校验成功命令的 stdout 或提示词工具返回的 `output`。 -本地命令步骤以 UTF-8 读取 stdout 和 stderr。JSON 字符串中的非 ASCII 文本会原样保存到输出中，后续步骤读取时也保持原文。 +本地命令步骤以 UTF-8 读取 stdout 和 stderr。JSON 字符串中的非 ASCII 文本会原样保存到输出中，后续步骤读取时也保持原文。运行 `teamai` 的命令运行的是驱动该步骤的那个 teamai。 提示词步骤可以设置 `model`：`models/aliases.yaml` 中的别名或模型名，不能含空格或模板。文件顶层的 `defaults: { model: fast }` 为其中的提示词步骤设置模型。提示词步骤使用以下第一个已设置的模型： @@ -53,6 +53,8 @@ steps: scorers: { names: [review-quality], batch: false } ``` +`scorers: { names: [review-quality], sampling: all }` 评判每一个成功的 pass，不管每个评分器的 `samplingRate`，运行会等这些评判完成后才运行它的 [`finally` 步骤](workflow-advanced.md#finally-步骤)并结束。见[一次运行和它的评审](scorers.md#一次运行和它的评审)。 + 第一个步骤之后的每个步骤都要用 `needs` 列出它等待的步骤。`needs: []` 表示另起一个没有依赖的分支。每次 `next` 租出文件中第一个依赖都已完成的步骤。`--driver local` 一次运行一个步骤，除非 `workflows.max_parallel` 大于 1。某个依赖失败，或所有依赖都被跳过时，该步骤会被跳过。被跳过的步骤不会运行。一个步骤失败不会停止其他分支。没有可运行的步骤时，只要有步骤失败，运行结果就是 `failed`，否则是 `succeeded`。缺少 `needs`、`needs` 中的 id 不是本工作流的步骤，或依赖成环，都会让该文件无效。缺少 `needs` 时，即使之前有有效版本，该工作流也没有可运行版本。 步骤可以带 `if` 表达式。表达式为假时跳过该步骤，这不会让运行失败。表达式可以读取 `steps..outcome`（`success`、`failed` 或 `skipped`）和 `steps..outputs.`，且只能读取该步骤直接或间接依赖的步骤。可用 `==`、`!=`、`<`、`<=`、`>`、`>=`、`&&`、`||`、`!`、括号，以及字面量 `'text'`、数字、`true`、`false`、`null` 和 `[]`。路径可以用下标读取数组元素，例如 `steps.review.outputs.files[0]`。函数 `length`、`any`、`all` 和 `filter` 读取数组。不存在的值读作 `null`。`false`、`null`、`0` 和 `''` 为假，其他值都为真。`==` 是严格比较，并比较整个值：`'1' == 1` 和 `'Build' == 'build'` 都为假。 @@ -166,6 +168,21 @@ teamai workflow start build --driver local teamai workflow start build --driver local --tools default=agy # 所有步骤都在 Antigravity CLI 上运行 ``` +提示词步骤可以指定自己的工具，以及一个确切的模型： + +```yaml +schemaVersion: v1alpha1 +name: second-opinion +description: Review the change on another tool, on an exact model. +steps: + - id: review + prompt: Review the change. + tool: codex + model: { name: gpt-5.5, effort: high } +``` + +`tool:` 优先于 `--tools`、`workflows.tools`、`TEAMAI_AI_TOOL` 和 `ai.tool`。只有在 `driver.can.tools` 中列出该工具的驱动才会租到这个步骤。使用 `--driver local` 时，工具无法在本机启动的步骤以 `tool_unavailable` 失败。`model: { name, effort }` 运行的就是这个模型和 effort，从不经过 `models/aliases.yaml` 解析；字符串形式的 `model` 仍是别名或模型名。 + `--driver local` 在当前检出目录里运行没有 `workspace` 的步骤，除非外层步骤打开了工作区。这里的命令是只读的：退出码为 0，且改了仓库文件、文件权限、符号链接目标、已暂存重命名的新路径、已脏子模块里的文件、暂存区或 HEAD 时，步骤失败，改动留在原地。已有未跟踪目录里的文件变化也算；该目录保持不变时，步骤不会因此失败。`workspace: new` 的步骤在自己的分支上改文件。见[工作区](#工作区)。共用当前检出目录的步骤一次只运行一个。不同工作树里的步骤可以一起运行，上限是 [`workflows.max_parallel`](workflow-advanced.md#并行运行本地命令)。每次 attempt 的 `result.json` 位于当前 scope 的 TeamAI 数据目录中，不在产品检出目录内。驱动调用的 `next` 和 `report` 与下面相同： ```text @@ -440,7 +457,7 @@ $ teamai workflow resume --next --tools default=claude # crontab，未设置 开启[使用统计上报](reporting.md)时，每次运行都会向团队的 `teamai-reports` 分支发布一条保护隐私的记录：开始时一次，结束时一次。关闭后不发布任何记录，返回中也不带 `record`。记录存放在 `workflows/runs/.json`，以 run id 为键，重试的发布会更新同一条记录而不是产生副本。 -记录只包含：run id、工作流名、定义里有别名时的 `alias`、`sourceHash`、可选的 `teamRevision`、`origin`（团队的定义，或[默认工作流](#默认工作流)及发布它的 teamai 版本）、阶段（`start` 或 `end`）、驱动的 kind、运行启动时的模式（`runMode`）、状态、运行到达的 end（到达时才有）、未到达 end 就失败的运行的失败代码（仅限 teamai 设定的代码，如循环用完次数时的 `loop_exhausted`，或本地驱动找不到工具时的 `tool_unavailable`，不含失败消息）、开始和结束时间、时长、逐步条目（step 路径如 `check.classify`、agent、tool、model、outcome、tokens，驱动在 `next` 上发送了时该 pass 的 `runMode`，以及该 pass 的产出（`outcomes`：它分支上的 commit 哈希、为它交付的拉取请求，以及 `teamai contribute` 在它的 attempt 中记录的经验 ID）——混合 agent 的运行会分别记录每步自己的字段；`tool` 仅在是本地驱动的工具之一时保留，`model` 是驱动报告的名称，仅在形如模型名时保留）、步骤汇总计数（succeeded、failed、skipped、cancelled）、累计已结算 attempt 数、运行交付的 pull request（`prs`），以及发布者用户名。`sourceHash` 标识实际运行的工作流和提示文件的精确字节。只有一个远端可达的团队提交同时保存父工作流和所有子工作流的这些精确字节时，才写入 `teamRevision`；本地或未推送的修改以及混合缓存版本都会省略它。每条条目都带 `pass` 编号，循环运行了多次的步骤每个 pass 各有一条条目。`for_each` 的项按该项自己运行的次数计 `pass`，所以前面某个 pass 的项较少时，项的 `pass` 可能比所属步骤的小。被取消步骤的条目 outcome 为 `cancelled`，被跳过的步骤没有 outcome。汇总计数按运行结束时的状态，每个步骤只计一次。从未开始的子工作流里的步骤不计入。提示词、命令文本、输入值、`env` 值、输出、diff、秘密值、本地工作区路径、会话 id 和主机名没有可落地的字段——记录由白名单构建，而不是从运行状态里过滤。 +记录只包含：run id、工作流名、定义里有别名时的 `alias`、`sourceHash`、可选的 `teamRevision`、`origin`（团队的定义，或[默认工作流](#默认工作流)及发布它的 teamai 版本）、阶段（`start` 或 `end`）、驱动的 kind、运行启动时的模式（`runMode`）、状态、运行到达的 end（到达时才有）、未到达 end 就失败的运行的失败代码（仅限 teamai 设定的代码，如循环用完次数时的 `loop_exhausted`，或本地驱动找不到工具时的 `tool_unavailable`，不含失败消息）、开始和结束时间、时长、逐步条目（step 路径如 `check.classify`、agent、tool、model、outcome、tokens，驱动在 `next` 上发送了时该 pass 的 `runMode`，以及该 pass 的产出（`outcomes`：它分支上的 commit 哈希、为它交付的拉取请求，以及 `teamai contribute` 在它的 attempt 中记录的经验 ID）——混合 agent 的运行会分别记录每步自己的字段；`tool` 仅在是本地驱动的工具之一时保留，`model` 是驱动报告的名称，仅在形如模型名时保留）、步骤汇总计数（succeeded、failed、skipped、cancelled）、累计已结算 attempt 数、运行交付的 pull request（`prs`）、[隔离运行](workflow-advanced.md#隔离运行)的 `isolated: true`，以及发布者用户名。`sourceHash` 标识实际运行的工作流和提示文件的精确字节。只有一个远端可达的团队提交同时保存父工作流和所有子工作流的这些精确字节时，才写入 `teamRevision`；本地或未推送的修改以及混合缓存版本都会省略它。每条条目都带 `pass` 编号，循环运行了多次的步骤每个 pass 各有一条条目。`for_each` 的项按该项自己运行的次数计 `pass`，所以前面某个 pass 的项较少时，项的 `pass` 可能比所属步骤的小。被取消步骤的条目 outcome 为 `cancelled`，被跳过的步骤没有 outcome。汇总计数按运行结束时的状态，每个步骤只计一次。从未开始的子工作流里的步骤不计入。提示词、命令文本、输入值、`env` 值、输出、diff、秘密值、本地工作区路径、会话 id 和主机名没有可落地的字段——记录由白名单构建，而不是从运行状态里过滤。 条目的 `agent` 是该 pass 实际运行时观察到的组合：步骤的 `agent:` 定义及工具如何拿到它（工具自己读取 agent 文件时为 `main`，指令拼接到提示词前时为 `prompt`）、工具及其版本、启动时请求的模型和 effort、工具报告的模型、检出最近一次 pull 的团队提交以及之后是否改过已下发的文件、项目提交、TeamAI 版本和驱动的 kind。没有观察到的值不写。`fingerprint` 对定义、工具、模型、effort 和团队提交做哈希，fingerprint 相同的两个 pass 记录下来的这些部分相同。模型或团队提交未知，或改过已下发的文件时，不写 `fingerprint`。没有任何观察的 pass，以及旧版 TeamAI 写的记录，只带定义的名称。步骤运行期间，如果该检出最近一次 pull 已完整下发到步骤启动的工具，该工具以及它在同一检出中启动的会话都会跳过会话开始时的 pull，步骤因此保持它记录的团队提交。 @@ -450,6 +467,14 @@ $ teamai workflow resume --next --tools default=claude # crontab，未设置 jq -r 'select((.prs // []) | index("")) | .run' workflows/runs/*.json ``` +[benchmark 运行](benchmarks.md)还会写一条 benchmark 记录 `benchmarks/runs/.json`，套件改动后仍能重建对比。它的 `report` 步骤把记录写到本机上该运行的目录中，并在使用统计上报开启时像运行自己的记录一样发布它：先排队，发布失败时在下一次 `teamai pull` 时重试。它包含套件的名称和 `sourceHash`、重复次数、套件设置了 [trial 限制](benchmarks.md#trial-限制)时的 `limits`、基线团队修订、评判其 trial 的每个评分器及其评分标准的 `sourceHash`（`scorers`：套件的评分器，以及每个按成功标准评判的任务的 `teamai:correctness:`）、每个任务的 id、起始提交、`sourceHash` 和 `baseline`、每个 arm 的名称、工具、团队（`pulled`、`none`、一个 ref 或 `without` 路径，以及它的修订）、运行工作流的 arm 的 `workflow`（名称和 `sourceHash`），以及它的模型别名在该修订上为它的工具解析出的模型和 effort 及其 `source`：`team`，或在 `benchmark run --member-aliases` 下为 `member`。`skillChanges` 列出每个 arm 新增、去掉或改变的团队 skill。每个计划的 trial 有一条条目：任务、arm、重复序号、顺序、trial 的步骤、它的 `result`、trial 限制结束了其 prompt 时的 `limit`、其驱动没有执行的 trial 限制 `unenforced`、prompt 的 `durationMs`（只计工具启动前后）、`tokens` 和 `cost`（`{ amount, unit }`），只按工具自己声明的写，工作流 arm 中有 prompt 未声明时各自带 `complete: false`，`used`：trial 加载的每个团队 skill（`loaded`，或者没加载它的 prompt 未被观察到时为 `unknown`），以及 `scoringCost`（`{ amount, unit, complete }`）：本安装上评判该 trial 的评判者声明的成本，含失败的评判；有评判未声明成本时 `complete: false`。它还包含 `schemaVersion`、`run`、运行在其 `finally` 步骤之前的 `outcome`、`teamaiVersion`、`startedAt`、`finishedAt`、`username`，以及 `workflow`：运行的 `benchmark` 工作流（`name`、`origin` 和 `sourceHash`）。`baseline` 是在任何 trial 之前，任务的 check 在其起始提交的全新克隆上的结果：在那里失败（任务本应如此）为 `valid`，通过为 `invalid`，没能运行为 `could_not_run`，没有驱动运行它为 `not_run`，按成功标准评判的任务为 `not_needed`。只有 `valid` 任务的 trial 会运行。任务的 check 跑出判定时，结果为 `passed` 或 `failed`；trial 或它的 check 没能完成时为 `error`；按成功标准评判的任务按评判者给出的正确性标签为 `passed` 或 `failed`，评判失败或运行先结束时为 `unscored`；从未开始的 trial 为 `not_run`。套件的评分器在 `report` 运行之前评判每一个 trial，不管其 `samplingRate`。trial 结果从不成为运行的状态或步骤的 outcome，benchmark 运行也不计入任何工作流总量。 + +benchmark 运行在启动时冻结计划：运行期间修改套件并 pull，不会改变它运行的任何内容。驱动租用之前被取消的 trial 为 `not_run`，租用之后被取消的为 `error`。要只从 benchmark 记录算出每个 arm 的通过率（与表格打印的一致），在团队仓库 `teamai-reports` 分支的检出中运行： + +```text +jq -r '. as $b | $b.arms[] | .name as $arm | [$b.trials[] | select(.arm == $arm)] | (map(select(.result == "passed")) | length) as $passed | "\($arm) \($passed / length * 100 | round)% (\($passed)/\(length))"' benchmarks/runs/.json +``` + 结束运行的操作在锁内排队保存结束记录，在释放锁后发布。发布失败时——没有远端、reports 锁被占用、push 被拒——记录会进入当前 scope 数据目录的本地队列，在下一次 `teamai pull` 时重试。此时 `start`、`next` 的返回会带 `record: pending` 而不是 `record: published`。运行本身不受影响：一台机器在运行开始后崩溃，该运行会以"已开始未结束"的状态保持可见。 Run id 是 `r` 加 16 位十六进制。此方案之前创建的运行用 8 位十六进制，仍然可以解析。 @@ -532,7 +557,7 @@ TeamAI 从宿主设置的会话变量得知会话所用的工具。找不到时 `teamai skill get workflow` 打印 agent 遵循的指令。`teamai workflow --help` 列出所有选项。 -skill 驱动声明 `env: "commands"`（见[要求和限制](workflow-advanced.md#要求和限制)）。命令步骤和 setup 像在本地驱动中一样获得各自的 `requires.env` 键。带 `requires.env` 的提示词步骤会等待，因为 TeamAI 只在自己启动的进程中限定团队 env。该驱动从不声明 `tokens`、`max_turns` 或 `network`，因为没有宿主会把子 agent 的 token 或轮次报告给 TeamAI。处于 `budget` 下的无头提示词步骤会等待，所以运行级的 `budget` 会让该运行的所有无头提示词步骤等待。`max_turns` 和 `requires.network: false` 会列入 `unenforced`。worktree 不是 sandbox：`requires.isolation: sandbox` 也会列入 `unenforced`。 +skill 驱动声明 `env: "commands"`（见[要求和限制](workflow-advanced.md#要求和限制)）。命令步骤和 setup 像在本地驱动中一样获得各自的 `requires.env` 键。带 `requires.env` 的提示词步骤会等待，因为 TeamAI 只在自己启动的进程中限定团队 env。该驱动从不声明 `tokens`、`max_turns` 或 `network`，因为没有宿主会把子 agent 的 token 或轮次报告给 TeamAI；也从不声明 `unattended`，所以带 `permissions: unattended` 的提示词步骤会等待。处于 `budget` 下的无头提示词步骤会等待，所以运行级的 `budget` 会让该运行的所有无头提示词步骤等待。`max_turns` 和 `requires.network: false` 会列入 `unenforced`。worktree 不是 sandbox：`requires.isolation: sandbox` 也会列入 `unenforced`。 当一个因 env 或 budget 而等待的步骤，在同一次运行中较早步骤打开的工作区里工作时，describe 会把整个运行交给本地驱动，由本人在自己的终端启动。否则会话先运行它能运行的部分。只剩等待的步骤时，它把运行交给本人，由本人在自己的终端用 `teamai workflow resume ` 继续。本地驱动用本地配置中 `workflows.tools` 指定的、或 `--tools` 选择的无头工具运行提示词步骤。`TEAMAI_AI_TOOL`、`workflows.tools.default` 和两处 `ai.tool` 都没有指定工具时，会话给出的本地启动命令和 `resume` 命令会带上 `--tools default=<会话的工具>`。 @@ -568,12 +593,13 @@ skill 驱动声明 `env: "commands"`（见[要求和限制](workflow-advanced.md ## 默认工作流 -teamai 自带两个工作流，团队不用自己编写就能使用。它们都以 `teamai-` 开头，方便和团队的工作流区分： +teamai 自带一些工作流，团队不用自己编写就能使用： | 工作流 | 作用 | 它的 skill 何时出现 | |---|---|---| | `teamai-share-learnings` | 把会话学到的东西分享为团队经验（一条或多条），然后运行 `teamai-share-session` | [recall](knowledge.md) 开启且来源可写时，因为它的第一步运行 `teamai skill get share`；团队的同名副本也一样 | | `teamai-share-session` | 发布会话的转录（已脱敏），并用团队的评分器给它评分 | 始终 | +| `teamai-benchmark` | 通过 `teamai benchmark run` 运行 [benchmark suite](benchmarks.md#benchmark-工作流) | 始终 | 团队写一个同名文件就会取代默认工作流： @@ -600,6 +626,8 @@ $ teamai list workflows ⚠ [workflows] workflows/teamai-share-session.yaml has no runnable version, so teamai's default teamai-share-session stays in effect ``` +例外是你接收的两个命名空间都定义了的名字：它没有可运行版本，启动它会以 `workflow_namespace_conflict` 拒绝，并指出两个文件。 + `teamai-share-learnings` 用它的五个输入运行 `teamai-share-session`，所以团队的 `teamai-share-session`（或别名为 `teamai-share-session` 的团队工作流）要声明同样的五个输入，类型也相同，用不到的也要声明： ```yaml diff --git a/docs/usage-guide.md b/docs/usage-guide.md index c4300c6c3..c25db7930 100644 --- a/docs/usage-guide.md +++ b/docs/usage-guide.md @@ -14,6 +14,7 @@ Start with [Getting Started](guide/getting-started.md): set up a team, join one, - [Workflows](guide/workflow.md) — command steps in `workflows/*.yaml`: pull, start, next, and report - [Advanced workflows](guide/workflow-advanced.md) — parallel steps, loops, `for_each`, ends that cancel open steps, and the expression reference - [Scorers](guide/scorers.md) — rubrics in `scorers/*.md` that judge workflow steps +- [Benchmarks](guide/benchmarks.md) — suites in `benchmarks/` that compare tools and models on your own tasks, and their CI jobs - [Team Context](guide/knowledge.md) — learnings, the codebase graph, teamwiki, and CI - [Team Culture](guide/team-culture.md) — Company and team culture injected into every agent - [Dashboard](guide/dashboard.md) — Local dashboard and session save diff --git a/docs/usage-guide.zh-CN.md b/docs/usage-guide.zh-CN.md index e812a0787..1108dda4c 100644 --- a/docs/usage-guide.zh-CN.md +++ b/docs/usage-guide.zh-CN.md @@ -14,6 +14,7 @@ - [工作流](guide/zh-CN/workflow.md) — `workflows/*.yaml` 里的 command 步骤：pull、start、next、report - [高级工作流](guide/zh-CN/workflow-advanced.md) — 并行步骤、循环、`for_each`、到达结局时取消未完成的步骤、表达式参考 - [评分器](guide/zh-CN/scorers.md) — `scorers/*.md` 里评判工作流步骤的评分标准 +- [Benchmark](guide/zh-CN/benchmarks.md) — `benchmarks/` 里用你自己的任务比较工具和模型的套件，以及它们的 CI 任务 - [团队上下文](guide/zh-CN/knowledge.md) — learnings、代码知识图谱、teamwiki、CI - [团队文化](guide/zh-CN/team-culture.md) — 注入到每个 Agent 的公司与团队文化 - [Dashboard](guide/zh-CN/dashboard.md) — 本地看板与会话存档 diff --git a/oxlint-plugin-teamai/failure-codes.js b/oxlint-plugin-teamai/failure-codes.js index 9d3bcf92e..5c9dd78b8 100644 --- a/oxlint-plugin-teamai/failure-codes.js +++ b/oxlint-plugin-teamai/failure-codes.js @@ -26,7 +26,7 @@ function literalStrings(file, name) { }); } -const diagnostics = ['unknown_key', 'missing_needs', 'false_if', 'bad_template', 'dotted_name', 'undeclared_output', 'undeclared_end', 'bad_loop', 'bad_workspace', 'workflow_namespace_conflict', 'prompt_namespace_conflict', 'bad_delivery', 'bad_invoke', 'bad_alias', 'alias_conflict']; +const diagnostics = ['unknown_key', 'missing_needs', 'false_if', 'bad_template', 'dotted_name', 'undeclared_output', 'undeclared_end', 'bad_loop', 'bad_workspace', 'workflow_namespace_conflict', 'prompt_namespace_conflict', 'benchmark_namespace_conflict', 'bad_delivery', 'bad_invoke', 'bad_alias', 'alias_conflict']; const answers = ['invalid_json', 'driver_running', 'invalid_run', 'repeated_response', 'agent_in_path', 'stale_response', 'retried', 'end_reached', 'repeated_report', 'wrong_attempt', 'invalid_report', 'lease_cancelled', 'interrupted', 'run_unreadable', 'driver_stuck', 'ambiguous_run', 'busy', 'workflow_recursion', 'workflow_not_found', // Refusals of a run handover, answered to the command or driver that asked for it. 'not_owner', 'unsupported_repo', 'reports_unreachable', 'reports_failed', 'run_not_found', 'run_not_published', 'run_mismatch', @@ -42,6 +42,10 @@ const answers = ['invalid_json', 'driver_running', 'invalid_run', 'repeated_resp 'invalid_arguments', 'invalid_pass', 'not_initialized', 'unsupported_driver', 'invalid_tools', // Refusals of teamai workflow start for a run whose pull requests would have no branch to target. 'no_base_branch', + // Refusals of a local start with a step the local driver could never take for a strict requirement, before any run exists. + 'requirement_unmet', + // Refusals of a start with an unattended prompt step that no strict sandbox covers, before any run exists. + 'unattended_unsandboxed', // Refusals of a start from an agent session that the workflow's invoke does not let start it. 'invoke_denied', // Refusals of an agent's start of an invoke: ask workflow in an unattended run, where no person can agree. @@ -56,6 +60,15 @@ const answers = ['invalid_json', 'driver_running', 'invalid_run', 'repeated_resp 'skill_call_failed', // Why the skill driver stopped judging a run, told to the agent in a judging-off event. 'judge_runner_gone', 'no_child_agent', + // Refusals of teamai benchmark run and plan, before any run exists. + 'team_revision_unpullable', 'benchmark_not_found', 'commit_unreachable', 'check_files_unwritable', + 'arm_model_missing', 'arm_unpublishable', 'arm_team_unresolved', 'arm_team_unreadable', 'arm_team_unclean', 'arm_workflow_unrunnable', + // Refusals of teamai benchmark try, which tries one task file and creates no run. + 'benchmark_invalid', 'task_try_failed', + // Refusals of teamai benchmark plan where the team a trial runs cannot be pinned. + 'team_ref_blocked', + // Refusals of teamai workflow publish and a claim of an isolated run, which stays on the machine that started it. + 'isolated_run', // Refusals of teamai score --run, answered to the member who asked for a pass to be judged. 'scoring_unavailable', 'pass_not_found', 'scorer_not_found', 'entry_leased', // Refusals of teamai score --session, answered to the member who asked for a session to be judged. diff --git a/oxlint-plugin-teamai/rules/workflow-boundaries.js b/oxlint-plugin-teamai/rules/workflow-boundaries.js index 4f759d541..e122cdf1b 100644 --- a/oxlint-plugin-teamai/rules/workflow-boundaries.js +++ b/oxlint-plugin-teamai/rules/workflow-boundaries.js @@ -8,6 +8,7 @@ const SKILL = 'src/workflow/drivers/skill/'; const LOCAL = 'src/workflow/drivers/local/'; const PLACE = 'src/workflow/place/'; const DISCOVERY = 'src/workflow/discovery/'; +const BENCHMARK = 'src/benchmark/'; const FLAT = 'src/workflow/'; /** Rule 1: what a module under `drivers/skill/` may import, besides its siblings, `zod` and `node:*`. */ @@ -25,7 +26,7 @@ const SKILL_IMPORTS = { 'src/workflow/store.ts': { names: ['takeRunLock', 'LockTaken', 'RunLock', 'attemptDir', 'attemptResultPath', 'skillStatePath', 'runsDir', 'RunIdSchema', 'RunId', 'AttemptIdSchema', 'AttemptId', 'withRunnerLock', 'runnerAlive'] }, 'src/workflow/step-model.ts': {}, 'src/env-resolution.ts': {}, - 'src/env-exec.ts': { names: ['stepEnvironment'] }, + 'src/env-exec.ts': { names: ['stepEnvironment', 'withRunningCli'] }, 'src/utils/redact.ts': {}, 'src/utils/session-id.ts': {}, // Equal values as equal text, so a kept end is compared with the engine's latest one. @@ -56,12 +57,34 @@ const SKILL_IMPORTS = { /** @type {Array<{ folder: string; entries: Record; testEntries: string[] }>} */ const ENTRIES = [ { folder: SKILL, entries: { 'skill-driver.ts': 'any', 'literals.ts': ['src/workflow/discovery/render.ts'] }, testEntries: ['skill-driver.ts', 'literals.ts'] }, - { folder: PLACE, entries: { 'place.ts': ['src/workflow/protocol.ts', 'src/workflow/drivers/local/local-driver.ts'] }, testEntries: ['place.ts'] }, + { + folder: PLACE, + // engine.ts stages, before it saves a run, the evidence a step that settled there left in its clone. + entries: { 'place.ts': ['src/workflow/engine.ts', 'src/workflow/protocol.ts', 'src/workflow/drivers/local/local-driver.ts', `${BENCHMARK}trial.ts`, `${BENCHMARK}start.ts`], 'setup.ts': [`${BENCHMARK}trial.ts`] }, + testEntries: ['place.ts', 'team-without.ts'], + }, // workflow-published-runs.ts judges an offered run by the tool the local driver would launch for its prompt steps. { folder: LOCAL, entries: { 'local-driver.ts': ['src/workflow-commands.ts', 'src/workflow-published-runs.ts'], 'local-tool-adapter.ts': [] }, testEntries: ['local-driver.ts', 'local-tool-adapter.ts'] }, { folder: DISCOVERY, entries: { 'workflow-skills.ts': 'any', 'generated-marker.ts': 'any' }, testEntries: ['workflow-skills.ts', 'generated-marker.ts', 'render.ts'] }, + // A suite's checks, start and report change state, so only the commands reach them. Pull, push and their checks sync suites with the team's other definitions. + { + folder: BENCHMARK, + entries: { + 'plan.ts': 'any', 'terms.ts': 'any', 'suite.ts': 'any', 'check.ts': 'any', 'table.ts': 'any', 'result.ts': 'any', 'history.ts': 'any', 'suite-cache.ts': 'any', + 'trial.ts': ['src/workflow-commands.ts'], + 'start.ts': ['src/workflow-commands.ts'], + 'report.ts': ['src/workflow-commands.ts'], + }, + testEntries: ['plan.ts', 'terms.ts', 'suite.ts', 'check.ts', 'table.ts', 'result.ts', 'trial.ts', 'start.ts', 'history.ts', 'run-plan.ts', 'conformance.ts', 'suite-cache.ts', 'arm-workflow.ts', 'team-without.ts'], + }, ]; +/** Whether ENTRIES names `from` among the importers of the entry `to`. */ +const namedEntry = (/** @type {string} */ from, /** @type {string} */ to) => ENTRIES.some(({ folder, entries }) => { + const importers = to.startsWith(folder) ? entries[to.slice(folder.length)] : undefined; + return Array.isArray(importers) && importers.includes(from); +}); + export default defineRule({ meta: { type: 'problem', docs: { description: 'Preserve workflow driver and entry module boundaries.' } }, @@ -89,7 +112,9 @@ export default defineRule({ } if (from === SKILL + 'transitions.ts' && !typeOnly) violation = true; const flat = from.startsWith(FLAT) && ![FLAT + 'drivers/', PLACE, DISCOVERY].some((folder) => from.startsWith(folder)); - if (flat && (to.startsWith(FLAT + 'drivers/') || to.startsWith(DISCOVERY) || (to.startsWith(PLACE) && !(from === 'src/workflow/protocol.ts' && to === PLACE + 'place.ts')))) violation = true; + if (flat && (to.startsWith(FLAT + 'drivers/') || to.startsWith(DISCOVERY) || (to.startsWith(PLACE) && !namedEntry(from, to)))) violation = true; + // The engine, the protocol and the drivers never import a feature built on them, such as benchmarks. + if (from.startsWith(FLAT) && to.startsWith(BENCHMARK)) violation = true; if (from === 'src/workflow/pr-delivery.ts' && to === 'src/workflow/engine.ts') violation = true; if (violation) context.report({ node, message: `Workflow module boundary forbids ${from} -> ${to}. Use the declared entry module or an explained inline disable.` }); if (from === 'src/workflow/snapshot.ts' && !typeOnly && resolved) { diff --git a/oxlint-plugin-teamai/rules/workflow-boundaries.test.ts b/oxlint-plugin-teamai/rules/workflow-boundaries.test.ts index 48a682311..fc43206a8 100644 --- a/oxlint-plugin-teamai/rules/workflow-boundaries.test.ts +++ b/oxlint-plugin-teamai/rules/workflow-boundaries.test.ts @@ -44,3 +44,7 @@ rule.valid('follows a shared TSX dependency without crossing the local driver', 'src/workflow/attempt-marker.ts': `export { guide as attempt } from '../guide.js';`, 'src/guide.tsx': `export const guide = ;`, } }); +rule.invalid('rejects a workflow module import of a benchmark module', `import { plan } from '../benchmark/plan.js';`, undefined, { filename: 'src/workflow/engine.ts' }); +rule.valid('allows the engine to reach the place entry', `import { place } from './place/place.js';`, { filename: 'src/workflow/engine.ts' }); +rule.invalid('rejects a benchmark entry from a module it does not name', `import { startBenchmark } from './benchmark/start.js';`, undefined, { filename: 'src/pull.ts' }); +rule.valid('allows a benchmark entry from the module it names', `import { startBenchmark } from './benchmark/start.js';`, { filename: 'src/workflow-commands.ts' }); diff --git a/scripts/doc-checks/readme-matrix.ts b/scripts/doc-checks/readme-matrix.ts index 18e97ae45..fdc2df258 100644 --- a/scripts/doc-checks/readme-matrix.ts +++ b/scripts/doc-checks/readme-matrix.ts @@ -5,6 +5,7 @@ import path from 'node:path'; import { ALL_MODEL_AGENTS } from '../../src/models/profile.js'; import { runsTeamHookOn } from '../../src/hooks.js'; +import { isLaunchTool } from '../../src/tool-launch/tools.js'; import { isToolId, TOOLS, type Support } from '../../src/tools.js'; import { DEFAULT_TOOL_PATHS, type ToolId } from '../../src/types.js'; @@ -28,6 +29,7 @@ const COLUMNS = [ 'usage', 'sessions', 'scorers', + 'benchmarks', ] as const; type Column = (typeof COLUMNS)[number]; @@ -115,5 +117,7 @@ for (const [label, row] of matrix) { const sessionJudge = decided(d.features.sessionJudge) === '✓'; // Without a session reader, a tool whose session judges its skill-driven runs scores only those (✓³). assert.equal(cell(row, 'scorers'), Array.isArray(d.sessions) ? '✓' : sessionJudge ? '✓³' : '—'); + // A benchmark arm runs on a tool the local driver launches. + assert.equal(cell(row, 'benchmarks'), isLaunchTool(tool) ? '✓' : '—'); }); } diff --git a/scripts/doc-checks/workflow-docs-examples.ts b/scripts/doc-checks/workflow-docs-examples.ts index bc4ae23dc..91d3eeee8 100644 --- a/scripts/doc-checks/workflow-docs-examples.ts +++ b/scripts/doc-checks/workflow-docs-examples.ts @@ -16,6 +16,7 @@ import { leaseNext, reportAttempt, startRun } from '../../src/workflow/engine.js import { formatDiagnostic } from '../../src/workflow/parse.js'; import type { DriverIdentity, WorkflowEnvelope } from '../../src/workflow/types.js'; import { knownSecrets } from '../../src/env-resolution.js'; +import { parseSuite, type SourceFile } from '../../src/benchmark/suite.js'; const ROOT = process.cwd(); const GUIDE = path.join(ROOT, 'docs', 'guide'); @@ -31,6 +32,7 @@ function guidePages(name: RegExp): string[] { const PAGES = guidePages(/^workflow.*\.md$/); const SCORER_PAGES = guidePages(/^scorers.*\.md$/); +const BENCHMARK_PAGES = guidePages(/^benchmarks.*\.md$/); interface Example { label: string; @@ -428,7 +430,7 @@ for (const page of ADVANCED) { ); } -for (const page of PAGES) { +for (const page of [...PAGES, ...BENCHMARK_PAGES]) { await check( `${page}: every job of a GitHub Actions workflow it shows says what it runs on`, () => { @@ -459,7 +461,7 @@ await check( cleanup, ); -for (const page of [...PAGES, ...SCORER_PAGES]) { +for (const page of [...PAGES, ...SCORER_PAGES, ...BENCHMARK_PAGES]) { await check( `${page}: every relative link to a .md file resolves`, () => { @@ -501,3 +503,69 @@ for (const page of SCORER_PAGES) { cleanup, ); } + +/** Each team file a page shows: a yaml fence whose first line is `# benchmarks/.yaml`. */ +function benchmarkFilesOf(page: string): SourceFile[] { + return examplesOf(page).flatMap(({ source }) => { + const relativePath = /^# (benchmarks\/\S+\.yaml)\n/.exec(source)?.[1]; + return relativePath === undefined ? [] : [{ relativePath, content: source }]; + }); +} + +/** The `npx teamai-cli` commands of the GitLab CI jobs a page shows: a yaml fence whose top-level job has a `script`. */ +function gitlabCommandsOf(page: string): string[] { + const JobsSchema = z.record(z.string(), z.unknown()).transform((jobs) => + Object.values(jobs).flatMap((job) => { + const script = z.object({ script: z.array(z.string()) }).safeParse(job); + return script.success ? [script.data.script] : []; + }), + ); + return examplesOf(page).flatMap(({ source }) => JobsSchema.parse(parseDocument(source).toJS() ?? {}).flat()); +} + +const teamaiCommands = (commands: string[]): string[] => commands.filter((command) => command.startsWith('npx teamai-cli ')); + +await check('finds the English and Chinese benchmark pages', () => { + assert.deepEqual(BENCHMARK_PAGES, ['docs/guide/benchmarks.md', 'docs/guide/zh-CN/benchmarks.md']); +}); + +for (const page of BENCHMARK_PAGES) { + await check(`${page}: each yaml block parses, and its suite and task files pass pull`, () => { + for (const example of examplesOf(page)) assert.deepEqual(parseDocument(example.source).errors, [], example.label); + const files = benchmarkFilesOf(page); + const [suite, ...others] = files.filter((file) => file.relativePath.endsWith('/suite.yaml')); + if (!suite || others.length > 0) throw new Error(`${page} shows ${others.length + (suite ? 1 : 0)} suite files, not one`); + const tasks = files.filter((file) => file !== suite); + assert.deepEqual( + tasks.map((task) => path.posix.dirname(task.relativePath)), + tasks.map(() => `${path.posix.dirname(suite.relativePath)}/tasks`), + page, + ); + + const parsed = parseSuite({ suite, tasks }); + + assert.deepEqual(parsed.diagnostics.map(formatDiagnostic), []); + assert.deepEqual(parsed.ok && parsed.warnings.map(formatDiagnostic), []); + assert.equal(parsed.ok && parsed.suite.tasks.length, tasks.length); + }); + + await check(`${page}: the GitLab CI job runs the teamai commands of the GitHub Actions job`, () => { + const github = teamaiCommands(ciRecipesOf(path.join(ROOT, page)).flatMap(runCommandsOf)); + assert.ok(github.length > 0, `${page} shows a GitHub Actions job that runs teamai`); + assert.deepEqual(teamaiCommands(gitlabCommandsOf(page)), github); + }); +} + +await check('shows the same benchmark files and CI commands in every language', () => { + const [english, ...others] = BENCHMARK_PAGES.map((page) => ({ + page, + files: benchmarkFilesOf(page), + commands: ciRecipesOf(path.join(ROOT, page)).map(runCommandsOf), + gitlab: gitlabCommandsOf(page), + })); + for (const other of others) { + assert.deepEqual(other.files, english?.files, other.page); + assert.deepEqual(other.commands, english?.commands, other.page); + assert.deepEqual(other.gitlab, english?.gitlab, other.page); + } +}); diff --git a/scripts/tool-support.ts b/scripts/tool-support.ts index 078544a18..e26a3504e 100644 --- a/scripts/tool-support.ts +++ b/scripts/tool-support.ts @@ -22,7 +22,7 @@ export const TOOL_SUPPORT_PAGES = { export type ToolSupportLang = keyof typeof TOOL_SUPPORT_PAGES; -const FEATURE_KEYS = ['env', 'workflows', 'recall', 'stats', 'tokens', 'aiBackend', 'votesJudge', 'sessionJudge', 'agentHooks'] as const satisfies readonly (keyof ToolFeatures)[]; +const FEATURE_KEYS = ['env', 'workflows', 'unattended', 'recall', 'stats', 'tokens', 'aiBackend', 'votesJudge', 'sessionJudge', 'agentHooks'] as const satisfies readonly (keyof ToolFeatures)[]; const TEXT = { en: { @@ -44,7 +44,7 @@ const TEXT = { notes: 'Notes:', columns: { tool: 'Tool', home: 'Home', skills: 'Skills', rules: 'Rules', blocks: 'Instruction blocks', agents: 'Agents', hooks: 'Hooks', mcp: 'MCP', sessions: 'Sessions', - env: 'Env', workflows: 'Workflows (local driver)', recall: 'Recall', stats: 'Stats', tokens: 'Tokens & cost', aiBackend: 'AI backend', votesJudge: 'Votes judge', sessionJudge: 'Session judge', agentHooks: 'HTTP agent hooks', + env: 'Env', workflows: 'Workflows (local driver)', unattended: 'Unattended prompts', recall: 'Recall', stats: 'Stats', tokens: 'Tokens & cost', aiBackend: 'AI backend', votesJudge: 'Votes judge', sessionJudge: 'Session judge', agentHooks: 'HTTP agent hooks', }, }, 'zh-CN': { @@ -66,7 +66,7 @@ const TEXT = { notes: '注：', columns: { tool: '工具', home: '主目录', skills: '技能', rules: '规则', blocks: '指令块', agents: 'Agents', hooks: 'Hooks', mcp: 'MCP', sessions: '会话', - env: '环境变量', workflows: '工作流（本地驱动）', recall: 'Recall', stats: '统计', tokens: 'Token 与费用', aiBackend: 'AI 后端', votesJudge: '投票评审', sessionJudge: '会话评审', agentHooks: 'HTTP agent hooks', + env: '环境变量', workflows: '工作流（本地驱动）', unattended: '无人值守提示', recall: 'Recall', stats: '统计', tokens: 'Token 与费用', aiBackend: 'AI 后端', votesJudge: '投票评审', sessionJudge: '会话评审', agentHooks: 'HTTP agent hooks', }, }, } as const satisfies Record; diff --git a/skill-data/core/references/commands.md b/skill-data/core/references/commands.md index dc08a21ba..0306523ea 100644 --- a/skill-data/core/references/commands.md +++ b/skill-data/core/references/commands.md @@ -235,11 +235,13 @@ Generated: do not edit by hand. Regenerate with - `--tools ` — Override local prompt tools for this run, as comma-separated name=tool pairs. With --driver skill: only default=, the tool this coding session works in, which judges its scored steps and which its steps' models resolve for when teamai installs agents for it. Pass it when the describe reply says the session names no tool - `--auto-approve` — Answer optional approve HITL requests yes and choose requests with their default: with --driver local in this process, with --driver skill for this run. A required approval still goes to the person - `--inputs ` — Values for the inputs the workflow declares, as a JSON object, for example {"issue":42}. Every declared input is required. + - `--timeout ` — The longest the run may take, such as 30m or 2h: its steps stop and it fails with timeout once the shorter of this and the workflow's own timeout passes. Its finally steps still run after. + - `--sandbox` — With --driver local, declare this machine a sandbox for this process: a disposable host such as a CI runner or container, where benchmark trials and steps that need sandbox isolation may run. workflows.sandbox in config declares it for every process. - `teamai workflow next ` — Get a lease, hitl, wait, or done JSON envelope. Show hitl.message and hitl.respond to a person, who replaces