SaulMoro · Oct 26, 2026 · Oct 26, 2026 · Oct 26, 2026 · Oct 26, 2026 · Oct 26, 2026
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -21,6 +21,7 @@ All notable changes to this project will be documented in this file. See [standa
 - teamai ships default workflows a team gets without writing them: `teamai-share-learnings` shares what a session taught as team learnings, one or several after one approval, each recorded as an outcome of the session, then runs `teamai-share-session`, which publishes the session's transcript and scores it. pull writes no `teamai-share-learnings` skill while recall is off or the source is read-only, since its learning step runs `share`, and `teamai recall enable`/`disable` add or remove that skill at once. A team file of the same name replaces a default; for one name the project's team file wins, then the user scope's, then teamai's default. A team `teamai-share-session` keeps the five inputs `teamai-share-learnings` passes it, with their types: with other inputs, or one of another type, pull warns and writes no `teamai-share-learnings` skill, `teamai-share-learnings` refuses to start with `invalid_inputs` before any learning is shared, and the friction reminder goes back to `teamai skill get share`. `teamai list skills -v` labels the skills pull generates for workflows `[workflow]`. `teamai list workflows` shows each workflow's source (`team` or `default`), `workflow start --describe` reports it as `source`, a run records it as `origin` with the teamai version of a default, and the dashboard's Workflow Runs page shows it in a Definition column, with the learnings a run shared beside its pull requests in an Outcomes column. A run of a default continued on a machine whose teamai ships another version of it refuses with `definition_mismatch` (for [#1388](https://github.com/Tencent/teamai-cli/issues/1388)).
 - `workflow start --describe teamai-share-learnings --driver skill`, and the same for `teamai-share-session`, proposes a `value` and a `why` for each input, for the calling session: the session and its tool, publishing when the attended publish policy is `ask`, the scorers that sample the session when attended scoring is `ask`, and whether teamai's own judge or the session's agent judges. A team override gets values for the inputs that keep their name and type (for [#1388](https://github.com/Tencent/teamai-cli/issues/1388)).
 - The dashboard's Overview has a **Member settings** panel: for the selected workspace, each member setting some level sets, its value, the level and file that set it, and what the levels below say, as `teamai doctor` lists them, plus the keys of a settings-only `~/.teamai/config.yaml` that nothing reads. The Session sharing panel names the setting behind the daily judging limit (for [#1459](https://github.com/Tencent/teamai-cli/issues/1459)).
+- `teamai benchmark run <suite>` runs a benchmark suite from `benchmarks/<suite>/`: each task, a starting commit with a check or success criteria, goes to every arm (a tool, model, team revision or team workflow) a few times, each trial in its own clean clone on a machine declared a sandbox. The run prints a pass-rate table and publishes its record for the dashboard's Benchmarks page. `try` checks one task before it joins a suite, `plan` and `report` print the plan and the table, and `--fail-on-regression` fails a CI job when an arm's pass rate dropped by more than chance against the last comparable published run. It runs as `teamai-benchmark`, a default workflow the team can replace (for [#1180](https://github.com/Tencent/teamai-cli/issues/1180)).
 
 ### Changed
 

diff --git a/README.ja.md b/README.ja.md
diff --git a/README.ko.md b/README.ko.md
diff --git a/README.md b/README.md
diff --git a/README.th.md b/README.th.md
diff --git a/README.zh-CN.md b/README.zh-CN.md
diff --git a/docs/dev/workflow-engine.md b/docs/dev/workflow-engine.md
diff --git a/docs/guide/benchmarks.md b/docs/guide/benchmarks.md
diff --git a/docs/guide/dashboard.md b/docs/guide/dashboard.md
@@ -9,7 +9,7 @@ teamai dashboard
 teamai dashboard --port 8080
 ```
 
-Open `http://localhost:3721`. The sidebar is Overview, Team Execution, Workflow Runs, Scorers, Team Context, Resources, Distribution, and Team Improvement. The numbers are from this machine.
+Open `http://localhost:3721`. The sidebar is Overview, Team Execution, Workflow Runs, Scorers, Benchmarks, Team Context, Resources, Distribution, and Team Improvement. The numbers are from this machine.
 
 ## Session sharing
 
@@ -50,7 +50,7 @@ The Workflow Runs page has three lists for the selected workspace. **Local runs*
 
 On the install that offered a run, the next step is `teamai workflow resume <run>` instead, which takes it back there. A local run that is offered, or that another install now holds, says so under its status. **A runner needs** (**Their machine needs** on a run that needs a person) lists the `requires.env` keys and prompt steps of the steps left: set those up on the runner, as [its job](workflow.md#take-offered-runs-from-ci) does. **Not on this machine** is what this machine lacks of them, as `teamai workflow resume` lists it as `cannot run here`. The status line counts the offered runs and the ones that need a person. The page claims nothing: the command runs a driver on the machine that takes the run.
 
-The page shows what a run record publishes, plus each step's live status and the options of a waiting HITL request. Inputs, `for_each` items, outputs, prompts, HITL messages, and local paths stay in the run file. Read a HITL question where the run's driver asks it. A run's details show what each pass delivered: the commits on its branch, the pull request delivered for it, and the learnings `teamai contribute` recorded inside its attempt.
+The page shows what a run record publishes, plus each step's live status and the options of a waiting HITL request. Inputs, `for_each` items, outputs, prompts, HITL messages, and local paths stay in the run file. Read a HITL question where the run's driver asks it. A run's details show what each pass delivered: the commits on its branch, the pull request delivered for it, and the learnings `teamai contribute` recorded inside its attempt. The details of a [benchmark](benchmarks.md) run that ended show its results, usage and task × arm tables, as the CLI prints them.
 
 A waiting approve or choose request can be answered from the run's details. The answer goes through the same `respond` operation as the CLI, with channel `dashboard` and `agent: false`, so it can answer a required approval. A dashboard started inside an agent session records `agent: true`, as an answer from that shell would, and cannot answer a required approval. It carries the pass the page showed: if that request was answered or replaced in the meantime, the page shows the refusal and reloads. Another website cannot send an answer. A request without an `Origin` header, such as a plain `curl`, is refused too. This marks an answer as sent from a browser; it does not stop a local script that sets the header.
 
@@ -97,6 +97,45 @@ Labels from benchmark runs are left out. The page reads labels and recorded conf
 
 Labels from sessions and workflow runs show how the tools did on the work that came along. To compare tools, models or team assets on the same tasks, run a benchmark.
 
+## Benchmarks
+
+The Benchmarks page shows every ended [benchmark](benchmarks.md) run of the selected workspace: the runs on this machine and the run records on `teamai-reports`, over 30, 90 or 365 days. Pick a suite in **Suites**. Each question below names the panel that answers it, with what that panel showed for a team that ran `fix-auth` every week and `migrate-db` on three Claude models:
+
+| Question | Panel | What it showed |
+|---|---|---|
+| Should the team switch its auth fixes to claude? | **Leading arm** | `claude 67% · 95% 57%–75% · not separated from codex`. Codex has 56% (47%–65%). The intervals overlap, so these 7 runs do not tell the two apart. Add repetitions or tasks before you switch. |
+| Did the auth skill we merged on October 7 help? | **Pass rate per run** | A dotted line, `Team assets changed · 62432de`, marks the first run with the skill. Compare each arm's points on either side. The **Runs** table marks that run's team revision `changed`. |
+| Which model is worth paying for on `migrate-db`? | **Pass rate against cost**, on **Cost** | sonnet passes 73% for 0.31 USD a trial, opus 78% for 0.89 USD, and their intervals overlap. The dashed line joins the arms nothing beats on both pass rate and cost: haiku, sonnet, opus. |
+| Which task needs fixing, in the code or in the suite? | **Tasks by arm** | `refresh-race` has 7/7 under **Unstable runs**: in every run, some arm both passed and failed it, so one trial says little. `session-fixation` is 0% to 19% on every arm: read its check before blaming the tools. |
+| Why did the September 23 run drop to 27%? | **Tasks per run** and **Runs** | `login-copy` is a hatched cell in that run's column, and **Runs** shows `1 not measured` under its pass rate: the task's check passed without changes in that run, so its [baseline](benchmarks.md#baselines) was invalid, and its trials count as planned but not passed. **Tasks by arm** marks it `not measured in 1/7`. Open the run from **Runs** to read its trials. |
+
+**Last run** shows the newest run's trials by result for each arm, with its usage, deviations and the tasks it did not measure. **Open the run** opens its details on the Workflow Runs page.
+
+The panels above are the **Arms** tab. Two more tabs answer what changed between arms, and between team revisions. Here, for `onboarding-docs`, a [matrix](benchmarks.md#cross-tools-with-team-assets) of claude and codex with the pulled team assets and with none, and for `fix-auth` again:
+
+| Question | Tab and panel | What it showed |
+|---|---|---|
+| How much do our team assets help on onboarding docs? | **Compare**, by **Team assets** | `pulled team against no team assets +41 pp · 95% +16 pp to +59 pp` on claude, over 27 paired trials, and `−60%` cost per passed trial. |
+| Do they help codex as much? | **Difference per pair of arms** | codex gains +15 pp, but its interval crosses zero: these runs do not tell its two arms apart. |
+| Which task do the assets fix? | **Each task** | `oncall-runbook` goes from 1/9 to 7/9 with the assets; `first-pr-guide` barely moves. |
+| Did the auth-tokens skill help claude on `fix-auth`? | **Team changes**, on `claude · claude-opus-5-5` | 67%, 62%, 70% over three team revisions, in commit order, none told apart from the one before it. Click `67ee2ce · Add the auth-tokens skill and a security reviewer`: it changed three files, and `session-fixation` went from 1/12 to 3/12. |
+| Did a tool update move the numbers? | **Arms**, **Pass rate per run** | A dashed `Tool updated · claude 2.4.0` line marks the first run on the new version. |
+
+**Compare** lists each two arms of the suite's current version that differ only in their team assets, tool, model or workflow, and pairs their trials by run, task and repetition. **Team changes** follows one tool, model and workflow across the team revisions it ran on with the pulled assets or a ref, and names each revision from this machine's team clone, with the files outside `benchmarks/` it changed since the one before it.
+
+How the numbers are counted:
+
+- A pass rate is passed trials over planned trials, as the run's own table prints it.
+- The pooled numbers, the task tables and the leader count only the runs of the suite's current version: the suite files of its newest run. **Pass rate per run** also draws the older runs, after a dashed `Suite changed` line.
+- An arm is its name, tool, model, effort and team assets. When one of them changed, the older runs are a separate line, marked `earlier configuration`.
+- **Pass rate by arm** also counts the arm's deviations and trials not verified over those runs, as each run's table counts them. A deviation is a trial whose prompt ran on another tool, model or team revision than its arm's.
+- The interval is a 95% Wilson score interval. Two arms whose intervals overlap are not told apart by these runs.
+- A difference in **Compare** is over paired trials, with a 95% interval for paired proportions (Newcombe). One whose interval crosses zero is `not separated`.
+- Usage is the median over the trials whose tool reported it. Cost is per trial.
+- A run on this machine that was also published shows once, from this machine.
+
+The page shows what a run record publishes. It never shows a prompt, a diff, or a transcript.
+
 ## Resources
 
 The Resources page lists every rule in the team repo for the selected workspace: its name, namespace, description, and whether `pull` delivers it there. Root rules reach everyone; namespaced rules follow the active knowledge namespaces, and a root rule is marked replaced when an active namespace rule of the same name stands for it. Rules left out by tag subscriptions are marked too.

diff --git a/docs/guide/distribution.md b/docs/guide/distribution.md
@@ -101,7 +101,7 @@ sharing:
 .teamai/mcp/mcp.yaml
 ```
 
-`teamai push` sends every distributed resource type — skills, rules, agents, docs, env, hooks, mcp, models, claudemd, learnings, wiki, workflows, prompts, and scorers — with `--project` and `--role` namespace placement.
+`teamai push` sends every distributed resource type — skills, rules, agents, docs, env, hooks, mcp, models, claudemd, learnings, wiki, workflows, prompts, scorers, and benchmarks — with `--project` and `--role` namespace placement.
 
 ### Git permissions
 
@@ -155,6 +155,8 @@ A project is the service they are working on. After this directory is set to tha
 
 `knowledge` covers rules and shared instructions (`claudemd/`). A project can also declare learnings, env, hooks, mcp, models, docs, and wiki on its own.
 
+Benchmark suites follow `resources.benchmarks`. `benchmarks/<suite>/` goes to everyone. Once a role or project lists `<ns>`, `benchmarks/<ns>/<suite>/` goes only to members who have `<ns>` active, and `benchmarks/<ns>/` holds suites, not a `suite.yaml` of its own.
+
 ```yaml
 # manifest/roles.yaml
 version: 1

diff --git a/docs/guide/scorers.md b/docs/guide/scorers.md
@@ -92,7 +92,7 @@ $ teamai score --lease s5d1e07c2a9b4a/a91c0d7e0 --json '{"driver":{"kind":"local
 {"data":{"label":{"scorer":"review-quality","label":"good","score":1,"passed":true,…},"published":true}}
 ```
 
-`samplingRate` picks the succeeded passes to judge, the same way on every machine. `workflow next` never hands out judge work. `--lease <ref>` judges here with the tool the lease names, in a [confined profile](#isolation), and settles the lease like any driver's answer. A failed judge leaves the lease open.
+`samplingRate` picks the succeeded passes to judge, the same way on every machine. `workflow next` never hands out judge work. `--lease <ref>` judges here with the tool the lease names, in a [confined profile](#isolation), and settles the lease like any driver's answer. A failed judge leaves the lease open. A step whose `scorers` sets `sampling: all` is the exception: its scorers judge every succeeded pass, and the run waits for them before its `finally` steps and its end, while `workflow next` answers `wait` with `judging`, the judge entries still open. The local driver judges them itself and settles a failed judge at once; another driver's are judged with `teamai score --next` on the install that ran them. A [benchmark](benchmarks.md) judges its trials this way.
 
 The label records `used.skills`, the team skills the pass or its subagents loaded, found as for [a session](#score-a-session) from what the checkout held as teamai delivered it when the pass started. A `workflow:` step's label lists every skill its child passes recorded, and has no `used` when none of them recorded one.
 
@@ -319,7 +319,7 @@ Session judging: attended auto, set by scorers/scoring.yaml sessions; unattended
 
 | What is judged | Set by |
 |---|---|
-| A workflow step's passes | The step's `scorers`, sampled by each scorer's `samplingRate`. `sessions` plays no part. |
+| A workflow step's passes | The step's `scorers`, sampled by each scorer's `samplingRate`, or every pass under `sampling: all`. `sessions` plays no part. |
 | A captured session: yours, or one a `CI` or `TEAMAI_MODE` job ran that teamai did not launch | `sessions` for the session's run mode, for every scorer with `sessions: true`, sampled by its `samplingRate`. |
 | A session teamai launched, such as a workflow step or a judge | Nothing. It writes no capture, and its step is judged as a pass. |
 | A session of a tool teamai reads no sessions from, such as Pi | Nothing. `teamai doctor` notes each such tool the team enables, in each mode whose publish or scoring policy is not `off`. With the defaults, that is both modes. |
@@ -350,7 +350,7 @@ Every unattended driver ends with `teamai score --drain`. The local driver runs
 1. In unattended mode, every session this data home captured as unattended counts as finished, the calling session included. Attended sessions keep the six-hour rule.
 2. It judges every open judge entry this machine can judge: sampled step passes, and sampled sessions the policies allow, within `daily_limit`. While another judge pass runs on this machine, such as the background judge, it judges nothing and says so. Run the drain again once that pass ends.
 3. It publishes every pending label, as `score --push` does, and the workflow run records a run could not publish, as `teamai pull` does.
-4. With `publish: auto` for the current mode, it archives on `teamai-reports`, scrubbed, the finished captured sessions of that mode and the step sessions of the runs driven here that are not archived yet. A step session it cannot archive from here, because teamai reads no sessions from its tool or this machine cannot read it, is listed once as skipped with the reason, and never tried again. The other step sessions of that run are still archived.
+4. With `publish: auto` for the current mode, it archives on `teamai-reports`, scrubbed, the finished captured sessions of that mode and the step sessions of the runs driven here that are not archived yet. A step session it cannot archive from here, because teamai reads no sessions from its tool or this machine cannot read it, is listed once as skipped with the reason, and never tried again. The other step sessions of that run are still archived. The trial sessions of a [benchmark](benchmarks.md) run stay on the machine that ran it.
 5. It reports usage to `stats/<member>.yaml`, as `teamai pull` does, so the counts of an ephemeral runner reach the team.
 6. It prints what is left: entries whose scorer pins a model no judge tool here runs (`judge_model_unavailable`), or whose isolation this machine does not meet, and every open entry when no judge tool is installed here. Those wait for another driver's `score --next`.
 
@@ -430,4 +430,4 @@ A `best-effort` tool cannot judge for a scorer that requires `verified`, nor can
 | `<dataHome>/sessions/captures/` | What each session started as, written by the session-start hook. Removed a day after the session's file is gone. |
 | `scores/<member>/<YYYY-MM>.jsonl` on `teamai-reports` | Published labels, the newest per scorer, rubric, subject and judge kind. Labels wait in `<dataHome>/scores/pending/` until a push lands, and stay there while usage reporting is off. |
 
-Labels never enter the run or its record. Workflow Runs shows each pass's labels, and what judging cost apart from the run's own tokens. For a run on this machine that counts every judge attempt; for a run a member published, only the labelled ones. The dashboard's [Scorers page](dashboard.md#scorers) shows each scorer's pass rate over time and by configuration.
+Labels never enter a workflow run or its record. A benchmark run freezes each trial's verdict when it ends, and its record carries the trial's result, not the label. Workflow Runs shows each pass's labels, and what judging cost apart from the run's own tokens. For a run on this machine that counts every judge attempt; for a run a member published, only the labelled ones. The dashboard's [Scorers page](dashboard.md#scorers) shows each scorer's pass rate over time and by configuration.