Repository navigation
Conversation
5959410 to
824495b
Compare
The PR description documents sufficient testing: it includes a built real-CLI verification record and representative E2E results. Deferred provider/agent coverage is not blocking. Reviewed only the specified diff; ran no PR code. |
The earlier exact The PR description documents sufficient testing, including a built real-CLI run. Deferred provider/agent coverage is not blocking. Reviewed only the specified diff; ran no PR code, builds, or installs. |
Earlier findings: The exact-submodule case, dot-prefixed path bypass, and local-tool deadline omission are resolved in the current diff. Testing: The PR description documents sufficient testing, including a built real-CLI run. Deferred provider/agent coverage is not blocking. Reviewed only the specified diff; ran no PR code, builds, or installs. |
Earlier findings: All are resolved: protected-submodule handling, dot-prefixed paths, escaped globs, local-tool deadlines, and both languages’ timeout guidance. Testing: The PR description includes sufficient built real-CLI verification. Deferred provider/agent coverage is not blocking. Reviewed only the specified diff; ran no PR code, builds or installs. |
621a620 to
79b2e12
Compare
e3d7f4a to
f7a7737
Compare
Earlier findings: All are resolved in the current diff, including the protected-submodule bypasses, local-tool deadline, bilingual timeout guidance and installation deadline. Testing: The PR description documents sufficient built real-CLI verification. Deferred provider/agent coverage is not blocking. Reviewed only the specified diff; ran no PR code, builds or installs. |
|
No new confident findings in the current diff. Earlier findings: All are resolved, including the protected-submodule bypasses, local-tool deadline, bilingual timeout guidance, installation deadline, judging deadline, and malformed-glob validation crash. Testing: The PR description documents sufficient testing, including built real-CLI verification. Coverage deferred to CI is not blocking. Reviewed only the specified diff against the trusted base review criteria. Ran no PR code, builds, tests, or installs. |
STiFLeR7
left a comment
STiFLeR7
left a comment
There was a problem hiding this comment.
Summary. Adds --benchmark to teamai workflow start: runs YAML-defined benchmark suites (tasks with a starting commit, an optional known-good fix, and a check or success-criteria judge) across tool/model "arms" and repetitions, each trial in an isolated clone with secrets stripped and git hooks/filters/fsmonitor neutered, scored and tabulated. Adds --try to validate one task file's check against its starting commit and fix before it joins a suite. Rebased on #1304, closes #1180.
Verified.
- Secret stripping genuinely happens before any tool/driver launch and can't be reintroduced by a step's
requires.env:negotiate()(src/workflow/requirements.ts:272-276) filters a step'srequires.envto exclude every name in the trial's withhold list (stripped) before building the granted lease'senvScope.provide— soprovideandwithholdare disjoint by construction, not just by call order instepEnvironment(src/env-exec.ts:347-360).TRIAL_WITHHELD_ENV(terms.ts:24-26) is threaded into the plan, prompt/setup scope, and--try. Confirmed by readinge2e/benchmark-env.test.tsin full: none of the withheld names appear in setup/tool/check env even when the team env declares them, and no git hook/fsmonitor/clean filter in the clone sees them either. - Isolation is real:
freshClone(trial.ts:482-504)git inits a fresh repo, fetches onlystartingCommitby SHA (never a branch ref) from the run's checkout or a commit cache, and detaches HEAD there — nogit remote addis ever called, and a SHA-only fetch cannot pull descendant commits, so the fix commit is never reachable in the trial's clone during the prompt phase.check.add/restoreProtectedonly write fix-derived bytes in after the prompt's report is already committed, i.e. after the agent's tool process has exited. check.add/source.fixverification is real, not a naive copy:writeAdded(trial.ts:799-823) andrestoreProtected(trial.ts:699-751) read the source commit into a private index viaread-tree, write withcheckout-index --stdin, and verify every written path byte-for-byte against the commit's blob hash viaholds()(trial.ts:840-853), including the executable bit and symlink-vs-file kind. Submodule paths are explicitly refused.- Path traversal is blocked at the schema level:
RepoPathSchema(suite.ts:30-33) rejects empty/./..segments, backslashes, and a leading:.check.addpaths must additionally fall under acheck.protectglob. - The per-copy lock genuinely serializes prepare/report/check/sweep on the same trial copy without serializing across different trials — confirmed via
e2e/benchmark-trial-copy.test.ts(a second concurrentnext --preparegetsbusyrather than a corrupted/double clone; a killed report can be retried cleanly). - Judge-at-timeout doesn't double-score (
e2e/benchmark-judges.test.ts: exactly one judge launch, run ends timeout, trialunscored); "Not logged in" doesn't corrupt subsequent trials (verified both by the PR's real-CLI evidence and the general error path intakePrompt/settle). - Spot-checked 3 of the Evidence table's test-file references against their actual bodies (
benchmark-env.test.ts,benchmark-checks.test.ts,benchmark-judges.test.ts) — all three test exactly what their row claims, including several of the listed adversarial cases. npx tsc --noEmitandnpx oxlint --type-aware --deny-warnings --report-unused-disable-directivesboth ran clean on this branch.
Findings. None I can back with a concrete failure scenario or a cited rule violation. I specifically looked for races in the per-copy lock, double-scoring at judge timeout, "Not logged in" corrupting later trials, and path traversal via check.protect/check.add — all are handled and tested.
Not covered / unverified.
- Could not execute
npm run buildorvitestmyself in my environment (a sandbox policy against running an unreviewed PR's own code) — statictsc/oxlintare clean, but I don't have independent confirmation of the claimed578 files, 16565 passed/383 passede2e numbers beyond reading the relevant test sources directly. - Didn't re-audit the pre-existing, unmodified
src/update.tsfile-lock internals (acquireLock/releaseLock) — treated as trusted existing infrastructure since this PR doesn't touch it. - Didn't review the other ~110 changed files outside
src/workflow/benchmark/*.ts(pull/review/doctor wiring, dashboard, docs) at the same depth — focused the time on the new module and its safety-critical claims. TRIAL_WITHHELD_ENVonly covers git/CI credential-shaped names, not generic cloud credentials (AWS_*,NPM_TOKEN, etc.). Looks like a deliberate scope choice perterms.ts's own comment ("credentials a trial could push, publish or authenticate with") rather than an oversight — flagging it as a question rather than a finding, in case it's worth widening in a follow-up.
Verdict. Comment (approve-leaning, no blocking findings). The two most safety-critical claims — the secret-strip-wins-over-requires.env guarantee and the no-remote/ancestors-only clone isolation — both check out at the actual git-plumbing and negotiation-logic level, not just by the PR description, and the adversarial test cases listed genuinely exist and test what they claim.
|
@STiFLeR7 thanks for the thorough review. You found nothing blocking, so could you approve it? |
STiFLeR7
left a comment
STiFLeR7
left a comment
There was a problem hiding this comment.
Confirmed — nothing blocking. Approving.
SaulMoro
left a comment
SaulMoro
left a comment
There was a problem hiding this comment.
This was generated by AI during triage.
Re-review at dde0913. The README benchmarks column, the CHANGELOG entry, the workflows.sandbox docs (EN and zh-CN), the isolated-run hint and the finally case are all resolved. I checked the hint, the refusals and the finally case with the built CLI.
[P1 blocking] A workflow arm that starts with a command gets the wrong tool's team assets. This is the CI review's clone.ts finding, reproduced. Setup: a member with enabledAgents: [claude], a suite with defaults: { tool: codex }, and an arm workflow: codex-flow whose first step is a workspace: continue command, followed by a prompt. The plain Codex arm's clone has .codex/skills/answer/SKILL.md. The workflow arm's clone has no Codex team skill, so its Codex prompt runs without the team's assets, and the table shows that trial as Not verified. Cause: src/workflow/place/clone.ts:362 falls back to member.enabledAgents when the lease has no tool, and a command-first lease never has one. Carry the arm's tool on the lease regardless of the first step, and add a case where enabledAgents lacks the arm's tool.
[P2 non-blocking] The model-less refusal says "Set model on the arm", but the suite rejects model next to workflow on an arm ("an arm changes at most one of the defaults keys"). defaults.model is what works; I ran it and the arm runs on that model. The message should point there.
[P3 nit]
- The isolated-run hint
teamai workflow start agent-fixdrops the run's inputs, so it fails withinputs.task is missing. The samerestartCommandtext is used by thenot_retryablerefusal. - The CHANGELOG says
--fail-on-regressionfails a job whose pass rate dropped. Per the body, it exits 1 only when an arm's paired drop has its whole 95% interval below zero. - The body still does not mention the
uninstallchange (src/uninstall.ts,src/hooks.ts), or the new refusal and hint.
CI at this head: no failures so far. macOS Node 22/24 and the E2E shards were still running when I checked.
…am assets Two causes made a workflow arm run without the team assets it compares: - The clone installed for the tool of the step that opened it, else the member's enabled agents. A workflow arm that opens its clone with a command got the member's tools, not the arm's. The plan now writes the arm's tool into the clone's team selection (enabledAgents), which the install delivers to. - Each step's commit puts the clone's agent files back as the commit has them, which takes the delivery out, and a later step that continued the clone was never given it again. The install now keeps a copy of what it delivered, and a later step gets it back, recorded as overlay, as a worktree step gets its overlay. Proved by a benchmark-workflow-arms e2e: a codex arm whose workflow starts with a command, on a member who installs only claude, ran its prompt with no team recorded and no team skill in its clone; it now runs on the pulled revision with the skill there, as the prompt arm does. The clone-install e2e files pass (61 tests).
The refusal of a workflow arm with no model whose workflow runs an agent said to set model on the arm, which the suite rejects: an arm with workflow changes no other defaults key. It now says to set model in the suite's defaults, which the arm inherits, and the guide says so too.
Start a new run: teamai workflow start <name> dropped the run's inputs, so for a workflow that takes any it failed with invalid_inputs. Both the not_retryable refusal and the failed-run hint now pass them with --inputs, quoted for a POSIX shell. Proved by the workflow-from case of a workflow with inputs, which expected the command without them and now expects it with them.
On Windows, removing a table test's team directory failed with EBUSY in three CI runs out of about seven, each time in this file. Each test ends its run, and record publication then runs git in the team directory, which these tests never read; no other process was seen there. They now run with usage reporting off, so no publication git runs there. The holder is not confirmed: git calls there are awaited and fail fast, as the directory is not a repository. If Windows still fails, the removal helper should name what holds the directory.
Earlier findings: Resolved, including tool installation for command-first workflow arms. Testing: The description documents sufficient built real-CLI verification. Coverage deferred to CI is not blocking. Reviewed only |
…flow ran A workflow step has no attempt of its own; its prompts do. So a workflow arm cancelled while its prompt held a lease read as not_run, and one whose tool the driver could not launch read as error, with no line naming the arm. A trial's result, and that line, now read the workflow step and every step under it: not run until one of them was leased. Proved by two benchmark-workflow-arms e2e cases, red before (error and not_run swapped) and green after; benchmark-lifecycle and benchmark-local, which pin the same rules for prompt arms, pass.
Team changes grouped arms by tool, model and effort only, so a prompt arm on one team revision and a workflow arm on the next drew one line, showing a workflow change as a team change. A line now also keeps its workflow, which its chip names. Proved by a benchmark-history case, red before (one line across both revisions) and green after.
Earlier findings: The previously reported concrete cases are resolved; the loop-history case is distinct. Testing: The description includes sufficient built real-CLI verification. Coverage deferred to CI is not blocking. Reviewed only |
…l is classified A trial cancelled between two loop passes has no current attempt, though an earlier pass of its work was leased and kept in passes or the run's archive. It read as not_run, which drops a trial that ran from the error count. Tested: benchmark-table cases for a pass kept on the step and one archived.
… deadline cloneHead ran git with no deadline, so a filter or config a step left in its clone could hold the local driver before launch past the suite's time budget. Tested: workflow-clone-place case with a git that never returns on status.
Earlier findings: Resolved in the current diff, including pre-launch observation deadlines and historical child-attempt classification. Testing: The PR description includes sufficient built real-CLI verification. Coverage deferred to CI is not blocking. Reviewed only |
runOutputs dropped every output that read null, because resolvePath reads a missing path as null too. A step declaring a json output can report null, and the run's done.outputs then lost it. pathValue reads a missing path as undefined, so only an output no step gave is left out. Tested: workflow-protocol case with a null output and one from a skipped step.
Earlier findings: The previously reported concrete cases are resolved, including null JSON outputs. The command-first classification case above is distinct. Testing: The description documents sufficient built real-CLI verification. Coverage deferred to CI is not blocking. Reviewed only the specified three-dot diff against the trusted base criteria. Ran no PR code, builds, tests, or installs. |
…} step A trial runs the arm's workflow from the benchmark plan's output, and admission refuses a from step there. The run started anyway and failed with invalid_output; the preflight now refuses the arm before any run, naming the step. Tested: benchmark-workflow-arms e2e case, which created a failed run before.
…un when it starts with a command The arm's first step, a command, is leased before its prompt finds the tool missing, so the trial counted as leased and read error, and its unavailable-tool notice was gone. Only a prompt's lease now shows the tool launched. Tested: benchmark-workflow-arms e2e case with a command-first workflow on codex, which read error before.
|
No new confident findings in the specified diff.
Reviewed only |
|
No new confident findings in the specified diff.
Reviewed only |
… its envelope A refused report surfaced only as a ZodError on data, which does not say which step was refused or why. The trial-timeout case failed that way on CI and does not reproduce locally, so the next failure needs to name its cause. Tested: benchmark-driver-contract e2e, 7 cases.
enabledAgents is new in workspace.clone.team.selection, and no key of selection was listed. Name them all, with the default enabledAgents takes.
… the run deadline Both cases ran a second arm the driver cannot run, so the run ended only at its 15 s timeout, and the claude trial had to report before it. With the trial timeout's 3 s hang, clone and check, that margin was thin on a loaded runner: the trial timeout case failed in 2 of 4 CI runs with a refused report. The codex arm proved nothing these cases name, and the first case still covers the suite timeout settling a trial not_run. One arm and a 60 s run deadline end the run with its trial. Tested: benchmark-driver-contract e2e, 7 cases; the two cases now take 8 s and 5.6 s, without waiting on the run deadline.
|
No new confident findings in the specified diff.
Reviewed only |
benchmarkRunConfig names a real GitHub URL as the team remote, so a run that ended published its record with git ls-remote against github.com from a unit test. On Windows the team directory then stayed busy when the test removed it (EBUSY), first in benchmark-table, now in benchmark-trials. The config now publishes nothing; the one test of the record queue opts in, and publishes nothing either. Tested: every user of benchmarkRunConfig (5 unit files, workflow-conformance unit and e2e) with a logging git on PATH: no ls-remote, fetch or push left.
|
No new confident findings in the specified diff.
Reviewed only |
SaulMoro
left a comment
SaulMoro
left a comment
There was a problem hiding this comment.
This was generated by AI during triage.
Re-review at 677af10. Everything from the earlier rounds is resolved, and the description now covers the clone redelivery, the null output, enabledAgents, the arm refusals and the restart hint.
The commits since my last comment are test and docs changes:
- The
selectionkeys are documented in both languages, and they matchsrc/workflow/place/clone.ts. - The outside-driver trial-limit cases now run one arm with a 60 s run
timeout, so only the trial limit can fire. That was the intermittentworkflow reportrefusal in E2E. - The in-process benchmark fixtures no longer publish, which removes the Windows
EBUSYon teardown.
CI is green at this head: all 18 checks. macOS Node 24 needed one rerun for a workflow-workspace timeout; it passed on the rerun.
[P3 nit] The restart hint quotes --inputs POSIX-style, as the other printed commands do, so it may not paste into PowerShell as-is.
@jeff-r2026 this one is ready for your merge.
Summary
A team could not tell whether a tool, a model, its own assets or one of its workflows fixes its real tasks better. Now
teamai benchmark run <suite>runs a suite from the team repo: every arm works the same tasks in fresh clones and is scored by a check it never sees. It is teamai's second default workflow,workflows/teamai-benchmark.yaml, which a team can replace, and the engine and drivers know nothing about benchmarks.An arm changes the tool, the model, the team assets or the workflow, and a
matrixcrosses them:The default workflow, which a team's
workflows/teamai-benchmark.yaml(orworkflows/<ns>/teamai-benchmark.yaml) replaces, for example withoutstrict:teamai benchmark runexitspending, and with--fail-on-regressionwhen nothing comparable is published (with a notice)workflow start);reportfailed or was cancelled;--fail-on-regressionfound an arm whose paired drop has its whole 95% interval below zeroThe dashboard's Benchmarks page. These screenshots are from this PR's earlier revision; since then Compare also pairs arms that differ only in
workflow, awithoutarm lists its paths, and the empty page suggeststeamai benchmark run.Compare: how much the team's assets add, trial by trial (
onboarding-docs, a seeded workspace).Team changes: one tool and model across the team revisions it ran on, with the files each changed.
Arms: every arm over time, as before.
+41 pp · 95% +16 pp to +59 ppover 27 paired trials,−60%cost per passed trial+15 pp, interval crosses zero:not separated67ee2cechanged 3 files,session-fixation1/12 to 3/12claude 67% · 95% 57%–75% · not separated from codexmigrate-db?Light theme, Cost axis, zh-CN, tablet and phone
Generic workflow keys this adds, usable by any workflow
workflow: { from: <step> }definitiona needed step printed; admitted when that step reports. A bundle may not holdfromorprompt_filesteps; a refused admission fails the producerinvalid_output. A run with afromstep is isolatedworkflow,jsonfinally:workflow cancelends themdone.outputsoutputs, resolved afterfinallyworkflow start --timeoutworkspace: { clone: { commit, repo?, team } }none, a ref or{ without: [paths] }tool:can.tools;tool_unavailablewhen no driver of analonerun can take itpermissions: unattendedcan.unattended; tools without a supported mode failtool_unavailablemodel: { name, effort? }on_limit: end_turntimeout,max_turns,budgetend the turn from tool launch and keep the work; on a workflow step, an allowance its prompt turns spend; drivers opt in withcan.end_turnrequires: { …, strict: true }isolationandnetwork; withholds the credential list under a strict sandbox; a local-only run that cannot meet it is refused at startscorers: { sampling: all }requiredisolated: trueteamaiteamai …gets the CLI that runs the step first on itsPATHBenchmark record
workflows/runs/<run id>/benchmark.jsonin the scope's data directory, andbenchmarks/runs/<run id>.jsononteamai-reportswhen usage reporting is on. The releasedBenchmarkRecordshape, field for field (limits,scoringCost, baselines, armaliasandsource), plusschemaVersion: 1,run,outcome,teamaiVersion,startedAt,finishedAt,username?,workflow: { name, origin, sourceHash },arms[].team(pulled | ref | without (paths) | none, with its revision),arms[].workflow?,trials[].agent?,trials[].used?(loaded | unknownper skill),completeon tokens and cost, andskillChanges. Pairs are derived fromtrials[]when read. No prompts, commands, credential URLs, local paths or hidden check bytes. The workflow run record dropsbenchmarkand gainsisolated.Type of Change
Test Plan
npx tsc --noEmitpassesnpm run lintpassesnpx vitest runpasses, except the GitLab request-deadline tests, which fail the same way onmainon this machineThe two commits after that add tests only. On them:
npm run test:e2e -- src/__tests__/e2e/benchmark-19 files, 166 passed;npx vitest run src/__tests__/benchmark-history.test.ts20 passed; tsc and lint ok.Real-CLI e2e record
The built
dist/index.jsin a throwawayHOME, with a real Claude login and default Claude settings (no permission allow list). A team repo with one skill,answer; a project whosenode test.jsfails; a suite with one task and two arms,pulledandteam: { without: [skills/answer] }.(Columns that were all 0 are left out above.)
permissionMode: bypassPermissions, frompermissions: unattendedunder the strict sandbox.teamai-reportsasbenchmarks/runs/<run id>.json, and no clone is left inTMPDIR.gitlabandgithubproviders, and Windows (the.cmdwrapper of a step'steamai). The e2e suite covers them with fake tools; real runs are left to CI.Issue criteria and the tests that cover them
e2e/benchmark-pull,benchmark-suitee2e/benchmark-local(eight trials)e2e/benchmark-team-assets,e2e/benchmark-locale2e/benchmark-enve2e/benchmark-checkse2e/benchmark-checkse2e/benchmark-modelsbenchmark-tablee2e/benchmark-judgese2e/benchmark-local,benchmark-tablee2e/benchmark-lifecycle,e2e/benchmark-judges,e2e/benchmark-skill-loadse2e/benchmark-driver-contract,workflow-enginee2e/benchmark-pull(review --branch),review-branche2e/benchmark-local,e2e/benchmark-run,benchmark-tablee2e/benchmark-rune2e/benchmark-lifecycle,e2e/benchmark-local,workflow-records,e2e/benchmark-ci-jobe2e/benchmark-parallele2e/benchmark-fail-on-regressione2e/benchmark-trye2e/benchmark-team-arms,e2e/benchmark-locale2e/benchmark-modelse2e/benchmark-team-armse2e/benchmark-team-arms(team branch, frozen models and scorers)e2e/benchmark-team-arms(team remote out of reach)e2e/benchmark-local,e2e/benchmark-team-armse2e/benchmark-team-assets(run ends, run cancelled halfway)benchmark-table,e2e/benchmark-team-armse2e/benchmark-team-arms(team branch, frozen models and scorers)e2e/benchmark-driver-contract(team branch)benchmark-history(run pairs and history)benchmark-historye2e/benchmark-skill-loadsbenchmark-suite,benchmark-historye2e/benchmark-workflow-armse2e/benchmark-workflow-armse2e/benchmark-workflow-armsbenchmark-team-without,e2e/benchmark-team-arms,e2e/benchmark-pull(review --branch)e2e/benchmark-workflow-armse2e/benchmark-limits,benchmark-tablereview-branch,e2e/benchmark-pullbenchmark-suite,e2e/benchmark-locale2e/benchmark-checks,benchmark-suitee2e/benchmark-pull(review --branch and run),review-branch,benchmark-suitebenchmark-suite,e2e/benchmark-localPaths are under
src/__tests__/.Related Issues
Closes #1180, closes #1181, closes #1400, closes #1401. #1402 builds on
teamai benchmark try,source.fixandcheck.add; #1469 on the generated scorer pins. Other features as default workflows: #1589.Notes for Reviewers
source.fix,check.add,trial,team(withwithoutpaths),matrix, armworkflowandparallelship in this first release.workflow start --benchmark,--member-aliasesand--tryleaveworkflow startforteamai benchmark run|try|plan|check|report; they were never released.workflows.max_parallel, as the guide already says;outputsnow reachdone.outputs, including anulla step gave;teamai …gets the running CLI first on itsPATH.teamai uninstallin a workspace whose hooks live in HOME now keeps HOME's built-in hooks while the user scope or another workspace on the machine still runs them, and removes only this workspace's team hooks.src/benchmark/imports onlysrc/workflow/'s public interface; the engine, protocol and drivers never importsrc/benchmark/.workflow-module-boundaries.test.tsfails closed on a new edge, with an empty allowlist. The suites cache moved tosrc/benchmark/.requirement_unmet), where it used to print a notice, run every trial tonot_runand exit 0.check.addfiles are written after setup, where they used to be written before. A run andteamai benchmark tryshare one order: clone, setup, restorecheck.protect(trial only), writecheck.add, check.benchmark_localis nowisolated_run.planrefuses every suite withteam_ref_blockedand leaves no run (Q19), where trials used to settlenot_run.--member-aliasesstays onteamai benchmark run; the default workflow takes it as inputaliases(team | member), and a team override without that input is refused under the flag.team: { without }takes team repo paths for any asset, namespaces included, in place of skill names.planuses pull's resolvers and refuses a path that changes nothing: an inactive namespace, a shared path an active namespace replaces ("list X too"), a path tags leave out, a skill the member excludes, or nested paths. It is exact underskills/,rules/,agents/,claudemd/anddocs/; elsewhere a path only has to exist. An entry inside a shared file still needs a branch.--fail-on-regressioncompares with the newest comparable record published onteamai-reports, leaving out this machine's pending queue, and exits 0 with a notice when there is none.parallelis 1 to 32, serial by default.teamai-benchmarkoverride resolve by namespace like any team workflow. Two active namespaces defining one name refuse at start withworkflow_namespace_conflict.permissions: unattended, including arm workflows. A team override without that floor keeps the tool's permission settings and reports it.task, hasfinallysteps, runsworkflow: { from }, or runs anagentwhile the arm sets nomodel(setmodelin the suite'sdefaults).continuestep after another one sees them. A benchmark installs them for the arm's tool, through the newworkspace.clone.team.selection.enabledAgents, even when the arm's workflow starts with a command.not_runwhen the tool could not launch, even after a first command ran, anderroronce a prompt was leased or the trial was cancelled after a lease, including an earlier loop pass.--inputs.can; its deadline, and anend_turntimeoutMs, count from the first lease, so a restart never extends a step.leasedAt, so the overrun is at most one prompt timeout per prompt.TEAMAI_WORKFLOW_ATTEMPTfrom the leaseenvfor its command steps,cloneincan.workspace,can.env: trueandcan.isolation: sandboxfor the strict sandbox,can.setup,can.unattendedfor unattended prompts,can.end_turnfor trial limits, the arm's tool incan.tools, and the lease workspace ascwd.--timeout.outputs.table.pairscarry the interval, and no arm is recommended. Skill loads are shown against the suite's defaults arm.workflows/teamai-benchmark.yaml, thensrc/benchmark/(start,plan,trial,report,team-without), then the generic keys insrc/workflow/.