Repository navigation
feat(score): lease a session, or a skill run, to the driver's own judge (#1386) - #1470
Conversation
…ent#1386) `score --next` with `run` finds a run of another project through `findRunConfig`, but `score --lease <ref>` and `--submit` looked only in this directory's scope and the user scope, so a driver that took the lease from project B could not judge or submit it from B. `leaseConfig` now also searches the project partitions `findRunConfig` scans. A run's lease is read in its run's scope, as `runConfig` resolves it. The partition scan moves out of `findRunConfig` so both share it.
`teamai score --session <id> --scorer <name> --json <driver>` now hands out a lease, as `score --next` does, instead of rejecting the payload. A driver whose tool teamai does not run as a verified judge here, such as a Cursor session or an external orchestrator, judges the session from `prompt.md` and the evidence only, then submits through the existing submit path. There is one lease path: `takeJudgeLease` takes the named session as a second entry source (`JudgeRequest.entries` with `source: session`). The explicit request skips the background filters (sampling, the six idle hours, `scoring.sessions`, the scorer's `sessions`, the daily limit check, judged once) but its attempt still counts toward the day's launches. It takes nothing on `isolation_unmet` (new at lease time), `judge_model_unavailable`, scorers of one `--batch` that need different models, or a live lease on the entry (`busy`). The driver that holds the lease gets the session again when it asks with other tools, as its earlier lease is released. The payload names no `run` or `canTakeLease`. The submit check that refused a session lease once any judge had labelled the session now counts only labels made since the lease was taken, so an explicit lease of a session labelled before can settle. A lapsed lease no longer settles once another lease holds the session. Without `--json`, `score --session` is unchanged.
…o verified judge here (Tencent#1386) A skill-driven run whose session tool has no verified judge on this machine, such as Cursor, reached `done` with its pass judge work left open. describe now answers `judging.kind: session` for it, says the session judges the run in a subagent at best-effort, and offers the "Same, but without judging" start (`--no-judge`). The start or resume keeps that in skill.json, and `done` then carries `score`: the `score --next` literal that leases the run's judge work to the session's own judge. The workflow skill tells the session to run it, judge each lease in a fresh subagent from `prompt.md` and the evidence only, and submit until `idle`, or release a lease it cannot answer and stop. A session that resumes or takes over the run gets the same `score`, and a `--no-judge` run gets none. When every scorer of the scored steps asks for `verified`, describe answers `elsewhere` as before, since the session cannot judge them. With some of them, the session judges the rest and describe says that work stays open for another driver. A resume reads the scorers its run pinned. With judging `here`, the confined judge stays the only path.
|
No findings I’m confident warrant reporting in the specified PR diff. The PR description documents sufficient testing: two representative real-CLI runs against built Review was read-only; I did not run, build, or install PR code. |
…Tencent#1386) GitHub Copilot CLI and Muse Code have no session reader, so score --session still cannot judge their sessions. A workflow they drive with the skill driver is now judged by the session itself, so their scorers cell gets a footnoted mark. The matrix test derives the mark from the agent session variable the skill driver reads, so a tool gains it only when the driver can name it.
The PR description documents sufficient testing: two representative real-CLI runs against built The earlier pass raised no findings, so there are no resolved findings to list. Review was read-only; I did not run, build, or install PR code. |
…cent#1386) TOOLS gains a sessionJudge feature cell, shown as the Session judge column of the tool support page. The README test takes the scorers mark from it, and checks it against the agent session variable the skill driver names the tool from.
|
No new findings I’m confident warrant reporting in the specified PR diff. The earlier [P2 non-blocking] descriptor finding is resolved: Copilot and Muse now have The PR description documents sufficient testing: two representative real-CLI runs against built Review was read-only; I did not run, build, or install PR code. |
SaulMoro
left a comment
SaulMoro
left a comment
There was a problem hiding this comment.
This was generated by AI during triage.
@hz89Dev Thanks, this meets #1386. We compared the built CLI with main:
- Session lease.
score --session <id> --json <driver>returns a lease, wheremainrefuses it withinvalid_arguments.- A second request while the lease is live gets
busy. - A scorer that requires
verifiedgetsisolation_unmetand writes no attempt. - An unknown tool gets
judge_model_unavailable. - Submitting the same lease twice writes one label.
- After the lease lapses a new one can be taken, and the old one is refused with
lease_expired. - Without
--json, the output matchesmain.
- A second request while the lease is live gets
- Skill-driven Cursor run. describe answers
sessionand offers both options, anddone.scoreleases and labels the pass. Averifiedscorer's work stays open, and--no-judgegives noscore. - Failure paths. A lapsed or released lease can be taken again, the run stays
done, and another driver can still take the open work. - Older CLI. An older CLI reads the new
skill.jsonfield without error.
tsc, lint, the unit files (also under a symlinked TMPDIR) and commands-reference pass. The new tests fail on main for the intended reasons. The two e2e files that timed out under load pass when run alone. The last three commits only touch the tool descriptor, the generated pages and the README matrix, and they resolve the CI review's descriptor note.
Non-blocking:
- [P3 nit]
donekeeps carryingscoreafter all the work is judged (src/workflow/drivers/skill/turn.ts:362), so an agent that re-reads it makes one extra call that returnsidle.
Not run here: a real Cursor judge answer (submits used hand-written JSON), and the Codex, CodeBuddy and OpenCode agents.
Summary
A Cursor session, or an orchestrator whose tool teamai has no verified judge for here, could not get its work judged.
score --session <id> --json <driver>was rejected, and a skill-driven run reacheddonewith its scored pass left open. Now the driver judges with its own tool, through the same lease and submit pathscore --nextuses:teamai score --session <id> --scorer <name> --json <driver>leases that session to the driver.donecarriesscore, thescore --nextliteral for the run's judge work. The workflow skill tells the session to judge each lease in a fresh subagent that reads only the evidence.It also fixes
score --lease/--submitfailing withlease_unknownfrom another project's directory, wherescore --nextwithrunalready found the run.The README scorers column marks GitHub Copilot CLI and Muse Code
✓³: they have no session reader, so their own sessions are still not scored, but a workflow they drive with the skill driver is now judged by the session. Pi Coding Agent stays—: the skill driver does not name a session tool without agent support. The other tools without a session reader set no agent session variable.TOOLSrecords this as asessionJudgefeature cell, shown as the Session judge column of the tool support page.score --session ... --jsoninvalid_argumentsinvalid_jsonwithoutcan.judge, or withrun/canTakeLeasescoring.sessions, scorersessions, judged-once and the daily limit check. Still counts toward the day's launcheslease_expiredlease_expired, and the live lease's answer is keptdone, pass left openjudging.kind: session,done.score, the session judgesverifiedelsewhere. With only some, that work stays open for another driverscore --lease/--submitfrom project B for a run of project Alease_unknownType of Change
Test Plan
Proof: two real-CLI runs on the built
dist/(below), plus unit and e2e tests. Each new test was seen failing without its fix.npx tsc --noEmitpasses, at each commitnpm run lintpassesnpx vitest runpasses: 590 files, 17541 testsnpm run test:e2epasses forscore-sessionandscore-external-driver(7 tests)Tests added:
session-scoring.test.ts, "with a driver's own judge":busy, then a new lease after release or labelisolation_unmetwith no attempt writtencursor, andjudge_model_unavailable--batchrules; scorer file and--pushrefused;session_not_foundworkflow-skill-transitions.test.ts:sessionfor a tool with no confined judgeelsewherewhen the scorer requiresverified, and when scoring is offdone.score,--no-judgedrops it, and the decision uses the scorers the run pinnedworkflow-skill-driver.test.ts: an unattended run its session judges reachesdonewith ascoreliteral whose payload leases the pass. A--no-judgestart has none.readme-matrix.test.ts: the scorers cell is✓³for a tool with no session reader whosefeatures.sessionJudgeis✓. That cell must agree with the agent session variable the skill driver names the tool from. Before the README edit the test failed for Copilot CLI and Muse Code. With Pi's cell set to✓tit fails for Pi.score-session.test.ts: lease,busy, payload errors, submit from the lease cwd, stored label.score-external-driver.test.ts: the lease is taken, judged (wrong_drivershows it was found) and submitted from a second initialized project. Without the fix it fails withlease_unknown.Real CLI: a Cursor session drives a scored workflow and judges it itself
macOS, Node 24,
cursor-agent2026.10.21 on PATH (best-effort judge profile),CURSOR_CONVERSATION_IDset. Team repo and member in temp dirs. The script runs each literal the CLI prints.Real CLI:
score --session --jsonon a hook-captured Claude Code sessionNot run here: the
gitlab/githubproviders and the Codex, CodeBuddy and OpenCode agents. Left to CI.Related Issues
Closes #1386
Notes for Reviewers
judging: sessionwith the reason "cursor has no verified judge profile". The skill judges in a fresh subagent, as the issue specifies.busynow has two causes: the attempts lock is held, or a live lease holds the named session. The messages differ.scoreliteral is not among the stub's pre-approved commands, so the agent asks before running it, as for anyteamai score.sessionJudge, and older files read as before. describe gains asessionkind, anddonegains an optionalscore.docs/guide/scorers.md(en and zh-CN, lines 88-107) are not valid refs: they ares<12 hex>/a<7 hex>, and the CLI takess<13>/a<8>. The new example uses a valid ref. I left the old ones alone to keep this diff on its goal.