[feat] Score a finished session on demand #1178

Parent: #1137

Goal

Scorers judge workflow steps (#1085). Most agent work happens in ordinary sessions, which no scorer can reach today. Let a member judge a finished session of Claude Code, Codex or OpenCode with a team scorer, keep the reasoning on their machine, and publish only the label.

This issue adds the on-demand path: what a session ran as, how its judge runs, and where its labels go. Scoring sessions in the background is #1179.

What a session ran as

A session's label carries the agent the session ran as (#1176): the tool and its version, model and effort, the team revision delivered to the checkout, the selection it was delivered for, whether it was edited, and the project commit. Most of that has to be read when the session starts, because a later pull changes it.

• Capture at session start. The session-start hook writes <dataHome>/sessions/captures/<platform>/<sessionId>/<captureId>.json: { captureId, platform, sessionId, capturedAt, agent }, where agent comes from [feat] Record the agent each workflow step ran as #1176's builder with driver: "session" and captureId is a random UUID.
• Before the pull. Today the hook dispatcher starts the background handlers, the pull among them, before the foreground ones (hook-dispatch-cli.ts:493-506). Write the capture's pull-record part before that pull starts. The tool version can be added after.
• What the agent had available. team.revision does not say which assets were delivered. The member's local config decides that: scope, primaryRole, additionalRoles, projects, subscribedTags, excludedSkills and inheritUserScope (LocalConfigSchema, types.ts). A pull writes them to the checkout's record in lastPullByWorkspace as selection when it delivers, and the builder copies them into team.selection from that record, with the workspace children the record already keeps (children, Proposal: a workspace mode for features that span several repos #913), never from the current config, because a member can change a role without pulling. A record without selection, written by an older CLI, gives no team and no fingerprint. The fingerprint covers team.selection. Workflow passes get it too, since they use the same builder. The definitions those names point to (projects.yaml, roles, tags.yaml) are in the team repo at team.revision.
• Not observed: the user scope. The builder reads the checkout's record and never the user scope's (teamDelivery, pull.ts). Assets delivered to HOME, which a session can also load, are not in agent.
• A pull during the session. When the pull a session started delivers a new revision to that checkout, it marks the session's captures teamChanged: true. The session may have loaded some assets from each revision.
• A resume writes another capture, since the model or the team config can change between the two starts. A SessionStart for compaction, where the tool reports it as such, writes none.
• No capture is written when TEAMAI_LAUNCHED_BY is set. A workflow step is scored as a workflow subject ([feat] Add workflow scorers and recorded quality labels #1085), and a judge is never scored.
• A capture that fails is logged at debug level and never fails the hook.

A label is about the whole session. Its agent is the captured observation only when every capture of the session holds the same observation and none is marked teamChanged. With teamChanged, agent keeps everything but team and fingerprint. With different observations, or none, the label has no agent. model.reported is set only when the session's messages name one model (metadata.model in the session format, ir.ts).

The command

$ teamai score --scorer review-quality                          # inside a session: scores this session
$ teamai score --session 7c1e… --source codex --scorer review-quality --push

teamai score comes from #1085. This issue adds --session, --source and --push.

• Which session. --session <id> names a session the tool keeps on this machine. Without it, the agent's session variable (CLAUDE_CODE_SESSION_ID, CODEX_SESSION_ID and the others agentSessionFromEnv reads) names the session and its platform, when the variables name exactly one. Otherwise the command fails with Pass --session <id>; scoring never guesses the session. It never falls back to the most recent session in the event log, since a guessed session spends a judge on, and publishes a label for, a session the member did not mean.
• --source takes the platform names the session commands use (session migrate --source). It can be left out when the agent's session variable or the captures name exactly one platform for that id. Otherwise the command fails and lists the platforms it found.
• <name> is a scorer in the team's scorers/ as cached by pull ([feat] Add workflow scorers and recorded quality labels #1085). An unknown or invalid name fails with the list of valid scorers.
• The session is read with AgentSession.find, redacted, and given to the judge as AgentSession.transcript entries (feat(session): safe session reads and copies for workflows, drivers, scorers, and benchmarks (#593 follow-up) #1135). Scoring does not depend on dashboard events: a session TeamAI never saw events for can still be scored.
• Subagents are part of the session's work. The evidence includes their transcripts, read as [feat] Add workflow scorers and recorded quality labels #1085 reads them for a step. Without them, a session that delegated its edits to a subagent reaches the judge as a session that did nothing.
• samplingRate is ignored: an explicit request always scores, including with samplingRate: 0.
• Evidence longer than 200,000 characters keeps the first and last 100,000, and the prompt says how much was cut.
• Before the judge runs, the command prints the session's redacted first prompt and directory, so the member sees which session is scored. After, it prints the label, the score, whether it passed, and the path of the reasoning file. When the label did not pass, it also prints To share what went wrong, ask your agent to read <reasoning file> and share it with the team (teamai contribute). The share workflow (skill-data/share) lists a failed label's reasoning file as something it can turn into a learning.
• --push publishes every pending label, as below.
• Scoring has no preview: the global dry-run guard refuses it before anything runs, with teamai score has no --dry-run preview, nothing was run.

Why a command of its own. This issue first put scoring on session save. That was dropped: session save records a summary of a session that ran on this machine, and scoring shares none of its behaviour. --force and --include-prompt would not apply to labels, --dry-run and the fallback to the most recent session would mean something else, and a pushed session (#1269) has no local events to summarize. A session subcommand was dropped too, because the subject is not always a session: Warp scores runs and, with includeDescendants, their child runs, and here workflow steps (#1085) and what a run delivered (#1247) are scored as well. teamai score takes one flag per kind of subject. This issue adds --session, #1269 adds --team-session and --label, and #1247 adds --run.

The judge

The judge is #1085's: same profiles, same evidence directory, same isolation policy. It runs on the member's machine, reads only the evidence directory, and answers.

• Profiles. [feat] Add workflow scorers and recorded quality labels #1085 keeps the ordered list of judge profiles described here, since workflow judges run first, and this issue uses it: one per tool, each used only for installed versions it was verified on. A tool without a verified profile cannot judge. With none available, the command fails: No installed tool can run a scorer judge with every tool turned off. Install a supported version of Claude Code, Codex or OpenCode. Find executables the way the workflow adapter does, not through the private candidate list in utils/ai-client.ts, which has no OpenCode.

• What a profile turns off. Running commands, editing files, reading outside the evidence directory, web access, subagents, MCP servers, and hooks or plugins from the member's or the project's config. The judge gets the prepared, redacted evidence and nothing else. It keeps only the login the tool needs to reach its model. An empty working directory and a cleared environment help, but they do not meet this on their own.

• Profiles as measured (verified in [feat] Add workflow scorers and recorded quality labels #1085):

  Tool	Profile	Access	Level
  Claude	-p --restricted --tools=Read,Grep,Glob --strict-mcp-config --no-session-persistence, cwd = evidence	read	verified
  Codex	-a never exec --ephemeral, a permission profile that reads only the evidence dir with network off, web_search = "disabled", [agents] enabled = false	read	verified
  OpenCode 2	run --standalone, isolated config and db, OPENCODE_DISABLE_PROJECT_CONFIG=1, evidence outside any checkout	read	verified
  CodeBuddy	Claude-like flags, no --restricted (it still lists about 70 tools)	full	best-effort until a run proves it

• It runs in a new empty temp directory, removed afterwards, with TEAMAI_LAUNCHED_BY=scorer, and without GITHUB_TOKEN, GH_TOKEN, GITLAB_TOKEN, GL_TOKEN, CI_JOB_TOKEN, SSH_AUTH_SOCK, GIT_ASKPASS and SSH_ASKPASS.

• Model. The scorer's model, resolved for that tool with the alias lookup [feat] Add workflow scorers and recorded quality labels #1085 uses. Without one, the tool's default. judge.model is the model the tool reports for the judge's answer, read with the per-tool output parsing the workflow adapter uses (local-tool-adapter.ts), or the concrete model requested when the tool reports none, so a change of a tool's default model does not merge two judges into one cohort.

• Untrusted evidence. The prompt fences the evidence as data, as votes-judge.ts does with <<<TEAMAI_UNTRUSTED>>>, since the judged agent wrote it.

• The answer. TeamAI reads the judge's stdout and writes the label. The judge writes no file. An answer that names no label of the scorer fails the command with a non-zero exit and writes no label. The raw answer is kept in a separate file for failed attempts.

Reuse of #1085

#1085 ships teamai score, the judge and its profiles, the evidence directory with subagents, the label type, and the label store with its outbox, all in src/scorers/. This issue adds the session subject on top. Session labels are #1085's label, with rubric and judge. Nothing here asks a workflow issue to change or wait.

Labels

The label is #1085's, with source: "session":

• subject: { platform, sessionId }.
• agent: as described above, or absent.
• used: { skills }, the team skills the session or its subagents loaded, found in their messages by the rules skill-use.ts applies to hook events: a call to the tool's skill tool, a read of a SKILL.md under a skill root, or a mention such as /name or Codex's $name. Only names of skills in the team repo are kept, so a member's own skills are not published. The session is the source because the hook's per-session claims file keeps recent turns only and is swept with the session's files, and the usage log carries no session id.
• rubric: { sourceHash, teamRevision? }, sourceHash being the hash of the scorer file.
• judge: { kind: "model", tool, model?, usage?, isolation, access, batched? }, with usage (inputTokens, outputTokens, cost: { amount, unit } in the tool's own unit) only as far as the tool reports it, or { kind: "human" } for a hand label ([feat] Compare scorers with hand labels, including on pushed sessions #1269). A union, so a hand label cannot name a tool or a model.
• The logical key is (scorer, rubric.sourceHash, subject, judge.kind). Scoring the same session with the same rubric again replaces the label of the same kind, so a rescore by the model never replaces a hand label, nor the reverse. An edited rubric gives a new one.

Where they are kept:

• Locally. <dataHome>/scores/labels.jsonl, one line per label, the newest per key winning when read. The judge's full answer goes to <dataHome>/scores/reasoning/<labelVersion>.md, one file per verdict, where labelVersion is the sha256 of the canonical label. labelId, the sha256 of the canonical logical key, names the pending file below. A replacement that fails keeps the earlier label and its reasoning. The reasoning file starts with a header of scorer, rubric.sourceHash, subject, label, scoredAt and the session's redacted first prompt, so a member finds a label's reasoning with grep. It is never published, like the rest of the file.
• Pending publication. <dataHome>/scores/pending/<labelId>.json, the same outbox pattern as run records (pendingDir and publishPendingRecords in workflow/records.ts). After a push, remove a pending file only if it still holds the label that was sent. A rescore written during the push stays pending.
• Published. scores/<member>/<YYYY-MM>.jsonl on teamai-reports, one JSON line per label, <member> being the username stats use and the month taken from scoredAt in UTC. Write with updateReports and read the base with readReportsOriginFile (utils/reports-branch.ts). Merge by key against origin, keeping the newest scoredAt, so a member's second machine cannot overwrite a newer label with an older one. Readers keep the newest scoredAt per key across all files.
• Opt-out. When usage reporting is off (isUsageReportEnabled), --push publishes nothing and prints Usage reporting is off, so labels stay on this machine. The labels stay pending.

Never published: the transcript, the prompt, the judge's answer and its reasoning.

Acceptance criteria

1. Starting a Claude session in a pulled project writes one capture whose agent.tool.name is claude and whose agent.team.revision is the full commit of that checkout's revision before the session's own pull. Resuming it writes a second capture. Compaction writes none.
2. A session the local workflow driver starts writes no capture.
3. teamai score --session <id> --source claude-code --scorer <name> prints a label, adds one line to labels.jsonl and one reasoning file, and changes nothing on teamai-reports. It works for a native Codex session TeamAI has no dashboard events for.
4. With --push, scores/<member>/<YYYY-MM>.jsonl on origin has that label. A phrase taken from the session's transcript does not appear anywhere on the reports branch.
5. Scoring the same session again with the same scorer leaves one line for that key on origin, with the newer scoredAt. After editing the scorer's prompt, the next score adds a second line.
6. (Owned by [feat] Add workflow scorers and recorded quality labels #1085, shared here.) For each profile, the effective configuration has every capability above turned off. A fixture transcript that tells the judge to run a command, read a file outside the evidence, edit a file and call an MCP server or subagent gets each attempt denied, and no private content appears in the answer. An installed version that does not honor the profile is reported unavailable. A judge that merely chose not to call a tool does not pass this criterion.
7. With no profiled tool available, the command fails with the message above and writes no label.
8. With usage reporting off, --push prints the message above and publishes nothing.
9. A session whose two captures differ in model gives a label with no agent. A session whose pull delivered a new revision mid-session gives a label with agent but no team and no fingerprint.
10. When the push fails, for example offline, the label stays pending, and the next --push publishes it. Two clones of the same member pushing labels for the same month keep the newest per key. A label written during a push stays pending.
11. Session labels and workflow labels share one label type: a reader parses both with it, and labels of both sources share the key (scorer, rubric.sourceHash, subject, judge.kind).
12. teamai --dry-run score --session <id> --source claude-code --scorer <name> --push exits non-zero with teamai score has no --dry-run preview, nothing was run (the global dry-run guard; score stays off its list), launches no judge, writes no label, reasoning or pending file, and publishes nothing.
13. Inside a Claude Code session with CLAUDE_CODE_SESSION_ID set, teamai score --scorer <name> scores that session without --source. Outside any session, without --session, it fails with the message above and launches no judge, even when the event log has sessions.
14. Two members on the same team revision, one with primaryRole: backend and one with primaryRole: frontend, get captures with different team.selection and different fingerprints. After teamai roles changes the role without a pull, the next capture keeps the selection of the last pull. A record written before this change gives no team.
15. A session that loaded a team skill through the skill tool and read another team skill's SKILL.md gives a label whose used.skills names both. A skill that is not in the team repo is not listed.
16. A Claude Code session whose only edit is made by a subagent gives the judge that subagent's tool call, and a team skill only the subagent loaded is in used.skills.
17. With a scorer without model, a fake judge that reports model m1 gives a label with judge.model: "m1".
18. A label that did not pass prints the next-step line above. A label that passed does not.

Test plan

• Unit tests for capture writing, skipping and marking, selection in the pull record and the builder, the agent rule across captures, the logical key, the merge against origin and the outbox.
• Judge launch tests with a fake executable that records argv and environment, and checks of each profile's effective configuration.
• E2E with a fixture session and a fake tool for criteria 3 to 5, 8, 10, 11, 13 and 15 to 18.
• Real-CLI runs for criterion 6 on each tool available locally, recorded in the PR. Say which tools were not run.

Watch out for

• The session-start hook must stay fast. Writing the capture's pull-record part first must not wait on --version.
• docs/guide/ and docs/guide/zh-CN/: add a section on teamai score to the scorer guide [feat] Add workflow scorers and recorded quality labels #1085 adds. Regenerate skill-data/core/references/commands.md with npx vitest run commands-reference -u.

Out of scope

Scoring part of a session, background scoring (#1179), scoring a subagent as its own subject, showing labels (#1182), and a spending budget beyond #1179's daily limit.

Dependencies

• Blocked by: [feat] Add workflow scorers and recorded quality labels #1085 (teamai score, scorer files, the judge and its profiles, the label type and store, subagent transcripts in the evidence), [feat] Record the agent each workflow step ran as #1176 (the agent observation)
• Unblocks: [feat] Score finished sessions in the background #1179, [feat] Show scorer results in the dashboard #1182, [feat] Score what a workflow run delivered #1247, [feat] Compare scorers with hand labels, including on pushed sessions #1269