Repository navigation
chore(harness): Stage 3 model/effort routing + audit workflows - #437
Conversation
Route models at the subagent/workflow tier (locked: no opusplan, Opus stays the
session driver; model routing lives entirely below the driver).
Effort routing on existing subagents:
- parity-checker, i18n-syncer -> effort: low (already model: haiku)
- contract-aligner, security-reviewer, design-reviewer -> effort: medium
- design-reviewer also pinned model: sonnet: it carried no model and would have
inherited the Opus driver, defeating the routing for an auto-firing reviewer.
Audit fan-outs -> dynamic workflows (the biggest single quota win):
- new .claude/workflows/audit.mjs: one generic engine keyed by kind. Haiku finder
per surface -> Haiku default-refuted skeptic per serious finding -> completeness
loop-until-dry -> returns verified structured findings.
- new .claude/workflows/prod-readiness.mjs: runs the four audit workflows in
parallel (one-level nesting) + ops-layer fan-out + ops skeptic verify.
- /audit-{security,tests,performance,code-quality} + /prod-readiness become thin
non-fork drivers: resolve scope -> invoke Workflow -> Opus synthesizes the report
from the compact return. Each keeps a headless/CI inline (Explore) fallback since
the Workflow tool is absent from CI tool-allowlists.
The multi-phase fan-out/verify/loop reasoning the Opus context:fork used to do is
now free deterministic JS; Opus spends only on synthesis, every leaf agent is Haiku.
Runtime mechanics smoke-tested via a 1-agent probe workflow.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nsmhto1amnzXEmbSvuJgw9
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
|
There was a problem hiding this comment.
Review: PR #437 — "chore(harness): Stage 3 model/effort routing + audit workflows"
Recommendation: Request changes
| Severity | Count |
|---|---|
| Critical | 0 |
| High | 2 |
| Medium | 1 |
| Low / Info | 0 |
The diff only touches .claude/ harness tooling (5 subagent frontmatter files, harness-upgrade-plan.md, 5 SKILL.md rewrites, and 2 new .claude/workflows/*.mjs files) — no apps/*, packages/shared, or orbit-api code, so all five gated review subagents (parity-checker, i18n-syncer, contract-aligner, security-reviewer, design-reviewer) are N/A, and dimensions 8–14 of the rubric are N/A. Lint/type-check/test are also N/A — nothing in this diff falls under any workspace's turbo run lint/type-check/test target.
Findings
[HIGH] .claude/workflows/audit.mjs's isApiSurface/isUiSurface classifier silently drops 4 of the 6 security surfaces (injection, secrets-config, ratelimit-ai-abuse, error-web-auth) whenever /audit-security (or /prod-readiness) is scoped to api/backend, because those surfaces' label isn't prefixed api- and their where text never contains the literal substring "orbit-api" — unlike the performance/tests/code-quality KIND configs, which all use the api- prefix consistently. An api-scoped security audit — the most common invocation for backend hardening — silently reports a much narrower posture than it claims (misses secrets/CORS-in-Program.cs, SQL/EF injection, and rate-limit/AI-abuse coverage), with no signal that those checklist sections never ran. Fix: tag each surface with an explicit repo field instead of inferring it from a label prefix / substring match on free text.
[HIGH] The entire Stage-3 mechanism (Workflow({...}) calls added to all five SKILL.md files, plus the two new .mjs files) depends on a Workflow tool that could not be confirmed as present in this harness build — a search for it turned up nothing in either the main tool list or the deferred-tool index. Both new .mjs files also end in a bare top-level return, which is a SyntaxError under normal Node/ESM — the PR's own commit message concedes node --check flags this and calls it "a false positive" on an unverified assumption, and separately admits "Live end-to-end fires on the next real /audit-* invocation." Each skill does carry a documented inline-fallback for when Workflow is unavailable, so this degrades gracefully rather than breaking outright — but the PR checks off "biggest single quota win" in harness-upgrade-plan.md on a mechanism that has never actually been run to completion.
[MEDIUM] .claude/skills/audit-code-quality/SKILL.md:38 says "run the fan-out inline per Phase 3's fallback," but in this file's new numbering the fallback text lives in Phase 2 ("Run the audit workflow"); Phase 3 is "Apply the rubric." Looks like copy-paste drift from audit-tests/SKILL.md, where Phase 3 genuinely is the right phase. One-line fix.
What's good
- The
effort:/model:frontmatter additions on the five subagents are correct and minimal —design-reviewerpicking upmodel: sonnetalongsideeffort: mediumis the right catch (it previously had nomodel:and would have inherited the Opus driver). - Collapsing five near-duplicate fan-out/verify/loop SKILL.md sections into one shared
audit.mjsengine is a real DRY win once the surface-classification bug is fixed. - The plan doc is honest that end-to-end verification is still pending, rather than overclaiming.
Notes
- Cross-repo dimensions (contract-aligner, orbit-api side of the backward-compat guard) are not applicable — this diff touches no API contract or code.
- A stray scratch file (
_review_audit.mjs, created during syntax verification of thereturn-at-top-level claim) could not be removed in this sandbox — please delete it manually if it shows up untracked at repo root.



Stage 3 of the harness overhaul (
.claude/research/harness-upgrade-plan.md): route models at the subagent/workflow tier. Locked: noopusplan, Opus stays the session driver — routing lives entirely below it.Effort routing on existing subagents
parity-checker,i18n-syncer→effort: low(alreadymodel: haiku)contract-aligner,security-reviewer,design-reviewer→effort: mediumdesign-revieweralso pinnedmodel: sonnet— it carried nomodeland would have inherited the Opus driver, defeating the routing for an auto-firing reviewer (matches its two effort-tier siblings).Audit fan-outs → dynamic workflows (the biggest single quota win)
.claude/workflows/audit.mjs— one generic engine keyed bykind: a Haiku finder per surface (reads that audit's rubric/checklist) → a Haiku default-refuted skeptic per serious finding → a completeness-critic loop-until-dry → returns verified structured findings + coverage + deferred..claude/workflows/prod-readiness.mjs— runs the four audit workflows in parallel (one-levelworkflow()nesting) + the ops-layer fan-out (observability / multi-instance / background-durability / staging; backups always Deferred) + ops skeptic verify./audit-{security,tests,performance,code-quality}+/prod-readinessbecome thin non-fork drivers: resolve scope → invokeWorkflow→ Opus synthesizes the report from the compact return. Each keeps a headless/CI inline (Explore) fallback since the Workflow tool is absent from CI tool-allowlists (only/pr-reviewruns in CI, viaTask).The whole multi-phase fan-out/verify/loop reasoning the Opus
context: forkused to do is now free deterministic JS; Opus spends only on synthesis, every leaf agent is Haiku.Validation
node --check"illegal return" is a false positive — the Workflow runtime allows top-levelreturn).agent({model:'haiku', effort:'low', schema})returns a schema-validated object with file-tool access. ✅/audit-*invocation.Paired PR (orbit-api): thomasluizon/orbit-api#312 — the lockstep
effort: mediumon the shared review agents.🤖 Generated with Claude Code
https://claude.ai/code/session_01Nsmhto1amnzXEmbSvuJgw9