Skip to content

chore(harness): Stage 3 model/effort routing + audit workflows - #437

Merged
thomasluizon merged 1 commit into
mainfrom
chore/harness-stage-3-model-effort-routing
Jul 9, 2026
Merged

thomasluizon merged 1 commit into
mainfrom
chore/harness-stage-3-model-effort-routing

Conversation

@thomasluizon

Copy link
Copy Markdown
Owner

Stage 3 of the harness overhaul (.claude/research/harness-upgrade-plan.md): route models at the subagent/workflow tier. Locked: no opusplan, Opus stays the session driver — routing lives entirely below it.

Effort routing on existing subagents

  • parity-checker, i18n-syncer → effort: low (already model: haiku)
  • contract-aligner, security-reviewer, design-reviewer → effort: medium
  • design-reviewer also pinned model: sonnet — it carried no model and would have inherited the Opus driver, defeating the routing for an auto-firing reviewer (matches its two effort-tier siblings).

Audit fan-outs → dynamic workflows (the biggest single quota win)

  • .claude/workflows/audit.mjs — one generic engine keyed by kind: a Haiku finder per surface (reads that audit's rubric/checklist) → a Haiku default-refuted skeptic per serious finding → a completeness-critic loop-until-dry → returns verified structured findings + coverage + deferred.
  • .claude/workflows/prod-readiness.mjs — runs the four audit workflows in parallel (one-level workflow() nesting) + the ops-layer fan-out (observability / multi-instance / background-durability / staging; backups always Deferred) + ops skeptic verify.
  • /audit-{security,tests,performance,code-quality} + /prod-readiness become thin non-fork drivers: resolve scope → invoke Workflow → Opus synthesizes the report from the compact return. Each keeps a headless/CI inline (Explore) fallback since the Workflow tool is absent from CI tool-allowlists (only /pr-review runs in CI, via Task).

The whole multi-phase fan-out/verify/loop reasoning the Opus context: fork used to do is now free deterministic JS; Opus spends only on synthesis, every leaf agent is Haiku.

Validation

  • Both workflow scripts full-syntax-validated (the node --check "illegal return" is a false positive — the Workflow runtime allows top-level return).
  • Runtime mechanics smoke-tested via a 1-agent probe workflow: agent({model:'haiku', effort:'low', schema}) returns a schema-validated object with file-tool access. ✅
  • Live end-to-end fires on the next real /audit-* invocation.

Paired PR (orbit-api): thomasluizon/orbit-api#312 — the lockstep effort: medium on the shared review agents.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Nsmhto1amnzXEmbSvuJgw9

Route models at the subagent/workflow tier (locked: no opusplan, Opus stays the
session driver; model routing lives entirely below the driver).

Effort routing on existing subagents:
- parity-checker, i18n-syncer -> effort: low (already model: haiku)
- contract-aligner, security-reviewer, design-reviewer -> effort: medium
- design-reviewer also pinned model: sonnet: it carried no model and would have
  inherited the Opus driver, defeating the routing for an auto-firing reviewer.

Audit fan-outs -> dynamic workflows (the biggest single quota win):
- new .claude/workflows/audit.mjs: one generic engine keyed by kind. Haiku finder
  per surface -> Haiku default-refuted skeptic per serious finding -> completeness
  loop-until-dry -> returns verified structured findings.
- new .claude/workflows/prod-readiness.mjs: runs the four audit workflows in
  parallel (one-level nesting) + ops-layer fan-out + ops skeptic verify.
- /audit-{security,tests,performance,code-quality} + /prod-readiness become thin
  non-fork drivers: resolve scope -> invoke Workflow -> Opus synthesizes the report
  from the compact return. Each keeps a headless/CI inline (Explore) fallback since
  the Workflow tool is absent from CI tool-allowlists.

The multi-phase fan-out/verify/loop reasoning the Opus context:fork used to do is
now free deterministic JS; Opus spends only on synthesis, every leaf agent is Haiku.
Runtime mechanics smoke-tested via a 1-agent probe workflow.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nsmhto1amnzXEmbSvuJgw9
@vercel

vercel Bot commented Jul 9, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
orbit-ui-mobile-web Ignored Ignored Jul 9, 2026 7:03pm

Request Review

@sonarqubecloud

sonarqubecloud Bot commented Jul 9, 2026

Copy link
Copy Markdown

@thomasluizon
thomasluizon merged commit df40137 into main Jul 9, 2026
19 checks passed
@thomasluizon
thomasluizon deleted the chore/harness-stage-3-model-effort-routing branch July 9, 2026 19:08

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: PR #437 — "chore(harness): Stage 3 model/effort routing + audit workflows"

Recommendation: Request changes

Severity Count
Critical 0
High 2
Medium 1
Low / Info 0

The diff only touches .claude/ harness tooling (5 subagent frontmatter files, harness-upgrade-plan.md, 5 SKILL.md rewrites, and 2 new .claude/workflows/*.mjs files) — no apps/*, packages/shared, or orbit-api code, so all five gated review subagents (parity-checker, i18n-syncer, contract-aligner, security-reviewer, design-reviewer) are N/A, and dimensions 8–14 of the rubric are N/A. Lint/type-check/test are also N/A — nothing in this diff falls under any workspace's turbo run lint/type-check/test target.

Findings

[HIGH] .claude/workflows/audit.mjs's isApiSurface/isUiSurface classifier silently drops 4 of the 6 security surfaces (injection, secrets-config, ratelimit-ai-abuse, error-web-auth) whenever /audit-security (or /prod-readiness) is scoped to api/backend, because those surfaces' label isn't prefixed api- and their where text never contains the literal substring "orbit-api" — unlike the performance/tests/code-quality KIND configs, which all use the api- prefix consistently. An api-scoped security audit — the most common invocation for backend hardening — silently reports a much narrower posture than it claims (misses secrets/CORS-in-Program.cs, SQL/EF injection, and rate-limit/AI-abuse coverage), with no signal that those checklist sections never ran. Fix: tag each surface with an explicit repo field instead of inferring it from a label prefix / substring match on free text.

[HIGH] The entire Stage-3 mechanism (Workflow({...}) calls added to all five SKILL.md files, plus the two new .mjs files) depends on a Workflow tool that could not be confirmed as present in this harness build — a search for it turned up nothing in either the main tool list or the deferred-tool index. Both new .mjs files also end in a bare top-level return, which is a SyntaxError under normal Node/ESM — the PR's own commit message concedes node --check flags this and calls it "a false positive" on an unverified assumption, and separately admits "Live end-to-end fires on the next real /audit-* invocation." Each skill does carry a documented inline-fallback for when Workflow is unavailable, so this degrades gracefully rather than breaking outright — but the PR checks off "biggest single quota win" in harness-upgrade-plan.md on a mechanism that has never actually been run to completion.

[MEDIUM] .claude/skills/audit-code-quality/SKILL.md:38 says "run the fan-out inline per Phase 3's fallback," but in this file's new numbering the fallback text lives in Phase 2 ("Run the audit workflow"); Phase 3 is "Apply the rubric." Looks like copy-paste drift from audit-tests/SKILL.md, where Phase 3 genuinely is the right phase. One-line fix.

What's good

  • The effort:/model: frontmatter additions on the five subagents are correct and minimal — design-reviewer picking up model: sonnet alongside effort: medium is the right catch (it previously had no model: and would have inherited the Opus driver).
  • Collapsing five near-duplicate fan-out/verify/loop SKILL.md sections into one shared audit.mjs engine is a real DRY win once the surface-classification bug is fixed.
  • The plan doc is honest that end-to-end verification is still pending, rather than overclaiming.

Notes

  • Cross-repo dimensions (contract-aligner, orbit-api side of the backward-compat guard) are not applicable — this diff touches no API contract or code.
  • A stray scratch file (_review_audit.mjs, created during syntax verification of the return-at-top-level claim) could not be removed in this sandbox — please delete it manually if it shows up untracked at repo root.

thomasluizon added a commit that referenced this pull request Sep 7, 2026
…#437) (#847)

* fix: verify wake source process identity for ticket 437

* fix: verify wake identity at the final sleep decision

* fix(tools): reject all Linux dead wake-source states (#437)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant