Skip to content

Name the remaining major Harnesses, one smoked turn at a time #457

Closed

Description

@omesser

Problem

ADR-0017's launch table has three named rows and an escape hatch. Only two of
those rows have ever completed a turn:

Name Standing today
claude Verified 2026-09-07, fresh and resumed
hermes Verified 2026-09-07 fresh; resumed is #448
opencode Unverified — not installed on the machine #433 or #434 was written on
anything else Works, as typed, with no row and no name

The ADR also names three protocol-compatible Harnesses that are reachable
today through the escape hatch and have simply never been smoked:

  • Grok Build — grok agent stdio
  • GitHub Copilot CLI — copilot --acp --stdio (public preview)
  • Gemini CLI — gemini --acp

"Reachable through the custom value" is not the same as supported. A user who
types one of those gets no row, no documentation, and no evidence anyone has
run it. That is the gap this closes.

The bar is already written

ADR-0017 sets it: "A row moves in this table when that command exits zero."
scripts/probe-harness.sh attaches the configured Harness with no overlay,
runs one fixed prompt, and prints the handshake, the stop reason, the reply,
and whether the reply parsed as a Behavior proposal.

So this is not a design task. It is: install each Harness, run the probe,
record what initialize advertised, and move the row.

Scope

For each of opencode, Grok Build, GitHub Copilot CLI and Gemini CLI:

  1. Run scripts/probe-harness.sh and get exit zero on a fresh session.
  2. Run it again on a resumed session, because that is where hermes broke
    (fix(harness): A session/load that succeeds without restoring pins every turn to a dead session #448) and where the fake agent's error-shaped refusal hid the bug.
  3. Record what its initialize advertises — loadSession,
    mcpCapabilities.http, authMethods, sessionCapabilities,
    promptCapabilities — because ADR-0017 already shows these differ enough
    to matter, and Attaching real harnesses: per-harness MCP client config for Claude Code, opencode, Hermes, Pi, and Grok #166 depends on the MCP half.
  4. Give it a named row in the launch table with its command and standing.

A Harness that fails gets a row too, saying so. An honest "does not work yet,
here is what it did" is worth more than absence.

Constraints from ADR-0017, not to be relitigated here

  • No auto-approve flags, ever. None of the always-approve, yolo or
    setSessionMode flags is passed by default. The Chat surface owns
    permissions.
  • Gemini needs care. It has open issues about stdout pollution corrupting
    its NDJSON. Smoke it before believing it.
  • Pi stays deferred. It has no first-party ACP and the research says it
    wants a second adapter, not a compromise. Reopening that is a separate
    argument.
  • A vendor extension we do not know (x.ai/* and the like) gets
    method-not-found, which the protocol allows.

Also unproven, and worth folding in

Step 5 of #434 never ran: every probe found no ai-buddy-mcp binary beside the
app and had AI_BUDDY_MCP_BIN unset, so no session ever got an MCP server and
no Harness ever called the speak tool. The stdio server is built and tested
on its own; that it is reachable through a session is still untested for
every Harness including the two verified ones.

Acceptance

Related

#166 (per-harness MCP client config), #448 (resumed-session bug), #434
(closed — the first verification pass), ADR-0017.

🤖 Generated with Claude Code

Activity

  1. added this to the V1 milestone on Sep 7, 2026
  2. omesser commented on Sep 8, 2026

    @omesser
    OwnerAuthor

    Codex named row shipped in #556 (merge 087b55e). Per that PR / #554, the smoked-turn gate for Codex is still outstanding (AI_BUDDY_HARNESS=codex scripts/probe-harness.sh). Leaving this epic open.

    — Cursor agent (Coder), on @omesser's behalf.

  3. added
    p1Next up: the default for work that should happen
    on Sep 9, 2026
  4. self-assigned this
    on Sep 9, 2026
  5. omesser commented on Sep 9, 2026

    @omesser
    OwnerAuthor

    Taking the next named-row slice: smoke opencode (installed here; still unmarked in the launch table). Copilot CLI and Gemini CLI are not on this machine; Grok Build can follow in a later turn.

    — Cursor agent (Coder), on @omesser's behalf.

  6. omesser commented on Sep 9, 2026

    @omesser
    OwnerAuthor

    opencode smoke (2026-09-09)

    AI_BUDDY_HARNESS=opencode scripts/probe-harness.sh twice. First attach was fresh (the probe dir still named a claude session, so saved_id did not load). Second attach reused ses_f783d375bffe961nB046ND5aG5. Both exited 0 (end_turn).

    Handshake (both runs)

    • agent: OpenCode
    • loadSession: true
    • mcp http: true
    • mcp handed: ai-buddy --mcp-stdio (the probe does not bind the loopback listener)
    • authMethods: Login with opencode (advertised while signed in; the turn still ran)
    • session: ses_f783d375bffe961nB046ND5aG5

    Turn (fresh and resumed)

    • prompt: Reply with exactly this one line and nothing else: Wave | Hello from the probe.
    • stop: end_turn
    • reply parsed as a Behavior proposal: Wave | Hello from the probe.

    speak was not called; the probe line came back as the turn text. MCP was the stdio shim cargo run already hands, not a separately built ai-buddy-mcp. No code change: attach, resume, and parse all worked.

    Standing is in README and ADR-0017's opencode cell. Grok Build, Copilot CLI, and Gemini CLI are still unverified.

    — Cursor agent (Coder), on @omesser's behalf.

  7. omesser commented on Sep 9, 2026

    @omesser
    OwnerAuthor

    What initialize advertises, for acceptance bullets 3 and 4

    Two of the five fields this issue names — sessionCapabilities and promptCapabilities — have no column in the README table and do not belong in an ADR (docs/agents/docs.md keeps verification detail on the Issue or PR). Recording them here so #166 can read them without digging through a merged PR.

    Read by hand off one initialize request per Harness, with no session/new and no flags. loadSession, mcpCapabilities and authMethods are also in the README capability table; the last two columns are only here.

    loadSession mcpCapabilities sessionCapabilities promptCapabilities agentInfo
    opencode 1.18.30 true http, sse close, fork, list, resume embeddedContext, image OpenCode 1.18.30
    Grok Build 1.0.25 true http, sse list, resume, close embeddedContext; image and audio false none; _meta.agentVersion 1.0.25

    Grok Build also sends vendor extensions under _meta (x.ai/hooks, x.ai/fs_notify, x.ai/capabilities) and an auth capability object. Nothing in ai-buddy reads any of them, and an unknown vendor method gets method-not-found, which the protocol allows.

    Both were smoked with scripts/probe-harness.sh, fresh and resumed, exit zero. #587 has the transcripts and moves the rows; #585 recorded opencode's standing.

    Still open here: GitHub Copilot CLI and Gemini CLI, neither installed on this machine, and the MCP bullet — no Harness has yet called speak through a session, because a probe binds no loopback listener and so hands every Harness the stdio shim.

    — Claude Code agent (Coder), on @omesser's behalf.

    🤖 Generated with Claude Code

  8. moved this from 👷 Todo to 🏗 In Progress in Fidgeton Sep 10, 2026
  9. moved this from 🏗 In Progress to 🏁 Done in Fidgeton Sep 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or requestp1Next up: the default for work that should happen

Projects

  • Status
    Done

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions