Skip to content

fix(verify): probe-harness serves no MCP endpoint, so every tool claim it made is untested #984

Closed

Description

@omesser

The problem

scripts/probe-harness.sh cannot test whether ai-buddy's MCP server reaches a Harness, and it does not say so. Every per-Harness claim in the README's tools table was measured through it.

Why

--probe-harness returns at src-tauri/src/main.rs:2959. mcp_http::serve is called at :3144, during app startup, 185 lines later. So on the probe path mcp_http::endpoint() is None.

choose_mcp at src-tauri/src/harness.rs:1570 then does this:

  • The HTTP branch needs handshake.mcp_http and an endpoint. The endpoint is None, so it is skipped even for a Harness that advertises http.
  • It falls through to stdio, where launch.env = endpoint.map(stdio_env).unwrap_or_default() leaves an empty environment.

--mcp-stdio is a shim that dials the running app (ADR-0026). Without AI_BUDDY_MCP_URL it exits immediately. Run by hand it answers:

{"jsonrpc":"2.0","id":1,"error":{"code":-32603,"message":"AI_BUDDY_MCP_URL is unset: ai-buddy is not running, or this Harness dropped the environment it was given"}}

So every probe hands its Harness a server that cannot start, and the Harness correctly shows no ai-buddy tools.

Why it matters

The probe prints mcp http true next to the stdio shim it actually sent, which reads as though MCP were wired up. It is not.

This produced a wrong conclusion once already. A Goose session reported NO SUCH TOOL for describe_screen, on a resumed session and on a genuinely fresh one, and I read that as Goose ignoring the mcpServers field ACP defines on session/new. It was the probe. Goose would have behaved like any other Harness. That inference reached a PR description and a review before it was caught.

What to change

Have the probe stand up the endpoint before it attaches, so choose_mcp takes the same branch the app takes. mcp_http::serve returns the Endpoint and takes an mpsc::Sender<Call>; the probe needs a receiver that answers calls, which can be as small as dispatching through the same path the app uses.

If that is more than the probe should carry, the alternative is to make the limitation loud: have the probe say that no MCP endpoint is served and that any tool claim is therefore untested. Silence is what caused the wrong conclusion.

The rows this casts doubt on

Anything in the README's per-Harness tools table that asserts something about ai-buddy's MCP arriving or not arriving. README.md:172 says pi "does not receive ai-buddy's MCP". That is tracked separately, but it is the same measurement.

— Claude Code agent, on @omesser's behalf.

Activity

  1. added
    bugSomething isn't working
    ready-for-agentFully specified, ready for an AFK agent
    p1Next up: the default for work that should happen
    on Sep 24, 2026
  2. added this to the V1 milestone on Sep 24, 2026
  3. omesser commented on Sep 24, 2026

    @omesser
    OwnerAuthor

    Hygiene after #986

    #986 merged on main (fc5503f8). Docs only: the Goose tools row now reports a measured MCP delivery (HTTP, seven ai-buddy__* tools listed), and a note under the table points at this issue for the probe's missing endpoint.

    Done (docs): the wrong "not measured" Goose claim is gone; the README now names this issue as the probe limit.

    Still open (this issue): the probe still starts no MCP endpoint. The Goose row reads from a patched local run, not from stock scripts/probe-harness.sh. Until the probe stands up mcp_http::serve (or the loud-probe alternative lands in the script itself), tool claims for other Harnesses remain the same untested shape.

    Leaving this open. #985 still tracks the pi row separately.

    — Cursor agent (Coder), on @omesser's behalf.

  4. omesser commented on Sep 24, 2026

    @omesser
    OwnerAuthor

    The throwaway patch that produced the evidence behind #986, recorded here before I prune the worktree it lived in.

    It does two things to src-tauri/src/harness.rs: stands up mcp_http::serve inside run_probe with a channel that answers every call, and swaps PROBE_PROMPT for a tool-list request. With it, the probe prints mcp http://127.0.0.1:59997/mcp and Goose answers with its own eighteen tools plus the seven ai-buddy__* ones. Without it, endpoint() is None, choose_mcp skips the HTTP branch, and every Harness gets a stdio shim with an empty environment.

    It is not the fix. It logs calls to stderr and answers them with a stub, so it proves delivery and nothing else. The real fix has to decide what the probe answers a tool call with, and whether a probe run should serve the endpoint always or behind a flag.

    diff --git a/src-tauri/src/harness.rs b/src-tauri/src/harness.rs
    index 944e273d..999cfcfe 100644
    --- a/src-tauri/src/harness.rs
    +++ b/src-tauri/src/harness.rs
    @@ -1225,7 +1225,7 @@ impl Completer for Session {
     /// Prompt so a reply that does not parse is the Harness's doing, not the
     /// prompt's. Asked for explicitly rather than left to the Harness.
     const PROBE_PROMPT: &str =
    -    "Reply with exactly this one line and nothing else: Wave | Hello from the probe.";
    +    "List every tool available to you in this session, by name, one per line, and nothing else.";
     
     /// How long shutdown waits for the child to be reaped before saying so.
     const REAP: Duration = Duration::from_secs(2);
    @@ -1243,6 +1243,19 @@ pub fn run_probe() -> i32 {
             eprintln!("probe-harness: {VAR} is unset, so there is no Harness to attach");
             return 2;
         };
    +    {
    +        let (tx, rx) = std::sync::mpsc::channel();
    +        let served = crate::mcp_http::serve(tx);
    +        eprintln!("probe: mcp endpoint served = {}", served.is_some());
    +        std::thread::spawn(move || {
    +            while let Ok(call) = rx.recv() {
    +                eprintln!("probe: MCP CALL RECEIVED -> {}", call.tool);
    +                let _ = call
    +                    .reply
    +                    .send(Ok(serde_json::json!({"probe": "answered", "tool": call.tool})));
    +            }
    +        });
    +    }
         let session = Arc::new(Session::new(
             target.launch,
             target.cwd,

    — Claude Code agent, on @omesser's behalf.

  5. moved this from In Progress to Done in Fidgeton Sep 26, 2026
  6. omesser commented on Sep 26, 2026

    @omesser
    OwnerAuthor

    Shipped on main via #1035 (8588715).

    Done (acceptance met): the probe serves mcp_http::serve before attach, answers calls through the same dispatch the frame loop uses, and prints mcp listed / mcp call. Stock scripts/probe-harness.sh now takes the same MCP branch the app takes. README and docs/DEVELOPMENT.md describe the new lines; the unit test asserts an http-advertising Harness gets the loopback URL.

    Measured on the PR: Goose / Codex / Copilot fetched tools/list over HTTP; Pi was handed stdio and never asked (separate from this fix — pi-acp does not pass session/new servers to pi, see #1019).

    Closing as completed (already closed by the merge keyword).

    — Cursor agent (Coder), on @omesser's behalf.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingharnessp1Next up: the default for work that should happenready-for-agentFully specified, ready for an AFK agent

    Projects

    • Status
      Done

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions