From 90c211c0eae3ae2c9465d999008270b0c5ae462b Mon Sep 17 00:00:00 2001 From: omesser Date: Wed, 23 Sep 2026 21:57:06 +0300 Subject: [PATCH 1/2] docs: Say ai-buddy never reads pixels and point computer use at cua-driver Closes #882. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/README.md b/README.md index 9a1cf5ec3..5bbff3bb1 100644 --- a/README.md +++ b/README.md @@ -20,6 +20,7 @@ Pick a Character with an authored personality. The Director chooses idle Behavio - **Reacts to gestures.** Poke, pick up, throw — it arcs, lands, and keeps going. - **Stays out of your way.** Fades for fullscreen, hides on Control-Option-Command-B. Appears in screenshots by default; opt-out available in settings. - **Lives its own life.** Walks, idles, sits, sleeps — even with the Director off. +- **Never reads your screen.** Sensing is window metadata, never pixels: no screenshots, no OCR. An agent that needs to see and act on your desktop gets that from its Harness or an MCP server you attach — see [Computer use](#computer-use). ## See It @@ -223,6 +224,14 @@ Two transports, two distinct axes: **Explicitly not served:** mouse/keyboard/Executor tools (ADR-0003). No click, no type, no input events by design. +### Computer use + +ai-buddy never reads screen pixels. Sensing is OS window metadata — owner, title, bounds, frontmost app, idle — so `describe_screen` describes the window layout, not what is on screen. The buddy takes no screenshots, runs no OCR, and embeds no vision model for desktop content (ADR-0031). The "Appear in screenshots and screen shares" setting is the other direction: whether the *sprite* shows up in captures you take. + +That bounds ai-buddy's own code, not the agent you attach to it. An agent that needs to see and act on your desktop still can — the capability comes from the Harness itself, or from a computer-use MCP server you attach to the Harness, never through ai-buddy, whose MCP serves no input events. + +The portable option across the Harnesses above is [cua-driver](https://github.com/trycua/cua) (MIT; macOS, Windows, Linux), attached over stdio MCP. [Connect your agent to Cua Driver](https://cua.ai/docs/how-to-guides/driver/connect-your-agent) carries the per-client registration, and [MCP tools](https://cua.ai/docs/reference/cua-driver/mcp-tools) lists what it exposes. Attach it deliberately — it drives the real desktop with your signed-in sessions, and its permission mode is chosen by the process that owns the driver runtime, not by the agent asking. Some Harnesses bring computer use of their own instead; the [Capture decision note](./docs/research/capture-drop-and-harness-cu-path.md) has the per-Harness table and the other drivers surveyed. + ## Platform Support What works today on each OS. Degraded and stub mean reduced or no-op — supported honesty, not a crash. From da8f459078c4dfaf49b0fd6f0d1bf5fc7b1e9f0c Mon Sep 17 00:00:00 2001 From: omesser Date: Thu, 24 Sep 2026 00:03:02 +0300 Subject: [PATCH 2/2] docs: Drop the ADR number from the user-facing pixels line ADRs are internal. A README reader has no use for the number and no reason to follow it. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 5bbff3bb1..578df7404 100644 --- a/README.md +++ b/README.md @@ -226,7 +226,7 @@ Two transports, two distinct axes: ### Computer use -ai-buddy never reads screen pixels. Sensing is OS window metadata — owner, title, bounds, frontmost app, idle — so `describe_screen` describes the window layout, not what is on screen. The buddy takes no screenshots, runs no OCR, and embeds no vision model for desktop content (ADR-0031). The "Appear in screenshots and screen shares" setting is the other direction: whether the *sprite* shows up in captures you take. +ai-buddy never reads screen pixels. Sensing is OS window metadata — owner, title, bounds, frontmost app, idle — so `describe_screen` describes the window layout, not what is on screen. The buddy takes no screenshots, runs no OCR, and embeds no vision model for desktop content. The "Appear in screenshots and screen shares" setting is the other direction: whether the *sprite* shows up in captures you take. That bounds ai-buddy's own code, not the agent you attach to it. An agent that needs to see and act on your desktop still can — the capability comes from the Harness itself, or from a computer-use MCP server you attach to the Harness, never through ai-buddy, whose MCP serves no input events.