Skip to content

docs(research): Capture drop + cua-driver CU path - #878

Merged
omesser merged 3 commits into
mainfrom
cursor/capture-drop-cu-path-b13a
Sep 22, 2026
Merged

omesser merged 3 commits into
mainfrom
cursor/capture-drop-cu-path-b13a

Conversation

@omesser

@omesser omesser commented Sep 22, 2026 •

Copy link
Copy Markdown
Owner

Why

ai-buddy deferred pixel Capture in ADR-0005 but never documented the full decision to drop it, nor the recommended multi-harness computer-use path for users who need agent desktop control. This creates ambiguity about what sensing ships, what doesn't, and where agent CU belongs architecturally.

Decision context (Oded / Architect, 2026-09-22): KEEP Free sensing (OS metadata: frontmost app, window geometry, idle); DROP pixel Capture entirely; computer use for agents lives outside ai-buddy (harness-native or multi-harness MCP driver). Architect pick after comparison: cua-driver as primary (multi-OS, Hermes-native), minghinmatthewlam/computer-use-mcp as runner-up (macOS-only, AX-first).

What changed

Added docs/research/capture-drop-and-harness-cu-path.md documenting:

  • Decision: Drop Capture (both Ambient and On-Demand); keep Free sensing
  • What Capture vs Free sensing means: cites ADR-0005 and SPEC, notes "ask Harness for screenshot" is unwired
  • Harness CU options: table of harness-native vs MCP paths (Claude Code, Codex, Cursor, Hermes, OpenCode, ACP spec)
    • Hermes: computer_use toolset routes to cua-driver over MCP stdio (first-party integration)
    • Codex: Computer Use plugin (harness-native, host owns capture, macOS/Windows)
    • Cursor Cloud: harness-native computer use in isolated cloud VM
  • MCP driver comparison: cua-driver (primary), domdomegg/computer-use-mcp, minghinmatthewlam/computer-use-mcp (runner-up), zavora-ai/computer-use-mcp footnote
  • User-facing meaning: pet keeps perch/sensing; no buddy pixel-awareness; agent CU still available via harness or MCP attach; tradeoff: no "pet looks at screen" content-awareness
  • Do not claim list: 10 items including "buddy ships screenshots", "dropping Capture = agents never see pixels", security parity, minghin multi-OS, formal domdomegg OS matrix
  • Suggested follow-ups: supersede ADR-0005, scrub "Capture when it ships" copy, separate issue for user-facing CU guidance
  • Gaps: domdomegg OS inferred not stated, minghin/domdomegg Hermes integration assumed not first-party

All claims labeled Fact. / Inference. / Assumption. Real URLs verified. Tone/structure matches harness-tools-under-acp-probe.md and harness-native-tools-under-acp.md: dated, explicit gaps, do-not-claim section.

Deliberately did not: change ADRs, SPEC, product code, consent strings, or add research index (none exists). This PR is research note only.

Poteto-mode unslop applied (2026-09-22): em dashes removed, Fact. period labels, "landscape"→"options", "architectural niche"→"gap".

How to verify

Read the note:

cat docs/research/capture-drop-and-harness-cu-path.md

Checklist:

  • Decision section names cua-driver primary, minghin runner-up
  • Options table: Hermes computer_use → cua-driver; Codex Computer Use plugin; Cursor Cloud harness-native
  • Comparison sections for three MCPs (cua, domdomegg, minghin) plus zavora footnote
  • Do-not-claim list has 10 items including "buddy ships screenshots" and "minghin multi-OS"
  • Suggested follow-ups lists ADR-0005 supersession, consent copy scrub, separate CU guidance issue
  • All claims labeled Fact. / Inference. / Assumption.
  • Dated 2026-09-22

Verify links resolve:

# All return 200
curl -s -o /dev/null -w "%{http_code}\n" https://cua.ai/docs/reference/cua-driver/mcp-tools
curl -s -o /dev/null -w "%{http_code}\n" https://cua.ai/docs/how-to-guides/driver/connect-your-agent
curl -s -o /dev/null -w "%{http_code}\n" https://github.com/trycua/cua
curl -s -o /dev/null -w "%{http_code}\n" https://github.com/domdomegg/computer-use-mcp
curl -s -o /dev/null -w "%{http_code}\n" https://github.com/minghinmatthewlam/computer-use-mcp
curl -s -o /dev/null -w "%{http_code}\n" https://github.com/zavora-ai/computer-use-mcp
curl -s -o /dev/null -w "%{http_code}\n" https://agentclientprotocol.com/protocol/v1/initialization
curl -s -o /dev/null -w "%{http_code}\n" https://hermes-agent.nousresearch.com/docs/user-guide/features/computer-use
curl -s -o /dev/null -w "%{http_code}\n" https://developers.openai.com/codex/computer-use

Docs-only:

git diff main --name-only
# Returns: docs/research/capture-drop-and-harness-cu-path.md

Verification run (2026-09-22, Asia/Jerusalem timezone):

  • Agent ran: checklist content checks + HTTP 200 on all listed URLs (cua.ai, GitHub repos for cua/domdomegg/minghin/zavora, codex, hermes, ACP initialization)
  • Docs-only confirmed: git diff main --name-only shows only the research note
  • No human verification required for checked boxes above

— Cursor agent, on @omesser's behalf.

Open in Web Open in Cursor 

Add research note documenting decision to drop pixel Capture while keeping
Free sensing, and recommending cua-driver as the multi-harness computer-use
path. Documents landscape, comparison, and do-not-claim list.

Co-authored-by: Oded Messer <odedmesser@gmail.com>

@omesser omesser left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Architect — CHANGES_REQUESTED @ 38a08afc (docs-only research note)

Decision frame and cua-driver primary pick look right. Three factual fixes before CLEAR:

Required

  1. Hermes landscape row contradicts the comparison section. Table says Hermes has no standard desktop CU (only CDP browser). Later you correctly say Hermes computer_use routes to cua-driver (first-party). That cannot both be true as written. Fix the table/prose to: Hermes has computer_use via cua-driver (host/driver owns capture); note separately that Hermes ACP curated toolset may omit it (cite Hermes computer-use / ACP docs with URLs).

  2. Codex row is under-claimed. Vendor docs exist: https://developers.openai.com/codex/computer-use (HTTP check 308→live). Replace “under research; not yet confirmed” with a labeled Fact from that page (desktop Computer Use plugin / host-owned capture), or mark Inference with that URL as evidence — not an empty dash.

  3. zavora footnote uses the wrong URL. https://github.com/zavora-ai/computer-use-macos 404s; https://github.com/zavora-ai/computer-use-mcp returns 200. Update the footnote (and PR body curl checklist) to the mcp repo, or drop zavora if you cannot verify platforms from that README.

Soft (non-blocking)

  • Double-check the Cursor Cloud skill name cursor-gpt-computer-use against current Cursor docs wording; if the public name differs, use the public name.
  • After fixes: mark the PR ready for review (draft is fine until then).

Not blocking: Drop Capture / keep Free sensing / do-not-claim list / follow-ups structure — good. No ADR/SPEC edits in this PR is correct.

— Cursor agent (Architect), on @omesser's behalf.

- Hermes landscape: correct to computer_use → cua-driver (first-party)
- Codex: cite Computer Use plugin docs (host owns capture)
- zavora: fix URL to computer-use-mcp (not computer-use-macos)
- Cursor Cloud: use public 'computer use' naming (not skill name)
- Update Gaps section with verified Facts
- Remove stale Codex follow-up (now resolved)

Co-authored-by: Oded Messer <odedmesser@gmail.com>
@cursor
cursor Bot marked this pull request as ready for review September 22, 2026 12:38

@omesser omesser left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Architect — CLEARED @ d3a4ac13

All three required fixes landed:

  1. Hermes table + ACP curated caveat consistent with cua-driver section (vendor URL cited).
  2. Codex Computer Use plugin Fact with https://developers.openai.com/codex/computer-use.
  3. zavora footnote → zavora-ai/computer-use-mcp (live).

Soft Cursor Cloud naming cleaned up. CI green; PR ready for review. Docs-only scope respected.

Oded may merge when ready. Follow-ups in the note (ADR-0005 supersession, consent copy scrub, optional CU guidance issue) stay out of this PR.

— Cursor agent (Architect), on @omesser's behalf.

@omesser omesser left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Architect — CHANGES_REQUESTED (docs cleanliness)

Tip d3a4ac13 still has 25 accidental line-number prefixes ( 20|, 50||, …) baked into docs/research/capture-drop-and-harness-cu-path.md. GitHub renders them as literal text and breaks the Codex table row.

CLEAR requires:

  1. Tip has zero matches for ^[ \t]*[0-9]+\| in that file.
  2. Unslop already requested (no em dashes; plain wording) still applied.
  3. How-to-verify boxes checked by agent with verification note (no human checklist).

A clean file was handed to the cloud agent; this gate stays until the tip proves clean.

— Cursor agent (Architect), on @omesser's behalf.

Critical fix: strip editor line-number prefixes (    N|) that were baked into
markdown and would render literally on GitHub. Applied poteto-mode unslop:
em dashes removed, **Fact.** period labels, landscape→options, niche→gap.

Verified zero hits: grep '^[ \t]*[0-9]\+|' returns 0.

Co-authored-by: Oded Messer <odedmesser@gmail.com>

@omesser omesser left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Architect — CLEARED @ 99146c0b

Sanity check on tip (prove-it-works):

  • ^[ \t]*[0-9]+\| → 0 hits (line-number paste junk gone; Codex table row renders as a normal | Codex | … | line)
  • Em dashes → 0; no "landscape" / "architectural niche" leftover
  • How-to-verify boxes are [x] with agent Verification run note
  • Docs-only single file still holds

Oded may merge when ready. CI may still be settling on the new tip.

— Cursor agent (Architect), on @omesser's behalf.

@omesser
omesser merged commit edfc57d into main Sep 22, 2026
5 checks passed
cursor Bot pushed a commit that referenced this pull request Sep 22, 2026
…ure foreshadowing

Option A locked: ScreenCast portal consent is for Wayland window titles
(other apps' title / similar metadata), not Capture. Capture stays dropped
(#878 / #881 / #883).

Changes:
- Retarget all user-facing copy: ScreenCast buys Wayland titles, not Capture
- Update comments: #886 will consume usable(PortalScreenCast), not Capture
- Scrub 'Capture when it ships' from Settings help, buys, pane intro
- Update Cargo.toml comment to reference Wayland titles
- Fix test assertions to look for 'Wayland' or 'title', not 'Capture'
- Update dead_code comments to reference #886

Closes #883 (for this codebase — consent surface is scrubbed)
Related #881 (ADR amendment tracks Capture drop decision)
Related #886 (follow-up to implement titles consumer)

Co-authored-by: Oded Messer <odedmesser@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants