Skip to content

feat(computer-use): 2.0 background delivery, activation, overlay, harness - #315

Merged
Tryanks merged 4 commits into
mainfrom
feat/computer-use-2
Aug 31, 2026
Merged

feat(computer-use): 2.0 background delivery, activation, overlay, harness#315
Tryanks merged 4 commits into
mainfrom
feat/computer-use-2

Conversation

@Tryanks

@Tryanks Tryanks commented Aug 31, 2026

Copy link
Copy Markdown
Owner

Computer Use 2.0 — background input delivery (macOS)

Follows the techniques in pi-computer-use and kwwk-computer-use-core. The computer-use tools now drive target apps without stealing focus or moving the real cursor.

What changed

  • Background delivery (backend/macos/background.rs): mouse/keyboard/scroll/drag fallbacks are addressed to the target process via CGEventPostToPid with the required window-addressing fields (pid 39, window 51, routing 58, mouse window 91/92) and optional SkyLight window-local coordinates. AX-first (AXPress/AXValue) behavior is unchanged.
  • Background activation without a foreground switch: per-PID event taps drop the focus messages (raw types 13/19/20) headed to the previously-frontmost app, plus an appKitDefined primer (subtype 1 on acquire / 2 on release) and a window-center primer. The user's frontmost window stays put.
  • Chromium/Electron AX activation (ax.rs): sets AXManualAccessibility/AXEnhancedUserInterface and registers an AXObserver so background Chromium exposes its full tree (private _AXObserverAddNotificationAndCheckRemote with public fallback).
  • Agent-cursor overlay (backend/macos/overlay/): a click-through, non-activating NSPanel cursor + colorful window halo showing where the agent acts (never moves the real pointer; all UI marshalled to the main queue; no-op when disabled).
  • Screenshots: scaled (long-edge 1568 / area cap) and JPEG-encoded (q≈0.8).
  • Structured outcomes: per-step delivery (ax/background_pid/foreground_hid/none) and transaction-level activation.
  • Per-PID scheduling lanes: different apps run concurrently; same-app ops serialize. Existing epoch staleness rejection preserved.
  • Harness annotations: <computer_use_harness> block with observation sequence, recent actions, candidate targets, and stable-label deltas (since previous / initial).
  • Settings (Computer Use page): show_agent_cursor (default on) and allow_foreground_fallback (default off — keyboard-only, never pointer), with en/zh-CN i18n.

Every private API (SkyLight, per-PID taps, appKitDefined, private AX observer symbol) degrades gracefully — missing symbol / nil / tap-create failure falls back or reports didnt; nothing panics.

Testing

  • Local CI parity green: cargo fmt --all --check, cargo clippy --workspace --all-targets --locked -- -D warnings, cargo test --workspace --locked, process-helper boundary check, --locked (no Cargo.lock drift).
  • Live functional test on a real backgrounded app (TextEdit, with Finder frontmost): background keyboard input landed in the document AND the frontmost app was unchanged (1297 → 1297); background pointer click likewise kept focus. Both reported delivery: background_pid.

Notes / limitations

  • macOS only; Windows/Linux input paths are unchanged (Windows still advertises PNG screenshots).
  • The overlay's on-screen rendering isn't exercised by the headless probe (needs the app's main run loop); it is observational-only and no-ops on any failure.

@Tryanks
Tryanks merged commit cab63b0 into main Aug 31, 2026
3 checks passed
@Tryanks
Tryanks deleted the feat/computer-use-2 branch August 31, 2026 20:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant