Skip to content

Make the sprite touchable: Grab, Throw and Poke - #49

Merged
omesser merged 1 commit into
mainfrom
feat/6-grab-throw-poke
Aug 29, 2026
Merged

omesser merged 1 commit into
mainfrom
feat/6-grab-throw-poke

Conversation

@omesser

@omesser omesser commented Aug 29, 2026

Copy link
Copy Markdown
Owner

Three of the five verbs existed in the Engine, with tests, and nothing produced them. WorldSnapshot.verbs was empty on every real tick, so the sprite could be watched and not touched.

Click it, pick it up, drag it, throw it.

The verbs come from polling, not the webview

That is what makes a drag survive.

  • CGEventSourceButtonState asks the window server whether the button is down. A state query, not an event tap, so no Accessibility grant — the same trade sensing already makes to read idle time without reading input.
  • The frame loop reads it beside the cursor position it takes anyway.
  • Routing through the webview would have failed where The five interaction verbs #6 warned: the overlay is click-through wherever the sprite is not drawn, so a drag that outran the art would stop delivering events mid-gesture.
  • A held Grab now suspends the hit-test, so the button keeps reaching the overlay instead of the app underneath.

A press stays ambiguous until it ends

Deciding on the press would snap the sprite to the cursor and drop it on every click.

Gesture Becomes
press, release without moving Poke
press, move 4pt — or hold 180ms Grab
release while still moving Throw
release after holding still drop

A test asserts a click emits no Grab at any point, hand-drift included.

Poke plays react

#6 left this to whichever of it and #8 landed second. react was in the Required Animation Set, shipped in the placeholder Character, and selected by nothing — so clicking a wide-awake sprite did nothing at all. Reacting is not a State: it draws over whatever the sprite is doing and changes nothing about where it is.

Two rectangles, one tick

The hit-test now runs twice, because the questions are about different moments:

  • Did this press belong to the sprite? Asks the sprite last drawn — the one the user pressed on.
  • Should the next click reach us? Asks the sprite about to be drawn — a cursor arriving over the art must not spend a frame passing clicks underneath.

Beyond what #6's table asks — say the word and they go

  • Pressing and holding still picks the sprite up, rather than requiring movement first.
  • A release only throws above a threshold speed. Nobody holds a mouse perfectly still, and testing for zero movement made every release a fling.

Not here

Menu waits on #18 (which waits on #12, #13); Summon needs the chat surface in #17. No Verb variant was added for Summon — nothing produces or consumes one, and an unreachable variant is the speculative generality this repo's reviews keep removing.

Verify

cargo test                    # 108 core + 18 shell
./scripts/verify-overlay.sh   # click-through still passes both ways

Then by hand, README manual checks 9–11. Exercised here against a real window server with synthetic events, not only in tests:

Gesture Observed
click verbs: [Poke], 32 frames of react
drag Grab on every tick held, 20 ticks of Dragged
flick Throw { velocity: (-1389, -1534) } — matching the drag
untouched 20s zero verbs

Part of #6

🤖 Generated with Claude Code

Three of the five verbs existed in the Engine, with tests, and nothing produced
them. `WorldSnapshot.verbs` was empty on every real tick, so the sprite could be
watched and not touched. This closes that half.

The verbs come from polling rather than from the webview, which is what makes a
drag survive. `CGEventSourceButtonState` asks the window server whether the
button is down — a state query and not an event tap, so it needs no
Accessibility grant, the same trade `sensing` already makes to read how long ago
the last input was without reading the input. The frame loop reads it beside the
cursor position it takes anyway.

Routing it through the webview would have failed exactly where #6 warned. The
overlay is click-through wherever the sprite is not drawn, so a drag that
outran the art would stop delivering events mid-gesture. Polling does not care
where the cursor is, and a held Grab now suspends the hit-test as well, so the
button keeps reaching the overlay instead of the application underneath.

A press stays ambiguous until it ends. Deciding on the press would snap the
sprite to the cursor and drop it on every click, so a press becomes a Grab only
once the hand has moved four points or held for 180ms, and a press that ends
before either is a Poke. A test asserts that a click emits no Grab at any point,
drift included.

Poke plays `react`, which #6 left to whichever of it and #8 landed second. The
Animation was in the Required Animation Set, shipped in the placeholder
Character, and was selected by nothing, so a click on a wide-awake sprite did
nothing at all. Reacting is not a State: it is drawn over whatever the sprite is
doing and changes nothing about where it is.

The hit-test now runs twice a tick against two rectangles, because the questions
are about different moments. Whether a press belongs to the sprite is about the
sprite the user pressed on, so it asks the one last drawn. Whether the next
click should reach us is about the sprite about to be drawn, so it asks this
one — a cursor arriving over the art must not spend a frame passing clicks
underneath.

Beyond what #6's table asks for, and easy to remove if unwanted: pressing and
holding still picks the sprite up, rather than requiring movement first. And a
release only throws above a threshold speed, because nobody holds a mouse
perfectly still and testing for no movement at all made every release a fling.

Menu and Summon are not here. #6 sequences Menu behind #18, and Summon needs the
chat surface in #17. No `Verb` variant was added for Summon: nothing produces or
consumes one, and an unreachable variant is the speculative generality this
repo's review keeps removing.

Verified against a real window server, not only in tests: a synthetic click
produces one Poke and 32 frames of `react`, a drag produces a Grab on every tick
it is held, and a flick produces a Throw at the speed the hand was moving.

Part of #6

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@omesser

omesser commented Aug 29, 2026 •

Copy link
Copy Markdown
Owner Author

Losing the Character between displays — fixed in cf722d9.

The cause was not the cursor conversion. Measured on this machine:

  • app.cursor_position() is NSEvent.mouseLocation (global points) multiplied by the primary display's scale, so dividing by that same factor recovers points exactly — on any DPI mix. The arithmetic was already right, and the trace confirmed it: the Character followed the cursor onto the second display and came to rest on its floor at pos(2600,1117).
  • What lost it was the window. macOS gives each display its own Space and draws a window spanning two of them on only one. Confirmed independently of ai-buddy with a bare NSPanel spanning both displays and an opaque square over each: only the square on the first display rendered, at floating level and at screen-saver level alike. The union-sized overlay was drawing the Character where nobody could see it.

The fix

  • The overlay covers one display and the frame loop moves it to whichever display the Character's feet are on (feet, not cursor, so a Throw crosses too).
  • overlay::display_for picks it — the containing display, or the nearest one when the point sits in a gap between displays or past the outermost edge.
  • display_union is gone; window_source::in_points replaces the per-display conversion inside it and usable_frame now shares it.
  • The per-tick primary_monitor() call is gone. It read NSScreen from the frame loop, which only the main thread may do; the cursor's scale factor now comes off the same cached main-thread read as the display frames.
  • Physics is untouched — still one point space across every display. Physics: true union bounds and cross-display sizing #4 is not touched.

Verified

  • cargo clippy --all-targets -- -D warnings, cargo test (128), node --test tests/*.test.js (6), pre-commit run --all-files — all pass.
  • scripts/verify-overlay.sh — all machine checks pass, including the check rewritten from "window spans the display union" to "window covers exactly one display" (origin included, so a right-sized window on the wrong display fails).
  • Synthetic drag from (960,950) on display 1 to (2600,500) on display 2, over 10,328 traced frames: the overlay was on the Character's display in every frame but one — the single 16ms tick at the crossing, which is the asynchronous window move landing. Window bounds afterwards were exactly 1728x1117 at (1920,0), and the sprite's window-local position (616,989).

Known and left alone

  • The sprite is clipped while it straddles the seam — one window cannot be on two displays. Marked ponytail:; one overlay per display is the upgrade.
  • Screenshots either side: before, the Character's own reported position on the second display showed bare wallpaper; after, Blip is there on screen where the trace says he is.

@omesser

omesser commented Aug 29, 2026

Copy link
Copy Markdown
Owner Author

The display work has moved out of this branch, to #56 (issue #55).

This PR is back to exactly the commit that was reviewed, a0bbcc9 — Grab, Throw and Poke, nothing else. Force-pushed, no amend: the commit message is untouched.

#56 is stacked on this branch rather than on main, because a Grab is what carries the Character across a boundary in the first place. Merge this first.

Re-verified on the reset branch: cargo clippy --all-targets -- -D warnings, cargo test (126), node --test tests/*.test.js (6), pre-commit run --all-files — all pass without the display commit.

@omesser
omesser merged commit 9853cca into main Aug 29, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant