Skip to content

Follow the Character to the display it is on - #56

Merged
omesser merged 3 commits into
mainfrom
feat/55-follow-the-character
Aug 29, 2026
Merged

omesser merged 3 commits into
mainfrom
feat/55-follow-the-character

Conversation

@omesser

@omesser omesser commented Aug 29, 2026 •

Copy link
Copy Markdown
Owner

The overlay now covers one display and follows the Character onto whichever display it is on. Dragging the Character to a second display used to lose it. macOS gives each display its own Space, and a window spanning two Spaces renders on only one, so a union-sized overlay drew the Character on a display that was never composited.

The first diagnosis was wrong

The first diagnosis blamed the cursor conversion: one scale factor applied across two different physical spaces. docs/SPEC.md:513 lists that bug class among the four that earned this repo's testing rule, so it is a fair suspicion. It is not this bug. tao-0.35.3/src/platform_impl/macos/util/mod.rs:101 returns global points multiplied by the primary display's scale factor and nothing else. Dividing by that factor recovers points exactly, on any mix of DPIs. The trace agreed: the Character came to rest at pos(2600,1117) on display 2, over bare wallpaper.

The real cause

A bare NSPanel spanning both displays, with an opaque square over each, rendered only the square on the first display. That held at floating level and at screen-saver level alike. The reproduction uses no ai-buddy code.

What changed

  • The overlay covers one display. Each frame moves it to the display the Character's feet are on. Feet rather than cursor, because a Throw crosses a boundary with nobody holding the Character.
  • overlay::display_for picks that display. It returns the containing display, or the nearest one when the point falls in a gap or past the outermost edge.
  • display_union is deleted. window_source::in_points replaces the per-display conversion inside it, and usable_frame shares it.
  • The per-tick primary_monitor() call is gone. It read NSScreen off the main thread, which is not allowed. The cursor's scale factor now comes from the same cached main-thread read as the display frames.
  • Physics is unchanged: one point space across every display.

Supersedes a written decision

docs/SPEC.md and DESIGN.md specify a union-sized overlay. macOS cannot provide one. Four places move with the code:

Where Was Is
docs/SPEC.md:298 overlay sized to the union of visible display frames overlay covers one display and follows the sprite
DESIGN.md §14 overlay sized to the union union is a physics concept only
DESIGN.md decision 24 overlay spans the union of display frames overlay covers one display; physics spans them all
scripts/verify-overlay.sh window spans the display union window covers exactly one display, origin included

Known limitation

The sprite is clipped while it straddles the seam between two displays, because one window cannot be on both. place_sprite centres the art on the contact point, so the clip is at most half the rendered width, or 64 points for the placeholder Character. The compromise is marked ponytail: in the code, with one overlay per display as the upgrade.

Verify

cargo clippy --all-targets -- -D warnings
cargo test                    # 128
node --test tests/*.test.js   # 6
./scripts/verify-overlay.sh   # all machine checks

verify-overlay.sh passes on a real two-display desktop, including the rewritten bounds check: window covers exactly one display 1920x1080 at (0,0). The old assertion passes on the broken build and fails on the fixed one, so it was replaced rather than added to.

A synthetic drag ran from (960,950) on display 1 to (2600,500) on display 2, over 10,328 traced frames. The overlay was on the Character's display in every frame but one: the 16ms tick at the crossing, while the asynchronous window move lands. Final window bounds were 1728x1117 at (1920,0), sprite window-local at (616,989).

#49 also touches main.rs, platform.rs and README.md. Whichever of the two merges second needs a rebase.

Closes #55

🤖 Generated with Claude Code

@omesser
omesser force-pushed the feat/55-follow-the-character branch from cf722d9 to da80c2e Compare August 29, 2026 19:52
@omesser
omesser changed the base branch from feat/6-grab-throw-poke to main August 29, 2026 19:52
@omesser

omesser commented Aug 29, 2026

Copy link
Copy Markdown
Owner Author

Rebased onto main. This no longer stacks on #49 and can be merged on its own, in either order.

  • The two commits were written one on top of the other but are independent work. Only src-tauri/src/main.rs conflicted, in the import block.
  • Nothing from the verbs work survives here: no input.rs, no pointer.rs, no Pointer in the frame loop, no react, no verbs parameter on assemble.
  • scripts/verify-overlay.sh passes on a real two-display desktop — window covers exactly one display 1920x1080 at (0,0).

#49 and this PR both touch main.rs, platform.rs and README.md, so whichever merges second will want a rebase. Neither contains the other's work.

Comment thread crates/core/src/overlay.rs Resolved
@omesser

omesser commented Aug 29, 2026

Copy link
Copy Markdown
Owner Author

Machine-specific audit

Two defects found, both in scripts/verify-overlay.sh, both fixed in 7839646. The Rust in this PR is clean: every display number it uses is queried, and the arithmetic holds for one display, three, a vertical stack, a negative origin and a Dock on any edge.

Fixed

The props were measured from the wrong rectangle. The harness placed the Perch 200 points and the furniture 60 points below the centre of displays[0]'s whole frame. The app starts the sprite at the centre of that display's usable frame — snapshot::starting_position reads usable_frames.first(). The two differ by half the menu bar's height and half the Dock's width, so the gap between the sprite and the props was whatever this desktop's furniture happened to make it. A Dock on the left edge shifts the sprite's start right by half the Dock's width while the props stay put; wide enough and the sprite starts beside the 400-point Perch instead of over it. Both props now come off the same usable frame the app starts the sprite in.

The drops were fixed points. +200 put the Perch's top edge 240 points above this display's floor, and the prop steps down 80 points during the run. On a display with less usable height the stepped top edge lands past the bottom, where macOS will not place a titled window. The prop would then not move, and a window that moves is noticed within about one poll interval would fail against a frame loop that is working. Both offsets are now fractions of the room between the sprite and the floor: room / 2 for the Perch, room / 6 for the furniture. On this desktop that computes 744 and 585 against the old 740 and 600, so the regression check still means what it meant.

The one assumption left is named in the comment: about 350 points of usable height, enough for the Perch to take its 80-point step and stay on screen. Every display macOS runs on clears that.

window covers exactly one display counted matches. Mirrored displays report the same rectangle, so len(covered) == 1 fails on a mirrored desktop for a correctly placed window. It now accepts any match — the equality against a single display's rectangle is what rules out a union, not the count.

Checked and cleared

  • overlay::display_for — pure arithmetic over reported rectangles. Nearest-display fallback handles gaps between non-flush displays and points thrown outside the outermost edge. No origin, count or ordering assumed. The seam tie-break is documented as arbitrary because both answers are equally true.
  • platform::read_displays — every frame converted by in_points with that monitor's own scale factor, never the primary's. frames and usable_frames are filled in one pass so their indices correspond.
  • Displays::cursor_scale reading primary_monitor() — this is a tao fact, not a machine fact. tao returns the global cursor in points multiplied by the primary display's scale factor whichever display the cursor is over, so the primary's is the only factor that undoes it. Queried, not hardcoded.
  • main::cover_display sizing before positioning — this is what fixes the origin offset the removed ponytail: note described. Growing a window anchors its bottom-left corner, so the old position-then-size order pushed the top edge up by the difference in heights. The general fix, not one tuned to a height.
  • displays[0] in the harness against usable_frames.first() in the app — the same display on any desktop. tao's available_monitors and inspect-window.swift both enumerate CGGetActiveDisplayList, which reports the main display first.
  • window_source::in_points and usable_frame tests, and overlay::display_for tests — literal inputs with expectations computed from them. Inputs, not assumptions. The 1920x1080 beside 1728x1117 fixture is documented as the arrangement the bug was reported on and the code never sees a scale factor.
  • The "numbers this machine reported" comments in window_source.rs — measured evidence, labelled as measured, with nothing depending on them.
  • screencapture -D "$i" — the count comes from the inspector and a failure is reported rather than failed. Not an assertion.
  • The usable floor, elevated prop and desktop furniture checks — all computed from the real desktop the run happens on. The floor check filters candidate displays by the sprite's x, so it holds for a vertical arrangement too.
  • DOCK_LEVEL=20 and layer == 3 — window levels, fixed by macOS, not by this desktop.
  • The #[cfg(target_os = "macos")] modules — one OS, which is the documented project decision, and no layout baked in.

Out of scope, for whoever owns it

src-tauri/src/main.rs reads outer_position() and scale_factor() at the top of the tick but cover_display may move the window later in the same tick. On the crossing frame, place_sprite and cursor_in_window use the previous display's origin and scale, so the sprite is drawn in the wrong place for one tick and self-corrects on the next. Not a machine assumption — it happens equally on any multi-display desktop — but it is a real one-frame glitch on every crossing. A continue after a successful cover_display would drop the stale frame instead.

Verification

cargo clippy --all-targets -- -D warnings clean. cargo test 117 passed. node --test tests/*.test.js 6 passed. pre-commit run --all-files all passed. ./scripts/verify-overlay.sh passed every check on this two-display mixed-scale desktop, including all twelve frame-loop checks and both hit-test probes.

Ready to merge.

@omesser

omesser commented Aug 29, 2026

Copy link
Copy Markdown
Owner Author

Follow-up on the crossing frame flagged in the audit. I tried the suggested fix, measured it, and reverted it.

The suggestion. origin and scale are read at the top of the tick, and cover_display moves the window later in the same tick, so place_sprite could use the origin of the display the window just left. The proposed fix was to continue after a successful move.

Measured, on a synthetic drag across the boundary with #49 merged in:

frames drawn off-window worst window-local x
with the continue 455 1 1950
without it 479 1 1950

No difference. cover_display is asynchronous: the move is queued rather than applied inside the tick. So the frame being skipped was still correct, because the window had not moved yet, and the one stale frame lands on a later tick that the continue never covers.

The continue therefore drops a good frame and fixes nothing. Reverted.

The single off-window frame out of ~460 is the asynchronous move landing. It is the same 16ms artifact already recorded as the ponytail: limitation on this PR, and one overlay per display remains the upgrade that removes it.

Worth recording that the two PRs compose: merging #49 and this one conflicts only in the import block of src-tauri/src/main.rs, and the merged build drags the Character from display 1 to display 2 correctly.

omesser and others added 3 commits August 29, 2026 23:51
Dragging the Character across a display boundary lost it. The Engine
never did: the frame trace shows it following the cursor onto the
second display and coming to rest on that display's floor. The window
lost it. macOS gives each display its own Space and draws a window
spanning two of them on only one, so an overlay sized to the union of
every display frame is invisible on every display but the one it
happens to belong to. The Character was being drawn the whole time,
where nobody could see it.

The overlay now covers one display, and the frame loop moves it to
whichever display the Character's feet are on. Physics is untouched --
it still runs in the one point space every display shares. Which
display that is comes from `overlay::display_for`, which answers with
the nearest when the point is in a gap between displays or past the
outermost edge, so a cursor between screens still leaves the overlay
somewhere.

The per-tick `primary_monitor()` call goes with it. It read `NSScreen`
from the frame loop, which only the main thread may do, and the scale
factor it fetched now arrives from the same cached main-thread read as
the display frames.

Without this, a Character dragged onto a second display is gone until
it is dragged back blind, and every desktop with more than one display
has screens the Character cannot be seen on at all.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The `display_for` fixture was called `mixed_dpi` and documented as "the desktop
the sprite went missing on". That points at hardware, and `display_for` never
sees a scale factor: it compares rectangles in points, so DPI is not what the
test covers. A reader has to check the values to find out whether the test
depends on the machine running it. It does not.

Both fixtures now say what arrangement they describe and that the values are
literals. Where a scale factor is the subject, in `in_points`, the doc keeps the
conversion and drops the hardware.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The verification harness placed its prop windows a fixed number of points
below the centre of the first display's whole frame. The sprite starts at
the centre of that display's usable frame instead, so the gap between the
two was whatever this desktop's menu bar and Dock happened to make it, and
a fixed drop past a shorter display's bottom edge is a window macOS will
not place, which reads as a frame loop that never noticed a move.

Both props are now measured from the same usable frame the app starts the
sprite in, at fractions of the room between the sprite and the floor. The
one assumption left is stated: about 350 points of usable height, which
every display macOS runs on clears.

Also accepts more than one matching display for the overlay's rectangle.
Mirrored displays report the same rectangle, and two matches there is one
answer said twice rather than a window spanning two screens.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@omesser
omesser force-pushed the feat/55-follow-the-character branch from 7839646 to 708c52d Compare August 29, 2026 20:52
@omesser
omesser merged commit 32c56df into main Aug 29, 2026
2 checks passed
omesser added a commit that referenced this pull request Aug 29, 2026
`docs/agents/comments.md` covers code comments. The same drift shows up in
commit messages, pull request descriptions and review replies, and nothing
written down covered those. Pull request #56 opened on "Not the arithmetic —
the window.", a sentence built for effect rather than to state a fact, and the
repo owner rejected it by hand. Without the rule written down the next
contributor repeats it, and the owner corrects it by hand again.

The prose rules are identical across the four surfaces, so they are stated
once. A sibling document rather than a section in `comments.md`, because a
file named for code comments is a surprising place to find the rule about
commit subjects.

Closes #63

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
omesser added a commit that referenced this pull request Aug 29, 2026
`window_source` handed every non-macOS platform `StubWindowSource` and an
empty `DisplayCache`, so the Shell knew of no displays at all. Since #56 the
setup hook calls `display_for` over `displays.read().frames` and returns
`Err("no displays reported")` when it finds none, so a Linux build panics at
startup — and before #56 it started with no floor and no walls, and the sprite
fell out of the world on the first tick. `docs/SPEC.md` and DESIGN.md both call
this a supported degraded mode rather than an error state.

`read_displays` was gated on macOS although its body is portable Tauri —
`available_monitors`, `work_area`, `scale_factor`. Ungated, it feeds a source
that reports the usable frames and no windows, which is what screen-edge
physics is. `Capabilities::default()` declares no `window_geometry`, so
`snapshot()` keeps the windows cleared.

No test: the arm needs a real window server, and the arithmetic it feeds
(`in_points`, `usable_frame`, `starting_position`, `display_for`) is already
covered in `crates/core`. The Linux CI leg compiles it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The Character disappears when dragged onto a second display

1 participant