From ec75a261da76e7a0f63d19fb7ac9b2a2c8ac5a36 Mon Sep 17 00:00:00 2001 From: omesser Date: Sun, 30 Aug 2026 03:22:29 +0300 Subject: [PATCH 1/2] Give every display its own overlay One window cannot be on two displays, so a Character on a seam was clipped to half of itself. Every display now gets an overlay covering it, each told where the sprite is in its own coordinates, and the halves meet. Co-Authored-By: Claude Opus 5 --- DESIGN.md | 16 +- README.md | 33 ++- crates/core/src/overlay.rs | 249 ++++++++++---------- docs/SPEC.md | 23 +- scripts/verify-overlay.sh | 71 +++--- src-tauri/capabilities/overlay.json | 4 +- src-tauri/src/main.rs | 337 ++++++++++++++++++---------- src/main.js | 18 +- 8 files changed, 466 insertions(+), 285 deletions(-) diff --git a/DESIGN.md b/DESIGN.md index b4d34a03a..a1c09b3e3 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -356,11 +356,15 @@ to being called spyware. Physics runs in a single coordinate space, the one every display shares. Reconciling per-display physics would be considerably harder. -The overlay window is the exception, and not by choice: macOS gives each display -its own Space and draws a window spanning two of them on only one, so the window -covers a single display and the frame loop moves it to whichever display the -sprite reaches. A sprite straddling the boundary is clipped, which one overlay -per display would fix and nothing else will. +The overlays are per display, and not by choice: macOS gives each display its +own Space and draws a window spanning two of them on only one, so a window sized +to the union is invisible everywhere but the display it belongs to. Every +display therefore gets its own overlay covering it, and every overlay is told +where the sprite is, in its own coordinates. A sprite straddling a boundary is +drawn by both and clipped by each to its own half, so the halves meet at the +seam. One code path builds and configures an overlay, because click-through, +window level, Spaces membership and hide rules have to be identical across all +of them. Two real problems to budget for: differing backing scale factors between displays, and gaps between non-aligned displays. Clamp to the union of visible @@ -474,5 +478,5 @@ records it. | 20 | Memory | One shared plaintext Markdown file the user owns; chat history session-scoped | | 21 | No Harness attached | Fully charming — full Spatial Layer, chat shows a connect nudge | | 23 | Art | True pixel art, integer nearest-neighbour — [ADR-0006](./docs/adr/0006-pixel-art-integer-scaling.md) | -| 24 | Displays | Overlay covers one display and follows the sprite; physics spans them all | +| 24 | Displays | One overlay per display, each covering it; physics spans them all | | 25 | Release staging | v1 is charm, chat, Memory, and Harness attach; voice and Ambient Capture deferred | diff --git a/README.md b/README.md index 5ce87b469..e23a99875 100644 --- a/README.md +++ b/README.md @@ -114,9 +114,10 @@ it when the overlay, the platform layer or the frame loop changes. scripts/verify-overlay.sh # or --keep to leave the app running ``` -It checks that exactly one overlay window exists at floating level, that its -bounds match one whole display — the one the Character is on — and that the app -is an accessory with no Dock tile or switcher entry. +It checks that there is one overlay window per display, that every one of them +is on screen at floating level, that each display is covered whole by one of +them and none of them covers anything else, and that the app is an accessory +with no Dock tile or switcher entry. Then it checks the frame loop against a real desktop. It opens a plain window of its own below where the sprite starts, so the sprite has a Perch to aim at, and @@ -136,9 +137,13 @@ Keep hands off the mouse while it runs. For a live view of what the app is deciding, set `AI_BUDDY_TRACE_HITTEST=1` for the click-through decision and `AI_BUDDY_TRACE_FRAMES=1` for the Engine's -frames — state, position and animation, once per tick. The first line or two of -the hit-test trace are emitted before the window frame settles and report a -stale origin; read the later ones. +frames — state, position and animation, once per tick. Both trace in the point +space every display shares, so the positions they report are comparable with +what the window server says about any display. + +There is one overlay per display and every one of them is told where the +sprite is, so the trace is one line per tick rather than one per overlay. Which +overlay the cursor is on is on the hit-test line. **A human** is still needed for the last step, because only the window server can answer it. Run the app, then confirm: @@ -195,6 +200,22 @@ can answer it. Run the app, then confirm: goes where your hand goes. Let go and it settles back onto the Dock's top edge, fully visible. +The last three need a second display, and only a window server can answer them: + +13. **A Character on a seam is whole.** Drag the sprite slowly across the + boundary between two displays and stop with it half on each. Both halves + are drawn, and they meet — no gap between them, nothing drawn twice, and + the art is not doubled or offset at the seam. Two displays with different + scale factors is the case worth doing this on. +14. **Either half can be clicked.** With the sprite straddling, click the half + on each display in turn. Both pick it up, and clicking beside it on either + display still reaches whatever is underneath. +15. **A display can come and go.** With the app running, unplug a display, or + turn one off in System Settings > Displays. The sprite carries on, and the + remaining display still passes clicks through where the sprite is not. + Plug it back in: the sprite can be dragged onto it again within a second + or so, without restarting the app. + The sprite starts in the middle of the first display and goes wherever gravity and your windows take it from there, or wherever you put it. To watch it fall without touching it, move or close the window it is sitting on. diff --git a/crates/core/src/overlay.rs b/crates/core/src/overlay.rs index 32b3a37db..0ccb7872d 100644 --- a/crates/core/src/overlay.rs +++ b/crates/core/src/overlay.rs @@ -6,56 +6,39 @@ //! decision is this module. It is pure arithmetic so it can be tested without a //! windowing system. //! -//! Placing the overlay is here for the same reason. Which display the window -//! covers and where the art lands inside it are two more sums the Shell would -//! otherwise do by hand beside the window server, where nothing can check them. +//! Placing the overlays is here for the same reason. There is one overlay per +//! display, so which one the cursor is on and where the art lands inside each of +//! them are sums the Shell would otherwise do by hand beside the window server, +//! where nothing can check them. +//! +//! Everything here is in points, the space every display shares and the Engine +//! works in. Physical pixels stop at `window_source::in_points`. use crate::window_source::Rect; -/// Convert a global cursor reading into overlay-window coordinates, in points. +/// The index of the display a point is on, or of the nearest one when it is on +/// none. /// -/// Both inputs are "physical pixels", but they are not the same physical space. -/// The windowing layer scales the global cursor by the *primary* display's scale -/// factor and the window's own origin by the *window's* scale factor. On a -/// mixed-DPI desktop those differ, so subtracting the two raw readings and -/// dividing once yields a hit region nowhere near the sprite. Each reading has -/// to be returned to points using the factor that produced it. -pub fn cursor_in_window( - cursor_physical: (f64, f64), - cursor_scale: f64, - window_origin_physical: (f64, f64), - window_scale: f64, -) -> (i32, i32) { - let cursor_x = cursor_physical.0 / cursor_scale; - let cursor_y = cursor_physical.1 / cursor_scale; - let origin_x = window_origin_physical.0 / window_scale; - let origin_y = window_origin_physical.1 / window_scale; - - ( - (cursor_x - origin_x).round() as i32, - (cursor_y - origin_y).round() as i32, - ) -} - -/// The display a point is on, or the nearest one when it is on none. +/// Every display has an overlay, so this names the overlay a point belongs to — +/// the one whose window a click there would reach. /// -/// The overlay covers one display at a time, so something has to name which. /// Containment answers it almost always, and the fallback is for the moments it /// cannot: displays need not be flush, so a cursor can sit in the gap between /// two of them, and a thrown sprite passes outside the outermost edge before -/// physics catches it. The nearest display is the answer there because the -/// point is on its way into or out of that one, and because the alternative — -/// no display — leaves the overlay nowhere to be and the Character invisible. +/// physics catches it. The nearest display is the answer there because the point +/// is on its way into or out of that one. /// -/// A point on a shared edge belongs to the first display that reported it, -/// which is arbitrary and has to be: both answers are equally true. +/// A point on a shared edge belongs to the first display that reported it, which +/// is arbitrary and has to be: both answers are equally true. It costs nothing +/// here, because both overlays are drawing the same sprite anyway. /// /// `None` only when no display was reported, which is a machine with no screen. -pub fn display_for(point: (f64, f64), displays: &[Rect]) -> Option { +pub fn display_index_for(point: (f64, f64), displays: &[Rect]) -> Option { displays .iter() - .min_by(|a, b| outside_by(point, a).total_cmp(&outside_by(point, b))) - .copied() + .enumerate() + .min_by(|(_, a), (_, b)| outside_by(point, a).total_cmp(&outside_by(point, b))) + .map(|(index, _)| index) } /// How far outside a rectangle a point lies, squared. Zero anywhere inside it. @@ -76,41 +59,58 @@ fn outside_by(point: (f64, f64), rect: &Rect) -> f64 { /// Where to draw the art, given where the Character's feet are. /// /// A `Frame` reports the contact point: the Character's feet, in the point space -/// every display shares. The webview draws in points from the overlay's -/// top-left, and the art hangs above the feet and is centred on them, so the -/// drawn rectangle sits half a width to the left and a whole height above. +/// every display shares. The art hangs above the feet and is centred on them, so +/// the drawn rectangle sits half a width to the left and a whole height above. /// -/// The window origin arrives in physical pixels against the window's own scale -/// factor, exactly as it does for `cursor_in_window`, and has to be returned to -/// points before it can be subtracted from a position that is already in them. -pub fn place_sprite( - contact: (f64, f64), - window_origin_physical: (f64, f64), - window_scale: f64, - art_size: (i32, i32), - scale: i32, -) -> SpriteRect { - let origin_x = window_origin_physical.0 / window_scale; - let origin_y = window_origin_physical.1 / window_scale; +/// The answer is in that same shared space, which is the space the hit-test asks +/// its question in. `SpriteRect::in_overlay` is what turns it into one overlay's +/// own coordinates for drawing. +pub fn place_sprite(contact: (f64, f64), art_size: (i32, i32), scale: i32) -> SpriteRect { let (width, height) = art_size; SpriteRect { - x: (contact.0 - origin_x).round() as i32 - width / 2, - y: (contact.1 - origin_y).round() as i32 - height, + x: contact.0.round() as i32 - width / 2, + y: contact.1.round() as i32 - height, scale, } } -/// Where the sprite sits in the overlay window, and how far its art is blown up. +/// Where the sprite sits, and how far its art is blown up. /// -/// `x` and `y` are the sprite's top-left corner in window coordinates. `scale` is -/// the integer nearest-neighbour factor the art is rendered at. +/// `x` and `y` are the sprite's top-left corner, in whichever space it was +/// placed in: `place_sprite` works in the shared point space, `in_overlay` moves +/// a copy into one overlay's own. `scale` is the integer nearest-neighbour +/// factor the art is rendered at. +#[derive(Clone, Copy)] pub struct SpriteRect { pub x: i32, pub y: i32, pub scale: i32, } +impl SpriteRect { + /// The same rectangle in one overlay's coordinates: points from the + /// top-left corner of the display that overlay covers. + /// + /// Every overlay is handed the sprite, including the ones it is nowhere + /// near, and each draws the part that falls inside it. That is what makes a + /// Character on a seam whole: both overlays are given the same rectangle in + /// their own coordinates, so the halves they clip meet at the seam instead + /// of overlapping or leaving a gap. + /// + /// No scale factor appears. Two displays with different backing factors + /// share one point space, and the overlay's origin is its display's because + /// the window is placed to cover that display exactly — which is what + /// `scripts/verify-overlay.sh` asserts on a real desktop. + pub fn in_overlay(&self, overlay: Rect) -> SpriteRect { + SpriteRect { + x: self.x - overlay.x.round() as i32, + y: self.y - overlay.y.round() as i32, + scale: self.scale, + } + } +} + /// Which pixels of an Animation frame are drawn, at the art's own resolution. #[derive(Debug)] pub struct AlphaMask { @@ -281,32 +281,11 @@ mod tests { ); } - /// Mixed-DPI is the case that matters: a 1x primary display beside a 2x - /// built-in. The two inputs arrive in different physical spaces, so each has - /// to be undone with the factor that produced it. - #[test] - fn cursor_and_window_are_converted_with_their_own_scale_factors() { - // Cursor sits at 500,300 in points. The primary display is 1x, so its - // physical reading is the same numbers. - // The overlay's top-left is at 100,50 in points on a 2x display, so its - // physical reading is doubled. - let (x, y) = cursor_in_window((500.0, 300.0), 1.0, (200.0, 100.0), 2.0); - - assert_eq!((x, y), (400, 250), "500-100 across, 300-50 down, in points"); - } - - #[test] - fn a_single_scale_desktop_still_converts() { - let (x, y) = cursor_in_window((800.0, 600.0), 2.0, (200.0, 100.0), 2.0); - - assert_eq!((x, y), (300, 250), "400-100 across, 300-50 down, in points"); - } - /// The art hangs above the feet and is centred on them, so a Character /// standing at a point is drawn above and to the left of it. #[test] fn the_art_hangs_above_the_contact_point_and_is_centred_on_it() { - let sprite = place_sprite((500.0, 300.0), (0.0, 0.0), 1.0, (128, 128), 4); + let sprite = place_sprite((500.0, 300.0), (128, 128), 4); assert_eq!( (sprite.x, sprite.y), @@ -315,21 +294,6 @@ mod tests { ); } - /// The same trap as `cursor_in_window`: the window origin arrives in - /// physical pixels against the window's own scale factor, and the Frame - /// arrives in points. Subtracting them raw puts the art nowhere near the - /// Character on any display that is not 1x. - #[test] - fn the_window_origin_is_undone_with_the_windows_own_scale() { - let sprite = place_sprite((500.0, 300.0), (200.0, 100.0), 2.0, (128, 128), 4); - - assert_eq!( - (sprite.x, sprite.y), - (336, 122), - "origin is 100,50 in points" - ); - } - fn rect(x: f64, y: f64, width: f64, height: f64) -> Rect { Rect { x, @@ -342,11 +306,11 @@ mod tests { /// Two displays side by side, the second starting where the first ends. /// /// Literal points, so this runs the same on any machine, including one with - /// a single display. `display_for` compares rectangles and never sees a - /// scale factor: converting a display into points is `window_source`'s job - /// and is tested there. These are the sizes that conversion produces for a - /// 1080p display beside a Retina one, which is the arrangement the bug was - /// reported on. + /// a single display. Nothing here sees a scale factor: converting a display + /// into points is `window_source`'s job and is tested there. These are the + /// sizes that conversion produces for a 1080p display beside a Retina one, + /// which is the arrangement the bug was reported on — 1x beside 2x, sharing + /// one point space. /// /// The seam at 1920 is the point of the fixture. Every case below is about /// which side of it a point falls on. @@ -357,26 +321,83 @@ mod tests { ] } + /// The issue this module is here for: a Character standing on the seam is + /// drawn by both overlays, and the halves have to meet. + #[test] + fn a_sprite_on_a_seam_is_two_halves_that_meet() { + let displays = two_displays(); + // Feet on the seam. 128 points of art, so 64 fall on each display. + let sprite = place_sprite((1920.0, 700.0), (128, 128), 4); + + let left = sprite.in_overlay(displays[0]); + let right = sprite.in_overlay(displays[1]); + + assert_eq!((left.x, left.y), (1856, 572), "overlay 0 draws from 1856"); + assert_eq!((right.x, right.y), (-64, 572), "overlay 1 draws from -64"); + assert_eq!( + 1920 - left.x, + -right.x, + "the columns overlay 0 runs out of room for are the ones overlay 1 hides \ + behind its own left edge: no gap and no overlap" + ); + } + + /// Displays stack vertically too, and the origin subtracted is the whole + /// origin rather than an x. + #[test] + fn an_overlay_below_another_subtracts_its_own_top_edge() { + let displays = [ + rect(0.0, 0.0, 1920.0, 1080.0), + rect(0.0, 1080.0, 1920.0, 1080.0), + ]; + let sprite = place_sprite((500.0, 1080.0), (128, 128), 4); + + assert_eq!( + sprite.in_overlay(displays[0]).y, + 952, + "128 up from the seam" + ); + assert_eq!(sprite.in_overlay(displays[1]).y, -128, "and 128 above it"); + } + + /// A display whose origin is not a whole number of points, which fractional + /// scaling produces. The sprite still lands on the pixel grid. + #[test] + fn a_fractional_display_origin_is_rounded_to_the_pixel_grid() { + let sprite = place_sprite((500.0, 300.0), (128, 128), 4); + + assert_eq!( + sprite.in_overlay(rect(100.4, 50.6, 800.0, 600.0)).x, + 336, + "436 - 100" + ); + assert_eq!( + sprite.in_overlay(rect(100.4, 50.6, 800.0, 600.0)).y, + 121, + "172 - 51" + ); + } + #[test] fn a_point_on_a_display_belongs_to_that_display() { let displays = two_displays(); - assert_eq!(display_for((960.0, 540.0), &displays), Some(displays[0])); - assert_eq!(display_for((2600.0, 500.0), &displays), Some(displays[1])); + assert_eq!(display_index_for((960.0, 540.0), &displays), Some(0)); + assert_eq!(display_index_for((2600.0, 500.0), &displays), Some(1)); assert_eq!( - display_for((1920.5, 500.0), &displays), - Some(displays[1]), + display_index_for((1920.5, 500.0), &displays), + Some(1), "half a point across the seam is across it" ); assert_eq!( - display_for((1920.0, 500.0), &displays), - Some(displays[0]), + display_index_for((1920.0, 500.0), &displays), + Some(0), "the seam itself goes to whichever display reported first" ); } /// Displays are not always flush, and a cursor can sit in the gap between - /// two of them. The overlay still has to be somewhere. + /// two of them. The click still belongs to one overlay. #[test] fn a_point_in_the_gap_between_displays_belongs_to_the_nearer_one() { let displays = [ @@ -384,8 +405,8 @@ mod tests { rect(2000.0, 0.0, 1728.0, 1117.0), ]; - assert_eq!(display_for((1930.0, 500.0), &displays), Some(displays[0])); - assert_eq!(display_for((1990.0, 500.0), &displays), Some(displays[1])); + assert_eq!(display_index_for((1930.0, 500.0), &displays), Some(0)); + assert_eq!(display_index_for((1990.0, 500.0), &displays), Some(1)); } /// A Throw carries the sprite past the outermost edge before physics @@ -395,19 +416,19 @@ mod tests { let displays = two_displays(); assert_eq!( - display_for((4000.0, 2000.0), &displays), - Some(displays[1]), + display_index_for((4000.0, 2000.0), &displays), + Some(1), "off the bottom-right corner of the desktop" ); assert_eq!( - display_for((-500.0, -500.0), &displays), - Some(displays[0]), + display_index_for((-500.0, -500.0), &displays), + Some(0), "off the top-left corner of the desktop" ); } #[test] - fn no_displays_is_nowhere_to_put_the_overlay() { - assert_eq!(display_for((960.0, 540.0), &[]), None); + fn no_displays_is_no_overlay_to_belong_to() { + assert_eq!(display_index_for((960.0, 540.0), &[]), None); } } diff --git a/docs/SPEC.md b/docs/SPEC.md index eea165699..56d813453 100644 --- a/docs/SPEC.md +++ b/docs/SPEC.md @@ -300,18 +300,27 @@ frames rather than to its bounding rectangle, so the sprite cannot enter gaps be non-aligned displays. Backing scale factor is resolved per display at render time, not in the Engine. -The overlay window covers one display and follows the sprite to the next. It cannot be +There is one overlay window per display, each covering that display. No window can be sized to the union: macOS gives each display its own Space and draws a window spanning two of them on only one, so a union-sized overlay is invisible on every display but the -one it belongs to. The Engine is unaffected — it works in the space all displays share, -and the Shell picks the display the sprite's contact point is on, or the nearest when it -is on none. +one it belongs to. The set follows the desktop, so a display attached or removed while +the app runs gains or loses its overlay without a restart. + +Every overlay is told where the sprite is, in its own coordinates, and each draws the +part that falls inside it. A Character straddling a seam is therefore whole: the two +halves are clipped by the two overlays and meet at the seam. The Engine is unaffected — +it works in the space all displays share and knows nothing about windows. ### Overlay and input -One always-on-top overlay window per Character Instance. On macOS: a non-activating panel -at floating level, joining all Spaces, stationary, excluded from the application switcher, -never accepting first responder status. +One always-on-top overlay window per display per Character Instance. On macOS: a +non-activating panel at floating level, joining all Spaces, stationary, excluded from the +application switcher, never accepting first responder status. Identical on every overlay, +which is why one code path builds and configures them all. + +Click-through is decided once per tick, in the shared point space, and applied to the +overlay the cursor is on; every other overlay passes clicks through, so a sprite on one +display never swallows a click on another. Click-through is per-window rather than per-pixel in Tauri. The overlay tracks the cursor and toggles ignore-mouse-events by hit-testing the sprite's current alpha, so transparent diff --git a/scripts/verify-overlay.sh b/scripts/verify-overlay.sh index 96d2e87e9..7f07eb667 100755 --- a/scripts/verify-overlay.sh +++ b/scripts/verify-overlay.sh @@ -308,20 +308,21 @@ lsappinfo list 2> /dev/null | grep -A 4 '"ai-buddy"' > "$OUT/lsappinfo.txt" # rests on the Dock (#39), so this crop is also the proof of that — the sprite # is whole, above the Dock, rather than three quarters buried in it. python3 - "$OUT" << 'PY' -import json, re, subprocess, sys +import re, subprocess, sys out = sys.argv[1] log = open(f"{out}/app.log").read() -windows = json.load(open(f"{out}/window.json"))["windows"] size = re.search(r"sprite (\d+)x(\d+)", log) +# The trace reports the sprite in the space every display shares, which is the +# space screencapture crops in. No overlay's origin is added back: with one +# overlay per display there is no single window to add. at = re.findall(r"^frame: \S+ Grounded \S+ sprite\((-?\d+),(-?\d+)\)", log, re.M) -if windows and size and at: +if size and at: width, height = (int(g) for g in size.groups()) x, y = (int(g) for g in at[-1]) pad = 40 - region = (f"{windows[0]['x'] + x - pad},{windows[0]['y'] + y - pad}," - f"{width + pad * 2},{height + pad * 2}") + region = (f"{x - pad},{y - pad},{width + pad * 2},{height + pad * 2}") subprocess.run(["screencapture", "-x", "-R", region, f"{out}/sprite.png"], check=False) PY @@ -350,29 +351,45 @@ for d in displays: print(f" {d['w']:.0f}x{d['h']:.0f} at ({d['x']:.0f},{d['y']:.0f})") print("\nChecks:") -check(len(windows) == 1, "exactly one overlay window", f"found {len(windows)}") +# One overlay per display, each covering its own: a Character straddling a seam +# is drawn by the overlay on each side, so both halves are on screen. One window +# could not do it — macOS gives each display its own Space and draws a window +# spanning two of them on only one, so an overlay wider than a display is +# invisible on every display but that one. +check(len(windows) == len(displays), + "one overlay window per display", + f"{len(windows)} windows, {len(displays)} displays") if not windows: sys.exit(1) -w = windows[0] -check(w["onscreen"], "window is on screen") -check(w["layer"] == 3, "floating window level", f"layer={w['layer']}") +# Identical rules on every one of them. A second window is a second place for +# the level and the on-screen flag to disagree, which is the whole risk of +# having more than one. +check(all(w["onscreen"] for w in windows), "every overlay is on screen") +levels = {w["layer"] for w in windows} +check(levels == {3}, "every overlay is at floating level", f"levels={sorted(levels)}") -# One display exactly, rather than the union of them: macOS gives each display -# its own Space and draws a window spanning two of them on only one, so an -# overlay wider than a display is invisible on every display but that one. The -# frame loop moves it to whichever display the Character is on, and the origin -# has to match too — a window covering the right area of the wrong display is -# the same disappearance. + +def bounds(r): + return (r["x"], r["y"], r["w"], r["h"]) + + +# The origin has to match too — a window covering the right area of the wrong +# display is the same disappearance. # -# Any match, not exactly one: mirrored displays report the same rectangle, and -# two matches there is one answer said twice rather than a window spanning two -# screens. What rules a union out is the equality itself. -covered = [d for d in displays - if (w["x"], w["y"], w["w"], w["h"]) == (d["x"], d["y"], d["w"], d["h"])] -check(bool(covered), - "window covers one whole display and no more", - f"{w['w']:.0f}x{w['h']:.0f} at ({w['x']:.0f},{w['y']:.0f})") +# Matched by set rather than pairwise: mirrored displays report the same +# rectangle, and which overlay answers which of them is not a fact worth +# asserting. What rules a union out is the equality itself. +uncovered = [d for d in displays if bounds(d) not in {bounds(w) for w in windows}] +check(not uncovered, + "every display has an overlay covering it whole", + f"{len(uncovered)} uncovered: " + ", ".join( + f"{d['w']:.0f}x{d['h']:.0f} at ({d['x']:.0f},{d['y']:.0f})" for d in uncovered)) + +astray = [w for w in windows if bounds(w) not in {bounds(d) for d in displays}] +check(not astray, + "no overlay covers anything but a whole display", + ", ".join(f"{w['w']:.0f}x{w['h']:.0f} at ({w['x']:.0f},{w['y']:.0f})" for w in astray)) ls = open(f"{out}/lsappinfo.txt").read() check('type="UIElement"' in ls, "accessory app: no Dock tile or switcher entry") @@ -411,16 +428,16 @@ echo "Hit-test pipeline:" BEFORE=$(swift scripts/cursor-position.swift) SPRITE_AT=$( python3 - "$OUT" << 'PY' -import json, re, sys +import re, sys out = sys.argv[1] log = open(f"{out}/app.log").read() -w = json.load(open(f"{out}/window.json"))["windows"][0] size = re.search(r"sprite (\d+)x(\d+)", log) at = re.findall(r"^frame: .* sprite\((-?\d+),(-?\d+)\)", log, re.M) if not (size and at): sys.exit(1) -# Where the art's top-left corner is on screen, and how big it is. -print(int(w["x"]) + int(at[-1][0]), int(w["y"]) + int(at[-1][1]), *size.groups()) +# Where the art's top-left corner is on screen, and how big it is. Already in +# the shared point space, which is the space the cursor is warped in. +print(int(at[-1][0]), int(at[-1][1]), *size.groups()) PY ) || SPRITE_AT="" diff --git a/src-tauri/capabilities/overlay.json b/src-tauri/capabilities/overlay.json index 284471337..0aefcac78 100644 --- a/src-tauri/capabilities/overlay.json +++ b/src-tauri/capabilities/overlay.json @@ -1,9 +1,9 @@ { "$schema": "../gen/schemas/desktop-schema.json", "identifier": "overlay", - "description": "Tauri denies the webview every core command unless a capability grants it, and with no capability at all the overlay could not hear the Frame the Shell pushes it each tick - it drew nothing, silently. Listening for events is the only thing it needs: it holds no state, reads no files, and calls no command of its own.", + "description": "Tauri denies the webview every core command unless a capability grants it, and with no capability at all the overlay could not hear the Frame the Shell pushes it each tick - it drew nothing, silently. Listening for events is the only thing it needs: it holds no state, reads no files, and calls no command of its own. The glob covers every overlay: there is one per display, labelled overlay-0, overlay-1 and so on, and they all draw the same Character under the same rules.", "windows": [ - "overlay" + "overlay-*" ], "permissions": [ "core:event:allow-listen", diff --git a/src-tauri/src/main.rs b/src-tauri/src/main.rs index c6df62544..619415eec 100644 --- a/src-tauri/src/main.rs +++ b/src-tauri/src/main.rs @@ -1,11 +1,11 @@ //! ai-buddy's overlay shell. //! -//! One transparent, always-on-top window renders the Character. Click-through on -//! macOS is per-window rather than per-pixel, so a screen-sized transparent -//! window would swallow every click. The shell therefore tracks the cursor and -//! toggles ignore-mouse-events by hit-testing the sprite's alpha, which is what -//! makes the overlay feel like a sprite on the desktop instead of a sheet of -//! glass over it. +//! One transparent, always-on-top window per display renders the Character. +//! Click-through on macOS is per-window rather than per-pixel, so a screen-sized +//! transparent window would swallow every click. The shell therefore tracks the +//! cursor and toggles ignore-mouse-events by hit-testing the sprite's alpha, +//! which is what makes the overlay feel like a sprite on the desktop instead of +//! a sheet of glass over it. //! //! It also owns the frame loop, which is the only thing that can: the Engine is //! pure and cannot read a clock, and `WindowSource` reports geometry and nothing @@ -24,7 +24,7 @@ use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH}; use ai_buddy_core::engine::{Engine, Point}; use ai_buddy_core::input::Pointer; -use ai_buddy_core::overlay::{cursor_in_window, display_for, place_sprite, SpriteRect}; +use ai_buddy_core::overlay::{display_index_for, place_sprite, SpriteRect}; use ai_buddy_core::snapshot::{starting_position, SnapshotAssembler}; use ai_buddy_core::window_source::{Rect, WindowSource}; use serde::Serialize; @@ -51,7 +51,17 @@ const ALPHA_THRESHOLD: u8 = 128; /// And the Engine advances on elapsed time, so something has to advance it. const ENGINE_TICK: Duration = Duration::from_millis(16); -const OVERLAY_LABEL: &str = "overlay"; +/// Every overlay's label: this and the index of the display it covers. +/// +/// One window per display, so the index is both the name and the way the frame +/// loop finds the overlay belonging to a display. `capabilities/overlay.json` +/// grants the same permissions to `overlay-*`. +const OVERLAY_LABEL: &str = "overlay-"; + +/// The label of the overlay covering the display at `index`. +fn overlay_label(index: usize) -> String { + format!("{OVERLAY_LABEL}{index}") +} /// The event carrying each `Frame` to the webview. const FRAME_EVENT: &str = "frame"; @@ -100,6 +110,89 @@ fn cover_display(window: &tauri::WebviewWindow, display: Rect) -> Result<(), tau window.set_position(LogicalPosition::new(display.x, display.y)) } +/// Build one overlay, configure it, and put it over its display. +/// +/// The only place an overlay is made. Click-through, window level, Spaces +/// membership and hide rules have to be identical on every overlay, and a +/// second window is a second place for them to disagree; one function called +/// once per display is what keeps them one set of rules instead of two. +/// +/// Main thread only: it builds a window and calls AppKit. +fn build_overlay( + app: &tauri::AppHandle, + label: &str, + display: Rect, +) -> Result<(), Box> { + let window = WebviewWindowBuilder::new(app, label, WebviewUrl::default()) + .title("ai-buddy") + .transparent(true) + .decorations(false) + .shadow(false) + .always_on_top(true) + .visible_on_all_workspaces(true) + .accept_first_mouse(true) + .focused(false) + .resizable(false) + .skip_taskbar(true) + .visible(false) + .build()?; + + // Click-through until the cursor is proven to be over the sprite. The wrong + // default swallows a click; this one loses nothing. + window.set_ignore_cursor_events(true)?; + cover_display(&window, display)?; + platform::configure_overlay(&window)?; + window.show()?; + + eprintln!( + "overlay: {label} covers {:.0}x{:.0} at ({:.0},{:.0})", + display.width, display.height, display.x, display.y, + ); + + Ok(()) +} + +/// One overlay per display, each covering that display. +/// +/// This is what keeps a Character on a seam whole: both overlays draw it, each +/// clipping its own half, and the halves meet. One window cannot do it, because +/// macOS gives each display its own Space and draws a window spanning two of +/// them on only one — a window sized to the display union is invisible +/// everywhere but the display it happens to belong to. +/// +/// Idempotent, because the desktop changes while the app runs: a display that +/// already has its overlay keeps it and is only re-covered, which is what a +/// display that moved or changed resolution needs. Overlays past the end of the +/// list belong to displays that have been unplugged. +/// +/// Main thread only; see `build_overlay`. +fn place_overlays( + app: &tauri::AppHandle, + displays: &[Rect], +) -> Result<(), Box> { + for (index, display) in displays.iter().enumerate() { + let label = overlay_label(index); + match app.get_webview_window(&label) { + Some(window) => cover_display(&window, *display)?, + None => build_overlay(app, &label, *display)?, + } + } + + // Labels are handed out in order, so the first missing one ends the set. + for index in displays.len().. { + let Some(window) = app.get_webview_window(&overlay_label(index)) else { + break; + }; + eprintln!( + "overlay: {} has no display left to cover", + overlay_label(index) + ); + window.close()?; + } + + Ok(()) +} + /// The frame loop: assemble a snapshot, tick the Engine, apply the `Frame`. /// /// Applying a `Frame` is two things at once, which is why they share a loop. @@ -117,17 +210,18 @@ fn run_frame_loop( source: impl WindowSource + Send + 'static, displays: platform::DisplayCache, start: Point, - home: Rect, + covered: Vec, ) { thread::spawn(move || { let mut engine = Engine::new(start).with_behaviors(cast.behaviors().clone()); let mut assembler = SnapshotAssembler::new(source); - // The display the overlay is currently over, as setup left it. - let mut home = home; + // The displays the overlays cover, as setup left them. + let mut covered = covered; - // `None` until the first decision, so the first tick always applies. - let mut ignoring: Option = None; + // One click-through flag per overlay, `None` until that overlay's first + // decision so the first tick always applies. + let mut ignoring: Vec> = vec![None; covered.len()]; let mut pointer = Pointer::default(); @@ -151,27 +245,21 @@ fn run_frame_loop( loop { thread::sleep(ENGINE_TICK); - let Some(window) = app.get_webview_window(OVERLAY_LABEL) else { - return; // window is gone; so is the reason to tick - }; - - let (Ok(cursor), Ok(origin), Ok(scale)) = ( - app.cursor_position(), - window.outer_position(), - window.scale_factor(), - ) else { + let Ok(cursor) = app.cursor_position() else { continue; }; // The windowing layer reports the global cursor against the primary - // display's scale factor but the window's origin against the - // window's own, so each needs undoing with its own factor. The - // primary's arrives from the cache rather than from a monitor here: - // asking a monitor its scale means asking `NSScreen`, and only the - // main thread may do that. + // display's scale factor, whichever display it is actually over, so + // that factor is what undoes it. It arrives from the cache rather + // than from a monitor here: asking a monitor its scale means asking + // `NSScreen`, and only the main thread may do that. let displays = displays.read(); let cursor_scale = displays.cursor_scale; + // One flag per overlay, and the desktop can gain or lose one. + ignoring.resize(displays.frames.len(), None); + // Wall time since the last tick that reached the Engine, not since // the last turn of this loop: a tick that could not read the // platform skips without advancing anything, and the time it spent @@ -181,24 +269,27 @@ fn run_frame_loop( let elapsed_ms = u32::try_from(last_tick.elapsed().as_millis()).unwrap_or(u32::MAX); last_tick = Instant::now(); - let (local_x, local_y) = cursor_in_window( - (cursor.x, cursor.y), - cursor_scale, - (origin.x as f64, origin.y as f64), - scale, - ); - - let pressed_sprite = drawn_last.as_ref().is_some_and(|last| { - cast.draw(last.animation, last.animation_ms) - .is_some_and(|art| art.mask.hit(&last.rect, local_x, local_y)) - }); - // The Engine works in points across every display, which is the // space the cursor reading becomes once its own scale is undone. let cursor_points = ai_buddy_core::engine::Point { x: cursor.x / cursor_scale, y: cursor.y / cursor_scale, }; + + // The hit-test asks its question in that shared space rather than + // in an overlay's. Every overlay is handed the same sprite in its + // own coordinates, so the answer is the same whichever overlay it + // is asked of, and asking once is one answer instead of one per + // window that could disagree. + let cursor_at = ( + cursor_points.x.round() as i32, + cursor_points.y.round() as i32, + ); + + let pressed_sprite = drawn_last.as_ref().is_some_and(|last| { + cast.draw(last.animation, last.animation_ms) + .is_some_and(|art| art.mask.hit(&last.rect, cursor_at.0, cursor_at.1)) + }); let verbs = pointer.update( pressed_sprite, platform::primary_button_down(), @@ -211,27 +302,27 @@ fn run_frame_loop( } let frame = engine.tick(&assembler.assemble(elapsed_ms, cursor_points, verbs)); - // The overlay covers one display and follows the Character to the - // next. macOS gives each display its own Space and draws a window - // that spans two of them on only one, so a window sized to the - // display union is invisible everywhere but the display it happens - // to belong to — which is how a Character dragged across a boundary - // was lost while the Engine still had it in hand. - // - // The feet decide rather than the cursor, because a Throw crosses - // with nobody holding it. + // A display can be plugged in, unplugged or rearranged while the + // app runs, and every display needs its overlay. Posted rather than + // done here: only the main thread may build a window. Recorded once + // the post is accepted, so a rejected one is retried next tick + // rather than latched as done. // - // ponytail: the sprite is clipped while it straddles the seam, - // because one window cannot be on two displays. One overlay per - // display would draw both halves; that is the upgrade if the pop - // reads badly. - if let Some(display) = - display_for((frame.position.x, frame.position.y), &displays.frames) - { - // Recorded only once the move is accepted, so a refused one is - // retried next tick rather than latched as done. - if display != home && cover_display(&window, display).is_ok() { - home = display; + // An empty read is ignored rather than obeyed. It is what a failed + // read of the desktop looks like as well as a machine with no + // screen, and tearing every overlay down costs two webviews and + // their art to rebuild — for a desktop that has nothing to draw on + // either way. + if !displays.frames.is_empty() && displays.frames != covered { + let handle = app.clone(); + let frames = displays.frames.clone(); + let posted = app.run_on_main_thread(move || { + if let Err(why) = place_overlays(&handle, &frames) { + eprintln!("overlay: {why}"); + } + }); + if posted.is_ok() { + covered = displays.frames.clone(); } } @@ -247,26 +338,14 @@ fn run_frame_loop( drawn.art_size.1 * SPRITE_SCALE, ); + // Placed once, in the space every display shares. Each overlay is + // handed it in its own coordinates below. let sprite = place_sprite( (frame.position.x, frame.position.y), - (origin.x as f64, origin.y as f64), - scale, (width, height), SPRITE_SCALE, ); - let _ = window.emit( - FRAME_EVENT, - Placement { - x: sprite.x, - y: sprite.y, - width, - height, - animation: frame.animation, - frame_index: drawn.index, - }, - ); - if tracing_frames { // Unix milliseconds, so that a prop window opened by the // verification script and this loop can be read against one @@ -294,7 +373,7 @@ fn run_frame_loop( // reach us is a question about where the art is going to be. A // cursor that has just arrived over it must not spend a frame // passing clicks to the application underneath. - let over_sprite = drawn.mask.hit(&sprite, local_x, local_y); + let over_sprite = drawn.mask.hit(&sprite, cursor_at.0, cursor_at.1); drawn_last = Some(Drawn { rect: sprite, animation: frame.animation, @@ -306,26 +385,68 @@ fn run_frame_loop( // the cursor over transparent pixels, hand the button to whatever // is underneath, and drop the sprite in the user's hand. let ignore = !(over_sprite || pointer.grabbing()); - let flipped = ignoring != Some(ignore); + let on_overlay = + display_index_for((cursor_points.x, cursor_points.y), &displays.frames); + let mut flipped = false; + + for (index, display) in displays.frames.iter().enumerate() { + let label = overlay_label(index); + let Some(window) = app.get_webview_window(&label) else { + continue; // a display whose overlay has not been built yet + }; + let local = sprite.in_overlay(*display); + + // Every overlay is told, including the ones the sprite is + // nowhere near: each draws the part that falls inside it, which + // is what leaves a Character on a seam whole instead of clipped + // to one display. + // + // Addressed rather than broadcast, because each overlay is told + // a different rectangle. src/main.js listens for its own label + // to match: a webview listening for nothing in particular hears + // broadcasts only. + let _ = window.emit_to( + label, + FRAME_EVENT, + Placement { + x: local.x, + y: local.y, + width, + height, + animation: frame.animation, + frame_index: drawn.index, + }, + ); - // Only record the new state once the platform accepted it. Recording - // it regardless would latch a failed toggle forever, leaving - // click-through stuck in whichever mode it happened to be in. - if flipped && window.set_ignore_cursor_events(ignore).is_ok() { - ignoring = Some(ignore); + // Click-through is per-window, and a click only ever lands on + // the overlay the cursor is on. Every other overlay passes + // clicks through whatever the sprite is doing, so a click on + // one display is never swallowed by a sprite on another. + let ignore = ignore || on_overlay != Some(index); + + // Only record the new state once the platform accepted it. + // Recording it regardless would latch a failed toggle forever, + // leaving click-through stuck in whichever mode it happened to + // be in. + if ignoring[index] != Some(ignore) { + flipped = true; + if window.set_ignore_cursor_events(ignore).is_ok() { + ignoring[index] = Some(ignore); + } + } } ticks = ticks.wrapping_add(1); if tracing && (flipped || ticks % 120 == 0) { eprintln!( - "hit-test: cursor({:.0},{:.0}) scale(cursor {:.1}, window {:.1}) \ - -> local({},{}) {} click-through {}{}", + "hit-test: cursor({:.0},{:.0}) scale {:.1} -> point({},{}) \ + on overlay {} {} click-through {}{}", cursor.x, cursor.y, cursor_scale, - scale, - local_x, - local_y, + cursor_at.0, + cursor_at.1, + on_overlay.map_or(-1, |index| index as i32), if over_sprite { "HIT " } else { "miss" }, if ignore { "on" } else { "OFF" }, if flipped { " <- flipped" } else { "" }, @@ -412,35 +533,20 @@ fn main() { #[cfg(target_os = "macos")] app.set_activation_policy(tauri::ActivationPolicy::Accessory); - let window = WebviewWindowBuilder::new(app, OVERLAY_LABEL, WebviewUrl::default()) - .title("ai-buddy") - .transparent(true) - .decorations(false) - .shadow(false) - .always_on_top(true) - .visible_on_all_workspaces(true) - .accept_first_mouse(true) - .focused(false) - .resizable(false) - .skip_taskbar(true) - .visible(false) - .build()?; - - // Click-through until the cursor is proven to be over the sprite. The - // wrong default swallows a click; this one loses nothing. - window.set_ignore_cursor_events(true)?; - - // Built before the window is placed rather than after the loop + // Read before the overlays are built rather than after the loop // starts: reading which part of a display is usable means asking // AppKit, and only the main thread may do that. let (source, displays) = platform::window_source(app.handle().clone()); let start = starting_position(&source.snapshot()); - // The overlay covers the display the Character starts on, and the - // frame loop moves it to whichever display the Character reaches. - let home = display_for((start.x, start.y), &displays.read().frames) - .ok_or("no displays reported")?; - cover_display(&window, home)?; + // One overlay per display, so a Character straddling a seam is + // drawn whole. The frame loop keeps the set in step with a desktop + // that gains or loses a display. + let covered = displays.read().frames; + if covered.is_empty() { + return Err("no displays reported".into()); + } + place_overlays(app.handle(), &covered)?; // The sprite size is the idle Animation's, blown up. Animations may // declare different frame sizes, so this is what the Character is @@ -450,21 +556,14 @@ fn main() { cast.draw("idle", 0).map_or((0, 0), |drawn| drawn.art_size); eprintln!( - "overlay: display {:.0}x{:.0} at ({:.0},{:.0}) of {}; character {}; sprite {}x{}", - home.width, - home.height, - home.x, - home.y, - displays.read().frames.len(), + "overlay: {} display(s); character {}; sprite {}x{}", + covered.len(), cast.name(), sprite_width * SPRITE_SCALE, sprite_height * SPRITE_SCALE, ); - platform::configure_overlay(&window)?; - window.show()?; - - run_frame_loop(app.handle().clone(), cast, source, displays, start, home); + run_frame_loop(app.handle().clone(), cast, source, displays, start, covered); Ok(()) }) .run(tauri::generate_context!()) diff --git a/src/main.js b/src/main.js index a6df007f4..2153718fb 100644 --- a/src/main.js +++ b/src/main.js @@ -59,10 +59,20 @@ function draw(now) { async function start() { art = await window.__TAURI__.core.invoke("character"); - await window.__TAURI__.event.listen("frame", ({ payload }) => { - previous = latest; - latest = { ...payload, at: performance.now() }; - }); + // There is one overlay per display and each is told where the sprite is in + // its own coordinates, so this asks for the frames addressed to this window + // and no others. Not optional: a listener registered with no target hears + // only broadcasts, and every frame is addressed. + const overlay = window.__TAURI__.webviewWindow.getCurrentWebviewWindow(); + + await window.__TAURI__.event.listen( + "frame", + ({ payload }) => { + previous = latest; + latest = { ...payload, at: performance.now() }; + }, + { target: overlay.label }, + ); requestAnimationFrame(draw); } From f906c1ac7a3876fa40f62da9d813f0ba7eac9fd6 Mon Sep 17 00:00:00 2001 From: omesser Date: Sun, 30 Aug 2026 03:46:23 +0300 Subject: [PATCH 2/2] Send a click on the seam to the overlay under it Review fixes: the seam column's click-through went to the wrong overlay, a failed reconcile was latched as done, the emit-target comment named a mechanism that is not the real one, and the verify script's new count check failed on mirrored displays. Co-Authored-By: Claude Opus 5 --- README.md | 9 ++-- crates/core/src/overlay.rs | 40 ++++++++++++----- scripts/verify-overlay.sh | 22 ++++++--- src-tauri/src/main.rs | 92 ++++++++++++++++++++++---------------- src/main.js | 15 ++++++- 5 files changed, 115 insertions(+), 63 deletions(-) diff --git a/README.md b/README.md index e23a99875..9ee5181db 100644 --- a/README.md +++ b/README.md @@ -203,10 +203,13 @@ can answer it. Run the app, then confirm: The last three need a second display, and only a window server can answer them: 13. **A Character on a seam is whole.** Drag the sprite slowly across the - boundary between two displays and stop with it half on each. Both halves + boundary between two displays and hold it there, half on each. Both halves are drawn, and they meet — no gap between them, nothing drawn twice, and - the art is not doubled or offset at the seam. Two displays with different - scale factors is the case worth doing this on. + the art is not doubled or offset at the seam. Watch them while your hand is + still moving, not only once it stops: each overlay interpolates between the + Engine's ticks on its own clock, so a moving sprite is the only thing that + can catch two overlays disagreeing about where it is. Two displays with + different scale factors or refresh rates is the case worth doing this on. 14. **Either half can be clicked.** With the sprite straddling, click the half on each display in turn. Both pick it up, and clicking beside it on either display still reaches whatever is underneath. diff --git a/crates/core/src/overlay.rs b/crates/core/src/overlay.rs index 0ccb7872d..4d777602a 100644 --- a/crates/core/src/overlay.rs +++ b/crates/core/src/overlay.rs @@ -28,17 +28,31 @@ use crate::window_source::Rect; /// physics catches it. The nearest display is the answer there because the point /// is on its way into or out of that one. /// -/// A point on a shared edge belongs to the first display that reported it, which -/// is arbitrary and has to be: both answers are equally true. It costs nothing -/// here, because both overlays are drawing the same sprite anyway. +/// Containment is half-open, because a window is: a display 1920 wide at x=0 +/// covers columns 0 to 1919, and column 1920 is the next display's first. The +/// shared edge has to go to the display whose window is actually there, since +/// this is what decides which overlay stops being click-through — naming the +/// neighbour leaves the overlay under the cursor passing clicks through, and a +/// click on the sprite falls to whatever is beneath it. /// /// `None` only when no display was reported, which is a machine with no screen. pub fn display_index_for(point: (f64, f64), displays: &[Rect]) -> Option { displays .iter() - .enumerate() - .min_by(|(_, a), (_, b)| outside_by(point, a).total_cmp(&outside_by(point, b))) - .map(|(index, _)| index) + .position(|display| covers(point, display)) + .or_else(|| { + displays + .iter() + .enumerate() + .min_by(|(_, a), (_, b)| outside_by(point, a).total_cmp(&outside_by(point, b))) + .map(|(index, _)| index) + }) +} + +/// Whether a display's window has this point, right and bottom edges excluded. +fn covers(point: (f64, f64), rect: &Rect) -> bool { + (rect.x..rect.x + rect.width).contains(&point.0) + && (rect.y..rect.y + rect.height).contains(&point.1) } /// How far outside a rectangle a point lies, squared. Zero anywhere inside it. @@ -366,13 +380,14 @@ mod tests { fn a_fractional_display_origin_is_rounded_to_the_pixel_grid() { let sprite = place_sprite((500.0, 300.0), (128, 128), 4); + // Both halves round up, so truncating either one is a wrong answer. assert_eq!( - sprite.in_overlay(rect(100.4, 50.6, 800.0, 600.0)).x, - 336, - "436 - 100" + sprite.in_overlay(rect(100.6, 50.6, 800.0, 600.0)).x, + 335, + "436 - 101" ); assert_eq!( - sprite.in_overlay(rect(100.4, 50.6, 800.0, 600.0)).y, + sprite.in_overlay(rect(100.6, 50.6, 800.0, 600.0)).y, 121, "172 - 51" ); @@ -391,8 +406,9 @@ mod tests { ); assert_eq!( display_index_for((1920.0, 500.0), &displays), - Some(0), - "the seam itself goes to whichever display reported first" + Some(1), + "the seam column is the second display's first column, not the first \ + display's last: that is where its window starts" ); } diff --git a/scripts/verify-overlay.sh b/scripts/verify-overlay.sh index 7f07eb667..797c0f925 100755 --- a/scripts/verify-overlay.sh +++ b/scripts/verify-overlay.sh @@ -350,15 +350,28 @@ print("\nDisplays:") for d in displays: print(f" {d['w']:.0f}x{d['h']:.0f} at ({d['x']:.0f},{d['y']:.0f})") + +def bounds(r): + return (r["x"], r["y"], r["w"], r["h"]) + + print("\nChecks:") # One overlay per display, each covering its own: a Character straddling a seam # is drawn by the overlay on each side, so both halves are on screen. One window # could not do it — macOS gives each display its own Space and draws a window # spanning two of them on only one, so an overlay wider than a display is # invisible on every display but that one. -check(len(windows) == len(displays), +# +# Counted over distinct rectangles rather than raw entries, because the two +# sides are enumerated differently: displays here come from +# CGGetActiveDisplayList and the overlays from NSScreen, and a mirrored pair is +# two of the first and one of the second. One overlay is the right answer for a +# mirrored pair, so counting the rectangles is what asks the question the app +# can actually answer. +screens = {bounds(d) for d in displays} +check(len(windows) == len(screens), "one overlay window per display", - f"{len(windows)} windows, {len(displays)} displays") + f"{len(windows)} windows, {len(screens)} distinct displays") if not windows: sys.exit(1) @@ -369,11 +382,6 @@ check(all(w["onscreen"] for w in windows), "every overlay is on screen") levels = {w["layer"] for w in windows} check(levels == {3}, "every overlay is at floating level", f"levels={sorted(levels)}") - -def bounds(r): - return (r["x"], r["y"], r["w"], r["h"]) - - # The origin has to match too — a window covering the right area of the wrong # display is the same disappearance. # diff --git a/src-tauri/src/main.rs b/src-tauri/src/main.rs index 619415eec..e916c964a 100644 --- a/src-tauri/src/main.rs +++ b/src-tauri/src/main.rs @@ -18,7 +18,7 @@ mod package; mod platform; use std::collections::BTreeMap; -use std::sync::Arc; +use std::sync::{Arc, Mutex}; use std::thread; use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH}; @@ -51,16 +51,13 @@ const ALPHA_THRESHOLD: u8 = 128; /// And the Engine advances on elapsed time, so something has to advance it. const ENGINE_TICK: Duration = Duration::from_millis(16); -/// Every overlay's label: this and the index of the display it covers. +/// The label of the overlay covering the display at `index`. /// /// One window per display, so the index is both the name and the way the frame /// loop finds the overlay belonging to a display. `capabilities/overlay.json` -/// grants the same permissions to `overlay-*`. -const OVERLAY_LABEL: &str = "overlay-"; - -/// The label of the overlay covering the display at `index`. +/// grants the same permissions to every `overlay-*`. fn overlay_label(index: usize) -> String { - format!("{OVERLAY_LABEL}{index}") + format!("overlay-{index}") } /// The event carrying each `Frame` to the webview. @@ -165,32 +162,42 @@ fn build_overlay( /// display that moved or changed resolution needs. Overlays past the end of the /// list belong to displays that have been unplugged. /// +/// Every display is attempted even after one fails, and the failures are +/// returned together. Stopping at the first would leave every display after it +/// without an overlay, which is a worse desktop than the one bad display. +/// /// Main thread only; see `build_overlay`. -fn place_overlays( - app: &tauri::AppHandle, - displays: &[Rect], -) -> Result<(), Box> { +fn place_overlays(app: &tauri::AppHandle, displays: &[Rect]) -> Result<(), String> { + let mut failed = Vec::new(); + for (index, display) in displays.iter().enumerate() { let label = overlay_label(index); - match app.get_webview_window(&label) { - Some(window) => cover_display(&window, *display)?, - None => build_overlay(app, &label, *display)?, + let placed = match app.get_webview_window(&label) { + Some(window) => cover_display(&window, *display).map_err(|why| why.to_string()), + None => build_overlay(app, &label, *display).map_err(|why| why.to_string()), + }; + if let Err(why) = placed { + failed.push(format!("{label}: {why}")); } } // Labels are handed out in order, so the first missing one ends the set. for index in displays.len().. { - let Some(window) = app.get_webview_window(&overlay_label(index)) else { + let label = overlay_label(index); + let Some(window) = app.get_webview_window(&label) else { break; }; - eprintln!( - "overlay: {} has no display left to cover", - overlay_label(index) - ); - window.close()?; + eprintln!("overlay: {label} has no display left to cover"); + if let Err(why) = window.close() { + failed.push(format!("{label}: {why}")); + } } - Ok(()) + if failed.is_empty() { + Ok(()) + } else { + Err(failed.join("; ")) + } } /// The frame loop: assemble a snapshot, tick the Engine, apply the `Frame`. @@ -216,13 +223,15 @@ fn run_frame_loop( let mut engine = Engine::new(start).with_behaviors(cast.behaviors().clone()); let mut assembler = SnapshotAssembler::new(source); - // The displays the overlays cover, as setup left them. - let mut covered = covered; - // One click-through flag per overlay, `None` until that overlay's first // decision so the first tick always applies. let mut ignoring: Vec> = vec![None; covered.len()]; + // The displays the overlays cover, as setup left them. Shared with the + // main thread, which is the only place that can change what they cover + // and so the only place that knows when this is true again. + let covered = Arc::new(Mutex::new(covered)); + let mut pointer = Pointer::default(); // Click-through is invisible: nothing on screen says whether the overlay @@ -304,26 +313,30 @@ fn run_frame_loop( // A display can be plugged in, unplugged or rearranged while the // app runs, and every display needs its overlay. Posted rather than - // done here: only the main thread may build a window. Recorded once - // the post is accepted, so a rejected one is retried next tick - // rather than latched as done. + // done here: only the main thread may build a window. + // + // Recorded by the closure, on success, rather than here once the + // post is accepted. An accepted post only means the work is queued, + // and every failure inside `place_overlays` is a log line rather + // than a refusal; recording it here would latch a desktop the + // overlays never reached and leave a hot-plugged display with no + // overlay until the display list changed again. The price is that a + // reconcile that keeps failing is posted again every tick, which is + // what retrying it means. // // An empty read is ignored rather than obeyed. It is what a failed // read of the desktop looks like as well as a machine with no // screen, and tearing every overlay down costs two webviews and // their art to rebuild — for a desktop that has nothing to draw on // either way. - if !displays.frames.is_empty() && displays.frames != covered { + if !displays.frames.is_empty() && *covered.lock().unwrap() != displays.frames { let handle = app.clone(); let frames = displays.frames.clone(); - let posted = app.run_on_main_thread(move || { - if let Err(why) = place_overlays(&handle, &frames) { - eprintln!("overlay: {why}"); - } + let placed = Arc::clone(&covered); + let _ = app.run_on_main_thread(move || match place_overlays(&handle, &frames) { + Ok(()) => *placed.lock().unwrap() = frames, + Err(why) => eprintln!("overlay: {why}"), }); - if posted.is_ok() { - covered = displays.frames.clone(); - } } // The Engine names an Animation and how long it has been playing; @@ -401,10 +414,11 @@ fn run_frame_loop( // is what leaves a Character on a seam whole instead of clipped // to one display. // - // Addressed rather than broadcast, because each overlay is told - // a different rectangle. src/main.js listens for its own label - // to match: a webview listening for nothing in particular hears - // broadcasts only. + // Addressed rather than emitted to all, because each overlay + // is told a different rectangle. src/main.js has to name its + // own label to match: an untargeted listener hears every emit, + // addressed elsewhere or not, and would draw whichever + // rectangle arrived last. let _ = window.emit_to( label, FRAME_EVENT, diff --git a/src/main.js b/src/main.js index 2153718fb..3ba264dae 100644 --- a/src/main.js +++ b/src/main.js @@ -19,6 +19,14 @@ let art = {}; // The hit-test runs against the unlagged position, so the click-through // rectangle leads what is on screen by one sample — a pixel or two at walking // speed, and always in the direction of travel. +// +// ponytail: every overlay interpolates on its own clock — its own arrival +// times and its own frame phase — so the two halves of a sprite on a seam can +// round a point apart while it is moving. At rest they cannot, there being +// nothing to interpolate. The upgrade is to carry the Engine's own timestamp +// on the frame and solve for it, which needs a per-webview offset between that +// clock and `performance.now()`; worth it if README item 13 ever shows a +// shimmer at the seam under a drag. let previous = null; let latest = null; @@ -61,8 +69,11 @@ async function start() { // There is one overlay per display and each is told where the sprite is in // its own coordinates, so this asks for the frames addressed to this window - // and no others. Not optional: a listener registered with no target hears - // only broadcasts, and every frame is addressed. + // and no others. Not optional, and not for the reason it looks like: a + // listener registered with no target is an `Any` listener, and tauri hands an + // `Any` listener every emit, addressed elsewhere or not. Without this each + // overlay would also hear its neighbours' rectangles and draw whichever + // arrived last, so every display would show the same display's half. const overlay = window.__TAURI__.webviewWindow.getCurrentWebviewWindow(); await window.__TAURI__.event.listen(