Skip to content

feat(shell): Cue each click on the Character, in sight and in sound #277

Closed

Description

@omesser

Why

Two of the three ways you can click the buddy do nothing you can see or hear.

Gesture Verb What it does today
Left click Poke Plays the react Required Animation (engine.rs:831)
Double click Poke + Summon Wakes a sleeping sprite. Nothing else
Right click Menu Wakes a sleeping sprite. Nothing else
Pickup Grab The sprite follows the cursor. No announcement
Drop none Nothing at all — see below, there is no drop verb

engine.rs:2232 asserts that silence on purpose:
a_summon_or_a_menu_wakes_the_sprite_without_moving_it ends with
assert_eq!(addressed.animation, "idle", "and nothing is played for it"). Both
verbs are accepted and shown nothing, because the surfaces they open — the chat
surface (#17) and the tray menu (#18) — were the response.

So a double-click that lands is indistinguishable from one that missed the
sprite. There is no audio anywhere in the tree: git grep -i "audio\|sound"
over *.rs returns nothing but the word "sound" in a comment. The buddy is
silent, and most of the ways you can touch it are also invisible.

Each interaction should announce itself — its own visual cue and its own sound,
so the gestures are told apart by feel rather than by their consequences.

Why procedural, and not three new animations

This cannot be Character art, and that is a constraint rather than a preference.

ADR-0002 fixed the Required Animation Set at eight because "every required
animation is a tax on every Character that will ever exist". engine.rs:2226
records the same reasoning for the verb set: five verbs, so "no Character ever
has to grow another Animation for a sixth". Three per-interaction cues authored
as art would be six to nine new required frames per Character, retroactively,
for every package that already exists.

Procedural is the way out. The Shell draws and synthesizes the cue; no Character
Package changes, and no Primitive is added.

The precedent is already in the tree. src/bubble.js draws speech bubbles and a
thinking indicator — Shell-owned visuals, over the sprite, in the webview, no
Character art involved. A cue is the same kind of thing.

Audio needs no dependency either. The sprite renders in a Tauri webview
(src/index.html, src/main.js), so the Web Audio API is already there. An
oscillator plus a gain envelope is a few lines and no crate, no sample files,
nothing to ship per Character. Procedural in the literal sense: the sound is
computed, not loaded.

The seam

The frame event already carries one-tick pulses, which is exactly the shape a
cue needs. src/main.js:297 on dialogue: "Dialogue rides exactly one tick, and
latest keeps only the newest placement" — the bubble machine latches the pulse
where every delivery is seen.

A cue pulse rides the same path:

Step Where
Decide the cue, from the tick's verbs and the Dragged transitions crates/core/src/engine.rs, beside addressed at :888
Add it to the per-instance payload src-tauri/src/main.rs:223 SpritePlacement, :330 Placed
Fill it in src-tauri/src/frame_loop.rs:1188
Latch it and start the cue src/main.js, alongside view.bubbles.event(owned)

The Engine decides which cue, not the webview. It is the only place that knows
both the verbs and the State transitions, it is pure, and the decision is
therefore testable without a window. The webview is told which cue to play and
plays it.

The cue vocabulary

Five cues, one per interaction. The verb set is fixed at five (#6), so the
vocabulary is the verb set and nothing has to be invented for a sixth.

Interaction Cue fires on
Left click Verb::Poke
Double click Verb::Summon
Right click Verb::Menu
Pickup Entering State::Dragged
Drop Leaving State::Dragged

A Throw is a drop that was moving. It gets the drop cue, played harder rather
than a sixth sound — say the same envelope with more gain and a higher pitch. If
that reads as too subtle in the hand, it becomes its own cue; that is a tuning
call, not a structural one.

Decided: a double-click emits Summon, not Poke and Summon

crates/core/src/input.rs:204 currently returns vec![Verb::Poke, Verb::Summon]
for the second click of a pair. That changes to vec![Verb::Summon], and the
comment above it — which argues for keeping the Poke — goes with it.

Two consequences the change has to carry, or it is a regression:

Summon has to start marking the buddy as addressed. engine.rs:834 sets
addressed = true inside the Verb::Poke branch, and nowhere else except the
cursor dwell at :644. addressed is what makes the next Director wake
reactive rather than ambient. Drop the Poke from a double-click and, as the code
stands, a double-click stops counting as the user addressing the buddy — the
loudest interaction there is would become invisible to the Director. Summon
must set addressed in the same way. Same for Menu, on the same reasoning:
engine.rs:2226 already calls both "the user reaching for the sprite".

Summon should play the react animation that the Poke used to bring. Today
the second click's Poke is what plays Primitive::React (engine.rs:831).
Without it, a double-click would visibly do less than a single click. Summon
plays React itself, so the sprite still notices you.

The first click of a pair still emits a Poke, and that cannot be fixed here.
A double-click is two releases: the first emits [Poke] at input.rs:206
before anything knows a second is coming, the second now emits [Summon].
Suppressing that first Poke would mean holding every left click for
DOUBLE_CLICK_MS (400 ms, input.rs:38) to see whether a partner arrives,
which makes every single click feel broken. So a double-click is a Poke cue
followed ~200 ms later by a Summon cue.

Handle it at the cue layer, not the verb layer: the Summon cue cancels a Poke
cue still in flight.
The verb stream stays honest, the ear hears one cue, and
the cancel is a few lines in the webview where both cues live. This is why the
cue machine needs to own its own timing rather than fire and forget.

A drag emits no Poke today — that part already holds

Worth stating because it looks like a bug and is not: a press that becomes a
drag never emits a Poke. input.rs:177-181 moves Phase::Pressed straight to
Phase::Grabbing and returns vec![Verb::Grab], and the Poke only comes from
the release branch at :193. So pickup already does not fire the click cue.

Two mechanics the pickup and drop cues have to respect:

Verb::Grab arrives every tick, not on the press. engine.rs:67 says so
outright: "present on every tick the sprite is held". A cue keyed on the verb
being present would fire every frame of a drag. The pickup cue fires on the
transition into State::Dragged.

There is no drop verb, and adding one is not allowed. input.rs:216
returns Vec::new() when a grab is released without enough speed to be a
Throw — the Engine drops the sprite because the Grab stopped arriving, so
nothing announces it. The verb set is fixed at five by #6 and ADR-0002, so a
sixth Drop verb is exactly what the project forbids. The drop cue therefore
keys on leaving State::Dragged, which is also what makes drop and throw
distinguishable: both leave Dragged, and only the throw carries a velocity.

Decided

Settled, not open:

  1. Audio is on by default. Mute lives in Settings.
  2. Do Not Disturb silences the audio cue and keeps the visual one. Do Not Disturb: the buddy stays, and stops starting things #84 is
    "quiet but not gone", and a visual cue cannot embarrass anyone in a meeting.
  3. The cue vocabulary is global. No per-Character pitch or color, no new
    manifest field. A Character-declared vocabulary would be its own issue and
    its own ADR conversation.

Acceptance

  • Left click, double click, right click, pickup, and drop each produce a
    distinct visual cue and a distinct sound.
  • A double-click emits Verb::Summon alone. input.rs:204 no longer
    returns a Poke with it.
  • Summon and Menu set addressed, so a double-click and a right-click
    still make the next Director wake reactive. A test covers it — this is the
    regression the verb change would otherwise cause.
  • Summon plays Primitive::React, so a double-click is not visibly less
    than a single click.
  • The Summon cue cancels a Poke cue still in flight, so one double-click is
    heard as one cue and not two.
  • Every cue is edge-triggered. Holding the button through a drag fires the
    pickup cue once, not once a tick.
  • The drop cue fires on leaving State::Dragged, for a still drop and for a
    throw alike. No sixth verb is added.
  • No Character Package changes. No new Primitive, no new Required Animation,
    no new manifest field.
  • No new dependency for audio.
  • Audio is on by default and mutable in Settings, and the mute takes effect
    without a restart.
  • Do Not Disturb silences the audio cue and leaves the visual one.
  • A hidden Character (Do Not Disturb: the buddy stays, and stops starting things #84 hidden) produces no cue of either kind.
  • The verb and state-transition mapping is unit-tested, in input.rs for
    the verbs a gesture emits and in engine.rs for addressed and the
    Dragged transitions. The drawing and the sound are not testable from
    Rust, and the issue does not pretend otherwise.
  • Works on macOS and on Linux X11.

Out of scope

  • The chat surface (Chat surface #17) and the tray menu (Tray and settings #18). This is the cue that the
    gesture registered, not the surface it opens.
  • Per-Character cue vocabulary, and any Character Manifest field for one.
  • A separate throw cue, unless the shared drop cue reads wrong in the hand.
  • Windows. docs/SPEC.md:569 keeps the arms stubs.

Related: #6 fixes the verb set at five, #17, #18, #84 for Do Not Disturb,
ADR-0002 for why the cue cannot be Character art, ADR-0006 for the pixel-art
scaling a procedural visual has to sit beside without clashing.

All file:line references are pinned to main at 873c05f1.

— Claude Code, on @omesser's behalf.

Activity

  1. added
    enhancementNew feature or request
    needs-triageMaintainer needs to evaluate this issue
    ready-for-agentFully specified, ready for an AFK agent
    and removed
    needs-triageMaintainer needs to evaluate this issue
    on Sep 3, 2026
  2. omesser commented on Sep 3, 2026

    @omesser
    OwnerAuthor

    Body updated with the decisions from review: audio on by default with a mute in Settings, Do Not Disturb silences the audio cue and keeps the visual one, the cue vocabulary is global, and a double-click emits Verb::Summon alone.

    Tracing the drag request turned up two things worth flagging before implementation:

    • Summon and Menu do not set addressed. engine.rs:834 sets it inside the Verb::Poke branch only. Removing the Poke from a double-click would stop a double-click counting as the user addressing the buddy, so the next Director wake would be ambient rather than reactive. Both verbs now have to set it, and Summon also has to play Primitive::React or a double-click does visibly less than a single click.
    • There is no drop verb, and a sixth is not allowed. input.rs:216 returns no verb when a grab is released too slowly to be a Throw — the Engine drops the sprite because the Grab stopped arriving. So the pickup and drop cues key on entering and leaving State::Dragged rather than on verbs, which keeps the verb set at five per The five interaction verbs #6 and ADR-0002.

    One constraint that cannot be designed away: the first click of a double-click has already emitted its Poke before anything knows a second is coming. Suppressing it would mean holding every left click for the 400 ms double-click window. The Summon cue therefore cancels a Poke cue still in flight, at the cue layer.

    Relabelled ready-for-agent.

    — Claude Code, on @omesser's behalf.

  3. omesser commented on Sep 3, 2026

    @omesser
    OwnerAuthor

    Cue design approved. The set is now fixed, and docs/design/cues.html in #282 is the spec — every duration, easing curve, frequency and envelope in its cues CSS block and Cues script block is signed off.

    Cue Visual Sound
    Poke one ring outward, 24→82 px, 260 ms sine 660→600 Hz, 140 ms decay
    Summon two rings inward, 90 ms apart, 430 ms triangle C5 then G5, second at 110 ms
    Menu thin filled ring segment, 4.5 px band in a 58 px circle, sweeping 360° two square-wave ticks, 320 then 240 Hz, low-passed
    Pickup four dashes rising 30 px from the feet, staggered 25 ms sine sweep 280→720 Hz
    Drop flat puff at the feet spreading to 92 px, 280 ms sine 160→55 Hz, 160 ms decay
    Throw the same puff to 130 px, 340 ms the same thud, 1.6× gain, 200 ms decay

    The grammar is direction: Poke goes out, Summon is Poke reversed and doubled, Menu goes around, Pickup goes up, Drop goes sideways. That is what lets the cues be told apart without being loud. Every stroke is white on ink, the pair src/main.css already uses for the speech bubble, so the overlay gains no new colors.

    The Menu cue went through one revision: it began as a thin three-quarter stroked arc, which read as low-resolution because a 2.5 px SVG stroke landed between pixels. It is now a conic gradient masked to a thin donut and snapped to whole pixels, which is crisp at any DPI.

    Prerequisites #278 and #280 are merged. The implementation is in flight as PR 3 of 3.

    — Claude Code, on @omesser's behalf.

  4. moved this from Todo to In Progress in Fidgeton Sep 3, 2026
  5. moved this from In Progress to Done in Fidgeton Sep 3, 2026
  6. added this to the V1 milestone on Sep 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestready-for-agentFully specified, ready for an AFK agent

    Projects

    • Status
      Done

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions