Skip to content

Richer AI interactions: what the model API and the Harness can carry #144

Open
4 of 5 issues completed

Description

@omesser

Related: #16 Harness attachment, #17 chat surface, #15 MCP tool surface, #119 speech bubbles, shipped as #127 and #137. ADR-0004 keeps the model outside the frame loop, ADR-0008 keeps one Harness session, and ADR-0003 rules out an Executor. Nothing below changes that.

This is an umbrella for triage: each accepted item becomes its own sub-issue. Triage verdicts are in the verdict comment and the Architect answer.

Implementation status

1. Structured replies

2. Sensing in the Character Prompt, V1

3. Memory write-backs and protocol, V1

4. Live narrator of Harness work, V2

5. Streaming dialogue into the bubble, V1 p0

6. Reaction repertoire for the verbs, V1

Why

A turn today is thin: one wake produces one BehaviorProposal { behavior, dialogue }, parsed from a plain-text reply, a name or say: line. The buddy can act or say one thing, once, and the model sees little of the moment it is reacting to. The pipes for more already exist: the MCP tools speak, play_behavior, list_windows, describe_screen, recall and remember, the speech bubble, and the one-session design. They are just not carrying much yet.

Original suggestions, each grounded in an existing seam

1. Structured replies instead of say: parsing

Every provider behind the Completer supports structured output, through JSON schema or tool calling. Replace the name-or-say: convention with one validated object: behavior, dialogue, an optional short play list, an optional mood, and an optional pacing hint such as "quiet for a while". Misparses stop being silent behavior refusals, and every later item on this list gets a field instead of a convention. Seam: ModelDirector::as_speech/wake and the Completer request body.

Triage: narrowed. The blanket swap is dropped. Keep the item for a pacing hint, and later remember, when a visible feature needs it.

2. Sensing in the Character Prompt

Context carries the frontmost app and recent windows, but the prompt spends little of it, and describe_screen/list_windows exist for the Harness only. Fold a denylisted window summary, time of day, and a couple of recall lines into ambient wakes, so unprompted remarks land on what is actually happening, like "that build has been red for twenty minutes", rather than generic filler. The denylist already gates what may enter any sensing result, and the same gate applies here.

3. Memory write-backs from turns

remember exists as a tool, but only a Harness can call it. Let a structured reply from item 1 carry an optional remember fact, so the HTTP stand-in also builds continuity across sessions. The buddy that knows you prefer the terminal at 2am is the buddy that feels alive. Bound writes, since the Memory file already has limits, and keep them visible in the Action Log #16.

Triage: writing facts down isn't enough. Spec the protocol first in #1374, with the Harness as the main reader.

4. The buddy as a live narrator of Harness work

Once #16 attaches a real session, the Harness can call speak/play_behavior mid-turn, not just at the end: narrate a long task while it runs, celebrate when tests go green, react when they fail, sit and watch while something compiles. This is the thinking indicator #119 generalized from "a turn is in flight" to "what the turn is doing". It needs nothing new below the tool surface. It is a prompt-side convention for the attached session, plus perhaps two or three named lifecycle Behaviors a Character may declare, working, succeeded and failed, with silent fallbacks, the same contract as climb.

5. Streaming dialogue into the bubble

The Completer returns one String, and the bubble waits for all of it. Providers stream. Stream tokens into Frame.dialogue so the bubble fills as the model talks and talk plays while it does. That is the difference between a companion speaking and a companion serving a document. #119 deferred this deliberately, and the bubble machine #137 now has the state seams to take it.

6. Reaction repertoire for the verbs

A Poke always plays react. The reactive wake it fires can already reach the model, but the reply arrives too late for the reflex. Keep the instant react, and let the turn add a follow-up quip or a second Behavior chosen from what actually happened, since happened is already in Context. Poked while asleep earns a different line than poked mid-walk. Recency suppression already exists to stop repeats.

Not in scope

  • Desktop control: none of any kind, ADR-0003 stands.
  • A second model endpoint: ADR-0008 stands. Chat, ambient life, and everything above share the one session.
  • Speaking under Do Not Disturb: Frame.dialogue is already gated, and new channels inherit the gate.

Suggested slicing

Item 1 first, since it is the substrate the others attach fields to. Items 2 and 6 are stand-in-friendly and cheap after it. Items 3 to 5 ride #16's Harness attachment and can wait for it.

Activity

  1. added
    enhancementNew feature or request
    ready-for-agentFully specified, ready for an AFK agent
    needs-triageMaintainer needs to evaluate this issue
    and removed
    ready-for-agentFully specified, ready for an AFK agent
    on Sep 1, 2026
  2. moved this from Todo to Human Triage in Fidgeton Sep 3, 2026
  3. added this to the V1 milestone on Sep 6, 2026
  4. added
    p1Next up: the default for work that should happen
    p0Drop everything: broken for users, or blocking
    and removed
    p1Next up: the default for work that should happen
    p0Drop everything: broken for users, or blocking
    on Sep 9, 2026
  5. omesser commented on Oct 6, 2026

    @omesser
    OwnerAuthor

    Sliced inbound harness wake into concrete bugfix: #1356 (between-turn session/update drop in acp_wire serve — stream/thought/speak never wake Director). Cron alone does not make Instance Prompt reminders work until that lands.

    — Cursor agent (Coder), on @omesser's behalf.

  6. omesser commented on Oct 6, 2026

    @omesser
    OwnerAuthor

    Verdict - calls for discussion

    1. Structured replies instead of say: parsing - Challenge. what behavior/UX improvement will we get from this? the downside is risk - structured replies with say will diverge chat contents from speech bubble more often, will possible bring more failure modes of breaking format on the harness/AI model side. What will the user gain? by this? Give concrete examples and Scenarios

    2. Sensing in the Character Prompt - Good idea! Open an issue for it. We'll implement that for V1 milestone even. Propose a summarized format - it should be dense and very token efficient to not cause failures for small models; short/concise/compacted and ok to be lossy, the model can still call the MCP server for details and it should remain meaningful.

    3. Memory write-backs from turns - Good! let's spec this out. just writing it down is not enough. We need to figure out the protocol of using the memory file (unused until now) - need to pass it to the AI at the very least

    4. The buddy as a live narrator of Harness work - this is great. create an issue for it. but let's push this to V2 milestone for now. it's also affected/blocked by 1 IMHO

    5. Streaming dialogue into the bubble - yes. accepted. open an issue and spec this out for V1. this is now p0.

    6. Reaction repertoire for the verbs - give 3 concrete examples of how that would change current behavior

  7. omesser commented on Oct 6, 2026

    @omesser
    OwnerAuthor

    Answering the verdict in your numbering. I'm reading the whole triage through one lens: idle behavior driven by the authored personality. Items 2 and 6 feed that directly, so they're the cheap wins. Item 4 is Harness-side, so V2 is right. I checked the claims below against current main (ab02e80) and corrected the original write-up where the code disagrees with it.

    New sub-issues, all on the board in Todo: #1373 (2, V1), #1374 (3, spec, V1), #1375 (4, V2), #1376 (5, V1, p0).

    1) Structured replies. Conceded, mostly.
    A substrate-first swap has no user gain of its own, and both risks you named are real: another format for small and local models to break, and another place for chat and bubble to drift apart. Checking the code also removed two of the gains the write-up claimed:

    • Acting and speaking in one wake already works. parse_proposal takes a Behavior name on its own line (or name | line) and treats the rest as speech. The Engine plays the Behavior and speaks the line either way.
    • Misparses aren't silent today. An undeclared name is spoken as speech and logged as near_miss in the Action Log. An empty reply logs failed, and the Static Director fills in. A Behavior the State gate refuses still has its line spoken. (One wart: a near miss speaks the raw reply, name included.)

    That leaves one concrete scenario, the pacing hint. You say "shush for an hour" or "quiet, I'm in a meeting", and the Director actually holds to it, because the reply can tell Pace not to wake before a given time. Today Pace grows on its own schedule and resets on every reactive wake, so the character can agree and then pipe up again a few minutes later. This is also the portable home for the time-based Instance Prompt examples #1354 left out ("a follow-up that lets the Director set its next wake"), and it works for a Model API Director, not only a Harness. It isn't a timer feature, just "don't wake me before X".

    Mitigations, if and when it lands:

    • Parse leniently and fall back to today's convention, so a model that ignores the field is no worse off.
    • With a Harness, use a tool call rather than JSON in text.
    • Keep the single Frame.dialogue string feeding both the bubble and the Chat row, as it does today, so the two can't drift.

    Proposal: drop the blanket JSON swap. Keep item 1 open, narrowed to "add a pacing hint (and later remember) when a visible feature needs it", and let the specs for 3 and 5 decide its shape.

    2) Sensing. Agreed, V1 → #1373.
    One dense, lossy line of about 40–60 tokens replaces today's separate time: and open: lines:

    now Tue 14:52 · front VS Code "main.rs — fidget" 41m · before Chrome "PR #1354", iTerm2 "cargo test" 3m · on Slack · idle 0m · mem: prefers terminal late
    

    Rules:

    • Titles are truncated, durations are in minutes, and the line carries at most 3 recent apps and 2 Memory lines.
    • Titles and other apps' names appear only with the Window and application names consent. The DenyList filters first.
    • A field is left out entirely when its consent is off or it has no value.

    Lossy is fine here: a Harness can still call list_windows / describe_screen / recall for detail.

    3) Memory. Agreed, writing it down isn't enough → #1374.
    The spec answers three questions:

    • Read: what slice of Memory goes into each wake. Today no wake gets any of it. The file has no size guard on purpose because it's user-owned, so the bound belongs at the prompt seam: top N lines.
    • Write: the Harness remember tool now. A reply field for Model API Directors comes only once item 1 has one.
    • Visibility: each write shows the fact itself in the Action Log, and you can edit or forget it. Today a remember call shows only as a tool-call title and status.

    Reminders stored as dated Memory facts stay the later follow-on #1356 already names. They're not in this scope.

    4) Narrator. Agreed, V2 → #1375.
    It depends on 1 only for named lifecycle Behaviors (working / succeeded / failed). Narration through speak / play_behavior mostly rides #1356.

    5) Streaming. Accepted, V1, p0 → #1376.
    Both transports already read deltas: ACP agent_message_chunk in acp_wire::note_update, and SSE in model.rs read_frames. Both feed one shared Answer, and both already stream thoughts live to the Thinking row. Only the answer is held until the end, so the Harness isn't meaningfully cheaper to do first; one answer-delta door serves both transports. The real work is holding a Behavior-name line back so it never flashes in the bubble, plus the bubble machine and cancel. Streaming one string to both bubble and Chat row also leaves less room for drift.

    6) Reaction repertoire. Three before/afters.
    Today a Poke always plays an instant react. The reactive wake then reaches the model with what just happened: poked, state, recent and standing on. What it never says is what the sprite was doing before the verb. The change: keep the instant react, and add that to the wake (state before the verb, the Behavior in progress, a poke streak) so the follow-up line or Behavior fits the moment.

    • (i) Poked while asleep.
      • Today: state is read after the tick, and a verb wakes a sleeping sprite into a fall, so the model sees state: falling or idle, never asleep, and answers with a generic quip.
      • After: react (startle), then a groggy "…five more minutes", and it goes quiet so it nods off again.
    • (ii) Poked three times in a row.
      • Today: recent holds the last three Behaviors the Engine played from proposals, but the Engine's own react is never recorded there, and a fresh Poke takes over a call that's still thinking (ADR-0016). Nothing tells the model this is the third poke.
      • After: a streak count in the wake makes it escalate: "hey", then "what?", then it turns its back and walks off, each step in the Character's voice.
    • (iii) Pulled off the wall mid-climb.
      • Today: the wake says picked up / state: held and nothing about the climb.
      • After: "I was almost at the top!", then it climbs again or sulks.
      • A plain Poke mid-climb doesn't make it fall: it pauses on the wall for the cooldown, then climbs on. A follow-up there should be a gesture or a line, not a move.

    Not a gain, because it already works today: poked while perched, it can already remark on the app it's standing on, since standing on: is in every wake.

    — Cursor agent (Architect), on @omesser's behalf.

  8. omesser commented on Oct 6, 2026

    @omesser
    OwnerAuthor
    1. memory:

    The spec answers three questions:

    Read: what slice of Memory goes into each wake. Today no wake gets any of it. The file has no size guard on purpose because it's user-owned, so the bound belongs at the prompt seam: top N lines.
    Write: the Harness remember tool now. A reply field for Model API Directors comes only once item 1 has one.
    Visibility: each write shows the fact itself in the Action Log, and you can edit or forget it. Today a remember call shows only as a tool-call title and status.

    Read - The main consumer here should be the agent/harness and we need to make this accessible. either read via mcp (recall) OR hooking it up into the harness existing memory subsystem - needs further research

  9. omesser commented on Oct 6, 2026

    @omesser
    OwnerAuthor

    decision update:

    6. Reaction repertoire for the verbs

    OK, looks good and very lean /easy to implement. but we need to decide about the recency - do we send last N states/actions or set of states/actions that happened over the last 10-20 seconds? with/without timestamps? and what else?

  10. added
    umbrellaAn over-arching issue containing references to several issues/PRs for implementation/research phases
    on Oct 6, 2026
  11. omesser commented on Oct 6, 2026

    @omesser
    OwnerAuthor

    Answering your two latest notes — memory Read, and item 6 recency. Checked against current main.

    3) Memory Read — reframed

    Agreed: the primary consumer is the agent/Harness, not stuffing Memory into every Model API wake. Updated #1374 accordingly.

    Spec research (explicit Done-when there):

    1. MCP recall already exists (dispatch → MemoryManifest::recall). Make sure the Harness is actually prompted / expected to use it when facts would help.
    2. Hook Fidget Memory into the Harness's own memory subsystem if one exists (Claude Code / ACP session memory). Needs further research.

    Ambient Character Prompt top-N stays a secondary, optional, bounded path for Model API Directors (and a couple of lines in the sensing line from #1373) — demoted vs Harness accessibility, not dropped. Write + Visibility unchanged from the earlier scope.

    6) Reaction repertoire — recency decision

    OK on lean / easy. Concrete recommendation (not a menu of equals). Opened #1380 (V1, Todo) to implement.

    Today's wake history (why last-N alone is wrong):

    Field What it is today Gap for the three scenarios
    what just happened Verb word from live.happened (prompt::follow_up / happened_word) Fine for "poked"; says nothing about before
    recent Last 3 Behavior names from proposals the Engine played (director::REMEMBERED + remember in frame_loop when frame.behavior is set) Misses Engine-owned react; can be hours-stale across quiet minutes
    state Post-tick frame.state (frame_loop: "After the tick so a Throw is already Falling…") Poke-while-asleep often shows falling/idle, never asleep
    Behavior in progress Frame.playing_behavior exists every tick of a named Behavior Not in the wake prompt
    Streak / time window None. since_state is only Static Director pacing (STATE_BOUND = 90s) No "third poke" signal

    Instant react on Poke/Summon stays (engine: "A Poke is answered…" → Primitive::React). That path does not set frame.behavior, so it never enters recent.

    Recommend this lean default for the reactive wake:

    1. Keep instant react.
    2. Add pre-verb state (snapshot before the tick / from live.last_state) — required for asleep.
    3. Add Behavior-in-progress when playing_behavior is set (climb / walk) — required for mid-climb and pull-off.
    4. Add a poke/verb streak count (new counter; not proposal-recent).
    5. For "what else happened": a ~10–15s window of distinct state/Behavior transitions, not last-N proposal Behaviors alone — last-N misses react and goes stale across quiet time.
    6. No wall-clock timestamps. Optional relative ages only if the window/streak needs them (2s ago). Hard cap ≤5 events / ≤~40 tokens.

    That covers the three accepted before/afters without bloating the Character Prompt. Shape details and Done-when live on #1380; say if you want a different window or cap before implement.

    — Cursor agent (Architect), on @omesser's behalf.

  12. omesser commented on Oct 8, 2026

    @omesser
    OwnerAuthor

    Item 6 (reaction repertoire) shipped via #1380 / #1425. The instant react is kept, and the reactive wake now carries was: (what the gesture found), a streak suffix on what just happened, doing:, and a 12s lately: window capped at 5 steps, with no timestamps.

    One change from the plan here: doing: and lately: also go on ambient follow-ups, not only reactive wakes. That's because a gate would only add a branch, and on ambient wakes those lines are usually empty anyway. This umbrella stays open for the other items.

    — Cursor agent (Coder), on @omesser's behalf.

  13. omesser commented on Oct 29, 2026

    @omesser
    OwnerAuthor

    Direction change for item 4, the live narrator in #1375, and for the speech work under it: see #1935.

    — Cursor agent (Architect), on @omesser's behalf.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestneeds-triageMaintainer needs to evaluate this issuep1Next up: the default for work that should happenumbrellaAn over-arching issue containing references to several issues/PRs for implementation/research phases

    Projects

    • Status
      In Progress

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions