Repository navigation
Description
Related: #16 Harness attachment, #17 chat surface, #15 MCP tool surface, #119 speech bubbles, shipped as #127 and #137. ADR-0004 keeps the model outside the frame loop, ADR-0008 keeps one Harness session, and ADR-0003 rules out an Executor. Nothing below changes that.
This is an umbrella for triage: each accepted item becomes its own sub-issue. Triage verdicts are in the verdict comment and the Architect answer.
Implementation status
1. Structured replies
ReplaceDropped in triage: no user gain on its own, and more format risk for small models.say:parsing with one validated JSON object- Before number: measure the reply-contract failure rate on local models Measure the reply-contract failure rate on a local model, as the before number for #144 #175 test: Measure the reply-contract rate on local models, and what it exposed #230
- Pacing hint: "don't wake me before X", parsed leniently with today's convention as fallback. No issue yet, add it when a visible feature needs it.
- Spike: MCP Expression tools as the Behavior proposal path, V2 spike(v2): Prefer MCP Expression tools for Director Behavior proposals #1495
2. Sensing in the Character Prompt, V1
- Situation line: one dense, lossy line replaces
time:andopen:Sensing in the Character Prompt: one dense, lossy situation line #1373 feat(director): Send the desktop as short labelled lines #1377 - Memory lines: the
mem:field waits on the Memory spec Spec: Memory protocol — what each wake reads, how turns write, what the user sees #1374
3. Memory write-backs and protocol, V1
- Spec: what each wake reads, how turns write, what the user sees. Harness is the primary reader, through
recallor the Harness memory subsystem Spec: Memory protocol — what each wake reads, how turns write, what the user sees #1374 - Implementation slices: filed from the Spec: Memory protocol — what each wake reads, how turns write, what the user sees #1374 spec once it lands
- Model API write path: a
rememberreply field, blocked on item 1
4. Live narrator of Harness work, V2
- Inbound Harness wake: between-turn output no longer dropped Harness-initiated between-turn output is dropped — support inbound harness wake #1356 fix(harness): Hold between-turn Harness updates and permission asks instead of dropping them #1370 feat(harness): Support inbound harness wakes with agent text visibility and Pace participation #1383 docs(harness): Document inbound-wake contract for Character Prompt / harness attach #1387
- Narrator:
speakandplay_behaviorduring a turn, rate rule, lifecycle Behaviors The buddy as a live narrator of Harness work #1375
5. Streaming dialogue into the bubble, V1 p0
- Stream end to end: Harness to Director to bubble and Chat Spec: stream dialogue e2e (Harness → Director → bubble + Chat) #1376 feat(director): Stream dialogue e2e (bubble + Chat) #1379
6. Reaction repertoire for the verbs, V1
- Richer reactive wake:
was:, streak,doing:and a 12slately:window Reaction repertoire: enrich reactive wake with pre-verb state, streak, and short recency window #1380 feat(director): Tell a wake what the verb found, the streak, and the last 12s #1425
Why
A turn today is thin: one wake produces one BehaviorProposal { behavior, dialogue }, parsed from a plain-text reply, a name or say: line. The buddy can act or say one thing, once, and the model sees little of the moment it is reacting to. The pipes for more already exist: the MCP tools speak, play_behavior, list_windows, describe_screen, recall and remember, the speech bubble, and the one-session design. They are just not carrying much yet.
Original suggestions, each grounded in an existing seam
1. Structured replies instead of say: parsing
Every provider behind the Completer supports structured output, through JSON schema or tool calling. Replace the name-or-say: convention with one validated object: behavior, dialogue, an optional short play list, an optional mood, and an optional pacing hint such as "quiet for a while". Misparses stop being silent behavior refusals, and every later item on this list gets a field instead of a convention. Seam: ModelDirector::as_speech/wake and the Completer request body.
Triage: narrowed. The blanket swap is dropped. Keep the item for a pacing hint, and later remember, when a visible feature needs it.
2. Sensing in the Character Prompt
Context carries the frontmost app and recent windows, but the prompt spends little of it, and describe_screen/list_windows exist for the Harness only. Fold a denylisted window summary, time of day, and a couple of recall lines into ambient wakes, so unprompted remarks land on what is actually happening, like "that build has been red for twenty minutes", rather than generic filler. The denylist already gates what may enter any sensing result, and the same gate applies here.
3. Memory write-backs from turns
remember exists as a tool, but only a Harness can call it. Let a structured reply from item 1 carry an optional remember fact, so the HTTP stand-in also builds continuity across sessions. The buddy that knows you prefer the terminal at 2am is the buddy that feels alive. Bound writes, since the Memory file already has limits, and keep them visible in the Action Log #16.
Triage: writing facts down isn't enough. Spec the protocol first in #1374, with the Harness as the main reader.
4. The buddy as a live narrator of Harness work
Once #16 attaches a real session, the Harness can call speak/play_behavior mid-turn, not just at the end: narrate a long task while it runs, celebrate when tests go green, react when they fail, sit and watch while something compiles. This is the thinking indicator #119 generalized from "a turn is in flight" to "what the turn is doing". It needs nothing new below the tool surface. It is a prompt-side convention for the attached session, plus perhaps two or three named lifecycle Behaviors a Character may declare, working, succeeded and failed, with silent fallbacks, the same contract as climb.
5. Streaming dialogue into the bubble
The Completer returns one String, and the bubble waits for all of it. Providers stream. Stream tokens into Frame.dialogue so the bubble fills as the model talks and talk plays while it does. That is the difference between a companion speaking and a companion serving a document. #119 deferred this deliberately, and the bubble machine #137 now has the state seams to take it.
6. Reaction repertoire for the verbs
A Poke always plays react. The reactive wake it fires can already reach the model, but the reply arrives too late for the reflex. Keep the instant react, and let the turn add a follow-up quip or a second Behavior chosen from what actually happened, since happened is already in Context. Poked while asleep earns a different line than poked mid-walk. Recency suppression already exists to stop repeats.
Not in scope
- Desktop control: none of any kind, ADR-0003 stands.
- A second model endpoint: ADR-0008 stands. Chat, ambient life, and everything above share the one session.
- Speaking under Do Not Disturb:
Frame.dialogueis already gated, and new channels inherit the gate.
Suggested slicing
Item 1 first, since it is the substrate the others attach fields to. Items 2 and 6 are stand-in-friendly and cheap after it. Items 3 to 5 ride #16's Harness attachment and can wait for it.
Metadata
Metadata
Assignees
Labels
Projects
- StatusShow more project fieldsIn Progress
Activity
omesser commentedon Oct 6, 2026
Sliced inbound harness wake into concrete bugfix: #1356 (between-turn
session/updatedrop inacp_wireserve — stream/thought/speak never wake Director). Cron alone does not make Instance Prompt reminders work until that lands.— Cursor agent (Coder), on @omesser's behalf.
omesser commentedon Oct 6, 2026
Verdict - calls for discussion
Structured replies instead of say: parsing - Challenge. what behavior/UX improvement will we get from this? the downside is risk - structured replies with say will diverge chat contents from speech bubble more often, will possible bring more failure modes of breaking format on the harness/AI model side. What will the user gain? by this? Give concrete examples and Scenarios
Sensing in the Character Prompt - Good idea! Open an issue for it. We'll implement that for V1 milestone even. Propose a summarized format - it should be dense and very token efficient to not cause failures for small models; short/concise/compacted and ok to be lossy, the model can still call the MCP server for details and it should remain meaningful.
Memory write-backs from turns - Good! let's spec this out. just writing it down is not enough. We need to figure out the protocol of using the memory file (unused until now) - need to pass it to the AI at the very least
The buddy as a live narrator of Harness work - this is great. create an issue for it. but let's push this to V2 milestone for now. it's also affected/blocked by 1 IMHO
Streaming dialogue into the bubble - yes. accepted. open an issue and spec this out for V1. this is now p0.
Reaction repertoire for the verbs - give 3 concrete examples of how that would change current behavior
omesser commentedon Oct 6, 2026
Answering the verdict in your numbering. I'm reading the whole triage through one lens: idle behavior driven by the authored personality. Items 2 and 6 feed that directly, so they're the cheap wins. Item 4 is Harness-side, so V2 is right. I checked the claims below against current
main(ab02e80) and corrected the original write-up where the code disagrees with it.New sub-issues, all on the board in Todo: #1373 (2, V1), #1374 (3, spec, V1), #1375 (4, V2), #1376 (5, V1, p0).
1) Structured replies. Conceded, mostly.
A substrate-first swap has no user gain of its own, and both risks you named are real: another format for small and local models to break, and another place for chat and bubble to drift apart. Checking the code also removed two of the gains the write-up claimed:
parse_proposaltakes a Behavior name on its own line (orname | line) and treats the rest as speech. The Engine plays the Behavior and speaks the line either way.near_missin the Action Log. An empty reply logsfailed, and the Static Director fills in. A Behavior the State gate refuses still has its line spoken. (One wart: a near miss speaks the raw reply, name included.)That leaves one concrete scenario, the pacing hint. You say "shush for an hour" or "quiet, I'm in a meeting", and the Director actually holds to it, because the reply can tell Pace not to wake before a given time. Today Pace grows on its own schedule and resets on every reactive wake, so the character can agree and then pipe up again a few minutes later. This is also the portable home for the time-based Instance Prompt examples #1354 left out ("a follow-up that lets the Director set its next wake"), and it works for a Model API Director, not only a Harness. It isn't a timer feature, just "don't wake me before X".
Mitigations, if and when it lands:
Frame.dialoguestring feeding both the bubble and the Chat row, as it does today, so the two can't drift.Proposal: drop the blanket JSON swap. Keep item 1 open, narrowed to "add a pacing hint (and later
remember) when a visible feature needs it", and let the specs for 3 and 5 decide its shape.2) Sensing. Agreed, V1 → #1373.
One dense, lossy line of about 40–60 tokens replaces today's separate
time:andopen:lines:Rules:
Lossy is fine here: a Harness can still call
list_windows/describe_screen/recallfor detail.3) Memory. Agreed, writing it down isn't enough → #1374.
The spec answers three questions:
remembertool now. A reply field for Model API Directors comes only once item 1 has one.remembercall shows only as a tool-call title and status.Reminders stored as dated Memory facts stay the later follow-on #1356 already names. They're not in this scope.
4) Narrator. Agreed, V2 → #1375.
It depends on 1 only for named lifecycle Behaviors (
working/succeeded/failed). Narration throughspeak/play_behaviormostly rides #1356.5) Streaming. Accepted, V1, p0 → #1376.
Both transports already read deltas: ACP
agent_message_chunkinacp_wire::note_update, and SSE inmodel.rsread_frames. Both feed one sharedAnswer, and both already stream thoughts live to the Thinking row. Only the answer is held until the end, so the Harness isn't meaningfully cheaper to do first; one answer-delta door serves both transports. The real work is holding a Behavior-name line back so it never flashes in the bubble, plus the bubble machine and cancel. Streaming one string to both bubble and Chat row also leaves less room for drift.6) Reaction repertoire. Three before/afters.
Today a Poke always plays an instant
react. The reactive wake then reaches the model withwhat just happened: poked,state,recentandstanding on. What it never says is what the sprite was doing before the verb. The change: keep the instantreact, and add that to the wake (state before the verb, the Behavior in progress, a poke streak) so the follow-up line or Behavior fits the moment.stateis read after the tick, and a verb wakes a sleeping sprite into a fall, so the model seesstate: fallingoridle, neverasleep, and answers with a generic quip.react(startle), then a groggy "…five more minutes", and it goes quiet so it nods off again.recentholds the last three Behaviors the Engine played from proposals, but the Engine's ownreactis never recorded there, and a fresh Poke takes over a call that's still thinking (ADR-0016). Nothing tells the model this is the third poke.picked up/state: heldand nothing about the climb.Not a gain, because it already works today: poked while perched, it can already remark on the app it's standing on, since
standing on:is in every wake.— Cursor agent (Architect), on @omesser's behalf.
omesser commentedon Oct 6, 2026
Read - The main consumer here should be the agent/harness and we need to make this accessible. either read via mcp (recall) OR hooking it up into the harness existing memory subsystem - needs further research
omesser commentedon Oct 6, 2026
decision update:
6. Reaction repertoire for the verbs
OK, looks good and very lean /easy to implement. but we need to decide about the recency - do we send last N states/actions or set of states/actions that happened over the last 10-20 seconds? with/without timestamps? and what else?
omesser commentedon Oct 6, 2026
Answering your two latest notes — memory Read, and item 6 recency. Checked against current
main.3) Memory Read — reframed
Agreed: the primary consumer is the agent/Harness, not stuffing Memory into every Model API wake. Updated #1374 accordingly.
Spec research (explicit Done-when there):
recallalready exists (dispatch→MemoryManifest::recall). Make sure the Harness is actually prompted / expected to use it when facts would help.Ambient Character Prompt top-N stays a secondary, optional, bounded path for Model API Directors (and a couple of lines in the sensing line from #1373) — demoted vs Harness accessibility, not dropped. Write + Visibility unchanged from the earlier scope.
6) Reaction repertoire — recency decision
OK on lean / easy. Concrete recommendation (not a menu of equals). Opened #1380 (V1, Todo) to implement.
Today's wake history (why last-N alone is wrong):
what just happenedlive.happened(prompt::follow_up/happened_word)recentdirector::REMEMBERED+rememberinframe_loopwhenframe.behavioris set)react; can be hours-stale across quiet minutesstateframe.state(frame_loop: "After the tick so a Throw is already Falling…")falling/idle, neverasleepFrame.playing_behaviorexists every tick of a named Behaviorsince_stateis only Static Director pacing (STATE_BOUND= 90s)Instant
reacton Poke/Summon stays (engine: "A Poke is answered…" →Primitive::React). That path does not setframe.behavior, so it never entersrecent.Recommend this lean default for the reactive wake:
react.live.last_state) — required for asleep.playing_behavioris set (climb / walk) — required for mid-climb and pull-off.recent).reactand goes stale across quiet time.2s ago). Hard cap ≤5 events / ≤~40 tokens.That covers the three accepted before/afters without bloating the Character Prompt. Shape details and Done-when live on #1380; say if you want a different window or cap before implement.
— Cursor agent (Architect), on @omesser's behalf.
omesser commentedon Oct 8, 2026
Item 6 (reaction repertoire) shipped via #1380 / #1425. The instant
reactis kept, and the reactive wake now carrieswas:(what the gesture found), a streak suffix onwhat just happened,doing:, and a 12slately:window capped at 5 steps, with no timestamps.One change from the plan here:
doing:andlately:also go on ambient follow-ups, not only reactive wakes. That's because a gate would only add a branch, and on ambient wakes those lines are usually empty anyway. This umbrella stays open for the other items.— Cursor agent (Coder), on @omesser's behalf.
omesser commentedon Oct 29, 2026
Direction change for item 4, the live narrator in #1375, and for the speech work under it: see #1935.
speaktool is removed. Whatever the parser doesn't sort as thinking prose, a startup banner, a bare Behavior name or the Behavior line streams to Chat and the bubble.speakor two channels.— Cursor agent (Architect), on @omesser's behalf.