feat(live-voice): answer an open asktool question by voice by vastsa · Pull Request #1270 · vastsa/PI-Desktop

vastsa · 2026-10-01T02:14:35Z

Summary

A Live Voice work call can now answer the AskTool question the bound session is waiting on, and both call surfaces point the user at that pending request instead of only telling them to finish it on the desktop. A reloaded renderer also rebuilds the AskTool and permission cards it is still waiting on.

Motivation & Root Cause

A work call could already report waiting-input, but the question existed only in that session's own Composer card, so a user in a call had to leave the call flow, find the session and answer by hand. The call announced a wait it could not act on.
The renderer held pending AskTool/permission cards in memory only, so a renderer reload lost them while the runtime still waited, leaving a wait with no visible card.

Key Changes

packages/host-runtime/src/live-work/intent.ts: new respond-input intent ({ questionIndex, options[] }) with strict fail-closed validation.
packages/host-runtime/src/live-work/coordinator.ts + context.ts: route respond-input as a control request (never a turn, steer or queue), and carry the bounded open question into the classifier input.
apps/desktop/electron/main/live-voice/spoken-answer.ts (new): option-only answer matching, bounded read-out, delivery-budget truncation.
apps/desktop/electron/main/live-voice/work-bridge.ts: resolve the session's single open question through the Host input path, fail closed on zero/several open questions and on unknown or partial answers, and deliver the question with its own options to the provider as interaction-required feedback.
packages/agent-host/src/agent-host.ts, apps/desktop/electron/main/agent-host-bridge.ts, ipc/agent-ipc.ts, lib/api.ts: bounded reads of open inputs and pending interactive requests, plus one-ask resolution.
Renderer: restorePendingInteractive merges Host state into the live queues without ever clearing them, at the same call sites as restorePendingPlan; the compact bar keeps a waiting line and Call Details shows the pending request with an action that opens the exact bound session.
Docs: ADR 0314, live-work-session, live-voice, component spec, E2E plan (English and zh-CN mirrors), unreleased notes, live-work evidence.
Tests: ask-answer.test.ts (intent + routing), spoken-answer.test.mjs (option-only matching and bounds), live-work-decision.test.mjs (what the call surfaces show), pending-interactive-restore.test.mjs (recovery).

Safety: permission, Plan and Goal approvals stay desktop-UI decisions, so spoken agreement is still never approval; a spoken answer may only select among the option labels the question itself offered, and the panel and bar still expose no approve or answer control.

Verification

pnpm build:js; pnpm -r --if-present typecheck; node scripts/check-style-tokens.mjs; pnpm docs:check (542 pages).
Package suites: desktop 3340, host-runtime 119, agent-host 57, shared 1166, racp 21, voice-runtime 51, i18n 28.
E2E: pnpm test:e2e 23/23 (2 skipped: needs PI_DESKTOP_TEST_API_KEY), pnpm test:e2e:live-voice 5/5 fixture journeys, pnpm test:e2e:live-voice-interaction 46/46.
NOT RUN: real provider account, microphone and device acceptance; no verify:ui:* was requested.

A Live Voice work call could report that the bound session was waiting on an AskTool question but could not act on it, so the user had to leave the call, find that session's card and answer by hand. The call surfaces now name the pending request and open the session that owns it, and a work call can resolve that session's single open question: the question and its own options are read out, and a spoken answer may only select among those labels. The answer goes through the same Host input path the desktop card uses, as a control request that never creates, steers or queues a turn. Permission, Plan and Goal approvals remain desktop-UI decisions, so spoken agreement is still never approval (ADR 0314). A reloaded renderer no longer drops the AskTool and permission cards a session is still waiting on: the pending interactive requests are read back from Host state and merged with the live queues without clearing anything.

Copilot

Copilot review overview

🟡 Changes recommended

Ask identity races, ambiguous delivery handling, inconsistent bounds, and contradictory user guidance can cause incorrect or unusable voice answers.

Review effort: Balanced
Findings: 1 · 5 · 3

Open (9)

Bind voice answers to the active pending request ID · New
Read pending asks after awaiting Host permission · New
Preserve exact option labels through voice answer validation · New
Ensure the first complete question fits the byte budget · New
Preserve unknown status for ambiguous dispatch failures · New
Enforce the byte budget for pendingQuestion · New
Update AskTool hint copy to support voice answering · New
Clarify that only visual controls cannot answer by voice · New
Remove the duplicated waiting-decision scenario · New

What changed in this PR

Adds option-only voice answers for pending AskTool requests, restores interactive cards after renderer reloads, and surfaces pending decisions in Live Voice UI.

Changes:

Adds respond-input intent routing and Host resolution.
Restores pending AskTool/permission cards and displays decision notices.
Adds tests, translations, specifications, and ADR 0314.

File	Description
`packages/shared/src/types/agent.ts`	Adds pending-interaction contract.
`packages/shared/src/protocol.ts`	Adds pending-interaction IPC channel.
`packages/i18n/src/locales/zh-TW/index.ts`	Adds decision UI translations.
`packages/i18n/src/locales/zh-CN/index.ts`	Adds decision UI translations.
`packages/i18n/src/locales/tr/index.ts`	Adds decision UI translations.
`packages/i18n/src/locales/pt-BR/index.ts`	Adds decision UI translations.
`packages/i18n/src/locales/ko/index.ts`	Adds decision UI translations.
`packages/i18n/src/locales/fr/index.ts`	Adds decision UI translations.
`packages/i18n/src/locales/es/index.ts`	Adds decision UI translations.
`packages/i18n/src/locales/en/index.ts`	Adds decision UI translations.
`packages/i18n/src/locales/de/index.ts`	Adds decision UI translations.
`packages/host-runtime/test/live-work-fixture.ts`	Supports input-response tests.
`packages/host-runtime/src/live-work/intent.ts`	Defines and validates `respond-input`.
`packages/host-runtime/src/live-work/coordinator.ts`	Routes spoken answers as controls.
`packages/host-runtime/src/live-work/context.ts`	Supplies pending questions to classification.
`packages/host-runtime/src/live-work/ask-answer.test.ts`	Tests answer intent and routing.
`packages/agent-host/src/agent-host.ts`	Exposes pending input requests.
`docs/zh-CN/spec/06-delivery/04-e2e-test-plan.md`	Updates Chinese E2E coverage.
`docs/zh-CN/spec/04-ux/08-component-spec.md`	Documents Chinese decision UI.
`docs/zh-CN/spec/03-runtime/live-work-session.md`	Documents Chinese answer semantics.
`docs/zh-CN/spec/03-runtime/live-voice.md`	Documents Chinese voice capability.
`docs/spec/06-delivery/04-e2e-test-plan.md`	Updates E2E scenarios.
`docs/spec/04-ux/08-component-spec.md`	Documents decision surfaces.
`docs/spec/03-runtime/live-work-session.md`	Defines spoken-answer behavior.
`docs/spec/03-runtime/live-voice.md`	Documents Live Voice integration.
`docs/project/unreleased.md`	Adds release notes.
`docs/implementation/live-work-evidence.md`	Records validation evidence.
`docs/adr/README.md`	Registers ADR 0314.
`docs/adr/0314-live-voice-spoken-asktool-answers.md`	Records the architecture decision.
`apps/desktop/test/spoken-answer.test.mjs`	Tests matching and bounds.
`apps/desktop/test/pending-interactive-restore.test.mjs`	Tests card restoration.
`apps/desktop/test/live-work-decision.test.mjs`	Tests decision presentation.
`apps/desktop/src/styles/voice.css`	Styles decision notices.
`apps/desktop/src/stores/slices/session-slice.ts`	Restores pending interactions.
`apps/desktop/src/stores/runtime/session-coordination.ts`	Triggers restoration on coordination.
`apps/desktop/src/stores/app-state.ts`	Adds restoration API state.
`apps/desktop/src/lib/api.ts`	Exposes pending-interaction IPC.
`apps/desktop/src/features/voice/live/LiveWorkDecisionNotice.tsx`	Renders pending decisions.
`apps/desktop/src/features/voice/live/LiveVoiceDetails.tsx`	Shows decisions in call details.
`apps/desktop/src/features/voice/live/LiveVoiceCallBar.tsx`	Shows persistent waiting state.
`apps/desktop/src/features/voice/live/live-work-decision.ts`	Derives bounded decision details.
`apps/desktop/electron/main/live-voice/work-profile.ts`	Updates provider instructions.
`apps/desktop/electron/main/live-voice/work-bridge.ts`	Resolves spoken answers through Host.
`apps/desktop/electron/main/live-voice/spoken-answer.ts`	Implements option matching and readout.
`apps/desktop/electron/main/live-voice/runtime.ts`	Forwards pending questions.
`apps/desktop/electron/main/live-voice/call-service.ts`	Connects question-aware feedback.
`apps/desktop/electron/main/live-voice/call-service-feedback.ts`	Produces interaction feedback.
`apps/desktop/electron/main/ipc/agent-ipc.ts`	Handles pending-interaction reads.
`apps/desktop/electron/main/agent-host-bridge.ts`	Bridges pending inputs and resolution.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+      const open = bridge.openAsk(request.sessionId);
+      if (!open) {


+    async pendingInteractiveRequests(sessionId: string): Promise<PendingInteractiveRequests> {
+      const asks = agentHost.pendingInputRequests(sessionId)
+        .slice(0, MAX_PENDING_INTERACTIVE)
+        .map((entry) => entry.original);
+      const pending = await listPendingToolRequests(options.getHost, sessionId);
+      return {
+        asks,
+        permissions: pending.slice(0, MAX_PENDING_INTERACTIVE).map(
+          ({ createdAt: _createdAt, expiresAt: _expiresAt, ...request }) => request,
+        ),
+      };
+    },


+      options: question.options
+        .slice(0, MAX_ASK_OPTIONS)
+        .map((option) => boundSpokenText(askToolOptionLabel(option), MAX_ASK_OPTION_CHARS)),


+  for (const section of sections) {
+    const candidate = `${text}\n${section}`;
+    if (encoder.encode(candidate).byteLength > MAX_SPOKEN_QUESTION_BYTES) break;
+    text = candidate;


+      try {
+        await bridge.resolveOpenAsk({ sessionId: request.sessionId, inputId: open.inputId, answers });
+        return { status: "resolved", message: "Answer delivered to the open question.", turnId: open.turnId };
+      } catch {
+        return { status: "unavailable", message: "The answer could not be delivered. Answer the question in the desktop." };


+  // The pending question is what a spoken answer has to be checked against:
+  // without its own option labels in the input, the classifier could only
+  // guess, so it travels with the request.
+  if (input.pendingQuestion) payload.pendingQuestion = input.pendingQuestion;


+    <div className="live-voice-decision" role="status">
+      <strong className="live-voice-decision-title">{t("liveVoice.decisionTitle")}</strong>
+      <p className="live-voice-work-summary">{liveWorkDecisionText(decision, t)}</p>
+      <p className="live-voice-hint">{t("liveVoice.decisionHint")}</p>


+  that exact session. The decision itself still belongs to the session's own
+  card, so answering or approving from the call remains impossible.



 - **前提：** 使用隔离的 Live Provider fixture、本地 AgentHost 会话和确定性意图分类器；候选请求须从现有 Live adapter 回调进入，不使用真实账号或付费端点。
- **步骤：** 验证纯通话没有工作作用域时会拒绝任务候选；再以明确选择的本地会话和关闭的上下文共享启动工作通话。提交一个已声明工具请求并检查回执、Host admission 和 `voiceOrigin`。覆盖忙时独立排队、过期 steer、精确回合 stop、submit 返回前到达的终态事件，以及基于已记录结果的只读查询。请求项目／会话列表，确认只返回标签和本次 call 的 opaque 引用；通过面板动作打开已列会话、在已列项目中新建会话，并确认工作绑定仍固定。分别保持 Provider 生成、用户讲话和本地播放活跃，确认反馈须待三者空闲和静默窗口结束后才发送。覆盖 silent、静默时的显式查询、过期反馈降级；结束通话时确认已受理工作仍保留。
+- **步骤：** 验证纯通话没有工作作用域时会拒绝任务候选；再以明确选择的本地会话和关闭的上下文共享启动工作通话。提交一个已声明工具请求并检查回执、Host admission 和 `voiceOrigin`。覆盖忙时独立排队、过期 steer、精确回合 stop、submit 返回前到达的终态事件，以及基于已记录结果的只读查询。请求项目／会话列表，确认只返回标签和本次 call 的 opaque 引用；通过面板动作打开已列会话、在已列项目中新建会话，并确认工作绑定仍固定。分别保持 Provider 生成、用户讲话和本地播放活跃，确认反馈须待三者空闲和静默窗口结束后才发送。覆盖 silent、静默时的显式查询、过期反馈降级；当提交处于等待用户决策状态时，确认紧凑栏保留等待提示、详情显示待决请求并提供打开该精确会话的操作，且其他会话的待决请求不会被归到绑定会话；当提交处于等待用户决策状态时，确认紧凑栏保留等待提示、详情显示待决请求并提供打开该精确会话的操作，且其他会话的待决请求不会被归到绑定会话。随后用语音回答该 AskTool 提问，确认会话卡片经由 Host 输入路径解决；再分别用提问从未提供的标签、只回答一部分、以及同时存在两个待答提问重试，确认每次都被拒绝、不产生写操作且不重试。结束通话时确认已受理工作仍保留。


Copilot AI balanced review requested due to automatic review settings October 1, 2026 02:14

Copilot started reviewing on behalf of vastsa October 1, 2026 02:15

View session

Copilot AI reviewed Oct 1, 2026

View reviewed changes

vastsa merged commit e37904e into main Oct 1, 2026
5 checks passed

vastsa deleted the feat/live-voice-asktool-answers branch October 6, 2026 16:56

Navigation Menu

Repository navigation

feat(live-voice): answer an open asktool question by voice - #1270

feat(live-voice): answer an open asktool question by voice#1270
vastsa merged 1 commit into
mainfrom
feat/live-voice-asktool-answers

vastsa commented Oct 1, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		that exact session. The decision itself still belongs to the session's own
		card, so answering or approving from the call remains impossible.

Conversation

vastsa commented Oct 1, 2026

Summary

Motivation & Root Cause

Key Changes

Verification

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants