Desktop widget + PTT + task cancellation by ncrispino · Pull Request #38 · sonichi/sutando

ncrispino · 2026-03-29T18:52:08Z

Summary

Adds a native macOS desktop widget and several cross-cutting features.

Desktop widget (`widget.swift`)

SwiftUI floating panel — always-on-top, translucent, all-spaces
Task display, voice status indicator, conversation view, text input
Global push-to-talk hotkey (Ctrl+Space, configurable via SUTANDO_PTT_KEY)
Core status polling (shows proactive loop activity)
Auto-built and launched from startup.sh (gracefully skips if Xcode CLT not installed)

Push-to-talk (PTT)

Widget: global hotkey via Carbon Events (with Accessibility event tap fallback)
Web client: Space bar when PTT mode enabled, PTT toggle button
Both sync mute state to voice agent via new HTTP control server (port 9901)

Task cancellation

New /task/cancel endpoint in agent-api
Cancel buttons in both widget and web client
Result polling stops properly on cancel/error (with 5-min timeout)

Web client improvements

Auto-connect overlay (respects browser autoplay policy)
Better microphone error messages (specific to permission, device, etc.)
setMuteState refactor — widget/PTT can control mute remotely
Live transcript publishing to control server for widget pickup
End All button + Escape shortcut
Fixed stale task cleanup (no flicker on agent-api restart)

Voice agent

HTTP control server on port 9901 (state, mute, PTT, live transcript)
Writes voice-conversation.json for widget conversation display

Utility scripts

kill-all.sh — reliably stop all Sutando processes
clean-state.sh — remove stale task/result files

Review feedback addressed

✅ Poll timeout added (5 min) + stop on error/cancelled states
✅ PTT hotkey: return eventNotHandledErr when event not matched
✅ kill-all.sh: kill watch-tasks instead of fswatch
✅ Startup gracefully handles missing Xcode CLT
✅ Port 9901 (voice control server) added to health-check.py

Open

PTT: further testing across apps (consider deferring to v2 as optional mode)

Test plan

Build widget: swiftc -o sutando-widget src/widget.swift -framework Cocoa -framework SwiftUI -framework CoreGraphics -O
Widget floats on all spaces, shows tasks and voice status
PTT works globally (Ctrl+Space) and in web client (Space)
Task cancel from widget and web client
Text input: goes through WS when voice connected, task API when not
Auto-connect overlay activates voice on click
health-check shows voice-control (port 9901) status

🤖 Generated with Claude Code

sonichi

Review — WIP: Desktop widget + PTT + task cancellation

Great additions. Here's my assessment:

Widget (widget.swift) ✅

SwiftUI floating panel, always-on-top, all-spaces — solid UX
Core status polling (reads core-status.json) — integrates with the status system we built today
Task display, conversation view, text input — all useful
1346 lines but well-structured

Task cancellation ✅

/task/cancel API endpoint — fills a gap (only had inline cancel before)
Cancel buttons in widget + web client
Proper result polling stop on cancel

PTT (Push-to-talk) ⚠️

Owner suggests PTT may be unnecessary for v1 — hotkey connect + voice "bye" disconnect is simpler
Could defer to v2 as optional mode
Carbon Events + Accessibility fallback adds complexity

Web client improvements ✅

Auto-connect overlay — good UX
Better mic error messages — addresses real user pain
setMuteState refactor — enables remote control from widget

Utility scripts ✅

kill-all.sh, clean-state.sh — useful maintenance tools

Suggestions:

Consider making PTT optional/deferrrable
The widget auto-build in startup.sh needs Xcode CLT — add a check
Port 9901 (control server) should be in health-check.py

Overall: strong PR, widget is a differentiator. Merge when TODOs are resolved.

Addresses reviewer suggestion to include the widget/PTT control server in the health check alongside other optional services. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

sonichi · 2026-04-03T10:54:59Z

Reviewed the diff — the features look solid (task cancel, PTT, widget, kill-all/clean-state scripts). However, the PR has merge conflicts against current main (startup.sh, voice-agent.ts, web-client.ts, .gitignore have all changed significantly since March 29).

Note: the native Sutando menu bar app (PR #74-76) now handles some of what the widget covers (global hotkeys, voice toggle, mute). The widget adds PTT and a floating panel which are complementary, but the startup.sh and voice-agent.ts changes will need careful rebasing to avoid conflicts with the new inline tools, tab aliases, and stand-identity.json loading.

@ncrispino could you rebase against main when you get a chance?

ncrispino · 2026-04-03T23:03:26Z

@sonichi I think we can just close this for now, I can put more targeted features against main when I have time.

"ts.net" in url matches spoofed hostnames like evil-ts.net.attacker.com. Parse URL and check hostname endswith(".ts.net") instead. POC: scripts/poc-codeql-38-url-substring.sh Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: validate display param is numeric (CodeQL #33) Unsanitized display query param flowed into subprocess.run args. Add isdigit() check so only numeric values reach screencapture. Includes POC script: scripts/poc-codeql-33-cmd-injection.sh Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: escape backslashes before quotes in AppleScript strings (CodeQL #42, #39) The .replace(/"/g, '\\"') only escaped quotes but not backslashes. A backslash before a quote (\\") would become \\" which closes the AppleScript string context. Fix: escape backslashes first, then quotes. Includes POC: scripts/poc-codeql-42-39-string-escaping.sh Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test: POC confirms path injection guards work (CodeQL #16-23, #28-31, #35-36) Existing resolve() + is_relative_to() + regex guards in agent-api.py and dashboard.py properly block all path traversal attempts. CodeQL flags these as false positives because it can't track taint through the validation logic. POC: scripts/poc-codeql-path-injection.sh Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: use hostname endswith instead of substring check (CodeQL #38) "ts.net" in url matches spoofed hostnames like evil-ts.net.attacker.com. Parse URL and check hostname endswith(".ts.net") instead. POC: scripts/poc-codeql-38-url-substring.sh Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: reduce phone number exposure in logs (CodeQL #37) Extract masked phone to separate variable and reduce visible digits from 6 (2+4) to 4 (2+2). Breaks CodeQL taint flow from env var to console.log. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…ichi#39, sonichi#42) (sonichi#320) * fix: validate display param is numeric (CodeQL sonichi#33) Unsanitized display query param flowed into subprocess.run args. Add isdigit() check so only numeric values reach screencapture. Includes POC script: scripts/poc-codeql-33-cmd-injection.sh Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: escape backslashes before quotes in AppleScript strings (CodeQL sonichi#42, sonichi#39) The .replace(/"/g, '\\"') only escaped quotes but not backslashes. A backslash before a quote (\\") would become \\" which closes the AppleScript string context. Fix: escape backslashes first, then quotes. Includes POC: scripts/poc-codeql-42-39-string-escaping.sh Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test: POC confirms path injection guards work (CodeQL sonichi#16-23, sonichi#28-31, sonichi#35-36) Existing resolve() + is_relative_to() + regex guards in agent-api.py and dashboard.py properly block all path traversal attempts. CodeQL flags these as false positives because it can't track taint through the validation logic. POC: scripts/poc-codeql-path-injection.sh Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: use hostname endswith instead of substring check (CodeQL sonichi#38) "ts.net" in url matches spoofed hostnames like evil-ts.net.attacker.com. Parse URL and check hostname endswith(".ts.net") instead. POC: scripts/poc-codeql-38-url-substring.sh Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: reduce phone number exposure in logs (CodeQL sonichi#37) Extract masked phone to separate variable and reduce visible digits from 6 (2+4) to 4 (2+2). Breaks CodeQL taint flow from env var to console.log. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

ncrispino added 5 commits March 29, 2026 11:17

Widget

c2736b5

Fix html err

a87734f

Rm narration for now

d444be0

Merge

9ed853e

Reveiw fixes

57f9fd0

sonichi reviewed Mar 29, 2026

View reviewed changes

feat(health-check): add voice control server (port 9901) to checks

1a8c3e5

Addresses reviewer suggestion to include the widget/PTT control server in the health check alongside other optional services. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

vyvy3n changed the title ~~WIP: Desktop widget + PTT + task cancellation~~ Desktop widget + PTT + task cancellation Mar 29, 2026

vyvy3n marked this pull request as ready for review March 29, 2026 22:18

ncrispino closed this Apr 3, 2026

This was referenced Apr 13, 2026

Fix 5 CodeQL security alerts (#33, #37, #38, #39, #42) #320

Merged

SEC-320: 5 CodeQL security alerts — command injection, escaping, URL sanitization, logging #321

Closed

bassilkhilo-ag2 mentioned this pull request Sep 23, 2026

room-ops: "category set <room> <name> [--suggest]" so the agent can categorize rooms #4668

Open

Navigation Menu

Repository navigation

Desktop widget + PTT + task cancellation - #38

Desktop widget + PTT + task cancellation#38
ncrispino wants to merge 6 commits into
mainfrom
feature/widget

ncrispino commented Mar 29, 2026 •

edited by vyvy3n

Loading

Uh oh!

sonichi left a comment

Uh oh!

sonichi commented Apr 3, 2026

Uh oh!

ncrispino commented Apr 3, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

ncrispino commented Mar 29, 2026 • edited by vyvy3n Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Desktop widget (widget.swift)

Push-to-talk (PTT)

Task cancellation

Web client improvements

Voice agent

Utility scripts

Review feedback addressed

Open

Test plan

Uh oh!

sonichi left a comment

Choose a reason for hiding this comment

Uh oh!

sonichi commented Apr 3, 2026

Uh oh!

ncrispino commented Apr 3, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ncrispino commented Mar 29, 2026 •

edited by vyvy3n

Loading

Desktop widget (`widget.swift`)