Skip to content

fix: stream voice transcription into chat composer - #866

Open
GUANGDIJI wants to merge 1 commit into
MemTensor:v1.2.8from
GUANGDIJI:codex/asr-realtime-streaming
Open

GUANGDIJI wants to merge 1 commit into
MemTensor:v1.2.8from
GUANGDIJI:codex/asr-realtime-streaming

Conversation

@GUANGDIJI

Copy link
Copy Markdown
Contributor

What changed

  • wire the existing live ASR transcription events into the main chat composer
  • keep partial and final transcript updates scoped to the conversation where recording started
  • replace only the exact provisional voice range so matching user-authored text is never overwritten
  • keep the existing final full-audio transcription and live-stream fallback behavior unchanged

Validation

  • @memmy/frontend-desktop typecheck passed
  • focused ASR/HomePage suite: 4 files, 125 tests passed
  • full desktop frontend suite: 296 files / 3416 tests passed; one unrelated PDF resource suite could not start because the test runtime does not provide DOMMatrix
  • PR scope guard passed against origin/v1.2.7 (1 commit, 2 changed files)

@Memtensor-AI

Copy link
Copy Markdown
Collaborator

πŸ€– Open Code Review

Target: PR #866
Task: 3c74d47eeed8e2a7
Base: v1.2.7
Head: codex/asr-realtime-streaming
Head SHA: 71a43b79c0d384eeba5c8abbcd26e27232297713

πŸ” OpenCodeReview found 6 issue(s) in this PR.


1. App/frontend/desktop/src/pages/home-page.tsx (L998-L999)

Each live onLine dispatches a whole new composer-draft state (setComposerDraftForScope β†’ dispatch(composerDraftUpdated)), which re-renders this very large page. The streaming transcriber emits partial events continuously (several per second, unlike the old ~6s chunked lines), so a full page re-render plus a requestAnimationFrame resize per partial can make dictation janky. Consider coalescing live updates, e.g. accumulate the latest text in a ref and flush at most once per animation frame (or debounce ~100ms) instead of dispatching on every partial.


2. App/frontend/desktop/src/pages/home-page.tsx (L3616-L3617)

When the recorded provisional range no longer matches (user edited before it, a slash command rewrote the draft, or the draft was cleared because the user hit Enter to send mid-dictation), this fallback appends next via mergeVoiceTranscript(current, nextText) β€” but next is the entire accumulated live transcript, not just the new remainder. All previously dictated words therefore appear a second time in the composer (and the final transcript replaces only that appended block, leaving the duplicate behind). Suggest trying to re-locate previous.text in current to recover the offset, and otherwise insert only the part of next that is not already present instead of re-merging the full text.


3. App/frontend/desktop/src/pages/home-page.tsx (L2840)

The recording is pinned to the scope that started it, but nothing cancels or reports it when chatScopeKey changes during dictation (e.g. the user switches conversations). The mic button in the new chat still renders as active because asrRecorder.isRecording is global, so pressing it stops the old recording and writes the whole transcript β€” plus any later live lines β€” into the previous chat's draft and error notices, invisible to the user. Consider cancelling voice input when the scope changes, alongside the existing useEffect(() => { setTaskRecording(null); }, [chatScopeKey]), or at least surfacing that recording continues in another chat.


4. App/frontend/desktop/src/pages/home-page.tsx (L2863-L2864)

These slash-menu/command-selection resets run unconditionally, while resize/focus just below are gated on scopeKey === chatScopeKey. When the recording belonged to a different chat scope, the current chat's slash-menu state is reset for no reason. Move them inside the same scope guard for consistency.


5. App/frontend/desktop/src/pages/home-page.tsx (L991-L992)

The live option supplies only onLine, so live-transcription failures are completely silent here. recording-entry.tsx passes onError for exactly this reason: when the relay cannot be opened or a segment request fails, the user keeps speaking while the composer stays blank (and the segmented fallback can fail too), with no hint that dictation is not being transcribed. Consider adding an onError that reports the first failure (e.g. via setComposerMediaErrorForScope), suppressed/throttled so the 6s fallback retries do not spam toasts (see isLiveAsrTimeout for the timeout case).


6. App/frontend/desktop/src/pages/home-page.tsx (L995-L996)

mergeLiveLines de-duplicates by startMs, which only holds while offsets are absolute from the start of the recording. When the streaming transcriber fails, useAsrRecorder silently swaps in createChunkedLiveTranscriber, whose consumedSamples restarts at 0, so its lines (0, 6000, 12000…) collide with and get sorted among the earlier stream lines. The merged text written into the composer is then re-ordered and partially overwritten (earlier sentences dropped, later ones shuffled), and because that whole string replaces the provisional range, the corruption is user-visible in the draft. Consider tracking revisions against the most recent line (replace when the offset does not advance, append when it does) instead of a global offset-keyed merge, or reset the accumulated lines when the recorder falls back.

Generated by cloud-assistant via Open Code Review.

@Memtensor-AI

Copy link
Copy Markdown
Collaborator

βœ… Automated Test Results: PASSED

Required tests passed (95/95 executed, 0 skipped). Duration: 13s

Branch: codex/asr-realtime-streaming

Tested source: 71a43b79c0d384eeba5c8abbcd26e27232297713
Tested base: 1f87dcf385db4339da63f7018339cdd63f781dd7
Tested commit: 71a43b79c0d384eeba5c8abbcd26e27232297713
These results apply only to this immutable source/base snapshot.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants