Skip to content

Realtime ASR with language auto: return the detected language per utterance #90

Open

Description

@gdevenyi

Request

Return the detected language for each utterance when Nemotron 3.5 ASR Streaming runs with language: auto:

  • in the realtime WebSocket (/v1/audio/transcriptions/realtime): a language field (for example "fr-FR") on each conversation.item.input_audio_transcription.completed event, and ideally on .delta events once known;
  • in nemo-speech transcribe --json with --stream/endpointing: the language per utterance or per word, not only a file-level list.

Why

docs/asr/models.md says that with auto, "Structured results include the selected or detected language". For live transcription of conversations that switch between languages (for example English and French), the client needs the language of each utterance, for display and to route translation. The model already conditions on a language-ID prompt, so the information exists inside the engine.

What I see (v0.2.0, nvidia/nemotron-3.5-asr-streaming-0.6b Q8_0, Vulkan)

Audio: 88 s, 60 s English (test_files/diar/ami_en2002d_2132.wav) followed by 28 s French.

  • Realtime WebSocket, session.update with "language": "auto": session.updated echoes language: auto. The 20 .completed events have only audio_processed, event_id, transcript, type and words (word, start, end, confidence, speaker). The French utterances are transcribed correctly in French, but no event says which language they are.
  • nemo-speech transcribe en_fr.wav --language auto --stream --endpointing --json: the result has a single "languages": ["fr-FR"] for the whole file, although most of the audio is English.

Suggested shape

{"type": "conversation.item.input_audio_transcription.completed", "transcript": "...", "language": "fr-FR", "words": [...]}

A per-word language would also cover a switch inside one utterance, if the decoder can tell.

Activity

  1. added a commit that references this issue on Oct 30, 2026
    adc1ab7
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions