Repository navigation
serve (Vulkan): concurrent realtime sessions with ASR batching on give no output or abort with GGML_ASSERT(ne3 == ne13) #89
Description
Summary
With nemo-speech serve on the Vulkan backend and the default ASR batching, two or more concurrent realtime transcription sessions break: one run aborted with GGML_ASSERT(ne3 == ne13) failed, and a second run produced no deltas and no words on either session. The same test with --asr.batching.enabled=false works: 2 and 3 concurrent sessions decode correctly, with the same latency as one session. One session with batching on also works.
This may share a root cause with #48 (same assert, backtrace there is in CacheAwareEncoder::EncoderBatcher), but the trigger here is different: no zero-PCM silence is involved, only concurrent sessions that the batcher combines.
Environment
- nemo-speech
0.2.0, prebuilt release archivenemo-speech-0.2.0-linux-x86_64-vulkan.tar.gz(SHA-256 matches the release) - Linux x86_64, Intel Core Ultra 7 258V, Intel Arc 140V iGPU (
Intel(R) Graphics (LNL)), Mesa Vulkan driver, API 1.4.354 - Models:
nvidia/nemotron-3.5-asr-streaming-0.6b(nemotron-3.5-asr-streaming-0.6b.q8_0.gguf) andnvidia/Nemotron-3-Diarization(Nemotron-3-Diarization.q8_0.gguf)
Repro
Server:
nemo-speech serve --asr-model nemotron-3.5-asr-streaming-0.6b.q8_0.gguf \
--diar-model Nemotron-3-Diarization.q8_0.gguf --device vulkan --port 8091 --no-ui \
--asr.endpointing.enable=true --asr.batching.enabled=true
Client: N WebSocket connections at the same time to /v1/audio/transcriptions/realtime. Each sends one session.update (word_timestamps: true, speaker_diarization: true or false), then streams the same 16 kHz mono PCM16 WAV as binary frames of 80 ms, paced at real time, then input_audio_buffer.commit. Audio: 88 s, test_files/diar/ami_en2002d_2132.wav from this repo followed by 28 s of French speech (the AMI file alone should also do).
| batching | diarization | sessions | result |
|---|---|---|---|
| on | on | 1 | OK: 20 finals, 191 words |
| on | on | 2 | first run: server aborted, /work/llama.cpp/ggml/src/ggml-cpu/ggml-cpu.c:1280: GGML_ASSERT(ne3 == ne13) failed (several times), SIGABRT. Second run: no abort, but each session got 0 deltas and one empty final |
| on | off | 2 | each session: 0 deltas, one empty final |
| off | on | 2 | OK: both sessions 20 finals, 191 words |
| off | on | 3 | OK: all sessions 20 finals, 191 words; latency equal to 1 session |
On the CPU backend, 2 sessions with batching on did not fail like this (they fell behind real time on this CPU, which is expected).
Expected
Concurrent realtime sessions decode correctly with batching on, or the server keeps sessions apart when it cannot batch them.
Workaround
--asr.batching.enabled=false.
Related
Separate from the above: in serve, asr.endpointing.enable is false by default, so a realtime session gets one final at commit, and the endpointing_ms field in session.update has no effect on its own. If that default is intended, a note in docs/api.md next to endpointing_ms would help.
Activity