Repository navigation
Arabic streaming RNNT drops words with Q4 and FP16 #71
Description
Hello,
We reproduced missing Arabic words through the unmodified
NeMo-Speech.cpp streaming API, without our app or runtime patches.
Could you help isolate the cause?
Runtime commit: 8642eaa
Model: nvidia/nemotron-3.5-asr-streaming-0.6b
Checkpoint: ea30d66debe3740a08b573244286791d423d6b3e
Q4_K SHA256:
fe2059ac5e16295957867c6c87244c83fe3e58e2fb3a9fed0933fa2475bc0429
Settings: CPU, two threads, 16 kHz mono, ar-AR, right context 0,
verbatim output and word timings enabled, diarization disabled,
token-silence endpointing at 1200 ms.
Input: four public Al-Ikhlas verse recordings, separated by
250 ms silence, repeated three times, converted to signed16 PCM.
Dataset: MuazAhmad7/Surah_Ikhlas-Labeled_Dataset
Revision: 973b5ef1d1532bcbe7e9a5a5b0dce125cc4d0dac
This is a constructed diagnostic, not natural continuous speech.
No expected transcript or Quran text is supplied to the decoder.
Q4_K returns 43 of 45 words, omitting ayah 112:2 in repetition 2.
The baseline repeat is exact. The unmodified upstream API and our
integration produce identical word records and timestamps.
A separate FP16 conversion from the same checkpoint, with tensor
shapes and metadata checked, returns 41 words with five text edits.
It also omits that fragment. Raising weight precision alone does
not resolve this case. Silence produces zero words with both.
We have not isolated model accuracy from converter/runtime
behaviour. Which reference NeMo streaming configuration or decoder
diagnostics should we compare against? We can provide a minimal
C++ reproducer if helpful.
Thank you.
Activity
AbdeensAcademy commentedon Oct 20, 2026
Follow-up with a controlled comparison against today's runtime
commit 2bdfb03.
We reproduced the exact original Q4_K SHA256 from the public
checkpoint and ran baseline/candidate/baseline/candidate on one
Linux x86-64 CPU host, using the same input and settings above.
All four runs still return 43 of 45 words, with identical observed
transcripts, word timestamps and received-audio frontiers.
The new mmap tests pass, including byte equality for all 657
model tensors. Anonymous memory falls by about 390 MiB; this is
not the total resident-memory reduction. We found no recognition
speed improvement. The first timed word still appears after
1760 ms of received audio, using 80 ms input chunks. This is an
unpaced file diagnostic, not measured microphone latency.
Could you recommend a reference NeMo streaming configuration or
diagnostic to distinguish a checkpoint limitation from converter
or decoder behaviour? We need to resolve the missing words before
enabling recitation-error feedback. The reproducer supplies no
expected Quran text to the recognizer.
Thank you.