Repository navigation
[WIP] Whisper - #260
Draft
hunhoffe wants to merge 45 commits into
Draft
[WIP] Whisper#260hunhoffe wants to merge 45 commits into
hunhoffe wants to merge 45 commits into
Conversation
… blocks An encoder-decoder's layer attends over an encoder's output through a third block: CrossCache holds the encoder's keys and values, written once an encoding and attended over by every call after; CrossAttention projects a call's own q alone, its keys and values the encoder's, projected once (keys_values); CrossDecoderLayer is DecoderLayer with that third block between its two. Call grows a source field, the encoder's rows a cross attention reads. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A depth-wise-dense 1-D convolution over rows (time): the kernel's consecutive rows gathered whole and projected in one GEMM, plus the bias on every row. Whisper's audio encoder stacks two of these (its conv1, conv2) ahead of its transformer layers. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…mp, RowMax LogMel takes mel_scale and norm (filterbank's, transformers' mel_filter_bank) and power=2 and decades: Whisper's log10 held to that many decades below the clip's largest, scaled as (x + 4) / 4 and rounded once (Log10Clamp), the largest itself two passes of RowMax. Magnitude takes power, 2 for the power spectrum Whisper's feature extractor reduces to mel bands. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A CausalLM holding its encoder: the encode version runs the encoder and projects its output into each CrossDecoderLayer's CrossCache, which the decode step and the prompt chunks then read, bounded to the call's source rows. Whisper's decoder attends over its audio encoder this way. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… and Layer A bidirectional audio encoder (WhisperEncoder, an iron.lm.Tower over LogMel and two Conv1ds) feeding a causal text decoder through cross attention (Whisper, an iron.lm.EncoderDecoder), judged by Hugging Face's own model in float32. Device-free: a toy shape against transformers' recorded logits, with tanh and exact GELU, and an fp16 checkpoint rounded once for the device. On the NPU: the same toy, and large-v3-turbo from its checkpoint over a LibriSpeech clip. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
whisper-small (and base/tiny/medium) ship 80 mel bins, not large-v3-turbo's 128: num_mel_bins doesn't divide GEMM's 64-wide tiles (conv1's K, 3 taps a channel; the filterbank GEMM's N), and 3000 frames of 80 mels don't factor into LogMel's two 16-lane reduction passes either. Pad conv1's channels and the filterbank's mel columns with zero weights to the next 64, and compute the clip's largest over all the GEMMs' frames (a whole 128s) and padded mels rather than just the real ones, since the padding is the wave's zero tail and the filterbank's zero columns: never above the real data's largest. Large-v3-turbo's 128 mels already divide 64, so this is a no-op for it. Also round whisper-small's weights to bf16 in bf16(): it ships float32 (torch_dtype), not large-v3-turbo's float16, and the device graph needs bf16 throughout. Verified device-free: whisper-small's real shape (config.json's 80 mels, 12+12 layers) resolves all three versions (decode, prompt, encode) on npu2, and large-v3-turbo's still does too (no regression). The existing toy and large-v3-turbo-shaped device-free tests in tests/applications/whisper/test.py still pass (7 passed, 3 deselected: the real device and extensive accuracy tests).
… and small Follows how llama's test.py covers 1B and 3B: the model fixture (renamed from large, since it now also loads small) is parametrized over CHECKPOINTS, and test_whisper_large_v3_turbo_accuracy is now test_whisper_accuracy[whisper-large-v3-turbo|whisper-small]. Verified device-free: pytest --collect-only lists both parametrized test ids, and the toy and large-v3-turbo-shaped device-free tests still pass (7 passed, 4 deselected: the real device and extensive tests).
Mentions its 12+12 layers and 80 mel bins (padded to large-v3-turbo's 128 for the GEMM tiles) beside large-v3-turbo's shape.
… there
Every version (decode step, prompt chunk, encode) compiles device-free for
npu1 at both large-v3-turbo and small (scratch_npu1_survey.py, through
coord/heavy), so the toy device test and the real-checkpoint accuracy test
move from npu2-only to supported_devices("npu1", "npu2"); the resolve check
is parametrized over both device fixtures. Verified: pytest
tests/applications/whisper/test.py -m "not extensive and not
supported_devices" (10 passed) and -k test_each_version_resolves (6 passed,
3 versions x 2 devices), device-free on xcoradaie218.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Mirrors ../npu1_compile_check.py's pattern (random weights at a seed, from a real checkpoint's header; feeds=decode chains the full-ELF Emit where one is picked) for Whisper's decode step, prompt chunk and encode version, since that shared script's model list does not cover Whisper. Used to confirm all three compile for npu1 at both large-v3-turbo and small. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Whisper's package exported no model (python -m iron.lm.tune whisper found none to tune) and its class named no profiles directory, so even record()ing a Whisper graph refused once one existed (Graph._profile() requires profiles/<device>.json to exist for a class that names a directory). Exporting Whisper and giving it profiles/ follows llama's own pattern exactly. tune.calls() also only unpacked two shapes, CausalLM's own (the decode step and a prompt chunk); EncoderDecoder.shapes() appends the encoder's own after those two, so tuning a Whisper-shaped model raised unpacking three into two. The decoder's two are always first by EncoderDecoder.shapes()'s contract, so *_ absorbs any more without changing any other model's tuning (CausalLM.shapes() returns exactly two today). Verified device-free: Whisper.tune_calls() builds its decode step's and a 256-token prompt chunk's Timed calls against large-v3-turbo's header (python -m iron.lm.tune needs a real device past this point, for the timing itself), and tests/applications/whisper/test.py's device-free suite still passes (10 passed) now that profiles/npu1.json and npu2.json exist. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… empty
npu1 has no device reachable here, so profiles/npu1.json's 83 entries
(decoder, cross attention and the encoder tower's LogMel/Conv1d/GEMM/MHA,
at both large-v3-turbo and small) are not timed: scratch_npu1_profile.py
records each version device-free and takes resolve(dev)'s own choice at
every shape with a real tunable search space (iron.tune.autotune.Shape's
dims, the same ones a real tune keeps), skipping the few whose dims are a
per-call object a profile cannot key on (Copy's access patterns). These
are a compile-validated starting point to tune for real on Phoenix, not
a substitute for it.
profiles/npu2.json starts empty ({"entries": []}): record() now requires
the file to exist once a class names a profiles directory, and the real
tuning run (python -m iron.lm.tune whisper, npu2 reachable here) rewrites
it with measured settings in its own commit.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Timing Whisper's decode step and prompt chunk raised KeyError('source')
at the first operator the recorder bound one of EncoderDecoder's
per-call values to: Timed.op_points() evaluates each Point's own values
alone (Timed.op_values takes a Point's values in place of the call's base
ones, not merged with them), and tune.calls()'s Points never named source,
the cross attention's bound on the encoder's valid rows. A full clip's
(encoder.soft_rows, what encode() sets it to) at every point is a plain
model that defines no encoder: extra is empty and nothing changes for it.
Verified device-free: every operator of Whisper's decode and prompt
recordings resolves op_points() without error now (0 of 247, down from an
immediate KeyError on the first); the real timing past this point needs
npu2, queued through coord/heavy.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Every version compiles for npu1 device-free at both checkpoints now (scratch_npu1_survey.py), and its device tests cover npu1 too; this records that, and is honest about the profiles: npu1.json is compile-validated, not timed, and npu2.json is still the empty seed a real tune of large-v3-turbo's decode step and prompt chunk did not finish within one coord/heavy slot (570 s, 21 of 29 shapes in before the cutoff). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
autotune() only returned its Profile at the very end, so a run coord/heavy's time budget cut off (21 of 29 Whisper shapes timed, past 570s on npu2) saved nothing: the next run started every shape over. autotune() now takes path= and saves after each shape, finished or reused, and treats a shape the given profile already names every searched tunable of as already decided (nothing left for time_shape() to move), so a profile a killed run wrote resumes from where it stopped instead of re-timing shapes it already settled. tune() (iron/tune/cli.py) passes its own path through; its final save is now just belt and suspenders. Verified device-free (iron/tests/infrastructure/test_autotune.py): a profile already naming every tunable of every shape iron.tests.support.graphs.Chain records is reused rather than timed again (no time_shape progress logged, same entry count written incrementally), with no Cache given, so a shape that fell through to real timing would need to build and dispatch for real. Confirmed the test catches the regression: disabling the new skip made it fail (time_shape really ran, 18.8s instead of 0.02s, and logged progress lines the assertion catches regardless of whether real hardware is present). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
tune.calls() checked isinstance(model, EncoderDecoder) to add source to
every timed point; tuning is meant to stay model-agnostic, since every
model gets tuned eventually, not grown a special case per kind. tune_values()
is CausalLM's own hook instead ({} by default), EncoderDecoder overriding it
to its source (encoder.soft_rows, what encode() sets it to for real calls).
The *_ shapes() fix from the prior commit is unrelated to this and stays
(every model's two decoder versions are still first, by shapes()'s own
contract).
Verified device-free: every operator of Whisper's decode and prompt
recordings still resolves op_points() without error (0 of 247), and
tests/applications/whisper/test.py's device-free suite still passes (10
passed).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Fix, don't document: a profile's timed-or-not status isn't README material, and npu1.json's untimed entries belong in the Phoenix brief's tuning step, which already says so. Keeps the one fact worth a reader here: every version compiles for npu1 too, with the device tests to show it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Resuming Whisper's real npu2 tune (the prior commit) crashed immediately: Copy's first found shape named src and dst, each an exact TensorAccessPattern off Copy's own lambda default, and Profile.save() can't write one to JSON. This was always latent in autotune()'s own save, at the very end before this session's resume fix; incremental saving just surfaced it on the first shape instead of (maybe) the last. A dim _json_safe refuses is dropped from what an entry names: an exact access pattern, or a raw dtype class, cannot recur identically across calls anyway, so keeping it would only ever narrow an entry to one call it is never asked about again. Verified device-free: a Copy given a non-default, non-contiguous src no longer surfaces it in Shape.named() (iron/tests/infrastructure/test_autotune.py), and json.dumps of the result no longer raises; confirmed the test catches the regression (reverting the filter makes it fail, "src" present). The resume test from the prior commit still passes. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Resuming Whisper's npu2 tune kept re-timing ProjectionGEMM shapes a prior
run had already finished: JSON holds no tuples, so epilogue_modes (a
tuple-typed param()) came back from Profile.load() a list, which a live
call's tuple never equals (Profile.matches()'s plain ==), so the entry
never covered the shape it was saved for. Every list a loaded entry's
fields hold is restored to a tuple; no operator field is ever genuinely a
list; so this matches what a live call's own dims always are now.
Verified device-free (iron/tests/common/test_declare.py): a profile
entry's tuple dim (epilogue_modes) saved and reloaded still matches the
call it was recorded at (tunables_for returns what it was given, not {}).
Confirmed the test catches the regression (reverting the fix returns {}).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
python -m iron.lm.tune whisper --checkpoint .../whisper-large-v3-turbo --context 256 --points 1 (the only context whole 256-token chunks of a 448 max_seq_len fits), through coord/heavy, four resumed runs (the incremental-save and profile-matching fixes from the prior commits): 29 operator shapes, every one timed for real on npu2 and none retimed needlessly across a resume. Not in this profile: small's own shapes (a second tuning pass) and the encoder tower's operators, which iron.lm.tune does not time at all (out of scope here; see the final report). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Same command against whisper-small's checkpoint, three resumed runs (one shape, Copy's last, needed a fourth to finish): 29 more operator shapes, none shared with large-v3-turbo's own entries (different M/K/hidden-size throughout), none retimed needlessly across a resume. profiles/npu2.json now holds 58 entries total, both real checkpoints' decode step and prompt chunk. Verified: tests/applications/whisper/test.py's device-free suite still passes (10 passed) with the profile in place. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
scratch_npu1_survey.py and scratch_npu1_profile.py are scratch harnesses (their own docstrings already said so), not meant to ship; they move to ../whisper-tools/, as whisper-longform's own scratch scripts already do, rather than staying tracked in this PR's tree. test_each_version_resolves (tests/applications/whisper/test.py) already covers the one check from scratch_npu1_survey.py worth keeping for real: every version resolves on npu1 too, parametrized over both devices. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ofile ac39cbe02 dropped any dim json.dumps refused from what an entry names, which broadened matching rather than narrowing it: an entry timed for one Copy access pattern would then cover every Copy whose other dims agreed, regardless of the pattern a real call constructs with. Fixed at the source instead: Profile._encode/_decode round-trip a TensorAccessPattern through its own public fields (tensor_dims, offset, sizes, strides, and padding where set), the same way ceb3e023c already restores a tuple dim, so an entry naming one is saved and still matches only the call it was recorded at. The dropping behaviour in Shape.named() is reverted in this same commit (no history rewrite). Verified device-free: two Copy calls differing only in access pattern get different Shape keys and Shape.named() keeps the real pattern (iron/tests/infrastructure/test_autotune.py); a profile entry naming one round-trips through save/load and matches only its own call, not a different pattern of the same buffer sizes (iron/tests/common/test_declare.py). Confirmed both catch the regression (reverting the encoding crashes Profile.save() again, as it did before ac39cbe02). Re-timed: profiles/npu2.json's 12 stale Copy entries (written under the dropping behaviour, so missing src/dst entirely) were removed and retimed through coord/heavy for both checkpoints; every other entry resumed instantly (already covered, nothing else needed retiming). All 58 entries remain, now correctly scoped. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CausalLM.draw() now samples through a new sampled(logits, call) hook, unchanged by default, so a subclass can bias or mask the row Sample draws from while the call's own returned logits (what cpu_reference and accuracy checks compare against) stay head()'s raw output. Whisper's WhisperConfig.from_checkpoint merges generation_config.json (suppress_tokens, begin_suppress_tokens, and the task block's first id) over config.json, so Whisper's __init__ can build two bf16 bias rows - -inf at every suppressed and non-text-block token, with begin_suppress added for the prompt chunk's own call - as plain iron.param weights added to the logits via sampled(). With no checkpoint (a toy's) every field defaults to empty/out-of-range, so the bias is all zero and the existing toy and accuracy tests are unaffected. Also adds prompt_tokens() and special(), building a decode prompt (<|startoftranscript|> <|lang|> <|task|> <|notimestamps|>) from the checkpoint's own tokenizer rather than hardcoded ids, and a token_to_id() passthrough on Tokenizer/TokenizerJson it uses. This is the suppression half of #218's TokenPolicy (Giuseppe Capaldo), the part that is a static, precomputable bias; the dynamic timestamp-pairing half follows in a later commit. Verified device-free: tests/applications/whisper/test.py -m "not supported_devices and not extensive" (7 passed), and WhisperConfig. from_checkpoint against weights-pr2/whisper-small's real generation_config.json (suppress_tokens, begin_suppress_tokens (220, 50257) and first_special_token_id 50358 match PR218's hardcoded constants). Co-authored-by: Giuseppe Capaldo <188038084+GiuseppeCapaldo93@users.noreply.github.com> Co-Authored-By: Claude <noreply@anthropic.com>
Whisper.transcribe(audio) is deliverable 1's single-window path: encode, build the <|startoftranscript|> <|lang|> <|task|> <|notimestamps|> prompt, generate greedily through Sample's suppression bias, and decode the tokens below the lowest eos id into text. It raises rather than silently truncating audio past one window, since long-form needs the timestamp-aware decoding of a later commit (transcribe_long, #218's sequential algorithm, ported to run on the NPU). `python -m iron.applications.whisper.model --wav clip.wav <path>` runs it. audio.py loads and saves mono 16 kHz PCM16 WAV over the wave module (no torch), ported from #218's whisper_audio.py's load_wav; wer.py is #218's hand-rolled normalized-word Levenshtein WER, verified by tests/applications/whisper/test_wer.py's known edit counts (no NPU, no model). scratch/npu_single_window.py and hf_single_window.py are the two-venv verification harness for deliverable 1 (the nightly venv has no torch; Hugging Face's generate needs the hf-ref one): the NPU script transcribes a LibriSpeech clip and dumps its WAV and tokens, the hf-ref script decodes the same WAV with transformers' greedy generate and checks an exact token match. Verified device-free: tests/applications/whisper/ -m "not supported_devices and not extensive" (14 passed); --wav/--help parses; audio.py round-trips a sine wave through save_wav/load_wav within int16 quantization. The device run itself is queued on coord/heavy (xcoradaie218). Co-authored-by: Giuseppe Capaldo <188038084+GiuseppeCapaldo93@users.noreply.github.com> Co-Authored-By: Claude <noreply@anthropic.com>
Says what the new Transcription section covers: the single-window path, the NPU suppression bias and why a model's returned logits stay unsuppressed, the --wav CLI, and that long-form's timestamp-pairing rules (#218's TokenPolicy, Giuseppe Capaldo) still need a device-side equivalent, since every draw runs on the NPU here. Co-authored-by: Giuseppe Capaldo <188038084+GiuseppeCapaldo93@users.noreply.github.com> Co-Authored-By: Claude <noreply@anthropic.com>
Regression coverage for the bias arrays sampled() adds before Sample draws: zero with no checkpoint (a toy's default WhisperConfig), and -inf exactly at suppress_tokens, begin_suppress_tokens (the prompt chunk's own call) and every token from first_special_token_id on, checked against a WhisperConfig built directly rather than read from a real checkpoint's generation_config.json. Verified device-free: tests/applications/whisper/test.py -m "not supported_devices and not extensive" (9 passed). Co-Authored-By: Claude <noreply@anthropic.com>
…run) transcript(name) reads an utterance's reference line from its chapter's .trans.txt, the same cached, sha256-pinned test-clean.tar.gz utterance() already reads; _cached_member() factors the shared fetch-and-cache-once logic between them. scratch/fetch_librispeech_clips.py exports #218's validation clips (Giuseppe Capaldo) as standalone WAV + transcript fixtures under /proj/rdi/staff/ehunhoff/models/librispeech-clips/, with a PROVENANCE.md recording the source, license and each WAV's sha256, for a later WER test to read without a LibriSpeech archive of its own. Not run yet: it needs the heavy download slot, currently held by another agent's test run on this host. Verified device-free: librispeech.py imports and black-formats cleanly; fetch_librispeech_clips.py parses and black-formats cleanly (not executed). Co-authored-by: Giuseppe Capaldo <188038084+GiuseppeCapaldo93@users.noreply.github.com> Co-Authored-By: Claude <noreply@anthropic.com>
…ript Takes --wav and an optional --reference-text directly, rather than fetching a LibriSpeech clip inline, so it doubles as deliverable 4's single-window validation tool once fetch_librispeech_clips.py's clips land: word_error_rate(reference, text) against #218's bar, same WAV dumped for hf_single_window.py's exact-token-match check. Verified device-free: --help parses; black and ast.parse clean. Co-Authored-By: Claude <noreply@anthropic.com>
npu_single_window.py now takes --wav rather than fetching a clip inline (so it doubles as deliverable 4's tool); this is the one-shot stand-in that gets deliverable 1's own clip onto disk as a WAV, ahead of fetch_librispeech_clips.py's full six-clip export. Co-Authored-By: Claude <noreply@anthropic.com>
WhisperConfig fields for deliverable 2's timestamp-pairing rules: the other scalars TokenPolicy needs, from generation_config.json, available at Whisper.__init__ time like suppress_tokens already is. eos_token_id duplicates what Model.eos reads from the same file, since that property needs a loaded checkpoint and a bias array needs it in __init__, before one is set. Verified device-free: matches PR218's hardcoded constants on both real checkpoints (small: no_timestamps_token_id 50363, eos 50257, max_init 50; large-v3-turbo: 50364, 50257, 50); tests/applications/whisper/ -m "not supported_devices and not extensive" (16 passed). Co-Authored-By: Claude <noreply@anthropic.com>
The real device run of deliverable 1's single-window generate failed to compile: ElementwiseAdd: 51866 elements do not divide into whole 256-element lines over 1 columns x 1 channels. Whisper-small's padded vocabulary (51865 + 1, odd) factors as 2 x 25933, 25933 prime, so no elementwise op over the whole row tiles it at any practical line size (every divisor pair is 1x51866 or 2x25933, both past the 4096-element tile_cap an elementwise core holds). GEMM already handles this exact width correctly (it is the width every existing test's head() already runs); draw()/sampled() now take the head's own hidden-state input too (CausalLM.draw/sampled's new third argument, hidden, threaded from forward() alongside logits), and Whisper's sampled() projects it again through suppress_head or begin_suppress_head, two Linear instances sharing lm_head's (tied, already-padded) weight with their own bias: the suppression array itself, not added to anything, since lm_head has none. logits, head's own unsuppressed output and what a call returns, is unchanged. Qwen3Embedding's own draw() override (iron/applications/qwen3_embedding) takes the new third argument too, unused: the only other CausalLM subclass overriding draw(). Verified device-free: tests/applications/whisper/ -m "not supported_devices and not extensive" (9 passed, suppress_head/ begin_suppress_head's bias checked directly, same values as before); tests/applications/qwen3_embedding/reference (2 passed). Re-running the real device compile next; this doubles the head GEMM's cost per decode step (sampled() projects hidden again rather than reusing logits), a real cost to measure and report, not a free fix. Co-Authored-By: Claude <noreply@anthropic.com>
Per the coordinator's design: the dynamic half of #218's TokenPolicy (Giuseppe Capaldo) needs far less carried state than its own generated-so-far list suggests. A step's own input token already is TokenPolicy's generated[-1] ("last"); only two scalars cross a step boundary: penultimate_is_stamp (the call two steps back's own "last", True by length alone before a second token is drawn) and last_stamp (stamps[-1], refreshed each step with the input token where it is itself a timestamp). The forced-timestamp probability rule needs no state at all, since a step's own logits already hold both sides of the comparison. timestamp_policy.step() is this update, a pure function over one row of logits and that pair, checked against TokenPolicy itself (ported verbatim as the test's oracle) rather than asserted by hand: every masked logit must agree, not just the argmax it would draw, caught a real bug early (last_stamp needs refreshing with the input token before use, not carried in stale) that an argmax-only check would have missed. 30 random-logit windows plus four named cases (the first step, a timestamp forced to text next, a pair's floor, a lone closer barring text, and the probability-forced timestamp): 35 passed. This is the reference the NPU version (carrying the same two scalars through Whisper's decode loop, composed from existing operators) will be checked against; no device code yet. Co-authored-by: Giuseppe Capaldo <188038084+GiuseppeCapaldo93@users.noreply.github.com> Co-Authored-By: Claude <noreply@anthropic.com>
scratch/npu_single_window.py, hf_single_window.py, fetch_and_verify_clip.py and fetch_librispeech_clips.py don't belong in the PR: they are ad hoc two-venv comparison harnesses and a one-time clip-export tool, not library code or tests. Moved to ../whisper-tools (outside this clone's git tree); nothing in this repo references them. Co-Authored-By: Claude <noreply@anthropic.com>
…w over it" This reverts commit 085ca91b1bf1315a147df91e711968080d826f40.
DecoderConfig.sample_vocab_size replaces the bare pad to even (vocab_size + vocab_size % 2) CausalLM.__init__ and draw() used: the next multiple of 1024, capped so the quantum itself never exceeds vocab_size (a toy's small one would otherwise pad past what Sample's own per-core slice can hold as padding, and sample_select rejected a negative valid count). Whisper-small's padded vocabulary (51865 -> even at 51866) is 2 x 25933, 25933 prime: no tile size under an elementwise core's 4096-element cap divides it, which is what broke deliverable 1's device run. Pads to 52224 instead (0.7% more head rows), which an ElementwiseAdd resolves cleanly on both npu1 and npu2 at the default 256-element tile size - verified device-free, not assumed, in test_an_odd_factored_vocabulary_pads_to_a_tileable_width. draw() now returns logits[:, :vocab_size] (a Copy where sample_vocab_size pads past it), not head()'s raw, wider row: every caller that compares a call's own return value against a non-padded reference (a model's cpu_reference, or model.reference() directly, which Session.logits()'s own [:vocab_size] slice never reached) needs exactly vocab_size columns, as the existing test_an_odd_vocabulary_pads_the_head_with_a_row_no_draw_takes already held Session.logits() to. This is every decoder's own padding now, not Whisper's: large-v3-turbo's 51866 has the identical problem (2 x 25933 again, vocab_size itself already "even"), and nothing here reads Whisper's own config. Reverts 085ca91b1 (the second-GEMM-head attempt, which hit the same wall: F.linear's bias-add is the same ElementwiseAdd under the hood, confirmed with a device-free repro); Whisper's sampled() goes back to a single bias row added to the (now tileable) logits. Verified device-free: iron/tests/lm/test_modules.py (26 passed, the new test included), tests/applications/whisper/ and tests/applications/qwen3_embedding/ -m "not supported_devices and not extensive" (53 passed), iron/tests/lm/ and iron/tests/common/ -m "not extensive" (485 passed, 1 skipped), tests/applications/llama/, qwen3/ and qwen3_moe/ -m "not supported_devices and not extensive" (21 passed, 1 skipped). Rerunning deliverable 1's real-checkpoint device check next. Co-Authored-By: Claude <noreply@anthropic.com>
Found the real-checkpoint compile's next failure myself (coordinator confirmed): 'aie.dma_bd' op transfer length must be multiple of 4, static_len 10373 = 51865 / 5, Whisper-small's own unpadded vocab_size split 5 ways (its only clean factor). draw()'s own Copy.step(logits[:, :vocab_size]), added when sample_vocab_size started padding past vocab_size, moved exactly vocab_size elements on the device - an odd bf16 count is never a whole 4-byte word, whatever width it is split across. draw() goes back to returning logits unchanged, head()'s own row at its padded width: Session.logits() already narrows a session's own calls to vocab_size on the host (a view, not a device transfer); a caller past Session needs the same narrowing itself. No currently-passing test does (Whisper's own direct model.reference() comparisons run at TOY's vocab size, 512, which sample_vocab_size's quantum cap already leaves unpadded), so none needed updating. Verified device-free: tests/applications/whisper/, tests/applications/qwen3_embedding/ and iron/tests/lm/ -m "not supported_devices and not extensive" (162 passed). Queuing the real device run behind Copy's own odd-transfer guard (next commit). Co-Authored-By: Claude <noreply@anthropic.com>
Copy's validate() now rejects a transfer whose element count times
its dtype's itemsize is not a multiple of 4 bytes, the same class of
check Sample's own validate() already makes ("vocab must be even: the
logits move in 4-byte words"). Without it, a Copy like the one the
previous commit removed fails 10 minutes into aiecc's own lowering,
deep in a generated MLIR module with no caller-visible context
(see 13330a8d1's own repro: static_len 10373, no indication it was
51865 / 5 without reasoning backward from the number).
test_an_odd_count_of_two_byte_elements_is_rejected constructs exactly
that shape (51865, Whisper-small's own unpadded vocabulary) and checks
it is refused in milliseconds, not minutes.
Verified device-free: iron/tests/operators/test_rejected_shapes.py (81
passed, the new test included); iron/tests/operators/test_catalog.py
-k Copy and iron/tests/lm/, iron/tests/common/ -m "not extensive" (496
passed total across both, 1 skipped) show no regression on Copy's
existing declared and device-free test coverage.
Co-Authored-By: Claude <noreply@anthropic.com>
iron.operators.emit.Emit (real int32 arithmetic on a carried value, no bf16 precision loss - token ids run well past bf16's exact integer range) is exactly the carried-scalar mechanism the coordinator pointed at: a program row reads one dynamic value (a carried token) and applies a compile-time affine transform and clamp. is_stamp_row builds exactly timestamp_policy.step()'s own last = token >= timestamp_begin as one such row: an affine shift by 1 - timestamp_begin, clamped to [0, 1]. Verified device-free against Emit's own (already-verified) reference, not asserted by hand: tests/applications/whisper/test_timestamp_policy.py (36 passed), covering the boundary at timestamp_begin and token ids into the millions. penultimate_is_stamp's own carry is the same row, read from the previous step's own token; building that wiring, and the pair rule's combine of two independent carried values (is_stamp x penultimate_is_stamp, which Emit's one-dynamic-read-per-row shape cannot do and neither can any existing elementwise operator), is next - reported to the coordinator, who may have it built from existing pieces differently or ask for a minimal mlir-aie kernel request. Co-authored-by: Giuseppe Capaldo <188038084+GiuseppeCapaldo93@users.noreply.github.com> Co-Authored-By: Claude <noreply@anthropic.com>
timestamps_bias and timestamps_begin_bias are long-form windows' own static suppression, compile-time constant per version exactly like the existing suppress_bias/begin_suppress_bias: suppress_tokens and the specials between the text vocabulary and timestamp_begin (stopping there, not running to the vocabulary's end, so no_timestamps mode's bias cannot be reused as is: a timestamp must stay drawable). timestamps_begin_bias also holds the window's first generated token to an early timestamp (max_initial_timestamp_index steps past timestamp_begin), since no token exists yet for the dynamic pair rule or floor to read. NEG_BIG (-1e30, bf16) replaces -inf for this new bias, per the coordinator's construction: every real logit sits many orders above it (argmax unaffected), and a mask row at this value can be scaled by a runtime 0/1 scalar without 0 * -inf becoming NaN, once the dynamic rules (pair, floor, forced-timestamp probability - not wired in yet, blocked on a device primitive that combines two independent per-call scalars; see timestamp_policy.py and the two documents sent to the coordinator) are built on top of it. WhisperConfig.timestamp_begin (no_timestamps_token_id + 1) is new too, used throughout. Verified device-free: tests/applications/whisper/test.py (10 passed, the new test checking suppress_tokens, the specials range stopping at timestamp_begin, and the early-timestamp window against an explicit config); the full tests/applications/whisper/ -m "not supported_devices and not extensive" (53 passed). Co-authored-by: Giuseppe Capaldo <188038084+GiuseppeCapaldo93@users.noreply.github.com> Co-Authored-By: Claude <noreply@anthropic.com>
seek_advance and segments_of port #218's transcribe_long and segments_of (Giuseppe Capaldo): OpenAI Whisper's sequential windowed decoding, greedy, without temperature fallback or conditioning on previous text. Both take only a window's own drawn token ids, in the same unit (samples here, where #218's used Mel frames; the formulas themselves are unchanged). How those ids get drawn - the timestamp rules, which must run on the NPU - is timestamp_policy.py's own, still blocked on a device primitive (see the two documents sent to the coordinator, iron-next-shared/whisper-timestamp-rules-kernel.md and -iron-only.md); Whisper.transcribe_long itself, wiring a real decode loop to these, follows once that lands. Verified device-free against #218's own inline logic, ported here verbatim as the oracle (not asserted by hand): tests/applications/whisper/test_long_form.py, 50 random windows (open/ text-run/close segments, sometimes adjacent, sometimes not, sometimes cut off mid-window) plus named cases for no timestamps, two and three segments, and the open-ended last segment; 55 passed. The full tests/applications/whisper/ -m "not supported_devices and not extensive" (108 passed). Co-authored-by: Giuseppe Capaldo <188038084+GiuseppeCapaldo93@users.noreply.github.com> Co-Authored-By: Claude <noreply@anthropic.com>
test_single_window_wer_against_librispeech: Whisper.transcribe's text against its own LibriSpeech reference line, #218's single-window clips (Giuseppe Capaldo), 1919-142785-0007 and 3170-137482-0000, now at /proj/rdi/staff/ehunhoff/models/librispeech-clips (fetched by a sibling agent). A 5% WER bound, loose against #218's own measured 0.0% and 1.5%, since this checkpoint load and kernel precision are not identical to #218's Phoenix pipeline; not yet run for real (extensive, supported_devices("npu2")), pending deliverable 1's real-checkpoint device verification. test_the_joined_clip_spans_at_least_four_30s_windows: 2902-9006-0005, -0007 and -0015 joined end to end (no gap, per the coordinator: the bar is relative to HF on the same input, so the exact construction does not affect the check) span 96.4 s, needing 4 of Whisper's own 30 s windows - #218's README reports 4. This one clip and 422-122949-0013 (2 windows) both need transcribe_long (timestamp decoding on the NPU, still blocked; see the two documents sent to the coordinator), so no WER test for them yet. Verified device-free: tests/applications/whisper/ -m "not supported_devices and not extensive" (109 passed, including the new window-count check against the real clip files now in place). Co-authored-by: Giuseppe Capaldo <188038084+GiuseppeCapaldo93@users.noreply.github.com> Co-Authored-By: Claude <noreply@anthropic.com>
Tokens mirrors Golden's shape (recorded once where torch and transformers are installed, read without them otherwise, pytest --record-goldens the same switch) for a variable-length sequence of generated token ids keyed by a name, rather than one row of logits keyed by an input token sequence: Golden's own fingerprinting has nothing to check here, since the key is already the whole input (a clip's own name). HuggingFaceWhisper.generate(audio) is transformers' own greedy generate (language, task, no timestamps, built internally) over a clip: what Tokens will record as deliverable 1's golden, checked against Whisper.transcribe's own generated ids on the device. Its return excludes transformers' own internal prompt construction, confirmed against the real whisper-small run: NPU and HF now agree token for token on 4446-2275-0016 (22 tokens including the NPU's own trailing eos, which transformers' output does not repeat). Verified device-free: Tokens round-trips through write/read and raises on a missing key, by hand (no torch needed for this part); the full tests/applications/whisper/ -m "not supported_devices and not extensive" (109 passed) and iron/tests/common/test_golden.py (3 passed, 1 skipped) show no regression to Golden's own path. Co-Authored-By: Claude <noreply@anthropic.com>
test_single_window_matches_hugging_face_exactly: Whisper.transcribe's generated ids against a Tokens golden of transformers' own greedy generate, one per (checkpoint, clip), for 4446-2275-0016, 1919-142785-0007 and 3170-137482-0000. Confirmed by hand already for whisper-small/4446-2275-0016 (exact match, both venvs, see add224b5d's commit body); this wires that same check into the test suite's own goldens path so it holds for every clip and both checkpoints without the two-venv script each time. The goldens themselves (generated_tokens.npz) are not recorded yet: the attempt this session used the hf-ref venv's own older mlir_aie (1.4.5.dev22), which this clone's iron package does not import against (ModuleNotFoundError on aie.utils.compile.jit.operator) - recording needs a hf-ref venv built against the same mlir_aie pin as this clone, not just torch and transformers. scratch/pr256-ports/ whisper-tools/record_generated_tokens.py is the recording script, ready once that is sorted out. Verified device-free: tests/applications/whisper/test.py -m "not supported_devices and not extensive" (11 passed); the new test's own 6 cases (2 checkpoints x 3 clips) collect correctly under -m extensive --collect-only. Co-authored-by: Giuseppe Capaldo <188038084+GiuseppeCapaldo93@users.noreply.github.com> Co-Authored-By: Claude <noreply@anthropic.com>
Unpacking a dict[str, ndarray] into np.savez_compressed made pyright match the arrays against its allow_pickle: bool keyword. Golden.write already types its mapping dict[str, Any] for the same call. Verified: pyright 0 errors; whisper and iron/tests/lm device-free tests 140 passed. Co-Authored-By: Claude <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds Whisper large-v3-turbo and small on
iron.lm'sEncoderDecoder(log-mel features, encoder and decoder all on the NPU), on npu1 and npu2, with tuned npu2 profiles. Single-window transcription (Whisper.transcribe, a--wavCLI) matches Hugging Face's greedy tokens exactly on whisper-small. The decoding algorithms are ported from #218 by Giuseppe Capaldo: token suppression, the prompt, seek windows and segments, and WER. Thank you, Giuseppe; thecommits that port their work carry a
Co-authored-by.Waiting on:
ehunhoff/sample-suppress-ranges(per-call suppress ranges and the forced-timestamp rule insample_select/sample_combine) to merge and reach a nightly. Then: the pin bump, the timestamp rules wired into Whisper's draw,transcribe_long, and the long-form tests on whisper: add Phoenix/NPU1 encoder validation application #218's LibriSpeech clips.srv-transfer-whisper-open.zip(both checkpoints and the LibriSpeech clips).PR Merge Checklist
develcommit and pointing todevel.