You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Granite 4.2 3B (iron/applications/granite/), Llama's dense decoder with Granite's four scalars (attention, residual, embedding and logits multipliers), projections at Precision.HIGH.
Granite 4.0-H 1B (iron/applications/granite_hybrid/), the Mamba2 hybrid, built on Granite's classes.
Granite 4.0-H 1B
Library pieces it runs:SelectiveScan and state_slices, CausalConv1d/Checkpoints and F.causal_conv1d/ConvState. Their kernels (aie.iron.kernels.ssm) are in the pinned wheel.
Dense only. The shipped 1B runs no experts, so its MoE branch (and with it Experts and TopKRouter) stays out; the config refuses a checkpoint with num_local_experts. The MoE toy reference test and its golden go with it.
npu1: an empty npu1 profile and a TestEachStep class, as Granite 4.2 3B got in 7c34b69.
A rate fix
generate counted only the tokens through eos over the time a device loop took to draw all of them. Granite 4.0-H 1B draws eos 8th after the benched 13-character prompt, so its TPS read 1.20 where the device draws 6.68. 5c7d8fd counts every token drawn where draws come at once, with a device test of it. Granite 4.2 3B's TPS is unchanged (5.71-5.73).
Measured (Strix Halo, npu2)
Granite 4.2 3B
Granite 4.0-H 1B
accuracy, 40 steps (KL mean / max, top-1 swaps)
0.00059 / 0.0018, 0
0.0024 / 0.0063, 2
per-step form, 20 steps
0.0013 / 0.0099, 1
0.0016 / 0.0048, 2
long_context deep decode KL, 4k to whole
0.0010 - 0.0022
0.0007 - 0.0069
decode
5.7 tok/s
6.7 tok/s
tests
15, plus 40 long_context, all pass
15, plus 40 long_context, all pass
Granite 4.0-H 1B's golden gains the per-step check's 20 rows, recorded device-free and teacher-forced on the reference's greedy token.
npu1
Every per-step version of both models compiles for npu1 with no device bound. Granite 4.2 3B's TestEachStep passed on Phoenix before (7c34b69); Granite 4.0-H 1B and both models' long_context deep checks first run on npu1 here, in CI. Each long_context comment says "npu1 still unmeasured" until then.
CI
Each runner needs granite-4.2-3b/ and granite-4.0-h-1b/ under /srv. srv-transfer-granite.zip holds both, with a SHA256SUMS; Granite 4.0-H 1B's golden.npz there replaces the 10-23 copy, which predates the per-step rows.
The nightly grows: long_context's whole-context prompt is about 50 min for Granite 4.2 3B and 12-14 min for Granite 4.0-H 1B on npu2, and longer per step on npu1.
Recurrent invalidation:Session.forget() now calculates the safe restart boundary using the original processed endpoint, before truncating history. The 100-token → forget(70) → 71-token logits regression rewinds to 64 and matches a fresh run. Tests also cover forgetting at zero, a chunk boundary, and the current endpoint. Checkpoints.restart() itself needed no change.
SelectiveScan blocks: compatibility checks reject nonpositive blocks, nonmultiples of SLICE, and block row counts that do not divide rows. Tests cover explicit and automatically resolved column counts, plus valid block controls.
CLI: added main(GraniteHybrid) under the module entry-point guard, with a subprocess --help regression.
Random weights: the checkpoint-specific early-EOS timing test now skips when random_weights is not None, including seed 0. The trained-checkpoint assertion and TPS calculation are unchanged.
Validation: all 139 relevant shape/shared-LM/Granite reference tests passed; the 21 targeted cases also passed after CodeQL. Temporarily removing the fixes reproduced the invalid-block, non-boundary-forget, and missing-CLI failures. Black, REUSE, secret scanning, and CodeQL passed (zero alerts).
Not done: no performance tuning, broad operator/library deduplication, or dependency changes. No NPU or checkpoint was available here, so real-device EOS timing and throughput were not rerun; random-weight device-test collection was checked, but its hardware execution was not.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Unresolved issues can cause unnecessary decoding after an initial EOS and graph-recording failures when convolution chunks are smaller than the required history.
No open issues 🎉
2 previously missed issues
In code that hasn't changed since last review
Handle chunks smaller than convolution history
iron/functional.py:157
The staging buffer assumes chunk >= K - 1, but neither CausalConv1d nor Checkpoints.of() enforces that constraint. For a four-tap convolution with a two-row chunk, first is -1 and flat[-1:2] has zero rows, so copying the three-row history fails while recording the graph. Please reserve at least K - 1 history rows independently of the chunk size, or explicitly reject this configuration at construction.
Avoid repeated decode steps after first-token EOS
iron/lm/model.py:533
When the first token is EOS, device_draws() has not yet entered loop.steps(max_tokens - 1). With at_once true, this condition now runs every remaining decode step even though generate() returns only the initial EOS, potentially delaying completion for a large token budget. Retain the early exit for first-token EOS while counting already-drawn tokens when EOS occurs later.
@copilot please address review findings:
Unresolved issues can cause unnecessary decoding after an initial EOS and graph-recording failures when convolution chunks are smaller than the required history.
No open issues 🎉
2 previously missed issues
In code that hasn't changed since last review
Medium severity Handle chunks smaller than convolution history
iron/functional.py:157
The staging buffer assumes chunk >= K - 1, but neither CausalConv1d nor Checkpoints.of() enforces that constraint. For a four-tap convolution with a two-row chunk, first is -1 and flat[-1:2] has zero rows, so copying the three-row history fails while recording the graph. Please reserve at least K - 1 history rows independently of the chunk size, or explicitly reject this configuration at construction.
Medium severity Avoid repeated decode steps after first-token EOS
iron/lm/model.py:533
When the first token is EOS, device_draws() has not yet entered loop.steps(max_tokens - 1). With at_once true, this condition now runs every remaining decode step even though generate() returns only the initial EOS, potentially delaying completion for a large token budget. Retain the early exit for first-token EOS while counting already-drawn tokens when EOS occurs later.
@copilot please address review findings:
Unresolved issues can cause unnecessary decoding after an initial EOS and graph-recording failures when convolution chunks are smaller than the re...
Convolution history: chose the explicit-rejection option. CausalConv1d now raises a descriptive ValueError at construction when prefill_chunk < kernel - 1, before graph recording. Eight cases cover invalid chunks (including four taps/two rows), exact-history boundaries, and valid larger chunks. Checkpoints remains unrestricted for other recurrent states. Smaller-than-history convolution chunks are deliberately not supported by this fix.
First-token EOS: generation now stops before entering loop.steps() when the initial draw is EOS. Later EOS in batched generation still counts every already-drawn token without returning tokens past EOS. Twelve device-free cases exercise the real device_draws() iterator with a fake loop/clock, checking decode calls, output, TTFT, and TPS for first/later/no EOS, streaming/generation, and one-/six-token budgets.
Validation: both regressions failed before their fixes. All 66 targeted tests passed, including both Granite reference suites. The broader available suite passed 82 tests, with 18 optional-tokenizer skips. Two additional checkpoint-dependent checks failed because this sandbox lacks the Gemma4/EmbeddingGemma2 files; excluding that file produced the green broader run. Black, REUSE, secret scanning, and CodeQL passed (zero alerts).
No dependency changes, performance tuning, or broad deduplication. No real-NPU execution or throughput measurements were performed.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
The shape checks allow state sizes not divisible by SLICE to reach the vectorized B/C gather, causing out-of-bounds loads and invalid alignment assumptions.
No open issues 🎉
1 previously missed issue
In code that hasn't changed since last review
Require state_size to be a multiple of SLICE
iron/operators/selective_scan.py:139
Please also require state_size to be a multiple of SLICE. For example, heads=2, head_dim=16, state_size=24, groups=1 passes both shape checks, but the B/C gather emits a 16-element load at offset 72 into an 82-element combined row; it also declares 32-byte alignment for misaligned C offsets. Reject these sizes here so they cannot reach the vectorized gather with out-of-bounds accesses.
a0162c7 moved eighteen applications out of #222 to their own pull
requests, each with its checkpoints, goldens and the library code only
it exercises. Granite goes first: this ports 46a5d87's model, profiles,
README and device-free tests back in, unchanged, against today's iron.lm.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Granite's GraniteConfig covers test_config.py's key-mapping check
alongside LlamaConfig's, and long_context's APPLICATIONS holds it to
the same one-compile, any-context guarantee as Llama and Gemma 4; its
deep_kl figures are 46a5d87's own, measured at 4095 positions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
8c25885 gave every model test fixture a model.release() after the
module's (and TestEachStep's) tests, so the next model's CREATE_BO
doesn't meet a full heap behind a reference cycle still awaiting GC.
Granite's fixtures, restored from before that commit, didn't have it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
754e3ae cut iron/lm's and the three #222 applications' docstrings to
what a caller relies on, the rationale left for the commits that made
it. Granite's, restored from before that commit, kept the longer form;
this brings its module and Granite class docstrings to the same register.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
74f7004's table (cut from b1e224c6 to #222's models) left out Granite's
row when it moved out in a0162c7. Restored at its measured figure
(86.5 s on this host today, against b1e224c6's 86.5 s): its deepest
prompt chunk, at its native 131072 context, as one command.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
bd6c053, d23627e and fdaaab1 widened long_context's five context
sizes to the per-step form (an xclbin a step, the host starting each on
the token the last one drew) on npu1 and npu2; Granite's entry, restored
from before that work, was still full-ELF only. Granite has an npu1
profile (empty, every operator's device defaults), so it joins at
each_step=("npu1", "npu2") like Llama's and Gemma 4's.
Both forms measured on this host's npu2, 4096 to the whole 131072
context: decode deep in the cache KL 0.0010 to 0.0022, its deepest prompt
chunk 1.7e-8 to 1.4e-4, top-1 agreeing throughout. npu1 is still
unmeasured on this host.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Profile.make merged the call's keywords over the profile's tunables, so a
tunable passed explicitly as None (mlir-aie's 'not given') overwrote the
profile's value with None, and resolve() then chose its own. Both
Profile.make and the graph recorder, which constructs every step through
it, now drop None keywords first, as Operator construction does.
Co-Authored-By: Claude <noreply@anthropic.com>
GEMV, GEMM, DepthwiseConv1d, Limbs and Magnitude declared cases at 8
columns, which NPU1 (4 columns) refuses, so 15 of its 165 default cases
skipped there: every decode-sized GEMV among them, and GEMV,
DepthwiseConv1d, Limbs and Magnitude had no bench case on Phoenix for
the trends to track. A case of 8 columns now takes all of a narrower
device. On NPU2 every case and label is unchanged, so the benchmark
history keeps its rows.
test_npu1_runs_every_case_of_an_operator_it_runs holds the rule: an
operator NPU1 runs at all resolves every case there. All 15 cases compile
for npu1 on xcoradaie211; none has run on a Phoenix device yet.
Co-Authored-By: Claude <noreply@anthropic.com>
Zero rounds is falsy, so the command line sampled instead of checking;
one round compared nothing and printed a zero denominator. Both now stop
at the parser, before the checkpoint loads.
Co-Authored-By: Claude <noreply@anthropic.com>
It polled for 120 s and then yielded either way, so a server that died or
never came up surfaced later as a refused request. It now raises when the
server exits early or stays silent, and stops a group that outlived it.
Co-Authored-By: Claude <noreply@anthropic.com>
run_test's comment named IRON_RUNTIME, which mlir-aie reads as
NPU_RUNTIME, and its docstring a call_reference that is _call_reference's
reference; Profile.load's Enum comment was garbled; kernel_cycles guarded
on ExternalFunction.recipe_key, which every pinned mlir-aie has; and the
heap-exhaustion test set IRON_XFAIL_DEVICE_HEAP, read by nothing since
169ae38.
Co-Authored-By: Claude <noreply@anthropic.com>
The black check's verdict was grep's, which matched black's own 'Oh no!'
line once the two filters had removed the rest, so the hook passed a
file black refused. It now reads black's exit status.
Co-Authored-By: Claude <noreply@anthropic.com>
Config.to_json asked each field's value whether it had a to_json, for
Llama3RopeScaling's, which existed only to put back the rope_type its
from_dict dropped. rope_type is now a field (always llama3, which
from_dict still checks), so dataclasses.asdict is the exact inverse; a
nested Config still writes its own to_json.
Co-Authored-By: Claude <noreply@anthropic.com>
Golden read IRON_RECORD_GOLDENS at construction, so its tests set and
cleared the variable with monkeypatch. It now takes record=, None
reading the variable as before (pytest --record-goldens is unchanged),
and the tests say which they mean. Two of them ran a model's graph
reference with no device bound, so they failed on a host without an NPU,
which AGENTS.md lists iron/tests/common as needing none; they take the
npu2 fixture. The checkpoint test that needs recording off skips under
--record-goldens instead.
Co-Authored-By: Claude <noreply@anthropic.com>
Each hasattr and getattr-with-a-default now states what reaches it:
Operand re-raises its own property by looking in vars(Operand); Handle's
reflected operators are vars(Handle)'s; a broadcast operand's or a
described argument's shape is read where is_operand says it has one; a
frozen parent is a dataclass's own parameter; an extent's word is an
ExtentWord's; a checkpoint tree's path and file are its _Module's or
_Layers'; a model's repr reads a DecoderConfig's context; the model to
tune is the export whose class defines tune_calls; and a kv kernel's
first entry is read from its own fields. The tests' probes follow.
Co-Authored-By: Claude <noreply@anthropic.com>
Imports inside functions are now at the top of their module unless one
of two reasons holds, which a comment beside it names: a cycle (the
graph and iron.operators, golden and checkpoint, causal and lm.tune,
testing and declare), or an optional dependency (the models extra's
jinja2 and tokenizers, the examples' soundfile, and torch and
transformers where only recording a golden needs them). record.py's
visualize import, golden's tempfile, uuid4, pkgutil and the tests'
support imports had neither. tools.shipped, a one-line wrapper kept
lazy only by that import, is gone: its callers make Shipped themselves.
Co-Authored-By: Claude <noreply@anthropic.com>
EncoderLayer set each of DecoderLayer's fields again by hand. With its
norm a LayerNorm (DecoderLayer.norm) and its MLP and NoCache passed in,
DecoderLayer's constructor makes the same layer, in the same order.
Co-Authored-By: Claude <noreply@anthropic.com>
Each xdist worker and the controller made its own CSVReporter over the
same path, each renaming its rows over the file, and the controller,
which ran no test and so held none, wrote last: pytest -n auto with
--csv-output left an empty file. The reporter is now a plugin the
controller (or a serial session) registers, recording from the reports
every worker sends it (pytest_runtest_logreport carries the
record_property figures and the bench mark); the tests running at once
are a set, each counted a failed check until it ends, as before.
Co-Authored-By: Claude <noreply@anthropic.com>
Its two fifo-type lambdas served one fifo each, its slices lambda made the
same pattern twice per core, now a list made once, and _slices(op) is the
core_slices property. Its default cases pass on npu2 as before.
Co-Authored-By: Claude <noreply@anthropic.com>
The previous commits' isinstance checks lost pyright's narrowing in two
places and named a field its stubs lack in a third: the extent word's
divisor is read under its own isinstance, a nested dataclass instance
(not its class) goes to asdict, and a frozen parent's dataclass
parameters are read through cast(Any, ...).
Co-Authored-By: Claude <noreply@anthropic.com>
Gather and GatherWords declare no Testing (a gather's control words name
device addresses), so no gather had a row in the benchmark history. A
prompt chunk's 2048 ids from EmbeddingGemma 2's table are now checked
against numpy each call and timed end to end, the host's encoding of the
ids included: 434 us on xcoradaie211 (npu2), 4.8 GB/s of rows.
Co-Authored-By: Claude <noreply@anthropic.com>
Recording.dot() had no caller outside its own test, so this drops
iron/common/graph/visualize.py, Recording.dot(), the test, and the
graphviz apt package the CI image installed only for its dot binary.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
The selective-scan benchmark counts only output elements rather than recurrence and readout arithmetic, understating reported throughput by at least 160×.
NPU_RUNTIME gets its explanation in the programming guide's "Getting
Set Up" alongside NPU_CACHE_HOME, with AGENTS.md trimmed to a pointer;
DOCS_GITHUB_REF stays an inline comment at its one call site, matching
the other CI-only vars.
SelectiveScan inherited Operator.ops()'s one-op-per-output-element
count, understating its bench throughput about 160x: the state update
and C readout do five ops per (row, head, head_dim, state_size)
element the output doesn't carry.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
mkdocs build --strict only ran on push to devel, so a broken doc link
or anchor surfaced after merge; ci-lint.yml now runs it as a PR check.
docs/models.md hand-duplicates README's models table with GitHub-
absolute links in place of relative ones, since it isn't a symlink of
README the way installation.md is. Each now points at the other so a
new model updates both.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Brings the Granite models in under the lint and type checks this
branch adds:
- Granite's new docstrings take the one-line summary and imperative
mood ruff's Google convention asks; the local `I` (Mamba's inner
width) is `d_inner`.
- `SelectiveScan` joins `iron.operators`'s literal `__all__`, so it is
catalogued as devel's computed list catalogued it.
- `SelectiveScan.compatible()` names the device it needs, as MHA's
build does, and its one arith product carries the operator ignore the
other eudsl arith sites do.
- devel's `scripts/hooks/pre-push` tweak and the test it added go with
the hook, which the pre-commit config replaces here; the visualize
module devel removed stays removed.
- devel's Docs job joins `ci-lint.yml` beside this branch's Types job.
ruff, black, REUSE and every pre-commit hook pass over the repository;
pyright reports no errors against the pinned wheel with mlir-aie#3999
laid over it; `mkdocs build --strict` passes; 2252 tests collect, and
the docs, common, lm, Granite reference and SelectiveScan tests pass
on npu2.
Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
hunhoffe
left a comment
This PR contains two models:
iron/applications/granite/), Llama's dense decoder with Granite's four scalars (attention, residual, embedding and logits multipliers), projections atPrecision.HIGH.iron/applications/granite_hybrid/), the Mamba2 hybrid, built on Granite's classes.Granite 4.0-H 1B
SelectiveScanandstate_slices,CausalConv1d/CheckpointsandF.causal_conv1d/ConvState. Their kernels (aie.iron.kernels.ssm) are in the pinned wheel.ExpertsandTopKRouter) stays out; the config refuses a checkpoint withnum_local_experts. The MoE toy reference test and its golden go with it.TestEachStepclass, as Granite 4.2 3B got in 7c34b69.A rate fix
generatecounted only the tokens througheosover the time a device loop took to draw all of them. Granite 4.0-H 1B drawseos8th after the benched 13-character prompt, so its TPS read 1.20 where the device draws 6.68. 5c7d8fd counts every token drawn where draws come at once, with a device test of it. Granite 4.2 3B's TPS is unchanged (5.71-5.73).Measured (Strix Halo, npu2)
Granite 4.0-H 1B's golden gains the per-step check's 20 rows, recorded device-free and teacher-forced on the reference's greedy token.
npu1
Every per-step version of both models compiles for npu1 with no device bound. Granite 4.2 3B's
TestEachSteppassed on Phoenix before (7c34b69); Granite 4.0-H 1B and both models' long_context deep checks first run on npu1 here, in CI. Each long_context comment says "npu1 still unmeasured" until then.CI
Each runner needs
granite-4.2-3b/andgranite-4.0-h-1b/under/srv.srv-transfer-granite.zipholds both, with aSHA256SUMS; Granite 4.0-H 1B'sgolden.npzthere replaces the 10-23 copy, which predates the per-step rows.The nightly grows: long_context's whole-context prompt is about 50 min for Granite 4.2 3B and 12-14 min for Granite 4.0-H 1B on npu2, and longer per step on npu1.