You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Lint and type-check as mlir-aie does, publish the API reference, and draw CI's benchmarks on the site - #262
This brings IRON's tooling in line with mlir-aie's, publishes a Python API reference generated from IRON's docstrings, and moves the benchmark history onto the website's own Benchmarks page. Getting the numbers there faithfully turned up a few reporting bugs in the CI scripts, fixed here in their own commits. The docs now leave current performance figures to CI.
Lint and types, as mlir-aie does
ruff lints every Python file with mlir-aie's rule set: E, F, I, N802, N816, and pydocstyle in the Google convention with D205, D401 and D417. The mechanical fixes are in their own commits. Docstring-only edits leave every file's syntax tree unchanged. ProjectionGEMM.pack_B is now pack_b.
.pre-commit-config.yaml, modeled on mlir-aie's, runs black, ruff, pyright, clang-format, REUSE, actionlint and the hygiene hooks. It takes over the checks scripts/hooks/pre-push and scripts/clang-format-wrapper.py carried until now.
CI runs every hook in the lint job. The Types job runs pyright on Python 3.11 to 3.14, against the mlir_aie and pyright versions requirements.txt pins.
Bugs found on the way:
Sample.runs could return None for an unsplittable tail.
flm/goldens.py iterated Testing.cases without resolving them.
EmbeddingGemma kept its text processor in Model.tokenizer, so generate() failed deep inside with an AttributeError. It now has its own processor attribute, and a model that doesn't generate says so.
Two regressions from the pyright pass, each fixed in its own commit:
Gemma 4's load_state_dict read self.layers, which CausalLM sets only after loading, so constructing Gemma4 failed. It reads the decoder's layers again, and Gemma4Layer declares per_layer_embedding so pyright still checks the assignment.
Operator.resolved() now refuses to resolve without a device, rather than passing None to mlir-aie's resolved(dev: Device). Two device-free tests relied on that; they now bind npu2, as AGENTS.md has device-free tests do.
flm.ProjectionGEMM names B's layout, and the mlir-aie bump
Xilinx/mlir-aie#3995 extends #3941's rule that a host shape may not depend on a tunable() to a Select() flag. ProjectionGEMM's B picked its host shape (pack_b's bfp16 blocks, or bf16) on bfp16_b, a tunable filled from the device, so on the new mlir-aie nothing imported.
The caller now says:bfp16_b is a param(), True for a packed B on NPU2 at precision="medium". It defaults to False because a graph step infers B's shape before any operator exists. resolve(dev) refuses a value the device doesn't run and names the one to pass, so a missing True on NPU2 is an error, not a slower path.
Models and NPU1 are unchanged:F.linear reads B as stored, bf16 everywhere. The NPU2 callers that pre-pack B (flm/gemm's tests, the dequant contract, the gemma4 engine, two toolchain tests) pass True.
The bump (requirements.txt) also takes Runtime.merge's new run and a Select()'s None branch.
The Python API reference
The site gains an API section built the way mlir-aie builds its own: mkdocstrings with griffe reading the source statically. The Pages build needs neither mlir-aie nor XRT.
Operator signatures: an operator's constructor is mlir-aie's generic Operator.__init__(**given). A griffe extension (docs/hooks/operator_fields.py) reads each operator's param() and tunable() declarations into a keyword signature and an Args table, the way griffe reads a dataclass. test_operator_fields.py holds every catalogued operator's documented signature to its runtime declaration.
Missing operators:resample/ and five flm/ operator directories become regular packages. griffe doesn't descend into a namespace subpackage, so Resample, ResampleTaps, PatchPositions and all six flm operators were missing from the reference. The same 2029 tests collect as before.
Annotations: parameters with an Args: entry but no annotation get the types their callers already pass, so the strict docs build passes.
Benchmarks on the site, faithfully
The standalone chart page (#223) settled the right shape for the data: one history per suite, one line per benched parametrization. This brings that onto the site's own Benchmarks page, in its theme and navigation. First, three fixes to what the CI scripts report:
One date per run. The application suites run each model in its own pytest process, and each process stamped its own start time. A run is read as the rows of one date, so the nightly's 10% timing gate compared only long_context, the last model to run. The trends and PR comment also reported "operators added and dropped" that were just other models of the same run. The test action now passes the run's one UTC date to every process (--csv-date).
Each metric's direction. A move was judged "higher is better" only if the metric's name began with TPS, so a drop in GenerateTPS, StreamTPS or Cosine read as an improvement. pretty_common.METRICS now names every metric's unit and direction once. Norm and Tokens have no better side and are marked neither way.
Retention, as mlir-aie keeps its kernel checks. Each benched test keeps its runs from the last 90 days, then its own newest run of each week, at most 400 runs. Every other test keeps the suite's last two runs, which is all the reports read. On today's ci branch that takes 64 MB of CSVs down to 8.3 MB.
The page (docs/benchmarks.md; build_pages.py writes its data.json before mkdocs build) shows:
Each suite's latest run on each device: when it ran, at which commit, how many tests failed, and what moved past the PR comment's own gates (at least 5%, and more than twice the measured spread).
Any benchmark's history, by suite and metric:
one chart per device on a time axis, with runs whose checks failed marked with a ✕, and each point linking to its commit;
a table of every parametrization's latest value and change, noting which ones the latest run no longer ran.
The PR comment now links each moved benchmark to its history on the page. The old benchmarks/charts/ URL redirects there, keeping an operator's link.
Current performance figures live in CI
flm/gemm's headline comparison at M=1024 K=1536 N=6144 (the port against the shipped overlay and iron.operators.GEMM) is now three benched cases in the extensive suite on the eight-column NPU2. On a Strix today: 1224 µs, 2682 µs and 3709 µs, every output checked. The README links that history instead of quoting a ratio.
Design-rationale figures in flm/gemm's README stay as the evidence for each choice, each stamped with the PR it was measured in.
Timeout table: the README's driver-timeout section links long_context's nightly TTFT in place of one run's table.
Model READMEs that mention timing tests link their history on the page.
After merge
The application runs already on the ci branch are still split by model: the latest examples run on each device, and the one nightly so far. Once this merges, I'll join each of them into its one date on ci with a one-off script. Until then the page lists those models as missing from their latest run.
Testing
On a Strix NPU2 (xcoradaie213), against the old pin: pytest -m "not extensive" iron/operators/ iron/tests/, what small.yml runs, gives 1470 passed, 203 skipped.
AGENTS.md's device-free suite, without XRT: 845 passed, 545 skipped.
The CI script tests (test_ci_results, test_timing_gate, test_build_pages, test_merge_results): 23 pass, each run as CI runs the scripts.
Each benchmark commit's tree passes them on its own.
The run-date test fails without its fix, missing the first model's move.
mkdocs build --strict passes.
I built the Benchmarks page from the real ci branch and checked it in headless Chromium, in light and dark mode, with and without data. There were no JavaScript errors.
The three new flm/gemm bench cases pass on npu2 (Strix).
The pin moves to mlir_aie==1.4.5.dev119+g064a172, the nightly on v1.4.5-alpha.2's commit (the release's own build was cancelled; its wheels are the same code), with llvm-aie==22.0.0.2026102001+4f5df13d. Against it pyright reports 0 errors over iron/ (93 against the old pin), and every pre-commit hook passes.
On a Strix NPU2 with the new pin: the PR suite gives 1470 passed, 203 skipped, and flm/gemm's extensive headline, overlay and dequant cases 16 passed.
The mechanical part of adopting mlir-aie's ruff rules (E, F, I, N802,
N816): imports sorted and unused ones dropped, `iron.operators` and
`iron.operators.flm` list `__all__` literally so checkers see the
exports, `ProjectionGEMM.pack_B` is `pack_b` (callers and READMEs
updated), and ambiguous single-letter names renamed. Numpy's `.T` and
MHA's `O` operand keep their names with a `noqa` saying why.
Co-Authored-By: Claude <noreply@anthropic.com>
Each docstring gets a one-line summary, a blank line, then its detail
(D205), and a function's summary is in the imperative mood (D401), as
mlir-aie's ruff configuration requires. Missing `Args:` entries are
added (D417). Docstrings only: with them blanked, every file's syntax
tree is unchanged.
Co-Authored-By: Claude <noreply@anthropic.com>
EmbeddingGemma and its Multimodal kept their task-prefixed
`TextProcessor` in `Model.tokenizer`, the slot `generate()` and
`stream()` read as a `Tokenizer`, so generating from an encoder failed
deep inside with an AttributeError. The processor is now `processor`
(the constructor argument too), `embed()` without one says so, and
`Model._draw` raises a clear NotImplementedError on a model that does
not generate.
Co-Authored-By: Claude <noreply@anthropic.com>
Fixes what pyright reports over `iron/` (tests aside) against the next
mlir-aie, whose Python API now types what IRON reads: Optionals narrowed
where a value is required, explicit checks where an invariant was
implicit, annotations corrected (`Self` returns, `Device | None`), and
base methods whose subclasses each name their own parameters
(`Operator.reference`, `Elementwise.reference`, `Tower.inputs`,
`HuggingFace.logits`) take `*args: Any, **kwargs: Any`.
`iron.common.declare.operand.concrete` reads a static pattern's sizes,
strides and offset as ints.
Bugs found on the way: `Sample.runs` could return None for an unsplittable
tail, and `flm/goldens.py` iterated `Testing.cases` instead of resolving them.
25 `# pyright: ignore` remain, each naming its rule: 12 for
eudsl-python-extras' untyped arith operators, 6 optional torch and
transformers imports, `pyxrt`, 3 traced graph `body` overrides and 3
`.mac_dims` attached to a function.
Co-Authored-By: Claude <noreply@anthropic.com>
ruff now lints every Python file with mlir-aie's rule set (E, F, I,
N802, N816, pydocstyle in the Google convention with D401), and pyright
type-checks `iron/` apart from its tests. `.pre-commit-config.yaml`,
modeled on mlir-aie's, runs black, ruff, pyright, clang-format, REUSE,
actionlint and the whitespace and line-ending hooks; one `pre-commit
install` registers them. It takes over from scripts/hooks/pre-push and
scripts/clang-format-wrapper.py, which carried those checks until now.
CI's lint job runs every hook over the repository. Its types job runs
pyright on Python 3.11 to 3.14, the versions the mlir-aie wheels are
built for, against the mlir_aie and pyright versions requirements.txt
pins, so the separate mlir-aie ref the job checked out is gone; CPU
torch and transformers are installed so the golden-recording modules
are checked too. black is pinned to the hook's 26.10.1 and targets
Python 3.11. CONTRIBUTING.md, AGENTS.md and the README describe it.
Pyright needs the next mlir-aie nightly, which types the API IRON reads
and ships MLIR's stubs; the pin moves with it.
Co-Authored-By: Claude <noreply@anthropic.com>
Each of these parameters had an Args entry but no annotation, which the
API reference's docstring parser reports and the strict docs build
fails on. They take the types their callers already pass: Device for a
device, str | Path for a path, the Callable record_property is, and so
on. Encoder's config stays Any, as Model's does, since each model's
config adds its own fields.
pyright's diagnostics are unchanged against the pinned wheel.
Co-Authored-By: Claude <noreply@anthropic.com>
resample/ and five of flm/'s operator directories had no __init__.py,
so Python imported them as namespace packages inside the regular
iron.operators. That works at run time, but griffe does not descend into
a namespace subpackage within a regular package, so the API reference
silently left out Resample, ResampleTaps, PatchPositions and all six flm
operators. Each is now a package, as AGENTS.md already describes them.
gemma4_engine/ stays a directory of build scripts.
pytest imports with --import-mode=importlib, so the same 2029 tests
collect as before.
Co-Authored-By: Claude <noreply@anthropic.com>
The site gains an API section generated from IRON's docstrings, as
mlir-aie's reference is: mkdocstrings with griffe reading the source
statically, so the Pages build needs neither mlir-aie nor XRT and the
existing workflow publishes it unchanged. Pages cover graphs,
iron.functional, the operator library, writing an operator, testing,
the language-model layer and tuning.
An operator's constructor is mlir-aie's Operator.__init__(**given), so
griffe alone would show that generic signature. A griffe extension
(docs/hooks/operator_fields.py) reads each Operator subclass's param()
and tunable() declarations, as griffe reads a dataclass's fields, into
its keyword signature and an Args table, and lists its operands and
per-call values, each described by the comment above its declaration.
test_operator_fields.py holds every catalogued operator's documented
signature to its runtime declaration: names, order and which are
required.
Python's and numpy's inventories link their types; mlir-aie's dev
inventory links aie.* once mlir-aie publishes its API under those names
(Xilinx/mlir-aie#3999).
Co-Authored-By: Claude <noreply@anthropic.com>
mlir-aie#3999 types the API IRON reads and ships MLIR's ir stubs.
Those stubs declare Context.attach_diagnostic_handler as returning
object, so places() now narrows the handler to the
ir.DiagnosticHandler it is before detaching it. _as_i32 is annotated
now, so the comment on why pyright inferred "| None" for it goes.
Against the pinned wheel with #3999's python/ and stubs laid over it,
pyright reports no errors over iron/. The autotune device tests, which
reach places() through fits(), pass on npu2.
Co-Authored-By: Claude <noreply@anthropic.com>
Brings the Granite models in under the lint and type checks this
branch adds:
- Granite's new docstrings take the one-line summary and imperative
mood ruff's Google convention asks; the local `I` (Mamba's inner
width) is `d_inner`.
- `SelectiveScan` joins `iron.operators`'s literal `__all__`, so it is
catalogued as devel's computed list catalogued it.
- `SelectiveScan.compatible()` names the device it needs, as MHA's
build does, and its one arith product carries the operator ignore the
other eudsl arith sites do.
- devel's `scripts/hooks/pre-push` tweak and the test it added go with
the hook, which the pre-commit config replaces here; the visualize
module devel removed stays removed.
- devel's Docs job joins `ci-lint.yml` beside this branch's Types job.
ruff, black, REUSE and every pre-commit hook pass over the repository;
pyright reports no errors against the pinned wheel with mlir-aie#3999
laid over it; `mkdocs build --strict` passes; 2252 tests collect, and
the docs, common, lm, Granite reference and SelectiveScan tests pass
on npu2.
Co-Authored-By: Claude <noreply@anthropic.com>
iron/lm/testing.py imports pytest where a model's device test needs it,
deferred so a model importing weights_dir does not; the Types job had
no pytest, so pyright reported the import unresolved there though it
resolves wherever IRON's requirements are installed. pytest is the one
third-party module iron/ imports that the job lacked.
Co-Authored-By: Claude <noreply@anthropic.com>
The applications suites run each model in its own pytest process, and
each process stamped its own start time. The reports take a run to be
the rows of one date, so the newest date named only the last model's
tests: the nightly's 10% timing gate compared long_context alone, and
trends.md and the pull request comment reported operators added and
dropped that were only other models of the same run.
conftest.py takes --csv-date; the test action passes the run's one UTC
date to every process. A session given none stamps its own time, now in
UTC, which the CI runners already keep.
The runs already on the ci branch stay split until they are joined
there once.
Co-Authored-By: Claude <noreply@anthropic.com>
The trends report judged a move by whether the metric's name began with
Bandwidth, Throughput or TPS, so a drop in GenerateTPS, StreamTPS or
Cosine read as an improvement, and any move in Tokens or Norm, figures
with no better side, as a regression.
pretty_common.METRICS names every recorded metric's unit and whether a
higher value is better; a metric with no direction is marked neither
way, and the report prints each value with its unit.
Rows of the application tests' earlier home, iron/applications/<name>/,
are named by that directory rather than 'test'.
Co-Authored-By: Claude <noreply@anthropic.com>
all.csv kept up to 1000 rows of every test for a year. Benched or not,
every row was rewritten on each run: 61 MB over the eight suites, 97% of
it pass/fail rows of tests no report reads past the last two runs.
merge_all.py --prune, which merge_results.sh applies to all.csv, keeps
a benched test's runs of the last 90 days, the newest of each ISO week
before that and at most 400 runs (mlir-aie's KEEP_DAYS and MAX_RUNS),
and every other test's last two runs, which is all the trends and the
pull request comment read of it. On the current branch that is 61 MB
down to about 13 MB. It replaces --max-age-days, whose only other
caller turned it off.
Co-Authored-By: Claude <noreply@anthropic.com>
The benchmark history was a standalone page beside the site, opened in
a new tab. This brings the same data and the decisions behind it (one
history per suite, one line per benched parametrization) onto the
site's own Benchmarks page, in its theme and navigation.
build_pages.py now writes benchmarks/data.json beside the page, and
publish-pages.yml writes it before `mkdocs build`. The page shows:
- each suite's latest run on each device: when, at which commit, how
many tests failed, and what moved past the pull request comment's
gates (pretty_trends' THRESHOLD and SIGMA, now named once);
- a benchmark's history, picked by suite and metric: one chart per
device on a time axis, runs whose checks failed marked with a cross,
and a table of every parametrization's latest value and change,
which also says which ones the latest run no longer ran.
A catalog parametrization's runs on each device are one series, the
'npuN-' prefix dropped. Suites are named for what they are: pull
requests, extensive, nightly. Benchmarks no longer run (the test
modules the catalog replaced) are listed apart. Metrics open on the
first of METRICS' order rather than the alphabet's, so a model opens on
TTFT rather than DifferingTokens.
benchmarks/charts/ redirects to the page, keeping an operator's link.
Co-Authored-By: Claude <noreply@anthropic.com>
The comment names what moved; the Benchmarks page now shows where it
moved from. pr_comment.py --benchmarks-page links each operator in the
table to its history in that suite and metric on the page, and ends
with a link to the page; pr-comment.yml passes the site's.
Co-Authored-By: Claude <noreply@anthropic.com>
flm/gemm's README compares flm.ProjectionGEMM, the shipped overlay and
iron.operators.GEMM at M=1024 K=1536 N=6144 on the eight-column NPU2,
a shape CI did not time. Each is now a benched case in the extensive
suite there, so the comparison is a tracked figure on the Benchmarks
page. On a Strix (xcoradaie213) today: 1224 us, 2682 us and 3709 us,
every output checked.
Co-Authored-By: Claude <noreply@anthropic.com>
The Benchmarks page is where a current figure lives. The docs now point
there rather than restate one:
- flm/gemm's README links its headline comparison, now benched, and
NPU1's benched shapes in place of their latencies, and drops the NPU1
table it said predated the k-slice change. Every figure kept as the
evidence for a design choice is stamped with the PR it was measured
in; op.py's comments point to the README rather than repeat them.
- README.md's driver-timeout section links long_context's nightly TTFT
in place of one Strix Halo run's deepest chunks.
- The model READMEs that name timing tests link their history.
- AGENTS.md's ~160 us is named as one dispatch's fixed cost, apart from
flm/gemm's ~107 us, that operator's own smallest dispatch in #200.
Co-Authored-By: Claude <noreply@anthropic.com>
hunhoffe
changed the title
Lint and type-check as mlir-aie does, and publish the Python API reference
Lint and type-check as mlir-aie does, publish the API reference, and draw CI's benchmarks on the site
Oct 30, 2026
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Suite-wide pruning can delete a benchmark’s entire older history, while removing type suppressions against the unchanged mlir-aie pin causes the Types job to fail.
Aligns IRON’s linting and type-checking tooling with mlir-aie, adds a generated Python API reference, and integrates CI benchmark history into the website.
Changes:
Introduces pre-commit checks, Ruff rules, and broader Pyright coverage, with related code and documentation cleanup.
Adds API reference pages, operator-signature validation, and discoverable operator packages.
Connects benchmark publication, shared run timestamps, retention, redirects, and performance-history links.
This pull request requires changes.
Outstanding findings:
Critical — ci/scripts/merge_results.sh:16 (2 votes): Publication pruning selects weekly dates across an entire suite rather than per test, potentially deleting a benchmark’s entire older history when coverage varies. Apply retention per row_key and test differing run dates before enabling destructive pruning.
Moderate — docs/assets/css/iron.css:108 (1 vote): The fixed 22rem chart-column minimum causes horizontal overflow in narrow content areas. Cap the minimum at the available width.
Critical — iron/common/graph/carried.py:79 (1 vote): The removed suppression remains necessary with the pinned mlir-aie wheel, which lacks #3999 and treats Emit.rows as a required constructor argument. Update the wheel and matching Peano pin before removing dependent suppressions.
Critical — pyproject.toml:41 (2 votes): The expanded Types job installs the older pinned wheel without the overlay used locally, so its matrix will fail on missing-rows diagnostics. Update mlir-aie and its matching LLVM-AIE pin, or retain compatibility with the pinned wheel.
prune() took the newest run of each ISO week across the whole suite, so
a benchmark that ran that week, but not in its last run, lost the week:
with A on June 1 and 8 and B on June 7 and 14, A kept neither. Each
benched parametrization now keeps its own newest run of each week, and
its own MAX_RUNS; the 90-day window is still measured from the suite's
newest run, and every other row still keeps the suite's last two.
On the ci branch today that is 64 MB down to 8.3 MB, measured on the
files directly; the 13 MB 1d2f0a3 gave counted the latest run twice.
Co-Authored-By: Claude <noreply@anthropic.com>
b2dc84b had Gemma4.load_state_dict iterate self.layers, which pyright
types as a bare list, in place of m.layers, which it types as
DecoderLayers that declare no per_layer_embedding. But CausalLM sets
self.layers only after load_state_dict returns, so constructing Gemma4
raised AttributeError: every device-free Gemma 4 reference test failed.
The loop reads m.layers again, each narrowed to the Gemma4Layer it is,
and Gemma4Layer declares per_layer_embedding, so pyright still checks
the assignment.
Checked: tests/applications/gemma4/reference/test.py, 16 passed (all
failed before); pyright reports nothing on the file.
Co-Authored-By: Claude <noreply@anthropic.com>
b2dc84b made Operator.resolved() refuse to resolve without a device
rather than hand None to mlir-aie's resolved(dev: Device), which only
worked for operators whose resolve() ignores the device. Two tests
built an OperatorDesign with no device bound, so on a host without an
NPU they now met that refusal before what they check: the full-ELF
dispatch-value refusal and an exported design's text. Each takes the
npu2 fixture, as AGENTS.md has a device-free test bind one.
Checked: without XRT, test_dispatch_params.py 1 passed and 6 device
tests skipped, test_declare.py 35 passed; with the NPU,
test_dispatch_params.py 7 passed.
Co-Authored-By: Claude <noreply@anthropic.com>
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
The mandatory type-check hook fails with the pinned mlir-aie dependency, so the documented installation cannot pass pre-push checks and Types jobs remain failing until the dependency pins and compatibility changes are updated.
Prevent the chart grid from overflowing narrow screens
docs/assets/css/iron.css:108
The fixed 22rem minimum prevents the chart grid from shrinking below that width. On narrow phones, or when zoom reduces the available content width, even its single-column layout overflows the page and cuts off part of the chart. Cap the minimum at the container width so the charts remain responsive.
B's host shape is pack_b's bfp16ebs8 blocks or K x N bf16 values, and
ProjectionGEMM chose between them with bfp16_b, a tunable() resolve(dev)
filled from the device. mlir-aie #3995 extends #3941's rule that a host
shape may not depend on tuning to a Select()'s flag, so on the next
mlir-aie no IRON operator imports.
bfp16_b is now a param() the caller sets: True for a packed B on AIE2P
at precision='medium', the bfp16 macs' fast path. It defaults to False,
since a graph step infers B's shape before any operator exists, and a
computed default is unknown there. resolve(dev) refuses a value the
device does not run, naming the one to pass, so a missing True on NPU2
is an error, not a slower path. b_dtype follows it as a derived param.
Models are unchanged: F.linear reads B as stored (b_col_maj), bf16 on
every device, and so does NPU1. The NPU2 callers that pre-pack B pass
True: flm/gemm's tests (each device's own value where they run on
both), the dequant contract, the gemma4 engine and two toolchain tests.
With no host shape on a tunable, the recorder's resolve-before-check
branch can no longer fire, so it goes, with param_names. The SwiGLU
test that leaned on it now resolves the recorded GEMM, which still
refuses a row count it cannot tile rather than pad it.
Checked, on this commit and mlir-aie dev64 (today's pin): device-free
suite 845 passed, 545 skipped; flm/gemm, dequant and the two toolchain
tests on a Strix NPU2, 27 passed (the extensive headline, overlay and
dequant cases among them). On dev119 with the bump that follows: the
PR suite on the NPU, 1470 passed, 203 skipped.
Co-Authored-By: Claude <noreply@anthropic.com>
The first nightly with Xilinx/mlir-aie#3999, which types the API IRON
reads and ships MLIR's stubs: against it pyright reports no errors over
iron/ (93 against dev64), so the Types job and the pyright hook pass.
The wheel is the nightly's: the v1.4.5-alpha.2 release, on the same
commit, has none, its build cancelled by the nightly's in a shared
concurrency group. llvm-aie moves to 22.0.0.2026102001+4f5df13d, the pin
mlir-aie validated at 064a172 (the same LLVM commit, a day's build
later).
Two API changes since the old pin reach IRON:
- Runtime.merge takes run, to issue steps joined on-chip together
(#3995); MergedRuntime.merge takes and passes it on.
- Select() may hold a None branch, no stream (#3995); axis_of refuses
one as resolve() and the declaration check do.
Checked on this commit: pyright 0 errors; every pre-commit hook passes;
device-free suite 845 passed, 545 skipped; on a Strix NPU2, the PR
suite (pytest -m "not extensive" iron/operators/ iron/tests/) 1470
passed, 203 skipped, and flm/gemm's extensive headline, overlay and
dequant cases 16 passed.
Co-Authored-By: Claude <noreply@anthropic.com>
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
The chart’s 22rem minimum width causes horizontal overflow on narrower screens or zoomed layouts, so it must be capped at the container width.
No open issues 🎉
1 previously missed issue
In code that hasn't changed since last review
Prevent chart overflow below the 22rem minimum width
docs/assets/css/iron.css:108
The 22rem minimum still applies when auto-fit falls back to a single column. On a screen or zoomed layout narrower than 22rem, the chart therefore extends beyond the content area and introduces horizontal overflow rather than shrinking to fit. Cap the minimum at the container width so the history remains usable on narrow screens.
Each chart's grid column was at least 22rem, so on a phone, or a
desktop zoomed in, the charts ran past the page and it scrolled
sideways. The column's minimum is now the smaller of 22rem and the
container, and a chart card may shrink below its content's width.
Co-Authored-By: Claude <noreply@anthropic.com>
The page drew a benchmark's history with no way to the operator or model
behind it. build_pages.py gives each benchmark still run a link: a
model to the Models page, an operator to its class in the operator
reference. A catalog benchmark is named for its class; the four named
for their test's directory map to theirs (flm/gemm is
flm.ProjectionGEMM). The explorer shows it beside the benchmark's
description, and the page's introduction links both pages.
Checked: every one of the 43 benchmarks the current history still runs
has a link, and each resolves to an anchor in the built site;
test_build_pages.py covers a model, a catalog operator, a directory-
named one and a retired one; mkdocs build --strict passes.
Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
hunhoffe
changed the title
This brings IRON's tooling in line with mlir-aie's, publishes a Python API reference generated from IRON's docstrings, and moves the benchmark history onto the website's own Benchmarks page. Getting the numbers there faithfully turned up a few reporting bugs in the CI scripts, fixed here in their own commits. The docs now leave current performance figures to CI.
Lint and types, as mlir-aie does
ProjectionGEMM.pack_Bis nowpack_b.iron/(tests aside) against mlir-aie's typed API (Type the Python API for downstream pyright users Xilinx/mlir-aie#3999). 25# pyright: ignoreremain, each naming its rule..pre-commit-config.yaml, modeled on mlir-aie's, runs black, ruff, pyright, clang-format, REUSE, actionlint and the hygiene hooks. It takes over the checksscripts/hooks/pre-pushandscripts/clang-format-wrapper.pycarried until now.requirements.txtpins.Sample.runscould returnNonefor an unsplittable tail.flm/goldens.pyiteratedTesting.caseswithout resolving them.Model.tokenizer, sogenerate()failed deep inside with anAttributeError. It now has its ownprocessorattribute, and a model that doesn't generate says so.load_state_dictreadself.layers, whichCausalLMsets only after loading, so constructingGemma4failed. It reads the decoder's layers again, andGemma4Layerdeclaresper_layer_embeddingso pyright still checks the assignment.Operator.resolved()now refuses to resolve without a device, rather than passingNoneto mlir-aie'sresolved(dev: Device). Two device-free tests relied on that; they now bindnpu2, as AGENTS.md has device-free tests do.flm.ProjectionGEMM names B's layout, and the mlir-aie bump
Xilinx/mlir-aie#3995 extends #3941's rule that a host shape may not depend on a
tunable()to aSelect()flag. ProjectionGEMM's B picked its host shape (pack_b's bfp16 blocks, or bf16) onbfp16_b, a tunable filled from the device, so on the new mlir-aie nothing imported.bfp16_bis aparam(),Truefor a packed B on NPU2 atprecision="medium". It defaults toFalsebecause a graph step infers B's shape before any operator exists.resolve(dev)refuses a value the device doesn't run and names the one to pass, so a missingTrueon NPU2 is an error, not a slower path.F.linearreads B as stored, bf16 everywhere. The NPU2 callers that pre-pack B (flm/gemm's tests, the dequant contract, the gemma4 engine, two toolchain tests) passTrue.requirements.txt) also takesRuntime.merge's newrunand aSelect()'sNonebranch.The Python API reference
The site gains an API section built the way mlir-aie builds its own: mkdocstrings with griffe reading the source statically. The Pages build needs neither mlir-aie nor XRT.
Operator.__init__(**given). A griffe extension (docs/hooks/operator_fields.py) reads each operator'sparam()andtunable()declarations into a keyword signature and an Args table, the way griffe reads a dataclass.test_operator_fields.pyholds every catalogued operator's documented signature to its runtime declaration.resample/and fiveflm/operator directories become regular packages. griffe doesn't descend into a namespace subpackage, soResample,ResampleTaps,PatchPositionsand all six flm operators were missing from the reference. The same 2029 tests collect as before.Args:entry but no annotation get the types their callers already pass, so the strict docs build passes.Benchmarks on the site, faithfully
The standalone chart page (#223) settled the right shape for the data: one history per suite, one line per benched parametrization. This brings that onto the site's own Benchmarks page, in its theme and navigation. First, three fixes to what the CI scripts report:
long_context, the last model to run. The trends and PR comment also reported "operators added and dropped" that were just other models of the same run. The test action now passes the run's one UTC date to every process (--csv-date).TPS, so a drop inGenerateTPS,StreamTPSorCosineread as an improvement.pretty_common.METRICSnow names every metric's unit and direction once.NormandTokenshave no better side and are marked neither way.cibranch that takes 64 MB of CSVs down to 8.3 MB.The page (
docs/benchmarks.md;build_pages.pywrites itsdata.jsonbeforemkdocs build) shows:The PR comment now links each moved benchmark to its history on the page. The old
benchmarks/charts/URL redirects there, keeping an operator's link.Current performance figures live in CI
iron.operators.GEMM) is now three benched cases in the extensive suite on the eight-column NPU2. On a Strix today: 1224 µs, 2682 µs and 3709 µs, every output checked. The README links that history instead of quoting a ratio.long_context's nightly TTFT in place of one run's table.After merge
The application runs already on the
cibranch are still split by model: the latest examples run on each device, and the one nightly so far. Once this merges, I'll join each of them into its one date onciwith a one-off script. Until then the page lists those models as missing from their latest run.Testing
pytest -m "not extensive" iron/operators/ iron/tests/, whatsmall.ymlruns, gives 1470 passed, 203 skipped.test_ci_results,test_timing_gate,test_build_pages,test_merge_results): 23 pass, each run as CI runs the scripts.mkdocs build --strictpasses.cibranch and checked it in headless Chromium, in light and dark mode, with and without data. There were no JavaScript errors.mlir_aie==1.4.5.dev119+g064a172, the nightly on v1.4.5-alpha.2's commit (the release's own build was cancelled; its wheels are the same code), withllvm-aie==22.0.0.2026102001+4f5df13d. Against it pyright reports 0 errors overiron/(93 against the old pin), and every pre-commit hook passes.