Repository navigation
Conversation
Add a `bench` pytest marker and record it in the results CSV. Reports and charts cover the marked parametrizations. Every operator carries at least one marker, on the largest input it runs. The choice comes from five months of results on the `ci` branch: an operator whose every parametrization finishes near the dispatch overhead gets the least noisy one. build_pages.py renders the marked results as one chart per operator and publish-pages.yml deploys them. The repository needs Pages enabled with "GitHub Actions" as its source. pretty_trends.py reports the operators a run adds or drops, and the benched metrics that move by 5% or more. A percentage gate alone reports mostly noise. Over one window of the results branch, 633 of 1179 consecutive latency pairs crossed 5%, and 17 of those also crossed the spread the runs measured. A change has to clear both gates. pr_comment.py assembles these reports into the pull request comment, which drops the pass/fail tables. merge_all.py deletes results older than a year. Its per-test limit grouped rows by parametrization name alone. Operators share those names, so their histories overwrote each other. The key carries the test path.
The `github-pages` environment admits `devel`. A run that starts from `main` reaches the deploy step and GitHub rejects it, which leaves a failed run behind every push to `main`.
One chart per architecture, sharing the metric selector. A krackan latency and a phoenix latency differ by enough that one pair of axes hides the smaller one. Each parametrization holds its colour across the two charts. The x axis carries the commit SHA of each run, and the tooltip carries its date. The y axis starts at zero.
Contributor
CI Test Results72976ed (2026_09_28_22_28_15) IRON - CI SummaryExamplesiron/applications/llama_3.2_1b
Smalliron/operators/axpy
iron/operators/dequant
iron/operators/elementwise_add
iron/operators/elementwise_mul
iron/operators/flm/dequant
iron/operators/flm/gemm
iron/operators/gelu
iron/operators/gemm
iron/operators/gemv
iron/operators/layer_norm
iron/operators/leaky_relu
iron/operators/mem_copy
iron/operators/mha
iron/operators/relu
iron/operators/repeat
iron/operators/rms_norm
iron/operators/rope
iron/operators/sigmoid
iron/operators/silu
iron/operators/softmax
iron/operators/strided_copy
iron/operators/swiglu_decode
iron/operators/swiglu_prefill
iron/operators/tanh
iron/operators/transpose
Krackan - SmallIRONTested on iron/operators/axpy
iron/operators/dequant
iron/operators/elementwise_add
iron/operators/elementwise_mul
iron/operators/flm/dequant
iron/operators/flm/gemm
iron/operators/gelu
iron/operators/gemm
iron/operators/gemv
iron/operators/layer_norm
iron/operators/leaky_relu
iron/operators/mem_copy
iron/operators/mha
iron/operators/relu
iron/operators/repeat
iron/operators/rms_norm
iron/operators/rope
iron/operators/sigmoid
iron/operators/silu
iron/operators/softmax
iron/operators/strided_copy
iron/operators/swiglu_decode
iron/operators/swiglu_prefill
iron/operators/tanh
iron/operators/transpose
Trends: Performance trendsOperators dropped: Krackan - ExamplesIRONTested on iron/applications/llama_3.2_1b
Trends: Performance trendsNo benchmark moved by 5% and by more than 2x the spread the runs measured. Phoenix - SmallIRONTested on iron/operators/axpy
iron/operators/dequant
iron/operators/elementwise_add
iron/operators/elementwise_mul
iron/operators/flm/dequant
iron/operators/flm/gemm
iron/operators/gelu
iron/operators/gemm
iron/operators/gemv
iron/operators/layer_norm
iron/operators/leaky_relu
iron/operators/mem_copy
iron/operators/mha
iron/operators/relu
iron/operators/repeat
iron/operators/rms_norm
iron/operators/rope
iron/operators/sigmoid
iron/operators/silu
iron/operators/softmax
iron/operators/strided_copy
iron/operators/swiglu_decode
iron/operators/swiglu_prefill
iron/operators/tanh
iron/operators/transpose
Trends: Performance trendsNo benchmark moved by 5% and by more than 2x the spread the runs measured. Phoenix - ExamplesIRONTested on Trends: Performance trendsNo results. |
andrej
commented
Sep 28, 2026
| """Build the benchmark history site from the CSVs on the results branch. | ||
|
|
||
| Reads every ``{arch}/{suite}/all.csv`` under a results root and writes a single | ||
| self-contained ``index.html``: one chart per operator, one line per |
Collaborator
Author
There was a problem hiding this comment.
it's no longer one chart -- it's one per architecture (phoenix, krackan).
Comment on lines
+19
to
+24
| SUITES = [ | ||
| ("krackan/small", "Krackan - Operators"), | ||
| ("krackan/examples", "Krackan - Applications"), | ||
| ("phoenix/small", "Phoenix - Operators"), | ||
| ("phoenix/examples", "Phoenix - Applications"), | ||
| ] |
Collaborator
Author
There was a problem hiding this comment.
move out into a common file
Comment on lines
+29
to
+30
| SUITES = ["small", "extensive", "examples"] | ||
| ARCHS = ["krackan", "phoenix"] |
Collaborator
Author
There was a problem hiding this comment.
move into a common file
Comment on lines
+79
to
+80
| if args.pages_url: | ||
| lines += [f"[Full benchmark history]({args.pages_url})", ""] |
Collaborator
Author
There was a problem hiding this comment.
I don't think the link is super relevant since the results off of a branch (for which this PR comment exists) won't show up (right?) on that page.
Can we somehow get the results for branches published on a github page like .html that gets deleted after the branch gets merged so we could actually link something relevant here?
If not, I think we should remove the link.
A pull request from a fork controls its own jobs, so it controls every byte of the artifact they upload. The commenting job holds a write token and read the commit, the pull request number and the date out of that artifact, then interpolated them into a shell command and into JavaScript source. A value of `"; command; #` ran. The commit and the pull request number now come from the API. The date comes from the workflow run. The job extracts one member per artifact, `latest.csv`, passes its arguments through the environment, and renders the report with the code in its own checkout against `all.csv` on the results branch. Every test name it prints passes through a function that cuts it to length and removes the characters that end a table cell. A suite that reported nothing is now named. A crashed suite read the same as a suite that moved no benchmark.
The extensive workflow runs the default suite's tests too, so one commit carries two measurements of every benched parametrization. Grouping by architecture alone drew both on one axis under one commit label: 28 of the Krackan identifiers and 8 of the Phoenix ones appear in both suites. Each suite gets its own chart, with its own runs along the x axis.
A dispatch costs about 160 us. Seventeen of the marked shapes ran in under 250 us, so their 2-sigma spread reached 25% to 48% and no trend survived it. Each operator now benchmarks a shape sized for its own limits: 2**23 elements for the elementwise operators, 4096x512 for rope and softmax, 8192x512 for transpose, 4Mi elements for strided_copy. flm/gemm keeps only its milliseconds-long shapes. Every benched shape measured on NPU2 now runs for at least 491 us. GEMV printed "Latency: N us" where its pattern reads "Latency (us): N", so all 216 of its rows carried an empty latency and its charts showed bandwidth and throughput alone. It prints what the other 24 test modules print. A pattern that matches nothing leaves a column empty, and an empty column disappears from a chart without failing anything. The session now names any benched metric that a passing test declared and never printed.
Name one architecture and suite list in pretty_common, which build_pages.py and pr_comment.py both read. The site's docstring now says one chart per architecture and suite. Drop the benchmark history link from the pull request comment. The site renders the results branch, which only a push to a trunk branch writes, so a pull request's own results never reach it. Benchmark every llama parametrization.
andrej
marked this pull request as ready for review
September 29, 2026 02:18
hunhoffe
added a commit
that referenced
this pull request
Oct 1, 2026
Brings in #220's remaining content (e93ecac), #223's bench marker and CI pages/trend scripts, and #224's flm GEMM k tile. iron-next's operator layout wins wherever devel edited files it had removed or collapsed: - conftest.py keeps iron-next's collection and adds devel's Bench column and the report of benched tests that passed without a metric. - flm/gemm takes k_tile as a param(array=True) on the collapsed op.py; it reaches the configuration and kernel-object names, -DMM_FUSED_TILE_K, the L2 tiles, the A taps and pack_b. Shipped pins it at 512. The bench shapes select on the device's architecture, since iron-next's get_params has no dev_name; test_gemm_k_tile runs K=256 and K=768 at k_tile=256. - The per-operator test.py files, llama_3.2_1b and the old iron/common modules devel touched stay deleted; relative_build_dir.py goes with the AIEContext it tests. - requirements.txt, README.md and AGENTS.md keep iron-next's side; AGENTS.md takes devel's CI and benchmark sections. Co-Authored-By: Claude <noreply@anthropic.com>
hunhoffe
added a commit
that referenced
this pull request
Oct 1, 2026
devel's #223 tracks the parametrizations marked bench and names each by its test directory. Here the operators share one test, catalog.py, so CI read them all as one "catalog" operator. operator_name now takes a catalog row's operator from its parametrization id, and a module under iron/lm or a flat iron/operators file by its path. Case gains bench=, which catalog.py turns into pytest.mark.bench. Sweep adds one default-suite case of BENCH_ELEMENTS (2^23) at the widest grid and most channels, devel's elementwise choice. The rest take devel's shapes: GEMM 2048^3, GEMV at every column with M*K >= 2048*8192, MHA at 16384, RoPE 4096x512, Softmax 4096x512, Transpose 8192x512, Copy 4 Mi, Repeat 2048*64, and the Llama shapes of GQAContext and Sample. Emit finishes near the dispatch cost at any size, so it is tracked on one case, as devel does for such operators. Every Llama generation test is benched. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Inspired by Xilinx/mlir-aie#3698, this adds a visualization to the CI results that we generate. Without a visualization, I wasn't really looking at the numbers, so we regressed on llama with my previous commit. This should help that not happening again. In addition, the action now should no longer answer to PRs with the long table of benchmark results (that I rarely looked at), but instead only flags performance results for those operators that were either added or dropped, or any operator whose change in performance is 5%+ (and the mean moves more than two standard deviations, to avoid noise).
Previous results also included highly-noisy small tests that were primarily dominated by dispatch overhead. This adds an new pytest
benchmarker and annotates only sufficiently large test parametrizations with it, so the results aren't polluted with meaningless small measurements.Lastly, drop results from
all.csvthat are older than a year. A follow-on, if we start to have a huge repository because of benchmark results, might be to squash and force-rewrite the history of thecibranch if it ever becomes needed, as suggested by @hunhoffe.Tested locally but will probably need some online testing on
develso expect a follow-on PR (or multiple).