Repository navigation
[Perf] Windows/x64: BenchStone Regressions on 1/29/2025 7:05:56 PM +00:00 #112136
Copy link
Copy link
Closed
Labels
arch-x64area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMICLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMIos-windowsruntime-coreclrspecific to the CoreCLR runtimespecific to the CoreCLR runtimetenet-performancePerformance related issuePerformance related issuetenet-performance-benchmarksIssue from performance benchmarkIssue from performance benchmark
Milestone
Description
Run Information
| Name | Value |
|---|---|
| Architecture | x64 |
| OS | Windows 10.0.22621 |
| Queue | TigerWindows |
| Baseline | bfa02766bdde4657ddd1f734e274d43e6ba3c036 |
| Compare | f3d90b2f622691253225ddcd61b8ae8a087955ae |
| Diff | Diff |
| Configs | CompilationMode:tiered, RunKind:micro |
Regressions in Benchstone.BenchI.Array2
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio |
|---|---|---|---|---|---|---|---|---|
| 918.94 ms | 1.28 secs | 1.40 | 0.01 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Benchstone.BenchI.Array2*'Details
Benchstone.BenchI.Array2.Test
ETL Files
Histogram
JIT Disasms
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Name | Value |
|---|---|
| Architecture | x64 |
| OS | Windows 10.0.22621 |
| Queue | TigerWindows |
| Baseline | bfa02766bdde4657ddd1f734e274d43e6ba3c036 |
| Compare | f3d90b2f622691253225ddcd61b8ae8a087955ae |
| Diff | Diff |
| Configs | CompilationMode:tiered, RunKind:micro |
Regressions in Benchstone.BenchI.NDhrystone
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio |
|---|---|---|---|---|---|---|---|---|
| 326.78 ms | 361.55 ms | 1.11 | 0.02 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Benchstone.BenchI.NDhrystone*'Details
Benchstone.BenchI.NDhrystone.Test
ETL Files
Histogram
JIT Disasms
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
arch-x64area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMICLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMIos-windowsruntime-coreclrspecific to the CoreCLR runtimespecific to the CoreCLR runtimetenet-performancePerformance related issuePerformance related issuetenet-performance-benchmarksIssue from performance benchmarkIssue from performance benchmark


Activity
[-][Perf] Windows/x64: 2 Regressions on 1/29/2025 7:05:56 PM +00:00[/-][+][Perf] Windows/x64: BenchStone Regressions on 1/29/2025 7:05:56 PM +00:00[/+]LoopedBard3 commentedon Feb 4, 2025
Likely #111873, though only 2 regressions on this configuration. FYI @amanasifkhalid
Related Regressions:
dotnet-policy-service commentedon Feb 4, 2025
Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.
amanasifkhalid commentedon Mar 24, 2025
I took a look at the Benchstone regressions as part of #113108, and the key change in #111873 is that we're calculating
fgCalledCount(the JIT's estimate of how many times a method was called before it tiered up) differently. If the profile data is already consistent (which is the case for the Benchstone benchmarks), the new method shouldn't produce different results, but small floating-point rounding errors can subtly changefgCalledCount, and thus change relative block weight computations. CSE is pretty sensitive to this, and it's a known issue that CSE mixes relative and absolute block weights in its cost computation, so trivial changes to how relative weights are computed can churn CSE quite a bit. I ran into this while working on #112151, and tried to quirk my way around it. I'll keep this issue open for now, and revisit it once #112151 is merged to see if those quirks improve any CSE-related regressions.amanasifkhalid commentedon Jun 25, 2025
2.03
1.12
1.13
1.18
1.32
1.09
1.09
1.08
1.71
1.15
1.35
1.10
1.10
1.40
1.63
1.32
1.19
1.11
1.18
1.60
1.18
1.14
1.66
1.15
1.17
1.14
amanasifkhalid commentedon Jun 25, 2025
I took another look at the Benchstone regressions, and I'm no longer seeing meaningful diffs in Tier1. However, #111936 changed OSR codegen such that
fgOptimizeUncondBranchToSimpleCondkicks in more often, making some loops unnatural and thus pessimizing downstream phases. Here are the metric diffs forBenchstone.BenchI.Array2.Test:In local runs of these benchmarks, I don't always make it out of OSR, so this affects the actual benchmark runs. It might make sense to delay tail duplication (and the rest of the non-simplifying transformations in
fgUpdateFlowGraph), but I'm not keen on opening that can of worms in .NET 10, considering the amount of additional loop cloning it may unlock.amanasifkhalid commentedon Jun 25, 2025
For
System.Collections.ContainsFalse(Int32).ImmutableSortedSet(Size: 512), we were previously underestimatingfgCalledCountby about tenfold. The new calculation means the loop body's relative weight is much lower, such that 3-opt doesn't align it as well as it previously did. We previously had the following layout:We now have this:
Considering the regression is only on Tiger machines, I'm inclined to believe the JCC erratum is partially to blame for this.
amanasifkhalid commentedon Jun 25, 2025
The Linq benchmarks never make it out of OSR on my machine, and the more aggressive tail duplication after #111936 changes (pessimizes?) a few downstream opts:
More tail duplication might mean fewer branches for RBO to consider eliminating, so this decrease doesn't look bad on its own. It's possible the additional CSE wasn't worth it, and is responsible for the (relatively small) regression in these benchmarks.
amanasifkhalid commentedon Jun 25, 2025
The diffs for
System.Collections.IterateFor(String).ImmutableList(Size: 512)stem from profile incorporation computing slightly different edge likelihoods for a few conditions in an inlinee:vs
This change is just enough to move a jump within a hot loop:
vs
I suspect the profile churn is just from execution count imprecision from run to run. As for the block layout changes, it's possible the conditional jump
BB09 -> BB11is incorrectly predicted to be taken if it is a backward jump.With a mainline JIT, the layout returns to the baseline:
I suspect we're getting unlucky from run to run with the edge likelihoods; this would explain the spikiness of this benchmark's graph.
amanasifkhalid commentedon Jun 25, 2025
I couldn't find any meaningful diffs for the
System.Text.Json.Document.Testsbenchmarks. I suspect #111023 could be to blame.amanasifkhalid commentedon Jun 25, 2025
I might try moving tail duplication later in the JIT to reduce its interference with loop finding, but I'm hesitant to introduce more loop-related churn into .NET 10, especially since I have inversion-related regressions to look at still. I think these regressions are small and limited enough in frequency to accept for now.