You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Benchmark Regressions from Profile Maintenance and Block Layout Changes #113108
Part of #107749. I've collated all the regression reports assigned to me since #103450 was merged. For now, I'm excluding reports on Tiger machine configs to skip JCC errata-related regressions.
Starting from the top, the System.Memory.Span.Reverse regressions are quite large, and were likely introduced by #111736. The only interesting change in that PR is we no longer mark blocks unreachable via normal flow as rarely run in optSetBlockWeights. As far as I can tell, this only makes a difference for handler regions.
Currently, profile synthesis will set the weight of handler region entries to something close to zero, but it won't mark them as rarely run. We probably don't want to change this, because marking them as rarely run would block inlining within handlers, which matters for finally regions. If we keep handler regions from being marked as rarely run after optSetBlockWeights, these blocks will be considered by a handful of later phases, in particular CSE. Considering CSEs in handler blocks is probably a poor use of our CSE budget, and we already treat handler regions as cold elsewhere, but #111736 had a decent number of improvements too, so I'm hesitant to reintroduce the previous behavior.
I wasn't able to repro the regression with EgorBot, but I'll take a look locally.
I wasn't able to repro the regression with EgorBot, but I'll take a look locally.
I'm able to repro the regression on my Windows AMD machine, but the difference is quite inconsistent. From run to run, the difference is either just a nanosecond, or over 2x. For example:
I'm failing to find any codegen diffs along the call stack of this benchmark, and the potential fix would likely have its own slew of regressions, so the return on pursuing these regressions likely aren't as high as others that are directly related to layout churn. I'll keep this one in mind for later...
PerfLabTests.CastingPerf2.CastingPerf.ScalarValueTypeObj regressed and then later improved due to flowgraph/profile churn. Its current regressed status is from #112060, and is tracked in #112657. This is also true for System.Linq.Tests.Perf_Enumerable.SingleWithPredicate_FirstElementMatches(input: Array). I'll remove these from the above chart.
System.Collections.Tests.Perf_BitArray.BitArrayNot(Size: 512) improved on both platforms once late profile repair was enabled. On Viper Windows, it regressed again around 2/27/2025. I can't find an issue tracking this regression, but nothing in the commit range is flowgraph-related.
There are a handful of PRs that regressed from the initial 3-opt or 4-opt PRs, and didn't budge all that much since. These likely expose the limitations of our cost model's ability to reflect actual performance.
Unsurprisingly, several regressions are attributed to changes in profile data. These could be regressions in block layout as well, and/or regressions elsewhere.
I looked at a few of the regressions from #103450, and for these cases, 3-opt is keen to make bottom-tested loops top-tested. For example, here's the initial layout for System.SpanHelpers:IndexOfAny, which is the hot call in System.Memory.Span.IndexOfAnyFourValues and System.Memory.Span.IndexOfAnyFiveValues:
For hot loops, 3-opt finds it massively profitable to create fallthrough along a loop backedge, since the entry edges are considerably lower in weight, but this gives a worse loop shape overall. I think we'd be justified in inhibiting the creation of fallthrough along loop backedges if it would make the loop top-tested. A targeted implementation would be to detect loop backedges in ThreeOptLayout::ConsiderEdge, and if the edge is a conditional jump, we don't consider it. I'd prefer ConsiderEdge continue to only check layout invariants and not introduce its own profitability checks, so we could also try tweaking the cost model to give loop backedges a cost of zero such that we don't try to create fallthrough along them, but I don't know all of the ramifications of that just yet.
Adding a pre-layout pass of fgOptimizeBranch (with some tweaks) seems to improve loop layout for the 4-opt regressions. Consider this snippet from System.Collections.IterateForNonGeneric<T>:ArrayList(Size: 512), before we've run any pre-layout flow opts:
Instead of entering the loop from BB14 to check if it will execute at all, fgOptimizeBranch can clone the condition in BB09 and convert BB14 into a pre-loop test. What fgOptimizeBranch currently misses is the newfound opportunity to compact BB06 and BB09. It's a trivial transformation, but it denies 3-opt the opportunity to align the loop with an unconditional jump at the bottom. This incentivizes 3-opt to keep the loop bottom-tested, and move the exit edge's target up as well. Here's the new layout:
#113491 is in, but it's a bit too early to see churn in the benchmarks. I've recomputed the above regression table with a few modifications to help track down the remaining interesting regressions:
I've also excluded regressions that I previously determined were caused by unrelated changes. For example, the math and Tensor primitive regressions in the original table surfaced from an intrinsic change that coincided with one of my flowgraph changes. These benchmarks are not layout-sensitive and can be ignored for our purposes.
For brevity's sake, I'm only looking at benchmarks that remain regressed by 10% or more from their baseline -- we have a pretty long tail of regressions within the margin of noise. Also, some benchmarks are noisy enough that they dip above and below the 10% threshold from run to run, and cannot be easily attributed to any change of mine; I've excluded these as well.
This yields a much shorter table, consisting of regressions smaller in magnitude than the original triage suggested.
For System.Reflection.Invoke.StaticMethod5_arrayNotCached_int_string_struct_class_bool, I'm not seeing any codegen diffs locally with profile repair disabled. The only interesting behavior I've noticed is sometimes we never make it from Tier1-OSR to Tier1. Perhaps fluctuation in time to Tier1 explains some of the spikiness of this benchmark on Viper Windows.
I looked at the ADX query for this benchmark to see if profile repair regressed it on other platforms, and I didn't see anything interesting. However, the win-amd configuration improved around the time #113491 went in, so I suspect this regression will be closed soon:
As I anticipated, the System.Collections.IterateForNonGeneric<T>.ArrayList(Size: 512) regressions look better now that #113491 is in. I'm going to consider those regressions fixed.
#113491 resolves System.Collections.IterateFor<Int32>.IList(Size: 512) on Viper Windows (bottom), and almost closes the gap on Linux Ampere (top):
In the latter case, it looks like 3-opt is failing to model some architecture-specific behavior, since I don't see similarly stubborn regressions on other platforms. This seems to be a theme for layout regressions: one platform regresses while another improves. Perhaps we could leverage hardware vendors' optimization guides to develop platform-specific cost models, though that's not in the cards for .NET 10.
For what it's worth, Tiger machines improved from #113491 as well:
#111873 destabilized Benchstone.BenchI.IniArray.Test by changing how fgCalledCount is computed. For this particular benchmark, fgCalledCount looks about the same as before, but subtle rounding errors can sometimes change relative block weight computations during CSE. When this rounding matters, we see the benchmark regress, and when it doesn't, we see it return to its baseline.
Either way, the layout for this benchmark looks good:
System.Reflection.Invoke.Field_SetStatic_int also regressed due to the JIT choosing different CSEs after the profile maintenance logic added in #111047 changed the CSE cost computations. Introducing a post-morph profile repair phase would level the playing field a bit by ensuring the CSE phase is working with a consistent profile relatively often; regressions remaining after this can be attributed to the mentioned flaws in CSE's cost model.
This leaves about 20 benchmarks that meaningfully regressed either due to 3-opt (#103450, #110277) or late profile repair (#111915), and many of the regressions stemming from the latter are layout-related. After analyzing the most significant regressions, it seems 3-opt successfully optimizes for its cost model often enough that we don't see cases in the perf lab where it clearly misses a globally-optimal solution. However, these remaining regressions show 3-opt's cost model does not always capture the nuances of the target architecture. @AndyAyersMS has proposed some ideas for building a more sophisticated model: Aside from trying to differentiate the costs of different types of branches, we could also try to model how each partition swap changes the lengths of the jumps within each partition. I still don't know of a way to compute this cheaply, but a brute-force solution could be useful for determining if we can gain anything from this.
For .NET 10, I don't plan on significantly changing 3-opt's cost model at this point. Thus, I'm going to start closing tracking issues for layout-related regressions, though I'll keep the profile/CSE ones open for now since I have some current/planned PRs that might help them. Before doing anything, I'll build a table of improvements corresponding to the PRs analyzed above so we can get a sense of what we're gaining from this tradeoff.
#116054 is doing a lot of heavy lifting here (thanks @EgorBo). #116017 is also kicking in, in some cases. Both of these changes surely regressed some of the above benchmarks, but I'll rely on newer issues to track those.
I don't see us changing our block layout strategy at this point; we've since been improving its results by making progress on upstream phases like loop opts, profile maintenance, etc. Since turning on graph-based loop inversion was my last major item planned for flowgraph modernization in .NET 10, I'm going to close this in favor of newer regression reports.
Activity
amanasifkhalid commentedon Mar 4, 2025
1.79
3.98
2.00
1.42
2.68
1.68
2.04
1.81
2.03
1.06
1.20
2.37
1.12
1.47
1.39
1.39
1.31
1.33
1.33
1.33
1.32
1.33
1.32
1.32
1.33
1.26
1.30
1.29
1.27
1.26
1.28
1.18
1.42
1.09
1.60
1.25
1.25
1.56
1.22
1.24
1.25
1.24
1.23
1.23
1.28
1.22
1.22
1.13
1.52
1.20
1.16
1.25
1.19
1.11
1.60
1.21
1.09
1.17
1.19
1.14
1.18
1.16
1.87
1.16
1.08
1.26
1.21
1.16
1.15
1.14
1.14
1.30
1.07
1.11
1.14
1.14
1.07
1.21
1.11
1.17
1.13
1.11
1.14
1.11
1.14
1.20
1.12
1.06
1.21
1.12
1.11
1.11
1.10
1.06
1.36
1.11
1.08
1.09
1.09
1.71
1.35
1.10
1.20
1.07
1.13
1.16
1.14
1.07
1.07
1.15
1.07
1.07
1.09
1.09
1.10
1.05
1.08
1.20
1.17
1.11
1.09
1.07
1.16
1.09
1.07
1.07
1.16
1.11
1.09
1.08
1.09
1.14
1.13
1.12
1.09
1.08
1.13
1.17
1.07
1.17
1.18
1.08
1.09
1.09
1.11
1.09
1.06
1.08
1.08
1.07
1.07
1.11
1.08
1.06
1.07
1.06
1.12
1.07
1.10
1.09
1.11
1.06
1.07
1.29
1.26
1.11
1.07
1.06
1.06
1.06
1.06
1.06
1.08
1.06
1.07
1.05
1.08
1.08
1.32
1.15
1.07
1.06
1.06
1.06
1.06
1.14
1.06
1.06
1.09
1.17
1.06
1.05
1.05
1.05
1.10
1.08
1.09
1.51
1.10
1.09
1.07
1.20
1.63
1.32
1.19
1.05
1.08
1.80
1.07
1.09
1.11
1.09
2.08
1.10
1.09
1.22
1.08
1.20
1.17
1.22
1.22
1.22
1.22
1.22
1.60
1.18
1.14
1.08
1.10
1.09
1.19
1.07
1.48
1.87
1.33
dotnet-policy-service commentedon Mar 4, 2025
Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.
amanasifkhalid commentedon Mar 4, 2025
Starting from the top, the
System.Memory.Span.Reverseregressions are quite large, and were likely introduced by #111736. The only interesting change in that PR is we no longer mark blocks unreachable via normal flow as rarely run inoptSetBlockWeights. As far as I can tell, this only makes a difference for handler regions.Currently, profile synthesis will set the weight of handler region entries to something close to zero, but it won't mark them as rarely run. We probably don't want to change this, because marking them as rarely run would block inlining within handlers, which matters for finally regions. If we keep handler regions from being marked as rarely run after
optSetBlockWeights, these blocks will be considered by a handful of later phases, in particular CSE. Considering CSEs in handler blocks is probably a poor use of our CSE budget, and we already treat handler regions as cold elsewhere, but #111736 had a decent number of improvements too, so I'm hesitant to reintroduce the previous behavior.I wasn't able to repro the regression with EgorBot, but I'll take a look locally.
amanasifkhalid commentedon Mar 4, 2025
I'm able to repro the regression on my Windows AMD machine, but the difference is quite inconsistent. From run to run, the difference is either just a nanosecond, or over 2x. For example:
I'm failing to find any codegen diffs along the call stack of this benchmark, and the potential fix would likely have its own slew of regressions, so the return on pursuing these regressions likely aren't as high as others that are directly related to layout churn. I'll keep this one in mind for later...
amanasifkhalid commentedon Mar 4, 2025
PerfLabTests.CastingPerf2.CastingPerf.ScalarValueTypeObj regressed and then later improved due to flowgraph/profile churn. Its current regressed status is from #112060, and is tracked in #112657. This is also true for System.Linq.Tests.Perf_Enumerable.SingleWithPredicate_FirstElementMatches(input: Array). I'll remove these from the above chart.
amanasifkhalid commentedon Mar 4, 2025
System.Collections.Tests.Perf_BitArray.BitArrayNot(Size: 512) improved on both platforms once late profile repair was enabled. On Viper Windows, it regressed again around 2/27/2025. I can't find an issue tracking this regression, but nothing in the commit range is flowgraph-related.
amanasifkhalid commentedon Mar 5, 2025
I've annotated the most-regressed benchmarks with culprit PRs based on when the last notable regression occurred.
optSetBlockWeights, funclet creation #111736 comes up much more often than I expected it to; I suspect it's pessimizing CSE quite a bit.I looked at a few of the regressions from #103450, and for these cases, 3-opt is keen to make bottom-tested loops top-tested. For example, here's the initial layout for
System.SpanHelpers:IndexOfAny, which is the hot call inSystem.Memory.Span.IndexOfAnyFourValuesandSystem.Memory.Span.IndexOfAnyFiveValues:Both loops are bottom-tested, and we fall out of the inner loop into the outer loop. 3-opt transforms this into the following:
It's the same case for
Span.Sorting.BubbleSortSpan:For hot loops, 3-opt finds it massively profitable to create fallthrough along a loop backedge, since the entry edges are considerably lower in weight, but this gives a worse loop shape overall. I think we'd be justified in inhibiting the creation of fallthrough along loop backedges if it would make the loop top-tested. A targeted implementation would be to detect loop backedges in
ThreeOptLayout::ConsiderEdge, and if the edge is a conditional jump, we don't consider it. I'd preferConsiderEdgecontinue to only check layout invariants and not introduce its own profitability checks, so we could also try tweaking the cost model to give loop backedges a cost of zero such that we don't try to create fallthrough along them, but I don't know all of the ramifications of that just yet.5 remaining items
amanasifkhalid commentedon Mar 13, 2025
Adding a pre-layout pass of
fgOptimizeBranch(with some tweaks) seems to improve loop layout for the 4-opt regressions. Consider this snippet fromSystem.Collections.IterateForNonGeneric<T>:ArrayList(Size: 512), before we've run any pre-layout flow opts:Instead of entering the loop from
BB14to check if it will execute at all,fgOptimizeBranchcan clone the condition inBB09and convertBB14into a pre-loop test. WhatfgOptimizeBranchcurrently misses is the newfound opportunity to compactBB06andBB09. It's a trivial transformation, but it denies 3-opt the opportunity to align the loop with an unconditional jump at the bottom. This incentivizes 3-opt to keep the loop bottom-tested, and move the exit edge's target up as well. Here's the new layout:System.Collections.IterateFor<Int32>.IList(Size: 512)looks similar:JIT: Re-introduce late `fgOptimizeBranch` pass (#113491)
amanasifkhalid commentedon Mar 21, 2025
#113491 is in, but it's a bit too early to see churn in the benchmarks. I've recomputed the above regression table with a few modifications to help track down the remaining interesting regressions:
optSetBlockWeights, funclet creation #111736 since I wasn't able to find any codegen changes for the most impacted benchmarks with that change included/reverted (comment).This yields a much shorter table, consisting of regressions smaller in magnitude than the original triage suggested.
amanasifkhalid commentedon Mar 21, 2025
1.13
1.52
1.29
1.22
1.29
1.25
1.22
1.09
1.17
1.20
1.21
1.09
1.19
1.16
1.25
1.11
1.07
1.07
1.14
1.08
1.26
1.21
1.14
1.11
1.12
1.10
1.13
1.16
1.14
1.07
1.07
amanasifkhalid commentedon Mar 21, 2025
For
System.Reflection.Invoke.StaticMethod5_arrayNotCached_int_string_struct_class_bool, I'm not seeing any codegen diffs locally with profile repair disabled. The only interesting behavior I've noticed is sometimes we never make it from Tier1-OSR to Tier1. Perhaps fluctuation in time to Tier1 explains some of the spikiness of this benchmark on Viper Windows.I looked at the ADX query for this benchmark to see if profile repair regressed it on other platforms, and I didn't see anything interesting. However, the win-amd configuration improved around the time #113491 went in, so I suspect this regression will be closed soon:
amanasifkhalid commentedon Mar 21, 2025
As I anticipated, the
System.Collections.IterateForNonGeneric<T>.ArrayList(Size: 512)regressions look better now that #113491 is in. I'm going to consider those regressions fixed.amanasifkhalid commentedon Mar 21, 2025
#113491 resolves
System.Collections.IterateFor<Int32>.IList(Size: 512)on Viper Windows (bottom), and almost closes the gap on Linux Ampere (top):In the latter case, it looks like 3-opt is failing to model some architecture-specific behavior, since I don't see similarly stubborn regressions on other platforms. This seems to be a theme for layout regressions: one platform regresses while another improves. Perhaps we could leverage hardware vendors' optimization guides to develop platform-specific cost models, though that's not in the cards for .NET 10.
For what it's worth, Tiger machines improved from #113491 as well:
amanasifkhalid commentedon Mar 21, 2025
System.Collections.IterateForEach<Int32>.ImmutableSortedSet(Size: 512)was improved by 3-opt, but enabling 4-opt destabilized it. It recently stabilized at a worse baseline, but the commit range doesn't show anything layout-related. I don't see anything actionable here.amanasifkhalid commentedon Mar 24, 2025
#111873 destabilized
Benchstone.BenchI.IniArray.Testby changing howfgCalledCountis computed. For this particular benchmark,fgCalledCountlooks about the same as before, but subtle rounding errors can sometimes change relative block weight computations during CSE. When this rounding matters, we see the benchmark regress, and when it doesn't, we see it return to its baseline.Either way, the layout for this benchmark looks good:
#112151 may address the CSE regressions; I'll continue this work with that PR.
amanasifkhalid commentedon Mar 24, 2025
System.Reflection.Invoke.Field_SetStatic_intalso regressed due to the JIT choosing different CSEs after the profile maintenance logic added in #111047 changed the CSE cost computations. Introducing a post-morph profile repair phase would level the playing field a bit by ensuring the CSE phase is working with a consistent profile relatively often; regressions remaining after this can be attributed to the mentioned flaws in CSE's cost model.amanasifkhalid commentedon Mar 24, 2025
This leaves about 20 benchmarks that meaningfully regressed either due to 3-opt (#103450, #110277) or late profile repair (#111915), and many of the regressions stemming from the latter are layout-related. After analyzing the most significant regressions, it seems 3-opt successfully optimizes for its cost model often enough that we don't see cases in the perf lab where it clearly misses a globally-optimal solution. However, these remaining regressions show 3-opt's cost model does not always capture the nuances of the target architecture. @AndyAyersMS has proposed some ideas for building a more sophisticated model: Aside from trying to differentiate the costs of different types of branches, we could also try to model how each partition swap changes the lengths of the jumps within each partition. I still don't know of a way to compute this cheaply, but a brute-force solution could be useful for determining if we can gain anything from this.
For .NET 10, I don't plan on significantly changing 3-opt's cost model at this point. Thus, I'm going to start closing tracking issues for layout-related regressions, though I'll keep the profile/CSE ones open for now since I have some current/planned PRs that might help them. Before doing anything, I'll build a table of improvements corresponding to the PRs analyzed above so we can get a sense of what we're gaining from this tradeoff.
amanasifkhalid commentedon Mar 24, 2025
0.90
0.89
0.91
0.92
0.93
0.81
0.87
0.91
0.86
0.64
0.67
0.84
0.93
0.94
0.87
0.89
0.94
0.94
0.91
0.89
0.91
0.87
0.84
0.84
0.92
0.88
0.88
0.89
0.85
0.83
0.94
0.88
0.88
0.89
0.89
0.81
0.90
0.88
0.85
0.89
0.89
0.85
0.89
0.86
0.88
0.84
0.90
0.90
0.89
0.90
0.85
0.88
0.92
0.82
0.85
0.88
0.87
0.86
0.86
0.79
0.89
0.83
0.82
0.87
0.90
0.82
0.83
0.82
0.83
0.80
0.74
0.71
0.80
0.79
0.80
0.86
0.80
0.79
0.77
0.86
0.78
0.60
0.83
0.84
0.60
0.84
0.79
0.83
0.81
0.84
0.80
0.75
0.63
0.84
0.89
0.92
0.73
0.75
0.64
0.82
0.91
0.69
0.75
0.74
0.80
0.61
0.74
0.76
0.75
0.50
0.81
0.67
0.50
0.50
0.70
0.79
0.72
0.68
0.58
0.67
0.61
0.82
0.76
0.56
0.67
0.69
0.70
0.62
0.50
0.50
0.48
0.36
0.26
amanasifkhalid commentedon Jun 5, 2025
Some of the stubborn regressions in this comment have recently improved:
System.Memory.Span<Int32>.IndexOfAnyFourValues(Size: 512):System.Collections.Tests.Add_Remove_SteadyState<String>.Queue(Count: 512):System.Memory.Span<Int32>.IndexOfAnyFiveValues(Size: 512):System.Reflection.Invoke.Field_SetStatic_int:MicroBenchmarks.Serializers.Xml_ToStream(IndexViewModel).XmlSerializer_:System.Collections.IterateForEachNonGeneric<String>.ArrayList(Size: 512)#116054 is doing a lot of heavy lifting here (thanks @EgorBo). #116017 is also kicking in, in some cases. Both of these changes surely regressed some of the above benchmarks, but I'll rely on newer issues to track those.
I don't see us changing our block layout strategy at this point; we've since been improving its results by making progress on upstream phases like loop opts, profile maintenance, etc. Since turning on graph-based loop inversion was my last major item planned for flowgraph modernization in .NET 10, I'm going to close this in favor of newer regression reports.