[Perf] Regressions from inliner policy change #114996

Activity

1. performanceautofiler

   added

   arch-arm64

   os-linuxLinux OS (any supported distro)Linux OS (any supported distro)

   runtime-coreclrspecific to the CoreCLR runtimespecific to the CoreCLR runtime

   untriagedNew issue has not been triaged by the area ownerNew issue has not been triaged by the area owner

   on Apr 24, 2025
2. performanceautofiler commented on Apr 24, 2025

   

   performanceautofilerBot

   on Apr 24, 2025 – with PerformanceAutoFiler

   Author

   More actions

   Run Information

   Name	Value
   Architecture	arm64
   OS	ubuntu 22.04
   Queue	AmpereUbuntu
   Baseline	27d23218a56d484638692d56b3abd27e0b21b591
   Compare	d9c4c3e73dcf09435a3cc1cabb23584c9f24b504
   Diff	Diff
   Configs	CompilationMode:tiered, RunKind:micro

   Regressions in System.Collections.Sort<BigStruct>

   Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
   LinqOrderByExtension - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	20.39 μs	22.32 μs	1.09	0.11	False
   LinqQuery - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	20.54 μs	21.97 μs	1.07	0.06	False

   
   
   Test Report

   Repro

   General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

   git clone https://github.com/dotnet/performance.git
   python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Sort<BigStruct>*'

   Details

   System.Collections.Sort<BigStruct>.LinqOrderByExtension(Size: 512)

   ETL Files

   Histogram

   JIT Disasms

   System.Collections.Sort<BigStruct>.LinqQuery(Size: 512)

   ETL Files

   Histogram

   JIT Disasms

   Docs

   Profiling workflow for dotnet/runtime repository
   Benchmarking workflow for dotnet/runtime repository

   ---

   Run Information

   Name	Value
   Architecture	arm64
   OS	ubuntu 22.04
   Queue	AmpereUbuntu
   Baseline	27d23218a56d484638692d56b3abd27e0b21b591
   Compare	d9c4c3e73dcf09435a3cc1cabb23584c9f24b504
   Diff	Diff
   Configs	CompilationMode:tiered, RunKind:micro

   Regressions in System.Text.Json.Serialization.Tests.WriteJson<ImmutableSortedDictionary<String, String>>

   Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
   SerializeToUtf8Bytes - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	9.13 μs	10.34 μs	1.13	0.29	False
   SerializeToString - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	10.27 μs	12.48 μs	1.21	0.28	False

   
   
   Test Report

   Repro

   General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

   git clone https://github.com/dotnet/performance.git
   python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson<ImmutableSortedDictionary<String, String>>*'

   Details

   System.Text.Json.Serialization.Tests.WriteJson<ImmutableSortedDictionary<String, String>>.SerializeToUtf8Bytes(Mode: SourceGen)

   ETL Files

   Histogram

   JIT Disasms

   System.Text.Json.Serialization.Tests.WriteJson<ImmutableSortedDictionary<String, String>>.SerializeToString(Mode: SourceGen)

   ETL Files

   Histogram

   JIT Disasms

   Docs

   Profiling workflow for dotnet/runtime repository
   Benchmarking workflow for dotnet/runtime repository
3. AndyAyersMS commented on Apr 24, 2025

   

   AndyAyersMS

   on Apr 24, 2025

   Member

   More actions

   #114821
4. AndyAyersMS

   removed

   untriagedNew issue has not been triaged by the area ownerNew issue has not been triaged by the area owner

   on Apr 24, 2025
5. AndyAyersMS

   transferred this issue fromdotnet/perf-autofiling-issueson Apr 24, 2025
6. ghost added

   needs-area-labelAn area label is needed to ensure this gets routed to the appropriate area ownersAn area label is needed to ensure this gets routed to the appropriate area owners

   on Apr 24, 2025
7. dotnet-policy-service

   added

   untriagedNew issue has not been triaged by the area ownerNew issue has not been triaged by the area owner

   on Apr 24, 2025
8. AndyAyersMS

   changed the title [-][Perf] Linux/arm64: 32 Regressions on 4/21/2025 2:03:19 AM +00:00[/-] [+][Perf] Regressions from inliner policy change[/+] on Apr 24, 2025
9. AndyAyersMS commented on Apr 24, 2025

   

   AndyAyersMS

   on Apr 24, 2025

   Last edited by DrewScoggins

   Member

   More actions

   Related regressions:

   • [Perf] Linux/x64: 20 Regressions on 4/21/2025 2:03:19 AM +00:00 perf-autofiling-issues#54110
   • [Perf] Windows/x64: 17 Regressions on 4/21/2025 2:03:19 AM +00:00 perf-autofiling-issues#54088
   • [Perf] Linux/x64: 24 Regressions on 4/21/2025 2:03:19 AM +00:00 perf-autofiling-issues#54061
   • [Perf] Windows/x64: 17 Regressions on 4/21/2025 2:03:19 AM +00:00 perf-autofiling-issues#54082
   • [Perf] Windows/x64: 5 Regressions on 4/19/2025 12:13:01 AM +00:00 perf-autofiling-issues#54081
   • [Perf] Windows/x64: 2 Regressions on 4/21/2025 9:32:06 AM +00:00 perf-autofiling-issues#54089
   • [Perf] Linux/x64: 1 Regression on 4/21/2025 9:32:06 AM +00:00 perf-autofiling-issues#54062
   • [Perf] Linux/x64: 1 Regression on 4/21/2025 9:32:06 AM +00:00 perf-autofiling-issues#54111
   • [Perf] Windows/x64: 1 Regression on 4/18/2025 6:27:32 PM +00:00 perf-autofiling-issues#54086
   • [Perf] Linux/arm64: 2 Regressions on 6/10/2025 1:47:10 PM +00:00 perf-autofiling-issues#57435

   More reports (possibly dups):

   • [Perf] Linux/x64: 25 Regressions on 4/21/2025 2:03:19 AM +00:00 perf-autofiling-issues#54630
   • [Perf] Windows/x64: 27 Regressions on 4/21/2025 2:03:19 AM +00:00 perf-autofiling-issues#54609
   • [Perf] Windows/x64: 30 Regressions on 4/21/2025 2:03:19 AM +00:00 perf-autofiling-issues#54600
   • [Perf] Linux/x64: 22 Regressions on 4/21/2025 2:03:19 AM +00:00 perf-autofiling-issues#54569
   • [Perf] Linux/x64: 5 Regressions on 4/21/2025 9:32:06 AM +00:00 perf-autofiling-issues#54570
   • [Perf] Windows/x64: 1 Regression on 4/21/2025 9:32:06 AM +00:00 perf-autofiling-issues#54610
   • [Perf] Linux/x64: 1 Regression on 4/20/2025 10:05:57 PM +00:00 perf-autofiling-issues#54629
10. jeffschwMSFT

    added

    area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMICLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI

    on Apr 25, 2025
11. dotnet-policy-service commented on Apr 25, 2025

    

    dotnet-policy-serviceBot

    on Apr 25, 2025 – with Dotnet Policy Service

    Contributor

    More actions

    Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
    See info in area-owners.md if you want to be subscribed.
12. JulieLeeMSFT

    assigned

    AndyAyersMS

    on Apr 25, 2025
13. 11 remaining items

    Load more

14. AndyAyersMS commented on Aug 9, 2025

    

    AndyAyersMS

    on Aug 9, 2025

    Last edited by AndyAyersMS

    Member

    More actions

    Actually this seems to be fixed(*) with latest main after all...

    Method	Job	Toolchain	value	Mean	Error	StdDev	Median	Min	Max	Ratio	RatioSD	Allocated	Alloc Ratio
    FormatterDecimal	Job-IHVJAJ	\base-rel\corerun.exe	123456.789	42.51 ns	0.853 ns	0.948 ns	42.26 ns	40.89 ns	44.51 ns	1.00	0.03	-	NA
    FormatterDecimal	Job-YBSOLP	\diff-rel\corerun.exe	123456.789	40.99 ns	0.842 ns	0.901 ns	40.88 ns	39.66 ns	42.77 ns	0.96	0.03	-	NA
    FormatterDecimal	Job-QYFRJD	\main-rel\corerun.exe	123456.789	39.35 ns	1.130 ns	1.301 ns	39.21 ns	37.59 ns	42.54 ns	0.93	0.04	-	NA

    (* the lab is not showing this... I will have to look again)
15. AndyAyersMS commented on Aug 9, 2025

    

    AndyAyersMS

    on Aug 9, 2025

    Member

    More actions

    System.Net.Tests.Perf_WebUtility.Decode_DecodingRequired

    Method	Job	Toolchain	Mean	Error	StdDev	Median	Min	Max	Ratio	RatioSD	Gen0	Allocated	Alloc Ratio
    Decode_DecodingRequired	Job-LJYJDT	\base-rel\corerun.exe	90.25 ns	1.850 ns	1.817 ns	90.58 ns	88.08 ns	94.74 ns	1.00	0.03	0.0344	216 B	1.00
    Decode_DecodingRequired	Job-AJGKZG	\diff-rel\corerun.exe	94.70 ns	1.679 ns	1.311 ns	94.95 ns	92.36 ns	96.28 ns	1.05	0.02	0.0341	216 B	1.00
    Decode_DecodingRequired	Job-LBOSPD	\main-rel\corerun.exe	103.55 ns	1.417 ns	1.256 ns	103.33 ns	101.89 ns	106.55 ns	1.15	0.03	0.0343	216 B	1.00

    Very simple root method. We run out of inline budget after inlining System.Net.WebUtility:UrlDecodeInternal and leave some of its key callees un-inlined.

    We perhaps should make a similar noinline fix here like we did in #118544 (say on the Url.Decode methods).

    👍React with 👍1Reacted by Tanner Gooding
16. AndyAyersMS

    added a commit that references this issue on Aug 11, 2025

    Mark UrlDecodeInternal noinline to work around inliner limitation

    ...

    6cd7713
17. AndyAyersMS commented on Aug 11, 2025

    

    AndyAyersMS

    on Aug 11, 2025

    Last edited by AndyAyersMS

    Member

    More actions

    System.Buffers.Text.Tests.Utf8FormatterTests.FormatterDecimal(value: 123456.789)
    Actually this seems to be fixed() with latest main after all...
    ...
    ( the lab is not showing this... I will have to look again)

    What I see locally is that the app is now bimodal, with a slow time (like the regression) and a fast time (like the above). Now to figure out why...

    In the "slow" case we're able to inline System.Globalization.CultureInfo:GetFormat(System.Type):System.Object:this into the benchmark root, and then subsequent inlines fail with budget issues:

                [INL22 IL=-572662307 TR=000474 06003503] [INLINED: call site: profitable inline DEVIRT] System.Globalization.CultureInfo:GetFormat(System.Type):System.Object:this
      ...
                  [INL00 IL=0023 TR=001460 0600064C] [FAILED: call site: inline exceeds budget] System.Math:DivRem(uint,uint):System.ValueTuple`2[uint,uint]
                  [INL00 IL=0064 TR=001488 0600064C] [FAILED: call site: inline exceeds budget] System.Math:DivRem(uint,uint):System.ValueTuple`2[uint,uint]
                [INL00 IL=0116 TR=001153 06001BC5] [FAILED: call site: inline exceeds budget] System.Number+NumberBuffer:get_DigitsPtr():ptr:this
              [INL00 IL=0055 TR=000698 06001E71] [FAILED: call site: inline exceeds budget] System.Span`1[byte]:.ctor(ptr,int):this
    

    This call to GetFormat comes from System.Globalization.NumberFormatInfo:<GetInstance>g__GetProviderNonNull|58_0 and in the slow case we don't have PGO data for that method; in the fast case we do (and it makes the call site cold).

    g__GetProviderNonNull|58_0 seems to be prejitted as the first event we see is a Tier1+Instr. So seems to (possibly) be the result of it taking a while for that code to make it live, and how much data it's been able to gather before the root method is jitted.

    My local benchmark setup may be perturbing things somewhat as I'm using a checked jit and enabling dumping for the benchmark root, which presumably slows down the compilation of that method some and makes seeing the fast result more likely (for me slow runs are rare). But lab data is flipped the other way, though it does show occasional fast runs (on some platforms, here viper windows):

    

    Assuming this is the problem it's not clear if there a good fix; ideally we'd want to defer rejitting of callers until instrumented callees had had a chance to run for a bit, but we have no way of knowing that. In realistic apps we can hope there is more "distance" in the rejit queue between callers and callees (hot callees should generally trigger rejitting before their callers).

    We might consider upping the budget yet again (like we did in #114191). So let me see if using release bits flips the distribution back to mostly slow/sometimes fast, and then see what the budget needs to be to flip it back.
18. AndyAyersMS commented on Aug 12, 2025

    

    AndyAyersMS

    on Aug 12, 2025

    Member

    More actions

    Seems like just a tiny increase in budget (from 20 -> 21) fixes this. Tempting. Let me see if that also helps fix Decode_DecodingRequired
19. AndyAyersMS

    added 2 commits that reference this issue on Aug 12, 2025

    JIT: slightly increase the inline budget

    ...

    02ab9f1

    JIT: slightly increase the inline budget (#118641)

    ...

    4a5eb9d
20. AndyAyersMS commented on Aug 13, 2025

    

    AndyAyersMS

    on Aug 13, 2025

    Member

    More actions

    After #118641

    

    Addressed most but not all of the regression, and on both of these we are now close release/release (compare to last August).

    Refreshing the full set of initial regressions, we see we're mainly seeing issues on tiger which at this point are lower priority (though it's important to cross-check in ADX what happens elsewhere).

    So plan is to focus on the viper regressions next. First I am going to look at the XML serialization test, it is now fairly unstable but seems worse on average (orange line)

    

    and then maybe at

    

    and

    

    Notes	Recent Score	Orig Score	arm64 ubuntu	tiger ubuntu	tiger windows	viper ubuntu	viper windows	Benchmark
    bimodal	2.77	2.59			2.77 2.59			System.Globalization.Tests.StringEquality.Compare_Same_Upper(Count: 1024, Options: (en-US, OrdinalIgnoreCase))
    bimodal	1.55	1.57					1.55 1.57	MicroBenchmarks.Serializers.Xml_ToStream(IndexViewModel).XmlSerializer_
    	1.45	1.94			1.45 1.94			System.Globalization.Tests.StringSearch.LastIndexOf_Word_NotFound(Options: (en-US, None, False))
    	1.45	1.63			1.45 1.63			System.Globalization.Tests.StringSearch.LastIndexOf_Word_NotFound(Options: (, None, False))
    	1.45	1.31		1.44 1.33				System.IO.Tests.Perf_Path.GetFullPathNoRedundantSegments
    	1.41	1.39		1.41 1.50				System.Buffers.Text.Tests.Utf8ParserTests.TryParseUInt64(value: 18446744073709551615)
    	1.40	1.48		1.61 1.66		1.23 1.25		System.Collections.ContainsTrue(String).Queue(Size: 512)
    	1.39	1.39			1.39 1.39			System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "\p{Sm}", Options: NonBacktracking)
    	1.37	1.73		1.37 1.73				System.Globalization.Tests.StringSearch.LastIndexOf_Word_NotFound(Options: (, IgnoreCase, False))
    	1.32	1.37		1.19 1.45		1.42 1.31		System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Sherlock", Options: NonBacktracking)
    	1.32	1.37				1.32 1.37		System.Buffers.Tests.RentReturnArrayPoolTests(Byte).SingleSerial(RentalSize: 4096, ManipulateArray: True, Async: True, UseSharedPool: True)
    	1.30	1.26			1.30 1.26			System.Net.Tests.Perf_WebUtility.Decode_DecodingRequired
    	1.26	1.12	1.37 1.17				1.17 1.07	System.Buffers.Tests.RentReturnArrayPoolTests(Object).MultipleSerial(RentalSize: 4096, ManipulateArray: False, Async: False, UseSharedPool: False)
    	1.26	1.23		1.26 1.23				System.Buffers.Tests.RentReturnArrayPoolTests(Byte).SingleSerial(RentalSize: 4096, ManipulateArray: False, Async: False, UseSharedPool: True)
    	1.25	1.29				1.22 1.27		System.Text.Json.Tests.Perf_Get.GetDecimal
    	1.22	1.12	1.03 1.12	1.45 1.12				System.Buffers.Tests.RentReturnArrayPoolTests(Byte).SingleParallel(RentalSize: 4096, ManipulateArray: False, Async: False, UseSharedPool: False)
    	1.21	1.12	1.35 1.21	1.09 1.08		1.21 1.10	1.20 1.08	System.Buffers.Tests.RentReturnArrayPoolTests(Byte).MultipleSerial(RentalSize: 4096, ManipulateArray: False, Async: False, UseSharedPool: False)
    	1.21	1.13	1.21 1.13					System.Buffers.Tests.RentReturnArrayPoolTests(Byte).SingleParallel(RentalSize: 4096, ManipulateArray: True, Async: False, UseSharedPool: False)
    	1.20	1.25		1.10 1.12		1.31 1.25		System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Sherlock", Options: None)
    	1.19	1.34		1.05 1.35		1.35 1.36		System.Collections.ContainsFalse(String).Queue(Size: 512)
    	1.19	1.25				1.19 1.25		System.Buffers.Text.Tests.Utf8FormatterTests.FormatterDecimal(value: 123456.789)
    	1.18	1.15		1.26 1.17		1.06 1.09		System.Text.RegularExpressions.Tests.Perf_Regex_Industry_SliceSlice.Count(Options: IgnoreCase, NonBacktracking)
    	1.17	1.18					1.16 1.18	System.Buffers.Tests.RentReturnArrayPoolTests(Byte).SingleParallel(RentalSize: 4096, ManipulateArray: True, Async: True, UseSharedPool: True)
    	1.17	1.33	0.99 1.09	1.38 1.63				System.Globalization.Tests.StringSearch.LastIndexOf_Word_NotFound(Options: (en-US, IgnoreCase, False))
    	1.16	1.15				1.16 1.17		System.Tests.Perf_Decimal.ToString(value: 123456.789)
    	1.15	1.14					1.15 1.13	System.IO.Tests.Perf_Directory.GetCurrentDirectory
    	1.15	1.10		1.15 1.10				System.Buffers.Tests.RentReturnArrayPoolTests(Object).SingleParallel(RentalSize: 4096, ManipulateArray: True, Async: True, UseSharedPool: False)
    	1.15	1.13	1.05 1.05	1.09 1.18		1.26 1.09		System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Sherlock Holmes", Options: NonBacktracking)
    	1.15	1.17		1.05 1.18		1.26 1.21		System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Sherlock Holmes", Options: None)
    	1.14	1.09				1.14 1.09		System.Net.Sockets.Tests.SocketSendReceivePerfTest.ReceiveAsyncThenSendAsync_Task
    	1.13	1.17			1.13 1.17			System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (en-US, Ordinal, False))
    	1.12	1.18					1.12 1.18	System.Text.Json.Serialization.Tests.WriteJson(ImmutableSortedDictionary(String, String)).SerializeToString(Mode: SourceGen)
    	1.11	1.14		1.11 1.14				System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Holmes", Options: NonBacktracking)
    	1.10	1.14			1.05 1.14	1.13 1.11	1.12 1.16	System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: False, SkipValidation: False, DataSize: 100000)
    	1.10	1.09			1.10 1.09			System.Tests.Perf_Enum.InterpolateIntoSpan_Flags(value: Red, Green)
    	1.10	1.14		1.10 1.19				System.Collections.CreateAddAndRemove(String).Queue(Size: 512)
    	1.09	1.15					1.11 1.18	System.Text.Tests.Perf_Encoding.GetBytes(size: 512, encName: "utf-8")
    	1.09	1.12		1.09 1.12				System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "zqj", Options: NonBacktracking)
    	1.09	1.13	1.08 1.15	1.12 1.06	1.07 1.21	1.13 1.15	1.07 1.14	System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: False, SkipValidation: True, DataSize: 100000)
    	1.09	1.14			1.09 1.14			System.Memory.Span(Char).LastIndexOfValue(Size: 512)
    	1.08	1.12		1.07 1.10	1.12 1.28		1.06 1.08	System.Tests.Perf_Double.ToStringWithFormat(value: -1.7976931348623157E+308, format: "F50")
    	1.08	1.06				1.08 1.06		System.Buffers.Tests.RentReturnArrayPoolTests(Byte).SingleParallel(RentalSize: 4096, ManipulateArray: False, Async: True, UseSharedPool: False)
    	1.08	1.42		1.07 1.44				System.Memory.Span(Byte).LastIndexOfValue(Size: 512)
    	1.07	1.23		1.31 1.31	0.88 1.15			Benchmark.GetChildKeysTests.AddChainedConfigurationEmpty
    	1.07	1.08		1.05 1.07	1.11 1.11		1.06 1.09	System.Tests.Perf_Double.ToStringWithFormat(value: 1.7976931348623157E+308, format: "F50")
    	1.07	1.11	1.05 1.11		1.06 1.18	1.10 1.07	1.07 1.10	System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: False, SkipValidation: True, DataSize: 100000)
    	1.07	1.07					1.07 1.07	System.Threading.Channels.Tests.BoundedChannelPerfTests.PingPong
    	1.07	1.19					1.06 1.18	System.Collections.CtorFromCollection(Int32).ImmutableArray(Size: 512)
    	1.07	1.08			1.07 1.08			System.Text.Tests.Perf_Encoding.GetByteCount(size: 512, encName: "utf-8")
    	1.07	1.11			1.07 1.11			System.Collections.Sort(Int32).LinqOrderByExtension(Size: 512)
    	1.06	1.12					1.06 1.12	System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: False, SkipValidation: False, DataSize: 100000)
    	1.06	1.08	1.03 1.05	1.09 1.12				System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Holmes", Options: None)
    	1.05	1.05			1.05 1.05			System.Perf_Convert.ToBase64String(formattingOptions: InsertLineBreaks)
    	1.05	1.14				1.05 1.17		System.Text.RegularExpressions.Tests.Perf_Regex_Industry_SliceSlice.Count(Options: IgnoreCase)
    	1.04	1.14			1.04 1.14			System.Collections.Sort(Int32).Array_Comparison(Size: 512)
    	1.04	1.19			1.03 1.18			System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (en-US, Ordinal, False))
    	1.04	1.29					1.04 1.30	System.Collections.CtorFromCollection(Int32).List(Size: 512)
    	1.03	1.22			1.03 1.22			System.Collections.Sort(Int32).LinqQuery(Size: 512)
    	1.03	1.11		0.92 1.11		1.14 1.11		System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "(?i)Twain", Options: None)
    	1.01	1.07					1.01 1.07	System.Threading.Tests.Perf_CancellationToken.CreateLinkedTokenSource3
    	1.01	1.23		1.01 1.26				System.IO.Tests.BinaryWriterExtendedTests.WriteAsciiCharArray(StringLengthInChars: 8000)
    	1.01	1.08	1.01 1.08					System.Text.Json.Tests.Utf8JsonReaderCommentsTests.Utf8JsonReaderCommentParsing(CommentHandling: Skip, SegmentSize: 0, TestCase: LongMultiLine)
    	1.00	1.45	1.00 1.25		1.00 1.69			System.Globalization.Tests.StringEquality.Compare_Same(Count: 1024, Options: (en-US, OrdinalIgnoreCase))
    	1.00	1.22		1.00 1.23				System.IO.Tests.BinaryWriterExtendedTests.WriteAsciiString(StringLengthInChars: 8000)
    	1.00	1.09	1.00 1.09					System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (en-US, IgnoreCase, False))
    	0.99	1.10	0.99 1.10					System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (, IgnoreCase, False))
    	0.98	1.07					0.98 1.07	System.Tests.Perf_Type.GetType_FullyQualifiedNames(input: typeof(System.Collections.Generic.Dictionary`2[]))
    	0.98	1.06		0.89 1.06		1.06 1.06		System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "(?i)Twain", Options: NonBacktracking)
    	0.98	2.71				0.98 2.71		System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value: 4)
    	0.98	1.12	0.98 1.12	0.92 1.07	1.03 1.13	0.99 1.14	0.97 1.20	System.Text.Json.Document.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: True, TestCase: DeepTree)
    	0.97	1.08					0.97 1.08	System.Tests.Perf_Type.GetType_FullyQualifiedNames(input: typeof(System.Tests.NestedGeneric(String, Boolean)))
    	0.97	1.28			1.01 1.21		0.96 1.38	System.Text.Json.Document.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: True, TestCase: Json4KB)
    	0.97	1.10	0.99 1.11		0.96 1.10			System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: False, SkipValidation: True, DataSize: 10)
    	0.96	1.08	0.98 1.09		0.94 1.07			System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: False, SkipValidation: False, DataSize: 10)
    	0.96	1.05			0.96 1.05			System.Perf_Convert.ChangeType
    	0.95	1.19		0.94 1.20				System.IO.Tests.Perf_Path.GetFullPathForLegacyLength
    	0.95	1.26				0.95 1.26		System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value: 2147483647)
    	0.94	1.17	0.97 1.22	0.99 1.23	0.98 1.15	0.94 1.20	0.90 1.17	System.Text.Json.Document.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: False, TestCase: Json4KB)
    	0.93	1.36					0.94 1.38	System.Collections.CtorFromCollection(Int32).Queue(Size: 512)
    	0.93	1.13					0.92 1.14	System.Collections.Tests.Perf_BitArray.BitArrayIntArrayCtor(Size: 512)
    	0.93	1.13	0.98 1.19	0.94 1.16	0.91 1.10	0.90 1.12		System.Text.Json.Document.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: False, TestCase: Json400KB)
    	0.92	1.14	0.92 1.14					System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (en-US, IgnoreNonSpace, False))
    	0.92	1.14	0.92 1.14					System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (en-US, None, False))
    	0.92	1.13	0.92 1.13					System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (, None, False))
    	0.92	1.54		1.03 1.34				System.Buffers.Tests.RentReturnArrayPoolTests(Object).SingleParallel(RentalSize: 4096, ManipulateArray: False, Async: False, UseSharedPool: False)
    	0.92	1.26			0.95 1.17		0.89 1.36	System.Text.Json.Document.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: True, TestCase: Json400KB)
    	0.92	1.36					0.93 1.38	System.Collections.CtorFromCollection(Int32).Stack(Size: 512)
    	0.90	1.12	0.90 1.12					System.Tests.Perf_Guid.ParseExactD
    	0.90	1.20	0.90 1.20					System.Memory.ReadOnlySpan.IndexOfString(input: "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAXAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", value: "x", comparisonType: InvariantCultureIgno
    	0.90	1.08					0.89 1.08	System.Reflection.Activator(EmptyClass).CreateInstanceNames
    	0.90	1.08	0.90 1.08					System.Tests.Perf_Guid.ctor_str
    	0.90	1.12					0.90 1.10	System.Reflection.Attributes.IsDefinedMethodBaseMiss
    	0.89	1.12	0.94 1.18		0.84 1.07	0.87 1.08		System.Memory.ReadOnlySpan.IndexOfString(input: "string1", value: "string2", comparisonType: InvariantCulture)
    	0.88	1.18	0.88 1.18					System.Memory.ReadOnlySpan.IndexOfString(input: "AAAAA5AAAA", value: "5", comparisonType: InvariantCulture)
    	0.86	1.39				0.86 1.39		System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value: -2147483648)
    	0.85	1.17	0.88 1.24			0.83 1.11		System.Memory.ReadOnlySpan.IndexOfString(input: "ABCDE", value: "c", comparisonType: InvariantCultureIgnoreCase)
    	0.85	1.33				0.85 1.34		System.Buffers.Text.Tests.Utf8ParserTests.TryParseDecimal(value: 123456.789)
    	0.82	1.12			0.82 1.12			System.Collections.Sort(IntClass).Array_ComparerClass(Size: 512)
    	0.81	1.12	0.92 1.15		0.72 1.10			System.Memory.ReadOnlySpan.IndexOfString(input: "foobardzsdzs", value: "rddzs", comparisonType: InvariantCulture)
    	0.78	1.10			0.79 1.11			System.Collections.Sort(IntClass).Array_Comparison(Size: 512)
    	0.69	1.11			0.69 1.11			System.Buffers.Text.Tests.Utf8ParserTests.TryParseUInt32Hex(value: FFFFFFFFFFFFFFFF)
    	0.61	1.07			0.58 1.06		0.63 1.09	System.Collections.Sort(IntClass).LinqOrderByExtension(Size: 512)
    	0.57	1.27		0.57 1.27				System.Buffers.Text.Tests.Utf8ParserTests.TryParseUInt64(value: 12345)
21. AndyAyersMS commented on Aug 15, 2025

    

    AndyAyersMS

    on Aug 15, 2025

    Member

    More actions

    Not going to have enough time to chase all these down in .NET 10, so moving to .NET 11. Will keep looking and see if there are any simple fixes we might still consider taking.
22. AndyAyersMS

    modified the milestones: 10.0.0, 11.0.0 on Aug 15, 2025
23. AndyAyersMS commented on Aug 19, 2025

    

    AndyAyersMS

    on Aug 19, 2025

    Member

    More actions

    Array pool benchmarks showed improvements on Viper (dotnet/perf-autofiling-issues#60843) with the latest PGO update.
24. AndyAyersMS commented on Jun 7, 2026

    

    AndyAyersMS

    on Jun 7, 2026

    Member

    More actions

    Note

    Status update prepared with assistance from GitHub Copilot CLI.

    Status update

    Coming back to this in .NET 11:

    1. The original PR's code change is no longer in main. The dynamicPgo NOINLINE-suppression block added in JIT: don't mark callees noinline for non-fatal observations with pgo #114821 was silently removed as part of Return false in PropagateNeverToRuntime for BBJ_THROW blocks #118280 (Aug 2025 — the PR title only mentions a BBJ_THROW change in PropagateNeverToRuntime, but the diff also dropped the entire suppression hunk in InlineResult::Report).

    2. Re-adding the JIT: don't mark callees noinline for non-fatal observations with pgo #114821 suppression on top of current main does not change perf on the worst remaining entry from the original list. Verified locally on a Zen3 box: HEAD vs HEAD + the JIT: don't mark callees noinline for non-fatal observations with pgo #114821 suppression restored, on all 8 (method × Formatted × SkipValidation) combos of Perf_Basic.WriteBasic* at DataSize=100000, produced deltas within ±2% (noise). So whatever mechanism originally introduced these regressions is no longer the current cause for any benchmark that's still showing one.

    3. The follow-up fixes (JIT: allow inliner to use up more of the lva "budget" #118515 / Mark the vectorized CRC update methods noinline #118544 / JIT: slightly increase the inline budget #118641) plus the Aug-2025 PGO data refresh ([Perf] Windows/x64: 19 Improvements on 8/15/2025 5:36:08 AM +00:00 perf-autofiling-issues#60843, now closed) account for the originally-listed benchmarks that have since recovered. The benchmarks in the original table that still show >5% delta against the pre-27d23218 baseline are now likely independent regressions, not symptoms of JIT: don't mark callees noinline for non-fatal observations with pgo #114821.

    4. Remaining items are mostly tracked by existing autofiling issues:

       • The post-Apr-2026 step on Perf_Basic.WriteBasicUtf8(Formatted: False, SkipValidation: True, DataSize: 100000) on viper-windows (~+12% on top of the older delta, between 0bda60e7 and 6130a50f) is filed as [Perf] Windows/x64: 8 Regressions on 4/27/2026 2:35:20 PM +00:00 perf-autofiling-issues#72462.
       • The remaining RentReturn* viper variants and the tiger-* string/regex entries from the original table don't appear to share a root cause with JIT: don't mark callees noinline for non-fatal observations with pgo #114821 and are largely covered by their own per-cluster autofiling issues.

    Given the above, I think this can be closed and any further work tracked in the per-benchmark autofiling issues.

    👍React with 👍1Reacted by Julie Lee
25. AndyAyersMS

    closed this as completedon Jun 7, 2026
26. github-actions

    locked and limited conversation to collaborators on Jul 8, 2026