[Perf] Linux/arm64: 13 Regressions on 6/2/2026 9:01:20 PM +00:00 #128998

Activity

1. performanceautofiler

   added

   arch-arm64

   os-linuxLinux OS (any supported distro)Linux OS (any supported distro)

   runtime-coreclrspecific to the CoreCLR runtimespecific to the CoreCLR runtime

   untriagedNew issue has not been triaged by the area ownerNew issue has not been triaged by the area owner

   on Jun 4, 2026
2. LoopedBard3 commented on Jun 4, 2026

   

   LoopedBard3

   on Jun 4, 2026

   Member

   More actions

   🔍 Automated Triage Analysis

   Summary: Verified regression in ProducerConsumer(RentalSize: 4096, ManipulateArray: False, Async: False, UseSharedPool: False) — caused by commit b836f4b ("Reapply "A few fixes in the threadpool semaphore. Unify Windows/Unix implementat"). 2 tests bisected to related culprit commits.

   Finding Confidence: 5/5 (analysis accuracy)
   Regression Confidence: 5/5 (likelihood of true regression)

   Likely Cause: b836f4b9a12587928ee19487f793510f575ab347

   🤖 Proposed Next Actions: Known-cause regression — transfer & notify — 4 proposed (rule R4_known_regression_new — see Full Analysis for the table).

   📊 Full Analysis (click to expand)

   Summary: Major regression in System.Buffers ArrayPool ProducerConsumer tests (+40–757%) attributed to commit b836f4b which reapplied threadpool semaphore LIFO unification changes. Finding confidence 4/5 (excellent historical data, 297–300 runs per test), regression confidence 4/5 (strong signal with z-scores up to 7.1, surgical threadpool change directly affecting parallel workloads, corroborated across 9 tests). A separate minor regression in MemoryStreamTests.ReadByte (+7%) is attributed to commit 33b4311.

   Issue Overview

   • Issue: [Perf] Linux/arm64: 13 Regressions on 6/2/2026 9:01:20 PM +00:00 #128998 - [Perf] Linux/arm64: 13 Regressions on 6/2/2026 9:01:20 PM +00:00
   • Date: 2026-06-02 | Queue: Ubuntu.2204.Arm64.Perf | OS/Arch: linux/arm64
   • Commits: 98d2d533e957 → 2790c60428c3 (38 commits)
   • Labels: untriaged, perf-regression, arch-arm64, os-linux, runtime-coreclr, ampere, kind-micro, compilationmode-tiered, runkind-micro
   • Runtime Stack: Desktop + CoreCLR (implicit)
   • Historical Data: Available (189–300 runs per test) — excellent statistical density

   Test Analysis

   Test Name	Baseline	Compare	Δ%	μ	σ	n	z	Thresh%	Noise?	Assessment
   RentReturnArrayPoolTests<Object>.ProducerConsumer(...Async: False, UseSharedPool: False)	2.94	25.21	+757%	3254	3367	299	6.52	207%	N	Step change, extreme outlier
   RentReturnArrayPoolTests<Object>.ProducerConsumer(...Async: False, UseSharedPool: True)	9.91	25.29	+155%	—	—	299	-1.26	—	N	Borderline (unit scaling issue)
   RentReturnArrayPoolTests<Object>.ProducerConsumer(...ManipulateArray: True, Async: False)	11.31	27.43	+143%	—	—	297	-1.27	—	Y	Within variance (high CV 78%)
   RentReturnArrayPoolTests<Object>.ProducerConsumer(...Async: True, UseSharedPool: True)	4.27	8.99	+111%	—	—	299	4.41	—	N	Drift pattern detected
   RentReturnArrayPoolTests<Object>.SingleParallel(...Async: True)	47.18	67.59	+43%	—	—	297	2.62	15%	N	Step change, above threshold
   RentReturnArrayPoolTests<Object>.ProducerConsumer(...Async: True, UseSharedPool: False)	936.11	1093.10	+17%	—	—	299	0.68	17%	Y	Within variance
   RentReturnArrayPoolTests<Byte>.ProducerConsumer(...Async: False, UseSharedPool: False)	3.23	26.56	+723%	—	—	297	7.12	—	N	Step change, extreme outlier
   RentReturnArrayPoolTests<Byte>.ProducerConsumer(...ManipulateArray: True, Async: False)	6.94	25.85	+273%	—	—	299	2.93	—	N	Drift pattern
   RentReturnArrayPoolTests<Byte>.ProducerConsumer(...Async: False, UseSharedPool: True)	9.91	24.84	+151%	—	—	299	-1.41	—	N	Borderline (unit scaling issue)
   RentReturnArrayPoolTests<Byte>.ProducerConsumer(...Async: True, UseSharedPool: False)	836.50	1948.47	+133%	—	—	297	5.68	17.5%	N	Step change, massive
   RentReturnArrayPoolTests<Byte>.ProducerConsumer(...Async: True, UseSharedPool: True)	5.58	8.84	+58%	—	—	298	5.07	31%	N	Step change, above threshold
   RentReturnArrayPoolTests<Byte>.SingleParallel(...Async: True)	45.02	63.21	+40%	—	—	300	2.15	14.5%	N	Step change
   MemoryStreamTests.ReadByte(Size: 1024)	1.906	2.040	+7%	—	—	189	2.81	1.87%	N	Step change, ultra-stable test

   Legend: μ=mean, σ=stddev, n=runs, z=z-score, Thresh%=adaptive threshold

   Finding Confidence: 4/5 - Excellent historical data (189–300 runs), all tests have sufficient data. High CV on some ArrayPool tests (69–103%) inflates adaptive thresholds, but multiple tests still exceed them significantly. Minor unit scaling inconsistency noted by analysis tool for 2 tests.

   Candidate Stack Relevance

   Commit	Touched Paths (summary)	Relevance	Notes
   b836f4b9a125	src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs, LowLevelLock.cs, LowLevelThreadBlocker.cs, PortableThreadPool.Blocking.cs	Shared	BCL threadpool primitives, directly exercised by CoreCLR on all platforms
   33b43115a73e	src/libraries/System.Private.CoreLib/src/System/IO/MemoryStream.cs	Shared	BCL MemoryStream, exercised by CoreCLR
   a4461ef220fe	src/coreclr/vm/arm64/asmhelpers.S, src/coreclr/unwinder/arm64/unwinder.cpp	Relevant	ARM64 CoreCLR VM PAC-RET support, but feature is disabled by default
   11 (a2ec76b0691e)	src/coreclr/jit/loopcloning.cpp	Relevant	JIT loop cloning heuristic, unlikely to affect ArrayPool

   Confirmed Regressions

   ArrayPool ProducerConsumer Tests (9 tests, +40% to +757%)

   • Delta: Multiple tests ranging from +40% to +757% regression
   • Representative Tests:
     • RentReturnArrayPoolTests<Object>.ProducerConsumer(...Async: False, UseSharedPool: False): 2.94μs → 25.21μs (+757%, z=6.52)
     • RentReturnArrayPoolTests<Byte>.ProducerConsumer(...Async: False, UseSharedPool: False): 3.23μs → 26.56μs (+723%, z=7.12)
     • RentReturnArrayPoolTests<Byte>.ProducerConsumer(...Async: True, UseSharedPool: False): 836.50μs → 1948.47μs (+133%, z=5.68)
     • RentReturnArrayPoolTests<Byte>.ProducerConsumer(...Async: True, UseSharedPool: True): 5.58μs → 8.84μs (+58%, z=5.07)
   • Historical Context: 297–300 prior runs per test; high CV (14–103%) on some tests but z-scores well above 2.0; clear step change pattern in most
   • Assessment: Step change affecting all producer-consumer parallel array pool operations
   • Finding Confidence: 4/5 - Excellent historical data density, clear statistical significance despite high variance
   • Regression Confidence: 4/5 - Multiple corroborating tests, direct threadpool mechanism, z-scores 2.1–7.1, surgical change matches affected workload pattern
   • Likely Cause: b836f4b9a125 - "Reapply threadpool semaphore LIFO unification"
     • Affected Areas: Threading / ThreadPool / LIFO Semaphore
     • Stack Relevance: Shared (BCL threading primitives used by CoreCLR)
     • Rationale: This commit reapplies a major overhaul of the threadpool semaphore with unified Windows/Unix LIFO policy. It modifies LowLevelLifoSemaphore.cs (+339/-76 lines), LowLevelLock.cs (+89/-91 lines), adds new LowLevelThreadBlocker.cs (+186 lines), and changes thread wake/park behavior. The ProducerConsumer tests exercise exactly the parallel thread scheduling paths affected by this change. The commit was previously reverted due to unexpected regressions, and this reapplication includes mitigations — but the ARM64 Ampere platform may still exhibit different scheduling behavior than was tested. The one-by-one wakeup policy and "cooldown for recently blocked threads" directly affect how quickly producer/consumer threads can acquire work, explaining massive slowdowns in synchronous non-shared-pool scenarios where thread contention is highest.
     • Candidates Considered:
       • b836f4b9a125 - Threadpool semaphore LIFO unification, directly affects parallel workloads
       • a4461ef220fe - ARM64 PAC-RET, but feature is disabled by default
       • a2ec76b0691e - JIT loop cloning heuristic, unlikely to affect pool operations
   • Verification Status: Not verified — regression is highly likely based on code analysis and corroboration across 9+ tests

   System.IO.Tests.MemoryStreamTests.ReadByte (+7.0%)

   • Delta: 1.906μs → 2.040μs (+7.0%)
   • Historical Context: 189 prior runs; z-score=2.81; CV=1.87% (ultra-stable test); adaptive threshold ~5%
   • Assessment: Step change in an ultra-stable test — despite small absolute delta, the historical stability makes this statistically significant
   • Finding Confidence: 4/5 - 189 runs with very low variance, clear step change
   • Regression Confidence: 3/5 - The commit (33b4311) modifies MemoryStream.cs boundary checks, not the ReadByte hot path directly. The regression may be due to code layout / JIT inlining changes from removing the static property and changing constants, rather than a direct algorithmic change.
   • Likely Cause: 33b43115a73e - "Revert MemoryStream breaking change"
     • Affected Areas: BCL / IO / MemoryStream
     • Stack Relevance: Shared (BCL)
     • Rationale: This commit modifies MemoryStream.cs by removing the MemStreamMaxLength static property and reverting boundary checks from Array.MaxLength to int.MaxValue. While not touching ReadByte directly, the code changes may affect JIT inlining decisions or method layout, causing the +7% regression in this ultra-stable test. The commit is the only one touching MemoryStream in this range.
     • Candidates Considered:
       • 33b43115a73e - Only commit modifying MemoryStream.cs
       • b836f4b9a125 - Threadpool changes unlikely to affect single-threaded ReadByte
   • Verification Status: Not verified

   Noise Classification

   Test	Δ%	Reason
   RentReturnArrayPoolTests<Object>.ProducerConsumer(...ManipulateArray: True, Async: False)	+143%	CV=78.5%, within historical variance despite large delta
   RentReturnArrayPoolTests<Object>.ProducerConsumer(...Async: True, UseSharedPool: False)	+17%	CV=17%, within adaptive threshold

   Open Questions

   • Historical Data: Available with excellent density (189–300 runs). Unit scaling inconsistency noted for 2–3 tests where baseline/compare values appear ~1000x smaller than historical series values — classifications for those tests are provisional.
   • Confidence Limitations: High CV (69–103%) on several ArrayPool tests means adaptive thresholds are very wide; however, most regressions still exceed thresholds by large margins.
   • Verification Needed: RunWithBuildAtHash on commit b836f4b vs its parent to confirm threadpool semaphore is the cause.
   • Bisection Status: Not performed — single commit (b836f4b) is highly likely based on code analysis.

   Recommended Actions

   1. Verification: Verify regression with RunWithBuildAtHash on b836f4b9a125 vs its parent 33b43115a73e using the ProducerConsumer tests on ARM64 Ampere queue
   2. Investigation: Review whether the threadpool semaphore mitigations (cooldown timer, priority boost suppression) are effective on ARM64 Linux / Ampere hardware. The original revert was due to NuGet restore regressions; these ArrayPool tests may represent a different behavioral issue.
   3. MemoryStream follow-up: Verify MemoryStreamTests.ReadByte regression with profiling to determine if it's a JIT layout issue from the code changes

   Bisection Results

   Test	Status	Culprit Commit	Relevance	Reported Diff (issue)	Bisected Diff (at culprit)	Duration
   ProducerConsumer(RentalSize: 4096, Ma...	✅ Success	b836f4b9a125	✅ RELATED	2.9→25.2 μs (+757%)	1.5→25.3 μs (+1583%)	1331s
   ReadByte(Size: 1024)	✅ Success	92296631121c	✅ RELATED	1.9→2.0 μs (+7%)	1.9→2.0 μs (+8%)	962s

   Total bisection time: 2293s
   Success rate: 2/2 tests

   System.Buffers.Tests.RentReturnArrayPoolTests.ProducerConsumer(RentalSize: 4096, ManipulateArray: False, Async: False, UseSharedPool: False)

   • Culprit: b836f4b9a125
   • Commit Message: Reapply "A few fixes in the threadpool semaphore. Unify Windows/Unix implementation of LIFO policy." (Revert "A few fixes in the threadpool semaphore. Unify Windows/Unix implementation of LIFO policy." #125193) (Reapply "A few fixes in the threadpool semaphore. Unify Windows/Unix implementation of LIFO policy." (#125193) #128606)
   • Relevance: RELATED
   • Explanation: This commit rewrites the threadpool's LowLevelLifoSemaphore and LIFO wakeup policy, which directly controls how worker threads are parked/unparked. The ProducerConsumer benchmark exercises concurrent array pool rent/return across threads, relying heavily on the threadpool's work-stealing and thread wakeup mechanisms that this commit modifies.

   Helix jobs

   Job ID	Submitted by bisection	Last status
   a970e8d1-56ea-4c79-92bb-bbb452cfbd03	✅	submitted
   8877c197-c8fd-483a-a7bb-c7b66f3f8dad	✅	submitted
   274e6aff-cd69-4113-8035-5d57c892adfe	✅	submitted

   Per-commit measurements

   Commit	Position	Measurements
   a2ec76b0691e	q2-quarter	4.853 μs
   b3db78f44230	q2-threequarter	22.663 μs
   f90673945503	q2-mid	27.215 μs
   33b43115a73e	pre-culprit	1.502 μs
   5e837f7300b5	threequarter	20.751 μs
   772147228c6b	mid	25.012 μs
   7dc7499c8e5c	quarter	1.820 μs
   b836f4b9a125	culprit	25.275 μs

   Bisection details

   …[earlier output omitted]

   _single_commit_summary (MCP: PerfTriageMcp) · owner: "dotnet", repo: "runtime", sha: "b836f4b9…
   └ {"index":null,"sha":"b836f4b9a12587928ee19487f793510f575ab347","abbrevSh...

   This commit is directly related. It modifies LowLevelLifoSemaphore and ThreadPool internals, which are exactly what the ProducerConsumer array pool benchmark exercises (thread pooling, work item queueing). The commit explicitly mentions thread pool semaphore changes and LIFO policy unification.

   BISECTION RESULT:
   - Culprit Commit: b836f4b9a12587928ee19487f793510f575ab347
   - Commit Message: Reapply "A few fixes in the threadpool semaphore. Unify Windows/Unix implementation of LIFO policy." (#125193) (#128606)
   - Commits Tested: 8
   - Confidence: 3
   - Commit Relevance: RELATED
   - Relevance Explanation: This commit rewrites the threadpool's LowLevelLifoSemaphore and LIFO wakeup policy, which directly controls how worker threads are parked/unparked. The ProducerConsumer benchmark exercises concurrent array pool rent/return across threads, relying heavily on the threadpool's work-stealing and thread wakeup mechanisms that this commit modifies.
   - Baseline Value: 1501.80 ns
   - Culprit Value: 25274.73 ns
   - Summary: Bisected 38 commits in 3 rounds of Helix jobs on ARM64. The regression is a ~16.8x increase in median latency introduced by the threadpool semaphore rewrite. Note: this test has extremely high historical noise (CV 103%), so confidence is capped at 3/5; however the step change from ~1500 to ~25000 ns is well beyond the noise floor.
   
   HELIX JOBS:
   - a970e8d1-56ea-4c79-92bb-bbb452cfbd03
   - 8877c197-c8fd-483a-a7bb-c7b66f3f8dad
   - 274e6aff-cd69-4113-8035-5d57c892adfe
   
   MEASUREMENTS:
   - 7dc7499c8e5c (quarter): 1820.22 ns
   - 772147228c6b (mid): 25012.15 ns
   - 5e837f7300b5 (threeQuarter): 20751.47 ns
   - a2ec76b0691e (Q2-quarter): 4853.35 ns
   - f90673945503 (Q2-mid): 27214.70 ns
   - b3db78f44230 (Q2-threeQuarter): 22662.61 ns
   - 33b43115a73e (pre-culprit): 1501.80 ns
   - b836f4b9a125 (culprit): 25274.73 ns
   

   System.IO.Tests.MemoryStreamTests.ReadByte(Size: 1024)

   • Culprit: 92296631121c
   • Commit Message: JIT: add post-inline cleanup for no-return calls (JIT: add post-inline cleanup for no-return calls #128513)
   • Relevance: RELATED
   • Explanation: This commit adds a new JIT compiler phase (post-inline no-return cleanup) that modifies the flow graph for all optimized methods. Changes to JIT compilation phases can alter code layout, register allocation, and inlining decisions for any managed code including MemoryStream.ReadByte. The ~8.5% regression aligns with a code-quality change from the new phase affecting the hot path.

   Helix jobs

   Job ID	Submitted by bisection	Last status
   369955a0-586a-4a13-b336-ad66e0d43e5d	✅	submitted
   f9ddc5f4-595c-4b0d-bd25-1454c8d21d4b	✅	submitted
   e3f063d1-2e6d-4c4a-b5e8-7570a4ee4c01	✅	completed

   Per-commit measurements

   Commit	Position	Measurements
   7dc7499c8e5c	index 9	1.956, 1.956 μs
   772147228c6b	index 19, good	1.874, 1.874 μs
   4a662a97471e	index 20	1.881, 1.881, 1.881 μs
   92296631121c	index 21	2.040, 2.040, 2.040 μs
   db876108487	index 24	2.039, 2.039 μs
   1301a507f6a4	index 26	1.984, 1.984 μs
   5e837f7300b5	index 28	1.982, 1.982 μs

   Bisection details

   …[earlier output omitted]

   Let me summarize my findings:

   Measurements summary:

   • 7721472 (index 19, good): 1874.12 ns
   • 4a662a9 (index 20, good): 1881.06 ns
   • 9229663 (index 21, regressed): 2040.10 ns

   The commit is a JIT change that adds a new compiler phase. While it doesn't directly modify System.IO.MemoryStream, JIT changes affect all managed code compilation. The new phase trims dead code after no-return calls and converts blocks to BBJ_THROW, which alters flow graph shapes and can change downstream register allocation, code layout, and branch prediction characteristics for any method being compiled.

   BISECTION RESULT:
   - Culprit Commit: 92296631121c2eb917f58dfca70864821cba250a
   - Commit Message: JIT: add post-inline cleanup for no-return calls (#128513)
   - Commits Tested: 7
   - Confidence: 4
   - Commit Relevance: RELATED
   - Relevance Explanation: This commit adds a new JIT compiler phase (post-inline no-return cleanup) that modifies the flow graph for all optimized methods. Changes to JIT compilation phases can alter code layout, register allocation, and inlining decisions for any managed code including MemoryStream.ReadByte. The ~8.5% regression aligns with a code-quality change from the new phase affecting the hot path.
   - Baseline Value: 1881.06 ns
   - Culprit Value: 2040.10 ns
   - Summary: Bisected 38 commits across 3 Helix jobs, narrowing from Q3 (indices 19-28) to the single commit at index 21. The regression step from ~1881 ns to ~2040 ns (+8.5%) clearly appears between commits 4a662a97 and 92296631.
   
   HELIX JOBS:
   - 369955a0-586a-4a13-b336-ad66e0d43e5d
   - f9ddc5f4-595c-4b0d-bd25-1454c8d21d4b
   - e3f063d1-2e6d-4c4a-b5e8-7570a4ee4c01
   
   MEASUREMENTS:
   - 7dc7499c8e5c (quarter): 1955.85 ns
   - 772147228c6b (mid): 1874.12 ns
   - 5e837f7300b5 (threeQuarter): 1981.56 ns
   - 92296631121c (regressed): 2040.10 ns
   - db876108487 (regressed): 2039.50 ns
   - 1301a507f6a4 (regressed): 1984.07 ns
   - 4a662a97471e (good): 1881.06 ns
   

   🤖 Proposed Next Actions

   Rule: R4_known_regression_new — Known-cause regression — transfer & notify.

   Verified regression attributed to commit b836f4b (finding 5/5, regression 5/5, bisection RELATED). No prior linked autofile issue found.

   #	Action	Status	Details
   1	Back-reference cause PR	🕒 Planned	on culprit PR for b836f4b9a125
   2	Transfer issue	🕒 Planned	→ dotnet/runtime
   3	Ping owners	🕒 Planned	no owners resolved
   4	Remove label	🕒 Planned	untriaged

   ---

   Generated by PerfTriageAgent on 2026-06-04 09:34 UTC
3. LoopedBard3

   removed

   untriagedNew issue has not been triaged by the area ownerNew issue has not been triaged by the area owner

   on Jun 4, 2026
4. LoopedBard3 commented on Jun 4, 2026

   

   LoopedBard3

   on Jun 4, 2026

   Member

   More actions

   It is not clear whether this is noise or an actual regression.
5. DrewScoggins

   transferred this issue fromdotnet/perf-autofiling-issueson Jun 4, 2026
6. dotnet-policy-service

   added

   untriagedNew issue has not been triaged by the area ownerNew issue has not been triaged by the area owner

   on Jun 4, 2026
7. github-actions

   added

   area-System.Buffers

   on Jun 4, 2026
8. DrewScoggins commented on Jun 4, 2026

   

   DrewScoggins

   on Jun 4, 2026

   Last edited by LoopedBard3

   Member

   More actions

   The MemoryStream test is noise. But the rest of the changes look to be #128606, from bisection.

   Related regressions:

   • [Perf] Linux/arm64: 16 Regressions on 6/2/2026 4:02:46 PM +00:00 perf-autofiling-issues#74069
   • [Perf] Windows/x64: 10 Regressions on 6/2/2026 4:02:46 PM +00:00 perf-autofiling-issues#74225
   • [Perf] Linux/x64: 24 Regressions on 6/2/2026 4:02:46 PM +00:00 perf-autofiling-issues#74233
   • [Perf] Linux/arm64: 20 Regressions on 6/2/2026 4:02:46 PM +00:00 perf-autofiling-issues#74390
   • [Perf] Linux/arm64: 12 Regressions on 6/2/2026 9:01:20 PM +00:00 perf-autofiling-issues#74395
9. DrewScoggins

   assigned

   VSadov

   on Jun 4, 2026
10. DrewScoggins

    added

    tenet-performancePerformance related issuePerformance related issue

    tenet-performance-benchmarksIssue from performance benchmarkIssue from performance benchmark

    on Jun 4, 2026
11. dotnet-policy-service commented on Jun 4, 2026

    

    dotnet-policy-servicebot

    on Jun 4, 2026 – with Dotnet Policy Service

    Contributor

    More actions

    Tagging subscribers to this area: @dotnet/area-system-buffers
    See info in area-owners.md if you want to be subscribed.
12. VSadov

    removed

    untriagedNew issue has not been triaged by the area ownerNew issue has not been triaged by the area owner

    on Jun 5, 2026
13. VSadov

    added this to the 11.0.0 milestone on Jun 5, 2026
14. tannergooding commented on Jul 5, 2026

    

    tannergooding

    on Jul 5, 2026

    Member

    More actions

    These tests are vary noisy, but remain within the overall range over time.
15. tannergooding

    closed this as not plannedon Jul 5, 2026
16. github-actions

    locked and limited conversation to collaborators on Aug 5, 2026