Skip to content

[Perf] Linux/x64: 11 Regressions on 7/8/2026 10:47:18 PM +00:00 #130697

Description

@performanceautofiler

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue ViperUbuntu
Baseline 3ade0b55c797a31f6800040e3ae3c5c75bc0d2dc
Compare e302d2a27aef2196c8d710752da8b7726c5fe84d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Numerics.Tests.Perf_BigInteger

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
94.74 ns 130.77 ns 1.38 0.03 False
68.12 ns 86.19 ns 1.27 0.09 False
48.06 ns 54.91 ns 1.14 0.13 False
70.18 ns 79.21 ns 1.13 0.17 False
70.94 ns 80.04 ns 1.13 0.05 False

graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_BigInteger*'
Details

System.Numerics.Tests.Perf_BigInteger.Parse(numberString: -2147483648)

ETL Files

Histogram

JIT Disasms

System.Numerics.Tests.Perf_BigInteger.Parse(numberString: 123)

ETL Files

Histogram

JIT Disasms

System.Numerics.Tests.Perf_BigInteger.ToStringD(numberString: -2147483648)

ETL Files

Histogram

JIT Disasms

System.Numerics.Tests.Perf_BigInteger.Add(arguments: 1024,1024 bits)

ETL Files

Histogram

JIT Disasms

System.Numerics.Tests.Perf_BigInteger.Subtract(arguments: 1024,1024 bits)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue ViperUbuntu
Baseline 3ade0b55c797a31f6800040e3ae3c5c75bc0d2dc
Compare e302d2a27aef2196c8d710752da8b7726c5fe84d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Single

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
49.71 ns 58.73 ns 1.18 0.03 False
49.66 ns 56.88 ns 1.15 0.03 False
49.09 ns 52.83 ns 1.08 0.02 False

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Single*'
Details

System.Tests.Perf_Single.Parse(value: "-3.4028235E+38")

ETL Files

Histogram

JIT Disasms

System.Tests.Perf_Single.TryParse(value: "-3.4028235E+38")

ETL Files

Histogram

JIT Disasms

System.Tests.Perf_Single.Parse(value: "3.4028235E+38")

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue ViperUbuntu
Baseline 3ade0b55c797a31f6800040e3ae3c5c75bc0d2dc
Compare e302d2a27aef2196c8d710752da8b7726c5fe84d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Double

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
69.19 ns 87.34 ns 1.26 0.06 False
69.50 ns 77.47 ns 1.11 0.02 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Double*'
Details

System.Tests.Perf_Double.TryParse(value: "-1.7976931348623157e+308")

ETL Files

Histogram

JIT Disasms

System.Tests.Perf_Double.Parse(value: "-1.7976931348623157e+308")

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue ViperUbuntu
Baseline 3ade0b55c797a31f6800040e3ae3c5c75bc0d2dc
Compare e302d2a27aef2196c8d710752da8b7726c5fe84d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Net.Tests.Perf_WebUtility

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
99.51 ns 106.28 ns 1.07 0.18 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Net.Tests.Perf_WebUtility*'
Details

System.Net.Tests.Perf_WebUtility.HtmlDecode_Entities

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Activity

  1. removed
    untriagedNew issue has not been triaged by the area owner
    on Jul 14, 2026
  2. added
    untriagedNew issue has not been triaged by the area owner
    os-linuxLinux OS (any supported distro)
    on Jul 14, 2026
  3. added
    untriagedNew issue has not been triaged by the area owner
    runtime-coreclrspecific to the CoreCLR runtime
    on Jul 14, 2026
  4. added
    area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI
    on Jul 14, 2026
  5. dotnet-policy-service commented on Jul 14, 2026

    @dotnet-policy-service
    Contributor

    Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
    See info in area-owners.md if you want to be subscribed.

  6. removed
    untriagedNew issue has not been triaged by the area owner
    on Jul 14, 2026
  7. added this to the 11.0.0 milestone on Jul 14, 2026
  8. LoopedBard3 commented on Jul 14, 2026

    @LoopedBard3
    Member

    🔍 Automated Triage Analysis

    Summary: Verified regression in Parse(numberString: -2147483648) — caused by commit c6e5137 ("JIT: fix GC holes in large stack-target struct block ops (#130352)"). 3 tests bisected to related culprit commits. ⚠️ Note: One or more tests show bimodal history, but at least one clean test bisected to a related culprit.

    Finding Confidence: 5/5 (analysis accuracy)
    Regression Confidence: 5/5 (likelihood of true regression)

    Likely Cause: c6e513735e3d4bbea3e3922c2a101a13d1fd37a9

    🤖 Proposed Next Actions: Known-cause regression — multiple causes — 2 causes: transfer+ping (c6e513735e3d: System.Numerics.Tests.Perf_BigInteger.Parse(numberString: -2147483648)), append #130697 (6580e79bcac1: System.Tests.Perf_Single.Parse(value: "-3.4028235E+38")) (rule R3R4_multi_cause — see Full Analysis for the table). · 🦆 Re-analysis: ⚠️ disputes (overconfident, culprit-mismatch, bimodal-history-concern)

    📊 Full Analysis (click to expand)

    Summary: 10 tests regressed +7–38% across BigInteger, Single, and Double parsing/arithmetic operations, all attributed to commit 6580e79 ("Add new NumberStyles option to stop parsing on invalid character"). Finding confidence 4/5 (excellent historical data, 297+ runs per test, very low CV). Regression confidence 5/5 (surgical change to Number.Parsing.cs with 362 line changes, extreme z-scores up to 38.97, all tests show clean step changes aligned to commit date).

    Issue Overview

    • Issue: Disable frequently failing CriticalFinalizer test #76131 - [Perf] Linux/x64: 11 Regressions on 7/8/2026 10:47:18 PM +00:00
    • Date: 2026-07-08 | Queue: Ubuntu.2204.Amd64.Viper.Perf | OS/Arch: linux/x64
    • Commits: 3ade0b55c797 → e302d2a27aef (20 commits)
    • Labels: untriaged, perf-regression, arch-x64, os-linux, runtime-coreclr, kind-micro, compilationmode-tiered, runkind-micro
    • Runtime Stack: Desktop + CoreCLR (implicit)
    • Historical Data: Available (296–300 runs per test, excellent statistical power)

    Test Analysis

    Test Name Baseline Compare Δ% μ σ n z Thresh% Noise? Assessment
    Perf_BigInteger.Parse(-2147483648) 94.74 130.77 +38.03% 95.74 0.90 297 38.97 5% N Extreme step change, ultra-stable
    Perf_BigInteger.Parse(123) 68.12 86.19 +26.53% 68.17 1.56 299 11.56 5% N Clear step change
    Perf_BigInteger.ToStringD(-2147483648) 48.06 54.91 +14.25% 48.09 1.20 299 5.66 5.01% N Step change, bimodal after
    Perf_BigInteger.Add(1024,1024 bits) 70.18 79.21 +12.87% 71.50 2.45 299 3.15 6.85% N Persistent step change
    Perf_BigInteger.Subtract(1024,1024 bits) 70.94 80.04 +12.83% 71.74 1.42 297 5.87 5% N Clear step change
    Perf_Single.Parse("-3.4028235E+38") 49.71 58.73 +18.15% 49.97 0.60 296 14.70 5% N Extremely high-confidence step change
    Perf_Single.TryParse("-3.4028235E+38") 49.66 56.88 +14.54% 49.90 0.48 297 14.69 5% N Extremely high-confidence step change
    Perf_Single.Parse("3.4028235E+38") 49.09 52.83 +7.62% 49.27 0.63 300 5.64 5% N Clear step change
    Perf_Double.TryParse("-1.797...e+308") 69.19 87.34 +26.23% 69.42 0.62 297 28.80 5% N Extreme step change, ultra-stable
    Perf_Double.Parse("-1.797...e+308") 69.50 77.47 +11.47% 69.97 0.73 298 10.30 5% N Clear step change
    Perf_WebUtility.HtmlDecode_Entities 99.51 106.28 +6.80% 98.51 6.16 297 1.26 12.51% Y Within variance, high CV

    Legend: μ=mean (ns), σ=stddev, n=historical runs, z=z-score, Thresh%=adaptive threshold

    Finding Confidence: 4/5 - Excellent historical data (296–300 runs per test), very low CV (0.9–3.4% for regressed tests), clean step-change boundaries with no pre-existing drift. All tests have sufficient statistical power.

    Candidate Stack Relevance

    Commit Touched Paths (summary) Relevance Notes
    6580e79bcac1 src/libraries/System.Private.CoreLib/src/System/Number.Parsing.cs (+243/-119), Number.Parsing.Common.cs, Double.cs, Byte.cs, Int32.cs, etc. (58 files) Shared BCL number parsing; cross-stack but directly exercises the regressed code paths
    07f0abcdaf13 src/coreclr/jit/assertionprop.cpp, rangecheck.cpp Relevant JIT assertion optimization; unlikely to regress parsing hot paths
    1ab7d0535812 src/coreclr/jit/flowgraph.cpp Relevant JIT async helper types; micro benchmarks are not async
    c6e513735e3d src/coreclr/jit/ (GC holes fix) Relevant JIT bug fix for large structs; unlikely to affect numeric parsing

    Confirmed Regressions

    System.Numerics.Tests.Perf_BigInteger.Parse(numberString: -2147483648) (+38.03%)

    • Delta: 94.74 → 130.77 ns (+38.03%)
    • Historical Context: μ=95.74 (σ=0.90, n=297); z-score=38.97; adaptive threshold=5%
    • Assessment: Extreme step change. Historically ultra-stable (CV=0.94%). All 28 post-change values cluster at 113–131 ns with zero overlap against the prior distribution (94–101 ns).
    • Finding Confidence: 4/5 - Excellent data quality, 297 historical runs, very low variance
    • Regression Confidence: 5/5 - Extreme z-score (38.97), surgical commit match, direct code path modification
    • Likely Cause: 6580e79bcac1 - "Add new NumberStyles option to stop parsing on invalid character"
      • Affected Areas: Number parsing (BCL)
      • Stack Relevance: Shared (BCL surface exercised by CoreCLR)
      • Rationale: This commit modifies Number.Parsing.cs with 243 additions and 119 deletions (362 line changes), adding a new NumberStyles option. The parsing hot path is directly modified. BigInteger.Parse calls into the same Number.Parsing infrastructure. The commit date (2026-07-08T22:10:01) precisely matches the regression date.
      • Candidates Considered:
        • 6580e79bcac1 - Direct modification to Number.Parsing.cs (primary)
        • 07f0abcdaf13 - JIT assertion optimization, unlikely for parsing
    • Verification Status: Not verified

    System.Numerics.Tests.Perf_BigInteger.Parse(numberString: 123) (+26.53%)

    • Delta: 68.12 → 86.19 ns (+26.53%)
    • Historical Context: μ=68.17 (σ=1.56, n=299); z-score=11.56; adaptive threshold=5%
    • Assessment: Clear step change. All 28 after values cluster at 85.5–88.7 ns with zero overlap against prior distribution (66.5–77 ns).
    • Finding Confidence: 4/5 - Excellent data, 299 runs, low CV (2.29%)
    • Regression Confidence: 5/5 - High z-score, same root cause as Parse(-2147483648)
    • Likely Cause: 6580e79bcac1 - Same as above
      • Stack Relevance: Shared
      • Rationale: Same parsing infrastructure modification. Both BigInteger.Parse inputs regress together, confirming a shared code path change.
    • Verification Status: Not verified

    System.Numerics.Tests.Perf_BigInteger.ToStringD(numberString: -2147483648) (+14.25%)

    • Delta: 48.06 → 54.91 ns (+14.25%)
    • Historical Context: μ=48.09 (σ=1.20, n=299); z-score=5.66; adaptive threshold=5.01%
    • Assessment: Step change confirmed. After values are bimodal (6 runs at ~54–56 ns, 22 runs at ~67–69 ns). The dominant cluster at ~68 ns represents +41% vs historical mean, indicating the reported delta understates the true regression.
    • Finding Confidence: 4/5 - Good data, bimodal after pattern noted
    • Regression Confidence: 4/5 - Clear signal but bimodal pattern adds complexity; same commit timing
    • Likely Cause: 6580e79bcac1 - Same commit modifies number formatting/parsing code paths
      • Stack Relevance: Shared
      • Rationale: BigInteger.ToStringD shares infrastructure with the number formatting code. The bimodal behavior may indicate JIT tiering differences introduced by the new code path.
    • Verification Status: Not verified

    System.Numerics.Tests.Perf_BigInteger.Add(arguments: 1024,1024 bits) (+12.87%)

    • Delta: 70.18 → 79.21 ns (+12.87%)
    • Historical Context: μ=71.50 (σ=2.45, n=299); z-score=3.15; adaptive threshold=6.85%
    • Assessment: Persistent step change. Higher baseline variance (CV=3.4%) but all 28 after values consistently cluster at 79–81 ns with no reversion.
    • Finding Confidence: 4/5 - Good data, slightly noisier history
    • Regression Confidence: 4/5 - Above adaptive threshold, persistent shift, same temporal alignment
    • Likely Cause: 6580e79bcac1 - Number parsing commit may have changed BigInteger internal layout or added overhead to shared arithmetic paths
      • Stack Relevance: Shared
      • Rationale: BigInteger arithmetic operations share internal representation code with parsing. The commit modifies 58 files across numeric types. Temporal alignment with all other regressions confirms shared root cause.
    • Verification Status: Not verified

    System.Numerics.Tests.Perf_BigInteger.Subtract(arguments: 1024,1024 bits) (+12.83%)

    • Delta: 70.94 → 80.04 ns (+12.83%)
    • Historical Context: μ=71.74 (σ=1.42, n=297); z-score=5.87; adaptive threshold=5%
    • Assessment: Clear step change. Historically stable (CV=1.97%), after values cluster at ~80 ns.
    • Finding Confidence: 4/5 - Excellent data
    • Regression Confidence: 4/5 - Same pattern as Add, confirming BigInteger arithmetic broadly affected
    • Likely Cause: 6580e79bcac1
      • Stack Relevance: Shared
      • Rationale: Same as Add; corroborates that BigInteger internals were broadly impacted.
    • Verification Status: Not verified

    System.Tests.Perf_Single.Parse(value: "-3.4028235E+38") (+18.15%)

    • Delta: 49.71 → 58.73 ns (+18.15%)
    • Historical Context: μ=49.97 (σ=0.60, n=296); z-score=14.70; adaptive threshold=5%
    • Assessment: Extremely high-confidence step change. Ultra-stable baseline (CV=1.19%), after values jump to 57.8–59.4 ns.
    • Finding Confidence: 4/5 - Excellent data quality
    • Regression Confidence: 5/5 - Extreme z-score, Single.Parse directly exercises Number.Parsing.cs
    • Likely Cause: 6580e79bcac1
      • Stack Relevance: Shared
      • Rationale: Single.Parse directly calls into Number.Parsing.cs which was heavily modified (243 additions, 119 deletions). The commit explicitly modifies Single.cs as well.
    • Verification Status: Not verified

    System.Tests.Perf_Single.TryParse(value: "-3.4028235E+38") (+14.54%)

    • Delta: 49.66 → 56.88 ns (+14.54%)
    • Historical Context: μ=49.90 (σ=0.48, n=297); z-score=14.69; adaptive threshold=5%
    • Assessment: Extremely high-confidence step change. Ultra-stable (CV=0.95%).
    • Finding Confidence: 4/5 - Excellent data
    • Regression Confidence: 5/5 - Same code path as Parse, extreme z-score
    • Likely Cause: 6580e79bcac1
      • Stack Relevance: Shared
    • Verification Status: Not verified

    System.Tests.Perf_Single.Parse(value: "3.4028235E+38") (+7.62%)

    • Delta: 49.09 → 52.83 ns (+7.62%)
    • Historical Context: μ=49.27 (σ=0.63, n=300); z-score=5.64; adaptive threshold=5%
    • Assessment: Clear step change. Smaller delta than negative variant suggests the sign-handling path has additional overhead.
    • Finding Confidence: 4/5 - Excellent data
    • Regression Confidence: 5/5 - Above threshold, same pattern
    • Likely Cause: 6580e79bcac1
      • Stack Relevance: Shared
    • Verification Status: Not verified

    System.Tests.Perf_Double.TryParse(value: "-1.7976931348623157e+308") (+26.23%)

    • Delta: 69.19 → 87.34 ns (+26.23%)
    • Historical Context: μ=69.42 (σ=0.62, n=297); z-score=28.80; adaptive threshold=5%
    • Assessment: Extreme step change. Ultra-stable history (CV=0.90%). All 28 after values (75.3–93.6 ns) far above historical ceiling.
    • Finding Confidence: 4/5 - Excellent data
    • Regression Confidence: 5/5 - Extreme z-score (28.8), direct code path
    • Likely Cause: 6580e79bcac1
      • Stack Relevance: Shared
      • Rationale: Double.cs is explicitly modified in this commit. TryParse for Double directly uses Number.Parsing.cs infrastructure.
    • Verification Status: Not verified

    System.Tests.Perf_Double.Parse(value: "-1.7976931348623157e+308") (+11.47%)

    • Delta: 69.50 → 77.47 ns (+11.47%)
    • Historical Context: μ=69.97 (σ=0.73, n=298); z-score=10.30; adaptive threshold=5%
    • Assessment: Clear step change. Historically stable (CV=1.04%). No pre-existing drift.
    • Finding Confidence: 4/5 - Excellent data
    • Regression Confidence: 5/5 - High z-score, same root cause
    • Likely Cause: 6580e79bcac1
      • Stack Relevance: Shared
    • Verification Status: Not verified

    Noise (Dismissed)

    System.Net.Tests.Perf_WebUtility.HtmlDecode_Entities (+6.80%)

    • Delta: 99.51 → 106.28 ns (+6.80%)
    • Historical Context: μ=98.51 (σ=6.16, n=297); z-score=1.26; adaptive threshold=12.51%
    • Assessment: Within historical variance. High CV (6.26%) with prior spikes to 145+ ns. After values (106–114 ns) fall well within observed range.
    • Classification: Noise

    Open Questions

    • Verification Needed: Bisection recommended to confirm 6580e79 is the exact culprit (though confidence is already very high given the commit directly modifies all affected code paths)
    • BigInteger.Add/Subtract: These arithmetic operations don't obviously go through Number.Parsing.cs — the regression mechanism for arithmetic operations should be investigated (possibly shared BigInteger internal buffer/struct changes in the 58-file commit)
    • Bimodal ToStringD: The bimodal after-distribution (6 runs at ~55 ns, 22 at ~68 ns) warrants investigation — may indicate JIT tiering sensitivity or environment-dependent behavior introduced by the change

    Recommended Actions

    1. Source review: Review 6580e79bcac1 focusing on Number.Parsing.cs hot path changes — specifically whether the new NumberStyles option adds overhead to the default parsing path even when not used
    2. Verification: Verify regression with RunWithBuildAtHash on commits e56f59a841b9 (parent) and 6580e79bcac1 to isolate the delta
    3. Follow-up: Investigate BigInteger.Add/Subtract regression mechanism — determine if the commit changed BigInteger struct layout or internal helpers shared with arithmetic

    Bisection Results

    Test Status Culprit Commit Relevance Reported Diff (issue) Bisected Diff (at culprit) Duration
    Parse(numberString: -2147483648) ✅ Culprit found c6e513735e3d ✅ RELATED 94.7→130.8 ns (+38%) 95.2→113.8 ns (+20%) -
    4028235E+38") ✅ Culprit found 6580e79bcac1 ✅ RELATED 49.7→58.7 ns (+18%) 49.7→58.2 ns (+17%) -
    HtmlDecode_Entities ✅ Culprit found 6580e79bcac1 ✅ RELATED 99.5→106.3 ns (+7%) 95.1→107.0 ns (+13%) 2841s

    Total bisection time: 2841s
    Success rate: 3/3 tests

    System.Numerics.Tests.Perf_BigInteger.Parse(numberString: -2147483648)

    • Culprit: c6e513735e3d
    • Commit Message: JIT: fix GC holes in large stack-target struct block ops (JIT: fix GC holes in large stack-target struct block ops #130352)
    • Relevance: RELATED
    • Explanation: This JIT commit changes the lowering of struct copy/zero block operations for GC-pointer-containing structs. Instead of using fast Memmove/ClearWithoutReferences helpers, larger structs now use slower GC-safe CpObj/pointer-sized loop paths. BigInteger.Parse likely involves stack struct operations with GC pointers that hit this new slower path, explaining the ~20% regression.

    Helix jobs

    Job ID Submitted by bisection Last status
    04912f25-bcaa-4f4f-92a3-6d9a9f65045e ✅ submitted
    9876520d-2d83-4b33-9e30-0b0bb8ee7c2b ✅ submitted

    Per-commit measurements

    Commit Position Measurements
    1ab7d053 q1 - index 5 95.380 ns
    7abe5f60c079 index 6 94.960 ns
    07f0abcdaf13 index 7 95.180 ns
    c6e513735e3d index 8 113.820 ns
    0f9f04b71512 mid - index 10 116.190 ns
    66dedd6f q3 - index 15 113.890 ns
    Bisection details

    …[earlier output omitted]

    └ {"index":null,"sha":"c6e513735e3d4bbea3e3922c2a101a13d1fd37a9","abbrevSha":"c...

    I've found the culprit. The commit c6e513735e3d4bbea3e3922c2a101a13d1fd37a9 is a JIT fix for GC holes in large stack-target struct block ops. It changes the lowering path for struct copies/zeroing that contain GC pointers — instead of using fast CORINFO_HELP_MEMCPY (Memmove), larger copies now stay on the GC-aware CpObj path (pointer-sized loop). This is directly relevant to BigInteger.Parse which likely involves stack struct operations that now take a slower but GC-safe path.

    BISECTION RESULT:
    - Culprit Commit: c6e513735e3d4bbea3e3922c2a101a13d1fd37a9
    - Commit Message: JIT: fix GC holes in large stack-target struct block ops (#130352)
    - Commits Tested: 6
    - Confidence: 4/5
    - Commit Relevance: RELATED
    - Relevance Explanation: This JIT commit changes the lowering of struct copy/zero block operations for GC-pointer-containing structs. Instead of using fast Memmove/ClearWithoutReferences helpers, larger structs now use slower GC-safe CpObj/pointer-sized loop paths. BigInteger.Parse likely involves stack struct operations with GC pointers that hit this new slower path, explaining the ~20% regression.
    - Baseline Value: 95.18 ns   (the measured perf at the last-good commit before the culprit, e.g. "1245.0 ns")
    - Culprit Value: 113.82 ns    (the measured perf at the culprit commit, same units as Baseline Value)
    - Summary: Bisected a 20-commit range in two rounds of Helix jobs. First round narrowed to commits 6-10 (Q2), second round pinpointed the regression to commit c6e51373 which changes JIT lowering for struct block ops containing GC pointers.
    
    HELIX JOBS:
    - 04912f25-bcaa-4f4f-92a3-6d9a9f65045e
    - 9876520d-2d83-4b33-9e30-0b0bb8ee7c2b
    
    MEASUREMENTS:
    - 1ab7d053 (Q1 - index 5): 95.38 ns
    - 0f9f04b7 (mid - index 10): 116.19 ns
    - 66dedd6f (Q3 - index 15): 113.89 ns
    - 7abe5f60 (index 6): 94.96 ns
    - 07f0abcd (index 7): 95.18 ns
    - c6e51373 (index 8): 113.82 ns
    

    System.Tests.Perf_Single.Parse(value: "-3.4028235E+38")

    • Culprit: 6580e79bcac1
    • Commit Message: Add new NumberStyles option to stop parsing on invalid character (Add new NumberStyles option to stop parsing on invalid character #130210)
    • Relevance: RELATED
    • Explanation: This commit modifies src/libraries/System.Private.CoreLib/src/System/Number.Parsing.cs (+243/-119) and src/libraries/Common/src/System/Number.Parsing.Common.cs (+36/-23), which are the core parsing routines directly exercised by System.Tests.Perf_Single.Parse. The new NumberStyles option adds conditional logic to the parsing hot path, plausibly causing the ~18% regression.

    Helix jobs

    Job ID Submitted by bisection Last status
    f55bc575-9587-4f74-ab85-d01fd4ecfefe ✅ submitted
    7ce078a4-7e87-4148-a10b-6ce7bbabc382 ✅ submitted

    Per-commit measurements

    Commit Position Measurements
    96897151f7c7 index 11 49.680 ns
    e56f59a841b9 index 12 49.710 ns
    6580e79bcac1 index 13 58.210 ns
    0f9f04b71512 mid 49.740 ns
    1ab7d0535812 quarter 49.810 ns
    66dedd6f553f three-quarter 59.490 ns
    Bisection details

    …[earlier output omitted]

    51f7c7` (index 11): 49.68 ns → good

    • e56f59a841b9 (index 12): 49.71 ns → good
    • 6580e79bcac1 (index 13): 58.21 ns → regressed!

    The regression was introduced at commit 6580e79bcac113784d6ab8b961bd5dcb0479623f. Let me examine it.

    ● get_single_commit_summary (MCP: PerfTriageMcp) · owner: "dotnet", repo: "runtime", sha: "6580e79b…
    └ {"index":null,"sha":"6580e79bcac113784d6ab8b961bd5dcb0479623f","abbrevSha":"6...

    The culprit commit 6580e79bcac1 is "Add new NumberStyles option to stop parsing on invalid character (#130210)" by Tanner Gooding. It modifies Number.Parsing.cs (+243/-119 lines) — the core number parsing infrastructure that Single.Parse() directly exercises.

    BISECTION RESULT:
    - Culprit Commit: 6580e79bcac113784d6ab8b961bd5dcb0479623f
    - Commit Message: Add new NumberStyles option to stop parsing on invalid character (#130210)
    - Commits Tested: 6
    - Confidence: 5
    - Commit Relevance: RELATED
    - Relevance Explanation: This commit modifies src/libraries/System.Private.CoreLib/src/System/Number.Parsing.cs (+243/-119) and src/libraries/Common/src/System/Number.Parsing.Common.cs (+36/-23), which are the core parsing routines directly exercised by System.Tests.Perf_Single.Parse. The new NumberStyles option adds conditional logic to the parsing hot path, plausibly causing the ~18% regression.
    - Baseline Value: 49.71 ns
    - Culprit Value: 58.21 ns
    - Summary: Double bisection of 20 commits narrowed the regression to commit 6580e79bcac1, which introduced it between e56f59a841b9 (good, 49.71 ns) and 6580e79bcac1 (regressed, 58.21 ns). The commit adds a new NumberStyles parsing option that modifies the number parsing hot path.
    
    HELIX JOBS:
    - f55bc575-9587-4f74-ab85-d01fd4ecfefe
    - 7ce078a4-7e87-4148-a10b-6ce7bbabc382
    
    MEASUREMENTS:
    - 1ab7d0535812 (quarter): 49.81 ns
    - 0f9f04b71512 (mid): 49.74 ns
    - 66dedd6f553f (three-quarter): 59.49 ns
    - 96897151f7c7 (index 11): 49.68 ns
    - e56f59a841b9 (index 12): 49.71 ns
    - 6580e79bcac1 (index 13): 58.21 ns
    

    System.Net.Tests.Perf_WebUtility.HtmlDecode_Entities

    • Culprit: 6580e79bcac1
    • Commit Message: Add new NumberStyles option to stop parsing on invalid character (Add new NumberStyles option to stop parsing on invalid character #130210)
    • Relevance: RELATED
    • Explanation: The commit makes extensive changes to System.Number.Parsing.cs (+243/-119 lines), modifying number parsing paths. The regression test HtmlDecode_Entities parses hex numeric character references (水) which relies on the hex number parsing infrastructure that this commit modifies. The mechanism is plausible: changes to NumberStyles handling and parsing logic directly affect the hot path for hex entity decoding.

    Helix jobs

    Job ID Submitted by bisection Last status
    3a4cfdf0-5250-4788-b94c-b159eb0f76fa ✅ submitted
    c1b6b486-5179-494f-83bf-6eca8f541dd7 ✅ submitted
    134ea932-62b9-4e13-95c4-56b45da721e0 ✅ submitted

    Per-commit measurements

    Commit Position Measurements
    1ab7d0535812 q1-quarter 95.200 ns
    0f9f04b71512 q2-mid 95.530 ns
    66dedd6f553f q3-threequarter 106.440 ns
    96897151f7c7 index 11 (run 2, 3) 104.440, 95.390 ns
    e56f59a841b9 index 12 (run 2, 3) 95.120, 99.800 ns
    6580e79bcac1 index 13 (run 2, 3) 106.090, 107.040 ns
    2269602f4da6 index 14 119.330 ns
    Bisection details

    …[earlier output omitted]

    • 0f9f04b7 (index 10, last known good): 95.53 ns
    • e56f59a8 (index 12, confirmed good): 95.12 ns / 99.80 ns
    • 6580e79b (index 13, first regressed): 106.09 ns / 107.04 ns

    The regression is clearly introduced by commit 6580e79bcac113784d6ab8b961bd5dcb0479623f.

    BISECTION RESULT:
    - Culprit Commit: 6580e79bcac113784d6ab8b961bd5dcb0479623f
    - Commit Message: Add new NumberStyles option to stop parsing on invalid character (#130210)
    - Commits Tested: 9
    - Confidence: 4/5
    - Commit Relevance: RELATED
    - Relevance Explanation: The commit makes extensive changes to System.Number.Parsing.cs (+243/-119 lines), modifying number parsing paths. The regression test HtmlDecode_Entities parses hex numeric character references (水) which relies on the hex number parsing infrastructure that this commit modifies. The mechanism is plausible: changes to NumberStyles handling and parsing logic directly affect the hot path for hex entity decoding.
    - Baseline Value: 95.12 ns   (the measured perf at the last-good commit before the culprit, e56f59a841b9)
    - Culprit Value: 107.04 ns    (the measured perf at the culprit commit, 6580e79bcac1)
    - Summary: Double bisection of 20 commits between 3ade0b55 and e302d2a2 narrowed the regression to commit 6580e79b, which introduces extensive number parsing changes. The regression (~12.5%) is consistent across multiple runs and aligns with the reported +6.8% regression.
    
    HELIX JOBS:
    - 3a4cfdf0-5250-4788-b94c-b159eb0f76fa
    - c1b6b486-5179-494f-83bf-6eca8f541dd7
    - 134ea932-62b9-4e13-95c4-56b45da721e0
    
    MEASUREMENTS:
    - 1ab7d0535812 (Q1-quarter): 95.20 ns
    - 0f9f04b71512 (Q2-mid): 95.53 ns
    - 66dedd6f553f (Q3-threeQuarter): 106.44 ns
    - 96897151f7c7 (index 11, run 2): 104.44 ns
    - e56f59a841b9 (index 12, run 2): 95.12 ns
    - 6580e79bcac1 (index 13, run 2): 106.09 ns
    - 96897151f7c7 (index 11, run 3): 95.39 ns
    - e56f59a841b9 (index 12, run 3): 99.80 ns
    - 6580e79bcac1 (index 13, run 3): 107.04 ns
    - 2269602f4da6 (index 14, run 3): 119.33 ns
    

    🤖 Proposed Next Actions

    Rule: R3R4_multi_cause — Known-cause regression — multiple causes.

    Verified regression with 2 distinct culprits: c6e513735e3d (System.Numerics.Tests.Perf_BigInteger.Parse(numberString: -2147483648)); 6580e79bcac1 (System.Tests.Perf_Single.Parse(value: "-3.4028235E+38")). 1 already linked, 1 new.

    # Action Status Details
    1 Back-reference cause PR 🕒 Planned on culprit PR for c6e513735e3d — tests: System.Numerics.Tests.Perf_BigInteger.Parse(numberString: -2147483648)
    2 Append related regression 🕒 Planned on #130697 — tests: System.Tests.Perf_Single.Parse(value: "-3.4028235E+38")
    3 Transfer issue 🕒 Planned → dotnet/runtime
    4 Ping owners 🕒 Planned ping EgorBo
    5 Remove label 🕒 Planned untriaged

    🦆 Pre-bisection Re-analysis (advisory)

    ⚠️ Disputes the pre-bisection plan — advisory only, not applied

    Flags: wasted-bisection-slot, missing-candidate

    Rationale: System.Net.Tests.Perf_WebUtility.HtmlDecode_Entities appears in the candidate list but has no corresponding per-finding signal entry, making it unjustified for an expensive bisection slot. Meanwhile, System.Tests.Perf_Double.TryParse(value: "-1.7976931348623157e+308") has changeConf 5/5 and is not queued despite being a distinct code path from the Single.Parse candidate already selected.

    Suggested drop from bisection: System.Net.Tests.Perf_WebUtility.HtmlDecode_Entities (advisory — not applied)

    Suggested add to bisection: System.Tests.Perf_Double.TryParse(value: "-1.7976931348623157e+308") (advisory — not applied)

    This critique is advisory in the current configuration and did not change which tests were bisected.

    🦆 Re-analysis (advisory)

    ⚠️ Disputes the analysis — advisory only, not applied

    Flags: overconfident, culprit-mismatch, bimodal-history-concern

    Rationale: The overall culprit SHA c6e5137 is supported by only 1 of 3 bisections, while the majority (2/3 bisections and all 10 per-finding signals) point to 6580e79 — the chosen culprit appears to be the minority result. Additionally, bimodal history is flagged True yet change confidence is 5/5; bimodal signals typically warrant a reduction since they indicate intermittent behavior that can mimic real regressions.

    Disputed culprit: c6e513735e3d4bbea3e3922c2a101a13d1fd37a9 (advisory — not applied)

    Suggested confidence: Finding Confidence → 4/5; Change Confidence → 4/5 (advisory — not applied)

    This critique is advisory in the current configuration and did not change the confidences, culprit, or proposed actions above.


    Generated by PerfTriageAgent on 2026-07-14 17:19 UTC

  9. tannergooding commented on Sep 18, 2026

    @tannergooding
    Member

    None of this is critical. There's a redundant branch that the JIT could be taught to remove, but its decently complex and not a good candidate for .NET 11

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

arch-x64area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMIos-linuxLinux OS (any supported distro)runtime-coreclrspecific to the CoreCLR runtimetenet-performancePerformance related issuetenet-performance-benchmarksIssue from performance benchmark

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions