Skip to content

[Perf] Changes at 7/5/2022 10:32:06 PM #72030

Closed
Closed
[Perf] Changes at 7/5/2022 10:32:06 PM#72030
Assignees
tannergooding
Labels
arch-x64area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMIos-linuxLinux OS (any supported distro)runtime-coreclrspecific to the CoreCLR runtimetenet-performancePerformance related issuetenet-performance-benchmarksIssue from performance benchmark
Milestone
8.0.0

Description

@performanceautofiler
performanceautofiler
bot
opened on Jul 12, 2022

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 338db1a1ba498f308797ce7a54c0ed54e7286d50
Compare ffcb0149763663a55a79784be216afe81f6bf76c
Diff Diff

Regressions in Benchstone.MDBenchF.MDLLoops

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Test - Duration of single invocation 658.14 ms 695.60 ms 1.06 0.01 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'Benchstone.MDBenchF.MDLLoops*'
Details

Payloads

Baseline
Compare

Histogram

Benchstone.MDBenchF.MDLLoops.Test

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 695.6042480769231 > 688.2603742326924.
IsChangePoint: Marked as a change because one of 5/11/2022 7:02:26 PM, 5/21/2022 4:54:11 AM, 6/7/2022 5:42:36 AM, 7/5/2022 7:01:28 PM, 7/12/2022 6:44:07 AM falls between 7/3/2022 4:26:07 AM and 7/12/2022 6:44:07 AM.
IsRegressionStdDev: Marked as regression because -40.861647117168786 (T) = (0 -696232577.1907164) / Math.Sqrt((12804781397326.438 / (31)) + (24144080383239.016 / (45))) is less than -1.9925434951804422 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (45) - 2, .025) and -0.06066058823673496 = (656414111.0853835 - 696232577.1907164) / 656414111.0853835 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 338db1a1ba498f308797ce7a54c0ed54e7286d50
Compare ffcb0149763663a55a79784be216afe81f6bf76c
Diff Diff

Regressions in Benchstone.MDBenchI.MDMulMatrix

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Test - Duration of single invocation 644.48 ms 861.47 ms 1.34 0.00 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'Benchstone.MDBenchI.MDMulMatrix*'
Details

Payloads

Baseline
Compare

Histogram

Benchstone.MDBenchI.MDMulMatrix.Test

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 861.4729290714286 > 676.9384600500001.
IsChangePoint: Marked as a change because one of 7/5/2022 7:01:28 PM, 7/12/2022 6:44:07 AM falls between 7/3/2022 4:26:07 AM and 7/12/2022 6:44:07 AM.
IsRegressionStdDev: Marked as regression because -318.2377784562592 (T) = (0 -865401784.0575421) / Math.Sqrt((225348633110.33362 / (31)) + (20803085904355.203 / (44))) is less than -1.99299712588925 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (44) - 2, .025) and -0.3419061262248373 = (644904861.1859034 - 865401784.0575421) / 644904861.1859034 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Activity

  1. performanceautofiler
    assigned
    tannergooding
    on Jul 12, 2022
  2. performanceautofiler
    added
    CoreClr
    refs/heads/main
    Regression
    RunKind=micro
    ubuntu 18.04
    untriagedNew issue has not been triaged by the area owner
    x64
    on Jul 12, 2022
  3. DrewScoggins
    removed
    untriagedNew issue has not been triaged by the area owner
    on Jul 12, 2022
  4. DrewScoggins
    transferred this issue fromdotnet/perf-autofiling-issueson Jul 12, 2022
  6. ghost added
    untriagedNew issue has not been triaged by the area owner
    on Jul 12, 2022
  7. DrewScoggins
    added
    tenet-performancePerformance related issue
    tenet-performance-benchmarksIssue from performance benchmark
    on Jul 12, 2022

  8. DrewScoggins commented on Jul 12, 2022

    @DrewScoggins
    DrewScoggins
    on Jul 12, 2022
    Member

    Related to this commit: #70271

  9. 3 remaining items

  10. JulieLeeMSFT
    added this to the 7.0.0 milestone on Jul 12, 2022
  11. tannergooding
    modified the milestones: 7.0.0, 8.0.0 on Aug 1, 2022

  12. mrsharm commented on Aug 8, 2022

    @mrsharm
    mrsharm
    on Aug 8, 2022
    Contributor

    This regression showed up in our perf report for preview 7. The details are as follows:

    Details

    Benchstone.MDBenchI.MDMulMatrix.Test

    Result Ratio Alloc Delta Operating System Bit Processor Name
    Faster 1.66 -288 Windows 11 Arm64 Microsoft SQ1 3.0 GHz
    Faster 1.71 +0 Windows 11 Arm64 Microsoft SQ1 3.0 GHz
    Faster 1.23 +672 macOS Monterey 12.3 Arm64 Apple M1 Max
    Slower 0.76 +0 Windows 10 X64 Intel Xeon CPU E5-1650 v4 3.60GHz
    Slower 0.75 +0 Windows 10 X64 Intel Core i7-6700 CPU 3.40GHz (Skylake)
    Slower 0.80 +0 Windows 10 X64 Intel Core i7-6700 CPU 3.40GHz (Skylake)
    Slower 0.79 +0 Windows 10 X64 Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R)
    Slower 0.76 +0 Windows 10 X64 Intel Core i9-10900K CPU 3.70GHz
    Slower 0.81 +0 Windows 11 X64 AMD Ryzen Threadripper PRO 3945WX 12-Cores
    Slower 0.86 +0 Windows 11 X64 AMD Ryzen 9 3950X
    Same 1.04 +0 Windows 11 X64 AMD Ryzen 9 5900X
    Same 0.99 +0 Windows 11 X64 AMD Ryzen 9 5950X
    Slower 0.81 +0 Windows 11 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
    Slower 0.77 +0 Windows 11 X64 Intel Core i9-10900K CPU 3.70GHz
    Slower 0.61 +0 Windows 11 X64 11th Gen Intel Core i9-11900H 2.50GHz
    Slower 0.74 +0 ubuntu 18.04 X64 Intel Xeon CPU E5-1650 v4 3.60GHz
    Slower 0.86 +0 ubuntu 18.04 X64 Intel Core i7-2720QM CPU 2.20GHz (Sandy Bridge)
    Slower 0.74 +0 ubuntu 18.04 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
    Same 0.97 +0 ubuntu 20.04 X64 AMD Ryzen 9 5900X
    Slower 0.72 +0 ubuntu 20.04 X64 Intel Core i9-10900K CPU 3.70GHz
    Same 0.99 +0 Windows 10 X86 Intel Xeon CPU E5-1650 v4 3.60GHz
    Same 1.03 +0 Windows 10 X86 Intel Core i7-6700 CPU 3.40GHz (Skylake)
    Same 0.98 +0 Windows 11 X86 AMD Ryzen Threadripper PRO 3945WX 12-Cores
    Slower 0.73 +0 macOS Big Sur 11.6.8 X64 Intel Core i5-4278U CPU 2.60GHz (Haswell)
    Slower 0.73 +0 macOS Monterey 12.3.1 X64 Intel Core i7-5557U CPU 3.10GHz (Broadwell)
    Slower 0.72 +672 macOS Monterey 12.4 X64 Intel Core i5-4278U CPU 2.60GHz (Haswell)

  13. BruceForstall commented on Aug 12, 2022

    @BruceForstall
    BruceForstall
    on Aug 12, 2022
    Contributor

    This regression is expected, as per discussion in #70271. Follow-up work items have been filed to improve this.

  14. BruceForstall
    closed this as completedon Aug 12, 2022
  15. ghost locked as resolved and limited conversation to collaborators on Sep 12, 2022
  16. jeffhandley
    added
    runtime-coreclrspecific to the CoreCLR runtime
    os-linuxLinux OS (any supported distro)
    arch-x64
    and removed
    CoreClr
    refs/heads/main
    Regression
    ubuntu 18.04
    x64
    on Dec 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

arch-x64area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMIos-linuxLinux OS (any supported distro)runtime-coreclrspecific to the CoreCLR runtimetenet-performancePerformance related issuetenet-performance-benchmarksIssue from performance benchmark

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions