Repository navigation
Regressions from "Avoid xchg for resolution" #81768
Description
Run Information
| Architecture | x64 |
|---|---|
| OS | ubuntu 18.04 |
| Baseline | 9e05d3347be231ae1bfcc7037ddecfb5c4ca51a1 |
| Compare | 120b7215c9d029fccee94f46864494d79074a0eb |
| Diff | Diff |
Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| Count - Duration of single invocation | 280.39 ms | 303.10 ms | 1.08 | 0.01 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig*'Details
Payloads
Histogram
System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: ".{0,2}(Tom|Sawyer|Huckleberry|Finn)", Options: Compiled)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 303.0988273333333 > 291.22724740673084.
IsChangePoint: Marked as a change because one of 1/11/2023 2:37:52 AM, 1/27/2023 7:12:54 PM, 2/1/2023 8:39:12 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -6.8991552954396305 (T) = (0 -303268185.5826008) / Math.Sqrt((192513142277717.06 / (27)) + (51382853519.585594 / (32))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (32) - 2, .025) and -0.06468243531225498 = (284843795.2239316 - 303268185.5826008) / 284843795.2239316 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
| Architecture | x64 |
|---|---|
| OS | ubuntu 18.04 |
| Baseline | 9e05d3347be231ae1bfcc7037ddecfb5c4ca51a1 |
| Compare | 120b7215c9d029fccee94f46864494d79074a0eb |
| Diff | Diff |
Regressions in System.Net.Http.Tests.SocketsHttpHandlerPerfTest
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| Get_EnumerateHeaders_Unvalidated - Duration of single invocation | 8.95 ms | 17.44 ms | 1.95 | 0.90 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Net.Http.Tests.SocketsHttpHandlerPerfTest*'Details
Payloads
Histogram
System.Net.Http.Tests.SocketsHttpHandlerPerfTest.Get_EnumerateHeaders_Unvalidated(ssl: True, chunkedResponse: True, responseLength: 100000)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 17.440992809375 > 11.242045705142887.
IsChangePoint: Marked as a change because one of 2/1/2023 8:39:12 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -3.548097165978861 (T) = (0 -15794333.253045475) / Math.Sqrt((17374797894054.414 / (27)) + (37834194257680.664 / (32))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (32) - 2, .025) and -0.4358447430604137 = (11000028.609904464 - 15794333.253045475) / 11000028.609904464 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Architecture | x64 |
|---|---|
| OS | ubuntu 18.04 |
| Baseline | 9e05d3347be231ae1bfcc7037ddecfb5c4ca51a1 |
| Compare | 120b7215c9d029fccee94f46864494d79074a0eb |
| Diff | Diff |
Regressions in SciMark2.kernel
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| benchSparseMult - Duration of single invocation | 609.18 ms | 696.21 ms | 1.14 | 0.00 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'SciMark2.kernel*'Details
Payloads
Histogram
SciMark2.kernel.benchSparseMult
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 696.2089355333333 > 639.6030126750002.
IsChangePoint: Marked as a change because one of 2/1/2023 8:39:12 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -657.0518642017826 (T) = (0 -696370455.7566222) / Math.Sqrt((209470639230.72266 / (26)) + (303351012897.6243 / (32))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (26) + (32) - 2, .025) and -0.14278876574405194 = (609360606.8162792 - 696370455.7566222) / 609360606.8162792 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Activity
[-][Perf] Linux/x64: 3 Regressions on 2/1/2023 10:12:25 AM[/-][+]Regressions from "Avoid xchg for resolution" [/+]EgorBo commentedon Feb 7, 2023
#81216 cc @jakobbotsch
jakobbotsch commentedon Mar 16, 2023
I think both of these are modalities:
The Ryzen regression from the linked issue does look real, it improved with #79346 but then regressed again with #81216:
I think to handle this we would need to change the strategy depending on exact model of x64 CPU, which we probably don't have appetite for yet.
Overall the benchmark is still better off than before #79346, though, so there is no long-term regression here, and I will close this.