Repository navigation
[Perf] Regressions in System.Tests.Perf_String #72026
Copy link
Copy link
Closed
Labels
area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMICLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMIruntime-coreclrspecific to the CoreCLR runtimespecific to the CoreCLR runtimetenet-performancePerformance related issuePerformance related issuetenet-performance-benchmarksIssue from performance benchmarkIssue from performance benchmark
Milestone
Description
Run Information
| Architecture | x64 |
|---|---|
| OS | ubuntu 18.04 |
| Baseline | 49288b4ebea7f1792fbd1e09726cb7ee21aa1088 |
| Compare | c9feedf55ea613a2b9220fe6b9e8feda2aa72287 |
| Diff | Diff |
Regressions in System.Tests.Perf_String
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| Replace_Char - Duration of single invocation | 7.29 ns | 8.85 ns | 1.21 | 0.04 | False | |||||
| Replace_Char - Duration of single invocation | 24.73 ns | 26.92 ns | 1.09 | 0.02 | False | |||||
| Trim_CharArr - Duration of single invocation | 13.10 ns | 14.28 ns | 1.09 | 0.04 | False | |||||
| TrimEnd_CharArr - Duration of single invocation | 12.03 ns | 13.16 ns | 1.09 | 0.03 | False | |||||
| TrimStart_CharArr - Duration of single invocation | 11.99 ns | 12.99 ns | 1.08 | 0.06 | False | |||||
| Insert - Duration of single invocation | 14.79 ns | 16.16 ns | 1.09 | 0.09 | False | |||||
| Trim_CharArr - Duration of single invocation | 15.00 ns | 16.25 ns | 1.08 | 0.05 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_String*'Details
Payloads
Histogram
System.Tests.Perf_String.Replace_Char(text: "This is a very nice sentence", oldChar: 'z', newChar: 'y')
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.845170871226863 > 7.670440403693207.
IsChangePoint: Marked as a change because one of 5/20/2022 9:09:19 PM, 6/6/2022 10:12:27 PM, 6/30/2022 12:21:07 PM, 7/9/2022 7:52:53 PM, 7/12/2022 6:44:07 AM falls between 7/3/2022 4:26:07 AM and 7/12/2022 6:44:07 AM.
IsRegressionStdDev: Marked as regression because -24.68730484131406 (T) = (0 -8.86006701982299) / Math.Sqrt((0.30394777561351416 / (62)) + (0.00028404941372391833 / (14))) is less than -1.9925434951804422 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (62) + (14) - 2, .025) and -0.24300139752005798 = (7.127962235199352 - 8.86006701982299) / 7.127962235199352 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_String.Replace_Char(text: "yfesgj0sg1ijslnjsb3uofdz3tbzf6ysgblu3at20nfab2wei1kxfbvsbpzwhanjczcqa2psra3aacxb67qnwbnfp2tok6v0a58l", oldChar: 'b', newChar: '+')
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 26.923045448271715 > 25.96829592730264.
IsChangePoint: Marked as a change because one of 5/23/2022 3:43:39 PM, 5/30/2022 12:57:05 PM, 6/2/2022 10:27:31 PM, 6/21/2022 2:34:11 AM, 6/30/2022 12:21:07 PM, 7/9/2022 7:52:53 PM, 7/12/2022 6:44:07 AM falls between 7/3/2022 4:26:07 AM and 7/12/2022 6:44:07 AM.
IsRegressionStdDev: Marked as regression because -16.902264164705286 (T) = (0 -26.85562320373283) / Math.Sqrt((0.2177592740561024 / (62)) + (0.11167489328952931 / (14))) is less than -1.9925434951804422 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (62) + (14) - 2, .025) and -0.0723409339300548 = (25.043922463454642 - 26.85562320373283) / 25.043922463454642 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_String.Trim_CharArr(s: " Test", c: [' ', ' '])
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.284826701512642 > 13.396557193427627.
IsChangePoint: Marked as a change because one of 5/26/2022 12:53:48 AM, 6/27/2022 8:54:23 AM, 7/6/2022 12:32:56 PM, 7/12/2022 6:44:07 AM falls between 7/3/2022 4:26:07 AM and 7/12/2022 6:44:07 AM.
IsRegressionStdDev: Marked as regression because -26.96687179926083 (T) = (0 -14.173591246385067) / Math.Sqrt((0.03788567263726484 / (37)) + (0.0383699250110538 / (39))) is less than -1.9925434951804422 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (39) - 2, .025) and -0.09319825995530753 = (12.965252292813306 - 14.173591246385067) / 12.965252292813306 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_String.TrimEnd_CharArr(s: "Test ", c: [' ', ' '])
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.163854667838171 > 12.618378592555835.
IsChangePoint: Marked as a change because one of 5/11/2022 11:37:12 AM, 6/27/2022 8:54:23 AM, 7/6/2022 7:05:00 PM, 7/12/2022 6:44:07 AM falls between 7/3/2022 4:26:07 AM and 7/12/2022 6:44:07 AM.
IsRegressionStdDev: Marked as regression because -22.782476194670256 (T) = (0 -12.927894127663626) / Math.Sqrt((0.034522014778461146 / (38)) + (0.03389912496939499 / (38))) is less than -1.9925434951804422 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (38) - 2, .025) and -0.08082221625195152 = (11.961166168932627 - 12.927894127663626) / 11.961166168932627 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_String.TrimStart_CharArr(s: " Test", c: [' ', ' '])
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.993530472880698 > 12.590734687620323.
IsChangePoint: Marked as a change because one of 5/31/2022 12:36:20 AM, 5/31/2022 10:47:41 AM, 6/27/2022 8:54:23 AM, 7/6/2022 7:05:00 PM, 7/12/2022 6:44:07 AM falls between 7/3/2022 4:26:07 AM and 7/12/2022 6:44:07 AM.
IsRegressionStdDev: Marked as regression because -18.62570938687287 (T) = (0 -12.940232668090372) / Math.Sqrt((0.041070098657543234 / (38)) + (0.060150082009386575 / (38))) is less than -1.9925434951804422 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (38) - 2, .025) and -0.08024834863923754 = (11.978942327836805 - 12.940232668090372) / 11.978942327836805 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_String.Insert(s1: "dzsdzsDDZSDZSDZSddsz", i: 7, s2: "Test")
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.15650202137041 > 15.507248952399829.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 6/27/2022 8:54:23 AM, 7/6/2022 12:32:56 PM, 7/12/2022 6:44:07 AM falls between 7/3/2022 4:26:07 AM and 7/12/2022 6:44:07 AM.
IsRegressionStdDev: Marked as regression because -22.399239937854425 (T) = (0 -16.11370988082571) / Math.Sqrt((0.1224892606655733 / (37)) + (0.06619022737766722 / (38))) is less than -1.99299712588925 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (38) - 2, .025) and -0.10963953337967557 = (14.521571551932283 - 16.11370988082571) / 14.521571551932283 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_String.Trim_CharArr(s: " Te st ", c: [' ', ' '])
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.250477173889717 > 15.71603280005312.
IsChangePoint: Marked as a change because one of 5/12/2022 8:50:50 AM, 5/20/2022 10:18:01 PM, 5/25/2022 1:42:32 AM, 6/7/2022 6:20:49 PM, 6/20/2022 2:50:37 PM, 6/23/2022 8:35:19 AM, 7/6/2022 7:05:00 PM, 7/12/2022 6:44:07 AM falls between 7/3/2022 4:26:07 AM and 7/12/2022 6:44:07 AM.
IsRegressionStdDev: Marked as regression because -20.812178686587128 (T) = (0 -16.05538192959856) / Math.Sqrt((0.03792612090128188 / (38)) + (0.054471320135789945 / (37))) is less than -1.99299712588925 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (37) - 2, .025) and -0.0688636415004556 = (15.020982383740028 - 16.05538192959856) / 15.020982383740028 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
category:performance
theme:benchmarks
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMICLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMIruntime-coreclrspecific to the CoreCLR runtimespecific to the CoreCLR runtimetenet-performancePerformance related issuePerformance related issuetenet-performance-benchmarksIssue from performance benchmarkIssue from performance benchmark







Activity
[-][Perf] Changes at 7/10/2022 1:04:01 AM[/-][+][Perf] Regressions in System.Buffers.Tests.ReadOnlySequenceTests<Char>[/+]DrewScoggins commentedon Jul 12, 2022
Seems related to #71659
DrewScoggins commentedon Jul 12, 2022
Related regressions dotnet/perf-autofiling-issues#6704
7 remaining items
tannergooding commentedon Jun 9, 2023
Regressions are all within an acceptable (and very small, sub 5ns) range. Many of them are now showing overall improvements as well.