Repository navigation
[Perf] Changes at 5/28/2022 1:41:21 AM #70162
Copy link
Copy link
Closed
Labels
arch-arm64area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMICLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMIruntime-coreclrspecific to the CoreCLR runtimespecific to the CoreCLR runtime
Milestone
Description
Run Information
| Architecture | arm64 |
|---|---|
| OS | ubuntu 20.04 |
| Baseline | 7f7c6a229b691bd060ee1bf4aeb2cea4c5e52168 |
| Compare | 748dfb78ef87b38599b89e85d3201cdb9f9c0b33 |
| Diff | Diff |
Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| Count - Duration of single invocation | 2.20 ms | 2.98 ms | 1.36 | 0.05 | True | |||||
| Count - Duration of single invocation | 153.31 μs | 191.44 μs | 1.25 | 0.01 | True | |||||
| Count - Duration of single invocation | 1.24 ms | 1.37 ms | 1.10 | 0.10 | True | |||||
| Count - Duration of single invocation | 24.70 ms | 34.02 ms | 1.38 | 0.10 | True | |||||
| Count - Duration of single invocation | 144.68 μs | 165.27 μs | 1.14 | 0.01 | False | |||||
| Count - Duration of single invocation | 240.75 μs | 269.14 μs | 1.12 | 0.01 | True | |||||
| Count - Duration of single invocation | 12.03 ms | 21.39 ms | 1.78 | 0.05 | True | |||||
| Count - Duration of single invocation | 47.29 ms | 59.27 ms | 1.25 | 0.44 | True | |||||
| Count - Duration of single invocation | 219.57 μs | 232.72 μs | 1.06 | 0.01 | True | |||||
| Count - Duration of single invocation | 89.40 μs | 123.35 μs | 1.38 | 0.01 | True | |||||
| Count - Duration of single invocation | 2.33 ms | 2.46 ms | 1.06 | 0.02 | True | |||||
| Count - Duration of single invocation | 177.12 μs | 211.80 μs | 1.20 | 0.01 | True | |||||
| Count - Duration of single invocation | 3.18 ms | 3.36 ms | 1.06 | 0.02 | True | |||||
| Count - Duration of single invocation | 3.08 ms | 3.48 ms | 1.13 | 0.13 | True | |||||
| Count - Duration of single invocation | 94.23 μs | 128.22 μs | 1.36 | 0.01 | True | |||||
| Count - Duration of single invocation | 374.51 μs | 422.05 μs | 1.13 | 0.01 | True | |||||
| Count - Duration of single invocation | 49.23 ms | 60.94 ms | 1.24 | 0.48 | True |
Repro
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock*'Details
Payloads
Histogram
System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "the\s+\w+", Options: NonBacktracking)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.981663075502008 > 2.30941796713219.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -28.080765181910134 (T) = (0 -2748213.418441032) / Math.Sqrt((563613961.9919467 / (25)) + (7259741238.014956 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.2509798087203997 = (2196848.7415093617 - 2748213.418441032) / 2196848.7415093617 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "[a-q][^u-z]{13}x", Options: NonBacktracking)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 191.43783472206997 > 160.95435806566604.
IsChangePoint: Marked as a change because one of 5/31/2022 5:02:26 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -77.0805759253954 (T) = (0 -190973.89435431754) / Math.Sqrt((6394852.873722199 / (39)) + (317440.2113534399 / (6))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (6) - 2, .025) and -0.23147466580807602 = (155077.40407229832 - 190973.89435431754) / 155077.40407229832 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "the", Options: NonBacktracking)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.3717235683241253 > 1.2718615059933995.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -18.20562778818909 (T) = (0 -1386204.0048313157) / Math.Sqrt((627918840.8777322 / (25)) + (1547500831.444926 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.15335016285121025 = (1201893.4487376017 - 1386204.0048313157) / 1201893.4487376017 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "\w+", Options: NonBacktracking)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 34.020709333333336 > 26.112056162500004.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -23.71893120000806 (T) = (0 -33687196.72236111) / Math.Sqrt((108679716720.46556 / (25)) + (2882756164569.5947 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.3723324199439214 = (24547402.825138893 - 33687196.72236111) / 24547402.825138893 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Street", Options: NonBacktracking)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 165.2708110011001 > 151.98199722719656.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 5/31/2022 5:02:26 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -24.825963965192063 (T) = (0 -166486.91418455934) / Math.Sqrt((34070055.2268123 / (39)) + (1392632.58256889 / (6))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (6) - 2, .025) and -0.1859572100506226 = (140381.88964461276 - 166486.91418455934) / 140381.88964461276 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "The", Options: NonBacktracking)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 269.14373965358675 > 252.81119025480768.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -17.8257511259632 (T) = (0 -263047.12470040156) / Math.Sqrt((10901377.745342152 / (25)) + (20715833.607267246 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.08957857224984643 = (241420.97816520146 - 263047.12470040156) / 241420.97816520146 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?s).*", Options: NonBacktracking)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 21.390846266666667 > 12.653760510955129.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -29.408494072301586 (T) = (0 -21969351.4441687) / Math.Sqrt((4309104161.758038 / (25)) + (2284619509591.901 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.8273756474998881 = (12022351.000587057 - 21969351.4441687) / 12022351.000587057 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "\p{L}", Options: NonBacktracking)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 59.271682283333334 > 52.54472064947369.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -19.252701802540358 (T) = (0 -60574629.67997309) / Math.Sqrt((5246988632455.257 / (25)) + (3218294459578.9824 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.23998483329436424 = (48851105.314764015 - 60574629.67997309) / 48851105.314764015 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Holmes", Options: NonBacktracking)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 232.71666604477616 > 231.6408227692195.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -35.65556806119108 (T) = (0 -233717.0214530257) / Math.Sqrt((1794347.206209707 / (25)) + (1702926.0782103387 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.06432041301297259 = (219592.72658446795 - 233717.0214530257) / 219592.72658446795 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "[a-q][^u-z]{13}x", Options: Compiled)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 123.34520114829395 > 93.96958350142047.
IsChangePoint: Marked as a change because one of 5/31/2022 5:02:26 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -215.78352088181776 (T) = (0 -123009.64032435244) / Math.Sqrt((94771.9133460141 / (39)) + (131928.7193142191 / (6))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (6) - 2, .025) and -0.3776321281885864 = (89290.62977508713 - 123009.64032435244) / 89290.62977508713 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)the", Options: NonBacktracking)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.464413328712871 > 2.403919837526334.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -16.011236332799633 (T) = (0 -2488036.238172627) / Math.Sqrt((1714946164.887705 / (25)) + (1986602740.6466775 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.09098012790848044 = (2280551.381758387 - 2488036.238172627) / 2280551.381758387 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Street", Options: None)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 211.80437533783785 > 185.6900772735253.
IsChangePoint: Marked as a change because one of 5/31/2022 5:02:26 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -146.412602916086 (T) = (0 -211632.1476028529) / Math.Sqrt((310405.9975140347 / (39)) + (288637.8608624683 / (6))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (6) - 2, .025) and -0.1959026234001499 = (176964.36437370421 - 211632.1476028529) / 176964.36437370421 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Holmes|Watson|Irene|Adler|John|Baker", Options: None)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.3579610954954955 > 3.3373258117721516.
IsChangePoint: Marked as a change because one of 5/23/2022 1:47:22 PM, 5/30/2022 4:33:23 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -4.01558145739983 (T) = (0 -3419948.6696357788) / Math.Sqrt((1361121048.566335 / (35)) + (40171019006.81633 / (10))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (35) + (10) - 2, .025) and -0.08082238832378143 = (3164209.685681739 - 3419948.6696357788) / 3164209.685681739 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "\p{Lu}", Options: NonBacktracking)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.477458443452381 > 3.1315628058333336.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -19.748340139107853 (T) = (0 -3432711.8964723423) / Math.Sqrt((5397452138.060062 / (25)) + (8744618333.829802 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.1723680192829514 = (2928015.640150157 - 3432711.8964723423) / 2928015.640150157 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Street", Options: Compiled)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 128.21670056352457 > 99.13207817865211.
IsChangePoint: Marked as a change because one of 5/31/2022 5:02:26 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -181.94302304862808 (T) = (0 -128889.10613489995) / Math.Sqrt((94724.04877640512 / (39)) + (204466.33468793487 / (6))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (6) - 2, .025) and -0.3693277797697027 = (94125.8243928834 - 128889.10613489995) / 94125.8243928834 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sher[a-z]+|Hol[a-z]+", Options: NonBacktracking)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 422.0465741242937 > 397.00675627836614.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -18.176481339885193 (T) = (0 -423290.6777829517) / Math.Sqrt((14052835.688902048 / (25)) + (123721501.26211147 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.1255541612870132 = (376073.13121115434 - 423290.6777829517) / 376073.13121115434 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "\p{Ll}", Options: NonBacktracking)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 60.93665580263158 > 48.91599193031251.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -19.59316576124937 (T) = (0 -59190504.20009433) / Math.Sqrt((3398672357602.4917 / (25)) + (3952651967907.543 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.2363756179036488 = (47874208.56814977 - 59190504.20009433) / 47874208.56814977 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
| Architecture | arm64 |
|---|---|
| OS | ubuntu 20.04 |
| Baseline | 7f7c6a229b691bd060ee1bf4aeb2cea4c5e52168 |
| Compare | 748dfb78ef87b38599b89e85d3201cdb9f9c0b33 |
| Diff | Diff |
Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| IsMatch - Duration of single invocation | 309.47 ns | 405.39 ns | 1.31 | 0.05 | True | |||||
| IsMatch - Duration of single invocation | 125.32 ns | 150.51 ns | 1.20 | 0.07 | True | |||||
| IsMatch - Duration of single invocation | 202.59 ns | 225.33 ns | 1.11 | 0.12 | False | |||||
| IsMatch - Duration of single invocation | 288.86 ns | 305.95 ns | 1.06 | 0.07 | True | |||||
| IsMatch - Duration of single invocation | 203.85 ns | 233.80 ns | 1.15 | 0.07 | False | |||||
| IsMatch - Duration of single invocation | 161.87 ns | 240.99 ns | 1.49 | 0.30 | True | |||||
| IsMatch - Duration of single invocation | 183.98 ns | 201.75 ns | 1.10 | 0.07 | True | |||||
| IsMatch - Duration of single invocation | 229.50 ns | 258.29 ns | 1.13 | 0.10 | False | |||||
| IsMatch - Duration of single invocation | 194.10 ns | 216.62 ns | 1.12 | 0.07 | False | |||||
| IsMatch - Duration of single invocation | 284.58 ns | 324.36 ns | 1.14 | 0.11 | True |
Repro
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple*'Details
Payloads
Histogram
System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 5, Options: NonBacktracking)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 405.3857795166146 > 320.66215880895163.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -40.418647382839566 (T) = (0 -411.15346009345296) / Math.Sqrt((32.35787412099058 / (25)) + (117.72592224435641 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.3576367213868147 = (302.84497584409996 - 411.15346009345296) / 302.84497584409996 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 0, Options: NonBacktracking)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 150.50710559372936 > 132.35765893001678.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -16.603267025237894 (T) = (0 -156.93849754400017) / Math.Sqrt((23.36308685349803 / (25)) + (45.54719564086743 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.23396226473638665 = (127.18257440192241 - 156.93849754400017) / 127.18257440192241 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 9, Options: NonBacktracking)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 225.3326525326946 > 216.42381707591662.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -5.17397456347185 (T) = (0 -229.7646021412044) / Math.Sqrt((330.28200735519516 / (25)) + (28.338981077832774 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.09424340910274114 = (209.97576976918418 - 229.7646021412044) / 209.97576976918418 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 2, Options: NonBacktracking)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 305.9530747031089 > 303.2243229249164.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -17.965470195742096 (T) = (0 -314.4026584100449) / Math.Sqrt((27.614845224254083 / (25)) + (27.37611344161136 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.0987403306997623 = (286.14828237879385 - 314.4026584100449) / 286.14828237879385 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 6, Options: NonBacktracking)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 233.79921100406875 > 216.0981848457079.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -9.520391677916491 (T) = (0 -235.10568109278287) / Math.Sqrt((17.71853965226845 / (25)) + (172.63511589388332 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.14123850434528068 = (206.0092436397956 - 235.10568109278287) / 206.0092436397956 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 11, Options: NonBacktracking)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 240.9901997942921 > 168.67270178875302.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -37.06145896087716 (T) = (0 -235.74694942321312) / Math.Sqrt((20.544573306682103 / (25)) + (63.292732744651765 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.45747758142672895 = (161.749966124652 - 235.74694942321312) / 161.749966124652 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 7, Options: NonBacktracking)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 201.74684363474523 > 192.63635665023534.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -19.338747961834603 (T) = (0 -208.1492449451767) / Math.Sqrt((16.669059602977708 / (25)) + (22.631321600765006 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.14232369415752283 = (182.21564168699942 - 208.1492449451767) / 182.21564168699942 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 10, Options: NonBacktracking)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 258.2913743473124 > 240.22440418405336.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -7.347499804753521 (T) = (0 -255.74952059699817) / Math.Sqrt((225.26819447156268 / (25)) + (28.97166097012453 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.10243003638235425 = (231.98707596560524 - 255.74952059699817) / 231.98707596560524 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 8, Options: NonBacktracking)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 216.62043721607395 > 210.04413335180266.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -7.4035224375679105 (T) = (0 -217.32059219734487) / Math.Sqrt((181.32903565283027 / (25)) + (20.14139467969238 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.10853866158707445 = (196.04241126440368 - 217.32059219734487) / 196.04241126440368 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 12, Options: NonBacktracking)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 324.3579286835487 > 288.0388752258521.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -21.307866588903774 (T) = (0 -317.91077286602257) / Math.Sqrt((60.57126691214733 / (25)) + (31.936782298424202 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.1552398254605527 = (275.1902815844173 - 317.91077286602257) / 275.1902815844173 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
arch-arm64area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMICLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMIruntime-coreclrspecific to the CoreCLR runtimespecific to the CoreCLR runtime






















Activity
DrewScoggins commentedon Jun 2, 2022
Changes that occurred on May 31st are most likey related to #68874
We are also seeing changes on May 27th in this issue, and this is the range of commits 1f303a1...77520a5
We are not sure which of these commits is the offender.
stephentoub commentedon Jun 2, 2022
Dup of #70160?
tannergooding commentedon Jun 16, 2022
Most of these were resolved in 2d5c8dc
A few were mostly fixed but still have a small regression related to #70182