Skip to content

[Perf] Changes at 5/28/2022 1:41:21 AM #70162

Closed

Description

@performanceautofiler

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 7f7c6a229b691bd060ee1bf4aeb2cea4c5e52168
Compare 748dfb78ef87b38599b89e85d3201cdb9f9c0b33
Diff Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Count - Duration of single invocation 2.20 ms 2.98 ms 1.36 0.05 True
Count - Duration of single invocation 153.31 μs 191.44 μs 1.25 0.01 True
Count - Duration of single invocation 1.24 ms 1.37 ms 1.10 0.10 True
Count - Duration of single invocation 24.70 ms 34.02 ms 1.38 0.10 True
Count - Duration of single invocation 144.68 μs 165.27 μs 1.14 0.01 False
Count - Duration of single invocation 240.75 μs 269.14 μs 1.12 0.01 True
Count - Duration of single invocation 12.03 ms 21.39 ms 1.78 0.05 True
Count - Duration of single invocation 47.29 ms 59.27 ms 1.25 0.44 True
Count - Duration of single invocation 219.57 μs 232.72 μs 1.06 0.01 True
Count - Duration of single invocation 89.40 μs 123.35 μs 1.38 0.01 True
Count - Duration of single invocation 2.33 ms 2.46 ms 1.06 0.02 True
Count - Duration of single invocation 177.12 μs 211.80 μs 1.20 0.01 True
Count - Duration of single invocation 3.18 ms 3.36 ms 1.06 0.02 True
Count - Duration of single invocation 3.08 ms 3.48 ms 1.13 0.13 True
Count - Duration of single invocation 94.23 μs 128.22 μs 1.36 0.01 True
Count - Duration of single invocation 374.51 μs 422.05 μs 1.13 0.01 True
Count - Duration of single invocation 49.23 ms 60.94 ms 1.24 0.48 True

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock*'
Details

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "the\s+\w+", Options: NonBacktracking)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.981663075502008 > 2.30941796713219.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -28.080765181910134 (T) = (0 -2748213.418441032) / Math.Sqrt((563613961.9919467 / (25)) + (7259741238.014956 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.2509798087203997 = (2196848.7415093617 - 2748213.418441032) / 2196848.7415093617 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "[a-q][^u-z]{13}x", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 191.43783472206997 > 160.95435806566604.
IsChangePoint: Marked as a change because one of 5/31/2022 5:02:26 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -77.0805759253954 (T) = (0 -190973.89435431754) / Math.Sqrt((6394852.873722199 / (39)) + (317440.2113534399 / (6))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (6) - 2, .025) and -0.23147466580807602 = (155077.40407229832 - 190973.89435431754) / 155077.40407229832 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "the", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.3717235683241253 > 1.2718615059933995.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -18.20562778818909 (T) = (0 -1386204.0048313157) / Math.Sqrt((627918840.8777322 / (25)) + (1547500831.444926 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.15335016285121025 = (1201893.4487376017 - 1386204.0048313157) / 1201893.4487376017 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "\w+", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 34.020709333333336 > 26.112056162500004.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -23.71893120000806 (T) = (0 -33687196.72236111) / Math.Sqrt((108679716720.46556 / (25)) + (2882756164569.5947 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.3723324199439214 = (24547402.825138893 - 33687196.72236111) / 24547402.825138893 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Street", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 165.2708110011001 > 151.98199722719656.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 5/31/2022 5:02:26 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -24.825963965192063 (T) = (0 -166486.91418455934) / Math.Sqrt((34070055.2268123 / (39)) + (1392632.58256889 / (6))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (6) - 2, .025) and -0.1859572100506226 = (140381.88964461276 - 166486.91418455934) / 140381.88964461276 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "The", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 269.14373965358675 > 252.81119025480768.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -17.8257511259632 (T) = (0 -263047.12470040156) / Math.Sqrt((10901377.745342152 / (25)) + (20715833.607267246 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.08957857224984643 = (241420.97816520146 - 263047.12470040156) / 241420.97816520146 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?s).*", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 21.390846266666667 > 12.653760510955129.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -29.408494072301586 (T) = (0 -21969351.4441687) / Math.Sqrt((4309104161.758038 / (25)) + (2284619509591.901 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.8273756474998881 = (12022351.000587057 - 21969351.4441687) / 12022351.000587057 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "\p{L}", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 59.271682283333334 > 52.54472064947369.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -19.252701802540358 (T) = (0 -60574629.67997309) / Math.Sqrt((5246988632455.257 / (25)) + (3218294459578.9824 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.23998483329436424 = (48851105.314764015 - 60574629.67997309) / 48851105.314764015 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Holmes", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 232.71666604477616 > 231.6408227692195.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -35.65556806119108 (T) = (0 -233717.0214530257) / Math.Sqrt((1794347.206209707 / (25)) + (1702926.0782103387 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.06432041301297259 = (219592.72658446795 - 233717.0214530257) / 219592.72658446795 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "[a-q][^u-z]{13}x", Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 123.34520114829395 > 93.96958350142047.
IsChangePoint: Marked as a change because one of 5/31/2022 5:02:26 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -215.78352088181776 (T) = (0 -123009.64032435244) / Math.Sqrt((94771.9133460141 / (39)) + (131928.7193142191 / (6))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (6) - 2, .025) and -0.3776321281885864 = (89290.62977508713 - 123009.64032435244) / 89290.62977508713 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)the", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.464413328712871 > 2.403919837526334.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -16.011236332799633 (T) = (0 -2488036.238172627) / Math.Sqrt((1714946164.887705 / (25)) + (1986602740.6466775 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.09098012790848044 = (2280551.381758387 - 2488036.238172627) / 2280551.381758387 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Street", Options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 211.80437533783785 > 185.6900772735253.
IsChangePoint: Marked as a change because one of 5/31/2022 5:02:26 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -146.412602916086 (T) = (0 -211632.1476028529) / Math.Sqrt((310405.9975140347 / (39)) + (288637.8608624683 / (6))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (6) - 2, .025) and -0.1959026234001499 = (176964.36437370421 - 211632.1476028529) / 176964.36437370421 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Holmes|Watson|Irene|Adler|John|Baker", Options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.3579610954954955 > 3.3373258117721516.
IsChangePoint: Marked as a change because one of 5/23/2022 1:47:22 PM, 5/30/2022 4:33:23 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -4.01558145739983 (T) = (0 -3419948.6696357788) / Math.Sqrt((1361121048.566335 / (35)) + (40171019006.81633 / (10))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (35) + (10) - 2, .025) and -0.08082238832378143 = (3164209.685681739 - 3419948.6696357788) / 3164209.685681739 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "\p{Lu}", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.477458443452381 > 3.1315628058333336.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -19.748340139107853 (T) = (0 -3432711.8964723423) / Math.Sqrt((5397452138.060062 / (25)) + (8744618333.829802 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.1723680192829514 = (2928015.640150157 - 3432711.8964723423) / 2928015.640150157 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Street", Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 128.21670056352457 > 99.13207817865211.
IsChangePoint: Marked as a change because one of 5/31/2022 5:02:26 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -181.94302304862808 (T) = (0 -128889.10613489995) / Math.Sqrt((94724.04877640512 / (39)) + (204466.33468793487 / (6))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (6) - 2, .025) and -0.3693277797697027 = (94125.8243928834 - 128889.10613489995) / 94125.8243928834 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sher[a-z]+|Hol[a-z]+", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 422.0465741242937 > 397.00675627836614.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -18.176481339885193 (T) = (0 -423290.6777829517) / Math.Sqrt((14052835.688902048 / (25)) + (123721501.26211147 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.1255541612870132 = (376073.13121115434 - 423290.6777829517) / 376073.13121115434 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "\p{Ll}", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 60.93665580263158 > 48.91599193031251.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -19.59316576124937 (T) = (0 -59190504.20009433) / Math.Sqrt((3398672357602.4917 / (25)) + (3952651967907.543 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.2363756179036488 = (47874208.56814977 - 59190504.20009433) / 47874208.56814977 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture arm64
OS ubuntu 20.04
Baseline 7f7c6a229b691bd060ee1bf4aeb2cea4c5e52168
Compare 748dfb78ef87b38599b89e85d3201cdb9f9c0b33
Diff Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IsMatch - Duration of single invocation 309.47 ns 405.39 ns 1.31 0.05 True
IsMatch - Duration of single invocation 125.32 ns 150.51 ns 1.20 0.07 True
IsMatch - Duration of single invocation 202.59 ns 225.33 ns 1.11 0.12 False
IsMatch - Duration of single invocation 288.86 ns 305.95 ns 1.06 0.07 True
IsMatch - Duration of single invocation 203.85 ns 233.80 ns 1.15 0.07 False
IsMatch - Duration of single invocation 161.87 ns 240.99 ns 1.49 0.30 True
IsMatch - Duration of single invocation 183.98 ns 201.75 ns 1.10 0.07 True
IsMatch - Duration of single invocation 229.50 ns 258.29 ns 1.13 0.10 False
IsMatch - Duration of single invocation 194.10 ns 216.62 ns 1.12 0.07 False
IsMatch - Duration of single invocation 284.58 ns 324.36 ns 1.14 0.11 True

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple*'
Details

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 5, Options: NonBacktracking)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 405.3857795166146 > 320.66215880895163.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -40.418647382839566 (T) = (0 -411.15346009345296) / Math.Sqrt((32.35787412099058 / (25)) + (117.72592224435641 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.3576367213868147 = (302.84497584409996 - 411.15346009345296) / 302.84497584409996 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 0, Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 150.50710559372936 > 132.35765893001678.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -16.603267025237894 (T) = (0 -156.93849754400017) / Math.Sqrt((23.36308685349803 / (25)) + (45.54719564086743 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.23396226473638665 = (127.18257440192241 - 156.93849754400017) / 127.18257440192241 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 9, Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 225.3326525326946 > 216.42381707591662.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -5.17397456347185 (T) = (0 -229.7646021412044) / Math.Sqrt((330.28200735519516 / (25)) + (28.338981077832774 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.09424340910274114 = (209.97576976918418 - 229.7646021412044) / 209.97576976918418 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 2, Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 305.9530747031089 > 303.2243229249164.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -17.965470195742096 (T) = (0 -314.4026584100449) / Math.Sqrt((27.614845224254083 / (25)) + (27.37611344161136 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.0987403306997623 = (286.14828237879385 - 314.4026584100449) / 286.14828237879385 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 6, Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 233.79921100406875 > 216.0981848457079.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -9.520391677916491 (T) = (0 -235.10568109278287) / Math.Sqrt((17.71853965226845 / (25)) + (172.63511589388332 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.14123850434528068 = (206.0092436397956 - 235.10568109278287) / 206.0092436397956 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 11, Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 240.9901997942921 > 168.67270178875302.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -37.06145896087716 (T) = (0 -235.74694942321312) / Math.Sqrt((20.544573306682103 / (25)) + (63.292732744651765 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.45747758142672895 = (161.749966124652 - 235.74694942321312) / 161.749966124652 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 7, Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 201.74684363474523 > 192.63635665023534.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -19.338747961834603 (T) = (0 -208.1492449451767) / Math.Sqrt((16.669059602977708 / (25)) + (22.631321600765006 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.14232369415752283 = (182.21564168699942 - 208.1492449451767) / 182.21564168699942 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 10, Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 258.2913743473124 > 240.22440418405336.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -7.347499804753521 (T) = (0 -255.74952059699817) / Math.Sqrt((225.26819447156268 / (25)) + (28.97166097012453 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.10243003638235425 = (231.98707596560524 - 255.74952059699817) / 231.98707596560524 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 8, Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 216.62043721607395 > 210.04413335180266.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -7.4035224375679105 (T) = (0 -217.32059219734487) / Math.Sqrt((181.32903565283027 / (25)) + (20.14139467969238 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.10853866158707445 = (196.04241126440368 - 217.32059219734487) / 196.04241126440368 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 12, Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 324.3579286835487 > 288.0388752258521.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -21.307866588903774 (T) = (0 -317.91077286602257) / Math.Sqrt((60.57126691214733 / (25)) + (31.936782298424202 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.1552398254605527 = (275.1902815844173 - 317.91077286602257) / 275.1902815844173 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Activity

  1. DrewScoggins commented on Jun 2, 2022

    @DrewScoggins
    Member

    Changes that occurred on May 31st are most likey related to #68874

    We are also seeing changes on May 27th in this issue, and this is the range of commits 1f303a1...77520a5

    We are not sure which of these commits is the offender.

  2. stephentoub commented on Jun 2, 2022

    @stephentoub
    Member

    Dup of #70160?

  3. added
    area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI
    on Jun 3, 2022
  4. removed
    untriagedNew issue has not been triaged by the area owner
    on Jun 3, 2022
  5. added this to the 7.0.0 milestone on Jun 3, 2022
  6. tannergooding commented on Jun 16, 2022

    @tannergooding
    Member

    Most of these were resolved in 2d5c8dc

    A few were mostly fixed but still have a small regression related to #70182

  7. ghost locked as resolved and limited conversation to collaborators on Jul 17, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

arch-arm64area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMIruntime-coreclrspecific to the CoreCLR runtime

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions