Skip to content

Regressions in System.Text.RegularExpressions #77910

Description

@performanceautofiler

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline fb36ccef4bab6bf69d890c30978daccdb62b5839
Compare e1ddbca5a008b4b15e8e6b309cbab12ba8b9b842
Diff Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Count - Duration of single invocation 332.03 ms 416.51 ms 1.25 0.01 False
Count - Duration of single invocation 7.07 ms 7.66 ms 1.08 0.00 False
Count - Duration of single invocation 339.16 ms 422.09 ms 1.24 0.00 False
Count - Duration of single invocation 84.12 ms 88.83 ms 1.06 0.05 False

graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig*'

Related Issues

Regressions

Improvements

Details

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: ".{0,2}(Tom|Sawyer|Huckleberry|Finn)", Options: Compiled)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 416.5054333333333 > 348.4545274038462.
IsChangePoint: Marked as a change because one of 10/27/2022 12:34:03 AM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -579.2051150485486 (T) = (0 -416808462.26582944) / Math.Sqrt((29667906565.438564 / (30)) + (431109292014.0141 / (21))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (21) - 2, .025) and -0.2560340022183109 = (331844887.5824176 - 416808462.26582944) / 331844887.5824176 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "(?i)Twain", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 7.6602848484848485 > 7.422380485714287.
IsChangePoint: Marked as a change because one of 8/17/2022 2:11:12 PM, 10/11/2022 8:42:28 PM, 10/25/2022 2:34:53 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -18.390301801937913 (T) = (0 -7601105.116898116) / Math.Sqrt((7020278732.863089 / (21)) + (10510234808.502586 / (30))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (30) - 2, .025) and -0.06758424314030985 = (7119911.300432262 - 7601105.116898116) / 7119911.300432262 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: ".{2,4}(Tom|Sawyer|Huckleberry|Finn)", Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 422.08876923076923 > 356.0478467307693.
IsChangePoint: Marked as a change because one of 10/27/2022 12:34:03 AM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -432.2606998714831 (T) = (0 -422918163.37345195) / Math.Sqrt((16255806192.92972 / (30)) + (778394737481.3641 / (21))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (21) - 2, .025) and -0.24721332168943 = (339090479.56654465 - 422918163.37345195) / 339090479.56654465 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "(?i)Tom|Sawyer|Huckleberry|Finn", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 88.82956166666668 > 88.30194718749999.
IsChangePoint: Marked as a change because one of 8/16/2022 10:54:40 PM, 9/3/2022 11:44:47 AM, 9/9/2022 3:12:19 AM, 10/25/2022 7:09:53 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -19.969261603093507 (T) = (0 -88999508.19202349) / Math.Sqrt((581619626110.0212 / (22)) + (893915401380.7828 / (29))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (29) - 2, .025) and -0.05673809283385816 = (84220970.92511655 - 88999508.19202349) / 84220970.92511655 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS Windows 10.0.19042
Baseline fb36ccef4bab6bf69d890c30978daccdb62b5839
Compare e1ddbca5a008b4b15e8e6b309cbab12ba8b9b842
Diff Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Common

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
OneNodeBacktracking - Duration of single invocation 73.39 ns 86.83 ns 1.18 0.08 False
Backtracking - Duration of single invocation 123.89 ns 138.00 ns 1.11 0.10 False
Backtracking - Duration of single invocation 107.89 ns 116.85 ns 1.08 0.09 False
IP_IsMatch - Duration of single invocation 105.58 ns 128.33 ns 1.22 0.06 False
Uri_IsMatch - Duration of single invocation 288.43 ns 311.47 ns 1.08 0.13 False
MatchesWord - Duration of single invocation 1.15 μs 1.31 μs 1.14 0.09 False
IP_IsMatch - Duration of single invocation 623.64 ns 688.23 ns 1.10 0.04 False
OneNodeBacktracking - Duration of single invocation 72.12 ns 81.15 ns 1.13 0.12 False
MatchesBoundary - Duration of single invocation 119.32 μs 141.46 μs 1.19 0.15 False
IP_IsNotMatch - Duration of single invocation 105.63 ns 127.01 ns 1.20 0.07 False
Uri_IsNotMatch - Duration of single invocation 290.86 ns 326.04 ns 1.12 0.24 False
IP_IsNotMatch - Duration of single invocation 617.02 ns 688.38 ns 1.12 0.03 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Common*'

Related Issues

Regressions

Improvements

Details

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Text.RegularExpressions.Tests.Perf_Regex_Common.OneNodeBacktracking(Options: Compiled)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 86.8275696278143 > 78.09999538957418.
IsChangePoint: Marked as a change because one of 8/17/2022 2:11:12 PM, 10/27/2022 12:34:03 AM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -21.57518390726645 (T) = (0 -86.48821966813284) / Math.Sqrt((2.7830911020001428 / (30)) + (3.528920341681947 / (21))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (21) - 2, .025) and -0.14599783051026738 = (75.46979354195031 - 86.48821966813284) / 75.46979354195031 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Backtracking(Options: IgnoreCase, Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 138.00026131064894 > 130.43276501654353.
IsChangePoint: Marked as a change because one of 8/17/2022 11:40:52 AM, 9/13/2022 12:23:36 PM, 9/22/2022 3:16:12 AM, 10/1/2022 2:31:50 PM, 10/11/2022 2:16:14 AM, 10/26/2022 2:14:52 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -12.988746098933982 (T) = (0 -137.91943666639156) / Math.Sqrt((4.155644390535771 / (27)) + (28.28187700971305 / (24))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (24) - 2, .025) and -0.1219621062533704 = (122.92700074065202 - 137.91943666639156) / 122.92700074065202 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Backtracking(Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 116.85494054206478 > 113.17838674501087.
IsChangePoint: Marked as a change because one of 8/30/2022 3:47:49 PM, 9/9/2022 3:12:19 AM, 9/23/2022 1:16:35 AM, 10/26/2022 2:14:52 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -7.830654846009884 (T) = (0 -113.71766160363575) / Math.Sqrt((0.9376591043423428 / (27)) + (17.022240467626517 / (24))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (24) - 2, .025) and -0.06314603557067641 = (106.96335009385074 - 113.71766160363575) / 106.96335009385074 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.IP_IsMatch(Options: IgnoreCase, Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 128.32724241101164 > 111.00532349382794.
IsChangePoint: Marked as a change because one of 10/27/2022 12:34:03 AM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -12.878304122450157 (T) = (0 -130.65359927667345) / Math.Sqrt((36.98744598474446 / (30)) + (45.387524679702196 / (21))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (21) - 2, .025) and -0.22189126668465434 = (106.92735338977795 - 130.65359927667345) / 106.92735338977795 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsMatch(Options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 311.4705102917103 > 304.4809632118888.
IsChangePoint: Marked as a change because one of 10/25/2022 2:34:53 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -12.838441152666068 (T) = (0 -318.5006447564052) / Math.Sqrt((31.157663241228725 / (21)) + (100.31298089502344 / (30))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (30) - 2, .025) and -0.09717079409603198 = (290.2926750058276 - 318.5006447564052) / 290.2926750058276 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesWord(Options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 1.3148710887082182 > 1.2113325437083777.
IsChangePoint: Marked as a change because one of 9/9/2022 3:12:19 AM, 9/13/2022 12:23:36 PM, 9/23/2022 1:16:35 AM, 9/28/2022 11:19:50 AM, 10/26/2022 4:56:52 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -9.42682913087762 (T) = (0 -1236.5753605706625) / Math.Sqrt((883.969054626842 / (28)) + (1130.592221572581 / (23))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (28) + (23) - 2, .025) and -0.07353049916484425 = (1151.8772513055376 - 1236.5753605706625) / 1151.8772513055376 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.IP_IsMatch(Options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 688.2267373119328 > 649.9391753240619.
IsChangePoint: Marked as a change because one of 8/30/2022 3:47:49 PM, 10/26/2022 7:11:30 AM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -29.20932783638292 (T) = (0 -707.8359076951109) / Math.Sqrt((75.79268832683788 / (25)) + (136.36896277319332 / (26))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (26) - 2, .025) and -0.13471062546622675 = (623.8030135694528 - 707.8359076951109) / 623.8030135694528 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.OneNodeBacktracking(Options: IgnoreCase, Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 81.15480073905125 > 76.024892831368.
IsChangePoint: Marked as a change because one of 8/17/2022 2:11:12 PM, 9/20/2022 6:55:31 AM, 9/29/2022 7:42:19 AM, 10/28/2022 6:19:52 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -11.456544689879665 (T) = (0 -81.29956025284886) / Math.Sqrt((6.233183332053246 / (40)) + (1.2138580131830443 / (11))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (11) - 2, .025) and -0.07840335594375587 = (75.38882349053914 - 81.29956025284886) / 75.38882349053914 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesBoundary(Options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 141.45801715176714 > 123.90950348505656.
IsChangePoint: Marked as a change because one of 10/12/2022 1:44:25 PM, 10/27/2022 5:53:52 AM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -4.621045369855946 (T) = (0 -136293.58076672902) / Math.Sqrt((15406308.356509058 / (32)) + (169005352.30472982 / (19))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (19) - 2, .025) and -0.11584827198857087 = (122143.47074610608 - 136293.58076672902) / 122143.47074610608 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.IP_IsNotMatch(Options: IgnoreCase, Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 127.0125895748806 > 110.91068039684593.
IsChangePoint: Marked as a change because one of 10/27/2022 12:34:03 AM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -16.919774417071256 (T) = (0 -131.36875012958367) / Math.Sqrt((2.499156880497458 / (30)) + (47.21611114095977 / (21))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (21) - 2, .025) and -0.2448183906845512 = (105.53246249626928 - 131.36875012958367) / 105.53246249626928 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsNotMatch(Options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 326.040303488356 > 305.709328866613.
IsChangePoint: Marked as a change because one of 8/23/2022 11:46:46 PM, 10/25/2022 2:34:53 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -14.36289221847867 (T) = (0 -320.0236477338281) / Math.Sqrt((59.920830731501 / (21)) + (16.947517034333945 / (30))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (30) - 2, .025) and -0.09048668039739656 = (293.46864430953406 - 320.0236477338281) / 293.46864430953406 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.IP_IsNotMatch(Options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 688.3812940853878 > 650.146447853521.
IsChangePoint: Marked as a change because one of 8/30/2022 3:47:49 PM, 10/26/2022 7:11:30 AM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -24.4078691693055 (T) = (0 -707.7392342546108) / Math.Sqrt((147.5822436663469 / (25)) + (149.2207036926858 / (26))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (26) - 2, .025) and -0.13336796992054428 = (624.4567104752638 - 707.7392342546108) / 624.4567104752638 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline fb36ccef4bab6bf69d890c30978daccdb62b5839
Compare e1ddbca5a008b4b15e8e6b309cbab12ba8b9b842
Diff Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Cache

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IsMatch - Duration of single invocation 41.58 ms 46.34 ms 1.11 0.37 False
IsMatch - Duration of single invocation 53.54 ms 59.27 ms 1.11 0.30 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Cache*'

Related Issues

Regressions

Improvements

Details

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Text.RegularExpressions.Tests.Perf_Regex_Cache.IsMatch(total: 400000, unique: 1, cacheSize: 15)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 46.342455 > 43.24550806985295.
IsChangePoint: Marked as a change because one of 9/1/2022 9:35:26 AM, 9/23/2022 1:16:35 AM, 10/27/2022 12:34:03 AM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -10.651290823344734 (T) = (0 -48154098.06689343) / Math.Sqrt((3565666914879.7876 / (30)) + (4421094351846.123 / (21))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (21) - 2, .025) and -0.14540495601817602 = (42041112.022331156 - 48154098.06689343) / 42041112.022331156 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Cache.IsMatch(total: 400000, unique: 7, cacheSize: 15)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 59.273963333333334 > 57.1792501875.
IsChangePoint: Marked as a change because one of 8/26/2022 9:04:17 AM, 10/26/2022 2:14:52 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -7.40053137352866 (T) = (0 -59191082.38388941) / Math.Sqrt((4451398538425.818 / (27)) + (6127025631905.805 / (24))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (24) - 2, .025) and -0.0881897846077196 = (54394080.16978136 - 59191082.38388941) / 54394080.16978136 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions