Repository navigation
[Perf] Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple #70160
Copy link
Copy link
Closed
Labels
arch-arm64area-System.Text.RegularExpressionsruntime-coreclrspecific to the CoreCLR runtimespecific to the CoreCLR runtimetenet-build-performanceImpacts build time: official, developer or CIImpacts build time: official, developer or CItenet-performance-benchmarksIssue from performance benchmarkIssue from performance benchmark
Milestone
Description
Run Information
| Architecture | arm64 |
|---|---|
| OS | Windows 10.0.19041 |
| Baseline | 7f7c6a229b691bd060ee1bf4aeb2cea4c5e52168 |
| Compare | 748dfb78ef87b38599b89e85d3201cdb9f9c0b33 |
| Diff | Diff |
Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| IsMatch - Duration of single invocation | 152.80 ns | 193.82 ns | 1.27 | 0.07 | False | |||||
| IsMatch - Duration of single invocation | 352.15 ns | 420.35 ns | 1.19 | 0.07 | False | |||||
| IsMatch - Duration of single invocation | 257.02 ns | 286.06 ns | 1.11 | 0.04 | False | |||||
| IsMatch - Duration of single invocation | 282.21 ns | 359.00 ns | 1.27 | 0.04 | False | |||||
| IsMatch - Duration of single invocation | 255.00 ns | 284.08 ns | 1.11 | 0.06 | False | |||||
| IsMatch - Duration of single invocation | 181.79 ns | 199.94 ns | 1.10 | 0.05 | False | |||||
| IsMatch - Duration of single invocation | 120.39 ns | 137.77 ns | 1.14 | 0.09 | False | |||||
| IsMatch - Duration of single invocation | 256.94 ns | 416.54 ns | 1.62 | 0.10 | False | |||||
| IsMatch - Duration of single invocation | 171.65 ns | 188.64 ns | 1.10 | 0.08 | False | |||||
| IsMatch - Duration of single invocation | 197.97 ns | 210.91 ns | 1.07 | 0.06 | False |
Repro
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple*'Details
Payloads
Histogram
System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 11, Options: NonBacktracking)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 193.82345578732298 > 160.51493849490748.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 4/19/2022 10:22:09 AM, 5/27/2022 6:33:44 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM.
IsRegressionStdDev: Marked as regression because -56.19568645452529 (T) = (0 -195.70128105261227) / Math.Sqrt((7.133098257454549 / (20)) + (4.724229991607184 / (18))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (18) - 2, .025) and -0.2918902520541079 = (151.48444749191881 - 195.70128105261227) / 151.48444749191881 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 3, Options: NonBacktracking)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 420.3466195942209 > 378.8891954189929.
IsChangePoint: Marked as a change because one of 2/23/2022 1:54:47 PM, 4/19/2022 10:22:09 AM, 5/27/2022 6:33:44 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM.
IsRegressionStdDev: Marked as regression because -30.38567503551837 (T) = (0 -427.89731912615565) / Math.Sqrt((31.25995226318362 / (20)) + (74.78254044265694 / (18))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (18) - 2, .025) and -0.20452782864423077 = (355.2407083925824 - 427.89731912615565) / 355.2407083925824 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 12, Options: NonBacktracking)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 286.06338400034446 > 267.8557193289962.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 4/19/2022 10:22:09 AM, 5/27/2022 6:33:44 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM.
IsRegressionStdDev: Marked as regression because -32.75362966900659 (T) = (0 -284.95062202007716) / Math.Sqrt((9.592799629074031 / (20)) + (6.133357952671183 / (18))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (18) - 2, .025) and -0.11621009500698831 = (255.2840395322649 - 284.95062202007716) / 255.2840395322649 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 5, Options: NonBacktracking)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 359.00123869551953 > 299.03102013628217.
IsChangePoint: Marked as a change because one of 2/23/2022 1:54:47 PM, 4/19/2022 10:22:09 AM, 5/27/2022 6:33:44 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM.
IsRegressionStdDev: Marked as regression because -49.8167233459315 (T) = (0 -359.2430371461935) / Math.Sqrt((23.83646758323746 / (20)) + (24.193796251581627 / (18))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (18) - 2, .025) and -0.283414266495181 = (279.9119867408318 - 359.2430371461935) / 279.9119867408318 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 13, Options: NonBacktracking)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 284.07699367758863 > 268.8931047557917.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 4/19/2022 10:22:09 AM, 5/27/2022 6:33:44 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM.
IsRegressionStdDev: Marked as regression because -14.707737837300268 (T) = (0 -284.6950606504841) / Math.Sqrt((72.20208076439859 / (20)) + (2.936603323654551 / (18))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (18) - 2, .025) and -0.11154517214224496 = (256.12549789749033 - 284.6950606504841) / 256.12549789749033 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 8, Options: NonBacktracking)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 199.93982018770097 > 190.74106671506482.
IsChangePoint: Marked as a change because one of 2/12/2022 8:07:31 PM, 3/18/2022 1:59:24 AM, 4/19/2022 10:22:09 AM, 5/27/2022 6:33:44 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM.
IsRegressionStdDev: Marked as regression because -6.23043356580579 (T) = (0 -204.18577959772213) / Math.Sqrt((10.69732540933686 / (20)) + (277.79032164012966 / (18))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (18) - 2, .025) and -0.13886241058668455 = (179.28924310754616 - 204.18577959772213) / 179.28924310754616 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 0, Options: NonBacktracking)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 137.77216215835156 > 126.05071223495933.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 5/27/2022 6:33:44 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM.
IsRegressionStdDev: Marked as regression because -19.701791772465455 (T) = (0 -139.42641042559248) / Math.Sqrt((9.086366063257891 / (20)) + (10.567192993391133 / (18))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (18) - 2, .025) and -0.16849776097197794 = (119.32107624204177 - 139.42641042559248) / 119.32107624204177 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 4, Options: NonBacktracking)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 416.5392651687832 > 273.34211778773965.
IsChangePoint: Marked as a change because one of 2/26/2022 10:27:55 AM, 4/19/2022 10:22:09 AM, 5/27/2022 6:33:44 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM.
IsRegressionStdDev: Marked as regression because -12.142152509147046 (T) = (0 -335.17522697922493) / Math.Sqrt((316.9666792966112 / (20)) + (427.06779958133274 / (18))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (18) - 2, .025) and -0.29515686537999125 = (258.7912213096184 - 335.17522697922493) / 258.7912213096184 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 7, Options: NonBacktracking)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 188.63875721318746 > 180.42681923567463.
IsChangePoint: Marked as a change because one of 2/22/2022 8:48:40 PM, 3/18/2022 1:59:24 AM, 4/13/2022 4:27:35 PM, 5/27/2022 6:33:44 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM.
IsRegressionStdDev: Marked as regression because -8.727248818384016 (T) = (0 -187.01306789463786) / Math.Sqrt((58.96324847137834 / (20)) + (2.4313152182161533 / (18))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (18) - 2, .025) and -0.08925628932185142 = (171.6887657459094 - 187.01306789463786) / 171.6887657459094 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 6, Options: NonBacktracking)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 210.90900129887 > 200.86860829041314.
IsChangePoint: Marked as a change because one of 2/23/2022 1:54:47 PM, 3/17/2022 6:54:53 PM, 4/19/2022 10:22:09 AM, 5/27/2022 6:33:44 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM.
IsRegressionStdDev: Marked as regression because -22.04937033521675 (T) = (0 -210.57787219141593) / Math.Sqrt((10.666133025825182 / (20)) + (5.760271628275522 / (18))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (18) - 2, .025) and -0.10708286804671203 = (190.2096746948584 - 210.57787219141593) / 190.2096746948584 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
arch-arm64area-System.Text.RegularExpressionsruntime-coreclrspecific to the CoreCLR runtimespecific to the CoreCLR runtimetenet-build-performanceImpacts build time: official, developer or CIImpacts build time: official, developer or CItenet-performance-benchmarksIssue from performance benchmarkIssue from performance benchmark










Activity
[-][Perf] Changes at 5/28/2022 1:41:21 AM[/-][+][Perf] Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple[/+]DrewScoggins commentedon Jun 2, 2022
This is the range of commits where the regression occurred 1f303a1...77520a5
stephentoub commentedon Jun 2, 2022
Is this not a dup of #70160?