Skip to content

[Perf] Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple #70160

Closed

Description

@performanceautofiler

Run Information

Architecture arm64
OS Windows 10.0.19041
Baseline 7f7c6a229b691bd060ee1bf4aeb2cea4c5e52168
Compare 748dfb78ef87b38599b89e85d3201cdb9f9c0b33
Diff Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IsMatch - Duration of single invocation 152.80 ns 193.82 ns 1.27 0.07 False
IsMatch - Duration of single invocation 352.15 ns 420.35 ns 1.19 0.07 False
IsMatch - Duration of single invocation 257.02 ns 286.06 ns 1.11 0.04 False
IsMatch - Duration of single invocation 282.21 ns 359.00 ns 1.27 0.04 False
IsMatch - Duration of single invocation 255.00 ns 284.08 ns 1.11 0.06 False
IsMatch - Duration of single invocation 181.79 ns 199.94 ns 1.10 0.05 False
IsMatch - Duration of single invocation 120.39 ns 137.77 ns 1.14 0.09 False
IsMatch - Duration of single invocation 256.94 ns 416.54 ns 1.62 0.10 False
IsMatch - Duration of single invocation 171.65 ns 188.64 ns 1.10 0.08 False
IsMatch - Duration of single invocation 197.97 ns 210.91 ns 1.07 0.06 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple*'
Details

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 11, Options: NonBacktracking)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 193.82345578732298 > 160.51493849490748.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 4/19/2022 10:22:09 AM, 5/27/2022 6:33:44 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM.
IsRegressionStdDev: Marked as regression because -56.19568645452529 (T) = (0 -195.70128105261227) / Math.Sqrt((7.133098257454549 / (20)) + (4.724229991607184 / (18))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (18) - 2, .025) and -0.2918902520541079 = (151.48444749191881 - 195.70128105261227) / 151.48444749191881 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 3, Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 420.3466195942209 > 378.8891954189929.
IsChangePoint: Marked as a change because one of 2/23/2022 1:54:47 PM, 4/19/2022 10:22:09 AM, 5/27/2022 6:33:44 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM.
IsRegressionStdDev: Marked as regression because -30.38567503551837 (T) = (0 -427.89731912615565) / Math.Sqrt((31.25995226318362 / (20)) + (74.78254044265694 / (18))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (18) - 2, .025) and -0.20452782864423077 = (355.2407083925824 - 427.89731912615565) / 355.2407083925824 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 12, Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 286.06338400034446 > 267.8557193289962.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 4/19/2022 10:22:09 AM, 5/27/2022 6:33:44 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM.
IsRegressionStdDev: Marked as regression because -32.75362966900659 (T) = (0 -284.95062202007716) / Math.Sqrt((9.592799629074031 / (20)) + (6.133357952671183 / (18))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (18) - 2, .025) and -0.11621009500698831 = (255.2840395322649 - 284.95062202007716) / 255.2840395322649 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 5, Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 359.00123869551953 > 299.03102013628217.
IsChangePoint: Marked as a change because one of 2/23/2022 1:54:47 PM, 4/19/2022 10:22:09 AM, 5/27/2022 6:33:44 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM.
IsRegressionStdDev: Marked as regression because -49.8167233459315 (T) = (0 -359.2430371461935) / Math.Sqrt((23.83646758323746 / (20)) + (24.193796251581627 / (18))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (18) - 2, .025) and -0.283414266495181 = (279.9119867408318 - 359.2430371461935) / 279.9119867408318 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 13, Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 284.07699367758863 > 268.8931047557917.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 4/19/2022 10:22:09 AM, 5/27/2022 6:33:44 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM.
IsRegressionStdDev: Marked as regression because -14.707737837300268 (T) = (0 -284.6950606504841) / Math.Sqrt((72.20208076439859 / (20)) + (2.936603323654551 / (18))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (18) - 2, .025) and -0.11154517214224496 = (256.12549789749033 - 284.6950606504841) / 256.12549789749033 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 8, Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 199.93982018770097 > 190.74106671506482.
IsChangePoint: Marked as a change because one of 2/12/2022 8:07:31 PM, 3/18/2022 1:59:24 AM, 4/19/2022 10:22:09 AM, 5/27/2022 6:33:44 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM.
IsRegressionStdDev: Marked as regression because -6.23043356580579 (T) = (0 -204.18577959772213) / Math.Sqrt((10.69732540933686 / (20)) + (277.79032164012966 / (18))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (18) - 2, .025) and -0.13886241058668455 = (179.28924310754616 - 204.18577959772213) / 179.28924310754616 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 0, Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 137.77216215835156 > 126.05071223495933.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 5/27/2022 6:33:44 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM.
IsRegressionStdDev: Marked as regression because -19.701791772465455 (T) = (0 -139.42641042559248) / Math.Sqrt((9.086366063257891 / (20)) + (10.567192993391133 / (18))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (18) - 2, .025) and -0.16849776097197794 = (119.32107624204177 - 139.42641042559248) / 119.32107624204177 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 4, Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 416.5392651687832 > 273.34211778773965.
IsChangePoint: Marked as a change because one of 2/26/2022 10:27:55 AM, 4/19/2022 10:22:09 AM, 5/27/2022 6:33:44 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM.
IsRegressionStdDev: Marked as regression because -12.142152509147046 (T) = (0 -335.17522697922493) / Math.Sqrt((316.9666792966112 / (20)) + (427.06779958133274 / (18))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (18) - 2, .025) and -0.29515686537999125 = (258.7912213096184 - 335.17522697922493) / 258.7912213096184 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 7, Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 188.63875721318746 > 180.42681923567463.
IsChangePoint: Marked as a change because one of 2/22/2022 8:48:40 PM, 3/18/2022 1:59:24 AM, 4/13/2022 4:27:35 PM, 5/27/2022 6:33:44 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM.
IsRegressionStdDev: Marked as regression because -8.727248818384016 (T) = (0 -187.01306789463786) / Math.Sqrt((58.96324847137834 / (20)) + (2.4313152182161533 / (18))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (18) - 2, .025) and -0.08925628932185142 = (171.6887657459094 - 187.01306789463786) / 171.6887657459094 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 6, Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 210.90900129887 > 200.86860829041314.
IsChangePoint: Marked as a change because one of 2/23/2022 1:54:47 PM, 3/17/2022 6:54:53 PM, 4/19/2022 10:22:09 AM, 5/27/2022 6:33:44 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM.
IsRegressionStdDev: Marked as regression because -22.04937033521675 (T) = (0 -210.57787219141593) / Math.Sqrt((10.666133025825182 / (20)) + (5.760271628275522 / (18))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (18) - 2, .025) and -0.10708286804671203 = (190.2096746948584 - 210.57787219141593) / 190.2096746948584 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Activity

  1. changed the title [-][Perf] Changes at 5/28/2022 1:41:21 AM[/-] [+][Perf] Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple[/+] on Jun 2, 2022
  2. DrewScoggins commented on Jun 2, 2022

    @DrewScoggins
    Member

    This is the range of commits where the regression occurred 1f303a1...77520a5

  3. stephentoub commented on Jun 2, 2022

    @stephentoub
    Member

    Is this not a dup of #70160?

  4. added this to the 7.0.0 milestone on Jun 15, 2022
  5. ghost removed
    untriagedNew issue has not been triaged by the area owner
    on Jun 15, 2022
  6. ghost locked as resolved and limited conversation to collaborators on Aug 11, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions