Repository navigation
[Perf] Regressions in Dictionary and Hashset on 9/24/2022 1:05:07 PM #76256
Description
Run Information
| Architecture | x64 |
|---|---|
| OS | ubuntu 18.04 |
| Baseline | 05aa1fdf72daa876e482f3a762d72d40e82c50b8 |
| Compare | 3922b81fc9c408639c0a090eebcd65f3c0db96ad |
| Diff | Diff |
Regressions in System.Collections.TryGetValueFalse<Int32, Int32>
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| IDictionary - Duration of single invocation | 3.94 μs | 6.19 μs | 1.57 | 0.21 | False | |||||
| Dictionary - Duration of single invocation | 3.91 μs | 5.31 μs | 1.36 | 0.10 | False |
_2.png)
_3.png)
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.TryGetValueFalse<Int32, Int32>*'Details
Payloads
Histogram
Edge Detector Info
System.Collections.TryGetValueFalse<Int32, Int32>.ConcurrentDictionary(Size: 512)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.5190243849206353 > 2.4036435081910237.
IsChangePoint: Marked as a change because one of 8/6/2022 4:25:34 AM, 8/15/2022 9:13:40 PM, 9/13/2022 12:23:36 PM, 9/24/2022 7:43:48 AM, 9/27/2022 3:32:36 AM falls between 9/18/2022 2:31:46 PM and 9/27/2022 3:32:36 AM.
IsRegressionStdDev: Marked as regression because -18.96761416807978 (T) = (0 -2531.2749920005836) / Math.Sqrt((4762.9302636739385 / (45)) + (102.93410430935235 / (13))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (45) + (13) - 2, .025) and -0.0868651462502898 = (2328.968778448312 - 2531.2749920005836) / 2328.968778448312 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Collections.TryGetValueFalse<Int32, Int32>.IDictionary(Size: 512)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.189232560069029 > 4.154752209147254.
IsChangePoint: Marked as a change because one of 9/23/2022 1:16:35 AM, 9/27/2022 3:32:36 AM falls between 9/18/2022 2:31:46 PM and 9/27/2022 3:32:36 AM.
IsRegressionStdDev: Marked as regression because -14.445665294315434 (T) = (0 -5399.457485776981) / Math.Sqrt((7800.283818254304 / (38)) + (188851.75236255076 / (20))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (20) - 2, .025) and -0.3564589621258156 = (3980.553512149795 - 5399.457485776981) / 3980.553512149795 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Collections.TryGetValueFalse<Int32, Int32>.Dictionary(Size: 512)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.312845970588235 > 4.10624376084124.
IsChangePoint: Marked as a change because one of 8/23/2022 11:46:46 PM, 9/1/2022 8:04:48 PM, 9/13/2022 12:23:36 PM, 9/24/2022 1:05:07 PM, 9/27/2022 3:32:36 AM falls between 9/18/2022 2:31:46 PM and 9/27/2022 3:32:36 AM.
IsRegressionStdDev: Marked as regression because -21.186422586815983 (T) = (0 -5116.308816871219) / Math.Sqrt((36197.789434224804 / (46)) + (24833.31452200058 / (12))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (12) - 2, .025) and -0.28421284578723527 = (3984.003768265432 - 5116.308816871219) / 3984.003768265432 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Activity
41 remaining items
stephentoub commentedon Apr 27, 2023
Remaining regression went away. Looks like there might be a new one, but that'll be covered by a different issue.
DrewScoggins commentedon Aug 24, 2023
These two tests, at least, were also regressed by this change. However, they did not improve after updated PGO data was pushed through.
https://pvscmdupload.blob.core.windows.net/reports/allTestHistory%2frefs%2fheads%2fmain_x64_ubuntu%2018.04%2fSystem.Collections.CtorDefaultSize(String).HashSet.html
https://pvscmdupload.blob.core.windows.net/reports/allTestHistory%2frefs%2fheads%2fmain_x64_ubuntu%2018.04%2fSystem.Collections.CtorDefaultSize(String).Dictionary.html
https://pvscmdupload.blob.core.windows.net/reports/allTestHistory%2frefs%2fheads%2fmain_x86_Windows%2010.0.18362%2fSystem.Collections.AddGivenSize(Int32).Dictionary(Size%3a%20512).html
ericstj commentedon Oct 10, 2023
@stephentoub can you take another look and see if there is anything we still need to do here to address a potential regression?
stephentoub commentedon Dec 11, 2023
I don't think there's anything further actionable here. Thanks.