Repository navigation
Regressions from "make SIP not on by default" #73784
Description
Run Information
| Architecture | arm64 |
|---|---|
| OS | ubuntu 20.04 |
| Baseline | 7f317997e9d0baeb21ce7a1889d3330a13ffd0a1 |
| Compare | 6274ea7365b7124d4451aab3e73712763e71183b |
| Diff | Diff |
Regressions in System.IO.Tests.StreamReaderReadToEndTests
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| ReadToEndAsync - Duration of single invocation | 67.94 ms | 75.94 ms | 1.12 | 0.08 | False |
Repro
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.IO.Tests.StreamReaderReadToEndTests*'Details
Payloads
Histogram
System.IO.Tests.StreamReaderReadToEndTests.ReadToEndAsync(LineLengthRange: [ 0, 1024])
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 75.93536301666667 > 70.84351980125.
IsChangePoint: Marked as a change because one of 6/22/2022 5:33:38 PM, 8/6/2022 1:07:13 AM, 8/11/2022 10:02:44 AM falls between 8/2/2022 9:32:02 PM and 8/11/2022 10:02:44 AM.
IsRegressionStdDev: Marked as regression because -24.83915370249787 (T) = (0 -76684335.4223952) / Math.Sqrt((1825236129188.6467 / (23)) + (2631504391413.2695 / (27))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (27) - 2, .025) and -0.15768388595046612 = (66239442.695046976 - 76684335.4223952) / 66239442.695046976 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Activity
[-][Perf] Regression on 8/6/2022 4:25:34 AM[/-][+]Regressions from "make SIP not on by default"[/+]13 remaining items
mangod9 commentedon Aug 12, 2022
from the chart this appears to be bimodel behavior from the test? Would be surprised in disabling SIP has caused this regression?
kunalspathak commentedon Aug 12, 2022
We see the improvements from #71029 and the spike comes from one of 638a4c1...667c170 so definitely it is sensitive to GC.
mangod9 commentedon Aug 12, 2022
Hmm interesting. Ok we can look if reverting the SIP change makes things better.
Maoni0 commentedon Aug 15, 2022
you don't need to revert the change, you can set the GCEnableSpecialRegions env var to turn it on as I mentioned in the PR -
set COMPlus_GCEnableSpecialRegions=1
mrsharm commentedon Aug 15, 2022
I was able to repro this issue locally and proved that after we set COMPLUS_GCEnableSpecialRegions=1, the regression dissapears:
Without COMPlus_GCEnableSpecialRegions=1:
With COMPlus_GCEnableSpecialRegions=1:
@kunalspathak: Should we close this issue as this issue is by design?
Maoni0 commentedon Aug 15, 2022
so when regions was enabled did we see the same degree of improvement with these tests? if so it would be fine to close.
mrsharm commentedon Aug 15, 2022
Yes, comparing net6.0 (w/o regions enabled by default) and the results above, I observe a definite improvement:
Repro:
py .\scripts\benchmarks_ci.py -f net6.0 --filter 'System.IO.Tests.StreamReaderReadToEndTests.ReadToEndAsync'Closing this issue.