Regressions from early liveness pass

Run Information

Architecture	x64
OS	ubuntu 18.04
Baseline	71bb0d481086b8b8a89a99a17ec80a861f32dc5d
Compare	5385f045f9aee91c22922d21552c4317198710f1
Diff	Diff

Regressions in System.Collections.ContainsTrue<String>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
ImmutableHashSet - Duration of single invocation	32.29 μs	36.41 μs	1.13	0.01	False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsTrue&lt;String&gt;*'

Details

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsTrue<String>.ImmutableHashSet(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 36.41235108981795 > 33.81515342706716.
IsChangePoint: Marked as a change because one of 1/11/2023 2:37:52 AM, 1/17/2023 6:49:05 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 6:49:05 AM.
IsRegressionStdDev: Marked as regression because -18.599642750176685 (T) = (0 -36102.01133372504) / Math.Sqrt((672170.3616818737 / (26)) + (321371.36263329774 / (27))) is less than -2.007583770314729 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (26) + (27) - 2, .025) and -0.11124265742351962 = (32487.96389568921 - 36102.01133372504) / 32487.96389568921 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information

Architecture	x64
OS	ubuntu 18.04
Baseline	71bb0d481086b8b8a89a99a17ec80a861f32dc5d
Compare	5385f045f9aee91c22922d21552c4317198710f1
Diff	Diff

Regressions in System.Hashing

Benchmark	Baseline	Test	Test/Base	Edge Detector
GetStringHashCode - Duration of single invocation	29.31 ns	35.29 ns	1.20	False
GetStringHashCode - Duration of single invocation	300.60 ns	367.64 ns	1.22	False
GetStringHashCode - Duration of single invocation	2.99 μs	3.67 μs	1.23	False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Hashing*'

Details

Payloads

Baseline
Compare

Histogram

System.Hashing.GetStringHashCode(BytesCount: 100)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 35.29313893613284 > 30.78401297711811.
IsChangePoint: Marked as a change because one of 1/11/2023 2:37:52 AM, 1/17/2023 6:49:05 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 6:49:05 AM.
IsRegressionStdDev: Marked as regression because -135.42838161529687 (T) = (0 -35.40427065306296) / Math.Sqrt((0.009162700930984043 / (28)) + (0.04616867251241285 / (27))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (28) + (27) - 2, .025) and -0.20868019077850827 = (29.291677751630186 - 35.40427065306296) / 29.291677751630186 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Hashing.GetStringHashCode(BytesCount: 1000)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 367.6354811368269 > 315.7770026743071.
IsChangePoint: Marked as a change because one of 1/11/2023 2:37:52 AM, 1/17/2023 6:49:05 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 6:49:05 AM.
IsRegressionStdDev: Marked as regression because -530.7547613004574 (T) = (0 -367.9386961273943) / Math.Sqrt((0.1649982330629939 / (28)) + (0.2721170652369591 / (27))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (28) + (27) - 2, .025) and -0.22294277091957193 = (300.8633804268116 - 367.9386961273943) / 300.8633804268116 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Hashing.GetStringHashCode(BytesCount: 10000)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.6740150402414486 > 3.1479971791020867.
IsChangePoint: Marked as a change because one of 1/11/2023 2:37:52 AM, 1/17/2023 6:49:05 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 6:49:05 AM.
IsRegressionStdDev: Marked as regression because -351.7087743229655 (T) = (0 -3670.848951389847) / Math.Sqrt((10.379123507797555 / (28)) + (88.93952000952916 / (27))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (28) + (27) - 2, .025) and -0.224614397896344 = (2997.5549509263255 - 3670.848951389847) / 2997.5549509263255 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture	x64
OS	ubuntu 18.04
Baseline	71bb0d481086b8b8a89a99a17ec80a861f32dc5d
Compare	5385f045f9aee91c22922d21552c4317198710f1
Diff	Diff

Regressions in Microsoft.Extensions.Logging.LoggingOverhead

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
TwoArguments_FilteredByLevel - Duration of single invocation	52.71 ns	56.03 ns	1.06	0.11	False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Microsoft.Extensions.Logging.LoggingOverhead*'

Details

Payloads

Baseline
Compare

Histogram

Microsoft.Extensions.Logging.LoggingOverhead.TwoArguments_FilteredByLevel

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 56.02761713687869 > 55.54323764013431.
IsChangePoint: Marked as a change because one of 1/11/2023 2:37:52 AM, 1/17/2023 6:49:05 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 6:49:05 AM.
IsRegressionStdDev: Marked as regression because -8.505790559807027 (T) = (0 -57.91890805119079) / Math.Sqrt((2.6196181816810222 / (28)) + (2.0835384659087244 / (27))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (28) + (27) - 2, .025) and -0.06459967498995105 = (54.40440140256242 - 57.91890805119079) / 54.40440140256242 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture	x64
OS	ubuntu 18.04
Baseline	71bb0d481086b8b8a89a99a17ec80a861f32dc5d
Compare	5385f045f9aee91c22922d21552c4317198710f1
Diff	Diff

Regressions in System.Globalization.Tests.StringHash

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
GetHashCode - Duration of single invocation	78.46 μs	96.20 μs	1.23	0.00	False
GetHashCode - Duration of single invocation	79.85 ns	97.32 ns	1.22	0.00	False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Globalization.Tests.StringHash*'

Details

Payloads

Baseline
Compare

Histogram

System.Globalization.Tests.StringHash.GetHashCode(Count: 131072, Options: (en-US, Ordinal))

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 96.20395003834356 > 82.53206515389446.
IsChangePoint: Marked as a change because one of 1/11/2023 2:37:52 AM, 1/17/2023 6:49:05 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 6:49:05 AM.
IsRegressionStdDev: Marked as regression because -927.8163766760983 (T) = (0 -96218.72445464086) / Math.Sqrt((4799.708115284992 / (28)) + (5107.2703618762525 / (27))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (28) + (27) - 2, .025) and -0.22414805808620092 = (78600.56127938196 - 96218.72445464086) / 78600.56127938196 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Globalization.Tests.StringHash.GetHashCode(Count: 128, Options: (en-US, Ordinal))

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 97.32120498547347 > 83.84882269177078.
IsChangePoint: Marked as a change because one of 1/11/2023 2:37:52 AM, 1/17/2023 6:49:05 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 6:49:05 AM.
IsRegressionStdDev: Marked as regression because -240.165019247504 (T) = (0 -97.35952570185772) / Math.Sqrt((0.022850832525283242 / (27)) + (0.12000637324947573 / (27))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (27) - 2, .025) and -0.2186684111980267 = (79.89008725199274 - 97.35952570185772) / 79.89008725199274 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture	x64
OS	ubuntu 18.04
Baseline	71bb0d481086b8b8a89a99a17ec80a861f32dc5d
Compare	5385f045f9aee91c22922d21552c4317198710f1
Diff	Diff

Regressions in System.Text.Json.Document.Tests.Perf_EnumerateArray

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
EnumerateArray - Duration of single invocation	555.94 ns	666.03 ns	1.20	0.03	False
EnumerateArray - Duration of single invocation	555.50 ns	666.11 ns	1.20	0.03	False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Document.Tests.Perf_EnumerateArray*'

Details

Payloads

Baseline
Compare

Histogram

System.Text.Json.Document.Tests.Perf_EnumerateArray.EnumerateArray(TestCase: ArrayOfNumbers)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 666.0286184872873 > 583.6842368983396.
IsChangePoint: Marked as a change because one of 10/31/2022 4:38:04 PM, 11/14/2022 1:59:26 PM, 1/11/2023 2:37:52 AM, 1/17/2023 6:49:05 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 6:49:05 AM.
IsRegressionStdDev: Marked as regression because -64.2180625971499 (T) = (0 -666.0556611937677) / Math.Sqrt((92.8182726497298 / (28)) + (0.2808685969007546 / (27))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (28) + (27) - 2, .025) and -0.21332508980405204 = (548.9507031469476 - 666.0556611937677) / 548.9507031469476 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Document.Tests.Perf_EnumerateArray.EnumerateArray(TestCase: ArrayOfStrings)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 666.1087253910298 > 583.492956795757.
IsChangePoint: Marked as a change because one of 10/31/2022 4:38:04 PM, 11/14/2022 1:59:26 PM, 1/11/2023 2:37:52 AM, 1/17/2023 6:49:05 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 6:49:05 AM.
IsRegressionStdDev: Marked as regression because -65.4371743574446 (T) = (0 -666.1879416527432) / Math.Sqrt((91.97880423420412 / (27)) + (0.2674594162412254 / (27))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (27) - 2, .025) and -0.22183673129443549 = (545.234829326969 - 666.1879416527432) / 545.234829326969 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture	x64
OS	ubuntu 18.04
Baseline	71bb0d481086b8b8a89a99a17ec80a861f32dc5d
Compare	5385f045f9aee91c22922d21552c4317198710f1
Diff	Diff

Regressions in Microsoft.Extensions.Primitives.StringSegmentBenchmark

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
GetSegmentHashCode - Duration of single invocation	8.42 ns	9.65 ns	1.15	0.04	False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Microsoft.Extensions.Primitives.StringSegmentBenchmark*'

Details

Payloads

Baseline
Compare

Histogram

Microsoft.Extensions.Primitives.StringSegmentBenchmark.GetSegmentHashCode

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.645745549271561 > 8.843247772400257.
IsChangePoint: Marked as a change because one of 1/11/2023 2:37:52 AM, 1/17/2023 6:49:05 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 6:49:05 AM.
IsRegressionStdDev: Marked as regression because -71.17016400876543 (T) = (0 -9.714728087433029) / Math.Sqrt((0.0002696614693663075 / (27)) + (0.00862587104426406 / (27))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (27) - 2, .025) and -0.15336985510447312 = (8.422907920159801 - 9.714728087433029) / 8.422907920159801 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture	x64
OS	ubuntu 18.04
Baseline	71bb0d481086b8b8a89a99a17ec80a861f32dc5d
Compare	5385f045f9aee91c22922d21552c4317198710f1
Diff	Diff

Regressions in System.Buffers.Tests.ReadOnlySequenceTests<Byte>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
IterateTryGetTenSegments - Duration of single invocation	34.61 ns	38.80 ns	1.12	0.01	False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Buffers.Tests.ReadOnlySequenceTests&lt;Byte&gt;*'

Details

Payloads

Baseline
Compare

Histogram

System.Buffers.Tests.ReadOnlySequenceTests<Byte>.IterateTryGetTenSegments

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 38.80313055040986 > 36.41515371063089.
IsChangePoint: Marked as a change because one of 1/11/2023 2:37:52 AM, 1/17/2023 6:49:05 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 6:49:05 AM.
IsRegressionStdDev: Marked as regression because -21.071054527202303 (T) = (0 -38.640534493705175) / Math.Sqrt((0.36738886862822207 / (27)) + (0.3706076692052171 / (27))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (27) - 2, .025) and -0.0990879572400807 = (35.15690827032206 - 38.640534493705175) / 35.15690827032206 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture	x64
OS	ubuntu 18.04
Baseline	71bb0d481086b8b8a89a99a17ec80a861f32dc5d
Compare	5385f045f9aee91c22922d21552c4317198710f1
Diff	Diff

Regressions in System.Collections.TryGetValueFalse<String, String>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
ConcurrentDictionary - Duration of single invocation	11.44 μs	13.06 μs	1.14	0.01	False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.TryGetValueFalse&lt;String, String&gt;*'

Details

Payloads

Baseline
Compare

Histogram

System.Collections.TryGetValueFalse<String, String>.ConcurrentDictionary(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.062030934553519 > 11.960065061368638.
IsChangePoint: Marked as a change because one of 1/11/2023 2:37:52 AM, 1/17/2023 6:49:05 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 6:49:05 AM.
IsRegressionStdDev: Marked as regression because -53.312222633056756 (T) = (0 -13146.895876043533) / Math.Sqrt((22317.200968174126 / (28)) + (5205.8544723019895 / (27))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (28) + (27) - 2, .025) and -0.1462392855878558 = (11469.59107172903 - 13146.895876043533) / 11469.59107172903 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture	x64
OS	ubuntu 18.04
Baseline	71bb0d481086b8b8a89a99a17ec80a861f32dc5d
Compare	5385f045f9aee91c22922d21552c4317198710f1
Diff	Diff

Regressions in System.Collections.ContainsKeyFalse<String, String>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
ConcurrentDictionary - Duration of single invocation	11.68 μs	13.31 μs	1.14	0.01	False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsKeyFalse&lt;String, String&gt;*'

Details

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsKeyFalse<String, String>.ConcurrentDictionary(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.31282633298023 > 12.255327886504563.
IsChangePoint: Marked as a change because one of 1/11/2023 2:37:52 AM, 1/17/2023 6:49:05 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 6:49:05 AM.
IsRegressionStdDev: Marked as regression because -67.26612019697792 (T) = (0 -13316.463397055422) / Math.Sqrt((13296.717586853552 / (27)) + (2520.04918235662 / (27))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (27) - 2, .025) and -0.13928956137100498 = (11688.392353064815 - 13316.463397055422) / 11688.392353064815 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@jakobbotsch

PR: #79346 cc @jakobbotsch

The GetStringHashCode regression does not repro for me on Ryzen and in fact looks to be an improvement there (screenshot thanks to Egor):

I can indeed repro the regression on my Intel laptop, so it might be jcc erratum. This is the diff:
https://www.diffchecker.com/ueIXGTMh/

There is a new jcc erratum candidate on line 135 in the diff version. However, there's also some significant new register allocation going on, resulting from a lot of new forward subs. We have many new forward subs in this code:

runtime/src/libraries/System.Private.CoreLib/src/System/Marvin.cs

Lines 126 to 137 in 6e773f2

    
           if (BitConverter.IsLittleEndian) 
        
           { 
        
               partialResult >>= 8; // make some room for the 0x80 byte 
        
               partialResult |= 0x8000_0000u; // put the 0x80 byte at the beginning 
        
               partialResult >>= (int)count & 0x1F; // shift out all previously consumed bytes 
        
           } 
        
           else 
        
           { 
        
               partialResult <<= 8; // make some room for the 0x80 byte 
        
               partialResult |= 0x80u; // put the 0x80 byte at the end 
        
               partialResult <<= (int)count & 0x1F; // shift out all previously consumed bytes 
        
           }

runtime/src/libraries/System.Private.CoreLib/src/System/Marvin.cs

Lines 216 to 237 in 6e773f2

    
           [MethodImpl(MethodImplOptions.AggressiveInlining)] 
        
           private static void Block(ref uint rp0, ref uint rp1) 
        
           { 
        
               // Intrinsified in mono interpreter 
        
               uint p0 = rp0; 
        
               uint p1 = rp1; 
        
               p1 ^= p0; 
        
               p0 = BitOperations.RotateLeft(p0, 20); 
        
               p0 += p1; 
        
               p1 = BitOperations.RotateLeft(p1, 9); 
        
               p1 ^= p0; 
        
               p0 = BitOperations.RotateLeft(p0, 27); 
        
               p0 += p1; 
        
               p1 = BitOperations.RotateLeft(p1, 19); 
        
               rp0 = p0; 
        
               rp1 = p1; 
        
           }

From what I can tell, it leads to slightly worse register allocation and a couple of extra register moves/resolution. But given that these should be handled via register renaming, I think it is more likely that the JCC erratum explains the perf difference.

I think it is more likely that the JCC erratum explains the perf difference

That could be true given the fact that same code is faster on AMD. However, JCC is really in the block that has lower bbWeight and not sure how much it contributes. I see an extra xchg (and of course 2 moves) that might be contributing to the perf difference.

I see an extra xchg (and of course 2 moves) that might be contributing to the perf difference.

Yes, that's the resolution I was hinting at above. I wonder if it would be better performance for LSRA to do such an exchange with a temporary register (if one is free).

@kunalspathak

I hacked LSRA to do the resolution via a free temporary register at that point instead. With that the regression disappears in the large cases, but remains partially for the small case:

Method	Job	Toolchain	BytesCount	Mean	Error	StdDev	Median	Min	Max	Ratio	RatioSD	Allocated	Alloc Ratio
GetStringHashCode	Job-MNICBP	\Core_Root\corerun.exe	10	3.515 ns	0.0798 ns	0.0666 ns	3.517 ns	3.387 ns	3.624 ns	1.08	0.03	-	NA
GetStringHashCode	Job-BZQGWO	\Core_Root_base\corerun.exe	10	3.275 ns	0.0751 ns	0.0702 ns	3.280 ns	3.194 ns	3.418 ns	1.00	0.00	-	NA

GetStringHashCode	Job-MNICBP	\Core_Root\corerun.exe	100	28.421 ns	0.5068 ns	0.5205 ns	28.360 ns	27.637 ns	29.388 ns	1.00	0.04	-	NA
GetStringHashCode	Job-BZQGWO	\Core_Root_base\corerun.exe	100	28.436 ns	0.5940 ns	0.6603 ns	28.385 ns	27.654 ns	29.795 ns	1.00	0.00	-	NA

GetStringHashCode	Job-MNICBP	\Core_Root\corerun.exe	1000	291.086 ns	6.0385 ns	6.9540 ns	289.770 ns	283.003 ns	305.492 ns	0.99	0.03	-	NA
GetStringHashCode	Job-BZQGWO	\Core_Root_base\corerun.exe	1000	294.442 ns	5.7260 ns	6.3644 ns	292.624 ns	286.687 ns	305.644 ns	1.00	0.00	-	NA

GetStringHashCode	Job-MNICBP	\Core_Root\corerun.exe	10000	2,892.172 ns	53.2998 ns	44.5077 ns	2,900.437 ns	2,815.856 ns	2,970.939 ns	0.98	0.02	-	NA
GetStringHashCode	Job-BZQGWO	\Core_Root_base\corerun.exe	10000	2,945.900 ns	52.8054 ns	46.8106 ns	2,946.043 ns	2,851.178 ns	3,023.474 ns	1.00	0.00	-	NA

Makes sense to me, after the loop is warmed up it should be hitting the µop cache where the extra movs have been register renamed away.

I agree with @kunalspathak that the xchg is probably larger part of the problem than the JCC erratum here, after analyzing the codegen some more I think the specific new JCC erratum is not part of the hot inner loop.
The above hack does not really help us know as using 3 movs instead of xchg also has the effect of mitigating the JCC erratum.

JIT: Avoid xchg for resolution

JIT: Avoid xchg for resolution (#81216)

I still need to look at `System.Collections.ContainsTrue.ImmutableHashSet:

GetStringHashCode was fixed by #81216.

LoggingOverhead.TwoArguments_FilteredByLevel has also been improved since:

There still seems to be a regression in System.Globalization.Tests.StringHash.GetHashCode, #81216 did not get us completely back to previous levels:

StringSegmentBenchmark.GetSegmentHashCode looks similar.

EnumerateArray ones look bimodal.

ReadOnlySequenceTests.IterateTryGetTenSegments also still has a regression:

TryGetValuseFalse and ContainsKeyFalse for ConcurrentDictionary look to have improved via #81557.

Most of these are improved overall now with tiered PGO, so will close this.

	if (BitConverter.IsLittleEndian)
	{
	partialResult >>= 8; // make some room for the 0x80 byte
	partialResult \|= 0x8000_0000u; // put the 0x80 byte at the beginning
	partialResult >>= (int)count & 0x1F; // shift out all previously consumed bytes
	}
	else
	{
	partialResult <<= 8; // make some room for the 0x80 byte
	partialResult \|= 0x80u; // put the 0x80 byte at the end
	partialResult <<= (int)count & 0x1F; // shift out all previously consumed bytes
	}

	[MethodImpl(MethodImplOptions.AggressiveInlining)]
	private static void Block(ref uint rp0, ref uint rp1)
	{
	// Intrinsified in mono interpreter
	uint p0 = rp0;
	uint p1 = rp1;

	p1 ^= p0;
	p0 = BitOperations.RotateLeft(p0, 20);

	p0 += p1;
	p1 = BitOperations.RotateLeft(p1, 9);

	p1 ^= p0;
	p0 = BitOperations.RotateLeft(p0, 27);

	p0 += p1;
	p1 = BitOperations.RotateLeft(p1, 19);

	rp0 = p0;
	rp1 = p1;
	}

Regressions from early liveness pass #80743

Description

Run Information

Regressions in System.Collections.ContainsTrue<String>

Repro

Payloads

Histogram

System.Collections.ContainsTrue<String>.ImmutableHashSet(Size: 512)

Description of detection logic

Docs

Regressions in System.Hashing

Repro

Payloads

Histogram

System.Hashing.GetStringHashCode(BytesCount: 100)

Description of detection logic

Description of detection logic

Description of detection logic

Docs

Run Information

Regressions in Microsoft.Extensions.Logging.LoggingOverhead

Repro

Payloads

Histogram

Microsoft.Extensions.Logging.LoggingOverhead.TwoArguments_FilteredByLevel

Description of detection logic

Docs

Run Information

Regressions in System.Globalization.Tests.StringHash

Repro

Payloads

Histogram

System.Globalization.Tests.StringHash.GetHashCode(Count: 131072, Options: (en-US, Ordinal))

Description of detection logic

Description of detection logic

Docs

Run Information

Regressions in System.Text.Json.Document.Tests.Perf_EnumerateArray

Repro

Payloads

Histogram

System.Text.Json.Document.Tests.Perf_EnumerateArray.EnumerateArray(TestCase: ArrayOfNumbers)

Description of detection logic

Description of detection logic

Docs

Run Information

Regressions in Microsoft.Extensions.Primitives.StringSegmentBenchmark

Repro

Payloads

Histogram

Microsoft.Extensions.Primitives.StringSegmentBenchmark.GetSegmentHashCode

Description of detection logic

Docs

Run Information

Regressions in System.Buffers.Tests.ReadOnlySequenceTests<Byte>

Repro

Payloads

Histogram

System.Buffers.Tests.ReadOnlySequenceTests<Byte>.IterateTryGetTenSegments

Description of detection logic

Docs

Run Information

Regressions in System.Collections.TryGetValueFalse<String, String>

Repro

Payloads

Histogram

System.Collections.TryGetValueFalse<String, String>.ConcurrentDictionary(Size: 512)

Description of detection logic

Docs

Run Information

Regressions in System.Collections.ContainsKeyFalse<String, String>

Repro

Payloads

Histogram

System.Collections.ContainsKeyFalse<String, String>.ConcurrentDictionary(Size: 512)

Description of detection logic

Docs

Activity

EgorBo commented on Jan 17, 2023

jakobbotsch commented on Jan 18, 2023