Repository navigation
System.Numerics.Tests.Perf_Matrix4x4.MultiplyByScalarOperatorBenchmark has regressed #65191
Description
System.Numerics.Tests.Perf_Matrix4x4.MultiplyByScalarOperatorBenchmark has regressed, it might be related to #59415
Repro:
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 net7.0 --filter System.Numerics.Tests.Perf_Matrix4x4.MultiplyByScalarOperatorBenchmark It seems that @performanceautofiler has missed the regression (cc @DrewScoggins ): https://pvscmdupload.blob.core.windows.net/reports/allTestHistory%2frefs%2fheads%2fmain_x64_Windows%2010.0.18362%2fSystem.Numerics.Tests.Perf_Matrix4x4.MultiplyByScalarOperatorBenchmark.html
System.Numerics.Tests.Perf_Matrix4x4.MultiplyByScalarOperatorBenchmark
| Result | Base | Diff | Ratio | Modality | Operating System | Bit | Processor Name |
|---|---|---|---|---|---|---|---|
| Slower | 4.67 | 8.10 | 0.58 | Windows 11 | X64 | AMD Ryzen Threadripper PRO 3945WX 12-Cores | |
| Slower | 3.72 | 7.00 | 0.53 | Windows 11 | X64 | AMD Ryzen 9 5900X | |
| Same | 5.56 | 7.91 | 0.70 | Windows 10 | X64 | Intel Xeon CPU E5-1650 v4 3.60GHz | |
| Slower | 7.99 | 11.94 | 0.67 | several? | Windows 11 | X64 | Intel Core i5-4300U CPU 1.90GHz (Haswell) |
| Same | 5.62 | 7.03 | 0.80 | Windows 10 | X64 | Intel Core i7-7700 CPU 3.60GHz (Kaby Lake) | |
| Same | 5.04 | 6.85 | 0.74 | Windows 11 | X64 | Intel Core i7-8700 CPU 3.20GHz (Coffee Lake) | |
| Slower | 5.20 | 8.06 | 0.65 | Windows 11 | X64 | Intel Core i9-9900T CPU 2.10GHz | |
| Slower | 7.80 | 10.54 | 0.74 | Windows 11 | X64 | Unknown processor | |
| Same | 9.87 | 9.32 | 1.06 | multimodal | Windows 11 | X64 | Unknown processor |
| Slower | 4.22 | 7.39 | 0.57 | ubuntu 20.04 | X64 | AMD Ryzen 9 5900X | |
| Slower | 5.40 | 10.62 | 0.51 | ubuntu 18.04 | X64 | Intel Xeon CPU E5-1650 v4 3.60GHz | |
| Slower | 9.77 | 12.18 | 0.80 | centos 7 | X64 | Intel Xeon CPU E5530 2.40GHz | |
| Slower | 7.62 | 11.41 | 0.67 | ubuntu 18.04 | X64 | Intel Core i7-2720QM CPU 2.20GHz (Sandy Bridge) | |
| Slower | 6.06 | 8.61 | 0.70 | alpine 3.13 | X64 | Intel Core i7-7700 CPU 3.60GHz (Kaby Lake) | |
| Slower | 5.99 | 8.41 | 0.71 | ubuntu 18.04 | X64 | Intel Core i7-7700 CPU 3.60GHz (Kaby Lake) | |
| Slower | 6.04 | 8.30 | 0.73 | ubuntu 20.04 | X64 | Intel Core i7-7700 CPU 3.60GHz (Kaby Lake) | |
| Same | 10.94 | 11.56 | 0.95 | Windows 10 | Arm64 | Microsoft SQ1 3.0 GHz | |
| Slower | 7.16 | 11.15 | 0.64 | bimodal | Windows 10 | X86 | Intel Xeon CPU E5-1650 v4 3.60GHz |
| Same | 34.35 | 31.63 | 1.09 | Windows 10 | Arm | Microsoft SQ1 3.0 GHz | |
| Slower | 7.07 | 11.71 | 0.60 | macOS Big Sur 11.6.3 | X64 | Intel Core i5-4278U CPU 2.60GHz (Haswell) | |
| Slower | 6.35 | 10.57 | 0.60 | macOS Big Sur 11.4 | X64 | Intel Core i7-5557U CPU 3.10GHz (Broadwell) |
Activity
adamsitnik commentedon Feb 11, 2022
Same goes for
System.Numerics.Tests.Perf_Matrix4x4.Transposehttps://pvscmdupload.blob.core.windows.net/reports/allTestHistory%2frefs%2fheads%2fmain_x64_ubuntu%2018.04%2fSystem.Numerics.Tests.Perf_Matrix4x4.Transpose.html
Details
adamsitnik commentedon Feb 11, 2022
And
System.Numerics.Tests.Perf_Matrix4x4.NegateBenchmark:https://pvscmdupload.blob.core.windows.net/reports/allTestHistory%2frefs%2fheads%2fmain_x64_Windows%2010.0.18362%2fSystem.Numerics.Tests.Perf_Matrix4x4.NegateBenchmark.html
adamsitnik commentedon Feb 11, 2022
There are more, I won't list them all here.
tannergooding commentedon Feb 11, 2022
I'll have to look at the codegen for these. They haven't been changed in a long time and so I expect this is something that's changed in CSE or similar.
Notably, they could/should likely be simplified to use the xplat helper intrinsics now, rather than individual AdvSimd/Sse code paths
jozkee commentedon Oct 14, 2022
Some benchmarks discussed here appeared in the 6.0 vs 7.0-rc2 perf report. Are they still under investigation? The historical data only goes back until october 2021 so we are unable to determine if this is noise nor to see what might've caused it.
System.Numerics.Tests.Perf_Matrix4x4.NegateBenchmarkSystem.Numerics.Tests.Perf_Matrix4x4.MultiplyByScalarOperatorBenchmarkSystem.Numerics.Tests.Perf_Matrix4x4.NegationOperatorBenchmarkSystem.Numerics.Tests.Perf_Matrix4x4.SubtractOperatorBenchmarkSystem.Numerics.Tests.Perf_Matrix4x4.TransposeSystem.Numerics.Tests.Perf_Matrix4x4.MultiplyByScalarOperatorBenchmark
Details
System.Numerics.Tests.Perf_Matrix4x4.NegateBenchmark
Details
System.Numerics.Tests.Perf_Matrix4x4.NegationOperatorBenchmark
Details
System.Numerics.Tests.Perf_Matrix4x4.SubtractOperatorBenchmark
Details
System.Numerics.Tests.Perf_Matrix4x4.Transpose
Details
dakersnar commentedon Dec 2, 2022
The .NET 7 GA report is also showing potential regression in the following. I ran into the same problem where the historical data does not go back far enough to help triage this. @DrewScoggins is this something we can change?
System.Numerics.Tests.Perf_Matrix4x4.AddOperatorBenchmark
Details
DrewScoggins commentedon Dec 5, 2022
We should figure out something that we can do. Naively, we could just extend the window of that history page from 12 to 18 months. However, that would not be feasible with the way those pages are currently generated, they would take too long to retrieve all of the data.
For these individual tests that we need more data for we can go ahead and generate longer trends, and just do them on a case by case basis, just get me a list of the tests that we need this for. However, I think the right way to fix this going forward is to update the generation method for the data on these pages so that we are not redoing work over and over (Currently, we regenerate the data every day, even though all we need to do is drop the earliest and grab the latest)
tannergooding commentedon Jul 24, 2023
This was resolved with my rewrite of Matrix4x4 so be able to utilize promotion, field enregistration, and general SIMD/vectorization