[ViveliDuCh]

Fixes #136062
Fixes #136061

main PR

No main PR is linked; these are test-only adjustments for release/8.0.

Description

Fix two test failures reported on release/8.0. Production behavior is unchanged.

• PKCS#7 certificate expectations: Expect single-certificate PKCS#7 loading to fail on Unix even when the compatibility switch allows non-certificate X509 data. The switch does not add support to the underlying Unix certificate importer. Preserve the successful Windows compatibility-switch case and retain all existing theory cases.
• Deep XML tests: Reduce the nesting depth from 100,000 to 10,000 for both root and non-root cases. Keep the original encryption setup and exception assertions unchanged. This reduces setup work to avoid timeouts while retaining depth-limit coverage and minimizing divergence from main for this release-branch fix.

Customer Impact

No production behavior changes. Without these test fixes, release validation remains subject to an incorrect Unix PKCS#7 expectation and a Windows x86 Checked timeout in deep XML test setup.

Regression

Addresses test failures reported on release/8.0; no product regression is fixed by this PR.

Testing

The unchanged baseline was built and tested before the initial fixes. The Unix PKCS#7 failure reproduced; the original CI XML timeout did not reproduce locally.

After the final 10,000-depth adjustment, the following command passed on Windows x86 with Debug libraries and the Checked CoreCLR testhost: 2 passed, 0 failed, 0 skipped. The root and non-root cases each completed in approximately 5 seconds.

dotnet build src\libraries\System.Security.Cryptography.Xml\tests\System.Security.Cryptography.Xml.Tests.csproj --no-restore -f net8.0 /t:Test /p:TargetArchitecture=x86 /p:Configuration=Debug /p:RuntimeConfiguration=Checked '/p:XUnitOptions=-method System.Security.Cryptography.Xml.Tests.EncryptedXmlTests.DeepNonRootDecryptedXml_Throws -method System.Security.Cryptography.Xml.Tests.EncryptedXmlTests.DeepRootDecryptedXml_Throws' /v:quiet /nologo

The full inner-loop suite was not rerun after the final depth-only adjustment. Earlier full-suite results and timings applied to the superseded byte-payload setup and are not claimed as validation of this final version.

Risk

Low: changes are limited to tests. The XML change is a single depth-constant reduction relative to the release-branch baseline; 10,000 levels still exceeds the default depth limit of 64. Both tests retain their original exception assertions. The PKCS#7 change aligns expectations with the platform importer without changing production code.

Package authoring signed off?

IMPORTANT: If this change touches code that ships in a NuGet package, please make certain that you have added any necessary package authoring and gotten it explicitly reviewed.

Not applicable: only test code changes; no package authoring changes are required.

[azure-pipelines]

Azure Pipelines:
Successfully started running 3 pipeline(s).
13 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[dotnet-policy-service]

Tagging subscribers to this area: @bartonjs, @vcsjones, @dotnet/area-system-security
See info in area-owners.md if you want to be subscribed.

[ViveliDuCh]

Test-only fix, adding servicing-approved label