ConditionalFact does not work from within VS

[weltkante]

• This issue is blocking
• This issue is causing unreasonable pain

ConditionalFact attributes are not working when tests are run inside VS 2022 - instead these tests are simply always run.

This was discovered here on the winforms repo, where tests that shouldn't be running were reported as failing.

Its unclear whether this is a VS 2022 issue or if the implementation of ConditionalFact needs to be updated to work under VS 2022

Example repro:

public class MachineSpecificTests
{
    public static bool Is32Bit => IntPtr.Size == 4;
    public static bool Is64Bit => IntPtr.Size == 8;

    [ConditionalFact(nameof(Is32Bit))]
    public void TestIntPtrSize32()
    {
        Assert.Equal(4, IntPtr.Size);
    }

    [ConditionalFact(nameof(Is64Bit))]
    public void TestIntPtrSize64()
    {
        Assert.Equal(8, IntPtr.Size);
    }
}

(The actual tests WinForms uses are here: https://github.com/dotnet/winforms/blob/main/src/System.Windows.Forms.Primitives/tests/UnitTests/Interop/PARAMTests.cs)

This is owned by the runtime team, assigning to #363
@akoeplinger FYI

[RussKie]

This is tracked under https://devdiv.visualstudio.com/DevDiv/_workitems/edit/1396904

[akoeplinger]

hm this is weird, I guess the ConditionalFactDiscoverer which marks the test as skipped is not running for some reason:

arcade/src/Microsoft.DotNet.XUnitExtensions/src/Discoverers/ConditionalFactDiscoverer.cs

Lines 18 to 23 in 9a94c6b

	if (ConditionalTestDiscoverer.TryEvaluateSkipConditions(discoveryOptions, DiagnosticMessageSink, testMethod, factAttribute.GetConstructorArguments().ToArray(), out string skipReason, out ExecutionErrorTestCase errorTestCase))
	{
	return skipReason != null
	? (IXunitTestCase) new SkippedTestCase(skipReason, DiagnosticMessageSink, discoveryOptions.MethodDisplayOrDefault(), discoveryOptions.MethodDisplayOptionsOrDefault(), testMethod)
	: new SkippedFactTestCase(DiagnosticMessageSink, discoveryOptions.MethodDisplayOrDefault(), discoveryOptions.MethodDisplayOptionsOrDefault(), testMethod); // Test case skippable at runtime.
	}