Reflection mode: TUnit.UnitTests fails 4 tests that pass in source-gen mode #7092

Summary

On main, tests/TUnit.UnitTests fails 4 tests on net10.0 when run with --reflection. The same tests pass in source-generated mode. Each failure has a separate cause in a reflection-mode code path, so reflection mode behaves differently from source-generated mode for these test shapes. These are real reflection-mode bugs, not tests that rely on source-gen behaviour. They are separate from #7089.

While reproducing them I hit two more problems in the same area. They are listed below the four failures.

Repro

cd tests/TUnit.UnitTests
dotnet build -f net10.0
dotnet run -f net10.0 --no-build -- --reflection

Result: total: 532, failed: 4.

Failing tests

1. EmptyDataSourceTests.EmptyStaticDataSource_ShouldYieldOneEmptyResult

InvalidOperationException: No members to generate were provided for TestParameters
   at TUnit.Core.MethodDataSourceAttribute.GetDataRowsAsync(...) MethodDataSourceAttribute.cs:170

Cause: MethodDataSourceAttribute.GetDataRowsAsync throws when DataGeneratorMetadata.MembersToGenerate is empty, and it is always empty for a test method with no parameters. Source-gen mode never reaches this code because it uses the generated Factory. In reflection mode, any [MethodDataSource] on a parameterless test method fails.

2. AsyncDataSourceTests.TestWithAsyncValueTaskDataSource

ArgumentException: Object of type 'System.Threading.Tasks.ValueTask`1[System.Collections.Generic.IEnumerable`1[System.Object[]]]' cannot be converted to type 'System.Int32'.

The test is discovered as TestWithAsyncValueTaskDataSource(System.Collections.Generic.List1[System.Object[]])` instead of two rows.

Cause: the reflection path in MethodDataSourceAttribute.GetDataRowsAsync handles IAsyncEnumerable<T>, Task<T> and IEnumerable, but not ValueTask<T>. A ValueTask<IEnumerable<...>> result is never awaited, so it is passed to the test as one argument. Source-gen mode uses DataSourceFactories.FromValueTask, which handles it.

3. GenericTestGenerationTests.GenericTestWithData<Int32>

ArgumentException: Object of type 'System.String' cannot be converted to type 'System.Int32'.

The test is GenericTestWithData<T>(T value) with [Arguments(5)], [Arguments("hello")], [GenerateGenericTest(typeof(int))] and [GenerateGenericTest(typeof(string))].

• Source-gen mode creates 2 tests: <int>(5) and <string>(hello).
• Reflection mode creates 4: every [Arguments] row for every [GenerateGenericTest] type. <Int32>(hello) then fails.

Cause: the generator pairs each [Arguments] row with the instantiation inferred from it. The reflection collector closes the method for each [GenerateGenericTest] type and then attaches every [Arguments] row, including rows whose values cannot bind to the closed parameter types.

4. RepositoryTestBase<T>.InitializeEntity (via CustomerRepositoryTests)

BeforeClassException: BeforeClass hook failed: Late bound operations cannot be performed on types or methods for which ContainsGenericParameters is true.
   at TUnit.Engine.Discovery.ReflectionHookDiscoveryService...CreateHookDelegate... ReflectionHookDiscoveryService.cs:974

The dependent test CanSetCustomerName is skipped because its dependency failed.

Cause: [Before(Class)] public static Task CleanEntities() is declared on the open generic RepositoryTestBase<T>. The reflection hook delegate calls the open-generic MethodInfo directly. The source-generated hook body walks context.ClassType's base types to the closed RepositoryTestBase<Customer> and calls the method there.

Other problems found while reproducing

5. Reflection discovery finds no tests when the checkout path contains "ref" or "runtimes"

ReflectionTestDataCollector.ShouldScanAssembly excludes any assembly whose Location contains "ref" or "runtimes" anywhere in the path. A checkout at C:\git\TUnit-worktrees\reflection-unittests\ (or ~/refactoring/, D:\src\preferences-app\, ...) therefore excludes every user assembly. With zero assemblies, new SemaphoreSlim(0, 0) then crashes discovery:

ArgumentOutOfRangeException: The maximumCount argument must be a positive number. (Parameter 'maxCount')
   at TUnit.Engine.Discovery.ReflectionTestDataCollector.CollectTestsAsync ReflectionTestDataCollector.cs:109

6. Source generator drops SkipIfEmpty on [MethodDataSource]

The generator's MethodDataSourceAttribute emission copies Arguments and DeferEnumeration, but not SkipIfEmpty. In source-gen mode a [MethodDataSource(..., SkipIfEmpty = true)] test with an empty source disappears from discovery instead of being reported as skipped. Reflection mode reports it as skipped. Example: EmptyDataSourceTests.EmptyStaticDataSource_WithSkipIfEmpty_ShouldBeSkipped.

Impact on users

Users who run in reflection mode (--reflection, VB/F# projects, or projects with source generation disabled) see:

• [MethodDataSource] fails on any parameterless test.
• Data source methods that return ValueTask<IEnumerable<T>> produce one wrong test instead of the real rows.
• [GenerateGenericTest] combined with [Arguments] produces extra failing tests.
• Static class hooks on generic base classes always fail, and every test in the class fails with them.
• Reflection mode finds no tests at all when the path contains "ref" or "runtimes".

Source-gen users silently lose SkipIfEmpty tests.

Open question (not changed here)

With an empty [MethodDataSource] and no SkipIfEmpty, the two modes still differ:

• Reflection mode yields one empty row, so the test runs once with no arguments. This is the "Test instance is null for test after instance creation" for test class with empty data source #2862 behaviour that EmptyDataSourceTests documents.
• Source-gen mode yields no rows, so the test does not exist.

The IDataSourceAttribute.SkipIfEmpty doc ("the test will be skipped instead of failing") suggests a third option: report it as a failure. Picking one changes user-visible behaviour, so it needs an owner decision.