`dotnet test` for MTP throws hard-to-interpret KeyNotFoundException

[Youssef1313]

So far I saw it only once, but I think it's worth investigating:

The following exception occurred when running the test module with RunCommand '/Users/runner/work/1/s/artifacts/bin/Microsoft.Testing.Extensions.UnitTests/Debug/net9.0/Microsoft.Testing.Extensions.UnitTests' and RunArguments '  --diagnostic --diagnostic-output-directory /Users/runner/work/1/s/artifacts/log/Debug --diagnostic-file-prefix Microsoft.Testing.Extensions.UnitTests_net9.0_Debug_x64 --diagnostic-verbosity trace --crashdump --hangdump --hangdump-timeout 15m --report-azdo --coverage --coverage-settings /Users/runner/work/1/s/test/coverage.config --coverage-output Microsoft.Testing.Extensions.UnitTests_net9.0_Debug_x64.coverage --report-trx --report-trx-filename Microsoft.Testing.Extensions.UnitTests_net9.0_x64.trx --results-directory /Users/runner/work/1/s/artifacts/TestResults/Debug/':

System.Collections.Generic.KeyNotFoundException: The given key 'ba98a3b7d1b0470a866ea837070091b8' was not present in the dictionary.
   at System.Collections.Concurrent.ConcurrentDictionary`2.ThrowKeyNotFoundException(TKey key)
   at System.Collections.Concurrent.ConcurrentDictionary`2.get_Item(TKey key)
   at Microsoft.DotNet.Cli.Commands.Test.Terminal.TerminalTestReporter.AssemblyRunCompleted(String executionId, Int32 exitCode, String outputData, String errorData)
   at Microsoft.DotNet.Cli.Commands.Test.TestApplicationHandler.OnTestProcessExited(Int32 exitCode, String outputData, String errorData)
   at Microsoft.DotNet.Cli.Commands.Test.TestApplication.RunAsync()
   at Microsoft.DotNet.Cli.Commands.Test.TestApplication.RunAsync()
   at Microsoft.DotNet.Cli.Commands.Test.TestApplicationActionQueue.Read(BuildOptions buildOptions, TestOptions testOptions, TerminalTestReporter output, Action`1 onHelpRequested)

In addition, this exception was just buried somewhere in the middle of the output and wasn't surfaced towards the end of the output.

[Youssef1313]

I have a theory here.

We started a test host controller, the test host controller failed for some reason and exited. In that case, we got a handshake from the controller, but we never called AssemblyRunStarted because the testhost didn't start. Then we called AssemblyRunCompleted, but the key won't be found in the dictionary.

What is missing here:

• Are we burying the reason why test host controller failed?
• The error should probably surface in a better way.
• The error should be re-printed towards the end.

[Youssef1313]

There are additional details in microsoft/testfx#6972.

[AndreasPetersen]

I have a test suite where I get this error

The following exception occurred when running the test module with RunCommand '/home/runner/work/MyTests' and RunArguments ' ':
System.Collections.Generic.KeyNotFoundException: The given key '90beda4603ef4b3faaa78373d263f779' was not present in the dictionary.
   at System.Collections.Concurrent.ConcurrentDictionary`2.ThrowKeyNotFoundException(TKey key)
   at System.Collections.Concurrent.ConcurrentDictionary`2.get_Item(TKey key)
   at Microsoft.DotNet.Cli.Commands.Test.Terminal.TerminalTestReporter.AssemblyRunCompleted(String executionId, Int32 exitCode, String outputData, String errorData)
   at Microsoft.DotNet.Cli.Commands.Test.TestApplicationHandler.OnTestProcessExited(Int32 exitCode, String outputData, String errorData)
   at Microsoft.DotNet.Cli.Commands.Test.TestApplication.RunAsync()
   at Microsoft.DotNet.Cli.Commands.Test.TestApplication.RunAsync()
   at Microsoft.DotNet.Cli.Commands.Test.TestApplicationActionQueue.Read(BuildOptions buildOptions, TestOptions testOptions, TerminalTestReporter output, Action`1 onHelpRequested)

I have tried enabling diagnostic logs, but I don't really get any more useful details from those. Is there anything I can do to see what is causing the issue so that I might fix it? Strangely enough I'm only getting this issue on my test suite for one of the test projects, and only when running on GitHub Actions: On WSL locally it runs just find. I'm kind of at a loss as to what to do, to further debug the issue.

[AndreasPetersen]

Okay I managed to find a workaround for my issue.

My projects were organized like this. Note that this is the actual directory structure, and not the solution structure, which doesn't necessarily follow the actual underlying directory structure:

src/Test project 1/
src/Test project 2/
src/Test project 3/
src/Test project 4/
src/A/Test project 5/
src/B/Test project 6
src/B/Test project 7

It was only Test project 6 that was failing. I then reorganized my solution to:

src/C/D/Test project 1/
src/C/E/Test project 2/
src/C/E/Test project 3/
src/F/Test project 4/
src/F/A/Test project 5/
src/B/Test project 6
src/B/Test project 7

Now I suddenly had three test projects with the error: Test project 6, 1, 5. Then I just flatly organized all projects under src/, and suddenly all test projects run successfully.

The solution folders did not seem to affect the issue.

[jackmtpt]

I've also started running into this on a multi-targeted test project. The .NET 10 assembly runs fine, but the .NET 8 one does not.

[Youssef1313]

@jackmtpt @AndreasPetersen The KeyNotFoundException here is just a sign of something else failing. Without providing more information/repro for your specific cases, it will be hard to say exactly what's wrong.

[jackmtpt]

 DBG   ID     OSID ThreadOBJ           State GC Mode     GC Alloc Context                  Domain           Count Apt Exception
   0    1      27a 000061F585D7C460    20020 Cooperative 000077C5C8D64D60:000077C5C8D65070 000061F585D2A260 -00001 Ukn System.OperationCanceledException 000077c5c8d43908

    1 System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
    1 System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task)
    1 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions)
    1 System.Runtime.CompilerServices.TaskAwaiter<System.Int32>.GetResult()
    1 my.namespace.MicrosoftTestingPlatformEntryPoint.<Main>(String[])

looks like the test assembly died for whatever reason, poking about in the crash dump for it now...