Some Helix work items are failing with exit code 8 (zero tests run) when all tests skipped.

[MichaelSimons]

See #54950 (comment). I've seen this in numerous PRs.

      Test results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1478091&view=ms.vss-test-web.build-test-results-tab
      ├─ dotnet-aot.Tests.dll.1 (Job: TestBuild: windows (x64) - windows.amd64.vs2026.pre.scout.open (985b0fb7-6413-42d8-b665-94c4e2ca8210)) (Finished, exit code 8)
      │  └─ Console: https://helixr1107v0xdeko0k025g8.blob.core.windows.net/dotnet-sdk-refs-pull-54950-merge-985b0fb7641342d8b6/dotnet-aot.Tests.dll.1/1/console.eb24658c.log?helixlogtype=result
      ├─ dotnet-aot.Tests.dll.1 (Job: TestBuild: linux (x64) - ubuntu.2204.amd64.open (aa4cd254-201f-476d-9295-c28cff15ebb6)) (Finished, exit code 8)
      │  └─ Console: https://helixr1107v0xdeko0k025g8.blob.core.windows.net/dotnet-sdk-refs-pull-54950-merge-aa4cd254201f476d92/dotnet-aot.Tests.dll.1/1/console.aff5f979.log?helixlogtype=result
      ├─ dotnet-aot.Tests.dll.1 (Job: TestBuild: macOS (arm64) - osx.15.arm64.open (ea0828db-ea88-45be-9830-d1130f5306c9)) (Finished, exit code 8)
      │  └─ Console: https://helixr1107v0xdeko0k025g8.blob.core.windows.net/dotnet-sdk-refs-pull-54950-merge-ea0828dbea8845be98/dotnet-aot.Tests.dll.1/1/console.30cd713a.log?helixlogtype=result
      ├─ dotnet-aot.Tests.dll.2 (Job: TestBuild: windows (x64) - windows.amd64.vs2026.pre.scout.open (985b0fb7-6413-42d8-b665-94c4e2ca8210)) (Finished, exit code 8)
      │  └─ Console: https://helixr1107v0xdeko0k025g8.blob.core.windows.net/dotnet-sdk-refs-pull-54950-merge-985b0fb7641342d8b6/dotnet-aot.Tests.dll.2/1/console.ab7f2343.log?helixlogtype=result
      ├─ dotnet-aot.Tests.dll.2 (Job: TestBuild: linux (x64) - ubuntu.2204.amd64.open (aa4cd254-201f-476d-9295-c28cff15ebb6)) (Finished, exit code 8)
      │  └─ Console: https://helixr1107v0xdeko0k025g8.blob.core.windows.net/dotnet-sdk-refs-pull-54950-merge-aa4cd254201f476d92/dotnet-aot.Tests.dll.2/1/console.0047ceaa.log?helixlogtype=result
      ├─ dotnet-aot.Tests.dll.3 (Job: TestBuild: linux (x64) - ubuntu.2204.amd64.open (aa4cd254-201f-476d-9295-c28cff15ebb6)) (Finished, exit code 8)
      │  └─ Console: https://helixr1107v0xdeko0k025g8.blob.core.windows.net/dotnet-sdk-refs-pull-54950-merge-aa4cd254201f476d92/dotnet-aot.Tests.dll.3/1/console.fe576a3e.log?helixlogtype=result
      ├─ dotnet-aot.Tests.dll.3 (Job: TestBuild: windows (x64) - windows.amd64.vs2026.pre.scout.open (985b0fb7-6413-42d8-b665-94c4e2ca8210)) (Finished, exit code 8)
      │  └─ Console: https://helixr1107v0xdeko0k025g8.blob.core.windows.net/dotnet-sdk-refs-pull-54950-merge-985b0fb7641342d8b6/dotnet-aot.Tests.dll.3/1/console.107fa6f3.log?helixlogtype=result
      ├─ dotnet-aot.Tests.dll.4 (Job: TestBuild: linux (x64) - ubuntu.2204.amd64.open (aa4cd254-201f-476d-9295-c28cff15ebb6)) (Finished, exit code 8)
      │  └─ Console: https://helixr1107v0xdeko0k025g8.blob.core.windows.net/dotnet-sdk-refs-pull-54950-merge-aa4cd254201f476d92/dotnet-aot.Tests.dll.4/1/console.dc1bce7f.log?helixlogtype=result
      ├─ dotnet-aot.Tests.dll.4 (Job: TestBuild: windows (x64) - windows.amd64.vs2026.pre.scout.open (985b0fb7-6413-42d8-b665-94c4e2ca8210)) (Finished, exit code 8)
      │  └─ Console: https://helixr1107v0xdeko0k025g8.blob.core.windows.net/dotnet-sdk-refs-pull-54950-merge-985b0fb7641342d8b6/dotnet-aot.Tests.dll.4/1/console.d3d4e1e0.log?helixlogtype=result
      ├─ dotnet-aot.Tests.dll.5 (Job: TestBuild: linux (x64) - ubuntu.2204.amd64.open (aa4cd254-201f-476d-9295-c28cff15ebb6)) (Finished, exit code 8)
      │  └─ Console: https://helixr1107v0xdeko0k025g8.blob.core.windows.net/dotnet-sdk-refs-pull-54950-merge-aa4cd254201f476d92/dotnet-aot.Tests.dll.5/1/console.542c484e.log?helixlogtype=result
      ├─ dotnet-aot.Tests.dll.5 (Job: TestBuild: windows (x64) - windows.amd64.vs2026.pre.scout.open (985b0fb7-6413-42d8-b665-94c4e2ca8210)) (Finished, exit code 8)
      │  └─ Console: https://helixr1107v0xdeko0k025g8.blob.core.windows.net/dotnet-sdk-refs-pull-54950-merge-985b0fb7641342d8b6/dotnet-aot.Tests.dll.5/1/console.8f1298b2.log?helixlogtype=result
      ├─ Microsoft.DotNet.Cli.Utils.Tests.dll.1 (Job: TestBuild: windows (x64) - windows.amd64.vs2026.pre.scout.open (985b0fb7-6413-42d8-b665-94c4e2ca8210)) (Finished, exit code 2)
      │  └─ Console: https://helixr1107v0xdeko0k025g8.blob.core.windows.net/dotnet-sdk-refs-pull-54950-merge-985b0fb7641342d8b6/Microsoft.DotNet.Cli.Utils.Tests.dll.1/1/console.803015fb.log?helixlogtype=result
      ├─ Microsoft.NET.Publish.Tests.dll.5 (Job: TestBuild: windows (x64) - windows.amd64.vs2026.pre.scout.open (985b0fb7-6413-42d8-b665-94c4e2ca8210)) (Finished, exit code 2)
      │  └─ Console: https://helixr1107v0xdeko0k025g8.blob.core.windows.net/dotnet-sdk-refs-pull-54950-merge-985b0fb7641342d8b6/Microsoft.NET.Publish.Tests.dll.5/1/console.da3d4d89.log?helixlogtype=result
      ├─ Microsoft.Win32.Msi.Tests.dll (Job: TestBuild: linux (x64) - ubuntu.2204.amd64.open (aa4cd254-201f-476d-9295-c28cff15ebb6)) (Finished, exit code 8)
      │  └─ Console: https://helixr1107v0xdeko0k025g8.blob.core.windows.net/dotnet-sdk-refs-pull-54950-merge-aa4cd254201f476d92/Microsoft.Win32.Msi.Tests.dll/1/console.e7263b43.log?helixlogtype=result
      └─ Microsoft.Win32.Msi.Tests.dll (Job: TestBuild: macOS (arm64) - osx.15.arm64.open (ea0828db-ea88-45be-9830-d1130f5306c9)) (Finished, exit code 8)
         └─ Console: https://helixr1107v0xdeko0k025g8.blob.core.windows.net/dotnet-sdk-refs-pull-54950-merge-ea0828dbea8845be98/Microsoft.Win32.Msi.Tests.dll/1/console.ee0e4d02.log?helixlogtype=result

[Evangelink]

Thanks, this is happening when a test app executes no tests at all (including only skipped tests). There is an issue on tested side to relax when some tests are included but skipped.

In the meantime, we should add the --ignore-exit-code 8 arg.

I'll resume some work tonight to do it if that's not picked up yet.

[nagilson]

https://learn.microsoft.com/en-us/dotnet/core/testing/microsoft-testing-platform-troubleshooting#exit-codes indeed, 8 is for when 0 tests execute. The helix work item and dotnet exec filter may make MTP find 0 runnable tests, or if all tests are 'skipped' due to flakes then it will also cause a failure. Also observed here. #54788

It took me a while to figure out what was going wrong and it was confusing because the test results tab showed everything was green : https://dev.azure.com/dnceng-public/public/_build/results?buildId=1479482&view=ms.vss-test-web.build-test-results-tab

[Evangelink]

I confirm I had the same thing and I might have merged PRs in this state as I was looking mainly st reported errors and build analysis tab.

I wonder if we could also improve helix reporting there.