Repository navigation
Build Analysis misclassifies known Helix test failures after telemetry category removal #17340
Description
Activity
@steveisok Yep. @ViktorHofer Likely fallout from the category removal. No good deed goes unpunished.
we didn't notice this because right after the change landed in dotnet/runtime we enabled the Job Monitor which doesn't hit this issue, but now that it is turned off again we're seeing it
Is the BuildAnalysis integration different with Job Monitor and doesn't rely on these telemetry categories?
Is the BuildAnalysis integration different with Job Monitor and doesn't rely on these telemetry categories?
No difference in integration (as in, there isn't anything that the monitor does special to integrate with BA). Maybe what is going on is that the submitting leg isn't actually failing, so there's no failure to classify.
Additional investigation from dotnet/runtime#132423 / Build Analysis check 102470876093 shows two related behaviors:
Confirmed: duplicate Helix test failures remain classified as build failures
Build 1587149 reports
baseservices/callconvs/TestCallingConventions/TestCallingConventions.cmdunder Known test errors, matched to dotnet/runtime#132801. The same failures also appear under Build Failures as:Microsoft.DotNet.Helix.Sdk/.../AzurePipelines.MultiQueue.targets(44,5): error : Test baseservices/callconvs/TestCallingConventions/TestCallingConventions.cmd has failed.This occurs for the Linux, macOS, and Windows legs and keeps the Build Analysis check red. This is direct evidence that the behavior described in this issue still occurs with Helix SDK
11.0.0-beta.26431.109.- PR: JIT: Improve modelling of AsyncResumedDef store in value numbering runtime#132423
- Check: https://github.com/dotnet/runtime/runs/102470876093
- Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1587149
Related missing known-issue match (cause not yet proven)
The same check leaves
SslStreamTlsResumeTests.DifferentEncryptionPolicy_NoResumefrom build 1586969 under Test Failures, although dotnet/runtime#133418 has a valid known-issue pattern and its automated validator reports successful matches for other builds.The pattern is:
{ "ErrorMessage": "System.Net.Security.Tests.SslStreamTlsResumeTests.DifferentEncryptionPolicy_NoResume [FAIL]", "BuildRetry": false, "ExcludeConsoleLog": false }The full pattern appears in the Helix console, but not in the Azure DevOps test result's
errorMessageorstackTrace. A possible explanation is the documented limit of 100 Helix logs per analysis, but this is a hypothesis, not confirmed. The issue was created within two hours of build 1586969 completing, and a rerun of Build Analysis the following day still did not associate it.- Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1586969
- Test result: run 43797732, result 174867
- Known issue: SslStreamTlsResumeTests.DifferentEncryptionPolicy_NoResume failing runtime#133418
The first behavior is a confirmed reproduction of this issue. The second may be a separate matching limitation or defect and likely needs Build Analysis telemetry to determine whether the console was skipped.
Fix is now tracked and in review:
- AzDO work item: https://dev.azure.com/dnceng/internal/_workitems/edit/12650
- PR: https://dev.azure.com/dnceng/internal/_git/dotnet-helix-service/pullrequest/65097
The PR updates Build Analysis to recognize the exact
CheckAzurePipelinesTestResultstimeline message emitted after the telemetry category marker was removed, while limiting de-duplication to jobs that have corresponding test results.https://dev.azure.com/dnceng/internal/_git/dotnet-helix-service/pullrequest/65097 is completed now - should this be closed?
Build Analysis misclassifies known Helix test failures after telemetry category removal
Description
Build Analysis can remain red when every failing Helix test is matched to a known issue. The Helix SDK reports each failed test through
CheckAzurePipelinesTestResultsusingLog.LogError(FailureCategory.Test, ...), but these errors now reach the Azure DevOps timeline as plain MSBuild errors:Build Analysis correctly matches the corresponding test results to known issues, but also treats the unclassified timeline messages as independent build failures. The check therefore remains red.
This appears related to:
Microsoft.DotNet.ArcadeLoggingand the(NETCORE_ENGINEERING_TELEMETRY=<Category>)decoration.The Helix task still supplies
FailureCategory.Test; that classification is no longer represented in the timeline message consumed by Build Analysis.Example
fAcquireLockfrom binderSetupBindingPathsruntime#13229011.0.0-beta.26381.1from Arcade commit93eebf1a31a5eaafd44326f1a81ca107913e098cThe build had two failed tests:
Loader/ContextualReflection/ContextualReflection/ContextualReflection.cmd, matched to Assert failure: !"OBJECTREF being accessed while thread is in preemptive GC mode." runtime#131925.Loader\classloader\StaticVirtualMethods\Regression\GitHub_130545\GitHub_130545.dll, matched to [ci-scan] Test failure: Loader.classloader.StaticVirtualMethods.Regression.GitHub_130545.DelegateOverStaticAbstractThroughGshare [Content truncated due to length] runtime#132030.Build Analysis displayed both under Known test errors, but also displayed the
AzurePipelines.MultiQueue.targets(44,5)messages under Build Failures. Those duplicate build failures kept the check red.Expected behavior
Once every failed test is matched to a known issue, the corresponding Helix task-level errors should not independently fail Build Analysis.
Actual behavior
The known test failures are matched, but their plain MSBuild timeline errors remain unmatched and cause Build Analysis to fail.
Possible fixes
Either:
FailureCategory.Testin a form Build Analysis can consume when forwarding MSBuild events to the Azure DevOps timeline, orCheckAzurePipelinesTestResultserrors against the corresponding failed test results.