Repository navigation
Build Analysis misclassifies known Helix test failures after telemetry category removal #17340
Description
Build Analysis misclassifies known Helix test failures after telemetry category removal
Description
Build Analysis can remain red when every failing Helix test is matched to a known issue. The Helix SDK reports each failed test through CheckAzurePipelinesTestResults using Log.LogError(FailureCategory.Test, ...), but these errors now reach the Azure DevOps timeline as plain MSBuild errors:
Microsoft.DotNet.Helix.Sdk/.../AzurePipelines.MultiQueue.targets(44,5): error : Test <test> has failed.
Build Analysis correctly matches the corresponding test results to known issues, but also treats the unclassified timeline messages as independent build failures. The check therefore remains red.
This appears related to:
- Remove the Arcade msbuild logger #16814, which removed
Microsoft.DotNet.ArcadeLoggingand the(NETCORE_ENGINEERING_TELEMETRY=<Category>)decoration. - Restore Arcade msbuild logger as a minimal error/warning-to-timeline translator #17110, which restored a minimal logger that forwards MSBuild errors and warnings to the Azure DevOps timeline, but did not restore failure-category propagation.
The Helix task still supplies FailureCategory.Test; that classification is no longer represented in the timeline message consumed by Build Analysis.
Example
- PR: Remove redundant
fAcquireLockfrom binderSetupBindingPathsruntime#132290 - Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1552976
- Helix SDK:
11.0.0-beta.26381.1from Arcade commit93eebf1a31a5eaafd44326f1a81ca107913e098c
The build had two failed tests:
Loader/ContextualReflection/ContextualReflection/ContextualReflection.cmd, matched to Assert failure: !"OBJECTREF being accessed while thread is in preemptive GC mode." runtime#131925.Loader\classloader\StaticVirtualMethods\Regression\GitHub_130545\GitHub_130545.dll, matched to [ci-scan] Test failure: Loader.classloader.StaticVirtualMethods.Regression.GitHub_130545.DelegateOverStaticAbstractThroughGshare [Content truncated due to length] runtime#132030.
Build Analysis displayed both under Known test errors, but also displayed the AzurePipelines.MultiQueue.targets(44,5) messages under Build Failures. Those duplicate build failures kept the check red.
Expected behavior
Once every failed test is matched to a known issue, the corresponding Helix task-level errors should not independently fail Build Analysis.
Actual behavior
The known test failures are matched, but their plain MSBuild timeline errors remain unmatched and cause Build Analysis to fail.
Possible fixes
Either:
- preserve
FailureCategory.Testin a form Build Analysis can consume when forwarding MSBuild events to the Azure DevOps timeline, or - update Build Analysis to recognize and de-duplicate
CheckAzurePipelinesTestResultserrors against the corresponding failed test results.
Activity
steveisok commentedon Aug 14, 2026
@mmitche is this the right place for this kind of issue?
mmitche commentedon Aug 14, 2026
@steveisok Yep. @ViktorHofer Likely fallout from the category removal. No good deed goes unpunished.
akoeplinger commentedon Aug 14, 2026
we didn't notice this because right after the change landed in dotnet/runtime we enabled the Job Monitor which doesn't hit this issue, but now that it is turned off again we're seeing it
ViktorHofer commentedon Aug 14, 2026
Is the BuildAnalysis integration different with Job Monitor and doesn't rely on these telemetry categories?
mmitche commentedon Aug 17, 2026
No difference in integration (as in, there isn't anything that the monitor does special to integrate with BA). Maybe what is going on is that the submitting leg isn't actually failing, so there's no failure to classify.
missymessa commentedon Sep 9, 2026
Additional investigation from dotnet/runtime#132423 / Build Analysis check 102470876093 shows two related behaviors:
Confirmed: duplicate Helix test failures remain classified as build failures
Build 1587149 reports
baseservices/callconvs/TestCallingConventions/TestCallingConventions.cmdunder Known test errors, matched to dotnet/runtime#132801. The same failures also appear under Build Failures as:This occurs for the Linux, macOS, and Windows legs and keeps the Build Analysis check red. This is direct evidence that the behavior described in this issue still occurs with Helix SDK
11.0.0-beta.26431.109.Related missing known-issue match (cause not yet proven)
The same check leaves
SslStreamTlsResumeTests.DifferentEncryptionPolicy_NoResumefrom build 1586969 under Test Failures, although dotnet/runtime#133418 has a valid known-issue pattern and its automated validator reports successful matches for other builds.The pattern is:
{ "ErrorMessage": "System.Net.Security.Tests.SslStreamTlsResumeTests.DifferentEncryptionPolicy_NoResume [FAIL]", "BuildRetry": false, "ExcludeConsoleLog": false }The full pattern appears in the Helix console, but not in the Azure DevOps test result's
errorMessageorstackTrace. A possible explanation is the documented limit of 100 Helix logs per analysis, but this is a hypothesis, not confirmed. The issue was created within two hours of build 1586969 completing, and a rerun of Build Analysis the following day still did not associate it.The first behavior is a confirmed reproduction of this issue. The second may be a separate matching limitation or defect and likely needs Build Analysis telemetry to determine whether the console was skipped.
missymessa commentedon Sep 9, 2026
Fix is now tracked and in review:
The PR updates Build Analysis to recognize the exact
CheckAzurePipelinesTestResultstimeline message emitted after the telemetry category marker was removed, while limiting de-duplication to jobs that have corresponding test results.nagilson commentedon Sep 25, 2026
https://dev.azure.com/dnceng/internal/_git/dotnet-helix-service/pullrequest/65097 is completed now - should this be closed?