Link to Failure log is not always present in Send to Helix logs of the failed CI runs #45357
Description
I've sent a very simple PR and the CI went red: https://github.com/dotnet/runtime/pull/45281/checks?check_run_id=1465351303
So I wanted to find out which tests has failed, but I was unable to find the failed test in the logs
Example of log with no link to the failed tests:
Starting: Send to Helix
==============================================================================
Task : Command line
Description : Run a command line script using Bash on Linux and macOS and cmd.exe on Windows
==============================================================================
Generating script.
Script contents:
./eng/common/msbuild.sh --ci /datadisks/disk1/workspace/_work/1/s/src/libraries/sendtohelix.proj /p:RuntimeFlavor=coreclr /p:TargetArchitecture=x64 /p:TargetRuntimeIdentifier= /p:Configuration=Debug /p:TargetOS=Linux /p:MonoForceInterpreter= /p:TestScope=innerloop /p:TestRunNamePrefixSuffix=CoreCLR_release /p:HelixBuild=20201127.21 /p:Creator=dotnet-bot /p:BuildTargetFramework=net6.0 /bl:/datadisks/disk1/workspace/_work/1/s/artifacts/log/Debug/SendToHelix.binlog
========================== Starting Command Output ===========================
/bin/bash --noprofile --norc /datadisks/disk1/workspace/_work/_temp/b0b3d0f2-1935-442b-be1d-164073b927e8.sh
/datadisks/disk1/workspace/_work/1/s/.dotnet/sdk/5.0.100-rc.2.20479.15/MSBuild.dll /nologo -logger:/datadisks/disk1/workspace/_work/1/s/.packages/microsoft.dotnet.arcade.sdk/6.0.0-beta.20573.2/tools/netcoreapp2.1/Microsoft.DotNet.Arcade.Sdk.dll -maxcpucount /m -verbosity:m /v:minimal /bl:/datadisks/disk1/workspace/_work/1/s/artifacts/log/Debug/SendToHelix.binlog /clp:Summary /nr:false /p:TreatWarningsAsErrors=true /p:ContinuousIntegrationBuild=true /p:RuntimeFlavor=coreclr /p:TargetArchitecture=x64 /p:TargetRuntimeIdentifier= /p:Configuration=Debug /p:TargetOS=Linux /p:MonoForceInterpreter= /p:TestScope=innerloop /p:TestRunNamePrefixSuffix=CoreCLR_release /p:HelixBuild=20201127.21 /p:Creator=dotnet-bot /p:BuildTargetFramework=net6.0 /warnaserror /datadisks/disk1/workspace/_work/1/s/src/libraries/sendtohelix.proj
Using Queues: (Alpine.312.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.12-helix-20200602002622-e06dc59
BuildSettings: net6.0-Linux-Debug-x64
BuildTargetFramework: net6.0
TestArchiveTestsRoot: /datadisks/disk1/workspace/_work/1/s/artifacts/helix/tests/
TestHostRootPath: /datadisks/disk1/workspace/_work/1/s/artifacts/bin/testhost/net6.0-Linux-Debug-x64/
TestArchiveRoot: /datadisks/disk1/workspace/_work/1/s/artifacts/helix/
TestArchiveRuntimeRoot: /datadisks/disk1/workspace/_work/1/s/artifacts/helix/runtime/
TestArchiveRuntimeFile: /datadisks/disk1/workspace/_work/1/s/artifacts/helix/runtime/test-runtime-net6.0-Linux-Debug-x64.zip
Compressing runtime directory
Creating directory /datadisks/disk1/workspace/_work/1/s/artifacts/helix/runtime/
Zipping directory "/datadisks/disk1/workspace/_work/1/s/artifacts/bin/testhost/net6.0-Linux-Debug-x64/" to "/datadisks/disk1/workspace/_work/1/s/artifacts/helix/runtime/test-runtime-net6.0-Linux-Debug-x64.zip".
Correlation directory prepared
Using Queues: (Alpine.312.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.12-helix-20200602002622-e06dc59
Build TargetFramework: net6.0
Building Helix work items
Using TestRunNamePrefix: net6.0-Linux-Debug-x64-CoreCLR_release-
Using HelixCorrelationPayload: /datadisks/disk1/workspace/_work/1/s/artifacts/helix/runtime/test-runtime-net6.0-Linux-Debug-x64.zip
Using HelixCommand: ./RunTests.sh --runtime-path "$HELIX_CORRELATION_PAYLOAD"
Using HelixType: test/functional/cli/innerloop/
Using WorkItemArchiveWildCard: /datadisks/disk1/workspace/_work/1/s/artifacts/helix/tests/**/*.zip
Using Timeout: 00:15:00
Done building Helix work items. Work item count: 240
Starting Azure Pipelines Test Run net6.0-Linux-Debug-x64-CoreCLR_release-(Alpine.312.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.12-helix-20200602002622-e06dc59
Using Queues: (Alpine.312.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.12-helix-20200602002622-e06dc59
Build TargetFramework: net6.0
Building Helix work items
Using TestRunNamePrefix: net6.0-Linux-Debug-x64-CoreCLR_release-
Using HelixCorrelationPayload: /datadisks/disk1/workspace/_work/1/s/artifacts/helix/runtime/test-runtime-net6.0-Linux-Debug-x64.zip
Using HelixCommand: ./RunTests.sh --runtime-path "$HELIX_CORRELATION_PAYLOAD"
Using HelixType: test/functional/cli/innerloop/
Using WorkItemArchiveWildCard: /datadisks/disk1/workspace/_work/1/s/artifacts/helix/tests/**/*.zip
Using Timeout: 00:15:00
Done building Helix work items. Work item count: 240
Uploading payloads for Job on (Alpine.312.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.12-helix-20200602002622-e06dc59...
Finished uploading payloads for Job on (Alpine.312.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.12-helix-20200602002622-e06dc59...
Sending Job to (Alpine.312.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.12-helix-20200602002622-e06dc59...
Sent Helix Job; see work items at https://helix.dot.net/api/jobs/4b3c383d-b7cb-48ce-8347-a56c4d098fe2/workitems?api-version=2019-06-17
Waiting for completion of job 4b3c383d-b7cb-48ce-8347-a56c4d098fe2 on (Alpine.312.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.12-helix-20200602002622-e06dc59
Job 4b3c383d-b7cb-48ce-8347-a56c4d098fe2 on (Alpine.312.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.12-helix-20200602002622-e06dc59 is completed with 241 finished work items.
Stopping Azure Pipelines Test Run net6.0-Linux-Debug-x64-CoreCLR_release-(Alpine.312.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.12-helix-20200602002622-e06dc59
/datadisks/disk1/workspace/_work/1/s/.packages/microsoft.dotnet.helix.sdk/6.0.0-beta.20573.2/tools/azure-pipelines/AzurePipelines.MultiQueue.targets(43,5): error : Test run 28802594 has one or more failing tests. [/datadisks/disk1/workspace/_work/1/s/src/libraries/sendtohelixhelp.proj]
##[error].packages/microsoft.dotnet.helix.sdk/6.0.0-beta.20573.2/tools/azure-pipelines/AzurePipelines.MultiQueue.targets(43,5): error : (NETCORE_ENGINEERING_TELEMETRY=Test) Test run 28802594 has one or more failing tests.
Build FAILED.
/datadisks/disk1/workspace/_work/1/s/.packages/microsoft.dotnet.helix.sdk/6.0.0-beta.20573.2/tools/azure-pipelines/AzurePipelines.MultiQueue.targets(43,5): error : Test run 28802594 has one or more failing tests. [/datadisks/disk1/workspace/_work/1/s/src/libraries/sendtohelixhelp.proj]
0 Warning(s)
1 Error(s)
Time Elapsed 00:11:31.37
##[error]Build failed (exit code '1').
##[error]Bash exited with code '1'.
Finishing: Send to Helix
I've asked @krwq for a sanity check and he found out that WASM builds produce the link to failed tests:
Done building Helix work items for scenario normal. Work item count: 213
Using Queues: Ubuntu.1804.Amd64.Open
Build TargetFramework: net6.0
Building Helix work items for scenario wasmtestonbrowser
Using TestRunNamePrefix: net6.0-Browser-Release-wasm-Mono_Release-wasmtestonbrowser-
Using HelixCorrelationPayload: /__w/1/s/artifacts/helix/runtime/test-runtime-net6.0-Browser-Release-wasm.zip
Using HelixCommand: ./RunTests.sh
Using HelixType: test/functional/cli/innerloop/
Using WorkItemArchiveWildCard: /__w/1/s/artifacts/helix/tests/**/*.zip
Using Timeout: 00:30:00
Done building Helix work items for scenario wasmtestonbrowser. Work item count: 213
Tool 'microsoft.dotnet.xharness.cli' (version '1.0.0-prerelease.20576.1') is already installed.
Tool 'microsoft.dotnet.xharness.cli' failed to install. This failure may have been caused by:
* You are attempting to install a preview release and did not use the --version option to specify the version.
* A package by this name was found, but it was not a .NET tool.
* The required NuGet feed cannot be accessed, perhaps because of an Internet connection problem.
* You mistyped the name of the tool.
For more reasons, including package naming enforcement, visit https://aka.ms/failure-installing-tool
Uploading payloads for Job on Ubuntu.1804.Amd64.Open...
Finished uploading payloads for Job on Ubuntu.1804.Amd64.Open...
Sending Job to Ubuntu.1804.Amd64.Open...
You can invoke the tool using the following command: xharness
Tool 'microsoft.dotnet.xharness.cli' (version '1.0.0-prerelease.20576.1') was successfully installed.
Uploading payloads for Job on Ubuntu.1804.Amd64.Open...
Finished uploading payloads for Job on Ubuntu.1804.Amd64.Open...
Sending Job to Ubuntu.1804.Amd64.Open...
Sent Helix Job; see work items at https://helix.dot.net/api/jobs/8c6ddb22-9b5b-40b9-b699-2f481727aa56/workitems?api-version=2019-06-17
Sent Helix Job; see work items at https://helix.dot.net/api/jobs/8733ee54-dcfe-404e-8b0d-adef4ad65c7c/workitems?api-version=2019-06-17
Waiting for completion of job 8c6ddb22-9b5b-40b9-b699-2f481727aa56 on Ubuntu.1804.Amd64.Open
Waiting for completion of job 8733ee54-dcfe-404e-8b0d-adef4ad65c7c on Ubuntu.1804.Amd64.Open
Job 8c6ddb22-9b5b-40b9-b699-2f481727aa56 on Ubuntu.1804.Amd64.Open is completed with 214 finished work items.
Stopping Azure Pipelines Test Run net6.0-Browser-Release-wasm-Mono_Release-wasmtestonbrowser-Ubuntu.1804.Amd64.Open
/__w/1/s/.packages/microsoft.dotnet.helix.sdk/6.0.0-beta.20573.2/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(76,5): error : Work item System.Runtime.Tests in job 8c6ddb22-9b5b-40b9-b699-2f481727aa56 has failed. [/__w/1/s/src/libraries/sendtohelixhelp.proj]
/__w/1/s/.packages/microsoft.dotnet.helix.sdk/6.0.0-beta.20573.2/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(76,5): error : Failure log: https://helix.dot.net/api/2019-06-17/jobs/8c6ddb22-9b5b-40b9-b699-2f481727aa56/workitems/System.Runtime.Tests/console [/__w/1/s/src/libraries/sendtohelixhelp.proj]
##[error].packages/microsoft.dotnet.helix.sdk/6.0.0-beta.20573.2/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(76,5): error : (NETCORE_ENGINEERING_TELEMETRY=Test) Work item System.Runtime.Tests in job 8c6ddb22-9b5b-40b9-b699-2f481727aa56 has failed.
Failure log: https://helix.dot.net/api/2019-06-17/jobs/8c6ddb22-9b5b-40b9-b699-2f481727aa56/workitems/System.Runtime.Tests/console
Job 8733ee54-dcfe-404e-8b0d-adef4ad65c7c on Ubuntu.1804.Amd64.Open is completed with 214 finished work items.
Stopping Azure Pipelines Test Run net6.0-Browser-Release-wasm-Mono_Release-normal-Ubuntu.1804.Amd64.Open
/__w/1/s/.packages/microsoft.dotnet.helix.sdk/6.0.0-beta.20573.2/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(76,5): error : Work item System.Runtime.Tests in job 8733ee54-dcfe-404e-8b0d-adef4ad65c7c has failed. [/__w/1/s/src/libraries/sendtohelixhelp.proj]
/__w/1/s/.packages/microsoft.dotnet.helix.sdk/6.0.0-beta.20573.2/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(76,5): error : Failure log: https://helix.dot.net/api/2019-06-17/jobs/8733ee54-dcfe-404e-8b0d-adef4ad65c7c/workitems/System.Runtime.Tests/console [/__w/1/s/src/libraries/sendtohelixhelp.proj]
##[error].packages/microsoft.dotnet.helix.sdk/6.0.0-beta.20573.2/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(76,5): error : (NETCORE_ENGINEERING_TELEMETRY=Test) Work item System.Runtime.Tests in job 8733ee54-dcfe-404e-8b0d-adef4ad65c7c has failed.
Failure log: https://helix.dot.net/api/2019-06-17/jobs/8733ee54-dcfe-404e-8b0d-adef4ad65c7c/workitems/System.Runtime.Tests/console
Build FAILED.
/__w/1/s/.packages/microsoft.dotnet.helix.sdk/6.0.0-beta.20573.2/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(76,5): error : Work item System.Runtime.Tests in job 8c6ddb22-9b5b-40b9-b699-2f481727aa56 has failed. [/__w/1/s/src/libraries/sendtohelixhelp.proj]
/__w/1/s/.packages/microsoft.dotnet.helix.sdk/6.0.0-beta.20573.2/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(76,5): error : Failure log: https://helix.dot.net/api/2019-06-17/jobs/8c6ddb22-9b5b-40b9-b699-2f481727aa56/workitems/System.Runtime.Tests/console [/__w/1/s/src/libraries/sendtohelixhelp.proj]
/__w/1/s/.packages/microsoft.dotnet.helix.sdk/6.0.0-beta.20573.2/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(76,5): error : Work item System.Runtime.Tests in job 8733ee54-dcfe-404e-8b0d-adef4ad65c7c has failed. [/__w/1/s/src/libraries/sendtohelixhelp.proj]
/__w/1/s/.packages/microsoft.dotnet.helix.sdk/6.0.0-beta.20573.2/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(76,5): error : Failure log: https://helix.dot.net/api/2019-06-17/jobs/8733ee54-dcfe-404e-8b0d-adef4ad65c7c/workitems/System.Runtime.Tests/console [/__w/1/s/src/libraries/sendtohelixhelp.proj]
0 Warning(s)
2 Error(s)
Time Elapsed 00:33:42.17
##[error]Build failed (exit code '1').
##[error]Bash exited with code '1'.
Finishing: Send to Helix
cc @ViktorHofer