New structured log output for Apple and Android platforms by matouskozak · Pull Request #1579 · dotnet/xharness

matouskozak · 2026-03-31T09:27:42Z

Problem

When investigating test failures in Helix, the XHarness console output lacked a consistent, structured summary of what happened during a run. Key information — exit codes, device details, produced files, and Helix URLs — was scattered across verbose log output, making it difficult for both humans and automated tooling to quickly parse results.

Additionally, Android logcat output was dumped in its entirety to the console log, burying the relevant DOTNET-tagged entries in thousands of lines of system noise.

Changes

New: `RunSummaryEmitter` (shared by both platforms)

Introduces a unified RunSummaryEmitter in Microsoft.DotNet.XHarness.Common that emits a structured JSON block delimited by <<XHARNESS_RESULT_START>> / <<XHARNESS_RESULT_END>> markers at the end of every run. The JSON contains:

Exit code and its name
Machine name and platform
Device info (name, OS version, architecture)
Manifest of produced files (test results, logs, etc.)
Helix job/work item URLs when running in Helix (for direct log access)

Both Android and Apple platforms use this shared emitter, ensuring consistent output across all run types.

Android improvements

Logcat filtering: Console output now shows only DOTNET-tagged logcat lines (the full unfiltered log is still written to the output file)
File tracking: InstrumentationRunner tracks all produced files (test results XMLs, logcat, bugreport) via DiagnosticsFile and includes them in the summary
Device metadata: Summary includes device serial, API version, and architecture (e.g., arm64-v8a, x86)

Apple improvements

CopyLogsToMainLog refactor: Now runs for all targets including simulators (previously skipped). MacCatalyst correctly copies SystemLog instead of ApplicationLog since MacCatalyst apps run as native macOS processes
Log collection: BaseOrchestrator collects all file-backed logs into the diagnostics summary with their types (executionlog, testlog, systemlog, applicationlog, xmllog, etc.)
MacCatalyst diagnostics: Populates device info with local machine name and macOS version

Common infrastructure

New DiagnosticsFile model in CommandDiagnostics.cs for tracking produced files
IDiagnosticsData.Files property added for file manifest support
xharness-result.json written to output directory for downstream tooling

Example output

Android device:

<<XHARNESS_RESULT_START>>
{
  "version": 1,
  "machineName": "DNCENGWIN-068",
  "exitCode": 0,
  "exitCodeName": "SUCCESS",
  "platform": "android",
  "helixWorkItemId": "System.Buffers.Tests-arm64-v8a",
  "helixJobId": "3cdb9380-8d1c-44af-89a8-caf90c38c09b",
  "helixConsoleUri": "https://helix.dot.net/api/2019-06-17/jobs/.../console",
  "helixFilesUri": "https://helix.dot.net/api/2019-06-17/jobs/.../files",
  "instrumentationExitCode": 0,
  "device": "09171JEC213198",
  "deviceOsVersion": "API 33",
  "architecture": "arm64-v8a",
  "files": [
    { "name": "testResults.xml", "type": "test-results" },
    { "name": "adb-logcat-net.dot.System.Buffers.Tests-net.dot.MonoRunner.log", "type": "logcat" }
  ]
}
<<XHARNESS_RESULT_END>>

Apple iOS simulator:

<<XHARNESS_RESULT_START>>
{
  "version": 1,
  "machineName": "dci-mac-build-342",
  "exitCode": 0,
  "exitCodeName": "SUCCESS",
  "platform": "apple",
  "device": "iPhone 11 Pro (iOS 18.1) - created by XHarness",
  "deviceOsVersion": "18.1",
  "files": [
    { "name": "test-ios-simulator-64.log", "type": "executionlog" },
    { "name": "System.Numerics.Vectors.Tests.log", "type": "systemlog" },
    { "name": "net.dot.System.Numerics.Vectors.Tests.log", "type": "applicationlog" },
    { "name": "xunit-test-ios-simulator-64-20260331_070927.xml", "type": "xmllog" }
  ]
}
<<XHARNESS_RESULT_END>>

Testing

AdbRunnerLogFilterTests — validates DOTNET line filtering from logcat output
InstrumentationRunnerSummaryTests — validates Android file tracking and JSON summary emission
CopyLogsToMainLogTests — validates Apple log copying behavior for simulators, devices, and MacCatalyst

Add structured Android run summaries and filter logcat output

Copilot

Pull request overview

Adds a shared, structured “run summary” output at the end of Apple and Android runs to make Helix investigations and automated parsing easier, while also reducing Android console noise by filtering logcat output.

Changes:

Introduces RunSummaryEmitter (Common) to emit a JSON result block delimited by <<XHARNESS_RESULT_START>> / <<XHARNESS_RESULT_END>> and write xharness-result.json.
Extends diagnostics infrastructure with DiagnosticsFile + IDiagnosticsData.Files to track produced artifacts.
Updates Android (InstrumentationRunner, AdbRunner) and Apple (BaseOrchestrator, TestOrchestrator) to emit summaries and improve log handling; adds targeted unit tests.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
tests/Microsoft.DotNet.XHarness.Apple.Tests/Orchestration/CopyLogsToMainLogTests.cs	New tests validating which Apple logs get copied into the main log (simulator/device vs MacCatalyst).
tests/Microsoft.DotNet.XHarness.Android.Tests/InstrumentationRunnerSummaryTests.cs	New tests validating JSON summary emission content (exit code, platform, device info, files, Helix URLs).
tests/Microsoft.DotNet.XHarness.Android.Tests/AdbRunnerLogFilterTests.cs	New tests validating logcat filtering to DOTNET-tagged lines.
src/Microsoft.DotNet.XHarness.Common/RunSummaryEmitter.cs	New shared emitter for structured JSON result block + file output.
src/Microsoft.DotNet.XHarness.Common/CommandDiagnostics.cs	Adds `DiagnosticsFile` model and `IDiagnosticsData.Files` to track produced files.
src/Microsoft.DotNet.XHarness.Apple/Orchestration/TestOrchestrator.cs	Copies the appropriate log type into the main log (ApplicationLog generally; SystemLog for MacCatalyst).
src/Microsoft.DotNet.XHarness.Apple/Orchestration/BaseOrchestrator.cs	Ensures Apple run summary emission happens (finally) and collects file-backed logs into the summary.
src/Microsoft.DotNet.XHarness.Android/InstrumentationRunner.cs	Tracks produced files and emits Android run summary + writes `xharness-result.json`.
src/Microsoft.DotNet.XHarness.Android/AdbRunner.cs	Filters console logcat output to DOTNET-tagged lines while keeping the full log in a file.

Comments suppressed due to low confidence (1)

src/Microsoft.DotNet.XHarness.Android/InstrumentationRunner.cs:222

PullResultXMLs returns a value that is assigned to failurePullingFiles, but the local variable is named success and is set to true in the exception path. This makes the control flow hard to reason about and easy to break in future edits. Rename the local to reflect what it actually represents (e.g., failurePullingFiles) and keep the return semantics consistent.

    private bool PullResultXMLs(string apkPackageName, string outputDirectory, IReadOnlyDictionary<string, string> resultValues, List<DiagnosticsFile> producedFiles)
    {
        bool success = false;

        foreach (string possibleResultKey in s_xmlOutputVariableNames)
        {
            if (!resultValues.TryGetValue(possibleResultKey, out string? resultFile))
            {
                continue;
            }

            _logger.LogInformation($"Found XML result file: '{resultFile}'(key: {possibleResultKey})");

            try
            {
                _runner.PullFiles(apkPackageName, resultFile, outputDirectory);
                producedFiles.Add(new DiagnosticsFile
                {
                    Name = Path.GetFileName(resultFile),
                    Type = "test-results",
                    Path = Path.Combine(outputDirectory, Path.GetFileName(resultFile)),
                });
            }
            catch (Exception toLog)
            {
                _logger.LogError(toLog, "Hit error (typically permissions) trying to pull {filePathOnDevice}", resultFile);
                success = true;
            }
        }

        return success;
    }

kotlarmilos

LGTM, thanks!

vitek-karas

Looks great - I didn't look at the code in detail, just the overall design and ideas though.

matouskozak added 4 commits March 26, 2026 11:27

Improve Android test diagnostics output

5757dcd

Add structured Android run summaries and filter logcat output

[WIP] Apple output log refactor

470a744

Unify Apple and Android summary logging

31800b7

CopyLogToMainLogTests

9bea51e

matouskozak self-assigned this Mar 31, 2026

matouskozak changed the title ~~[WIP] New log output structure~~ New structured log output for Apple and Android platforms Apr 1, 2026

remove human summary

176d705

matouskozak requested a review from Copilot April 1, 2026 10:56

Copilot started reviewing on behalf of matouskozak April 1, 2026 10:57 View session

Copilot AI reviewed Apr 1, 2026

View reviewed changes

matouskozak added 2 commits April 1, 2026 12:26

Address copilot review

d39e495

restore env variables in test

3272dd0

matouskozak marked this pull request as ready for review April 1, 2026 17:10

matouskozak requested review from akoeplinger, kotlarmilos and vitek-karas April 1, 2026 17:10

kotlarmilos approved these changes Apr 2, 2026

View reviewed changes

Comment thread src/Microsoft.DotNet.XHarness.Apple/Orchestration/BaseOrchestrator.cs

vitek-karas approved these changes Apr 2, 2026

View reviewed changes

matouskozak merged commit e396a85 into dotnet:main Apr 2, 2026
17 checks passed

matouskozak deleted the new-log-output-structure branch April 2, 2026 12:26

vitek-karas mentioned this pull request Apr 7, 2026

Add --enable-coverage flag for code coverage collection from device tests #1565

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New structured log output for Apple and Android platforms#1579

New structured log output for Apple and Android platforms#1579
matouskozak merged 7 commits intodotnet:mainfrom
matouskozak:new-log-output-structure

matouskozak commented Mar 31, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kotlarmilos left a comment

Uh oh!

Uh oh!

vitek-karas left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

matouskozak commented Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Changes

New: RunSummaryEmitter (shared by both platforms)

Android improvements

Apple improvements

Common infrastructure

Example output

Testing

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kotlarmilos left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vitek-karas left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

matouskozak commented Mar 31, 2026 •

edited

Loading

New: `RunSummaryEmitter` (shared by both platforms)