Skip to content

Commit 42c80a7

Browse files
Andy-Jostclaude
andcommitted
ci: put the test reports on the run's Summary page and post our own comment
Check runs created from inside a workflow are filed under the oldest workflow that ran on the commit. On a copy-pr-bot mirror that is always one of the fork PR's own workflows, so both "Test Results" checks showed up under the assignee/label check. Neither reporter can choose the suite, so neither creates a check run any more: EnricoMi writes its counts to the job summary, dorny writes its per-configuration report there too, and a short "how to read" paragraph precedes both. The PR comment is now ours: ci/tools/junit_comment.py renders a plain English summary from the same JUnit files (configurations, tests that did not pass and where, execution totals, a link to the Summary page) and marocchino/sticky-pull-request-comment posts it with recreate, so it reappears at the bottom of the thread on every run. Co-Authored-By: Claude Fable 5.1 <[email protected]>
1 parent ce642a7 commit 42c80a7

3 files changed

Lines changed: 403 additions & 18 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 60 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -662,7 +662,6 @@ jobs:
662662
runs-on: ubuntu-latest
663663
permissions:
664664
actions: read # download the test-results artifacts
665-
checks: write # publish the "Test Results" checks
666665
contents: read
667666
pull-requests: write # test-results comment on the PR
668667
needs:
@@ -683,9 +682,19 @@ jobs:
683682
- doc
684683
- precommit-windows
685684
steps:
686-
# Merge every test job's JUnit XML into two checks on the commit. Both
687-
# steps are informational: continue-on-error keeps them from changing
688-
# the gate below, and neither fails on test failures.
685+
# Merge every test job's JUnit XML into two reports on this run's Summary
686+
# page, and post one comment on the PR that links there. Nothing here
687+
# creates a check run: GitHub files API-created checks under the oldest
688+
# workflow that ran on the commit, which on a copy-pr-bot mirror is never
689+
# CI. Every step is informational: continue-on-error keeps it away from
690+
# the gate below, and none of them fails on test failures.
691+
- name: Checkout repository
692+
if: ${{ needs.should-skip.outputs.skip != 'true' }}
693+
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
694+
with:
695+
fetch-depth: 1
696+
persist-credentials: false
697+
689698
- name: Download test results
690699
if: ${{ needs.should-skip.outputs.skip != 'true' }}
691700
continue-on-error: true
@@ -694,9 +703,22 @@ jobs:
694703
pattern: test-results-*
695704
path: test-results
696705

706+
- name: Explain the test reports
707+
if: ${{ needs.should-skip.outputs.skip != 'true' }}
708+
continue-on-error: true
709+
run: |
710+
cat >> "$GITHUB_STEP_SUMMARY" <<'EOF'
711+
## How to read the test reports below
712+
713+
CI runs the test suites once per configuration: a platform, a Python version, a CUDA version, and a GPU. The first report merges every configuration, so its counts are test executions, and a test that fails on three configurations counts three times. The second report has one table per configuration and lists the failing tests under each.
714+
715+
A test *fails* when an assertion or exception happens inside the test body. A test *errors* when the exception happens outside it: fixture setup or teardown, or collection.
716+
EOF
717+
697718
- name: Publish test results (aggregate)
698719
# One entry per distinct test across the whole matrix, with the
699-
# number of runs that failed; also posts a PR comment on failures.
720+
# number of runs that failed. Job summary only: no check run and no
721+
# comment of its own (the comment below is ours).
700722
if: ${{ needs.should-skip.outputs.skip != 'true' }}
701723
continue-on-error: true
702724
uses: EnricoMi/publish-unit-test-result-action@d0a4676d0e0b938bc201470d88276b7c74c712b3 # v2.24.0
@@ -707,17 +729,13 @@ jobs:
707729
# by message text, and platform-specific paths in the traceback split
708730
# one failure into several "k out of N runs" entries.
709731
report_individual_runs: false
710-
check_run_annotations: none
711-
comment_mode: failures
732+
check_run: false
733+
comment_mode: "off"
712734
compare_to_earlier_commit: false
713-
# The run is on copy-pr-bot's pull-request/N mirror of the PR head, so
714-
# the PR lives in a fork: find it by commit search, not by branch.
715-
search_pull_requests: true
716-
pull_request_build: commit
717735

718736
- name: Publish test results (by configuration)
719737
# One section per test-results file, so a failure is listed under
720-
# the job configuration that produced it.
738+
# the job configuration that produced it. Job summary only.
721739
if: ${{ needs.should-skip.outputs.skip != 'true' }}
722740
continue-on-error: true
723741
uses: dorny/test-reporter@a6ddd83ac95ff4586f5d3aceeb314d9a1841db95 # v3.0.0
@@ -729,13 +747,37 @@ jobs:
729747
list-suites: failed
730748
list-tests: failed
731749
max-annotations: 0
732-
# The check turns red when any test failed; the step's continue-on-error
733-
# keeps that away from the gate below.
734-
fail-on-error: true
750+
fail-on-error: false
735751
fail-on-empty: false
736-
# false = publish as a check run on the commit; true writes the same
737-
# report to the job summary instead and creates no check.
738-
use-actions-summary: false
752+
# true writes the report to the job summary and creates no check run.
753+
use-actions-summary: true
754+
755+
# One comment on the PR, in our own words, recreated at the bottom of the
756+
# thread on every run. Only mirror branches have a PR to comment on.
757+
- name: Get PR number
758+
if: ${{ needs.should-skip.outputs.skip != 'true' && startsWith(github.ref_name, 'pull-request/') }}
759+
continue-on-error: true
760+
uses: ./.github/actions/get_pr_number
761+
762+
- name: Write the test-results comment
763+
if: ${{ needs.should-skip.outputs.skip != 'true' && startsWith(github.ref_name, 'pull-request/') }}
764+
continue-on-error: true
765+
run: |
766+
python ci/tools/junit_comment.py \
767+
--results test-results \
768+
--commit "${{ github.sha }}" \
769+
--run-url "${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}" \
770+
--output "${RUNNER_TEMP}/test-results-comment.md"
771+
772+
- name: Post the test-results comment
773+
if: ${{ needs.should-skip.outputs.skip != 'true' && startsWith(github.ref_name, 'pull-request/') && env.PR_NUMBER != '' }}
774+
continue-on-error: true
775+
uses: marocchino/sticky-pull-request-comment@d4d6b0936434b21bc8345ad45a440c5f7d2c40ff # v3.0.3
776+
with:
777+
header: test-results
778+
number: ${{ env.PR_NUMBER }}
779+
path: ${{ runner.temp }}/test-results-comment.md
780+
recreate: true
739781

740782
- name: Exit
741783
env:

‎ci/tools/junit_comment.py‎

Lines changed: 177 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,177 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
"""Write a plain-English pull-request comment from the JUnit files of a CI run.
5+
6+
Reads every ``*.xml`` under the test-results directory that the final CI job
7+
downloads, one subdirectory per test job, and writes a short Markdown comment:
8+
how many configurations ran, which tests did not pass and where, the execution
9+
totals, and a link to the run's Summary page where the full reports live.
10+
11+
Usage::
12+
13+
python ci/tools/junit_comment.py --results test-results \\
14+
--commit <sha> --run-url <url> --output comment.md
15+
"""
16+
17+
from __future__ import annotations
18+
19+
import argparse
20+
import sys
21+
import xml.etree.ElementTree as ET
22+
from dataclasses import dataclass, field
23+
from pathlib import Path
24+
25+
ARTIFACT_PREFIX = "test-results-"
26+
STANDARD_MODE_PREFIX = "standard-"
27+
MAX_LISTED = 25
28+
29+
30+
@dataclass
31+
class Totals:
32+
passed: int = 0
33+
failed: int = 0
34+
errors: int = 0
35+
skipped: int = 0
36+
37+
38+
@dataclass
39+
class Results:
40+
files: int = 0
41+
configurations: dict[str, Totals] = field(default_factory=dict)
42+
# test id -> {configuration: "failed" | "error"} for tests that did not pass somewhere
43+
not_passed: dict[str, dict[str, str]] = field(default_factory=dict)
44+
45+
46+
def configuration_label(results_dir: Path, xml_path: Path) -> str:
47+
"""The test job's configuration, taken from the artifact directory name."""
48+
parts = xml_path.relative_to(results_dir).parts
49+
name = parts[0] if len(parts) > 1 else ""
50+
if name.startswith(ARTIFACT_PREFIX):
51+
name = name[len(ARTIFACT_PREFIX) :]
52+
if name.startswith(STANDARD_MODE_PREFIX):
53+
name = name[len(STANDARD_MODE_PREFIX) :]
54+
return name or "(unknown configuration)"
55+
56+
57+
def outcome(case: ET.Element) -> str:
58+
"""JUnit: <error> is an exception outside the test body, <failure> one inside it."""
59+
tags = {child.tag for child in case}
60+
if "error" in tags:
61+
return "error"
62+
if "failure" in tags:
63+
return "failed"
64+
if "skipped" in tags:
65+
return "skipped"
66+
return "passed"
67+
68+
69+
def test_id(case: ET.Element) -> str:
70+
classname = case.get("classname") or ""
71+
name = case.get("name") or "(unnamed test)"
72+
return f"{classname}::{name}" if classname else name
73+
74+
75+
def collect(results_dir: Path) -> Results:
76+
results = Results()
77+
for xml_path in sorted(results_dir.rglob("*.xml")):
78+
label = configuration_label(results_dir, xml_path)
79+
totals = results.configurations.setdefault(label, Totals())
80+
results.files += 1
81+
# The files are pytest's own output from this run, not untrusted input.
82+
for case in ET.parse(xml_path).getroot().iter("testcase"): # noqa: S314
83+
state = outcome(case)
84+
if state == "passed":
85+
totals.passed += 1
86+
elif state == "failed":
87+
totals.failed += 1
88+
elif state == "error":
89+
totals.errors += 1
90+
else:
91+
totals.skipped += 1
92+
if state in ("failed", "error"):
93+
results.not_passed.setdefault(test_id(case), {})[label] = state
94+
return results
95+
96+
97+
def plural(count: int, word: str) -> str:
98+
return f"{count:,} {word}" if count == 1 else f"{count:,} {word}s"
99+
100+
101+
def render(results: Results, commit: str, run_url: str) -> str:
102+
lines = [f"## Test results for commit {commit[:7]}", ""]
103+
if results.files == 0:
104+
lines += ["No test result files were found for this run.", "", f"Run: {run_url}"]
105+
return "\n".join(lines) + "\n"
106+
107+
n_conf = len(results.configurations)
108+
configurations = sorted(results.configurations)
109+
lines.append(f"{plural(n_conf, 'configuration')} ran the test suites: {'; '.join(configurations)}.")
110+
lines.append("")
111+
112+
passed = sum(t.passed for t in results.configurations.values())
113+
failed = sum(t.failed for t in results.configurations.values())
114+
errors = sum(t.errors for t in results.configurations.values())
115+
skipped = sum(t.skipped for t in results.configurations.values())
116+
117+
if not results.not_passed:
118+
lines.append(
119+
f"No test failed. Counting every test execution across all configurations: "
120+
f"{passed:,} passed, {skipped:,} skipped."
121+
)
122+
else:
123+
errored = {t for t, where in results.not_passed.items() if "error" in where.values()}
124+
only_failed = len(results.not_passed) - len(errored)
125+
lines.append(
126+
f"{plural(len(results.not_passed), 'test')} did not pass in at least one configuration. "
127+
f"{only_failed} failed inside the test body (an assertion or an exception). "
128+
f"{len(errored)} hit an error outside the test body (fixture setup or teardown, or collection)."
129+
)
130+
lines.append("")
131+
lines.append(
132+
f"Counting every test execution across all configurations: "
133+
f"{passed:,} passed, {failed:,} failed, {errors:,} errored, {skipped:,} skipped."
134+
)
135+
lines.append("")
136+
lines.append("Tests that did not pass:")
137+
lines.append("")
138+
listed = sorted(results.not_passed)
139+
for name in listed[:MAX_LISTED]:
140+
where = results.not_passed[name]
141+
state = "error" if "error" in where.values() else "failed"
142+
if n_conf > 1 and len(where) == n_conf:
143+
place = f"in all {n_conf} configurations"
144+
else:
145+
place = "in " + ", ".join(sorted(where))
146+
lines.append(f"- `{name}`: {state} {place}")
147+
if len(listed) > MAX_LISTED:
148+
lines.append(f"- and {len(listed) - MAX_LISTED:,} more")
149+
150+
lines.append("")
151+
lines.append(f"The per-configuration tables and the failing-test details are on the run's Summary page: {run_url}")
152+
lines.append("")
153+
lines.append(
154+
"A test *fails* when an assertion or exception happens inside the test body. "
155+
"A test *errors* when the exception happens outside it: fixture setup or teardown, or collection."
156+
)
157+
return "\n".join(lines) + "\n"
158+
159+
160+
def main(argv: list[str] | None = None) -> int:
161+
parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
162+
parser.add_argument(
163+
"--results", required=True, type=Path, help="directory holding the downloaded test-results-* artifacts"
164+
)
165+
parser.add_argument("--commit", required=True, help="commit SHA the results belong to")
166+
parser.add_argument("--run-url", required=True, help="URL of the workflow run whose Summary page holds the reports")
167+
parser.add_argument("--output", required=True, type=Path, help="file to write the Markdown comment to")
168+
args = parser.parse_args(argv)
169+
170+
text = render(collect(args.results), args.commit, args.run_url)
171+
args.output.write_text(text, encoding="utf-8")
172+
sys.stdout.write(text)
173+
return 0
174+
175+
176+
if __name__ == "__main__":
177+
raise SystemExit(main())

0 commit comments

Comments
 (0)