Skip to content

Commit 9b5bb58

Browse files
Andy-Jostclaude
Andy-Jost
and
claude
committed
ci: demo an Allure test report published to GitHub Pages
The final job renders every test job's JUnit XML as an Allure Report 3 report and publishes it to docs/test-report/pr-N/ on the gh-pages branch, the way the docs previews are published, with a sticky comment that links to it. ci/tools/allure_prepare.py tags every suite with its CI configuration and writes an allurerc.mjs that maps the tags to Allure environments; without that, Allure folds the same test from every configuration into retries of one result. Allure is a CLI from npm run as a plain step, so the actions allowlist does not apply, and it reads JUnit XML directly. Every step is continue-on-error; the gate is unchanged. Co-Authored-By: Claude Fable 5.1 <[email protected]>
1 parent 0bee809 commit 9b5bb58

3 files changed

Lines changed: 300 additions & 0 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -660,6 +660,10 @@ jobs:
660660
name: Check job status
661661
if: ${{ always() && github.repository_owner == 'nvidia' }}
662662
runs-on: ubuntu-latest
663+
permissions:
664+
actions: read # download the test-results artifacts
665+
contents: write # publish the report to the gh-pages branch
666+
pull-requests: write # sticky comment with the report link
663667
needs:
664668
- ci-vars
665669
- should-skip
@@ -678,6 +682,102 @@ jobs:
678682
- doc
679683
- precommit-windows
680684
steps:
685+
# DEMO: render every test job's JUnit XML as an Allure report and publish
686+
# it to GitHub Pages next to the docs previews. Every step is
687+
# informational: continue-on-error keeps it away from the gate below, and
688+
# none of them fails on test failures.
689+
- name: Checkout repository
690+
if: ${{ needs.should-skip.outputs.skip != 'true' }}
691+
continue-on-error: true
692+
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
693+
with:
694+
fetch-depth: 1
695+
696+
- name: Download test results
697+
if: ${{ needs.should-skip.outputs.skip != 'true' }}
698+
continue-on-error: true
699+
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
700+
with:
701+
pattern: test-results-*
702+
path: test-results
703+
704+
- name: Prepare the Allure results
705+
# One flat directory of JUnit files, each suite tagged with its CI
706+
# configuration, plus an allurerc.mjs that maps the tags to Allure
707+
# environments so the configurations stay apart in the report.
708+
if: ${{ needs.should-skip.outputs.skip != 'true' }}
709+
continue-on-error: true
710+
run: |
711+
python3 ci/tools/allure_prepare.py \
712+
--results test-results \
713+
--output allure-results \
714+
--config allurerc.mjs \
715+
--report-name "cuda-python CI test results (${{ github.sha }})"
716+
717+
- name: Generate the Allure report
718+
# Allure Report 3 is a CLI from npm, run here as a plain command.
719+
if: ${{ needs.should-skip.outputs.skip != 'true' }}
720+
continue-on-error: true
721+
run: |
722+
npx --yes [email protected] generate allure-results -o allure-report
723+
size=$(du -sh allure-report | cut -f1)
724+
files=$(find allure-report -type f | wc -l)
725+
compressed=$(tar czf - allure-report | wc -c | awk '{ printf "%.0f MB", $1 / 1048576 }')
726+
echo "ALLURE_REPORT_SIZE=${size}" >> "$GITHUB_ENV"
727+
echo "ALLURE_REPORT_FILES=${files}" >> "$GITHUB_ENV"
728+
echo "ALLURE_REPORT_COMPRESSED=${compressed}" >> "$GITHUB_ENV"
729+
{
730+
echo "## Allure report"
731+
echo
732+
echo "Size on disk: ${size} in ${files} files (${compressed} compressed)."
733+
echo
734+
echo '```json'
735+
cat allure-report/widgets/statistic.json
736+
echo
737+
echo '```'
738+
} >> "$GITHUB_STEP_SUMMARY"
739+
740+
- name: Upload the Allure report
741+
if: ${{ needs.should-skip.outputs.skip != 'true' }}
742+
continue-on-error: true
743+
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
744+
with:
745+
name: allure-report
746+
path: allure-report/
747+
if-no-files-found: ignore
748+
retention-days: 14
749+
750+
# Only mirror branches have a PR to publish for and to comment on.
751+
- name: Get PR number
752+
if: ${{ needs.should-skip.outputs.skip != 'true' && startsWith(github.ref_name, 'pull-request/') }}
753+
continue-on-error: true
754+
uses: ./.github/actions/get_pr_number
755+
756+
- name: Publish the Allure report to GitHub Pages
757+
# Same mechanism as the docs preview: a folder in the gh-pages branch,
758+
# served at https://nvidia.github.io/cuda-python/test-report/pr-N/.
759+
if: ${{ needs.should-skip.outputs.skip != 'true' && startsWith(github.ref_name, 'pull-request/') && env.PR_NUMBER != '' }}
760+
continue-on-error: true
761+
uses: JamesIves/github-pages-deploy-action@fa24774553152dd7873cd16ebd8d959b010c5445 # v4.9.0
762+
with:
763+
git-config-name: cuda-python-bot
764+
git-config-email: [email protected]
765+
folder: allure-report
766+
target-folder: docs/test-report/pr-${{ env.PR_NUMBER }}/
767+
commit-message: "Deploy test report for PR ${{ env.PR_NUMBER }} (${{ github.sha }})"
768+
769+
- name: Post the test-report comment
770+
if: ${{ needs.should-skip.outputs.skip != 'true' && startsWith(github.ref_name, 'pull-request/') && env.PR_NUMBER != '' }}
771+
continue-on-error: true
772+
uses: marocchino/sticky-pull-request-comment@d4d6b0936434b21bc8345ad45a440c5f7d2c40ff # v3.0.3
773+
with:
774+
header: test-report
775+
number: ${{ env.PR_NUMBER }}
776+
message: |
777+
Allure test report for ${{ github.sha }}: https://nvidia.github.io/cuda-python/test-report/pr-${{ env.PR_NUMBER }}/
778+
779+
The report is ${{ env.ALLURE_REPORT_SIZE }} on disk in ${{ env.ALLURE_REPORT_FILES }} files (${{ env.ALLURE_REPORT_COMPRESSED }} compressed), all committed to the gh-pages branch.
780+
681781
- name: Exit
682782
env:
683783
NEEDS_JSON: ${{ toJSON(needs) }}

‎ci/tools/allure_prepare.py‎

Lines changed: 121 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,121 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
#
3+
# SPDX-License-Identifier: Apache-2.0
4+
5+
"""Prepare the JUnit XML files from every CI configuration for Allure Report 3.
6+
7+
The test workflows upload one ``test-results-<configuration>`` artifact per
8+
job, each holding one JUnit XML file per suite. Allure reads JUnit XML
9+
directly, but it identifies a test by class name and test name only, so the
10+
same test from several configurations would collapse into "retries" of one
11+
result. This script gives every ``<testsuite>`` a ``package`` attribute that
12+
names its configuration, which Allure turns into a ``package`` label, and
13+
writes an ``allurerc.mjs`` whose ``environments`` map that label back to one
14+
environment per configuration. The report then keeps the configurations
15+
apart and shows a test's result in every configuration side by side.
16+
17+
Usage::
18+
19+
python ci/tools/allure_prepare.py --results test-results \\
20+
--output allure-results --config allurerc.mjs
21+
"""
22+
23+
from __future__ import annotations
24+
25+
import argparse
26+
import json
27+
import re
28+
import sys
29+
import xml.etree.ElementTree as ET
30+
from pathlib import Path
31+
32+
ARTIFACT_PREFIX = "test-results-"
33+
MODE_PREFIX = "standard-"
34+
35+
36+
def configuration_name(artifact_dir: Path) -> str:
37+
"""Return the configuration name encoded in an artifact directory name."""
38+
name = artifact_dir.name
39+
if name.startswith(ARTIFACT_PREFIX):
40+
name = name[len(ARTIFACT_PREFIX) :]
41+
if name.startswith(MODE_PREFIX):
42+
name = name[len(MODE_PREFIX) :]
43+
return name
44+
45+
46+
def environment_id(name: str, taken: set[str]) -> str:
47+
"""Return a unique Allure environment id: ``[A-Za-z0-9_-]``, at most 64 characters."""
48+
base = re.sub(r"[^A-Za-z0-9_-]", "-", name)[:60] or "configuration"
49+
candidate = base
50+
counter = 2
51+
while candidate in taken:
52+
candidate = f"{base}-{counter}"
53+
counter += 1
54+
taken.add(candidate)
55+
return candidate
56+
57+
58+
def prepare(results: Path, output: Path) -> dict[str, int]:
59+
"""Copy every JUnit file into ``output`` with its configuration as the suite package.
60+
61+
Returns the number of files copied per configuration.
62+
"""
63+
output.mkdir(parents=True, exist_ok=True)
64+
counts: dict[str, int] = {}
65+
for xml_path in sorted(results.rglob("*.xml")):
66+
configuration = configuration_name(xml_path.parent)
67+
# The files come from our own test jobs, not from an untrusted source.
68+
tree = ET.parse(xml_path) # noqa: S314
69+
for suite in tree.iter("testsuite"):
70+
suite.set("package", configuration)
71+
tree.write(output / f"{configuration}-{xml_path.name}", encoding="utf-8", xml_declaration=True)
72+
counts[configuration] = counts.get(configuration, 0) + 1
73+
return counts
74+
75+
76+
def render_config(report_name: str, configurations: list[str]) -> str:
77+
"""Return the allurerc.mjs text with one environment per configuration."""
78+
taken: set[str] = set()
79+
entries = []
80+
for configuration in configurations:
81+
env_id = json.dumps(environment_id(configuration, taken))
82+
value = json.dumps(configuration)
83+
entries.append(
84+
f" {env_id}: {{\n"
85+
f" name: {value},\n"
86+
f' matcher: ({{ labels }}) => labels.some(({{ name, value }}) => name === "package" && value === {value}),\n'
87+
f" }},"
88+
)
89+
return (
90+
"// Generated by ci/tools/allure_prepare.py: one environment per CI configuration.\n"
91+
"export default {\n"
92+
f" name: {json.dumps(report_name)},\n"
93+
" environments: {\n" + "\n".join(entries) + "\n },\n};\n"
94+
)
95+
96+
97+
def main(argv: list[str] | None = None) -> int:
98+
parser = argparse.ArgumentParser(description="Prepare CI JUnit XML files for Allure Report 3.")
99+
parser.add_argument(
100+
"--results", type=Path, required=True, help="directory with the downloaded test-results-* artifacts"
101+
)
102+
parser.add_argument(
103+
"--output", type=Path, required=True, help="flat directory of JUnit files for 'allure generate'"
104+
)
105+
parser.add_argument("--config", type=Path, required=True, help="path of the allurerc.mjs to write")
106+
parser.add_argument("--report-name", default="cuda-python test results", help="title of the report")
107+
args = parser.parse_args(argv)
108+
109+
counts = prepare(args.results, args.output)
110+
if not counts:
111+
print(f"error: no JUnit XML files under {args.results}", file=sys.stderr)
112+
return 1
113+
args.config.write_text(render_config(args.report_name, sorted(counts)), encoding="utf-8")
114+
for configuration, count in sorted(counts.items()):
115+
print(f"{configuration}: {count} files")
116+
print(f"{len(counts)} configurations -> {args.output}; config written to {args.config}")
117+
return 0
118+
119+
120+
if __name__ == "__main__":
121+
sys.exit(main())

‎ci/tools/tests/test_allure_prepare.py‎

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
#
3+
# SPDX-License-Identifier: Apache-2.0
4+
5+
from __future__ import annotations
6+
7+
import os
8+
import sys
9+
import xml.etree.ElementTree as ET
10+
11+
import pytest
12+
13+
sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
14+
from allure_prepare import configuration_name, environment_id, main, prepare, render_config
15+
16+
JUNIT = """<?xml version="1.0" encoding="utf-8"?>
17+
<testsuites><testsuite name="pytest" errors="0" failures="1" skipped="0" tests="2" time="0.1">
18+
<testcase classname="tests.test_a" name="test_ok" time="0.01" />
19+
<testcase classname="tests.test_a" name="test_bad" time="0.02"><failure message="boom">boom</failure></testcase>
20+
</testsuite></testsuites>
21+
"""
22+
23+
24+
def write_artifacts(root):
25+
for artifact in ("test-results-standard-linux-64-py3.12", "test-results-nightly-cuda-core-win-64"):
26+
directory = root / artifact
27+
directory.mkdir(parents=True)
28+
(directory / "junit-core.xml").write_text(JUNIT, encoding="utf-8")
29+
(directory / "junit-bindings.xml").write_text(JUNIT, encoding="utf-8")
30+
31+
32+
@pytest.mark.agent_authored(model="claude-fable-5-1")
33+
def test_configuration_name_strips_the_artifact_and_standard_prefixes(tmp_path):
34+
assert configuration_name(tmp_path / "test-results-standard-linux-64-py3.12") == "linux-64-py3.12"
35+
assert configuration_name(tmp_path / "test-results-nightly-cuda-core-win-64") == "nightly-cuda-core-win-64"
36+
37+
38+
@pytest.mark.agent_authored(model="claude-fable-5-1")
39+
def test_prepare_flattens_the_files_and_tags_every_suite_with_its_configuration(tmp_path):
40+
write_artifacts(tmp_path / "in")
41+
counts = prepare(tmp_path / "in", tmp_path / "out")
42+
assert counts == {"linux-64-py3.12": 2, "nightly-cuda-core-win-64": 2}
43+
assert sorted(p.name for p in (tmp_path / "out").iterdir()) == [
44+
"linux-64-py3.12-junit-bindings.xml",
45+
"linux-64-py3.12-junit-core.xml",
46+
"nightly-cuda-core-win-64-junit-bindings.xml",
47+
"nightly-cuda-core-win-64-junit-core.xml",
48+
]
49+
tree = ET.parse(tmp_path / "out" / "linux-64-py3.12-junit-core.xml") # noqa: S314
50+
assert [suite.get("package") for suite in tree.iter("testsuite")] == ["linux-64-py3.12"]
51+
assert len(list(tree.iter("testcase"))) == 2
52+
53+
54+
@pytest.mark.agent_authored(model="claude-fable-5-1")
55+
def test_render_config_has_one_environment_per_configuration():
56+
text = render_config("demo", ["linux-64-py3.12", "win-64 (TCC)"])
57+
assert 'name: "demo"' in text
58+
assert '"linux-64-py3-12": {' in text
59+
assert '"win-64--TCC-": {' in text
60+
assert 'value === "win-64 (TCC)"' in text
61+
assert text.count("matcher:") == 2
62+
63+
64+
@pytest.mark.agent_authored(model="claude-fable-5-1")
65+
def test_environment_id_is_sanitized_and_unique():
66+
taken: set[str] = set()
67+
assert environment_id("a.b", taken) == "a-b"
68+
assert environment_id("a_b", taken) == "a_b"
69+
assert environment_id("a-b", taken) == "a-b-2"
70+
71+
72+
@pytest.mark.agent_authored(model="claude-fable-5-1")
73+
def test_main_fails_when_there_are_no_results(tmp_path, capsys):
74+
(tmp_path / "empty").mkdir()
75+
rc = main(
76+
["--results", str(tmp_path / "empty"), "--output", str(tmp_path / "out"), "--config", str(tmp_path / "rc.mjs")]
77+
)
78+
assert rc == 1
79+
assert "no JUnit XML" in capsys.readouterr().err

0 commit comments

Comments
 (0)