Skip to content

Summarize test failures in the CI job summary and annotations - #2791

Merged
Benjamin Elder (BenTheElder) merged 6 commits into
agent-substrate:mainfrom
BenTheElder:ci-test-failure-summary
Oct 17, 2026
Merged

Benjamin Elder (BenTheElder) merged 6 commits into
agent-substrate:mainfrom
BenTheElder:ci-test-failure-summary

Conversation

@BenTheElder

@BenTheElder Benjamin Elder (BenTheElder) commented Oct 17, 2026 •

Copy link
Copy Markdown
Collaborator

Failures in the CI step logs have gotten hard to find, the unit test step alone is ~35k lines for a single failure. junittool report lists each failed test with just its t.Error / t.Fatal output in the job summary and as annotations.

  • go test only marks t.Error output in -json without -v, so the gotestsum runs drop -v
  • the unit test logs print one line per package plus failed test output, the full output is still in the uploaded json
  • the unit test JUnit files are registered now, so verify actually checks them

NOTE: There are some off the shelf tools, but we already had one, so tests aside this isn't much code and avoids taking another third party action dependency, while keeping our existing junit collection etc. It also folds a parent test into its failed subtests, names the tests that never finished on a timeout, and points panic annotations at the right line.

Tested locally with sample failures, against the artifacts from a failed main run, and on this PR with a since-dropped commit of failing unit tests.

AI-assisted.

  • Tests pass
  • Appropriate changes to documentation are included in the PR

drop -v from the gotestsum runs, with -json it replaces -test.v=test2json and go test stops marking t.Error output
otherwise verify only checks root.xml, and fails on a missing manifest when the unit tests fail
@BenTheElder Benjamin Elder (BenTheElder) added area/tests Enhancing / fixing test coverage. area/dev-infra labels Oct 17, 2026
@BenTheElder

Copy link
Copy Markdown
Collaborator Author

Example, with temporary intentionally failing test commit:
https://github.com/agent-substrate/substrate/actions/runs/38992445744/job/117277569036?pr=2791

Which also annotate and report back to the sources:
image

@BenTheElder
Benjamin Elder (BenTheElder) marked this pull request as ready for review October 17, 2026 02:14
Comment thread tools/junittool/report.go Resolved
Comment thread tools/junittool/report.go Resolved
Comment thread tools/junittool/report.go Resolved
Comment thread tools/junittool/report.go Resolved
Comment thread .github/workflows/pr-workflow.yaml Resolved
@BenTheElder
Benjamin Elder (BenTheElder) added this pull request to the merge queue Oct 17, 2026
Merged via the queue into agent-substrate:main with commit 2f547d9 Oct 17, 2026
15 checks passed
@BenTheElder
Benjamin Elder (BenTheElder) deleted the ci-test-failure-summary branch October 17, 2026 06:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/dev-infra area/tests Enhancing / fixing test coverage.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants