Skip to content

Report tests that pass on retry as flaky - #2801

Merged
natebosch merged 5 commits into
masterfrom
test-reporting-retried
Oct 22, 2026
Merged

Report tests that pass on retry as flaky#2801
natebosch merged 5 commits into
masterfrom
test-reporting-retried

Conversation

@natebosch

@natebosch natebosch commented Oct 21, 2026

Copy link
Copy Markdown
Member

A test that failed and then passed on retry was reported inconsistently.
A setUpAll or tearDownAll stayed in Engine.failed because the code
that removes a test from failed when a later attempt passes was skipped
for callbacks that are modeled as tests in the runner, but aren't what the
user considers a test case. These can cause failures when there is an
error, but don't count as success when they run without exception.

The summaries then reported a failure even though the overall run
succeeded, for example ::error::2 tests passed, 1 failed. from the
GitHub reporter or +2 -1: All tests passed! from the compact and
expanded reporters. Regular tests that passed on retry were counted as
passed with no indication that they had failed.

  • Always remove a test from failed when it completes successfully, and
    track tests that passed after a failed attempt in new
    LiveSuite.flaky and Engine.flaky sets. A test that fails and is
    then skipped on retry moves from failed to skipped instead of
    being counted in both.
  • Add an AttemptCounter utility for reporters to track attempts
    without changing LiveTest or the protocol between the runner and
    test isolates.
  • compact, expanded, failures-only: show [E] (attempt N of M) for
    tests with retry, except for errors after the final attempt, and add
    (N flaky) to the final line. The expanded reporter also lists flaky
    tests.
  • github: title flaky test groups ⚠️ <name> (passed on attempt N of M)
    and tests that failed every attempt (failed after N attempts). Print
    output and errors in the order they were emitted for all tests, add a
    flaky count to the summary, and use a ::warning:: annotation when
    the only problems are flaky tests.
  • json: add testDone.retried and bump the protocol to 0.1.2. A
    synthetic test that passes on retry is now hidden, as the protocol
    documents.

A retried tearDownAll doesn't re-run its callbacks, so it always
passes on retry; that is tracked separately in
#2800.

@natebosch
Report tests that pass on retry as flaky
d4bd7d2 
A test that failed and then passed on retry was reported inconsistently.
A `setUpAll` or `tearDownAll` stayed in `Engine.failed` because the code
that removes a test from `failed` when a later attempt passes was skipped
for callbacks that are modeled as tests in the runner, but aren't what the
user considers a test case. These can cause failures when there is an
error, but don't count as success when they run without exception.

The summaries then reported a failure even though the overall run
succeeded, for example `::error::2 tests passed, 1 failed.` from the
GitHub reporter or `+2 -1: All tests passed!` from the compact and
expanded reporters. Regular tests that passed on retry were counted as
passed with no indication that they had failed.

- Always remove a test from `failed` when it completes successfully, and
  track tests that passed after a failed attempt in new
  `LiveSuite.flaky` and `Engine.flaky` sets. A test that fails and is
  then skipped on retry moves from `failed` to `skipped` instead of
  being counted in both.
- Add an `AttemptCounter` utility for reporters to track attempts
  without changing `LiveTest` or the protocol between the runner and
  test isolates.
- compact, expanded, failures-only: show `[E] (attempt N of M)` for
  tests with `retry`, except for errors after the final attempt, and add
  `(N flaky)` to the final line. The expanded reporter also lists flaky
  tests.
- github: title flaky test groups `⚠️ <name> (passed on attempt N of M)`
  and tests that failed every attempt `(failed after N attempts)`. Print
  output and errors in the order they were emitted for all tests, add a
  flaky count to the summary, and use a `::warning::` annotation when
  the only problems are flaky tests.
- json: add `testDone.retried` and bump the protocol to 0.1.2. A
  synthetic test that passes on retry is now hidden, as the protocol
  documents.

A retried `tearDownAll` doesn't re-run its callbacks, so it always
passes on retry; that is tracked separately in
#2800.
@natebosch
natebosch marked this pull request as ready for review October 21, 2026 22:51
@natebosch
natebosch requested a review from a team as a code owner October 21, 2026 22:51
@natebosch
natebosch requested a review from kevmoo October 21, 2026 22:51
@github-actions

github-actions Bot commented Oct 21, 2026 •
edited
Loading

Copy link
Copy Markdown

PR Health

Changelog Entry ✔️
Package Changed Files

Changes to files need to be accounted for in their respective changelogs.

This check can be disabled by tagging the PR with skip-changelog-check (for example, for non-user-visible changes when the package version is already at -wip).

kevmoo
kevmoo requested changes Oct 21, 2026
View reviewed changes
Comment thread pkgs/test_core/lib/src/runner/reporter/github.dart Resolved
Comment thread pkgs/test/test/runner/github_reporter_test.dart Resolved
Comment thread pkgs/test_core/lib/src/runner/reporter/attempt_counter.dart Resolved
Comment thread pkgs/test/test/runner/engine_test.dart Resolved
Comment thread pkgs/test_core/lib/src/runner/reporter/compact.dart Resolved
@natebosch
Update expected source locations in retry tests
2633dea 
Combining `attempts++` and the check into `if (++attempts == 1)` removed a
line from each test source and moved the `throw` two columns right, so the
expected line and column numbers in stack traces and JSON `testStart`
events were stale.
@natebosch
Make new class final
24e8ee7
@natebosch
Address review feedback on flaky retry reporting
3c3608d 
- github: title a test that fails and is then skipped on retry
  `⏭️ <name> (skipped on attempt N of M)` instead of
  `❌ <name> (failed after N attempts)`, matching the summary's skipped
  count.
- AttemptCounter: if `watch` attaches after the test has started, count
  the current attempt as the first one rather than missing it.
- Share the `(N flaky)` suffix between the compact, expanded, and
  failures-only reporters through a `flakySuffix` helper in
  `failure_summary.dart`.
- Narrow the changelog entries to `setUpAll`. A retried `tearDownAll`
  doesn't re-run its callbacks, tracked in
  #2800.
kevmoo
kevmoo approved these changes Oct 22, 2026
View reviewed changes

@kevmoo kevmoo left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yay!

@natebosch
natebosch merged commit e18fd11 into master Oct 22, 2026
71 checks passed
@natebosch
natebosch deleted the test-reporting-retried branch October 22, 2026 22:38
@natebosch natebosch mentioned this pull request Oct 28, 2026
Merged
natebosch added a commit that referenced this pull request Oct 28, 2026
@natebosch
Fix a race in an expanded reporter retry test (#2810)
6d74720 
Fixes a flake in `expanded_reporter_test.dart` "retry: does not report an
attempt for errors after completion", added in #2801. It failed on master
in https://github.com/dart-lang/test/actions/runs/40722683531/job/122938155175:
the `+1: second test ...` line was missing and the output went straight to
`+0 -1: second test ...`.

The first test scheduled its error with `Future(() { ... })`, so the test
isolate raised it right after the test completed. The runner starts the
next test one event-loop turn after it receives the `complete` message. If
the error message was already queued by then, the first test moved to
failed before the second test started. Both orders are correct output, so
this changes the test rather than the runner.

The second test now triggers the first test's error, using the same
`Completer` pattern as `retry_test.dart`. Only the stack trace lines in the
expected output change.

Locally, the old fixture reproduced the CI output in 1 of 40 runs. The new
fixture produced identical output in 60 of 60 runs.
copybara-service Bot pushed a commit to dart-lang/sdk that referenced this pull request Oct 28, 2026
@natebosch dart-scoped@luci-project-accounts.iam.gserviceaccount.com
[deps] rev core, dartdoc, ecosystem, i18n, native, protobuf, shelf, t…
add7e59 
…est, tools, webkit_inspection_protocol

Revisions updated by `dart tools/rev_sdk_deps.dart`.

core (https://github.com/dart-lang/core/compare/04da676..d7bd45c):
  d7bd45c1  Wed Oct 28 13:23:54 2026 +0530  Chandra ObulaReddy  [args] Wrap separators to usageLineLength (dart-lang/core#1032)
  6903c6c7  Fri Oct 23 20:10:24 2026 +0000  Jonas Finnemann Jensen  [crypto] Prepare to publish 3.0.8 (dart-lang/core#1026)
  7c1c3700  Fri Oct 23 11:14:15 2026 -0700  Kevin Moore  Tighten pull_request_label workflow permissions (dart-lang/core#1029)
  f01024fc  Thu Oct 22 16:31:50 2026 -0700  Kevin Moore  [collection] Share Iterable and Set tests across wrapper_test.dart (dart-lang/core#1028)
  c782eab4  Thu Oct 22 13:25:34 2026 +0200  Jonas Finnemann Jensen  Update convert package README and add examples (dart-lang/core#999)

dartdoc (https://github.com/dart-lang/dartdoc/compare/c0e19cf..db94437):
  db94437c  Wed Oct 28 11:37:17 2026 +0100  Sarah Zakarias   Add the doc-reference-target-differs warning (dart-lang/dartdoc#4306)
  b6e3980c  Tue Oct 27 14:59:21 2026 +0100  Sarah Zakarias  Only treat a doc comment as library documentation when it precedes a library directive (dart-lang/dartdoc#4303)
  84a0141f  Tue Oct 27 11:22:04 2026 +0100  Sarah Zakarias  Add the unscoped-doc-reference warning (dart-lang/dartdoc#4305)
  b25bcd99  Thu Oct 22 15:56:17 2026 +0300  Sergey G. Grekhov  Update/replace TODO in co19 tests (dart-lang/dartdoc#4299)
  1e9e771e  Thu Oct 22 12:18:48 2026 +0300  Sergey G. Grekhov  `#4134`. Add more placement tests for `library` directives (dart-lang/dartdoc#4297)
  17629ced  Wed Oct 21 10:29:05 2026 -0700  Bob Nystrom  Use newer function type syntax for parameters. (dart-lang/dartdoc#4290)
  f5928b1d  Wed Oct 21 15:44:39 2026 +0300  Sergey G. Grekhov  `#4134`. Add placement tests for enum values (dart-lang/dartdoc#4223)
  24ae37f3  Wed Oct 21 11:18:07 2026 +0200  dependabot[bot]  Bump test_reflective_loader from 0.4.0 to 0.6.0 (dart-lang/dartdoc#4296)

ecosystem (https://github.com/dart-lang/ecosystem/compare/0ab0cfd..24b9ba8):
  24b9ba8  Mon Oct 26 18:40:19 2026 -0700  Kevin Moore  fix(repo_manage): increment patch version when transitioning to -wip (dart-lang/ecosystem#474)
  e1d4a04  Fri Oct 23 20:18:20 2026 +0000  dependabot[bot]  Bump the github-actions group across 1 directory with 8 updates (dart-lang/ecosystem#473)
  4b44357  Fri Oct 23 11:13:26 2026 -0700  Kevin Moore  Tighten pull_request_label workflow permissions (dart-lang/ecosystem#472)

i18n (https://github.com/dart-lang/i18n/compare/1f5ea2f..4c7b4ad):
  4c7b4ad1  Fri Oct 23 11:14:00 2026 -0700  Kevin Moore  Tighten pull_request_label workflow permissions (dart-lang/i18n#1113)
  899251b1  Thu Oct 8 11:30:18 2026 +0200  Moritz  [intl4x] Fix doc comment examples and verify them via {@example} in tests (dart-lang/i18n#1107)
  5c8eb9d5  Thu Oct 8 10:27:58 2026 +0200  Moritz  [intl4x] Fix YearStyle.withEra on ECMA when length is short or omitted (dart-lang/i18n#1105)
  b80bb04a  Wed Sep 23 14:37:42 2026 +0200  Michael Thomsen  Remove experiment flag no longer needed (dart-lang/i18n#1097)
  766fd507  Wed Sep 23 13:59:56 2026 +0200  Moritz  [intl4x] Mark DisplayNames API as @experimental and trigger CI on PR base edits (dart-lang/i18n#1098)
  08de2f6f  Fri Sep 18 12:49:50 2026 +0200  Moritz  Rev package:intl4x to 1.0.0 and update status to preview (dart-lang/i18n#1088)

native (https://github.com/dart-lang/native/compare/3cbe319..e0ca7c6):
  e0ca7c60f  Wed Oct 28 12:08:25 2026 +0100  Michael Goderbauer  [infra] Pin ubuntu-24.04 in workflows installing clang-15 (dart-lang/native#3771)
  eb09430a9  Tue Oct 27 10:43:22 2026 +0100  Michael Goderbauer  [jni] Publish v1.1.1 (dart-lang/native#3765)
  94b5f4337  Tue Oct 27 10:21:16 2026 +0100  Michael Goderbauer  [jnigen] Pin ubuntu-24.04 in CI workflow (dart-lang/native#3766)
  74cc078ee  Tue Oct 27 09:29:23 2026 +0100  Michael Goderbauer  [jni] Add ProGuard consumer rules for Kotlin classes used by suspend functions (dart-lang/native#3756)
  0aa672fed  Fri Oct 23 11:13:43 2026 -0700  Kevin Moore  [infra] Tighten pull_request_label workflow permissions (dart-lang/native#3764)
  8b11b6e01  Fri Oct 23 13:38:47 2026 +0200  Michael Goderbauer  [jnigen] Bump AGP to 9.1.1 and KGP to 2.4.10 for Flutter beta (dart-lang/native#3758)

protobuf (https://github.com/dart-lang/protobuf/compare/a73edc9..1d52301):
  1d52301  Wed Oct 28 08:18:02 2026 -0700  Kevin Moore  test(protoc_plugin): replace dart:mirrors in reserved_names_test (google/protobuf.dart#1104)

shelf (https://github.com/dart-lang/shelf/compare/b66665b..73b2382):
  73b2382  Thu Oct 22 10:43:29 2026 -0700  Kevin Moore  shelf_router_generator: ignore invalid_use_of_internal_member on RouterEntry (dart-lang/shelf#548)

shelf is held back at 73b2382, before dart-lang/shelf#547, because
`pkgs/shelf/test/shelf_io_test.dart` now imports the unpublished
`package:_shelf_compliance`, which breaks the shared `shelf_tests` target in
google3.

test (https://github.com/dart-lang/test/compare/d1afeb1..5967927):
  5967927e  Mon Oct 26 17:50:52 2026 -0700  Nate Bosch  Diagnose browser connection failures and time them out with the suite load timeout (dart-lang/test#2788)
  0692e8ae  Mon Oct 26 16:29:27 2026 -0700  Kevin Moore  Remove mirror_matchers.dart from package:matcher (dart-lang/test#2802)
  da21f85a  Fri Oct 23 11:13:38 2026 -0700  Kevin Moore  Tighten pull_request_label workflow permissions (dart-lang/test#2804)
  e18fd112  Thu Oct 22 15:38:51 2026 -0700  Nate Bosch  Report tests that pass on retry as flaky (dart-lang/test#2801)
  b4ef80c0  Fri Oct 23 02:44:25 2026 +0600  Pranta Dutta  Fix compact reporter crash when the terminal is too narrow (dart-lang/test#2799)
  e3a7f2b6  Tue Oct 20 15:51:34 2026 -0700  Nate Bosch  Check that web entrypoints don't import VM-only libraries (dart-lang/test#2797)

tools (https://github.com/dart-lang/tools/compare/eee87b1..3980dca):
  3980dca39  Wed Oct 28 11:43:38 2026 +0100  Jonas Finnemann Jensen  [html] Move contributors list from LICENSE to AUTHORS (dart-lang/tools#2733)
  2866a7745  Wed Oct 28 11:13:19 2026 +0100  Jonas Finnemann Jensen  [extension_discovery] Prepare for v2.1.1 release (dart-lang/tools#2727)
  530feb74e  Wed Oct 28 13:38:07 2026 +0530  Chandra ObulaReddy  [csslib] Fix the aliceblue, antiquewhite and peru color values (dart-lang/tools#2732)
  d5c210271  Tue Oct 27 15:07:32 2026 -0700  Kevin Moore  [source_span] Fix CR/CRLF highlighting, point-span context, and subspan (dart-lang/tools#2634)
  cbea5b08e  Tue Oct 27 15:07:27 2026 -0700  Kevin Moore  [string_scanner] Validate error() span bounds and test eager lastSpan (dart-lang/tools#2715)
  22fd7a6a8  Tue Oct 27 15:07:20 2026 -0700  Kevin Moore  [csslib] Fix parser, analyzer, polyfill, and printer crashes (dart-lang/tools#2638)

webkit_inspection_protocol (https://github.com/google/webkit_inspection_protocol.dart/compare/762115a..2c255f2):
  2c255f2  Thu Oct 22 14:18:44 2026 -0700  Kevin Moore  Remove dart:mirrors and deprecate dom_model.dart and forwarder.dart (google/webkit_inspection_protocol.dart#141)

web is held back at 4e60e61, before dart-lang/web#596, because it changes
`package:web` signatures (e.g. `innerHTML=`, `parseFromString`, `Worker()`)
from `JSAny` to `String` in ways that break callers in google3.

Change-Id: Ia4253b35a91d7c05e34fe0e150ea4cb3e6628015
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/569660
Commit-Queue: Nate Bosch <[email protected]>
Reviewed-by: Brian Quinlan <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kevmoo kevmoo kevmoo approved these changes

Assignees

No one assigned

Labels

package:test

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@natebosch @kevmoo