fix(playwright): ignore intermediate retry failures in aggregate status#9231
Conversation
Overall package sizeSelf size: 6.57 MB Dependency sizes| name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.3.0 | 117.14 kB | 432.47 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |🤖 This report was automatically generated by heaviest-objects-in-the-universe |
BenchmarksBenchmark execution time: 2026-07-07 08:42:54 Comparing candidate commit 32096e4 in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 2245 metrics, 41 unstable metrics.
|
🎉 All green!🧪 All tests passed 🎯 Code Coverage (details) 🔗 Commit SHA: 32096e4 | Docs | Datadog PR Page | Give us feedback! |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #9231 +/- ##
==========================================
- Coverage 93.55% 93.53% -0.03%
==========================================
Files 900 900
Lines 52941 52944 +3
Branches 12491 12492 +1
==========================================
- Hits 49531 49523 -8
- Misses 3410 3421 +11 Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
@codex review |
|
Codex Review: Didn't find any major issues. You're on a roll. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 32096e43fd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| } else { | ||
| testSuiteToTestStatuses.set(testSuiteAbsolutePath, [testStatus]) | ||
| } | ||
| if (!willRetry) { |
There was a problem hiding this comment.
This drops the failed attempt from suite aggregation as soon as Playwright reports that it can retry, but that retry is not guaranteed to emit a later testEnd (for example, the run can be stopped by max-failures after another parallel failure or interrupted before the retry starts). In that case the test remains in remainingTestsByFile and the end-of-run fallback reports it as skipped, so the suite can finish without the original failure/error even though a failed test event was emitted. Consider keeping the pending failure and clearing it only when a later retry actually passes.
Useful? React with 👍 / 👎.
![datadog-prod-us1-4[bot]](https://avatars.githubusercontent.com/u/365230?s=60&v=4){kind=small}
There was a problem hiding this comment.
More details
The fix correctly defers suite status and error recording until tests complete retries, ensuring that intermediate failures don't pollute aggregate events. When a Playwright test fails then passes on retry, the final session/module/suite events now correctly report all-pass status with no error message, fixing the contradictory event state that previously occurred.
📊 Validated against 3 scenarios · Open Bits AI session
🤖 Datadog Autotest · Commit 32096e4 · What is Autotest? · Any feedback? Reach out in #autotest
2 participants
What does this PR do?
Fixes Playwright Test Optimization aggregate reporting for flaky tests that pass after retry.
Failed retry attempts are still reported as failed test events, but they no longer make the final suite, module, or session events look failed when Playwright eventually passes the test.
Motivation
For example, consider a Playwright test that fails on the first attempt and passes on retry. The test-level events are expected to show both attempts:
test.status=failtest.status=pass,test.is_retry=true,test.retry_reason=auto_test_retry,test.final_status=passBefore this fix, the aggregate events could still reflect the intermediate failed attempt:
test_suite_endwas reported as failedtest_module_endandtest_session_endcould reporttest.status=passwhile also carryingerror=1andTest suites failed: 1.That made a flaky-successful Playwright run look contradictory: the test retry passed, but the suite/session payload still looked failed. The root cause was that failed intermediate retry attempts were recorded into aggregate suite status and error state before Playwright had finished retrying the test.
This change only records suite status and suite errors once Playwright is done retrying that test, and only attaches session/module aggregate errors when the final session status failed.
Additional Notes
Added integration coverage for the flaky-success case so the final suite, module, and session events are passing and error-free when the final Playwright retry passes. Existing coverage still verifies that tests which exhaust all ATR attempts continue to report failure.