fix(playwright): handle disabled tests in serial retries#9313
Conversation
Overall package sizeSelf size: 6.71 MB Dependency sizes| name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.3.1 | 122.62 kB | 438.86 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |🤖 This report was automatically generated by heaviest-objects-in-the-universe |
🎉 All green!🧪 All tests passed 🎯 Code Coverage (details) 🔗 Commit SHA: 3544f04 | Docs | Datadog PR Page | Give us feedback! |
BenchmarksBenchmark execution time: 2026-07-14 06:34:04 Comparing candidate commit 3544f04 in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 2313 metrics, 45 unstable metrics.
|
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bb725a5cca
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
juan-fernandez
changed the title
|
@codex review |
|
Codex Review: Didn't find any major issues. Already looking forward to the next diff. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
![datadog-prod-us1-6[bot]](https://avatars.githubusercontent.com/u/365230?s=60&v=4){kind=small}
There was a problem hiding this comment.
More details
PR correctly fixes a crash in serial retry scenarios where disabled test siblings reappear in the worker. The fix adds early returns for unknown test events, tracks disabled test IDs through retry payloads, and marks reintroduced disabled tests as skipped before execution. Code analysis and synthetic validation confirm no regressions in normal test paths and correct handling of the disabled sibling scenario.
📊 Validated against 6 scenarios · Open Bits AI session
🤖 Datadog Autotest · Commit fb2f80b · What is Autotest? · Any feedback? Reach out in #autotest
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fb2f80be65
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
|
Codex Review: Didn't find any major issues. Keep them coming! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
2 participants
What does this PR do?
Handles disabled tests that Playwright reintroduces while retrying a serial suite. Unknown worker events no longer crash the instrumentation, reintroduced disabled tests remain skipped in the worker, and their skipped/disabled test event is reported exactly once without duplicating disabled attempt-to-fix skips.
Motivation
Fixes #9308.
Test Management removes disabled tests from the initial dispatcher groups. When an enabled test in a serial suite fails and retries, Playwright rebuilds the retry candidates from the full serial suite and schedules the disabled sibling. Its worker then emits events for a test absent from Datadog's filtered test map. Both handlers dereferenced
test.expectedStatus, crashing the Node.js process.Ignoring the unknown events prevents the crash, but by itself allows the worker to execute the disabled test and report it as passing. This change carries disabled IDs in the retry payload so the worker preserves their skipped status, then uses the existing summary path to emit the skipped/disabled event. The fallback is limited to tests actually filtered from dispatcher groups so skipped disabled attempt-to-fix executions are not reported twice.
Additional Notes
The regression runs on Playwright >=1.39 because older versions do not yet ignore unknown worker events in Playwright's own dispatcher.