feat(test-optimization): add EFD retry count override#9344
Conversation
🎉 All green!🧪 All tests passed 🎯 Code Coverage (details) 🔗 Commit SHA: f071805 | Docs | Datadog PR Page | Give us feedback! |
|
Bits Code status: ✅ Done Comment @DataDog to request changes |
Overall package sizeSelf size: 6.77 MB Dependency sizes| name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.3.1 | 122.62 kB | 438.86 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |🤖 This report was automatically generated by heaviest-objects-in-the-universe |
BenchmarksBenchmark execution time: 2026-07-15 10:22:32 Comparing candidate commit f071805 in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 2308 metrics, 50 unstable metrics.
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #9344 +/- ##
==========================================
+ Coverage 96.61% 96.64% +0.03%
==========================================
Files 920 923 +3
Lines 122266 122855 +589
Branches 21008 21343 +335
==========================================
+ Hits 118128 118738 +610
+ Misses 4138 4117 -21
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b2c14f5c42
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
|
Codex Review: Didn't find any major issues. Keep it up! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
There was a problem hiding this comment.
More details
PR adds DD_TEST_EARLY_FLAKE_DETECTION_RETRY_COUNT to override EFD retry counts across all duration buckets. Implementation correctly filters the backend's duration-based policy, applies the override uniformly, and preserves existing behavior when unset. All 8 new tests pass (unit, config, and integration), edge cases are handled properly, and the feature integrates cleanly with existing EFD and flaky retry systems.
📊 Validated against 30 scenarios · Open Bits AI session
🤖 Datadog Autotest · Commit f071805 · What is Autotest? · Any feedback? Reach out in #autotest
What does this PR do?
Adds
DD_TEST_EARLY_FLAKE_DETECTION_RETRY_COUNT, an optional non-negative integer that overrides the number of Early Flake Detection retries in each supported duration bucket.When set, the
5s,10s,30s, and5mbuckets all use this value. Tests lasting at least five minutes retain the existing no-retry behavior. When the variable is unset, the backend-provided duration policy remains unchanged. Setting it to0disables EFD retries.Motivation
DD_CIVISIBILITY_FLAKY_RETRY_COUNTcontrols Automatic Test Retries, but it does not limit Early Flake Detection. This makes it possible for EFD's duration buckets to run a new test more times than the configured ATR retry count, increasing CI duration unexpectedly.The new setting provides a separate, explicit EFD retry limit while preserving the existing backend policy by default.
Additional Notes
The override is applied at the shared library-configuration boundary, so it covers Playwright, Jest, Mocha, Cucumber, Vitest, and Cypress.
The generated retry policy uses only the standard duration-bucket keys; it does not introduce an additional sentinel key.
Validation performed:
1 !== 2) and passes afterward.git diff --check.