feat(test-optimization): create final_status tag on test event for jest#7356
Conversation
BenchmarksBenchmark execution time: 2026-02-02 12:16:01 Comparing candidate commit 980e0d5 in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 229 metrics, 31 unstable metrics. |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #7356 +/- ##
==========================================
- Coverage 80.56% 80.56% -0.01%
==========================================
Files 731 731
Lines 31236 31240 +4
==========================================
+ Hits 25166 25167 +1
- Misses 6070 6073 +3 Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Overall package sizeSelf size: 4.49 MB Dependency sizes| name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 2.0.3 | 76.87 kB | 808.03 kB | | dc-polyfill | 0.1.10 | 26.73 kB | 26.73 kB |🤖 This report was automatically generated by heaviest-objects-in-the-universe |
This comment has been minimized.
This comment has been minimized.
calvinbayer
force-pushed
the
calvin.bayer/feat/SDTEST-2980/create-final-status-tag-for-jest
branch
3 times, most recently
from
603eda3 to
284540b
Compare
juan-fernandez
left a comment
juan-fernandez
left a comment
There was a problem hiding this comment.
👏 Nice! LGTM other than some minor comments
| } else if (shouldFailSometimes) { | ||
| assert.ok(!('TEST_HAS_FAILED_ALL_RETRIES' in test.meta)) | ||
| assert.strictEqual(test.meta[TEST_MANAGEMENT_ATTEMPT_TO_FIX_PASSED], 'false') | ||
| assert.strictEqual(test.meta[TEST_FINAL_STATUS], isQuarantined ? 'skip' : 'fail') |
There was a problem hiding this comment.
❓ Does the spec say that if the status is to be ignored because the test is quarantined, its final_status should be skip?
There was a problem hiding this comment.
ok I see that this is explicitly tested later on, so nvm 😄
There was a problem hiding this comment.
Yes, updated the test name to be more explicit about the expected state
integration-tests/jest/jest.spec.js
Outdated
| assert.strictEqual(exitCode, 1) | ||
| }) | ||
|
|
||
| it('sets final_status for quarantined tests', async () => { |
There was a problem hiding this comment.
| it('sets final_status for quarantined tests', async () => { | |
| it('sets final_status to skip for quarantined tests', async () => { |
| } | ||
|
|
||
| // For ATR: The last execution's status is what the framework reports | ||
| return status |
There was a problem hiding this comment.
this function is such a beauty 😄
| numberOfExecutedRetries >= (testManagementAttemptToFixRetries + 1) | ||
|
|
||
| // When no retry features are active, every test execution is final | ||
| const noRetryFeaturesActive = !isEfdEnabled && !isAttemptToFixEnabled |
There was a problem hiding this comment.
could we return earlier in this case? (just thinking of ways to simplify this)
|
|
||
| // For ATR: The last execution's status is what the framework reports | ||
| return status | ||
| } |
There was a problem hiding this comment.
I'm thinking maybe we split the function by feature? So we can understand what each feature does more easily
calvinbayer
force-pushed
the
calvin.bayer/feat/SDTEST-2980/create-final-status-tag-for-jest
branch
from
284540b to
27e9b2a
Compare
calvinbayer
deleted the
calvin.bayer/feat/SDTEST-2980/create-final-status-tag-for-jest
branch
…st (#7356) * feat(test-optimization): create final_status tag on test event for jest * fix: errors after resolving rebase conflicts * test: add assertion to test name * refactor: extract functions to get final state for better readability * refactor: merge finalStatus function into restuls function for better readability * fix: efd enabled but not active caused final_status tag to be set on every retry * style: fix lint violations
…st (#7356) * feat(test-optimization): create final_status tag on test event for jest * fix: errors after resolving rebase conflicts * test: add assertion to test name * refactor: extract functions to get final state for better readability * refactor: merge finalStatus function into restuls function for better readability * fix: efd enabled but not active caused final_status tag to be set on every retry * style: fix lint violations
…st (#7356) * feat(test-optimization): create final_status tag on test event for jest * fix: errors after resolving rebase conflicts * test: add assertion to test name * refactor: extract functions to get final state for better readability * refactor: merge finalStatus function into restuls function for better readability * fix: efd enabled but not active caused final_status tag to be set on every retry * style: fix lint violations
…st (#7356) * feat(test-optimization): create final_status tag on test event for jest * fix: errors after resolving rebase conflicts * test: add assertion to test name * refactor: extract functions to get final state for better readability * refactor: merge finalStatus function into restuls function for better readability * fix: efd enabled but not active caused final_status tag to be set on every retry * style: fix lint violations
2 participants
What does this PR do?
Create a
@test.final_statustag that is added to every test event that represents the final execution of that test (there can be multiple executions of a single test due to retry mechanisms being active)Motivation
Tracers send each test execution as a span to the backend. When retry mechanisms are enabled a single test can run several times with different outcomes. These mechanisms differ in behavior. Some stop once the test reaches the desired state such as ATR ending when a test passes. Others always run a fixed number of attempts such as EFD or Attempt to fix.
To ensure the correct final result for the customer some intermediate outcomes are suppressed and the testing framework receives an adjusted final status. For example if an early EFD attempt passes but later attempts fail the test is still reported as passed and the failing attempts are hidden to avoid failing CI pipelines.
Both external and internal users have requested a way to query tests in Datadog by this final status. Their main goal is to build monitors and alerts for hard failures on default branches.
To support this a new tag will be added to the last retry of a test named:
@test.final_status:pass|fail|skip