[test optimization] Fix attempt to fix in vitest #7785
Merged
juan-fernandez merged 2 commits intomasterfrom Mar 16, 2026
Merged
[test optimization] Fix attempt to fix in vitest #7785juan-fernandez merged 2 commits intomasterfrom
juan-fernandez merged 2 commits intomasterfrom
Conversation
The shouldFlipStatus flag was resetting task.result.state to 'pass' for ALL attempt-to-fix tests during retries. Per spec, only quarantined or disabled test results should be ignored. Split into two flags: - shouldTrackStatuses: tracks retry outcomes for all ATF/EFD tests (needed to compute attemptToFixPassed/Failed) - shouldFlipStatus: only resets vitest's internal state for EFD and quarantined/disabled ATF tests, so plain ATF failures still propagate to the exit code Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Contributor
Overall package sizeSelf size: 4.98 MB Dependency sizes| name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.0.0 | 81.15 kB | 815.98 kB | | dc-polyfill | 0.1.10 | 26.73 kB | 26.73 kB |🤖 This report was automatically generated by heaviest-objects-in-the-universe |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #7785 +/- ##
==========================================
- Coverage 80.36% 80.34% -0.02%
==========================================
Files 743 743
Lines 32296 32296
==========================================
- Hits 25955 25949 -6
- Misses 6341 6347 +6 Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
This comment has been minimized.
This comment has been minimized.
BenchmarksBenchmark execution time: 2026-03-16 11:21:53 Comparing candidate commit 326f14c in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 234 metrics, 26 unstable metrics. |
Adds a regression test that fails the first attempt but passes all retries. Before the fix, shouldFlipStatus would reset intermediate states to 'pass' for all ATF tests, causing vitest to exit with code 0 even though the test is not quarantined/disabled. The new test asserts exit code 1 for this case. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
gnufede
approved these changes
Mar 16, 2026
Merged
juan-fernandez
added a commit
that referenced
this pull request
Mar 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fixes the vitest instrumentation to only suppress attempt-to-fix (ATF) test failures when the test is also quarantined or disabled, aligning with the test management spec.
Motivation
The spec states: "If test is tagged with
test.test_management.is_test_disabledortest.test_management.is_quarantined, then the test results are ignored."The
shouldFlipStatusflag was resettingtask.result.stateto'pass'for ALL ATF tests during retries, causing vitest to exit with code 0 even for plain ATF tests that fail. Only quarantined/disabled ATF tests should have their failures suppressed.