[test optimization] Fix quarantined tests being skipped rather than ignored in cypress#7442
Conversation
Overall package sizeSelf size: 4.57 MB Dependency sizes| name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 2.0.6 | 81.92 kB | 813.08 kB | | dc-polyfill | 0.1.10 | 26.73 kB | 26.73 kB |🤖 This report was automatically generated by heaviest-objects-in-the-universe |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #7442 +/- ##
=======================================
Coverage 80.41% 80.41%
=======================================
Files 732 732
Lines 31051 31051
=======================================
Hits 24971 24971
Misses 6080 6080 Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
BenchmarksBenchmark execution time: 2026-02-05 11:35:57 Comparing candidate commit 1658d98 in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 228 metrics, 32 unstable metrics. |
This comment has been minimized.
This comment has been minimized.
…gnored in cypress (#7442)
…with code 0
When a test is marked as attempt-to-fix by test management and fails,
the spec requires exit code 0 (failures should be ignored for session
outcome). This extends the existing quarantine error suppression pattern
to also cover attempt-to-fix tests: errors are caught in Cypress.on('fail')
and not re-thrown, while the actual failure status is still reported to
Datadog via the afterEach hook.
Also removes stale TODO comments for quarantine/disabled handling that
were fixed in PR #7442.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
2 participants
What does this PR do?
Changes quarantined test handling in Cypress to run tests but ignore their failures, rather than skipping them entirely. Quarantined tests now:
Cypress.on('fail')handlerTEST_STATUS: 'fail'to Datadog withTEST_MANAGEMENT_IS_QUARANTINED: 'true'Motivation
Aligns Cypress behavior with other frameworks, which run quarantined tests but ignore their failures for CI exit codes. Previously, Cypress skipped quarantined tests because there was no known mechanism to run tests while ignoring failures.
Additional Notes
The implementation uses
Cypress.on('fail')insupport.jsto intercept test failures. For quarantined tests, we:afterEachafterSpec(prevent Cypress from overwriting it)