What problem are you facing?

The e2e tests for Crossplane have not been as reliable as we'd like lately. For example, bumping the crossplane-runtime dependency in preparation for the v1.16 release was delayed due to e2e test failures that had been present in master for awhile (#5652 (comment)).

Ideally, we would be getting consistent green builds where all tests pass, but recently we've been getting more failures that are more likely to be test issues as opposed to genuine issues in the code we ship.

The less reliable our tests are, the less confidence we have in what we're shipping.

How could Crossplane help solve your problem?

This is a high level epic for improving our confidence and reliability in the e2e tests.

Framework improvements

• Set up buildpulse to start tracking test results (Fix unit test coverage reports, add E2E flake reports #5668)
• e2e tests should fail fast #5672
• e2e: wait for deployment to rollout #6090
• Re-run flaky E2E tests up to 3 times #6405
• test/e2e/funcs: add helpful error context to ApplyResources and others, vs. features.go:94 #5722
• Push all E2E packages to xpkg.crossplane.io #4257
• e2e: cover downgrades in the lifecycle test case #6160
• Bump sigs.k8s.io/e2e-framework dependency to v0.5.0 (test/e2e/funcs: add helpful error context to ApplyResources and others, vs. features.go:94 #5722 (comment))

Flaky tests

• TestDowngrade E2E frequently failing #6413
• TestCrossplaneLifecycle is broken #6451
• more flaky tests from https://app.buildpulse.io/@crossplane/crossplane
• TestUsageStandalone e2e test is flaky #6245
• Consistent e2e failures on old branches #6085
• E2E improvements for claim server side apply #6182
• Realtime compositions E2E tests flaky #5228
• TestCompositionInvalidComposed Increase e2e test reliability #5671 (comment)
• TestConfigurationWithDependency Bump crossplane-runtime to v1.17.0-rc.0 #5652 (comment)