What happened?
The TestUsageStandalone e2e test has been flaky recently, as shown in BuildPulse at: https://app.buildpulse.io/@crossplane/crossplane?q=TestUsageStandalone
I've seen two types of failures after looking through a bunch of e2e test results and logs.
Usage is not deleted
The test deletes the Usage object and waits 30 seconds for it to be fully deleted and cleaned up:
|
funcs.ResourcesDeletedWithin(30*time.Second, manifests, "with-by/usage.yaml"), |
Example failure: https://github.com/crossplane/crossplane/actions/runs/12938291341/job/36088096821
Log snippet:
+e2e *failed* | usage_test.go:71: Waiting 30s for Usage.apiextensions.crossplane.io using-uses-used to be deleted...
...
+e2e *failed* | usage_test.go:71: resources not deleted: client rate limiter Wait returned an error: context deadline exceeded:
I'm not sure what is causing this, and it's odd because the Usage object has replayDeletion: true, so we'd expect the deletion to be retried and completed within the 30 second waiting period we give it.
Fail to reach provider-nop webhook
The test installs provider-nop as a pre-req and waits for the provider to become healthy and active:
|
funcs.ResourcesCreatedWithin(30*time.Second, manifests, "setup/*.yaml"), |
|
funcs.ResourcesHaveConditionWithin(2*time.Minute, manifests, "setup/provider.yaml", pkgv1.Healthy(), pkgv1.Active()), |
I believe there's a race here between that check succeeding, then the test creating a NopResource before provider-nop's webhook is actually reachable.
Example failure: https://github.com/crossplane/crossplane/actions/runs/12837888189/job/35802430800
Logs snippet:
+e2e *failed* | usage_test.go:71: failed to decode file "with-by/used.yaml": Internal error occurred:
failed calling webhook "nopresources.nop.crossplane.io": failed to call webhook:
Post "https://provider-nop.crossplane-system.svc:9443/validate-nop-crossplane-io-v1alpha1-nopresource?timeout=10s":
dial tcp 10.96.200.225:9443: connect: connection refused
This may be fixable by adding a more thorough check to ensure the provider's webhook is reachable before creating a resource, or simply adding a retry for creating the resource.
What happened?
The
TestUsageStandalonee2e test has been flaky recently, as shown in BuildPulse at: https://app.buildpulse.io/@crossplane/crossplane?q=TestUsageStandaloneI've seen two types of failures after looking through a bunch of e2e test results and logs.
Usageis not deletedThe test deletes the
Usageobject and waits 30 seconds for it to be fully deleted and cleaned up:crossplane/test/e2e/usage_test.go
Line 43 in 9c7626f
Example failure: https://github.com/crossplane/crossplane/actions/runs/12938291341/job/36088096821
Log snippet:
I'm not sure what is causing this, and it's odd because the
Usageobject hasreplayDeletion: true, so we'd expect the deletion to be retried and completed within the 30 second waiting period we give it.Fail to reach
provider-nopwebhookThe test installs
provider-nopas a pre-req and waits for the provider to become healthy and active:crossplane/test/e2e/usage_test.go
Lines 79 to 80 in 9c7626f
I believe there's a race here between that check succeeding, then the test creating a
NopResourcebeforeprovider-nop's webhook is actually reachable.Example failure: https://github.com/crossplane/crossplane/actions/runs/12837888189/job/35802430800
Logs snippet:
This may be fixable by adding a more thorough check to ensure the provider's webhook is reachable before creating a resource, or simply adding a retry for creating the resource.