What problem are you facing?
In our e2e test framework, is that it looks like a failure won't stop the rest of the checks from being performed, so the test continues after what should be a fatal failure.
For example, In Assess("foo", funcs.AllOf(.... It seems reasonable to fail quickly if we've already detected a failed condition, instead of wasting more time on further checks that could take minute(s) each.
In the test output copied for convenience below, we fail an assessment (context deadline exceeded) and then proceed to perform more checks like Waiting 5m0s for... and Ensuring foo does not change within 1m0s.....
feature.go:741: XNopResource.nop.example.org propagate-fields-removals-l2dkr doesn't yet have desired value at field path spec.parameters.tags[newtag]: any(
- s"NotFound",
+ string("v2"),
)
env.go:428: XR did not have desired value "NotFound" at field path spec.parameters.tags[newtag]: context deadline exceeded:
<ommitted>
env.go:428: Ensuring claim NopResource.nop.example.org default/propagate-fields-removals does not change within 1m0s
env.go:428: Claim NopResource.nop.example.org default/propagate-fields-removals did not change within 1m0s
env.go:428: Waiting 5m0s for NopResource.nop.example.org default/propagate-fields-removals to have value "I'm cool!" at field path status.coolerField...
env.go:428: 1 resources have desired value "I'm cool!" at field path status.coolerField after 0.506s
env.go:428: Ensuring composite resource XNopResource.nop.example.org propagate-fields-removals-l2dkr does not change within 1m0s
env.go:428: Composite resource XNopResource.nop.example.org propagate-fields-removals-l2dkr did not change within 1m0s
How could Crossplane help solve your problem?
If an assessment fails in an e2e test then we should fail the entire test quickly. Performing further checks after a failure isn't ideal because:
- it extends the running time of the test when we know the outcome isn't going to be successful
- it adds additional output to the test artifacts that makes it harder to find the original/actual failure
What problem are you facing?
In our e2e test framework, is that it looks like a failure won't stop the rest of the checks from being performed, so the test continues after what should be a fatal failure.
For example, In
Assess("foo", funcs.AllOf(....It seems reasonable to fail quickly if we've already detected a failed condition, instead of wasting more time on further checks that could take minute(s) each.In the test output copied for convenience below, we fail an assessment (
context deadline exceeded) and then proceed to perform more checks likeWaiting 5m0s for...andEnsuring foo does not change within 1m0s.....How could Crossplane help solve your problem?
If an assessment fails in an e2e test then we should fail the entire test quickly. Performing further checks after a failure isn't ideal because: