Re-run flaky E2E tests up to 3 times#6405
Conversation
Also, report slowest tests and use standard verbose logging for maximum insight into progress. This feels a little like giving up, but realistically right now ~80% of our E2E test runs fail. When they do, we mostly just re-run the entire failed suites. Each suite run takes ~20-25 mins to complete. This argument will only re-run the specific test that failed, automatically. This should result in less time spent running and waiting for E2Es overall. Signed-off-by: Nic Cope <[email protected]>
|
Well this change worked as expected, except the I think we should make the change though - if we'd manually retried this test four times it would've run the whole suite four times and thus taken 80-100 minutes rather than 40. |
For three total runs - not four. Signed-off-by: Nic Cope <[email protected]>
I find it a lot easier to spot failed tests with this format, and I also appreciate that it gives a summary of how long each test took to run. Default to standard-verbose for when folks run locally. Signed-off-by: Nic Cope <[email protected]>
jbw976
left a comment
jbw976
left a comment
There was a problem hiding this comment.
This seems like a reasonable trade-off for more reliability now and we can keep making progress on #5671 as we find the time.
From your experience with this retry behavior, do we get any indication if we succeeded but had to retry a test to get there? or is that not very obvious unless you go looking into the logs?
| ARG GOARCH=${TARGETARCH} | ||
| ARG GOOS=${TARGETOS} | ||
| ARG FLAGS="-test-suite=base" | ||
| ARG GOTESTSUM_FORMAT="standard-verbose" |
There was a problem hiding this comment.
nice, i also like that by default when running e2e tests locally we'll get the more realtime streaming logging output 👍
It's just what you see in the runs for this PR. So no, it's not very easy to spot. I'm hoping BuildPulse will still highlight tests that're failing a lot though. |
2 participants
Description of your changes
This feels a little like giving up, but realistically right now ~80% of our E2E test runs fail. When they do, we mostly just re-run the entire failed suites. Each suite run takes ~20-25 mins to complete.
This argument will only re-run the specific test that failed, automatically. This should result in less time spent running and waiting for E2Es overall.
I have:
earthly +reviewableto ensure this PR is ready for review.Added or updated unit tests.Added or updated e2e tests.Linked a PR or a docs tracking issue to document this change.Addedbackport release-x.ylabels to auto-backport this PR.Followed the API promotion workflow if this PR introduces, removes, or promotes an API.Need help with this checklist? See the cheat sheet.