#320 contains some, not all, incidents of flaky tests in various PRs.
Our e2e tests remain brittle.
Run a ralph loop lasting 6 hours in total, by recording the "start" time and not stopping until "stop" time exceeds 6 hours of duration.
Inside this loop, you must:
- Run the e2e CI steps only (
just ci e2e -- which runs on both Linux and macOS), 4 times in the row. Gather all flaky failures. Invesgitate their root cause, and device a way such that they never occur again (so no freaking timeouts). Try not to change the server/ or client/ or common/ code here. Ideally, code changes happen in tests only. Then, commit your changes, and re-run e2e CI steps only 4 times in the row.
- Rinse and repeat, during the whole of 6 hours!
IMPORTANT: DO NOT STOP until whole six hours duration has passed.
You MUST open a PR from the get go. For each change you make, commit & push - with commit message containing the e3e run measurements (duration, successes, fails, etc.). Each improvement goes in a commit message.
#320 contains some, not all, incidents of flaky tests in various PRs.
Our e2e tests remain brittle.
Run a ralph loop lasting 6 hours in total, by recording the "start" time and not stopping until "stop" time exceeds 6 hours of duration.
Inside this loop, you must:
just ci e2e-- which runs on both Linux and macOS), 4 times in the row. Gather all flaky failures. Invesgitate their root cause, and device a way such that they never occur again (so no freaking timeouts). Try not to change the server/ or client/ or common/ code here. Ideally, code changes happen in tests only. Then, commit your changes, and re-run e2e CI steps only 4 times in the row.IMPORTANT: DO NOT STOP until whole six hours duration has passed.
You MUST open a PR from the get go. For each change you make, commit & push - with commit message containing the e3e run measurements (duration, successes, fails, etc.). Each improvement goes in a commit message.