Repository navigation
Fix flaky tests: e2e demo suite timeouts and syncer poll timeout #799
Description
Summary
CI failure analysis (hack/metrics/ci-failure-analysis.py on branch docs/velocity-metrics of the aditya-shantanu fork) identified the following tests failing repeatedly across 40 sampled failed CI runs:
| Test | Failures | Job | Root cause |
|---|---|---|---|
TestDurableDirLifecycle |
6× | e2e-test | 503 from atenet-router — xDS routes not yet propagated when actor reaches STATUS_RUNNING |
TestActorLifecycle |
4× | e2e-test | Same: 503 after resume, callActor 30s deadline too tight |
TestMultipleDurableDirLifecycle |
2× | e2e-test | Same |
TestSyncer_UpdateWorker_RetryOnVersionConflict |
2× | run-tests | wait.PollUntilContextTimeout 5s deadline too tight on loaded CI runners |
TestLoaderConcurrentHandshakes |
2× | run-tests | Concurrent symlink rotation — exact failure mode unknown, needs investigation |
Root Cause
e2e demo tests (demo_test.go)
After waitForActorStatus returns STATUS_RUNNING, the atenet-router's xDS route for the actor isn't always ready immediately. callActor retries for 30 seconds, but on a loaded CI runner 503s can persist longer than that. The xDS route propagation delay was already identified and fixed for the networking ingress tests in #724 — the same race affects the demo lifecycle tests.
TestSyncer_UpdateWorker_RetryOnVersionConflict
The test asserts via wait.PollUntilContextTimeout(..., 5*time.Second, ...). The syncer's rate-limiting work queue adds backoff between retries; on a loaded runner the 5s window is sometimes exhausted before the second attempt completes.
TestLoaderConcurrentHandshakes
Failure details unavailable from CI logs (logs were truncated / run was re-triggered). Needs a reproduction to diagnose. Leaving for follow-up.
Fix
See the linked PR. Changes:
internal/e2e/suites/demo/demo_test.go:callActordeadline 30s → 90s;waitForActorStatustimeout 60s → 120scmd/ateapi/internal/controlapi/syncer_test.go: final assertion poll timeout 5s → 15s
Activity
hack/metrics: add ci-failure-analysis script (#801)
TestSyncer_Lifecycle no longer flakes on a worker that is still UNAVA…