Skip to content

fix: envoy shutdown flaky test#3646

Merged
arkodg merged 1 commit intoenvoyproxy:mainfrom
guydc:fix-shutdown-flake
Jun 22, 2024
Merged

fix: envoy shutdown flaky test#3646
arkodg merged 1 commit intoenvoyproxy:mainfrom
guydc:fix-shutdown-flake

Conversation

@guydc
Copy link
Copy Markdown
Contributor

@guydc guydc commented Jun 21, 2024

What this PR does / why we need it:

This PR delays readiness of envoy to allow XDS sync as discussed in #2810. This is done specifically for the upgrade test suite gateway class.

Test flakiness seems to be related to 404 responses in early stages of the load test, which likely indicates that new proxy instances are ready before being programmed.

--- FAIL: TestEGUpgrade (230.50s)
    --- PASS: TestEGUpgrade/EGUpgrade (152.90s)
        --- PASS: TestEGUpgrade/EGUpgrade/Upgrade_from_an_older_eg_release_should_succeed (152.89s)
    --- FAIL: TestEGUpgrade/EnvoyShutdown (68.06s)
        --- FAIL: TestEGUpgrade/EnvoyShutdown/All_requests_must_succeed (68.04s)
        utils.go:176: failed to create load: error 404 for http://172.18.255.208/envoy-shutdown (82 bytes)
        utils.go:185: Load completed after 0s with 2 requests, 0 success, 2 failures and 2 errors

Which issue(s) this PR fixes:
Fixes #3262

@guydc guydc requested a review from a team as a code owner June 21, 2024 18:33
arkodg
arkodg approved these changes Jun 21, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@arkodg arkodg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM thanks !

@arkodg arkodg requested review from a team June 21, 2024 18:34
@arkodg
Copy link
Copy Markdown
Contributor

arkodg commented Jun 21, 2024

the 5s delay may be helpful for this e2e, but will need to be tweaked for user environments where the xDS config is large and many envoys are connecting to the control plane at once
A more robust solution would be to use the listener level health check recently added https://gateway.envoyproxy.io/latest/api/extension_types/#healthchecksettings as a active health check in the external loadbalancer. I dont think this can be added to metal lb, but will be useful for cloud environments

@codecov
Copy link
Copy Markdown

codecov bot commented Jun 21, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 68.30%. Comparing base (c752b88) to head (5ea7a67).

Additional details and impacted files 
@@            Coverage Diff             @@
##             main    #3646      +/-   ##
==========================================
+ Coverage   68.27%   68.30%   +0.03%     
==========================================
  Files         170      170              
  Lines       20760    20760              
==========================================
+ Hits        14174    14181       +7     
+ Misses       5566     5560       -6     
+ Partials     1020     1019       -1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@arkodg arkodg merged commit 432b7e3 into envoyproxy:main Jun 22, 2024
bjlhlin pushed a commit to bjlhlin/gateway that referenced this pull request Jun 24, 2024
@guydc @bjlhlin1
fix: envoy shutdown flaky test (envoyproxy#3646)
65a727e 
delay readdiness to allow XDS sync

Signed-off-by: Guy Daich <[email protected]>
Signed-off-by: “bjlhlin” <“[email protected]”>
Signed-off-by: bjlhlin <[email protected]>
bjlhlin pushed a commit to bjlhlin/gateway that referenced this pull request Jun 26, 2024
@guydc @bjlhlin1
fix: envoy shutdown flaky test (envoyproxy#3646)
2afa501 
delay readdiness to allow XDS sync

Signed-off-by: Guy Daich <[email protected]>
Signed-off-by: bjlhlin <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@arkodg arkodg arkodg approved these changes

@shawnh2 shawnh2 shawnh2 approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

flaky: e2e TestEGUpgrade/EnvoyShutdown

3 participants

@guydc @arkodg @shawnh2