integration-cli: fix swarm tests flakiness #39531

tonistiigi · 2019-07-15T21:19:57Z

relates to / addresses:

Flaky test: TestAPISwarmServicesCreate #36903 Flaky test: TestAPISwarmServicesCreate
Flaky test: TestAPISwarmRaftQuorum #34988 Flaky test: TestAPISwarmRaftQuorum
Flaky test: TestServicePlugin, TestCreateServiceSecretFileMode #37132 Flaky test: TestServicePlugin, TestCreateServiceSecretFileMode
Flaky test: TestAPISwarmLeaderElection #32673 Flaky test: TestAPISwarmLeaderElection
Flaky test: DockerSwarmSuite.TestSwarmRotateUnlockKey #28240 Flaky test: DockerSwarmSuite.TestSwarmRotateUnlockKey
Flaky test: DockerSwarmSuite.TestSwarmClusterRotateUnlockKey #38885 Flaky test: DockerSwarmSuite.TestSwarmClusterRotateUnlockKey
- related fix for this test: Fix TestSwarmClusterRotateUnlockKey #39616 Fix TestSwarmClusterRotateUnlockKey

SwarmKit bump:

full diff: moby/swarmkit@fb584e7...7dded76

includes:

Fix flaky tests swarmkit#2808 Fix flaky tests
Swap gometalinter for golangci-lint swarmkit#2866 Swap gometalinter for golangci-lint
Increase max recv gRPC message size to initialize connection broker swarmkit#2869 Increase max recv gRPC message size to initialize connection broker
Fix update out of sequence swarmkit#2870 Fix update out of sequence

thaJeztah · 2019-07-15T22:53:45Z

DockerSwarmSuite.TestAPISwarmRaftQuorum failed on Janky and Experimental

DockerSwarmSuite.TestAPISwarmServicesCreate also failed on Experimental

codecov · 2019-07-16T00:25:51Z

Codecov Report

❗ No coverage uploaded for pull request base (master@42a0544). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master   #39531   +/-   ##
=========================================
  Coverage          ?   37.31%           
=========================================
  Files             ?      609           
  Lines             ?    45234           
  Branches          ?        0           
=========================================
  Hits              ?    16881           
  Misses            ?    26062           
  Partials          ?     2291

tonistiigi · 2019-07-16T00:33:41Z

Pushed updates for: TestAPISwarmLeaderElection and TestAPISwarmRaftQuorum. TestAPISwarmLeaderElection now allows temporary context deadline exceeded and no-leader errors while the leader is switching. Ideally, these should never happen and we can create an issue to track if it is possible to guarantee that. The test still checks that leader election eventually happens and cluster state normalizes. For TestAPISwarmRaftQuorum I increased the timeout from default 30s like some other tests do as well. Probably for similar reasons as TestAPISwarmLeaderElection it sometimes gets deadline exceeded errors and when this happens it should have been already waited for 20sec, plus possible wait for heartbeats. If this doesn't help then temporarily I wouldn't mind allowing deadline exceeded error as well in addition to the specific error. More important part of the test is to check that the state normalizes.

For TestAPISwarmServicesCreate this seems actual swarm error with no workaround in the test. In the logs I see.

time="2019-07-15T22:22:17.270689016Z" level=error msg="failed transaction: update task desired state to REMOVE" error="update out of sequence" module=node node.id=puuje6xefqjhl5k42bfmqgh9v
time="2019-07-15T22:22:17.270715610Z" level=error msg="failed transaction: update task desired state to REMOVE" error="update out of sequence" module=node node.id=puuje6xefqjhl5k42bfmqgh9v

when it tries to remove a service.

tonistiigi · 2019-07-16T04:47:22Z

In experimental TestServicePlugin now failed from integration suite. The cause seems same as TestAPISwarmServicesCreate before.

time="2019-07-16T03:20:19.393117770Z" level=error msg="failed transaction: update task desired state to REMOVE" error="update out of sequence" module=node node.id=740tzgyr852ufnycchjw39kbp
time="2019-07-16T03:20:19.393133366Z" level=error msg="failed transaction: update task desired state to REMOVE" error="update out of sequence" module=node node.id=740tzgyr852ufnycchjw39kbp
time="2019-07-16T03:20:19.393148882Z" level=error msg="failed transaction: update task desired state to REMOVE" error="update out of sequence" module=node node.id=740tzgyr852ufnycchjw39kbp

If I understand correctly this is a completely internal error in swarm that causes the state change to be ignored with no way to control or get an error from the client.

All of these are covered in https://gist.github.com/tonistiigi/849f6551003fff2da43cbe0e21172e23

dperny · 2019-07-16T17:30:47Z

Opened a PR to fix the Update out of sequence error: moby/swarmkit#2870

tonistiigi · 2019-07-16T21:10:10Z

Reverted memberlist change to check if it has an effect on tests or can be worked on separately.

Signed-off-by: Tonis Tiigi <[email protected]>

andrewhsu

LGTM w.r.t. reducing the flakiness of tests

thaJeztah

LGTM

crosbymichael · 2019-07-18T14:09:30Z

LGTM

…3 branch) full diff: moby/swarmkit@4fb9e96...bbe3418 changes included: - moby/swarmkit#2889 [19.03 backport] Fix update out of sequence and increase max recv gRPC message size for nodes and secrets Which relates to - moby#39531 integration-cli: fix swarm tests flakiness - docker-archive#345 [19.03 backport] integration-cli: fix swarm tests flakiness And includes backports of - moby/swarmkit#2808 Fix flaky tests - moby/swarmkit#2866 Swap gometalinter for golangci-lint - moby/swarmkit#2869 Increase max recv gRPC message size to initialize connection broker - related / similar to moby#38103 / docker-archive#102 cluster: set bigger grpc limit for array requests - related / similar to moby#39306 Increase max recv gRPC message size for nodes and secrets - fixes moby/swarmkit#2733 Error generated when messages size is too big - moby/swarmkit#2870 Fix update out of sequence Signed-off-by: Sebastiaan van Stijn <[email protected]>

…3 branch) full diff: moby/swarmkit@4fb9e96...bbe3418 changes included: - moby/swarmkit#2889 [19.03 backport] Fix update out of sequence and increase max recv gRPC message size for nodes and secrets Which relates to - moby/moby#39531 integration-cli: fix swarm tests flakiness - docker-archive/engine#345 [19.03 backport] integration-cli: fix swarm tests flakiness And includes backports of - moby/swarmkit#2808 Fix flaky tests - moby/swarmkit#2866 Swap gometalinter for golangci-lint - moby/swarmkit#2869 Increase max recv gRPC message size to initialize connection broker - related / similar to moby/moby#38103 / docker-archive/engine#102 cluster: set bigger grpc limit for array requests - related / similar to moby/moby#39306 Increase max recv gRPC message size for nodes and secrets - fixes moby/swarmkit#2733 Error generated when messages size is too big - moby/swarmkit#2870 Fix update out of sequence Signed-off-by: Sebastiaan van Stijn <[email protected]> Upstream-commit: f7dbee3eeaa1dd218116f85b8f60361acbd5b214 Component: engine

thaJeztah · 2019-10-28T18:19:44Z

integration-cli/docker_api_swarm_test.go

 			for _, d := range nodes {
-				if d.GetNode(c, d.NodeID()).ManagerStatus.Leader {
+				n := d.GetNode(c, d.NodeID(), func(err error) bool {
+					if strings.Contains(errors.Cause(err).Error(), context.DeadlineExceeded.Error()) || strings.Contains(err.Error(), "swarm does not have a leader") {


Perhaps we should update this to take the same approach as #39168

GordonTheTurtle added the status/0-triage label Jul 15, 2019

thaJeztah added the area/testing label Jul 15, 2019

tonistiigi force-pushed the swarm-ci-check branch from c2ebe91 to fc74a0b Compare July 16, 2019 21:16

tonistiigi added 3 commits July 17, 2019 11:52

integration-cli: allow temporary errors on leader switch

3df1095

Signed-off-by: Tonis Tiigi <[email protected]>

integration-cli: allow temporary no-leader error

52e0dfe

Signed-off-by: Tonis Tiigi <[email protected]>

vendor: update swarmkit to 7dded76

096a7af

Signed-off-by: Tonis Tiigi <[email protected]>

tonistiigi force-pushed the swarm-ci-check branch from fc74a0b to 096a7af Compare July 17, 2019 18:54

tonistiigi changed the title ~~integration-cli: check memberlist bindport in ci~~ integration-cli: fix swarm tests flakiness Jul 17, 2019

tonistiigi marked this pull request as ready for review July 17, 2019 20:26

andrewhsu mentioned this pull request Jul 18, 2019

PR checks on windowsRS1 fail to start then time out #39556

Closed

andrewhsu approved these changes Jul 18, 2019

View reviewed changes

thaJeztah approved these changes Jul 18, 2019

View reviewed changes

thaJeztah added process/cherry-pick status/2-code-review and removed status/0-triage labels Jul 18, 2019

andrewhsu mentioned this pull request Jul 18, 2019

flaky DockerSwarmSuite tests #39499

Closed

crosbymichael merged commit fd6f0b1 into moby:master Jul 18, 2019

This was referenced Jul 18, 2019

Rewrite swarm leadership test #39550

Open

skip frequently flaky DockerSwarmSuite tests #39501

Closed

Dockerfile: use GO_VERSION build-arg for overriding Go version #39548

Merged

thaJeztah mentioned this pull request Jul 26, 2019

Fix TestSwarmClusterRotateUnlockKey #39616

Merged

This was referenced Sep 1, 2019

[19.03 backport] integration-cli: fix swarm tests flakiness docker-archive/engine#345

Merged

[18.09 backport] integration-cli: fix swarm tests flakiness docker-archive/engine#346

Merged

thaJeztah mentioned this pull request Sep 6, 2019

[19.03 backport] Fix update out of sequence and increase max recv gRPC message size for nodes and secrets moby/swarmkit#2889

Merged

thaJeztah mentioned this pull request Sep 12, 2019

[19.03 backport] bump swarmkit to bbe341867eae1615faf8a702ec05bfe986e73e06 (bump_v19.03 branch) docker-archive/engine#353

Merged

This was referenced Sep 25, 2019

[18.09 backport] Fix update out of sequence and increase max recv gRPC message size for nodes and secrets moby/swarmkit#2900

Merged

[18.03 backport] Fix update out of sequence moby/swarmkit#2902

Merged

thaJeztah added process/cherry-picked and removed process/cherry-pick labels Oct 8, 2019

thaJeztah mentioned this pull request Oct 10, 2019

[18.09 backport] integration-cli: Skip TestAPIImagesSaveAndLoad on RS3 and older docker-archive/engine#394

Merged

thaJeztah reviewed Oct 28, 2019

View reviewed changes

thaJeztah added this to the 20.03.0 milestone Apr 2, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

integration-cli: fix swarm tests flakiness #39531

integration-cli: fix swarm tests flakiness #39531

Uh oh!

tonistiigi commented Jul 15, 2019 •

edited by thaJeztah

Loading

Uh oh!

thaJeztah commented Jul 15, 2019

Uh oh!

codecov bot commented Jul 16, 2019 •

edited

Loading

Uh oh!

tonistiigi commented Jul 16, 2019

Uh oh!

tonistiigi commented Jul 16, 2019

Uh oh!

dperny commented Jul 16, 2019

Uh oh!

tonistiigi commented Jul 16, 2019

Uh oh!

andrewhsu left a comment

Uh oh!

thaJeztah left a comment

Uh oh!

crosbymichael commented Jul 18, 2019

Uh oh!

thaJeztah Oct 28, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

integration-cli: fix swarm tests flakiness #39531

integration-cli: fix swarm tests flakiness #39531

Uh oh!

Conversation

tonistiigi commented Jul 15, 2019 • edited by thaJeztah Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thaJeztah commented Jul 15, 2019

Uh oh!

codecov bot commented Jul 16, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

tonistiigi commented Jul 16, 2019

Uh oh!

tonistiigi commented Jul 16, 2019

Uh oh!

dperny commented Jul 16, 2019

Uh oh!

tonistiigi commented Jul 16, 2019

Uh oh!

andrewhsu left a comment

Choose a reason for hiding this comment

Uh oh!

thaJeztah left a comment

Choose a reason for hiding this comment

Uh oh!

crosbymichael commented Jul 18, 2019

Uh oh!

thaJeztah Oct 28, 2019

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

tonistiigi commented Jul 15, 2019 •

edited by thaJeztah

Loading

codecov bot commented Jul 16, 2019 •

edited

Loading