Skip to content

reduce flakiness of TestSwarmLockUnlockCluster and TestSwarmJoinPromoteLocked#38546

Merged
AkihiroSuda merged 1 commit intomoby:masterfrom
thaJeztah:wait_a_second
Jan 15, 2019
Merged

reduce flakiness of TestSwarmLockUnlockCluster and TestSwarmJoinPromoteLocked#38546
AkihiroSuda merged 1 commit intomoby:masterfrom
thaJeztah:wait_a_second

Conversation

@thaJeztah
Copy link
Member

I noticed that this test failed, because the node was in status "pending".

The test checks for the node's status immediately after it was restarted, so
possibly it needs some time to unlock.

14:07:10 FAIL: docker_cli_swarm_test.go:1128: DockerSwarmSuite.TestSwarmLockUnlockCluster
...
14:07:10 docker_cli_swarm_test.go:1168:
14:07:10     checkSwarmLockedToUnlocked(c, d)
14:07:10 docker_cli_swarm_test.go:1017:
14:07:10     c.Assert(getNodeStatus(c, d), checker.Equals, swarm.LocalNodeStateActive)
14:07:10 ... obtained swarm.LocalNodeState = "pending"
14:07:10 ... expected swarm.LocalNodeState = "active"

This patch adds a waitAndAssert for the node's status, with a 1 second timeout.

relates to #34051 and #35263

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a temporary commit to trigger our new "flaky check" 🎉 @olljanat 🤞

Copy link
Contributor

@olljanat olljanat Jan 12, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@thaJeztah integration-cli/ is deprecated so flaky check is only looking for new tests under integration/ folder...

How ever you can easily test that old test on locally by running commands:

export TEST_REPEAT="5"
export TIMEOUT="30m"
export TESTFLAGS="-test.count 5 -check.f TestSwarmJoinPromoteLocked"
make test-integration

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LOL, yes, just thought of that hahaha

19:15:21 The following new tests were added to integration-cli:

19:15:21 +func (s *DockerSwarmSuite) TestUnlockEngineAndUnlockedSwarm_FLAKY(c *check.C) {
19:15:21 +func (s *DockerSwarmSuite) TestSwarmJoinPromoteLocked_FLAKY(c *check.C) {
19:15:21 
19:15:21 integration-cli is deprecated. Please add an API integration test to
19:15:21 ./integration/COMPONENT/. See ./TESTING.md for more details.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But you can of course also set those to

: ${TEST_REPEAT:=1}
: ${TESTFLAGS:=}
if you want stress test old test with CI.

@thaJeztah thaJeztah changed the title reduce flakiness of TestUnlockEngineAndUnlockedSwarm and TestSwarmJoinPromoteLocked reduce flakiness of TestSwarmLockUnlockCluster and TestSwarmJoinPromoteLocked Jan 12, 2019
…teLocked

I noticed that this test failed, because the node was in status "pending".

The test checks for the node's status immediately after it was restarted, so
possibly it needs some time to unlock.

    14:07:10 FAIL: docker_cli_swarm_test.go:1128: DockerSwarmSuite.TestSwarmLockUnlockCluster
    ...
    14:07:10 docker_cli_swarm_test.go:1168:
    14:07:10     checkSwarmLockedToUnlocked(c, d)
    14:07:10 docker_cli_swarm_test.go:1017:
    14:07:10     c.Assert(getNodeStatus(c, d), checker.Equals, swarm.LocalNodeStateActive)
    14:07:10 ... obtained swarm.LocalNodeState = "pending"
    14:07:10 ... expected swarm.LocalNodeState = "active"

This patch adds a `waitAndAssert` for the node's status, with a 1 second timeout.

Signed-off-by: Sebastiaan van Stijn <[email protected]>
@thaJeztah
Copy link
Member Author

So far so good; I'll only have to push once more (as I made a copy/pasta error in my commit message 😂)

make TEST_REPEAT=5 TIMEOUT=30m TESTFLAGS="-test.count 5 -check.f TestSwarmAlternateLockUnlock" test-integration

INFO: Testing against a local daemon
PASS: docker_cli_swarm_test.go:1394: DockerSwarmSuite.TestSwarmAlternateLockUnlock	41.862s
OK: 1 passed
PASS: docker_cli_swarm_test.go:1394: DockerSwarmSuite.TestSwarmAlternateLockUnlock	40.127s
OK: 1 passed
PASS: docker_cli_swarm_test.go:1394: DockerSwarmSuite.TestSwarmAlternateLockUnlock	39.261s
OK: 1 passed
PASS: docker_cli_swarm_test.go:1394: DockerSwarmSuite.TestSwarmAlternateLockUnlock	38.891s
OK: 1 passed
PASS: docker_cli_swarm_test.go:1394: DockerSwarmSuite.TestSwarmAlternateLockUnlock	40.428s
OK: 1 passed
PASS

make TEST_REPEAT=5 TIMEOUT=30m TESTFLAGS="-test.count 5 -check.f TestSwarmInitLocked" test-integration

PASS: docker_cli_swarm_test.go:1053: DockerSwarmSuite.TestSwarmInitLocked	15.855s
OK: 1 passed
PASS: docker_cli_swarm_test.go:1053: DockerSwarmSuite.TestSwarmInitLocked	17.692s
OK: 1 passed
PASS: docker_cli_swarm_test.go:1053: DockerSwarmSuite.TestSwarmInitLocked	17.059s
OK: 1 passed
PASS: docker_cli_swarm_test.go:1053: DockerSwarmSuite.TestSwarmInitLocked	16.337s
OK: 1 passed
PASS: docker_cli_swarm_test.go:1053: DockerSwarmSuite.TestSwarmInitLocked	16.426s
OK: 1 passed
PASS

make TEST_REPEAT=5 TIMEOUT=30m TESTFLAGS="-test.count 5 -check.f TestSwarmJoinPromoteLocked" test-integration

INFO: Testing against a local daemon
PASS: docker_cli_swarm_test.go:1189: DockerSwarmSuite.TestSwarmJoinPromoteLocked	44.767s
OK: 1 passed
PASS: docker_cli_swarm_test.go:1189: DockerSwarmSuite.TestSwarmJoinPromoteLocked	46.623s
OK: 1 passed
PASS: docker_cli_swarm_test.go:1189: DockerSwarmSuite.TestSwarmJoinPromoteLocked	43.387s
OK: 1 passed
PASS: docker_cli_swarm_test.go:1189: DockerSwarmSuite.TestSwarmJoinPromoteLocked	51.256s
OK: 1 passed
PASS: docker_cli_swarm_test.go:1189: DockerSwarmSuite.TestSwarmJoinPromoteLocked	48.165s
OK: 1 passed
PASS

make TEST_REPEAT=5 TIMEOUT=30m TESTFLAGS="-test.count 5 -check.f TestSwarmLockUnlockCluster" test-integration

INFO: Testing against a local daemon
PASS: docker_cli_swarm_test.go:1128: DockerSwarmSuite.TestSwarmLockUnlockCluster	137.996s
OK: 1 passed
PASS: docker_cli_swarm_test.go:1128: DockerSwarmSuite.TestSwarmLockUnlockCluster	116.312s
OK: 1 passed
PASS: docker_cli_swarm_test.go:1128: DockerSwarmSuite.TestSwarmLockUnlockCluster	119.550s
OK: 1 passed
PASS: docker_cli_swarm_test.go:1128: DockerSwarmSuite.TestSwarmLockUnlockCluster	132.593s
OK: 1 passed
PASS: docker_cli_swarm_test.go:1128: DockerSwarmSuite.TestSwarmLockUnlockCluster	129.903s
OK: 1 passed
PASS

@codecov
Copy link

codecov bot commented Jan 12, 2019

Codecov Report

Merging #38546 into master will increase coverage by 0.02%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #38546      +/-   ##
==========================================
+ Coverage   36.62%   36.65%   +0.02%     
==========================================
  Files         608      608              
  Lines       45173    45173              
==========================================
+ Hits        16545    16558      +13     
+ Misses      26341    26333       -8     
+ Partials     2287     2282       -5

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants