Flaky Test: TestSwarmPublishDuplicatePorts on s390 #30427

ddingel · 2017-01-24T23:48:05Z

The CI fails sometime on the TestSwarmPublishDuplicatePorts test case like during Build 1781.

23:01:28 ----------------------------------------------------------------------
23:01:28 FAIL: docker_cli_swarm_test.go:1576: DockerSwarmSuite.TestSwarmPublishDuplicatePorts
23:01:28 
23:01:28 [ddc1e2218a2e3] waiting for daemon to start
23:01:28 [ddc1e2218a2e3] daemon started
23:01:28 
23:01:28 docker_cli_swarm_test.go:1584:
23:01:28     // make sure task has been deployed.
23:01:28     waitAndAssert(c, defaultReconciliationTimeout, d.CheckActiveContainerCount, checker.Equals, 1)
23:01:28 docker_utils_test.go:1125:
23:01:28     c.Assert(v, checker, args...)
23:01:28 ... obtained int = 0
23:01:28 ... expected int = 1
23:01:28 
23:01:28 [ddc1e2218a2e3] exiting daemon
23:01:29 
23:01:29 ----------------------------------------------------------------------

Something interesting is also that: while this test case fail might happen alone, sometimes it does happen also with [Flaky test: TestServiceUpdatePort on s390 #26601](https://github.com/docker/docker/issues/26601) like during [Build 1802](https://jenkins.dockerproject.org/job/Docker%20Master%20(z)/1802/).

17:09:43 ----------------------------------------------------------------------
17:09:43 FAIL: docker_cli_service_update_test.go:14: DockerSwarmSuite.TestServiceUpdatePort
17:09:43 
17:09:43 [d2a63063bb214] waiting for daemon to start
17:09:43 [d2a63063bb214] daemon started
17:09:43 
17:09:43 docker_cli_service_update_test.go:23:
17:09:43     waitAndAssert(c, defaultReconciliationTimeout, d.CheckActiveContainerCount, checker.Equals, 1)
17:09:43 docker_utils_test.go:1125:
17:09:43     c.Assert(v, checker, args...)
17:09:43 ... obtained int = 0
17:09:43 ... expected int = 1
17:09:43 
17:09:43 [d2a63063bb214] exiting daemon
17:09:45 
17:09:45 ----------------------------------------------------------------------

On he other hand only failing TestServiceUpdatePort didn't happen for at least the last 29 builds.

The text was updated successfully, but these errors were encountered:

vdemeester · 2017-03-28T13:08:49Z

still happening https://jenkins.dockerproject.org/job/Docker%20Master%20(s390x)/2238/console

vdemeester · 2017-03-28T13:09:18Z

/cc @aaronlehmann @aboch

tophj-ibm · 2017-04-03T14:17:23Z

definitely not just a z issue, seen on power as well https://jenkins.dockerproject.org/job/Docker%20Master%20(powerpc)/3513/console

michael-holzheu · 2017-04-06T15:07:05Z

This night I tried to reproduce the issue with Docker commit 833f1f4. I ran the test 1000 times on a s390x Debian Jessie host and kernel 4.6 without any failure.
TestSwarmPublishDuplicatePorts-s390x.txt

tophj-ibm · 2017-04-06T15:16:27Z

Thanks @michael-holzheu. This seems to point to another test influencing it somehow, maybe something in the swarm isn't getting cleaned up properly

michael-holzheu · 2017-04-06T15:43:33Z

Maybe obvious, but looking at the logs from failed runs like 2306, 2311, 2313 or 2320 we see that it always took 31 seconds from the test before until we see the FAIL message. For example here the output of run 2320 (09:11:47 to 09:12:18 = 31 seconds):

09:11:47 PASS: docker_cli_swarm_test.go:226: DockerSwarmSuite.TestSwarmPublishAdd	1.048s
09:12:18 
09:12:18 ----------------------------------------------------------------------
09:12:18 FAIL: docker_cli_swarm_test.go:1680: DockerSwarmSuite.TestSwarmPublishDuplicatePorts

So the failure reason is the timeout:

~/docker-fork/integration-cli$ git grep "defaultReconciliationTimeout ="
docker_api_swarm_test.go:var defaultReconciliationTimeout = 30 * time.Second

michael-holzheu · 2017-04-07T09:31:21Z

This seems to point to another test influencing it somehow, maybe something in the swarm isn't getting cleaned up properly

@tophj-ibm : I now tried to run the full DockerSwarmSuite in a loop:

#!/bin/bash
for i in {1..1000}
do
        echo "---------------------------------------------"
        echo "TEST: $i"
        echo "---------------------------------------------"
        TESTFLAGS='-check.f DockerSwarmSuite.*'  \
        DOCKER_GRAPHDRIVER=vfs DOCKER_EXECDRIVER=native TIMEOUT="120m" \
        hack/make.sh test-integration-cli
done

12 loops have been successful, but with run 13 the test cases began to fail

---------------------------------------------
TEST: 13
---------------------------------------------
FAIL: docker_cli_prune_unix_test.go:28: DockerSwarmSuite.TestPruneNetwork
FAIL: docker_cli_service_update_test.go:14: DockerSwarmSuite.TestServiceUpdatePort
FAIL: docker_cli_swarm_test.go:317: DockerSwarmSuite.TestSwarmContainerAttachByNetworkId
FAIL: docker_cli_swarm_test.go:270: DockerSwarmSuite.TestSwarmContainerAutoStart
FAIL: docker_cli_swarm_test.go:292: DockerSwarmSuite.TestSwarmContainerEndpointOptions
FAIL: docker_cli_swarm_test.go:1588: DockerSwarmSuite.TestSwarmNetworkCreateDup
FAIL: docker_cli_swarm_test.go:1464: DockerSwarmSuite.TestSwarmNetworkIPAMOptions
FAIL: docker_cli_swarm_test.go:1680: DockerSwarmSuite.TestSwarmPublishDuplicatePorts
OOPS: 104 passed, 9 skipped, 8 FAILED
--- FAIL: Test (2399.67s)

For runs 14-23 always the following 15 tests fail:

FAIL: docker_api_swarm_test.go:822: DockerSwarmSuite.TestAPIDuplicateNetworks
FAIL: docker_api_swarm_service_test.go:27: DockerSwarmSuite.TestAPIServiceUpdatePort
FAIL: docker_api_swarm_test.go:873: DockerSwarmSuite.TestAPISwarmHealthcheckNone
FAIL: docker_cli_swarm_test.go:1759: DockerSwarmSuite.TestNetworkInspectWithDuplicateNames
FAIL: docker_cli_swarm_test.go:345: DockerSwarmSuite.TestOverlayAttachable
FAIL: docker_cli_swarm_test.go:367: DockerSwarmSuite.TestOverlayAttachableOnSwarmLeave
FAIL: docker_cli_swarm_test.go:394: DockerSwarmSuite.TestOverlayAttachableReleaseResourcesOnFailure
FAIL: docker_cli_prune_unix_test.go:28: DockerSwarmSuite.TestPruneNetwork
FAIL: docker_cli_service_update_test.go:14: DockerSwarmSuite.TestServiceUpdatePort
FAIL: docker_cli_swarm_test.go:317: DockerSwarmSuite.TestSwarmContainerAttachByNetworkId
FAIL: docker_cli_swarm_test.go:270: DockerSwarmSuite.TestSwarmContainerAutoStart
FAIL: docker_cli_swarm_test.go:292: DockerSwarmSuite.TestSwarmContainerEndpointOptions
FAIL: docker_cli_swarm_test.go:1588: DockerSwarmSuite.TestSwarmNetworkCreateDup
FAIL: docker_cli_swarm_test.go:1464: DockerSwarmSuite.TestSwarmNetworkIPAMOptions
FAIL: docker_cli_swarm_test.go:1680: DockerSwarmSuite.TestSwarmPublishDuplicatePorts
OOPS: 97 passed, 9 skipped, 15 FAILED

TestSwarmPublishDuplicatePorts-s390x-full-DockerSwarmSuite.txt

I am not sure if this observation helps us for the TestSwarmPublishDuplicatePorts problem ...

michael-holzheu · 2017-05-16T14:30:07Z

Update

The last failing PR build was run 2020 (Apr 28, 2017): https://jenkins.dockerproject.org/job/Docker-PRs-s390x/2020/

jenkins-s390x-pr$ grep -w "FAIL:" * | grep DuplicatePorts
2020:FAIL: docker_cli_swarm_test.go:1724: DockerSwarmSuite.TestSwarmPublishDuplicatePorts

Since then we had 848 runs where the DuplicatePorts test was successful.

thaJeztah · 2017-05-16T15:21:01Z

Thanks @michael-holzheu - I think we can close this one then, but ping me if I closed prematurely :D

tophj-ibm · 2017-07-27T13:33:45Z

seen failing again on arm in #33892 :(

thaJeztah · 2017-07-27T14:05:45Z

Interesting; let me reopen

Changes included: - libnetwork#2147 Adding logs for ipam state - libnetwork#2143 Fix race conditions in the overlay network driver - possibly addresses moby#36743 services do not start: ingress-sbox is already present - possibly addresses moby#30427 Flaky Test: TestSwarmPublishDuplicatePorts on s390 - possibly addresses moby#36501 Flaky tests: Service "port" tests - libnetwork#2142 Add wait time into xtables lock warning - libnetwork#2135 filter xtables lock warnings when firewalld is active - libnetwork#2140 Switch from x/net/context to context - libnetwork#2134 Adding a recovery mechanism for a split gossip cluster Signed-off-by: Sebastiaan van Stijn <[email protected]>

gianarb added the area/testing label Jan 25, 2017

This was referenced Apr 7, 2017

Flaky test: TestServiceUpdatePort on s390 #26601

Closed

Flaky Test: DockerSuite.TestAPIImagesImportBadSrc #32267

Closed

thaJeztah closed this as completed May 16, 2017

tophj-ibm mentioned this issue Jul 27, 2017

Bump Go to 1.9.2 #33892

Merged

thaJeztah reopened this Jul 27, 2017

tonistiigi mentioned this issue Mar 30, 2018

services do not start: ingress-sbox is already present #36743

Open

This was referenced Apr 20, 2018

Skip some tests requires root uid when run as user… #36913

Merged

Flaky tests: Service "port" tests #36501

Open

thaJeztah mentioned this issue May 20, 2018

bump libnetwork to eb6b2a57955e5c149d47c3973573216e8f8baa09 #37111

Merged

thaJeztah mentioned this issue Jun 18, 2018

[epic] flaky tests #37306

Open

thaJeztah mentioned this issue Feb 1, 2019

Bump SwarmKit to fix proto capitalization #38673

Merged

thaJeztah mentioned this issue Jun 27, 2019

Flaky test: DockerSwarmSuite.TestSwarmNetworkIPAMOptions #39425

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Flaky Test: TestSwarmPublishDuplicatePorts on s390 #30427

Flaky Test: TestSwarmPublishDuplicatePorts on s390 #30427

ddingel commented Jan 24, 2017 •

edited by vdemeester

Loading

vdemeester commented Mar 28, 2017

vdemeester commented Mar 28, 2017

tophj-ibm commented Apr 3, 2017

michael-holzheu commented Apr 6, 2017

tophj-ibm commented Apr 6, 2017

michael-holzheu commented Apr 6, 2017

michael-holzheu commented Apr 7, 2017 •

edited

Loading

michael-holzheu commented May 16, 2017

thaJeztah commented May 16, 2017

tophj-ibm commented Jul 27, 2017

thaJeztah commented Jul 27, 2017

Flaky Test: TestSwarmPublishDuplicatePorts on s390 #30427

Flaky Test: TestSwarmPublishDuplicatePorts on s390 #30427

Comments

ddingel commented Jan 24, 2017 • edited by vdemeester Loading

vdemeester commented Mar 28, 2017

vdemeester commented Mar 28, 2017

tophj-ibm commented Apr 3, 2017

michael-holzheu commented Apr 6, 2017

tophj-ibm commented Apr 6, 2017

michael-holzheu commented Apr 6, 2017

michael-holzheu commented Apr 7, 2017 • edited Loading

michael-holzheu commented May 16, 2017

thaJeztah commented May 16, 2017

tophj-ibm commented Jul 27, 2017

thaJeztah commented Jul 27, 2017

ddingel commented Jan 24, 2017 •

edited by vdemeester

Loading

michael-holzheu commented Apr 7, 2017 •

edited

Loading