Skip to content

bump defaultReconciliationTimeout to 60 sec#36788

Closed
andrewhsu wants to merge 1 commit intomoby:masterfrom
andrewhsu:l
Closed

bump defaultReconciliationTimeout to 60 sec#36788
andrewhsu wants to merge 1 commit intomoby:masterfrom
andrewhsu:l

Conversation

@andrewhsu
Copy link
Copy Markdown
Contributor

@andrewhsu andrewhsu commented Apr 4, 2018

- What I did

bump defaultReconciliationTimeout from 30 sec to 60 sec

To accommodate for increase in leader election from 3 seconds to 10
seconds
. This will give more cycles to find another leader.

- How I did it

in the tests

- How to verify it

run DockerSwarmSuite.TestAPISwarmLeaderElection test

- Description for the changelog

N/A

- A picture of a cute animal (not mandatory but encouraged)
🐋

To accomodate for increase in leader election from 3 seconds to 10
seconds. This will give more cycles to find another leader.

Signed-off-by: Andrew Hsu <[email protected]>
@andrewhsu
Copy link
Copy Markdown
Contributor Author

andrewhsu commented Apr 4, 2018

cc @cyli @chungers @anshulpundir

Copy link
Copy Markdown
Contributor

@cyli cyli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - I'm not sure if this bumps the tests up enough - it was 30 seconds before, which means that if the leader election tick were 3 seconds, it's possible to get up to 10 leader elections. With 60 seconds and a 10 second election tick, it's possible to get up to 6.

I think this will be sufficient, but just wanted to mention that it doesn't give us the exact number of possible elections we might have had before.

)

var defaultReconciliationTimeout = 30 * time.Second
var defaultReconciliationTimeout = 60 * time.Second
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a comment for how this timeout is used.

@andrewhsu
Copy link
Copy Markdown
Contributor Author

@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 5, 2018

Codecov Report

Merging #36788 into master will decrease coverage by 0.29%.
The diff coverage is n/a.

@@            Coverage Diff            @@
##           master   #36788     +/-   ##
=========================================
- Coverage   35.22%   34.93%   -0.3%     
=========================================
  Files         614      614             
  Lines       45645    45645             
=========================================
- Hits        16080    15944    -136     
- Misses      27435    27607    +172     
+ Partials     2130     2094     -36

@tonistiigi
Copy link
Copy Markdown
Member

If this is for a specific test then there is no need to raise it for everything. We already raise it in couple of places where needed, eg.

waitAndAssert(c, defaultReconciliationTimeout*2, reducedCheck(sumAsIntegers, d1.CheckActiveContainerCount, d2.CheckActiveContainerCount), checker.Equals, instances)

@anshulpundir
Copy link
Copy Markdown
Contributor

I agree with @tonistiigi's suggestion to only raise it for the test in question. @andrewhsu

@cyli
Copy link
Copy Markdown
Contributor

cyli commented Apr 26, 2018

I am fine with bumping it up only for this test - but there might be some other tests that do not specifically test leader election, but may trigger leader elections (because we are taking down nodes), which could become more flaky as well.

Update: Actually I think I may be confusing the swarmkit integration tests with these, the two in particular that I was thinking of do seem to try to preserve the leader. So possibly not an issue, but something to keep in mind.

@anshulpundir
Copy link
Copy Markdown
Contributor

Agree with @cyli. We can always change increase the timeout for tests as and when necessary.

BTW, there are concerns that we might be masking bugs. And so maybe try with a smaller timeout increase ?

@andrewhsu andrewhsu closed this Jul 12, 2019
@andrewhsu andrewhsu deleted the l branch July 12, 2019 21:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants