Skip to content

stress: increase timeout for server waiting after TERM#43277

Merged
alexey-milovidov merged 3 commits intoClickHouse:masterfrom
azat:install/increase-timeout
Nov 27, 2022
Merged

stress: increase timeout for server waiting after TERM#43277
alexey-milovidov merged 3 commits intoClickHouse:masterfrom
azat:install/increase-timeout

Conversation

@azat
Copy link
Copy Markdown
Member

@azat azat commented Nov 16, 2022

Changelog category (leave one):

  • Not for changelog (changelog entry is not required)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

stress: increase timeout for server waiting after TERM

Greater timeout after TERM may reduce about of KILL (like in 1), let's try.

@robot-ch-test-poll robot-ch-test-poll added the pr-not-for-changelog This PR should not be mentioned in the changelog label Nov 16, 2022
@alesapin alesapin self-assigned this Nov 16, 2022
@tavplubix
Copy link
Copy Markdown
Member

(like in 1)

In this example Stress tests have found a real issue: after #42222 removal outdated parts from DiskS3 slowed down significantly. However, the only reason why we call MergeTreeData::clearOldPartsFromFilesystem in StorageMergeTree::shutdown is to avoid resurrection of parts dropped by DROP PARTITION after server restart. Of course it does not work and parts still can resurrect if server was killed. After #41145 we will be able to simply remove this call to clearOldPartsFromFilesystem.

I would prefer to fix the issue instead of increasing timeout. But maybe it's okay temporarily increase it and then decrease back after removing clearOldPartsFromFilesystem.

@azat
Copy link
Copy Markdown
Member Author

azat commented Nov 18, 2022

I would prefer to fix the issue instead of increasing timeout.

Yep, this is just a workaround to avoid noisy CI, and I think eventually it will be a problem even with increased timeout, but just not that frequent.

@alesapin
Copy link
Copy Markdown
Member

Why 5 functional tests failed????

@alesapin
Copy link
Copy Markdown
Member

Let's see is it stable

@tavplubix
Copy link
Copy Markdown
Member

Why 5 functional tests failed????

Some of these tests are known to be flaky in debug build (they are just too slow), and usually they fail all together

@alexey-milovidov alexey-milovidov mentioned this pull request Nov 21, 2022
@alesapin
Copy link
Copy Markdown
Member

@azat std::exception. Code: 1001, type: boost::wrapexcept<boost::program_options::unknown_option>, e.what() = unrecognised option '--timeout' (version 22.11.1.1360 (official build))

@azat azat marked this pull request as draft November 22, 2022 18:40
@azat azat force-pushed the install/increase-timeout branch from 52f12e5 to 7346762 Compare November 26, 2022 19:31
@azat
Copy link
Copy Markdown
Member Author

azat commented Nov 26, 2022

std::exception. Code: 1001, type: boost::wrapexceptboost::program_options::unknown_option, e.what() = unrecognised option '--timeout' (version 22.11.1.1360 (official build))

Yes, this is because it was used with the binary from previous version. Fixed.

@azat azat marked this pull request as ready for review November 26, 2022 19:33
azat added 3 commits November 27, 2022 09:40
Increasing timeout may help in case of slow server (debug, sanitizers).

Signed-off-by: Azat Khuzhin <[email protected]>
Greater timeout after TERM may reduce about of KILL, let's try.

Signed-off-by: Azat Khuzhin <[email protected]>
@azat azat force-pushed the install/increase-timeout branch from 7346762 to ca16f51 Compare November 27, 2022 08:40
@alexey-milovidov alexey-milovidov merged commit 2b5fe56 into ClickHouse:master Nov 27, 2022
@azat azat deleted the install/increase-timeout branch November 30, 2022 04:55
@azat
Copy link
Copy Markdown
Member Author

azat commented Nov 30, 2022

FWIW Not a lot of difference after merging this (play).

@alexey-milovidov
Copy link
Copy Markdown
Member

It did not help.

@azat
Copy link
Copy Markdown
Member Author

azat commented Feb 16, 2023

@alexey-milovidov It hasn't been used at that time, since it was broken in #44197, , and was gotten back in minor cleanup - #45136

yokofly added a commit to timeplus-io/proton that referenced this pull request Mar 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-not-for-changelog This PR should not be mentioned in the changelog

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants