Skip to content

Add image to run jepsen tests#22373

Merged
alesapin merged 23 commits intomasterfrom
jepsen_for_ci
Apr 4, 2021
Merged

Add image to run jepsen tests#22373
alesapin merged 23 commits intomasterfrom
jepsen_for_ci

Conversation

@alesapin
Copy link
Copy Markdown
Member

@alesapin alesapin commented Mar 30, 2021

I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en

Changelog category (leave one):

  • Build/Testing/Packaging Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Add on-demand check for clickhouse Keeper.

@robot-clickhouse robot-clickhouse added the pr-build Pull request with build/testing/packaging improvement label Mar 30, 2021
@alesapin alesapin changed the title Add image to run jepsen tests [WIP] Add image to run jepsen tests Mar 30, 2021
@alesapin alesapin marked this pull request as draft March 30, 2021 16:22
@alesapin
Copy link
Copy Markdown
Member Author

alesapin commented Mar 31, 2021 •
edited
Loading

For some reason sometimes clickhouse keeper is not able to start on nodes after binary download. When we run from already downloaded everything looks good.

@alesapin
Copy link
Copy Markdown
Member Author

alesapin commented Mar 31, 2021 •
edited
Loading

Sigh https://clickhouse-test-reports.s3.yandex.net/22373/019d4b3ebd0c5a04e49add5af39631592cecdf95/clickhouse_keeper_jepsen/jepsen_run_all_tests.log

7 successes
0 unknown
3 crashed <-- something wrong with Jepsen or startup
0 failures

@alesapin
Copy link
Copy Markdown
Member Author

alesapin commented Mar 31, 2021 •
edited
Loading

This check should happen each second but something happens with VM:

INFO [2021-03-31 19:27:47,035] jepsen node n1 - jepsen.clickhouse-keeper.utils Checking server alive on 
??? what is going on here?
INFO [2021-03-31 19:28:10,370] jepsen node n1 - jepsen.clickhouse-keeper.utils Checking server alive on

@alesapin
Copy link
Copy Markdown
Member Author

alesapin commented Apr 1, 2021

Found deadlock eBay/NuRaft#190.

@alesapin
Copy link
Copy Markdown
Member Author

alesapin commented Apr 2, 2021

Ok https://clickhouse-test-reports.s3.yandex.net/22373/1fd7e7ccf10122271ec169807327ebe9068ab59d/clickhouse_keeper_jepsen.html

@alesapin alesapin changed the title [WIP] Add image to run jepsen tests Add image to run jepsen tests Apr 2, 2021
@alesapin alesapin added the jepsen-test Need to test this PR with jepsen tests label Apr 2, 2021
@alesapin alesapin marked this pull request as ready for review April 2, 2021 09:27
@alesapin
Copy link
Copy Markdown
Member Author

alesapin commented Apr 2, 2021 •
edited
Loading 

# Successful tests

store/clickhouse-keeper-quorum=true-counter-simple-partitioner/20210402T105726.000Z
store/clickhouse-keeper-quorum=true-total-queue-random-node-killer/20210402T110101.000Z
store/clickhouse-keeper-quorum=true-total-queue-all-nodes-hammer-time/20210402T110424.000Z
store/clickhouse-keeper-quorum=true-unique-ids-simple-partitioner/20210402T110659.000Z
store/clickhouse-keeper-quorum=true-unique-ids-random-node-killer/20210402T111013.000Z
store/clickhouse-keeper-quorum=true-set-simple-partitioner/20210402T111341.000Z
store/clickhouse-keeper-quorum=true-unique-ids-bridge-partitioner/20210402T111811.000Z
store/clickhouse-keeper-quorum=true-counter-bridge-partitioner/20210402T112123.000Z
store/clickhouse-keeper-quorum=true-counter-random-node-killer/20210402T112425.000Z
store/clickhouse-keeper-quorum=true-unique-ids-blind-others-partitioner/20210402T112714.000Z
store/clickhouse-keeper-quorum=true-total-queue-blind-others-partitioner/20210402T113019.000Z
store/clickhouse-keeper-quorum=true-set-all-nodes-hammer-time/20210402T113308.000Z
store/clickhouse-keeper-quorum=true-set-blind-others-partitioner/20210402T113629.000Z
store/clickhouse-keeper-quorum=true-counter-blind-others-partitioner/20210402T113938.000Z
store/clickhouse-keeper-quorum=true-counter-all-nodes-hammer-time/20210402T114252.000Z
store/clickhouse-keeper-quorum=true-total-queue-bridge-partitioner/20210402T114549.000Z
store/clickhouse-keeper-quorum=true-unique-ids-blind-node-partitioner/20210402T114803.000Z
store/clickhouse-keeper-quorum=true-set-random-node-killer/20210402T115030.000Z
store/clickhouse-keeper-quorum=true-counter-blind-node-partitioner/20210402T115316.000Z
store/clickhouse-keeper-quorum=true-total-queue-simple-partitioner/20210402T115642.000Z
store/clickhouse-keeper-quorum=true-total-queue-blind-node-partitioner/20210402T120225.000Z
store/clickhouse-keeper-quorum=true-unique-ids-all-nodes-hammer-time/20210402T120532.000Z

# Crashed tests

clickhouse-keeper-quorum=true-set-bridge-partitioner
clickhouse-keeper-quorum=true-set-blind-node-partitioner

22 successes
0 unknown
2 crashed

@robot-clickhouse robot-clickhouse added the submodule changed At least one submodule changed in this PR. label Apr 3, 2021
@alesapin
Copy link
Copy Markdown
Member Author

alesapin commented Apr 4, 2021

One or two tests always crashed https://clickhouse-test-reports.s3.yandex.net/22373/75cd37f359c2cc3690734e3b50cf320dc21cbce6/clickhouse_keeper_jepsen.html

@alesapin
Copy link
Copy Markdown
Member Author

alesapin commented Apr 4, 2021

Yes, when we don't download binary before the test everything works as expected https://clickhouse-test-reports.s3.yandex.net/22373/c7da76e633602f22f56003ce5e7f0428376e752c/clickhouse_keeper_jepsen.html. Seems like binary download is a huge stress for our virtual machines...

@alesapin
Copy link
Copy Markdown
Member Author

alesapin commented Apr 4, 2021

One more time

@alesapin
Copy link
Copy Markdown
Member Author

alesapin commented Apr 4, 2021 •
edited
Loading

Good, and takes 45 minutes https://clickhouse-test-reports.s3.yandex.net/22373/76d0e60e264a5b9bf40ef1fcef875512bd7a8517/clickhouse_keeper_jepsen.html. I think I can add more workload + nemesis combinations.

@alesapin alesapin merged commit caaa19e into master Apr 4, 2021
@alesapin alesapin deleted the jepsen_for_ci branch April 4, 2021 15:35
@alesapin alesapin mentioned this pull request Apr 12, 2021
Closed
@alexey-milovidov alexey-milovidov mentioned this pull request Nov 24, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

jepsen-test Need to test this PR with jepsen tests pr-build Pull request with build/testing/packaging improvement submodule changed At least one submodule changed in this PR.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@alesapin @robot-clickhouse