Skip to content

Fix rare case when watch response received before request response#28197

Merged
alesapin merged 2 commits intomasterfrom
fix_requests_push
Aug 27, 2021
Merged

Fix rare case when watch response received before request response#28197
alesapin merged 2 commits intomasterfrom
fix_requests_push

Conversation

@alesapin
Copy link
Copy Markdown
Member

I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en

Changelog category (leave one):

  • Bug Fix (user-visible misbehaviour in official stable or prestable release)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Fix a rare bug in clickhouse-keeper when the client can receive a watch response before request-response.

@robot-clickhouse robot-clickhouse added the pr-bugfix Pull request with bugfix, not backported by default label Aug 26, 2021
@alesapin alesapin added the jepsen-test Need to test this PR with jepsen tests label Aug 26, 2021
@alesapin
Copy link
Copy Markdown
Member Author

alesapin commented Aug 27, 2021

Why did so many random tests failed without logs?

@alesapin
Copy link
Copy Markdown
Member Author

@Mergifyio update

@mergify
Copy link
Copy Markdown
Contributor

mergify bot commented Aug 27, 2021

Command update: success

Branch has been successfully updated

@alesapin
Copy link
Copy Markdown
Member Author

#28220

@tavplubix tavplubix self-assigned this Aug 27, 2021
@alesapin
Copy link
Copy Markdown
Member Author

Integration tests (release) — Timeout :( -- unrelated to changes

@alesapin alesapin merged commit fde3cc1 into master Aug 27, 2021
@alesapin alesapin deleted the fix_requests_push branch August 27, 2021 17:20
@tavplubix
Copy link
Copy Markdown
Member

Looks like docker daemon hung:

2021-08-27 14:07:53,006 Executing cmd: cd /place/sandbox-data/ramdrive/1057923636/ClickHouse/tests/integration && timeout -s 9 1h ./runner --tmpfs  --docker-image-version=latest --docker-compose-images-tags=yandex/clickhouse-mysql-golang-client:latest --docker-compose-images-tags=yandex/clickhouse-mysql-java-client:latest --docker-compose-images-tags=yandex/clickhouse-mysql-js-client:latest --docker-compose-images-tags=yandex/clickhouse-mysql-php-client:latest --docker-compose-images-tags=yandex/clickhouse-postgresql-java-client:latest --docker-compose-images-tags=yandex/clickhouse-integration-test:latest --docker-compose-images-tags=yandex/clickhouse-kerberos-kdc:latest --docker-compose-images-tags=yandex/clickhouse-integration-helper:latest  -t test_merge_tree_s3_failover/test.py::test_move_failover test_merge_tree_s3_failover/test.py::test_write_failover test_merge_tree_s3_restore/test.py::test_full_restore test_merge_tree_s3_restore/test.py::test_migrate_to_restorable_schema test_merge_tree_s3_restore/test.py::test_restore_another_bucket_path test_merge_tree_s3_restore/test.py::test_restore_different_revisions test_merge_tree_s3_restore/test.py::test_restore_mutations test_merge_tree_s3_restore/test.py::test_restore_to_detached test_merge_tree_s3_with_cache/test.py::test_read_after_cache_is_wiped test_merge_tree_s3_with_cache/test.py::test_write_is_cached test_multiple_disks/test.py::test_alter_double_move_partition test_multiple_disks/test.py::test_alter_move test_multiple_disks/test.py::test_alter_move_half_of_partition test_multiple_disks/test.py::test_alter_policy test_multiple_disks/test.py::test_background_move test_multiple_disks/test.py::test_concurrent_alter_modify test_multiple_disks/test.py::test_concurrent_alter_move test_multiple_disks/test.py::test_concurrent_alter_move_and_drop test_multiple_disks/test.py::test_detach_attach test_multiple_disks/test.py::test_download_appropriate_disk test_multiple_disks/test.py::test_freeze test_multiple_disks/test.py::test_jbod_overflow test_multiple_disks/test.py::test_kill_while_insert test_multiple_disks/test.py::test_max_data_part_size test_multiple_disks/test.py::test_move_across_policies_does_not_work test_multiple_disks/test.py::test_move_while_merge test_multiple_disks/test.py::test_mutate_to_another_disk test_multiple_disks/test.py::test_no_merges_in_configuration_allow_from_query_with_reload test_multiple_disks/test.py::test_no_merges_in_configuration_allow_from_query_without_reload test_multiple_disks/test.py::test_no_warning_about_zero_max_data_part_size test_multiple_disks/test.py::test_query_parser test_multiple_disks/test.py::test_rename test_multiple_disks/test.py::test_round_robin test_multiple_disks/test.py::test_simple_merge_tree_merges_are_disabled test_multiple_disks/test.py::test_simple_replication_and_moves test_multiple_disks/test.py::test_start_stop_moves test_multiple_disks/test.py::test_system_tables test_multiple_disks/test.py::test_yes_merges_in_configuration_disallow_from_query_with_reload test_multiple_disks/test.py::test_yes_merges_in_configuration_disallow_from_query_without_reload test_mutations_hardlinks/test.py::test_delete_and_drop_mutation test_mutations_hardlinks/test.py::test_drop_mutation test_mutations_hardlinks/test.py::test_modify_mutation test_mutations_hardlinks/test.py::test_update_mutation test_mutations_in_partitions_of_merge_tree/test.py::test_trivial_alter_in_partition_merge_tree_with_where test_mutations_in_partitions_of_merge_tree/test.py::test_trivial_alter_in_partition_merge_tree_without_where test_mutations_in_partitions_of_merge_tree/test.py::test_trivial_alter_in_partition_replicated_merge_tree test_mutations_with_merge_tree/test.py::test_mutations_with_merge_background_task test_mutations_with_merge_tree/test.py::test_mutations_with_truncate_table test_mysql_database_engine/test.py::test_bad_arguments_for_mysql_database_engine test_mysql_database_engine/test.py::test_clickhouse_ddl_for_mysql_database test_mysql_database_engine/test.py::test_clickhouse_dml_for_mysql_database test_mysql_database_engine/test.py::test_clickhouse_join_for_mysql_database test_mysql_database_engine/test.py::test_clickhouse_mysql_no_connection test_mysql_database_engine/test.py::test_column_comments_for_mysql_database_engine test_mysql_database_engine/test.py::test_data_types_support_level_for_mysql_database_engine test_mysql_database_engine/test.py::test_mysql_ddl_for_mysql_database test_mysql_database_engine/test.py::test_mysql_types  --parallel 5  '-rfEp --run-id=0 --color=no --durations=0 ' | tee /place/sandbox-data/ramdrive/1057923636/ClickHouse/tests/integration/parallel6_0.nfo
2021-08-27 15:07:53,013 Run parallel6 group successfully
2021-08-27 15:09:09,250 Seems like all tests passed but some of them are skipped or deselected. Ignoring them and finishing group.
2021-08-27 15:09:09,251 Tests from group parallel6 stats, ERROR count 0
2021-08-27 15:09:09,251 Totally have 0 with status ERROR
2021-08-27 15:09:09,251 Tests from group parallel6 stats, PASSED count 0
2021-08-27 15:09:09,251 Totally have 1314 with status PASSED
2021-08-27 15:09:09,251 Tests from group parallel6 stats, FAILED count 0
2021-08-27 15:09:09,251 Totally have 0 with status FAILED
2021-08-27 15:09:09,251 Tests from group parallel6 stats, SKIPPED count 0
2021-08-27 15:09:09,251 Totally have 0 with status SKIPPED
2021-08-27 15:09:09,251 Tests from group parallel6 stats, FLAKY count 0
2021-08-27 15:09:09,251 Totally have 1 with status FLAKY
2021-08-27 15:09:09,252 Totally finished tests 1315/1534
2021-08-27 15:09:09,252 Running test group test_system_replicated_fetches/test.py countaining 1 tests
2021-08-27 15:09:09,252 Can run with custom docker image version latest
2021-08-27 15:09:09,253 Running test group test_system_replicated_fetches/test.py for the 0 retry
2021-08-27 15:09:09,255 Dump iptables after run b'Chain INPUT (policy ACCEPT)\ntarget     prot opt source               destination         \n\nChain FORWARD (policy ACCEPT)\ntarget     prot opt source               destination         \nDOCKER-USER  all  --  anywhere             anywhere            \nDOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere            \nACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED\nDOCKER     all  --  anywhere             anywhere            \nACCEPT     all  --  anywhere             anywhere            \nACCEPT     all  --  anywhere             anywhere            \nACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED\nDOCKER     all  --  anywhere             anywhere            \nACCEPT     all  --  anywhere             anywhere            \nACCEPT     all  --  anywhere             anywhere            \nACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED\nDOCKER     all  --  anywhere             anywhere            \nACCEPT     all  --  anywhere             anywhere            \nACCEPT     all  --  anywhere             anywhere            \nACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED\nDOCKER     all  --  anywhere             anywhere            \nACCEPT     all  --  anywhere             anywhere            \nACCEPT     all  --  anywhere             anywhere            \n\nChain OUTPUT (policy ACCEPT)\ntarget     prot opt source               destination         \n\nChain DOCKER (4 references)\ntarget     prot opt source               destination         \n\nChain DOCKER-ISOLATION-STAGE-1 (1 references)\ntarget     prot opt source               destination         \nDOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            \nDOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            \nRETURN     all  --  anywhere             anywhere            \n\nChain DOCKER-ISOLATION-STAGE-2 (2 references)\ntarget     prot opt source               destination         \nDROP       all  --  anywhere             anywhere            \nDROP       all  --  anywhere             anywhere            \nRETURN     all  --  anywhere             anywhere            \n\nChain DOCKER-USER (1 references)\ntarget     prot opt source               destination         \nRETURN     all  --  anywhere             anywhere            \n'
2021-08-27 15:09:09,256 Killing all alive docker containers
Killed
2021-08-27 15:19:09,312 docker kill excepted: Command 'timeout -s 9 10m docker kill $(docker ps -q)' returned non-zero exit status 137
2021-08-27 15:19:09,312 Removing all docker containers
Killed
2021-08-27 15:29:09,368 docker rm excepted: Command 'timeout -s 9 10m docker rm $(docker ps -a -q) --force' returned non-zero exit status 137
2021-08-27 15:29:09,368 Stopping docker daemon
2021-08-27 15:29:24,476 Restarting docker 0

@tavplubix
Copy link
Copy Markdown
Member

From another task (which is also going to fail with timeout due to hung docker):

[gw2] [ 65%] FAILED test_mysql_database_engine/test.py::test_clickhouse_mysql_no_connection 
test_mysql_database_engine/test.py::test_column_comments_for_mysql_database_engine Killed

robot-clickhouse pushed a commit that referenced this pull request Aug 30, 2021
robot-clickhouse pushed a commit that referenced this pull request Aug 30, 2021
vitlibar pushed a commit that referenced this pull request Aug 31, 2021
Backport #28197 to 21.9: Fix rare case when watch response received before request response
vitlibar pushed a commit that referenced this pull request Aug 31, 2021
Backport #28197 to 21.8: Fix rare case when watch response received before request response
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jepsen-test Need to test this PR with jepsen tests pr-bugfix Pull request with bugfix, not backported by default

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants