Skip to content

Maybe fix RabbitMQ#70336

Merged
kssenii merged 2 commits intoClickHouse:masterfrom
filimonov:maybe_fix_rabbitmq
Oct 4, 2024
Merged

Maybe fix RabbitMQ#70336
kssenii merged 2 commits intoClickHouse:masterfrom
filimonov:maybe_fix_rabbitmq

Conversation

@filimonov
Copy link
Copy Markdown
Contributor

@filimonov filimonov commented Oct 3, 2024 •
edited
Loading

Changelog category:

  • CI Fix or Improvement (changelog entry is not required)

Changelog entry:

Attempt to fix flaky RabbitMQ tests. Maybe closes #45160.

Documentation entry for user-facing changes:

It seems to just time out currently – see: #68694 (comment).

Changes:

  • Updated to the most recent RabbitMQ version (also newer Erlang, which may improve speed).
  • Disabled all plugins except the one (rabbitmq_consistent_hash_exchange) we really need (should improve startup speed).
  • Disabled all feature flags except for a few mandatory ones (should improve startup speed).
  • Introduced a memory limit for RabbitMQ (just in case).
  • Simplified/refactored the Python wiring for RabbitMQ readiness checks.
  • Increased the timeout from 60 to 90 seconds (in a few failures i checked, the actual startup time was about 65-70 seconds, which are outliers as it usually doesn't fail).
for the record - local run 
$ docker run --rm --name clickhouse_integration_tests_if53wt --privileged     --dns-search='.'  --volume=$(pwd)/docker/test/integration/runner/misc/rabbitmq:/misc/rabbitmq  --volume=$(pwd)/build/programs/clickhouse-odbc-bridge:/clickhouse-odbc-bridge     --volume=$(pwd)/build/programs/clickhouse:/clickhouse     --volume=$(pwd)/build/programs/clickhouse-library-bridge:/clickhouse-library-bridge     --volume=$(pwd)/programs/server:/clickhouse-config     --volume=$(pwd)/tests/integration:/ClickHouse/tests/integration     --volume=$(pwd)/utils/backupview:/ClickHouse/utils/backupview     --volume=$(pwd)/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2     --volume=/run:/run/host:ro     --volume=clickhouse_integration_tests_volume:/var/lib/docker     -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d     -e DOCKER_HELPER_TAG=634724fd9222     -e DOCKER_BASE_TAG=2d0a5b22f529     -e DOCKER_KERBERIZED_HADOOP_TAG=5105c1c1cfaa     -e DOCKER_KERBEROS_KDC_TAG=310561e6eb75     -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=a63320938f8d     -e DOCKER_MYSQL_JAVA_CLIENT_TAG=807c80c2c10c     -e DOCKER_MYSQL_JS_CLIENT_TAG=d7dd5dbc48a5     -e DOCKER_MYSQL_PHP_CLIENT_TAG=99abb5178265     -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519     -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=f81480176616     -e DOCKER_PYTHON_BOTTLE_TAG=3b7722e65ba1     -e DOCKER_CLIENT_TIMEOUT=300     -e COMPOSE_HTTP_TIMEOUT=600     -e PYTHONUNBUFFERED=1     -e PYTEST_ADDOPTS="--dist=loadfile -n 5 -rfEps --run-id=0 --color=no --durations=0 test_storage_rabbitmq/test.py"     clickhouse/integration-tests-runner:e873dae8d34a
Start tests
============================= test session starts ==============================
platform linux -- Python 3.10.12, pytest-7.4.4, pluggy-1.5.0
rootdir: /ClickHouse/tests/integration
configfile: pytest.ini
plugins: xdist-3.5.0, reportlog-0.4.0, timeout-2.2.0, random-0.2, repeat-0.9.3, order-1.0.0
timeout: 900.0s
timeout method: signal
timeout func_only: False
created: 5/5 workers
5 workers [55 items]

scheduling tests via LoadFileScheduling

test_storage_rabbitmq/test.py::test_rabbitmq_select[0] 
[gw0] [  1%] PASSED test_storage_rabbitmq/test.py::test_rabbitmq_select[0] 
test_storage_rabbitmq/test.py::test_rabbitmq_select[1] 
[gw0] [  3%] PASSED test_storage_rabbitmq/test.py::test_rabbitmq_select[1] 
test_storage_rabbitmq/test.py::test_rabbitmq_select_empty 
[gw0] [  5%] PASSED test_storage_rabbitmq/test.py::test_rabbitmq_select_empty 
test_storage_rabbitmq/test.py::test_rabbitmq_json_without_delimiter 
[gw0] [  7%] PASSED test_storage_rabbitmq/test.py::test_rabbitmq_json_without_delimiter 
test_storage_rabbitmq/test.py::test_rabbitmq_csv_with_delimiter 
[gw0] [  9%] PASSED test_storage_rabbitmq/test.py::test_rabbitmq_csv_with_delimiter 
test_storage_rabbitmq/test.py::test_rabbitmq_tsv_with_delimiter 
[gw0] [ 10%] PASSED test_storage_rabbitmq/test.py::test_rabbitmq_tsv_with_delimiter 
test_storage_rabbitmq/test.py::test_rabbitmq_macros 
[gw0] [ 12%] PASSED test_storage_rabbitmq/test.py::test_rabbitmq_macros 
test_storage_rabbitmq/test.py::test_rabbitmq_materialized_view 
[gw0] [ 14%] PASSED test_storage_rabbitmq/test.py::test_rabbitmq_materialized_view 
test_storage_rabbitmq/test.py::test_rabbitmq_materialized_view_with_subquery 
[gw0] [ 16%] PASSED test_storage_rabbitmq/test.py::test_rabbitmq_materialized_view_with_subquery 
test_storage_rabbitmq/test.py::test_rabbitmq_many_materialized_views 
[gw0] [ 18%] PASSED test_storage_rabbitmq/test.py::test_rabbitmq_many_materialized_views 
test_storage_rabbitmq/test.py::test_rabbitmq_big_message 
[gw0] [ 20%] PASSED test_storage_rabbitmq/test.py::test_rabbitmq_big_message 
test_storage_rabbitmq/test.py::test_rabbitmq_sharding_between_queues_publish 
[gw0] [ 21%] PASSED test_storage_rabbitmq/test.py::test_rabbitmq_sharding_between_queues_publish 
test_storage_rabbitmq/test.py::test_rabbitmq_mv_combo 
[gw0] [ 23%] PASSED test_storage_rabbitmq/test.py::test_rabbitmq_mv_combo 
test_storage_rabbitmq/test.py::test_rabbitmq_insert 
[gw0] [ 25%] PASSED test_storage_rabbitmq/test.py::test_rabbitmq_insert 
test_storage_rabbitmq/test.py::test_rabbitmq_insert_headers_exchange 
[gw0] [ 27%] PASSED test_storage_rabbitmq/test.py::test_rabbitmq_insert_headers_exchange 
test_storage_rabbitmq/test.py::test_rabbitmq_many_inserts 
[gw0] [ 29%] PASSED test_storage_rabbitmq/test.py::test_rabbitmq_many_inserts 
test_storage_rabbitmq/test.py::test_rabbitmq_overloaded_insert 
[gw0] [ 30%] PASSED test_storage_rabbitmq/test.py::test_rabbitmq_overloaded_insert 
test_storage_rabbitmq/test.py::test_rabbitmq_direct_exchange 
[gw0] [ 32%] PASSED test_storage_rabbitmq/test.py::test_rabbitmq_direct_exchange 
test_storage_rabbitmq/test.py::test_rabbitmq_fanout_exchange 
[gw0] [ 34%] PASSED test_storage_rabbitmq/test.py::test_rabbitmq_fanout_exchange 
test_storage_rabbitmq/test.py::test_rabbitmq_topic_exchange 
[gw0] [ 36%] PASSED test_storage_rabbitmq/test.py::test_rabbitmq_topic_exchange 
test_storage_rabbitmq/test.py::test_rabbitmq_hash_exchange 
[gw0] [ 38%] PASSED test_storage_rabbitmq/test.py::test_rabbitmq_hash_exchange 
test_storage_rabbitmq/test.py::test_rabbitmq_multiple_bindings 
[gw0] [ 40%] PASSED test_storage_rabbitmq/test.py::test_rabbitmq_multiple_bindings 
test_storage_rabbitmq/test.py::test_rabbitmq_headers_exchange 
[gw0] [ 41%] PASSED test_storage_rabbitmq/test.py::test_rabbitmq_headers_exchange 
test_storage_rabbitmq/test.py::test_rabbitmq_virtual_columns 
[gw0] [ 43%] PASSED test_storage_rabbitmq/test.py::test_rabbitmq_virtual_columns 
test_storage_rabbitmq/test.py::test_rabbitmq_virtual_columns_with_materialized_view 
[gw0] [ 45%] PASSED test_storage_rabbitmq/test.py::test_rabbitmq_virtual_columns_with_materialized_view 
test_storage_rabbitmq/test.py::test_rabbitmq_many_consumers_to_each_queue 
[gw0] [ 47%] PASSED test_storage_rabbitmq/test.py::test_rabbitmq_many_consumers_to_each_queue 
test_storage_rabbitmq/test.py::test_rabbitmq_restore_failed_connection_without_losses_1 
[gw0] [ 49%] PASSED test_storage_rabbitmq/test.py::test_rabbitmq_restore_failed_connection_without_losses_1 
test_storage_rabbitmq/test.py::test_rabbitmq_restore_failed_connection_without_losses_2 
[gw0] [ 50%] PASSED test_storage_rabbitmq/test.py::test_rabbitmq_restore_failed_connection_without_losses_2 
test_storage_rabbitmq/test.py::test_rabbitmq_commit_on_block_write 
[gw0] [ 52%] PASSED test_storage_rabbitmq/test.py::test_rabbitmq_commit_on_block_write 
test_storage_rabbitmq/test.py::test_rabbitmq_no_connection_at_startup_1 
[gw0] [ 54%] PASSED test_storage_rabbitmq/test.py::test_rabbitmq_no_connection_at_startup_1 
test_storage_rabbitmq/test.py::test_rabbitmq_no_connection_at_startup_2 
[gw0] [ 56%] PASSED test_storage_rabbitmq/test.py::test_rabbitmq_no_connection_at_startup_2 
test_storage_rabbitmq/test.py::test_rabbitmq_format_factory_settings 
[gw0] [ 58%] PASSED test_storage_rabbitmq/test.py::test_rabbitmq_format_factory_settings 
test_storage_rabbitmq/test.py::test_rabbitmq_vhost 
[gw0] [ 60%] PASSED test_storage_rabbitmq/test.py::test_rabbitmq_vhost 
test_storage_rabbitmq/test.py::test_rabbitmq_drop_table_properly 
[gw0] [ 61%] PASSED test_storage_rabbitmq/test.py::test_rabbitmq_drop_table_properly 
test_storage_rabbitmq/test.py::test_rabbitmq_queue_settings 
[gw0] [ 63%] PASSED test_storage_rabbitmq/test.py::test_rabbitmq_queue_settings 
test_storage_rabbitmq/test.py::test_rabbitmq_queue_consume 
[gw0] [ 65%] PASSED test_storage_rabbitmq/test.py::test_rabbitmq_queue_consume 
test_storage_rabbitmq/test.py::test_rabbitmq_produce_consume_avro 
[gw0] [ 67%] PASSED test_storage_rabbitmq/test.py::test_rabbitmq_produce_consume_avro 
test_storage_rabbitmq/test.py::test_rabbitmq_bad_args 
[gw0] [ 69%] PASSED test_storage_rabbitmq/test.py::test_rabbitmq_bad_args 
test_storage_rabbitmq/test.py::test_rabbitmq_issue_30691 
[gw0] [ 70%] PASSED test_storage_rabbitmq/test.py::test_rabbitmq_issue_30691 
test_storage_rabbitmq/test.py::test_rabbitmq_drop_mv 
[gw0] [ 72%] PASSED test_storage_rabbitmq/test.py::test_rabbitmq_drop_mv 
test_storage_rabbitmq/test.py::test_rabbitmq_random_detach 
[gw0] [ 74%] PASSED test_storage_rabbitmq/test.py::test_rabbitmq_random_detach 
test_storage_rabbitmq/test.py::test_rabbitmq_predefined_configuration 
[gw0] [ 76%] PASSED test_storage_rabbitmq/test.py::test_rabbitmq_predefined_configuration 
test_storage_rabbitmq/test.py::test_rabbitmq_msgpack 
[gw0] [ 78%] PASSED test_storage_rabbitmq/test.py::test_rabbitmq_msgpack 
test_storage_rabbitmq/test.py::test_rabbitmq_address 
[gw0] [ 80%] PASSED test_storage_rabbitmq/test.py::test_rabbitmq_address 
test_storage_rabbitmq/test.py::test_format_with_prefix_and_suffix 
[gw0] [ 81%] PASSED test_storage_rabbitmq/test.py::test_format_with_prefix_and_suffix 
test_storage_rabbitmq/test.py::test_max_rows_per_message 
[gw0] [ 83%] PASSED test_storage_rabbitmq/test.py::test_max_rows_per_message 
test_storage_rabbitmq/test.py::test_row_based_formats 
[gw0] [ 85%] PASSED test_storage_rabbitmq/test.py::test_row_based_formats 
test_storage_rabbitmq/test.py::test_block_based_formats_1 
[gw0] [ 87%] PASSED test_storage_rabbitmq/test.py::test_block_based_formats_1 
test_storage_rabbitmq/test.py::test_block_based_formats_2 
[gw0] [ 89%] PASSED test_storage_rabbitmq/test.py::test_block_based_formats_2 
test_storage_rabbitmq/test.py::test_rabbitmq_flush_by_block_size 
[gw0] [ 90%] PASSED test_storage_rabbitmq/test.py::test_rabbitmq_flush_by_block_size 
test_storage_rabbitmq/test.py::test_rabbitmq_flush_by_time 
[gw0] [ 92%] PASSED test_storage_rabbitmq/test.py::test_rabbitmq_flush_by_time 
test_storage_rabbitmq/test.py::test_rabbitmq_handle_error_mode_stream 
[gw0] [ 94%] PASSED test_storage_rabbitmq/test.py::test_rabbitmq_handle_error_mode_stream 
test_storage_rabbitmq/test.py::test_attach_broken_table 
[gw0] [ 96%] PASSED test_storage_rabbitmq/test.py::test_attach_broken_table 
test_storage_rabbitmq/test.py::test_rabbitmq_nack_failed_insert 
[gw0] [ 98%] PASSED test_storage_rabbitmq/test.py::test_rabbitmq_nack_failed_insert 
test_storage_rabbitmq/test.py::test_rabbitmq_reject_broken_messages 
[gw0] [100%] PASSED test_storage_rabbitmq/test.py::test_rabbitmq_reject_broken_messages 

============================== slowest durations ===============================
92.28s call     test_storage_rabbitmq/test.py::test_rabbitmq_mv_combo
78.71s call     test_storage_rabbitmq/test.py::test_rabbitmq_random_detach
64.41s call     test_storage_rabbitmq/test.py::test_rabbitmq_restore_failed_connection_without_losses_2
60.40s call     test_storage_rabbitmq/test.py::test_row_based_formats
54.99s call     test_storage_rabbitmq/test.py::test_rabbitmq_handle_error_mode_stream
40.78s call     test_storage_rabbitmq/test.py::test_rabbitmq_sharding_between_queues_publish
31.50s call     test_storage_rabbitmq/test.py::test_rabbitmq_drop_table_properly
25.07s call     test_storage_rabbitmq/test.py::test_rabbitmq_restore_failed_connection_without_losses_1
22.65s call     test_storage_rabbitmq/test.py::test_rabbitmq_hash_exchange
19.06s call     test_storage_rabbitmq/test.py::test_rabbitmq_flush_by_time
17.76s call     test_storage_rabbitmq/test.py::test_rabbitmq_many_consumers_to_each_queue
17.44s call     test_storage_rabbitmq/test.py::test_rabbitmq_overloaded_insert
17.21s call     test_storage_rabbitmq/test.py::test_block_based_formats_2
17.19s call     test_storage_rabbitmq/test.py::test_rabbitmq_topic_exchange
15.37s call     test_storage_rabbitmq/test.py::test_rabbitmq_drop_mv
13.56s call     test_storage_rabbitmq/test.py::test_rabbitmq_many_inserts
12.16s call     test_storage_rabbitmq/test.py::test_rabbitmq_multiple_bindings
11.44s call     test_storage_rabbitmq/test.py::test_rabbitmq_no_connection_at_startup_2
10.84s call     test_storage_rabbitmq/test.py::test_rabbitmq_fanout_exchange
10.69s call     test_storage_rabbitmq/test.py::test_rabbitmq_direct_exchange
9.93s call     test_storage_rabbitmq/test.py::test_rabbitmq_reject_broken_messages
8.38s call     test_storage_rabbitmq/test.py::test_attach_broken_table
7.95s setup    test_storage_rabbitmq/test.py::test_rabbitmq_select[0]
7.60s call     test_storage_rabbitmq/test.py::test_rabbitmq_queue_consume
7.37s call     test_storage_rabbitmq/test.py::test_rabbitmq_nack_failed_insert
7.08s call     test_storage_rabbitmq/test.py::test_rabbitmq_produce_consume_avro
7.02s call     test_storage_rabbitmq/test.py::test_rabbitmq_headers_exchange
6.99s call     test_storage_rabbitmq/test.py::test_rabbitmq_select[1]
6.87s call     test_storage_rabbitmq/test.py::test_rabbitmq_select[0]
6.73s call     test_storage_rabbitmq/test.py::test_rabbitmq_predefined_configuration
6.49s call     test_storage_rabbitmq/test.py::test_rabbitmq_queue_settings
6.40s call     test_storage_rabbitmq/test.py::test_rabbitmq_csv_with_delimiter
5.30s call     test_storage_rabbitmq/test.py::test_rabbitmq_big_message
5.09s call     test_storage_rabbitmq/test.py::test_rabbitmq_address
5.04s call     test_storage_rabbitmq/test.py::test_rabbitmq_msgpack
4.94s teardown test_storage_rabbitmq/test.py::test_rabbitmq_reject_broken_messages
4.27s call     test_storage_rabbitmq/test.py::test_rabbitmq_commit_on_block_write
4.23s call     test_storage_rabbitmq/test.py::test_rabbitmq_no_connection_at_startup_1
3.74s call     test_storage_rabbitmq/test.py::test_max_rows_per_message
3.65s call     test_storage_rabbitmq/test.py::test_rabbitmq_virtual_columns
3.60s call     test_storage_rabbitmq/test.py::test_rabbitmq_virtual_columns_with_materialized_view
3.26s call     test_storage_rabbitmq/test.py::test_rabbitmq_many_materialized_views
2.85s call     test_storage_rabbitmq/test.py::test_rabbitmq_format_factory_settings
2.69s call     test_storage_rabbitmq/test.py::test_rabbitmq_materialized_view
2.66s call     test_storage_rabbitmq/test.py::test_rabbitmq_insert_headers_exchange
2.61s call     test_storage_rabbitmq/test.py::test_rabbitmq_insert
2.56s call     test_storage_rabbitmq/test.py::test_block_based_formats_1
2.55s call     test_storage_rabbitmq/test.py::test_format_with_prefix_and_suffix
2.50s call     test_storage_rabbitmq/test.py::test_rabbitmq_json_without_delimiter
2.48s call     test_storage_rabbitmq/test.py::test_rabbitmq_macros
2.45s call     test_storage_rabbitmq/test.py::test_rabbitmq_materialized_view_with_subquery
2.42s call     test_storage_rabbitmq/test.py::test_rabbitmq_tsv_with_delimiter
2.14s call     test_storage_rabbitmq/test.py::test_rabbitmq_flush_by_block_size
1.46s call     test_storage_rabbitmq/test.py::test_rabbitmq_issue_30691
1.39s call     test_storage_rabbitmq/test.py::test_rabbitmq_vhost
1.39s call     test_storage_rabbitmq/test.py::test_rabbitmq_select_empty
1.33s call     test_storage_rabbitmq/test.py::test_rabbitmq_bad_args
0.58s teardown test_storage_rabbitmq/test.py::test_row_based_formats
0.29s teardown test_storage_rabbitmq/test.py::test_rabbitmq_big_message
0.28s teardown test_storage_rabbitmq/test.py::test_rabbitmq_restore_failed_connection_without_losses_1
0.23s teardown test_storage_rabbitmq/test.py::test_rabbitmq_random_detach
0.18s teardown test_storage_rabbitmq/test.py::test_block_based_formats_2
0.18s teardown test_storage_rabbitmq/test.py::test_rabbitmq_select_empty
0.18s teardown test_storage_rabbitmq/test.py::test_max_rows_per_message
0.18s teardown test_storage_rabbitmq/test.py::test_rabbitmq_restore_failed_connection_without_losses_2
0.18s teardown test_storage_rabbitmq/test.py::test_rabbitmq_predefined_configuration
0.13s teardown test_storage_rabbitmq/test.py::test_attach_broken_table
0.13s teardown test_storage_rabbitmq/test.py::test_rabbitmq_topic_exchange
0.13s teardown test_storage_rabbitmq/test.py::test_rabbitmq_flush_by_block_size
0.13s teardown test_storage_rabbitmq/test.py::test_rabbitmq_no_connection_at_startup_2
0.13s teardown test_storage_rabbitmq/test.py::test_rabbitmq_mv_combo
0.13s teardown test_storage_rabbitmq/test.py::test_rabbitmq_macros
0.13s teardown test_storage_rabbitmq/test.py::test_rabbitmq_msgpack
0.13s teardown test_storage_rabbitmq/test.py::test_rabbitmq_bad_args
0.13s teardown test_storage_rabbitmq/test.py::test_rabbitmq_commit_on_block_write
0.13s teardown test_storage_rabbitmq/test.py::test_rabbitmq_tsv_with_delimiter
0.13s teardown test_storage_rabbitmq/test.py::test_rabbitmq_select[1]
0.13s teardown test_storage_rabbitmq/test.py::test_rabbitmq_many_consumers_to_each_queue
0.13s teardown test_storage_rabbitmq/test.py::test_rabbitmq_format_factory_settings
0.13s teardown test_storage_rabbitmq/test.py::test_rabbitmq_produce_consume_avro
0.13s teardown test_storage_rabbitmq/test.py::test_rabbitmq_select[0]
0.13s teardown test_storage_rabbitmq/test.py::test_rabbitmq_drop_table_properly
0.13s teardown test_storage_rabbitmq/test.py::test_rabbitmq_drop_mv
0.13s teardown test_storage_rabbitmq/test.py::test_rabbitmq_no_connection_at_startup_1
0.13s teardown test_storage_rabbitmq/test.py::test_rabbitmq_many_materialized_views
0.13s teardown test_storage_rabbitmq/test.py::test_rabbitmq_insert_headers_exchange
0.13s teardown test_storage_rabbitmq/test.py::test_rabbitmq_virtual_columns
0.13s teardown test_storage_rabbitmq/test.py::test_rabbitmq_address
0.13s teardown test_storage_rabbitmq/test.py::test_rabbitmq_issue_30691
0.13s teardown test_storage_rabbitmq/test.py::test_rabbitmq_direct_exchange
0.13s teardown test_storage_rabbitmq/test.py::test_rabbitmq_many_inserts
0.13s teardown test_storage_rabbitmq/test.py::test_rabbitmq_materialized_view
0.13s teardown test_storage_rabbitmq/test.py::test_rabbitmq_hash_exchange
0.13s teardown test_storage_rabbitmq/test.py::test_rabbitmq_overloaded_insert
0.13s teardown test_storage_rabbitmq/test.py::test_rabbitmq_flush_by_time
0.13s teardown test_storage_rabbitmq/test.py::test_rabbitmq_headers_exchange
0.13s teardown test_storage_rabbitmq/test.py::test_rabbitmq_materialized_view_with_subquery
0.13s teardown test_storage_rabbitmq/test.py::test_rabbitmq_json_without_delimiter
0.13s teardown test_storage_rabbitmq/test.py::test_rabbitmq_handle_error_mode_stream
0.13s teardown test_storage_rabbitmq/test.py::test_rabbitmq_csv_with_delimiter
0.13s teardown test_storage_rabbitmq/test.py::test_rabbitmq_queue_consume
0.13s teardown test_storage_rabbitmq/test.py::test_rabbitmq_vhost
0.13s teardown test_storage_rabbitmq/test.py::test_rabbitmq_multiple_bindings
0.13s teardown test_storage_rabbitmq/test.py::test_rabbitmq_sharding_between_queues_publish
0.13s teardown test_storage_rabbitmq/test.py::test_rabbitmq_virtual_columns_with_materialized_view
0.13s teardown test_storage_rabbitmq/test.py::test_block_based_formats_1
0.13s teardown test_storage_rabbitmq/test.py::test_format_with_prefix_and_suffix
0.13s teardown test_storage_rabbitmq/test.py::test_rabbitmq_fanout_exchange
0.13s teardown test_storage_rabbitmq/test.py::test_rabbitmq_nack_failed_insert
0.13s teardown test_storage_rabbitmq/test.py::test_rabbitmq_queue_settings
0.13s teardown test_storage_rabbitmq/test.py::test_rabbitmq_insert

(54 durations < 0.005s hidden.  Use -vv to show these durations.)
=========================== short test summary info ============================
PASSED test_storage_rabbitmq/test.py::test_rabbitmq_select[0]
PASSED test_storage_rabbitmq/test.py::test_rabbitmq_select[1]
PASSED test_storage_rabbitmq/test.py::test_rabbitmq_select_empty
PASSED test_storage_rabbitmq/test.py::test_rabbitmq_json_without_delimiter
PASSED test_storage_rabbitmq/test.py::test_rabbitmq_csv_with_delimiter
PASSED test_storage_rabbitmq/test.py::test_rabbitmq_tsv_with_delimiter
PASSED test_storage_rabbitmq/test.py::test_rabbitmq_macros
PASSED test_storage_rabbitmq/test.py::test_rabbitmq_materialized_view
PASSED test_storage_rabbitmq/test.py::test_rabbitmq_materialized_view_with_subquery
PASSED test_storage_rabbitmq/test.py::test_rabbitmq_many_materialized_views
PASSED test_storage_rabbitmq/test.py::test_rabbitmq_big_message
PASSED test_storage_rabbitmq/test.py::test_rabbitmq_sharding_between_queues_publish
PASSED test_storage_rabbitmq/test.py::test_rabbitmq_mv_combo
PASSED test_storage_rabbitmq/test.py::test_rabbitmq_insert
PASSED test_storage_rabbitmq/test.py::test_rabbitmq_insert_headers_exchange
PASSED test_storage_rabbitmq/test.py::test_rabbitmq_many_inserts
PASSED test_storage_rabbitmq/test.py::test_rabbitmq_overloaded_insert
PASSED test_storage_rabbitmq/test.py::test_rabbitmq_direct_exchange
PASSED test_storage_rabbitmq/test.py::test_rabbitmq_fanout_exchange
PASSED test_storage_rabbitmq/test.py::test_rabbitmq_topic_exchange
PASSED test_storage_rabbitmq/test.py::test_rabbitmq_hash_exchange
PASSED test_storage_rabbitmq/test.py::test_rabbitmq_multiple_bindings
PASSED test_storage_rabbitmq/test.py::test_rabbitmq_headers_exchange
PASSED test_storage_rabbitmq/test.py::test_rabbitmq_virtual_columns
PASSED test_storage_rabbitmq/test.py::test_rabbitmq_virtual_columns_with_materialized_view
PASSED test_storage_rabbitmq/test.py::test_rabbitmq_many_consumers_to_each_queue
PASSED test_storage_rabbitmq/test.py::test_rabbitmq_restore_failed_connection_without_losses_1
PASSED test_storage_rabbitmq/test.py::test_rabbitmq_restore_failed_connection_without_losses_2
PASSED test_storage_rabbitmq/test.py::test_rabbitmq_commit_on_block_write
PASSED test_storage_rabbitmq/test.py::test_rabbitmq_no_connection_at_startup_1
PASSED test_storage_rabbitmq/test.py::test_rabbitmq_no_connection_at_startup_2
PASSED test_storage_rabbitmq/test.py::test_rabbitmq_format_factory_settings
PASSED test_storage_rabbitmq/test.py::test_rabbitmq_vhost
PASSED test_storage_rabbitmq/test.py::test_rabbitmq_drop_table_properly
PASSED test_storage_rabbitmq/test.py::test_rabbitmq_queue_settings
PASSED test_storage_rabbitmq/test.py::test_rabbitmq_queue_consume
PASSED test_storage_rabbitmq/test.py::test_rabbitmq_produce_consume_avro
PASSED test_storage_rabbitmq/test.py::test_rabbitmq_bad_args
PASSED test_storage_rabbitmq/test.py::test_rabbitmq_issue_30691
PASSED test_storage_rabbitmq/test.py::test_rabbitmq_drop_mv
PASSED test_storage_rabbitmq/test.py::test_rabbitmq_random_detach
PASSED test_storage_rabbitmq/test.py::test_rabbitmq_predefined_configuration
PASSED test_storage_rabbitmq/test.py::test_rabbitmq_msgpack
PASSED test_storage_rabbitmq/test.py::test_rabbitmq_address
PASSED test_storage_rabbitmq/test.py::test_format_with_prefix_and_suffix
PASSED test_storage_rabbitmq/test.py::test_max_rows_per_message
PASSED test_storage_rabbitmq/test.py::test_row_based_formats
PASSED test_storage_rabbitmq/test.py::test_block_based_formats_1
PASSED test_storage_rabbitmq/test.py::test_block_based_formats_2
PASSED test_storage_rabbitmq/test.py::test_rabbitmq_flush_by_block_size
PASSED test_storage_rabbitmq/test.py::test_rabbitmq_flush_by_time
PASSED test_storage_rabbitmq/test.py::test_rabbitmq_handle_error_mode_stream
PASSED test_storage_rabbitmq/test.py::test_attach_broken_table
PASSED test_storage_rabbitmq/test.py::test_rabbitmq_nack_failed_insert
PASSED test_storage_rabbitmq/test.py::test_rabbitmq_reject_broken_messages
======================== 55 passed in 818.73s (0:13:38) ========================

@Algunenano Algunenano added the can be tested Allows running workflows for external contributors label Oct 3, 2024
@robot-ch-test-poll3
Copy link
Copy Markdown
Contributor

robot-ch-test-poll3 commented Oct 3, 2024 •
edited by robot-clickhouse-ci-1
Loading

This is an automated comment for commit fd5defd with description of existing statuses. It's updated for the latest CI running

✅ Click here to open a full report in a separate page

Successful checks
Check nameDescriptionStatus
BuildsThere's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS✅ success
Docs checkBuilds and tests the documentation✅ success
Fast testNormally this is the first check that is ran for a PR. It builds ClickHouse and runs most of stateless functional tests, omitting some. If it fails, further checks are not started until it is fixed. Look at the report to see which tests fail, then reproduce the failure locally as described here✅ success
Flaky testsChecks if new added or modified tests are flaky by running them repeatedly, in parallel, with more randomization. Functional tests are run 100 times with address sanitizer, and additional randomization of thread scheduling. Integration tests are run up to 10 times. If at least once a new test has failed, or was too long, this check will be red. We don't allow flaky tests, read the doc✅ success
Integration testsThe integration tests report. In parenthesis the package type is given, and in square brackets are the optional part/total tests✅ success
Style checkRuns a set of checks to keep the code style clean. If some of tests failed, see the related log from the report✅ success
Unit testsRuns the unit tests for different release types✅ success
Upgrade checkRuns stress tests on server version from last release and then tries to upgrade it to the version from the PR. It checks if the new server can successfully startup without any errors, crashes or sanitizer asserts✅ success

@clickhouse-ci clickhouse-ci bot added the manual approve Manual approve required to run CI label Oct 3, 2024
@kssenii kssenii self-assigned this Oct 3, 2024
@filimonov
Copy link
Copy Markdown
Contributor Author

filimonov commented Oct 3, 2024

Several recent failure samples:

  • 1 2024-09-27 19:01:45.444239+00:00 [info] <0.9.0> Time to start RabbitMQ: 70850225 us

  • 2 2024-10-02 10:45:22.153876+00:00 [info] <0.9.0> Time to start RabbitMQ: 67713508 us

  • 3 2024-10-02 16:10:12.332082+00:00 [info] <0.9.0> Time to start RabbitMQ: 69651090 us

All longer than 60 sec, so for those that PR should help.

But there is also one different, @kssenii - may be you can check it?

https://s3.amazonaws.com/clickhouse-test-reports/0/8236c9049d9810612a455ac7b67a44b81e5b308f/integration_tests__tsan__[2_6]//home/ubuntu/actions-runner/_work/_temp/test/output_dir/integration_run_parallel4_0.log

@filimonov
Copy link
Copy Markdown
Contributor Author

filimonov commented Oct 3, 2024 •
edited
Loading 

2024-10-03 20:55:41.107099+00:00 [debug] <0.254.0> Loading the following plugins: [rabbitmq_consistent_hash_exchange]
...
2024-10-03 20:55:41.148062+00:00 [info] <0.318.0> Feature flags: automatic enablement of feature flags limited to the following list: [feature_flags_v2,
2024-10-03 20:55:41.148062+00:00 [info] <0.318.0>                                                                                      message_containers]
...
2024-10-03 20:55:41.849184+00:00 [info] <0.9.0> Time to start RabbitMQ: 3740 ms

Seems to work.

@kssenii kssenii added this pull request to the merge queue Oct 4, 2024
Merged via the queue into ClickHouse:master with commit cbd1212 Oct 4, 2024
@robot-clickhouse robot-clickhouse added the pr-synced-to-cloud The PR is synced to the cloud repo label Oct 4, 2024
@kssenii
Copy link
Copy Markdown
Member

kssenii commented Oct 4, 2024 •
edited
Loading

In the next commit after merge commit of this PR there is https://s3.amazonaws.com/clickhouse-test-reports/0/0cc5940fba34346f59d9e45dce3cab4e3e0bb4be/integration_tests__tsan__[2_6].html, seems different than before, so it could be because of this PR.
I do not see the reason for failure because of absence of logs.

@filimonov
Copy link
Copy Markdown
Contributor Author

filimonov commented Oct 8, 2024 •
edited
Loading

Here
https://s3.amazonaws.com/clickhouse-test-reports/0/0cc5940fba34346f59d9e45dce3cab4e3e0bb4be/integration_tests__tsan__[2_6]/integration_run_parallel4_0.log
we see the following:

[gw0] [ 87%] PASSED test_storage_rabbitmq/test.py::test_rabbitmq_reject_broken_messages 
test_storage_rabbitmq/test.py::test_rabbitmq_restore_failed_connection_without_losses_1 
[gw0] [ 88%] FAILED test_storage_rabbitmq/test.py::test_rabbitmq_restore_failed_connection_without_losses_1 
[gw0] [ 88%] ERROR test_storage_rabbitmq/test.py::test_rabbitmq_restore_failed_connection_without_losses_1 
test_storage_rabbitmq/test.py::test_rabbitmq_restore_failed_connection_without_losses_2 
[gw0] [ 89%] FAILED test_storage_rabbitmq/test.py::test_rabbitmq_restore_failed_connection_without_losses_2 
[gw0] [ 89%] ERROR test_storage_rabbitmq/test.py::test_rabbitmq_restore_failed_connection_without_losses_2 
test_storage_rabbitmq/test.py::test_rabbitmq_select[0] 
[gw0] [ 90%] PASSED test_storage_rabbitmq/test.py::test_rabbitmq_select[0] 
[gw0] [ 90%] ERROR test_storage_rabbitmq/test.py::test_rabbitmq_select[0] 
test_storage_rabbitmq/test.py::test_rabbitmq_select[1] 
[gw0] [ 91%] SKIPPED test_storage_rabbitmq/test.py::test_rabbitmq_select[1] Killed

I.e. the first failed test is test_rabbitmq_restore_failed_connection_without_losses_1.

In rabbitmq logs (
https://s3.amazonaws.com/clickhouse-test-reports/0/0cc5940fba34346f59d9e45dce3cab4e3e0bb4be/integration_tests__tsan__[2_6]/integration_run_parallel4_0.tar.zst /test_storage_rabbitmq/_instances-0-gw0/rabbitmq/logs/rabbit.log ) we see that the rabbit was restarted

2024-10-04 12:07:16.687711+00:00 [notice] <0.254.0> Logging: configured log handlers are now ACTIVE
2024-10-04 12:07:18.549637+00:00 [info] <0.9.0> Time to start RabbitMQ: 4643 ms
...
2024-10-04 12:19:34.924328+00:00 [debug] <0.9544.0> Consistent hashing exchange: adding binding from exchange exchange 'producer_reconnect_test_consume' in vhost '/' to destination queue '1_test_consume' in vhost '/' with routing key '1'
2024-10-04 12:19:34.924609+00:00 [debug] <0.9544.0> Consistent hashing exchange: adding binding from exchange exchange 'producer_reconnect_test_consume' in vhost '/' to destination queue '1_test_consume' in vhost '/' with routing key '1'
...
2024-10-04 12:19:41.257623+00:00 [notice] <0.64.0> SIGTERM received - shutting down
...
2024-10-04 12:19:49.709558+00:00 [notice] <0.254.0> Logging: configured log handlers are now ACTIVE
...
2024-10-04 12:19:51.054984+00:00 [debug] <0.436.0> Consistent hashing exchange: adding binding from exchange exchange 'producer_reconnect_test_consume' in vhost '/' to destination queue '1_test_consume' in vhost '/' with routing key '1'
2024-10-04 12:19:51.055058+00:00 [debug] <0.436.0> Consistent hashing exchange: adding binding from exchange exchange 'producer_reconnect_test_consume' in vhost '/' to destination queue '1_test_consume' in vhost '/' with routing key '1'
2024-10-04 12:19:51.055132+00:00 [debug] <0.436.0> Consistent hashing exchange: NOT adding binding from exchange exchange 'producer_reconnect_test_consume' in vhost '/' to destination queue '1_test_consume' in vhost '/' with routing key '1' because this binding (possibly with a different routing key) already exists
...
2024-10-04 12:19:51.270299+00:00 [info] <0.9.0> Time to start RabbitMQ: 3750 ms

Later according to clickhouse logs the query was executed

2024.10.04 12:19:51.467897 [ 1021 ] {be3059f5-dd21-4e89-ab50-3f5eb45efed3} <Debug> executeQuery: (from 172.16.3.1:63640) SELECT count(DISTINCT key) FROM test.view (stage: Complete)
...
2024.10.04 12:19:52.170134 [ 1591 ] {} <Error> StorageRabbitMQ (rabbit_queue): Library error report: Name or service not known
...
695 repetitions of SELECT count(DISTINCT key) FROM test.view
...
2024.10.04 12:34:33.584002 [ 1021 ] {43bb2051-acf0-49a3-b293-9e815ddf2c10} <Debug> executeQuery: (from 172.16.3.1:62490) SELECT count(DISTINCT key) FROM test.view (stage: Complete)

It seems like a deadloop in the test here leading to the timeouts etc:

ClickHouse/tests/integration/test_storage_rabbitmq/test.py

Lines 2038 to 2042 in a23e807

while True:
result = instance.query("SELECT count(DISTINCT key) FROM test.view")
time.sleep(1)
if int(result) == messages_num:
break

and some issue with the queue after the RabbitMQ restart (maybe some change in the behavior of rabbitmq? Or somehow related to some of the plugins/feature flags which used to work before?)

The last messages in RabbitMQ log is

2024-10-04 12:57:52.645636+00:00 [debug] <0.1110.0> Closing 1 channel(s) because connection '172.16.3.4:35640 -> 172.16.3.2:5672' has been closed

For sure it sounds like a good idea to add maximum number of iterations to the RabbitMQ tests, maybe it will not be killed then, and a better diagnostic will be possible.

@filimonov filimonov mentioned this pull request Oct 9, 2024
Merged
@filimonov
Copy link
Copy Markdown
Contributor Author

filimonov commented Oct 9, 2024

For sure it sounds like a good idea to add maximum number of iterations to the RabbitMQ tests, maybe it will not be killed then, and a better diagnostic will be possible.

https://github.com/ClickHouse/ClickHouse/pull/70510/files

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kssenii kssenii kssenii approved these changes

Assignees

@kssenii kssenii

Labels

can be tested Allows running workflows for external contributors manual approve Manual approve required to run CI pr-ci pr-synced-to-cloud The PR is synced to the cloud repo

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

test_storage_rabbitmq is broken

6 participants

@filimonov @robot-ch-test-poll3 @kssenii @Algunenano @robot-clickhouse @robot-ch-test-poll