Infrastructure Failures

• Connection reset by peer while downloading test data from s3: Added a test for #11135 #12411 Need to add retries. Fixed Retries in s3 downloader #14578.
• Yandex Synchronization Check sometimes show bogus results @exprmntr
• TSan stress test sometimes killed by OOM: https://sandbox.yandex-team.ru/task/670887196/hardware
• Sometimes the check finish with SUCCESS but did not publish the result on GitHub due to GitHub API requests limit (5000 per hour).
• Step 13/14 : COPY build.sh / COPY failed: stat /place/sandbox-data/tasks/8/4/776403748/docker_root/tmp/docker-builder438288327/build.sh: no such file or directory: https://clickhouse-builds.s3.yandex.net/15027/ef800b508351108e59769a3587c5c1d0ff6dbab3/clickhouse_build_check/report.html#fail1

Test Failures

• 00682_empty_parts_merge https://clickhouse-test-reports.s3.yandex.net/17175/73c34acafc8725c501a7e94406f96daa672cb517/functional_stateless_tests_(thread).html#fail1
• test_https_replication/test.py::test_replication_after_partition https://clickhouse-test-reports.s3.yandex.net/15407/6e70f2ce7afab4202cd925f098f7b4f523b19b55/integration_tests_(release).html#fail1
• test_reload_zookeeper/test.py::test_reload_zookeeper https://clickhouse-test-reports.s3.yandex.net/14962/cf24789c1b6fe1dc81443cbf1ce6b4b0f10dd97d/integration_tests_(release).html#fail1
• test_s3_with_proxy/test.py::test_s3_with_proxy_list[s3] https://clickhouse-test-reports.s3.yandex.net/15158/0752fa01371d07c340217239b1fe6450f1e5684f/integration_tests_(release).html#fail1 https://clickhouse-test-reports.s3.yandex.net/15300/97671a17719eb9c101095e01cac7baa282908126/integration_tests_(release).html#fail1
• test_rename_column/test.py::test_rename_distributed_parallel_insert_and_select https://clickhouse-test-reports.s3.yandex.net/15062/6e1bd79d51958603be38907c622493bcad9b0c8e/integration_tests_(asan).html#fail1 -- @CurtizJ is fixing it https://t.me/c/1078904674/63413
• test_dictionaries_all_layouts_and_sources/test.py::test_simple_dictionaries[3]. Incorrect result with thread sanitizer. https://clickhouse-test-reports.s3.yandex.net/0/4a73fe7477d52d806d71d2b2d57a8a2ca6e679e9/integration_tests_(thread).html. @alesapin and https://clickhouse-test-reports.s3.yandex.net/9516/2cda3c962ceb1c420919e3a187b89ef746421e9c/integration_tests_(asan).html#fail1 and test_dictionaries_all_layouts_and_sources/test.py::test_complex_dictionaries[6] https://clickhouse-test-reports.s3.yandex.net/0/1c2435d4f7d5363ceec1c5a3f4273daad5f6cd0f/integration_tests_(thread).html
• test_zookeeper_config/test.py::test_secure_connection broken with tsan link thread sanitizer trace
• No route to host -- test_mysql_protocol - https://clickhouse-test-reports.s3.yandex.net/11163/ec894e7cb3e0e3f9f7c880b00d44ee3b83e028ad/integration_tests_(asan).html#fail1
• 00652_replicated_mutations_zookeeper.stdout https://clickhouse-test-reports.s3.yandex.net/0/d2be6c036b3b2f4bec8de985a9c620f9ae2cd28d/functional_stateless_tests_(release).html
  and https://clickhouse-test-reports.s3.yandex.net/14363/efd59eca62465e4dff860a588683215a4acdb353/functional_stateless_tests_(ubsan).html#fail1
• test_distributed_respect_user_timeouts/test.py::test_reconnect[configs-default-node2-remote] +
  [configs-default-node1-remote] + [configs_secure-default-node2-remote]
  https://clickhouse-test-reports.s3.yandex.net/0/ff0262626ab08919909d967b2859ee107656e659/integration_tests_(thread).html https://clickhouse-test-reports.s3.yandex.net/0/1a3a19391c144c3aed96d29db8bb47f5b0b899d7/integration_tests_(thread).html
• 01064_incremental_streaming_from_2_src_with_feedback https://clickhouse-test-reports.s3.yandex.net/0/561d4b5b0108b300746d9a03978d7dfa75d58a36/functional_stateless_tests_(release,_databaseatomic).html
• use-of-uninitialized-value DB::parseIPv6 https://clickhouse-test-reports.s3.yandex.net/0/e460d7c2bd58b379cc68a2a2e144305af36f9c02/stress_test_(memory).html
• 00738_lock_for_inner_table https://clickhouse-test-reports.s3.yandex.net/12045/987e64acfe789689b8710add5d04371b8f21b604/functional_stateless_tests_(debug).html#fail1
• 01318_long_failing_mutation_zookeeper https://clickhouse-test-reports.s3.yandex.net/0/6f2d9999be75efdab1d4753620d6943592e674d0/functional_stateless_tests_(unbundled).html (someone just stopped merges globally and doesn't start them back, probably it's fixed)
• 01280_ttl_where_group_by https://clickhouse-test-reports.s3.yandex.net/0/ad53ef2c14da180fd56efa6439307e4650052a3d/functional_stateless_tests_(release,_databaseatomic).html
• multiSearch, MSan: https://clickhouse-test-reports.s3.yandex.net/12292/eaa13d81769af71d6c620881e7064ff47285c9aa/stress_test_(memory).html#fail1 (something related to Hyperscan?)
• 01133_max_result_rows: https://clickhouse-test-reports.s3.yandex.net/11908/5a7e921135d27de93cf25479fd46a0eb7527ba3b/functional_stateless_tests_(release,_polymorphic_parts_enabled).html#fail1
• Hyperscan and async unwinding: https://clickhouse-test-reports.s3.yandex.net/12440/e8a3f0abb3e3169f0ade88bc2edd076ea4a0cff1/functional_stateful_tests_(release).html#fail1
• 00944_clear_index_in_partition (database atomic) https://clickhouse-test-reports.s3.yandex.net/0/21d3a797944066cb0b520e372570717ab165f3c4/functional_stateless_tests_(release,_databaseatomic).html#fail1
• 01040_dictionary_invalidate_query_switchover_long https://clickhouse-test-reports.s3.yandex.net/0/166a7b469ff7de99cd1d995eba950a31c52055a3/functional_stateless_tests_(release).html
• MemorySanitizer: use-of-uninitialized-value MemorySanitizer: use-of-uninitialized-value #14006
• zkutil.MultiAsync https://clickhouse-test-reports.s3.yandex.net/0/21d3a797944066cb0b520e372570717ab165f3c4/unit_tests_msan_clang-10.html
• materialize_index: https://clickhouse-test-reports.s3.yandex.net/12484/767bc7a905cfce8634b5348eccafa4fb9e7210ca/functional_stateless_tests_(address).html#fail1
• 01307_multiple_leaders https://clickhouse-test-reports.s3.yandex.net/12953/4bd6261dc164c64172c641adafec271df4c12301/functional_stateless_tests_(debug).html#fail1
• 01305_replica_create_drop_zookeeper https://clickhouse-test-reports.s3.yandex.net/0/795c09fdbbbe2c1f156d8b0c795cd9992297cb20/functional_stateless_tests_(unbundled).html#fail1
• 00960_live_view_watch_events_live https://clickhouse-test-reports.s3.yandex.net/13075/d3bcf89ae427dcb2b9cd7def3ac7296de3def723/functional_stateless_tests_(address).html#fail1
• test_storage_rabbitmq/test.py::test_rabbitmq_many_materialized_views:
  https://clickhouse-test-reports.s3.yandex.net/13167/b57219e008866b46b8e7c0b4ed5c175429e1c9e1/integration_tests_(release).html#fail1
  https://clickhouse-test-reports.s3.yandex.net/13235/8cbcd4027a52c09d141d5a10d440f30591ed76af/integration_tests_(release).html
  https://clickhouse-test-reports.s3.yandex.net/13611/de73fdae0c88e4286ef0de03c98c71003b4d78e3/integration_tests_(release).html#fail1
• test_version_update_after_mutation - https://clickhouse-test-reports.s3.yandex.net/13234/d28c4c4a6c82f7812562ba9ba7698e657670ee45/integration_tests_(asan).html#fail1
• https://clickhouse-test-reports.s3.yandex.net/0/73c349fd7fcc5bb0c66df2b7a0f0f575fc5a47e2/integration_tests_(asan).html
  Timeouts in test_storage_rabbitmq/test.py::test_rabbitmq_tsv_with_delimiter
  test_storage_rabbitmq/test.py::test_rabbitmq_many_materialized_views
• test_storage_rabbitmq/test.py::test_rabbitmq_headers_exchange
  https://clickhouse-test-reports.s3.yandex.net/14432/cd812d94deb7e03ae9fea5c77df1e6765aac5dbd/integration_tests_(asan).html#fail1
  WRITE locking attempt on "test.destination" has timed out! (120000ms) Possible deadlock avoided
• 00620_optimize_on_nonleader_replica_zookeeper
  https://clickhouse-test-reports.s3.yandex.net/0/73c349fd7fcc5bb0c66df2b7a0f0f575fc5a47e2/functional_stateless_tests_(unbundled).html
• test_mysql_protocol/test.py::test_mysql_federated https://clickhouse-test-reports.s3.yandex.net/13530/b633b49e072bdca7419f04fb93469ddfaacc5b39/integration_tests_(release).html
• 00633_materialized_view_and_too_many_parts_zookeeper - https://clickhouse-test-reports.s3.yandex.net/13156/dbf966cde31c1d46790ef8fa0fce6d68b79783ed/functional_stateless_tests_(release,_databaseatomic).html#fail1
• 00151_replace_partition_with_different_granularity: [ FAIL ] 600.92 sec. - Timeout!
  https://clickhouse-test-reports.s3.yandex.net/13611/de73fdae0c88e4286ef0de03c98c71003b4d78e3/functional_stateful_tests_(debug).html#fail1
• mutations_and_errors_zookeeper: https://clickhouse-test-reports.s3.yandex.net/0/8f16c542b6969ef11c0f3051a1f530fd82ea9d96/fast_test/runlog.out.log
• test_materialize_mysql_database
  https://clickhouse-test-reports.s3.yandex.net/10373/a2467f205f4ab120443afbb426cec3d5a506478d/integration_tests_(release).html#fail1
  https://clickhouse-test-reports.s3.yandex.net/13792/f63db2f4781b3d0cbfd4a91e6ddf3f9bb9f97066/integration_tests_(release).html#fail1
• 00956_sensitive_data_masking is still flacky https://clickhouse-test-reports.s3.yandex.net/10373/348ef1256ea8fb8f61109c33bbdd28daf46bdc8e/functional_stateless_tests_(debug).html#fail1
• move_partition_from_table_zookeeper https://clickhouse-test-reports.s3.yandex.net/13867/3ba9e9f6ced82b0c458bbce4bd99366baf6d5951/functional_stateless_tests_(release,_databaseatomic).html#fail1
• https://clickhouse-test-reports.s3.yandex.net/13647/e77ab608c8e579caca7131cc2036dbac3d32e582/stress_test_(thread).html#fail1 and https://clickhouse-test-reports.s3.yandex.net/13928/99b31480fd18b04677ba74bad21e5b3d7d928312/stress_test_(thread).html#fail1
  stress test ThreadSanitizer: data race
  Repair subtree2 #1 DB::CacheDictionary::getSource() const /build/obj-x86_64-linux-gnu/../src/Dictionaries/CacheDictionary.h💯78 (clickhouse+0xf5e732d)
• crash in CacheDictionary::update https://clickhouse-test-reports.s3.yandex.net/14412/382ebc700e5cc890df20d68d4ffd026c368c2b7c/stress_test_(memory).html#fail1 https://pastebin.com/raw/D9nj5pnY @nikitamikhaylov
• system.clusters data race: https://clickhouse-test-reports.s3.yandex.net/13880/ad30dcf434d24ca447c7533d7c258a7f0d8c67ca/stress_test_(thread)/stderr.log
• Segmentation fault during table drop in lazy database in stress test with msan:
  Faster test under TSan #14027 (comment)
  https://clickhouse-test-reports.s3.yandex.net/14027/ffd8c193852283fa7ec675814b6e5e3a46dc511b/stress_test_(memory).html#fail1
  cc: @nikvas0
• 01056_prepared_statements_null_and_escaping
• Stateless tests under TSan take 3 hours to run: https://clickhouse-test-reports.s3.yandex.net/14048/f95acff93cb17afda047f6fe2d40ebb59e3fb2f8/functional_stateless_tests_(thread)/test_run.txt.out.log
• 00633_materialized_view_and_too_many_parts_zookeeper with database atomic https://clickhouse-test-reports.s3.yandex.net/14049/e5bc5ea419d49c807dada50b1e33966537fd0ada/functional_stateless_tests_(release,_databaseatomic).html#fail1
• 01193_metadata_loading not fixed
  https://clickhouse-test-reports.s3.yandex.net/14167/bfe4e94806444d1225d93a083a716ff8d770be3c/functional_stateless_tests_(release,_databaseatomic).html#fail1
• 01442_merge_detach_attach test highlights issues if we enable Thread Fuzzer: https://clickhouse-test-reports.s3.yandex.net/9813/e412017200ef3300d1b36bcafdf46410dd5f4cfc/fast_test/runlog.out.log
• 00693_max_block_size_system_tables_columns flapped
  https://clickhouse-test-reports.s3.yandex.net/14219/f51e0bfeb0d7b93d8e0da9ceae0e4013d7d57927/fast_test.html#fail1
• ttl_where_group_by: https://clickhouse-test-reports.s3.yandex.net/14233/67a35303dd927f7ff447903d9f4f570eb9b48193/functional_stateless_tests_(address).html#fail1
• 00816_long_concurrent_alter_column https://clickhouse-test-reports.s3.yandex.net/0/5a529f171ce44c21f2039b05135b40813f2ab0d1/functional_stateless_tests_(debug).html
• test_concurrent_ttl_merges/test.py::test_no_ttl_merges_in_busy_pool https://clickhouse-test-reports.s3.yandex.net/14703/d0c259974d1486dc90ae29958a3f1dd196052961/integration_tests_(asan).html#fail1 @alesapin
• 00652_replicated_mutations_zookeeper need to check test https://clickhouse-test-reports.s3.yandex.net/14609/393c213866ff56ef98ae2f66ee50afebc9459319/functional_stateless_tests_(thread).html#fail1
• test_materialize_mysql_database/test.py::test_materialize_database_ddl_with_empty_transaction_8_0
  https://clickhouse-test-reports.s3.yandex.net/14809/2b113f7553f7f73c9c21a15e11999de10dfe5953/integration_tests_(asan).html#fail1
• test_materialize_mysql_database/test.py::test_materialize_database_ddl_with_empty_transaction_5_7
  https://clickhouse-test-reports.s3.yandex.net/14809/2b113f7553f7f73c9c21a15e11999de10dfe5953/integration_tests_(asan)/integration_run.3.txt.out.log
• test_s3_with_proxy - https://clickhouse-test-reports.s3.yandex.net/14731/6657be90a1e9621fa60321d891a30266174f8525/integration_tests_(release).html#fail1
• 01370_client_autocomplete_word_break_characters - https://clickhouse-test-reports.s3.yandex.net/15063/3fd85430f2431af8b8ef11a1c80966a0820be6e5/functional_stateless_tests_(thread).html#fail1
• Try fix race on Context::settings #15243 Stress tests ThreadSanitizer: data race /build/obj-x86_64-linux-gnu/../src/Core/SettingsFields.h:43:52 in DB::SettingFieldNumber::operator=(unsigned long) https://clickhouse-test-reports.s3.yandex.net/14265/494e59e7c12e0887a32b2564463a84aea500118b/stress_test_(thread)/stderr.log
• Stress tests DB::LazyPipeFDs::close() data race: https://clickhouse-test-reports.s3.yandex.net/14265/e2ee97202bf7b81cae3ab05458b6c4d8609c2617/stress_test_(thread)/stderr.log
• MemorySanitizer - use-of-uninitialized-value in MergedBlockOutputStream::writeWithPermutation(...)
  https://clickhouse-test-reports.s3.yandex.net/15240/dce6a436f37713224c242092446e1a9fcf8b4a4a/stress_test_(memory)/stderr.log
• MemorySanitizer src/Common/hex.h:57:9 in void writeHexUIntImpl Unitialized memory in system.parts for QueryLog #15284
• MemorySanitizer - use-of-uninitialized-value when logging an exception https://clickhouse-test-reports.s3.yandex.net/15003/fb826557b7ce3a3015fe00df78609ed42d9c5412/stress_test_(memory)/stderr.log
• 01249_bad_arguments_for_bloom_filter https://clickhouse-test-reports.s3.yandex.net/0/788b62b931c28b63ae47f1adce4ce5ca80cd3c68/functional_stateless_tests_(release,_databaseatomic).html
• 01455_shard_leaf_max_rows_bytes_to_read https://clickhouse-test-reports.s3.yandex.net/0/651518566a527f8a0c225f1d8a130712c8b810ee/functional_stateless_tests_(address).html
• Stress (thread) https://clickhouse-test-reports.s3.yandex.net/15242/0db8b6d647b67ffb480790626015a13b2b929035/stress_test_(thread).html#fail1
• MergeTreeThreadSelectBlockInputProcessor.cpp:75 member access ... which does not point to an object of type 'const DB::MergeTreeData' #15475 Stress undefined
• 00933_test_fix_extra_seek_on_compressed_cache - https://clickhouse-test-reports.s3.yandex.net/15642/e00008e61dc13cedb8e9f4c8afd6501f9f7b6a6e/fast_test.html#fail1 (refs Fix race condition in test #13747)
• Crash in quantilesDeterministic Assert in quantilesDeterministic #15683
• 01249_flush_interactive Make a test slightly more reliable #13760