Skip to content

01455_opentelemetry_distributed is flaky #93452

@rschu1ze

Description

@rschu1ze

Test case history

Example failure (from here):

2026-01-03 17:34:36 Reason: result differs with reference:  
2026-01-03 17:34:36 --- /home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/tests/queries/0_stateless/01455_opentelemetry_distributed.reference	2026-01-03 17:03:32.833158342 +0700
2026-01-03 17:34:36 +++ /home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/tests/queries/0_stateless/01455_opentelemetry_distributed.stdout	2026-01-03 17:34:36.106227629 +0700
2026-01-03 17:34:36 @@ -1,19 +1,19 @@
2026-01-03 17:34:36  ===http===
2026-01-03 17:34:36  {"query":"select 1 from remote('127.0.0.2', system, one) settings enable_analyzer = 1 format Null\n","status":"QueryFinish","tracestate":"some custom state","sorted_by_start_time":1}
2026-01-03 17:34:36  {"query":"DESC TABLE system.one","status":"QueryFinish","tracestate":"some custom state","sorted_by_start_time":1}
2026-01-03 17:34:36  {"query":"SELECT 1 AS `1` FROM `system`.`one` AS `__table1`","status":"QueryFinish","tracestate":"some custom state","sorted_by_start_time":1}
2026-01-03 17:34:36  {"query":"DESC TABLE system.one","query_status":"QueryFinish","tracestate":"some custom state","sorted_by_finish_time":1}
2026-01-03 17:34:36  {"query":"SELECT 1 AS `1` FROM `system`.`one` AS `__table1`","query_status":"QueryFinish","tracestate":"some custom state","sorted_by_finish_time":1}
2026-01-03 17:34:36  {"query":"select 1 from remote('127.0.0.2', system, one) settings enable_analyzer = 1 format Null\n","query_status":"QueryFinish","tracestate":"some custom state","sorted_by_finish_time":1}
2026-01-03 17:34:36  {"total spans":3,"unique spans":3,"unique non-zero parent spans":3}
2026-01-03 17:34:36 -{"initial query spans with proper parent":2}
2026-01-03 17:34:36 +{"initial query spans with proper parent":1}
2026-01-03 17:34:36  {"unique non-empty tracestate values":1}
2026-01-03 17:34:36  ===native===
2026-01-03 17:34:36  {"query":"select * from url('http:\/\/127.0.0.2:8123\/?query=select%201%20format%20Null', CSV, 'a int')","status":"QueryFinish","tracestate":"another custom state","sorted_by_start_time":1}
2026-01-03 17:34:36  {"query":"select 1 format Null\n","status":"QueryFinish","tracestate":"another custom state","sorted_by_start_time":1}
2026-01-03 17:34:36  {"query":"select 1 format Null\n","status":"QueryFinish","tracestate":"another custom state","sorted_by_start_time":1}
2026-01-03 17:34:36  {"query":"select 1 format Null\n","query_status":"QueryFinish","tracestate":"another custom state","sorted_by_finish_time":1}
2026-01-03 17:34:36  {"query":"select 1 format Null\n","query_status":"QueryFinish","tracestate":"another custom state","sorted_by_finish_time":1}
2026-01-03 17:34:36  {"query":"select * from url('http:\/\/127.0.0.2:8123\/?query=select%201%20format%20Null', CSV, 'a int')","query_status":"QueryFinish","tracestate":"another custom state","sorted_by_finish_time":1}
2026-01-03 17:34:36  {"total spans":3,"unique spans":3,"unique non-zero parent spans":3}
2026-01-03 17:34:36  {"initial query spans with proper parent":1}
2026-01-03 17:34:36 
2026-01-03 17:34:36 
2026-01-03 17:34:36 /home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/tests/queries/0_stateless/test_xtl0nwyf/01455_opentelemetry_distributed.sh.debuglog:
2026-01-03 17:34:36 + [2026-01-03 17:34:16] [:77] echo ===http===
2026-01-03 17:34:36 ++ [2026-01-03 17:34:16] [:78] /home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ci/tmp/clickhouse-client --send_logs_level=warning --database=test_xtl0nwyf --log_comment 01455_opentelemetry_distributed.sh-test_xtl0nwyf --max_insert_threads 1 --group_by_two_level_threshold 418615 --group_by_two_level_threshold_bytes 21813284 --distributed_aggregation_memory_efficient 0 --fsync_metadata 0 --output_format_parallel_formatting 0 --input_format_parallel_parsing 0 --min_chunk_bytes_for_parallel_parsing 3670170 --max_read_buffer_size 837872 --prefer_localhost_replica 0 --max_block_size 69027 --max_joined_block_size_rows 17053 --joined_block_split_single_row 1 --join_output_by_rowlist_perkey_rows_threshold 803 --max_threads 3 --optimize_append_index 1 --use_hedged_requests 0 --optimize_if_chain_to_multiif 1 --optimize_if_transform_strings_to_enum 0 --optimize_read_in_order 1 --optimize_or_like_chain 0 --optimize_substitute_columns 1 --enable_multiple_prewhere_read_steps 0 --read_in_order_two_level_merge_threshold 82 --optimize_aggregation_in_order 0 --aggregation_in_order_max_block_bytes 8810288 --use_uncompressed_cache 0 --min_bytes_to_use_direct_io 1330170029 --min_bytes_to_use_mmap_io 10737418240 --local_filesystem_read_method mmap --remote_filesystem_read_method read --local_filesystem_read_prefetch 1 --filesystem_cache_segments_batch_size 2 --read_from_filesystem_cache_if_exists_otherwise_bypass_cache 0 --throw_on_error_from_cache_on_write_operations 1 --remote_filesystem_read_prefetch 1 --allow_prefetched_read_pool_for_remote_filesystem 0 --filesystem_prefetch_max_memory_usage 64Mi --filesystem_prefetches_limit 0 --filesystem_prefetch_min_bytes_for_single_read_task 16Mi --filesystem_prefetch_step_marks 50 --filesystem_prefetch_step_bytes 0 --compile_expressions 1 --compile_aggregate_expressions 0 --compile_sort_description 1 --merge_tree_coarse_index_granularity 2 --optimize_distinct_in_order 0 --max_bytes_before_remerge_sort 2168140869 --min_compress_block_size 682886 --max_compress_block_size 1414072 --merge_tree_compact_parts_min_granules_to_multibuffer_read 121 --optimize_sorting_by_input_stream_properties 0 --http_response_buffer_size 2606618 --http_wait_end_of_query False --enable_memory_bound_merging_of_aggregation_results 0 --min_count_to_compile_expression 0 --min_count_to_compile_aggregate_expression 0 --min_count_to_compile_sort_description 0 --session_timezone Africa/Juba --use_page_cache_for_disks_without_file_cache False --page_cache_inject_eviction True --merge_tree_read_split_ranges_into_intersecting_and_non_intersecting_injection_probability 0.8 --prefer_external_sort_block_bytes 1 --cross_join_min_rows_to_compress 1 --cross_join_min_bytes_to_compress 100000000 --min_external_table_block_size_bytes 0 --max_parsing_threads 10 --optimize_functions_to_subcolumns 0 --parallel_replicas_local_plan 0 --query_plan_join_swap_table false --enable_vertical_final 0 --optimize_extract_common_expressions 0 --use_async_executor_for_materialized_views 1 --use_query_condition_cache 0 --secondary_indices_enable_bulk_filtering 1 --use_skip_indexes_if_final 1 --use_skip_indexes_on_data_read 1 --optimize_rewrite_like_perfect_affix 1 --input_format_parquet_use_native_reader_v3 1 --enable_lazy_columns_replication 1 --allow_special_serialization_kinds_in_output_formats 1 --temporary_files_buffer_size 1002763 --max_bytes_before_external_sort 0 --max_bytes_before_external_group_by 0 --max_bytes_ratio_before_external_sort 0.0 --max_bytes_ratio_before_external_group_by 0.2 --use_skip_indexes_if_final_exact_mode 1 --allow_merge_tree_settings --ratio_of_defaults_for_sparse_serialization 1.0 --prefer_fetch_merged_part_size_threshold 3656422907 --vertical_merge_algorithm_min_rows_to_activate 1 --vertical_merge_algorithm_min_columns_to_activate 100 --allow_vertical_merges_from_compact_to_wide_parts 0 --min_merge_bytes_to_use_direct_io 1 --index_granularity_bytes 30934714 --merge_max_block_size 9495 --index_granularity 37824 --min_bytes_for_wide_part 1073741824 --marks_compress_block_size 77176 --primary_key_compress_block_size 77226 --replace_long_file_name_to_hash 1 --max_file_name_length 50 --min_bytes_for_full_part_storage 270405481 --compact_parts_max_bytes_to_buffer 57669858 --compact_parts_max_granules_to_buffer 124 --compact_parts_merge_max_bytes_to_prefetch_part 25561565 --cache_populated_by_fetch 1 --concurrent_part_removal_threshold 37 --old_parts_lifetime 267 --prewarm_mark_cache 0 --use_const_adaptive_granularity 1 --enable_index_granularity_compression 0 --enable_block_number_column 0 --enable_block_offset_column 0 --use_primary_key_cache 0 --prewarm_primary_key_cache 0 --object_serialization_version v3 --object_shared_data_serialization_version map --object_shared_data_serialization_version_for_zero_level_parts advanced --object_shared_data_buckets_for_compact_part 24 --object_shared_data_buckets_for_wide_part 2 --dynamic_serialization_version v3 --serialization_info_version basic --string_serialization_version single_stream --nullable_serialization_version basic --enable_shared_storage_snapshot_in_query 0 --allow_repeated_settings --client_logs_file=/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/tests/queries/0_stateless/01455_opentelemetry_distributed.stderr-fatal -q 'select lower(hex(reverse(reinterpretAsString(generateUUIDv4())))) settings enable_analyzer = 1'
2026-01-03 17:34:36 + [2026-01-03 17:34:17] [:78] trace_id=8e6aa972cc61ecf0be17b10c2c3b4be8
2026-01-03 17:34:36 + [2026-01-03 17:34:17] [:85] curl -q -s --max-time 120 --header 'traceparent: 00-8e6aa972cc61ecf0be17b10c2c3b4be8-0000000000000073-01' --header 'tracestate: some custom state' 'http://localhost:8123/?max_insert_threads=1&group_by_two_level_threshold=418615&group_by_two_level_threshold_bytes=21813284&distributed_aggregation_memory_efficient=0&fsync_metadata=0&output_format_parallel_formatting=0&input_format_parallel_parsing=0&min_chunk_bytes_for_parallel_parsing=3670170&max_read_buffer_size=837872&prefer_localhost_replica=0&max_block_size=69027&max_joined_block_size_rows=17053&joined_block_split_single_row=1&join_output_by_rowlist_perkey_rows_threshold=803&max_threads=3&optimize_append_index=1&use_hedged_requests=0&optimize_if_chain_to_multiif=1&optimize_if_transform_strings_to_enum=0&optimize_read_in_order=1&optimize_or_like_chain=0&optimize_substitute_columns=1&enable_multiple_prewhere_read_steps=0&read_in_order_two_level_merge_threshold=82&optimize_aggregation_in_order=0&aggregation_in_order_max_block_bytes=8810288&use_uncompressed_cache=0&min_bytes_to_use_direct_io=1330170029&min_bytes_to_use_mmap_io=10737418240&local_filesystem_read_method=mmap&remote_filesystem_read_method=read&local_filesystem_read_prefetch=1&filesystem_cache_segments_batch_size=2&read_from_filesystem_cache_if_exists_otherwise_bypass_cache=0&throw_on_error_from_cache_on_write_operations=1&remote_filesystem_read_prefetch=1&allow_prefetched_read_pool_for_remote_filesystem=0&filesystem_prefetch_max_memory_usage=64Mi&filesystem_prefetches_limit=0&filesystem_prefetch_min_bytes_for_single_read_task=16Mi&filesystem_prefetch_step_marks=50&filesystem_prefetch_step_bytes=0&compile_expressions=1&compile_aggregate_expressions=0&compile_sort_description=1&merge_tree_coarse_index_granularity=2&optimize_distinct_in_order=0&max_bytes_before_remerge_sort=2168140869&min_compress_block_size=682886&max_compress_block_size=1414072&merge_tree_compact_parts_min_granules_to_multibuffer_read=121&optimize_sorting_by_input_stream_properties=0&http_response_buffer_size=2606618&http_wait_end_of_query=False&enable_memory_bound_merging_of_aggregation_results=0&min_count_to_compile_expression=0&min_count_to_compile_aggregate_expression=0&min_count_to_compile_sort_description=0&session_timezone=Africa%2FJuba&use_page_cache_for_disks_without_file_cache=False&page_cache_inject_eviction=True&merge_tree_read_split_ranges_into_intersecting_and_non_intersecting_injection_probability=0.8&prefer_external_sort_block_bytes=1&cross_join_min_rows_to_compress=1&cross_join_min_bytes_to_compress=100000000&min_external_table_block_size_bytes=0&max_parsing_threads=10&optimize_functions_to_subcolumns=0&parallel_replicas_local_plan=0&query_plan_join_swap_table=false&enable_vertical_final=0&optimize_extract_common_expressions=0&use_async_executor_for_materialized_views=1&use_query_condition_cache=0&secondary_indices_enable_bulk_filtering=1&use_skip_indexes_if_final=1&use_skip_indexes_on_data_read=1&optimize_rewrite_like_perfect_affix=1&input_format_parquet_use_native_reader_v3=1&enable_lazy_columns_replication=1&allow_special_serialization_kinds_in_output_formats=1&temporary_files_buffer_size=1002763&max_bytes_before_external_sort=0&max_bytes_before_external_group_by=0&max_bytes_ratio_before_external_sort=0.0&max_bytes_ratio_before_external_group_by=0.2&use_skip_indexes_if_final_exact_mode=1&database=test_xtl0nwyf&log_comment=01455_opentelemetry_distributed.sh-test_xtl0nwyf' --get --data-urlencode 'query=select 1 from remote('\''127.0.0.2'\'', system, one) settings enable_analyzer = 1 format Null'
2026-01-03 17:34:36 + [2026-01-03 17:34:17] [:90] check_log
2026-01-03 17:34:36 + [2026-01-03 17:34:17] [:12] /home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ci/tmp/clickhouse-client --send_logs_level=warning --database=test_xtl0nwyf --log_comment 01455_opentelemetry_distributed.sh-test_xtl0nwyf --max_insert_threads 1 --group_by_two_level_threshold 418615 --group_by_two_level_threshold_bytes 21813284 --distributed_aggregation_memory_efficient 0 --fsync_metadata 0 --output_format_parallel_formatting 0 --input_format_parallel_parsing 0 --min_chunk_bytes_for_parallel_parsing 3670170 --max_read_buffer_size 837872 --prefer_localhost_replica 0 --max_block_size 69027 --max_joined_block_size_rows 17053 --joined_block_split_single_row 1 --join_output_by_rowlist_perkey_rows_threshold 803 --max_threads 3 --optimize_append_index 1 --use_hedged_requests 0 --optimize_if_chain_to_multiif 1 --optimize_if_transform_strings_to_enum 0 --optimize_read_in_order 1 --optimize_or_like_chain 0 --optimize_substitute_columns 1 --enable_multiple_prewhere_read_steps 0 --read_in_order_two_level_merge_threshold 82 --optimize_aggregation_in_order 0 --aggregation_in_order_max_block_bytes 8810288 --use_uncompressed_cache 0 --min_bytes_to_use_direct_io 1330170029 --min_bytes_to_use_mmap_io 10737418240 --local_filesystem_read_method mmap --remote_filesystem_read_method read --local_filesystem_read_prefetch 1 --filesystem_cache_segments_batch_size 2 --read_from_filesystem_cache_if_exists_otherwise_bypass_cache 0 --throw_on_error_from_cache_on_write_operations 1 --remote_filesystem_read_prefetch 1 --allow_prefetched_read_pool_for_remote_filesystem 0 --filesystem_prefetch_max_memory_usage 64Mi --filesystem_prefetches_limit 0 --filesystem_prefetch_min_bytes_for_single_read_task 16Mi --filesystem_prefetch_step_marks 50 --filesystem_prefetch_step_bytes 0 --compile_expressions 1 --compile_aggregate_expressions 0 --compile_sort_description 1 --merge_tree_coarse_index_granularity 2 --optimize_distinct_in_order 0 --max_bytes_before_remerge_sort 2168140869 --min_compress_block_size 682886 --max_compress_block_size 1414072 --merge_tree_compact_parts_min_granules_to_multibuffer_read 121 --optimize_sorting_by_input_stream_properties 0 --http_response_buffer_size 2606618 --http_wait_end_of_query False --enable_memory_bound_merging_of_aggregation_results 0 --min_count_to_compile_expression 0 --min_count_to_compile_aggregate_expression 0 --min_count_to_compile_sort_description 0 --session_timezone Africa/Juba --use_page_cache_for_disks_without_file_cache False --page_cache_inject_eviction True --merge_tree_read_split_ranges_into_intersecting_and_non_intersecting_injection_probability 0.8 --prefer_external_sort_block_bytes 1 --cross_join_min_rows_to_compress 1 --cross_join_min_bytes_to_compress 100000000 --min_external_table_block_size_bytes 0 --max_parsing_threads 10 --optimize_functions_to_subcolumns 0 --parallel_replicas_local_plan 0 --query_plan_join_swap_table false --enable_vertical_final 0 --optimize_extract_common_expressions 0 --use_async_executor_for_materialized_views 1 --use_query_condition_cache 0 --secondary_indices_enable_bulk_filtering 1 --use_skip_indexes_if_final 1 --use_skip_indexes_on_data_read 1 --optimize_rewrite_like_perfect_affix 1 --input_format_parquet_use_native_reader_v3 1 --enable_lazy_columns_replication 1 --allow_special_serialization_kinds_in_output_formats 1 --temporary_files_buffer_size 1002763 --max_bytes_before_external_sort 0 --max_bytes_before_external_group_by 0 --max_bytes_ratio_before_external_sort 0.0 --max_bytes_ratio_before_external_group_by 0.2 --use_skip_indexes_if_final_exact_mode 1 --allow_merge_tree_settings --ratio_of_defaults_for_sparse_serialization 1.0 --prefer_fetch_merged_part_size_threshold 3656422907 --vertical_merge_algorithm_min_rows_to_activate 1 --vertical_merge_algorithm_min_columns_to_activate 100 --allow_vertical_merges_from_compact_to_wide_parts 0 --min_merge_bytes_to_use_direct_io 1 --index_granularity_bytes 30934714 --merge_max

Metadata

Metadata

Assignees

Labels

flaky testflaky test found by CItestingSpecial issue with list of bugs found by CI

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions