-
Notifications
You must be signed in to change notification settings - Fork 8.3k
Extremely rare intersecting parts #26876
Copy link
Copy link
Closed
Labels
fuzzProblem found by one of the fuzzersProblem found by one of the fuzzers
Description
We have an extremely rare reproduction case of intersecting parts in queue: https://clickhouse-test-reports.s3.yandex.net/26691/f61ecee9892f7adeacf70ca413364c3626671e79/functional_stateless_tests_(release).html#fail1
From logs
2021.07.22 15:05:41.778101 [ 308 ] {} <Trace> test_w17phc.alter_table_1 (fb007db4-7546-4dda-bb00-7db47546bdda): Created log entry for mutation -1_32_32_0_61
2021.07.22 15:05:41.781976 [ 89955 ] {} <Trace> test_w17phc.alter_table_9 (7e898522-0a5a-420c-be89-85220a5a020c): Executing log entry to mutate part -1_32_32_0 to -1_32_32_0_61
2021.07.22 15:05:41.781998 [ 89955 ] {} <Debug> test_w17phc.alter_table_9 (7e898522-0a5a-420c-be89-85220a5a020c): Source part -1_32_32_0 for -1_32_32_0_61 is not ready; will try to fetch it instead
2021.07.22 15:05:41.782216 [ 272 ] {} <Trace> test_w17phc.alter_table_9 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000468 for part -1_32_32_0 because it is covered by part -1_32_32_0_61 that is currently executing.
2021.07.22 15:05:41.784196 [ 89882 ] {} <Trace> test_w17phc.alter_table_6 (2b9ce506-edfd-40b0-ab9c-e506edfd10b0): Executing log entry to mutate part -1_32_32_0 to -1_32_32_0_61
2021.07.22 15:05:41.784222 [ 89882 ] {} <Debug> test_w17phc.alter_table_6 (2b9ce506-edfd-40b0-ab9c-e506edfd10b0): Source part -1_32_32_0 for -1_32_32_0_61 is not ready; will try to fetch it instead
2021.07.22 15:05:41.784998 [ 327 ] {} <Trace> test_w17phc.alter_table_6 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000567 for part -1_32_32_0 because it is covered by part -1_32_32_0_61 that is currently executing.
2021.07.22 15:05:41.785940 [ 8418 ] {} <Trace> test_w17phc.alter_table_1 (fb007db4-7546-4dda-bb00-7db47546bdda): Executing log entry to mutate part -1_32_32_0 to -1_32_32_0_61
2021.07.22 15:05:41.786593 [ 232 ] {} <Trace> test_w17phc.alter_table_1 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000522 for part -1_32_32_0 because it is covered by part -1_32_32_0_61 that is currently executing.
2021.07.22 15:05:41.787231 [ 315 ] {} <Trace> test_w17phc.alter_table_9 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000468 for part -1_32_32_0 because it is covered by part -1_32_32_0_61 that is currently executing.
2021.07.22 15:05:41.809082 [ 89533 ] {} <Debug> test_w17phc.alter_table_6 (2b9ce506-edfd-40b0-ab9c-e506edfd10b0): Fetching part -1_32_32_0_61 from /clickhouse/tables/00993_system_parts_race_condition_drop_zookeeper_test_w17phc/alter_table/replicas/r_1
2021.07.22 15:05:41.813336 [ 89533 ] {} <Trace> test_w17phc.alter_table_6 (2b9ce506-edfd-40b0-ab9c-e506edfd10b0): Renaming temporary part tmp_fetch_-1_32_32_0_61 to -1_32_32_0_61.
2021.07.22 15:05:41.846894 [ 255 ] {} <Trace> test_w17phc.alter_table_1 (fb007db4-7546-4dda-bb00-7db47546bdda): Will try to insert a log entry to DROP_RANGE for part: -1_32_32_0_61
2021.07.22 15:05:41.851334 [ 274 ] {} <Trace> test_w17phc.alter_table_9 (7e898522-0a5a-420c-be89-85220a5a020c): Will try to insert a log entry to DROP_RANGE for part: -1_32_32_0_61
2021.07.22 15:05:41.869251 [ 301 ] {} <Trace> test_w17phc.alter_table_6 (2b9ce506-edfd-40b0-ab9c-e506edfd10b0): Will try to insert a log entry to DROP_RANGE for part: -1_32_32_0_61
2021.07.22 15:05:41.877614 [ 89487 ] {} <Debug> test_w17phc.alter_table_6 (ReplicatedMergeTreeQueue): Removed 1 entries from queue. Waiting for 0 entries that are currently executing.
2021.07.22 15:05:41.877670 [ 89487 ] {} <Debug> test_w17phc.alter_table_6 (2b9ce506-edfd-40b0-ab9c-e506edfd10b0): Removing parts.
2021.07.22 15:05:41.879818 [ 89487 ] {} <Debug> test_w17phc.alter_table_6 (2b9ce506-edfd-40b0-ab9c-e506edfd10b0): Removed 1 parts inside -1_32_32_0_61.
2021.07.22 15:05:42.034238 [ 265 ] {} <Trace> test_w17phc.alter_table_6 (2b9ce506-edfd-40b0-ab9c-e506edfd10b0): Found 51 old parts to remove.
2021.07.22 15:05:42.034287 [ 265 ] {} <Debug> test_w17phc.alter_table_6 (2b9ce506-edfd-40b0-ab9c-e506edfd10b0): Removing 51 old parts from ZooKeeper
2021.07.22 15:05:42.038197 [ 265 ] {} <Debug> test_w17phc.alter_table_6 (2b9ce506-edfd-40b0-ab9c-e506edfd10b0): There is no part -1_32_32_0_61 in ZooKeeper, it was only in filesystem
2021.07.22 15:05:42.039110 [ 265 ] {} <Debug> test_w17phc.alter_table_6 (2b9ce506-edfd-40b0-ab9c-e506edfd10b0): There is no part -9_18_18_0 in ZooKeeper, it was only in filesystem
2021.07.22 15:05:42.039131 [ 265 ] {} <Debug> test_w17phc.alter_table_6 (2b9ce506-edfd-40b0-ab9c-e506edfd10b0): There is no part 1_23_23_0 in ZooKeeper, it was only in filesystem
2021.07.22 15:05:42.039144 [ 265 ] {} <Debug> test_w17phc.alter_table_6 (2b9ce506-edfd-40b0-ab9c-e506edfd10b0): There is no part 8_26_26_0 in ZooKeeper, it was only in filesystem
2021.07.22 15:05:42.041398 [ 265 ] {} <Debug> test_w17phc.alter_table_6 (2b9ce506-edfd-40b0-ab9c-e506edfd10b0): Removed 51 old parts from ZooKeeper. Removing them from filesystem.
2021.07.22 15:05:42.041441 [ 265 ] {} <Debug> test_w17phc.alter_table_6 (2b9ce506-edfd-40b0-ab9c-e506edfd10b0): Removing part from filesystem -1_0_5_1_24
2021.07.22 15:05:42.041871 [ 265 ] {} <Debug> test_w17phc.alter_table_6 (2b9ce506-edfd-40b0-ab9c-e506edfd10b0): Removing part from filesystem -1_0_16_2_24
2021.07.22 15:05:42.042122 [ 265 ] {} <Debug> test_w17phc.alter_table_6 (2b9ce506-edfd-40b0-ab9c-e506edfd10b0): Removing part from filesystem -1_0_16_2_40
2021.07.22 15:05:42.042373 [ 265 ] {} <Debug> test_w17phc.alter_table_6 (2b9ce506-edfd-40b0-ab9c-e506edfd10b0): Removing part from filesystem -1_0_16_2_43
2021.07.22 15:05:42.042591 [ 265 ] {} <Debug> test_w17phc.alter_table_6 (2b9ce506-edfd-40b0-ab9c-e506edfd10b0): Removing part from filesystem -1_8_8_0_24
2021.07.22 15:05:42.042904 [ 265 ] {} <Debug> test_w17phc.alter_table_6 (2b9ce506-edfd-40b0-ab9c-e506edfd10b0): Removing part from filesystem -1_9_9_0_24
2021.07.22 15:05:42.043195 [ 265 ] {} <Debug> test_w17phc.alter_table_6 (2b9ce506-edfd-40b0-ab9c-e506edfd10b0): Removing part from filesystem -1_10_10_0_24
2021.07.22 15:05:42.043505 [ 265 ] {} <Debug> test_w17phc.alter_table_6 (2b9ce506-edfd-40b0-ab9c-e506edfd10b0): Removing part from filesystem -1_11_11_0_24
2021.07.22 15:05:42.043807 [ 265 ] {} <Debug> test_w17phc.alter_table_6 (2b9ce506-edfd-40b0-ab9c-e506edfd10b0): Removing part from filesystem -1_13_13_0
2021.07.22 15:05:42.044220 [ 265 ] {} <Debug> test_w17phc.alter_table_6 (2b9ce506-edfd-40b0-ab9c-e506edfd10b0): Removing part from filesystem -1_13_13_0_24
2021.07.22 15:05:42.044613 [ 265 ] {} <Debug> test_w17phc.alter_table_6 (2b9ce506-edfd-40b0-ab9c-e506edfd10b0): Removing part from filesystem -1_14_14_0
2021.07.22 15:05:42.045065 [ 265 ] {} <Debug> test_w17phc.alter_table_6 (2b9ce506-edfd-40b0-ab9c-e506edfd10b0): Removing part from filesystem -1_14_14_0_24
2021.07.22 15:05:42.048232 [ 265 ] {} <Debug> test_w17phc.alter_table_6 (2b9ce506-edfd-40b0-ab9c-e506edfd10b0): Removing part from filesystem -1_32_32_0_61
2021.07.22 15:05:43.857503 [ 285 ] {} <Trace> test_w17phc.alter_table_6 (2b9ce506-edfd-40b0-ab9c-e506edfd10b0): Log entry is not created for merge -1_28_38_1 because log was updated
2021.07.22 15:05:43.875269 [ 287 ] {} <Trace> test_w17phc.alter_table_6 (2b9ce506-edfd-40b0-ab9c-e506edfd10b0): Will try to insert a log entry to DROP_RANGE for part: -5_76_76_0
2021.07.22 15:05:43.889751 [ 230 ] {} <Debug> test_w17phc.alter_table_6 (2b9ce506-edfd-40b0-ab9c-e506edfd10b0) (MergerMutator): Selected 10 parts from -1_28_28_0 to -1_38_38_0
2021.07.22 15:05:43.895635 [ 230 ] {} <Trace> test_w17phc.alter_table_6 (2b9ce506-edfd-40b0-ab9c-e506edfd10b0): Created log entry /clickhouse/tables/00993_system_parts_race_condition_drop_zookeeper_test_w17phc/alter_table/log/log-0000001708 for merge -1_28_38_1
2021.07.22 15:05:43.898020 [ 287 ] {} <Debug> test_w17phc.alter_table_6 (ReplicatedMergeTreeQueue): Pulling 4 entries to queue: log-0000001705 - log-0000001708
2021.07.22 15:05:43.901541 [ 287 ] {} <Error> test_w17phc.alter_table_6 (ReplicatedMergeTreeQueue): Code: 49. DB::Exception: Part -1_28_38_1 intersects next part -1_32_32_0_61. It is a bug. (LOGICAL_ERROR), Stack trace (when copying this message, always include the lines below):
2021.07.22 15:05:43.902364 [ 314318 ] {} <Fatal> BaseDaemon: 1. gsignal @ 0x4618b in /usr/lib/x86_64-linux-gnu/libc-2.31.so
2021.07.22 15:05:43.902414 [ 314318 ] {} <Fatal> BaseDaemon: 2. abort @ 0x25859 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
2021.07.22 15:05:43.902465 [ 314318 ] {} <Fatal> BaseDaemon: 3. terminate_handler() @ 0xfaed848 in /usr/lib/debug/.build-id/6f/14cac530f243bccae792f9337218c3ee6e0634.debug
2021.07.22 15:05:43.902514 [ 314318 ] {} <Fatal> BaseDaemon: 4. std::__terminate(void (*)()) @ 0x15be7a83 in /usr/lib/debug/.build-id/6f/14cac530f243bccae792f9337218c3ee6e0634.debug
2021.07.22 15:05:43.902559 [ 314318 ] {} <Fatal> BaseDaemon: 5. std::terminate() @ 0x15be79ec in /usr/lib/debug/.build-id/6f/14cac530f243bccae792f9337218c3ee6e0634.debug
2021.07.22 15:05:43.902599 [ 314318 ] {} <Fatal> BaseDaemon: 6. DB::ReplicatedMergeTreeQueue::pullLogsToQueue(std::__1::shared_ptr<zkutil::ZooKeeper>, std::__1::function<void (Coordination::WatchResponse const&)>) @ 0x10fd256c in /usr/lib/debug/.build-id/6f/14cac530f243bccae792f9337218c3ee6e0634.debug
2021.07.22 15:05:43.902638 [ 314318 ] {} <Fatal> BaseDaemon: 7. DB::ReplicatedMergeTreeMergePredicate::ReplicatedMergeTreeMergePredicate(DB::ReplicatedMergeTreeQueue&, std::__1::shared_ptr<zkutil::ZooKeeper>&) @ 0x10fec3c5 in /usr/lib/debug/.build-id/6f/14cac530f243bccae792f9337218c3ee6e0634.debug
2021.07.22 15:05:43.902683 [ 314318 ] {} <Fatal> BaseDaemon: 8. DB::StorageReplicatedMergeTree::dropPartImpl(std::__1::shared_ptr<zkutil::ZooKeeper>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, DB::ReplicatedMergeTreeLogEntry&, bool, bool) @ 0x10be6ef8 in /usr/lib/debug/.build-id/6f/14cac530f243bccae792f9337218c3ee6e0634.debug
2021.07.22 15:05:43.902718 [ 314318 ] {} <Fatal> BaseDaemon: 9. DB::StorageReplicatedMergeTree::dropPartNoWaitNoThrow(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0x10be69a2 in /usr/lib/debug/.build-id/6f/14cac530f243bccae792f9337218c3ee6e0634.debug
2021.07.22 15:05:43.902782 [ 314318 ] {} <Fatal> BaseDaemon: 10. DB::MergeTreeData::clearEmptyParts() @ 0x10e10e57 in /usr/lib/debug/.build-id/6f/14cac530f243bccae792f9337218c3ee6e0634.debug
2021.07.22 15:05:43.902826 [ 314318 ] {} <Fatal> BaseDaemon: 11. DB::ReplicatedMergeTreeCleanupThread::iterate() @ 0x10fa2921 in /usr/lib/debug/.build-id/6f/14cac530f243bccae792f9337218c3ee6e0634.debug
2021.07.22 15:05:43.902888 [ 314318 ] {} <Fatal> BaseDaemon: 12. DB::ReplicatedMergeTreeCleanupThread::run() @ 0x10fa1402 in /usr/lib/debug/.build-id/6f/14cac530f243bccae792f9337218c3ee6e0634.debug
2021.07.22 15:05:43.902931 [ 314318 ] {} <Fatal> BaseDaemon: 13. DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x10132080 in /usr/lib/debug/.build-id/6f/14cac530f243bccae792f9337218c3ee6e0634.debug
2021.07.22 15:05:43.902972 [ 314318 ] {} <Fatal> BaseDaemon: 14. DB::BackgroundSchedulePool::threadFunction() @ 0x101340f7 in /usr/lib/debug/.build-id/6f/14cac530f243bccae792f9337218c3ee6e0634.debug
2021.07.22 15:05:43.903036 [ 314318 ] {} <Fatal> BaseDaemon: 15. void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<ThreadFromGlobalPool::ThreadFromGlobalPool<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, unsigned long, char const*)::$_1>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, unsigned long, char const*)::$_1&&)::'lambda'(), void ()> >(std::__1::__function::__policy_storage const*) @ 0x10134e74 in /usr/lib/debug/.build-id/6f/14cac530f243bccae792f9337218c3ee6e0634.debug
2021.07.22 15:05:43.903082 [ 314318 ] {} <Fatal> BaseDaemon: 16. ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0x90a833f in /usr/lib/debug/.build-id/6f/14cac530f243bccae792f9337218c3ee6e0634.debug
2021.07.22 15:05:43.903123 [ 314318 ] {} <Fatal> BaseDaemon: 17. void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()> >(void*) @ 0x90abc23 in /usr/lib/debug/.build-id/6f/14cac530f243bccae792f9337218c3ee6e0634.debug
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
fuzzProblem found by one of the fuzzersProblem found by one of the fuzzers