Repository navigation
Description
Search before asking
- I had searched in the issues and found no similar issues.
Version
版本1.2.1,下载的官方安装包
What's Wrong?
3fe-5be集群5个be节点全部崩溃掉,重新启动5个be节点始终会有其中2个节点不能启动,be.out日志如下:
start time: 2023年 01月 09日 星期一 22:21:51 CST
F0113 22:17:07.946166 50147 column_vector.cpp:389] Size of filter doesn't match size of column. data size: 0, filter size: 1 @ 0x7f26967789ea doris::vectorized::ColumnVector<>::filter()
@ 0x7f2696700c6b doris::vectorized::ColumnNullable::filter()
@ 0x7f26967a7cbb doris::vectorized::Block::filter_block_internal()
@ 0x7f26967ac85c doris::vectorized::Block::filter_block()
@ 0x7f26967ac902 doris::vectorized::Block::filter_block()
@ 0x7f2697522e06 doris::vectorized::VExprContext::filter_block()
@ 0x7f269912f011 doris::vectorized::VScanner::get_block()
@ 0x7f269912c39f doris::vectorized::ScannerScheduler::_scanner_scan()
@ 0x7f269597f805 doris::ThreadPool::dispatch_thread()
@ 0x7f2695975bef doris::Thread::supervise_thread()
@ 0x7f268f900dc5 start_thread
@ 0x7f268fc0c73d __clone
@ (nil) (unknown)
*** Check failure stack trace: ***
@ 0x7f26995e8e0d google::LogMessage::Fail()
@ 0x7f26995eb349 google::LogMessage::SendToLog()
@ 0x7f26995e8976 google::LogMessage::Flush()
@ 0x7f26995eb9b9 google::LogMessageFatal::~LogMessageFatal()
@ 0x7f2696778a21 doris::vectorized::ColumnVector<>::filter()
@ 0x7f2696700c6b doris::vectorized::ColumnNullable::filter()
@ 0x7f26967a7cbb doris::vectorized::Block::filter_block_internal()
@ 0x7f26967ac85c doris::vectorized::Block::filter_block()
@ 0x7f26967ac902 doris::vectorized::Block::filter_block()
@ 0x7f2697522e06 doris::vectorized::VExprContext::filter_block()
@ 0x7f269912f011 doris::vectorized::VScanner::get_block()
@ 0x7f269912c39f doris::vectorized::ScannerScheduler::_scanner_scan()
@ 0x7f269597f805 doris::ThreadPool::dispatch_thread()
@ 0x7f2695975bef doris::Thread::supervise_thread()
@ 0x7f268f900dc5 start_thread
@ 0x7f268fc0c73d __clone
@ (nil) (unknown)
*** Query id: 4159b2473e6a4e06-a77d142ed2aad4b2 ***
*** Aborted at 1673619428 (unix time) try "date -d @1673619428" if you are using GNU date ***
*** Current BE git commitID: Unknown ***
*** SIGABRT unkown detail explain (@0xc376) received by PID 50038 (TID 0x7f2637df8700) from PID 50038; stack trace: ***
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/doris/be/src/common/signal_handler.h:420
1# 0x00007F268FB4A250 in /lib64/libc.so.6
2# raise in /lib64/libc.so.6
3# __GI_abort in /lib64/libc.so.6
4# 0x00007F26995F37F9 in /opt/apache-doris-be-1.2.1-bin-x86_64/lib/doris_be
5# 0x00007F26995E8E0D in /opt/apache-doris-be-1.2.1-bin-x86_64/lib/doris_be
6# google::LogMessage::SendToLog() in /opt/apache-doris-be-1.2.1-bin-x86_64/lib/doris_be
7# google::LogMessage::Flush() in /opt/apache-doris-be-1.2.1-bin-x86_64/lib/doris_be
8# google::LogMessageFatal::~LogMessageFatal() in /opt/apache-doris-be-1.2.1-bin-x86_64/lib/doris_be
9# doris::vectorized::ColumnVector::filter(doris::vectorized::PODArray<unsigned char, 4096ul, Allocator<false, false>, 15ul, 16ul> const&, long) const at /root/doris/be/src/vec/columns/column_vector.cpp:389
10# doris::vectorized::ColumnNullable::filter(doris::vectorized::PODArray<unsigned char, 4096ul, Allocator<false, false>, 15ul, 16ul> const&, long) const at /root/doris/be/src/vec/columns/column_nullable.cpp:309
11# doris::vectorized::Block::filter_block_internal(doris::vectorized::Block*, std::vector<unsigned int, std::allocator > const&, doris::vectorized::PODArray<unsigned char, 4096ul, Allocator<false, false>, 15ul, 16ul> const&) at /root/doris/be/src/vec/core/block.cpp:641
12# doris::vectorized::Block::filter_block(doris::vectorized::Block*, std::vector<unsigned int, std::allocator > const&, int, int) at /root/doris/be/src/vec/core/block.cpp:712
13# doris::vectorized::Block::filter_block(doris::vectorized::Block*, int, int) at /root/doris/be/src/vec/core/block.cpp:726
14# doris::vectorized::VExprContext::filter_block(doris::vectorized::VExprContext*, doris::vectorized::Block*, int) at /root/doris/be/src/vec/exprs/vexpr_context.cpp:122
15# doris::vectorized::VScanner::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /root/doris/be/src/vec/exec/scan/vscanner.cpp:64
16# doris::vectorized::ScannerScheduler::_scanner_scan(doris::vectorized::ScannerScheduler*, doris::vectorized::ScannerContext*, doris::vectorized::VScanner*) at /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:234
17# doris::ThreadPool::dispatch_thread() at /root/doris/be/src/util/threadpool.cpp:542
18# doris::Thread::supervise_thread(void*) at /root/doris/be/src/util/thread.cpp:455
19# start_thread in /lib64/libpthread.so.0
20# clone in /lib64/libc.so.6
What You Expected?
希望开发组能尽快定位问题
How to Reproduce?
No response
Anything Else?
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
xiongzegang commentedon Jan 14, 2023
尝试把集群降级到1.2版本,集群依然会有2个be节点会自动退出be.out日志:
start time: 2023年 01月 14日 星期六 11:07:27 CST
F0114 11:11:42.756687 793 column_impl.h:67] Size of selector: 1, is larger than size of column:0
*** Check failure stack trace: ***
@ 0x7f5a146e3d4d google::LogMessage::Fail()
@ 0x7f5a146e6289 google::LogMessage::SendToLog()
@ 0x7f5a146e38b6 google::LogMessage::Flush()
@ 0x7f5a146e68f9 google::LogMessageFatal::~LogMessageFatal()
@ 0x7f5a11301f8a doris::vectorized::ColumnVector<>::append_data_by_selector()
@ 0x7f5a118dce05 doris::vectorized::Block::append_block_by_selector()
@ 0x7f5a14081c9b doris::stream_load::VNodeChannel::add_block()
@ 0x7f5a14087bb0 doris::stream_load::VOlapTableSink::send()
@ 0x7f5a1089a2b9 doris::PlanFragmentExecutor::open_vectorized_internal()
@ 0x7f5a1089ba25 doris::PlanFragmentExecutor::open()
@ 0x7f5a10873dbc doris::FragmentExecState::execute()
@ 0x7f5a108771eb doris::FragmentMgr::_exec_actual()
@ 0x7f5a1087788a _ZNSt17_Function_handlerIFvvEZN5doris11FragmentMgr18exec_plan_fragmentERKNS1_23TExecPlanFragmentParamsESt8functionIFvPNS1_20PlanFragmentExecutorEEEEUlvE_E9_M_invokeERKSt9_Any_data
@ 0x7f5a10b22815 doris::ThreadPool::dispatch_thread()
@ 0x7f5a10b18bff doris::Thread::supervise_thread()
@ 0x7f5a0ac0ddc5 start_thread
@ 0x7f5a0af1973d __clone
@ (nil) (unknown)
*** Query id: 388e20e98c4642aa-bd5f30fd9aabf906 ***
*** Aborted at 1673665903 (unix time) try "date -d @1673665903" if you are using GNU date ***
*** Current BE git commitID: Unknown ***
*** SIGABRT unkown detail explain (@0x216) received by PID 534 (TID 0x7f59795ec700) from PID 534; stack trace: ***
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/doris/be/src/common/signal_handler.h:420
1# 0x00007F5A0AE57250 in /lib64/libc.so.6
2# raise in /lib64/libc.so.6
3# __GI_abort in /lib64/libc.so.6
4# 0x00007F5A146EE739 in /opt/apache-doris-be-1.2.0-bin-x86_64/lib/doris_be
5# 0x00007F5A146E3D4D in /opt/apache-doris-be-1.2.0-bin-x86_64/lib/doris_be
6# google::LogMessage::SendToLog() in /opt/apache-doris-be-1.2.0-bin-x86_64/lib/doris_be
7# google::LogMessage::Flush() in /opt/apache-doris-be-1.2.0-bin-x86_64/lib/doris_be
8# google::LogMessageFatal::~LogMessageFatal() in /opt/apache-doris-be-1.2.0-bin-x86_64/lib/doris_be
9# doris::vectorized::ColumnVector::append_data_by_selector(COWdoris::vectorized::IColumn::mutable_ptrdoris::vectorized::IColumn&, doris::vectorized::PODArray<unsigned long, 4096ul, Allocator<false, false>, 15ul, 16ul> const&) const at /root/doris/be/src/vec/columns/column_vector.h:355
10# doris::vectorized::Block::append_block_by_selector(std::vector<COWdoris::vectorized::IColumn::mutable_ptrdoris::vectorized::IColumn, std::allocator<COWdoris::vectorized::IColumn::mutable_ptrdoris::vectorized::IColumn > >&, doris::vectorized::PODArray<unsigned long, 4096ul, Allocator<false, false>, 15ul, 16ul> const&) const at /root/doris/be/src/vec/core/block.cpp:658
11# doris::stream_load::VNodeChannel::add_block(doris::vectorized::Block*, std::pair<std::unique_ptr<doris::vectorized::PODArray<unsigned long, 4096ul, Allocator<false, false>, 15ul, 16ul>, std::default_delete<doris::vectorized::PODArray<unsigned long, 4096ul, Allocator<false, false>, 15ul, 16ul> > >, std::vector<long, std::allocator > > const&) at /root/doris/be/src/vec/sink/vtablet_sink.cpp:202
12# doris::stream_load::VOlapTableSink::send(doris::RuntimeState*, doris::vectorized::Block*) at /root/doris/be/src/vec/sink/vtablet_sink.cpp:595
13# doris::PlanFragmentExecutor::open_vectorized_internal() at /root/doris/be/src/runtime/plan_fragment_executor.cpp:309
14# doris::PlanFragmentExecutor::open() at /root/doris/be/src/runtime/plan_fragment_executor.cpp:250
15# doris::FragmentExecState::execute() at /root/doris/be/src/runtime/fragment_mgr.cpp:251
16# doris::FragmentMgr::_exec_actual(std::shared_ptrdoris::FragmentExecState, std::function<void (doris::PlanFragmentExecutor*)>) at /root/doris/be/src/runtime/fragment_mgr.cpp:498
17# std::_Function_handler<void (), doris::FragmentMgr::exec_plan_fragment(doris::TExecPlanFragmentParams const&, std::function<void (doris::PlanFragmentExecutor*)>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) at /var/local/ldb-toolchain/include/c++/11/bits/std_function.h:291
18# doris::ThreadPool::dispatch_thread() at /root/doris/be/src/util/threadpool.cpp:542
19# doris::Thread::supervise_thread(void*) at /root/doris/be/src/util/thread.cpp:455
20# start_thread in /lib64/libpthread.so.0
21# clone in /lib64/libc.so.6
LemonLiTree commentedon Jan 17, 2023
什么操作导致be奔掉的?
xiongzegang commentedon Jan 17, 2023
经排查可能是因为datetimev2和datev2导致的,之前表是使用datetime和date。之前使用datetime集群正常跑业务,后面改成datetimev2后,就出现这样问题。后面又把datetimev2改回datetime了,集群稳定跑了3天了。数据导入主要使用routine load和insert into xxx select * from xxx。
liuchunhua commentedon Jul 14, 2023
version: 1.2.4
insert into innertable select * from hive
lide-reed commentedon Jul 28, 2023
Program terminated with signal 11, Segmentation fault.
#0 0x00007f35379f6b66 in __memmove_ssse3_back () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install glibc-2.17-324.el7_9.x86_64 libgcc-4.8.5-44.el7.x86_64
(gdb) bt
#0 0x00007f35379f6b66 in __memmove_ssse3_back () from /lib64/libc.so.6
#1 0x00005622fb2b4670 in _mm_storeu_si128 (__B=..., _P=) at /data/doris-1.x/be/src/vec/common/memcpy_small.h:72
#2 memcpy_small_allow_read_write_overflow15_impl (n=, src=, dst=) at /data/doris-1.x/be/src/vec/common/memcpy_small.h:57
#3 memcpy_small_allow_read_write_overflow15 (n=, src=, dst=) at /data/doris-1.x/be/src/vec/common/memcpy_small.h:72
#4 doris::vectorized::ColumnString::insert_from (this=this@entry=0x7f2f78803d20, src=..., n=) at /data/doris-1.x/be/src/vec/columns/column_string.h:153
#5 0x00005622fb2aa15b in doris::vectorized::ColumnString::insert_indices_from (this=0x7f2f78803d20, src=..., indices_begin=, indices_end=0x7f33f8aac070)
at /data/doris-1.x/be/src/vec/columns/column_string.cpp:110
#6 0x00005622fb343edd in doris::vectorized::MutableBlock::add_rows (this=this@entry=0x7f33cbdb6d00, block=block@entry=0x7f3149ecd0b0, row_begin=0x7f33f8aac060, row_end=0x7f33f8aac070)
at /data/doris-1.x/be/src/vec/common/cow.h:208
#7 0x00005622f9d4c40c in doris::MemTable::insert (this=0x7f33cbdb6b00, input_block=input_block@entry=0x7f3149ecd340, row_idxs=...) at /var/local/ldb-toolchain/include/c++/11/bits/stl_vector.h:1172
#8 0x00005622f9e32bd6 in doris::DeltaWriter::write (this=0x7f337357ac00, block=block@entry=0x7f3149ecd340, row_idxs=...) at /var/local/ldb-toolchain/include/c++/11/bits/unique_ptr.h:421
#9 0x00005622fa345582 in doris::TabletsChannel::add_batch<doris::PTabletWriterAddBlockRequest, doris::PTabletWriterAddBlockResult> (this=0x7f30c8840c00, request=..., response=response@entry=0x7f306aa944a0)
at /var/local/ldb-toolchain/include/c++/11/bits/hashtable_policy.h:337
#10 0x00005622fa415c46 in doris::LoadChannel::add_batch<doris::PTabletWriterAddBlockRequest, doris::PTabletWriterAddBlockResult> (this=0x7f30c83c5e00, request=..., response=response@entry=0x7f306aa944a0)
at /var/local/ldb-toolchain/include/c++/11/bits/shared_ptr_base.h:1290
#11 0x00005622fa415f93 in doris::LoadChannelMgr::add_batch<doris::PTabletWriterAddBlockRequest, doris::PTabletWriterAddBlockResult> (this=0x7f3518befd00, request=..., response=0x7f306aa944a0)
at /var/local/ldb-toolchain/include/c++/11/bits/shared_ptr_base.h:1290
#12 0x00005622fa406aa0 in operator() (__closure=0x7f303fa1d920) at /data/doris-1.x/be/src/runtime/exec_env.h:160
#13 __invoke_impl<void, doris::PInternalServiceImpl::_tablet_writer_add_block(google::protobuf::RpcController*, const doris::PTabletWriterAddBlockRequest*, doris::PTabletWriterAddBlockResult*, google::protobuf::Closure*)::<lambda()>&> (__f=...) at /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:61
#14 __invoke_r<void, doris::PInternalServiceImpl::_tablet_writer_add_block(google::protobuf::RpcController*, const doris::PTabletWriterAddBlockRequest*, doris::PTabletWriterAddBlockResult*, google::protobuf::Closure*)::<lambda()>&> (__fn=...) at /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:111
#15 std::_Function_handler<void(), doris::PInternalServiceImpl::_tablet_writer_add_block(google::protobuf::RpcController*, const doris::PTabletWriterAddBlockRequest*, doris::PTabletWriterAddBlockResult*, google::protobuf::Closure*)::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /var/local/ldb-toolchain/include/c++/11/bits/std_function.h:291
#16 0x00005622fa21e265 in operator() (this=0x7f3149ecd888) at /var/local/ldb-toolchain/include/c++/11/bits/std_function.h:556
#17 doris::PriorityThreadPool::work_thread (this=0x7f35361b9d90, thread_id=) at /data/doris-1.x/be/src/util/priority_thread_pool.hpp:138
#18 0x0000562301ec9370 in execute_native_thread_routine ()
#19 0x00007f3537686ea5 in start_thread () from /lib64/libpthread.so.0
#20 0x00007f35379999fd in clone () from /lib64/libc.so.6
zhaodageng commentedon Sep 27, 2023
haoshuai5358 commentedon Nov 9, 2023
我也遇到了相似的错误堆栈,1.2.6
*** Query id: 798ba6186f174202-a6cbf439b8193a86 ***
*** Aborted at 1689148586 (unix time) try "date -d @1689148586" if you are using GNU date ***
*** Current BE git commitID: Unknown ***
*** SIGSEGV address not mapped to object (@0x7f861d804000) received by PID 32341 (TID 0x7f7e9c91f700) from PID 494944256; stack trace: ***
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/doris/be/src/common/signal_handler.h:420
1# os::Linux::chained_handler(int, siginfo_t*, void*) in /usr/lib/jvm/java-1.8.0/jre/lib/amd64/server/libjvm.so
2# JVM_handle_linux_signal in /usr/lib/jvm/java-1.8.0/jre/lib/amd64/server/libjvm.so
3# signalHandler(int, siginfo_t*, void*) in /usr/lib/jvm/java-1.8.0/jre/lib/amd64/server/libjvm.so
4# 0x00007F88992B7400 in /lib64/libc.so.6
5# __memmove_ssse3_back in /lib64/libc.so.6
6# doris::vectorized::ColumnString::insert_from(doris::vectorized::IColumn const&, unsigned long) at /root/doris/be/src/vec/columns/column_string.h:156
7# doris::vectorized::ColumnNullable::append_data_by_selector(COWdoris::vectorized::IColumn::mutable_ptrdoris::vectorized::IColumn&, doris::vectorized::PODArray<unsigned long, 4096ul, Allocator<false, false>, 15ul, 16ul> const&) const at /root/doris/be/src/vec/columns/column_nullable.h:197
zhaodageng commentedon Nov 9, 2023
zhaodageng commentedon Sep 15, 2025