Repository navigation
merge - #1
Merged
Merged
merge#1
Conversation
wyxxxcat
pushed a commit
that referenced
this pull request
Jul 23, 2024
## Proposed changes Currently, Arrow use separate Jemalloc, and use non-standard methods `mallocx`, `sdallocx`, `rallocx` in `memory_pool_jemalloc.cc` to optimize memory allocation. But this may be incompatible with older versions of the Linux kernel. when we use Arrow on Arm Kirin v10 or Centos 7.4, it will get stuck on the Jemalloc Lock, with the stack below, it will appear when calling `arrow::RecordBatch::MakeEmpty`. the kernel version of Arm Kylin v10 is 4.19.90, and the kernel version of Centos 7.4 is 4.14. ``` #0 0x0000ffffae3ceff8 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x0000ffffae3c9b50 in pthread_mutex_lock () from /lib64/libpthread.so.0 #2 0x0000ffffae61c834 in pthread_mutex_lock () from /lib64/libc.so.6 #3 0x0000aaaac99bc1e0 in je_arrow_private_je_malloc_mutex_lock_slow () #4 0x0000aaaac99af3a4 in ?? () apache#5 0x0000aaaac99b576c in je_arrow_mallocx () apache#6 0x0000aaaac99a8aec in ?? () apache#7 0x0000aaaac99a9858 in arrow::AllocateResizableBuffer(long, arrow::MemoryPool*) () apache#8 0x0000aaaac399f8b8 in arrow::BufferBuilder::Resize(long, bool) () apache#9 0x0000aaaac983715c in arrow::BaseBinaryBuilder<arrow::BinaryType>::Resize(long) () apache#10 0x0000aaaac39a47e0 in arrow::BaseBinaryBuilder<arrow::BinaryType>::Append(unsigned char const*, int) () ``` After disable separate Jemalloc when compiling Arrow, the above error disappeared, and Arrow will use the default memory allocator, which is Doris Jemalloc.
wyxxxcat
pushed a commit
that referenced
this pull request
Jul 23, 2024
## Proposed changes
- Null pointer of type 'doris::StringRef' in orc reader. The root cause
is error will throw when `num_values == 0` in
`_decode_string_non_dict_encoded_column`.
```
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1046:9: runtime error: reference binding to null pointer of type 'doris::StringRef'
#0 0x562516fa9770 in std::vector<doris::StringRef, std::allocator<doris::StringRef> >::operator[](unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1046:2
#1 0x562516fa9770 in doris::Status doris::vectorized::OrcReader::_decode_string_non_dict_encoded_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn> const&, orc::TypeKind const&, orc::EncodedStringVectorBatch*, unsigned long) /root/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1164:39
#2 0x562516f9c08b in doris::Status doris::vectorized::OrcReader::_decode_string_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn> const&, orc::TypeKind const&, orc::ColumnVectorBatch*, unsigned long) /root/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1116:16
#3 0x562516f91d73 in doris::Status doris::vectorized::OrcReader::_fill_doris_data_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>&, std::shared_ptr<doris::vectorized::IDataType const> const&, orc::Type const*, orc::ColumnVectorBatch*, unsigned long) /root/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1357:16
#4 0x562516c79a0c in doris::Status doris::vectorized::OrcReader::_orc_column_to_doris_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, std::shared_ptr<doris::vectorized::IDataType const> const&, orc::Type const*, orc::ColumnVectorBatch*, unsigned long) /root/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1524:5
apache#5 0x562516f9339a in doris::Status doris::vectorized::OrcReader::_fill_doris_data_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>&, std::shared_ptr<doris::vectorized::IDataType const> const&, orc::Type const*, orc::ColumnVectorBatch*, unsigned long) /root/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1402:9
apache#6 0x562516c79a0c in doris::Status doris::vectorized::OrcReader::_orc_column_to_doris_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, std::shared_ptr<doris::vectorized::IDataType const> const&, orc::Type const*, orc::ColumnVectorBatch*, unsigned long) /root/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1524:5
...
```
- Shift exponent 128 is too large for 128-bit type 'ValueCopyType' (aka
'__int128') in parquet reader. The root cause is error will throw when
`len == 0`.
```
/root/doris/be/src/vec/exec/format/parquet/parquet_column_convert.h:413:27: runtime error: shift exponent 128 is too large for 128-bit type 'ValueCopyType' (aka '__int128')
#0 0x56251760fbc7 in doris::vectorized::parquet::StringToDecimal<doris::vectorized::Decimal128V3, (doris::vectorized::DecimalScaleParams::ScaleType)1>::physical_convert(COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&) /root/doris/be/src/vec/exec/format/parquet/parquet_column_convert.h:413:27
#1 0x562517290dc4 in doris::vectorized::parquet::PhysicalToLogicalConverter::convert(COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, doris::TypeDescriptor, std::shared_ptr<doris::vectorized::IDataType const> const&, COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, bool) /root/doris/be/src/vec/exec/format/parquet/parquet_column_convert.h:209:9
#2 0x562517284a6d in doris::vectorized::ScalarColumnReader::read_column_data(COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, std::shared_ptr<doris::vectorized::IDataType const>&, doris::vectorized::ColumnSelectVector&, unsigned long, unsigned long*, bool*, bool) /root/doris/be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:569:24
#3 0x56251725ae7e in doris::vectorized::RowGroupReader::_read_column_data(doris::vectorized::Block*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, unsigned long, unsigned long*, bool*, doris::vectorized::ColumnSelectVector&) /root/doris/be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:421:13
#4 0x56251724d6d2 in doris::vectorized::RowGroupReader::next_batch(doris::vectorized::Block*, unsigned long, unsigned long*, bool*) /root/doris/be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:321:9
apache#5 0x56251708eb97 in doris::vectorized::ParquetReader::get_next_block(doris::vectorized::Block*, unsigned long*, bool*) /root/doris/be/src/vec/exec/format/parquet/vparquet_reader.cpp:530:36
apache#6 0x56253036772d in doris::vectorized::VFileScanner::_get_block_wrapped(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/be/src/vec/exec/scan/vfile_scanner.cpp:311:13
apache#7 0x562530366549 in doris::vectorized::VFileScanner::_get_block_impl(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/be/src/vec/exec/scan/vfile_scanner.cpp:253:17
apache#8 0x5625176e79c8 in doris::vectorized::VScanner::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/be/src/vec/exec/scan/vscanner.cpp:117:17
apache#9 0x5625176e6fc1 in doris::vectorized::VScanner::get_block_after_projects(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/be/src/vec/exec/scan/vscanner.cpp:84:12
apache#10 0x562517698047 in doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>) /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:250:5
apache#11 0x56251769bc1f in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:172:25
apache#12 0x56251769bc1f in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:171:35
...
```
wyxxxcat
pushed a commit
that referenced
this pull request
Aug 6, 2024
… StreamLoadContext when operate_txn_2pc (apache#38033) /root/doris/be/src/runtime/stream_load/stream_load_executor.cpp:384:14: runtime error: load of value 3200171710, which is not a valid value for type 'TLoadType::type' #0 0x55be4e2cf808 in doris::StreamLoadExecutor::collect_load_stat(doris::StreamLoadContext*, doris::TTxnCommitAttachment*) /root/doris/be/src/runtime/stream_load/stream_load_executor.cpp:384:14 #1 0x55be4e2d2666 in doris::StreamLoadExecutor::rollback_txn(doris::StreamLoadContext*) /root/doris/be/src/runtime/stream_load/stream_load_executor.cpp:364:9 #2 0x55be80413748 in doris::CloudStreamLoadExecutor::operate_txn_2pc(doris::StreamLoadContext*) /root/doris/be/src/cloud/cloud_stream_load_executor.cpp:80:33 #3 0x55be50cad893 in doris::StreamLoad2PCAction::handle(doris::HttpRequest*) /root/doris/be/src/http/action/stream_load_2pc.cpp:85:49
wyxxxcat
pushed a commit
that referenced
this pull request
Aug 6, 2024
…out range of _buf (apache#37936) if we select nested type such as map/array/struct after large string , when string type in mysql_row_buf reserve make buffer size is not large enough , which will lead nested type open_dynamic_mode make _pos pointer out range of mysql_row_buf, then nested type call push_string, and reserve() will make heap_buffer_overflow ``` ==200769==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x62d0051c12ba at pc 0x55a77788692d bp 0x7fb52f474a30 sp 0x7fb52f4741f8 READ of size 36541 at 0x62d0051c12ba thread T2309 (Pipe_normal [wo) #0 0x55a77788692c in __asan_memcpy (/mnt/disk1/wangqiannan/amory/doris/output/be/lib/doris_be+0x60c1c92c) (BuildId: 4513940b6b9e22fa) #1 0x55a7a1f622fa in doris::MysqlRowBuffer<false>::reserve(long) /mnt/disk1/wangqiannan/amory/doris/be/src/util/mysql_row_buffer.cpp:140:5 #2 0x55a7a1f638eb in doris::MysqlRowBuffer<false>::push_string(char const*, long) /mnt/disk1/wangqiannan/amory/doris/be/src/util/mysql_row_buffer.cpp:473:5 #3 0x55a7a21f16eb in doris::Status doris::vectorized::DataTypeMapSerDe::_write_column_to_mysql<false>(doris::vectorized::IColumn const&, doris::MysqlRowBuffer<false>&, int, bool, doris::vectorized::DataTypeSerDe::FormatOptions const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/data_types/serde/data_type_map_serde.cpp:410:21 #4 0x55a7a21e4c1e in doris::vectorized::DataTypeMapSerDe::write_column_to_mysql(doris::vectorized::IColumn const&, doris::MysqlRowBuffer<false>&, int, bool, doris::vectorized::DataTypeSerDe::FormatOptions const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/data_types/serde/data_type_map_serde.cpp:478:12 apache#5 0x55a7a22070e6 in doris::Status doris::vectorized::DataTypeNullableSerDe::_write_column_to_mysql<false>(doris::vectorized::IColumn const&, doris::MysqlRowBuffer<false>&, int, bool, doris::vectorized::DataTypeSerDe::FormatOptions const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/data_types/serde/data_type_nullable_serde.cpp:300:9 apache#6 0x55a7a21fbc5e in doris::vectorized::DataTypeNullableSerDe::write_column_to_mysql(doris::vectorized::IColumn const&, doris::MysqlRowBuffer<false>&, int, bool, doris::vectorized::DataTypeSerDe::FormatOptions const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/data_types/serde/data_type_nullable_serde.cpp:317:12 apache#7 0x55a7c2e97e6c in doris::vectorized::VMysqlResultWriter<false>::write(doris::RuntimeState*, doris::vectorized::Block&) /mnt/disk1/wangqiannan/amory/doris/be/src/vec/sink/vmysql_result_writer.cpp:216:17 apache#8 0x55a7c8031b83 in doris::pipeline::ResultSinkOperatorX::sink(doris::RuntimeState*, doris::vectorized::Block*, bool) /mnt/disk1/wangqiannan/amory/doris/be/src/pipeline/exec/result_sink_operator.cpp:142:5 apache#9 0x55a7c99a81d6 in doris::pipeline::PipelineTask::execute(bool*)::$_1::operator()() const /mnt/disk1/wangqiannan/amory/doris/be/src/pipeline/pipeline_task.cpp:361:38 apache#10 0x55a7c99a4b27 in doris::pipeline::PipelineTask::execute(bool*) /mnt/disk1/wangqiannan/amory/doris/be/src/pipeline/pipeline_task.cpp:364:22 apache#11 0x55a7c9a23a2b in doris::pipeline::TaskScheduler::_do_work(unsigned long) /mnt/disk1/wangqiannan/amory/doris/be/src/pipeline/task_scheduler.cpp:138:9 apache#12 0x55a7c9a269ca in doris::pipeline::TaskScheduler::start()::$_0::operator()() const /mnt/disk1/wangqiannan/amory/doris/be/src/pipeline/task_scheduler.cpp:64:9 apache#13 0x55a7c9a2694e in void std::__invoke_impl<void, doris::pipeline::TaskScheduler::start()::$_0&>(std::__invoke_other, doris::pipeline::TaskScheduler::start()::$_0&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 apache#14 0x55a7c9a268ae in std::enable_if<is_invocable_r_v<void, doris::pipeline::TaskScheduler::start()::$_0&>, void>::type std::__invoke_r<void, doris::pipeline::TaskScheduler::start()::$_0&>(doris::pipeline::TaskScheduler::start()::$_0&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 apache#15 0x55a7c9a26635 in std::_Function_handler<void (), doris::pipeline::TaskScheduler::start()::$_0>::_M_invoke(std::_Any_data const&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 apache#16 0x55a777b226da in std::function<void ()>::operator()() const /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9 apache#17 0x55a77e95ec94 in doris::FunctionRunnable::run() /mnt/disk1/wangqiannan/amory/doris/be/src/util/threadpool.cpp:48:27 apache#18 0x55a77e941015 in doris::ThreadPool::dispatch_thread() /mnt/disk1/wangqiannan/amory/doris/be/src/util/threadpool.cpp:543:24 apache#19 0x55a77e97eb23 in void std::__invoke_impl<void, void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(std::__invoke_memfun_deref, void (doris::ThreadPool::*&)(), doris::ThreadPool*&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74:14 apache#20 0x55a77e97e928 in std::__invoke_result<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>::type std::__invoke<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(void (doris::ThreadPool::*&)(), doris::ThreadPool*&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 apache#21 0x55a77e97e860 in void std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>::__call<void, 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:420:11 apache#22 0x55a77e97e655 in void std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>::operator()<void>() /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:503:17 apache#23 0x55a77e97e54e in void std::__invoke_impl<void, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&>(std::__invoke_other, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 apache#24 0x55a77e97e48e in std::enable_if<is_invocable_r_v<void, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&>, void>::type std::__invoke_r<void, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&>(std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 apache#25 0x55a77e97dd85 in std::_Function_handler<void (), std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>>::_M_invoke(std::_Any_data const&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 apache#26 0x55a777b226da in std::function<void ()>::operator()() const /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9 apache#27 0x55a77e8fb841 in doris::Thread::supervise_thread(void*) /mnt/disk1/wangqiannan/amory/doris/be/src/util/thread.cpp:498:5 apache#28 0x7fc1c3a111c9 in start_thread (/lib64/libpthread.so.0+0x81c9) (BuildId: 823fccea3475e5870a4167dfe47df20e53222db0) apache#29 0x7fc1c4400e72 in clone (/lib64/libc.so.6+0x39e72) (BuildId: ec3d7025354f1f1985831ff08ef0eb3b50aefbce) 0x62d0051c12ba is located 0 bytes after 36538-byte region [0x62d0051b8400,0x62d0051c12ba) allocated by thread T2309 (Pipe_normal [wo) here: #0 0x55a7778c20bd in operator new[](unsigned long) (/mnt/disk1/wangqiannan/amory/doris/output/be/lib/doris_be+0x60c580bd) (BuildId: 4513940b6b9e22fa) #1 0x55a7a1f621c1 in doris::MysqlRowBuffer<false>::reserve(long) /mnt/disk1/wangqiannan/amory/doris/be/src/util/mysql_row_buffer.cpp:137:21 #2 0x55a7a1f638eb in doris::MysqlRowBuffer<false>::push_string(char const*, long) /mnt/disk1/wangqiannan/amory/doris/be/src/util/mysql_row_buffer.cpp:473:5 #3 0x55a7a1fd0d75 in doris::Status doris::vectorized::DataTypeStringSerDeBase<doris::vectorized::ColumnStr<unsigned int>>::_write_column_to_mysql<false>(doris::vectorized::IColumn const&, doris::MysqlRowBuffer<false>&, int, bool, doris::vectorized::DataTypeSerDe::FormatOptions const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/data_types/serde/data_type_string_serde.h:260:16 #4 0x55a7a1fccc1e in doris::vectorized::DataTypeStringSerDeBase<doris::vectorized::ColumnStr<unsigned int>>::write_column_to_mysql(doris::vectorized::IColumn const&, doris::MysqlRowBuffer<false>&, int, bool, doris::vectorized::DataTypeSerDe::FormatOptions const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/data_types/serde/data_type_string_serde.h:215:16 apache#5 0x55a7c2e97e6c in doris::vectorized::VMysqlResultWriter<false>::write(doris::RuntimeState*, doris::vectorized::Block&) /mnt/disk1/wangqiannan/amory/doris/be/src/vec/sink/vmysql_result_writer.cpp:216:17 apache#6 0x55a7c8031b83 in doris::pipeline::ResultSinkOperatorX::sink(doris::RuntimeState*, doris::vectorized::Block*, bool) /mnt/disk1/wangqiannan/amory/doris/be/src/pipeline/exec/result_sink_operator.cpp:142:5 apache#7 0x55a7c99a81d6 in doris::pipeline::PipelineTask::execute(bool*)::$_1::operator()() const /mnt/disk1/wangqiannan/amory/doris/be/src/pipeline/pipeline_task.cpp:361:38 ``` ## Proposed changes Issue Number: close #xxx <!--Describe your changes.-->
wyxxxcat
pushed a commit
that referenced
this pull request
Aug 6, 2024
…on in cloud mode (apache#37293)" (apache#38828) We have to figure out why it causes a SEGV when running cloud_p0 later ``` #0 doris::cloud::TabletCompactionJobPB::_internal_input_versions(int) const /root/doris/cloud/../gensrc/build/gen_cpp/cloud.pb.h:48193:33 #1 doris::cloud::MetaServiceImpl::start_tablet_job(google::protobuf::RpcController*, doris::cloud::StartTabletJobRequest const*, doris::cloud::StartTabletJobResponse*, google::protobuf::Closure*) /root/doris/cloud/src/meta-service/meta_service_job.cpp:436:9 #2 void doris::cloud::MetaServiceProxy::call_impl<doris::cloud::StartTabletJobRequest, doris::cloud::StartTabletJobResponse>(void (doris::cloud::MetaService::*)(google::protobuf::RpcController*, doris::cloud::StartTabletJobRequest const*, doris::cloud::StartTabletJobResponse*, google::protobuf::Closure*), google::protobuf::RpcController*, doris::cloud::StartTabletJobRequest const*, doris::cloud::StartTabletJobResponse*, google::protobuf::Closure*) /root/doris/cloud/src/meta-service/meta_service.h:684:13 #3 doris::cloud::MetaServiceProxy::start_tablet_job(google::protobuf::RpcController*, doris::cloud::StartTabletJobRequest const*, doris::cloud::StartTabletJobResponse*, google::protobuf::Closure*) /root/doris/cloud/src/meta-service/meta_service.h:478:9 #4 doris::cloud::MetaService::CallMethod(google::protobuf::MethodDescriptor const*, google::protobuf::RpcController*, google::protobuf::Message const*, google::protobuf::Message*, google::protobuf::Closure*) /root/doris/gensrc/build/gen_cpp/cloud.pb.cc:0:7 ``` This PR also add some FE log
wyxxxcat
pushed a commit
that referenced
this pull request
Aug 13, 2024
…pache#38935) ``` 1 __pthread_kill_internal (signo=6, threadid=140314099000896) at ./nptl/pthread_kill.c:78 #2 __GI___pthread_kill (threadid=140314099000896, signo=signo@entry=6) at ./nptl/pthread_kill.c:89 #3 0x00007fa1e2e8e476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26 #4 0x00007fa1e2e747f3 in __GI_abort () at ./stdlib/abort.c:79 apache#5 0x000055f77d0e71ea in __gnu_cxx::__verbose_terminate_handler () at ../../../../libstdc++-v3/libsupc++/vterminate.cc:95 apache#6 0x000055f77d0e5956 in __cxxabiv1::__terminate (handler=<optimized out>) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48 apache#7 0x000055f77d0e59c1 in std::terminate () at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:58 apache#8 0x000055f76ffc5fbe in __clang_call_terminate () apache#9 0x000055f77667ef44 in doris::Defer<doris::vectorized::FunctionVariantElement::get_element_column(doris::vectorized::ColumnObject const&, COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn> const&, COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>*)::{lambda()#1}>::~Defer() (this=0x7f9d6c06a900) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/util/defer_op.h:37 ```
wyxxxcat
pushed a commit
that referenced
this pull request
Aug 13, 2024
## Proposed changes Undefined behavior occurs if there is a null value in the list. ``` /root/doris/be/src/vec/common/string_ref.h:271:54: runtime error: null pointer passed as argument 2, which is declared to never be null /var/local/ldb-toolchain/bin/../usr/include/string.h:64:33: note: nonnull attribute specified here #0 0x5616d072245d in doris::StringRef::eq(doris::StringRef const&) const /root/doris/be/src/vec/common/string_ref.h:271:41 #1 0x5616d072245d in doris::StringRef::operator==(doris::StringRef const&) const /root/doris/be/src/vec/common/string_ref.h:274:60 #2 0x5616d072245d in doris::FixedContainer::find(doris::StringRef const&) const /root/doris/be/src/exprs/hybrid_set.h:76:36 #3 0x5616d072245d in void doris::StringValueSet>::_find_batch(doris::vectorized::IColumn const&, unsigned long, doris::vectorized::PODArray, 16ul, 15ul> const*, doris::vectorized::PODArray, 16ul, 15ul>&) /root/doris/be/src/exprs/hybrid_set.h:688:63 #4 0x5616d0747857 in doris::vectorized::FunctionIn::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/in.h:170:21 apache#5 0x5616c741fa3a in doris::vectorized::DefaultExecutable::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/function.h:462:26 apache#6 0x5616cbb5b650 in doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp apache#7 0x5616cbb4e14e in doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:244:12 apache#8 0x5616cbb4e3c2 in doris::vectorized::PreparedFunctionImpl::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:250:12 apache#9 0x5616c741cd68 in doris::vectorized::IFunctionBase::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.h:190:19 apache#10 0x5616c74cf712 in doris::vectorized::VInPredicate::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vin_predicate.cpp:130:5 apache#11 0x5616c740d5c0 in doris::vectorized::VectorizedFnCall::_do_execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*, std::vector>&) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:183:9 apache#12 0x5616c740ecf5 in doris::vectorized::VectorizedFnCall::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:215:12 apache#13 0x5616c7462e24 in doris::vectorized::VCompoundPred::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vcompound_pred.h:127:38 apache#14 0x5616c74bccec in doris::vectorized::VExprContext::execute(doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vexpr_context.cpp:54:5 apache#15 0x5616c74c1dcc in doris::vectorized::VExprContext::execute_conjuncts(std::vector, std::allocator>> const&, std::vector, 16ul, 15ul>, std::allocator, 16ul, 15ul>>> const*, bool, doris::vectorized::Block*, doris::vectorized::PODArray, 16ul, 15ul>, bool) /root/doris/be/src/vec/exprs/vexpr_context.cpp:169:9 apache#16 0x5616c74c5108 in doris::vectorized::VExprContext::execute_conjuncts_and_filter_block(std::vector, std::allocator>> const&, doris::vectorized::Block*, std::vector>&, int, doris::vectorized::PODArray, 16ul, 15ul>&) /root/doris/be/src/vec/exprs/vexpr_context.cpp:322:5 apache#17 0x5616ad8a7f1a in doris::segment_v2::SegmentIterator::_execute_common_expr(unsigned short*, unsigned short&, doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2680:5 apache#18 0x5616ad89e86e in doris::segment_v2::SegmentIterator::_next_batch_internal(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2582:25 apache#19 0x5616ad892f5c in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*)::$_0::operator()() const /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2315:9 apache#20 0x5616ad892f5c in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2314:19 apache#21 0x5616ad6dd9cc in doris::segment_v2::LazyInitSegmentIterator::next_batch(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/lazy_init_segment_iterator.h:44:33 apache#22 0x5616ad269d67 in doris::BetaRowsetReader::next_block(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/beta_rowset_reader.cpp:380:29 apache#23 0x5616de6de110 in doris::vectorized::VCollectIterator::Level0Iterator::_refresh() /root/doris/be/src/vec/olap/vcollect_iterator.h apache#24 0x5616de6c967f in doris::vectorized::VCollectIterator::Level0Iterator::refresh_current_row() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:514:24 apache#25 0x5616de6ca8a6 in doris::vectorized::VCollectIterator::Level0Iterator::ensure_first_row_ref() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:493:14 apache#26 0x5616de6d7008 in doris::vectorized::VCollectIterator::Level1Iterator::ensure_first_row_ref() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:692:27 apache#27 0x5616de6bd200 in doris::vectorized::VCollectIterator::build_heap(std::vector, std::allocator>>&) /root/doris/be/src/vec/olap/vcollect_iterator.cpp:186:9 apache#28 0x5616de651b6c in doris::vectorized::BlockReader::_init_collect_iter(doris::TabletReader::ReaderParams const&) /root/doris/be/src/vec/olap/block_reader.cpp:157:5 apache#29 0x5616de65526f in doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams const&) /root/doris/be/src/vec/olap/block_reader.cpp:229:19 apache#30 0x5616e175a0f9 in doris::vectorized::NewOlapScanner::open(doris::RuntimeState*) /root/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:237:32 apache#31 0x5616c736ad34 in doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr, std::shared_ptr) /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:236:5 apache#32 0x5616c736f05e in doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:176:21 apache#33 0x5616c736f05e in doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:175:31 apache#34 0x5616c736f05e in void std::_invoke_impl, std::shared_ptr)::$_1::operator()() const::'lambda'()&>(std::_invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 apache#35 0x5616c736f05e in std::enable_if, std::shared_ptr)::$1::operator()() const::'lambda'()&>, void>::type std::_invoke_r, std::shared_ptr)::$_1::operator()() const::'lambda'()&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 apache#36 0x5616c736f05e in std::_Function_handler, std::shared_ptr)::$_1::operator()() const::'lambda'()>::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 apache#37 0x5616aeed6a3b in doris::ThreadPool::dispatch_thread() /root/doris/be/src/util/threadpool.cpp:543:24 apache#38 0x5616aeeae4f7 in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:498:5 apache#39 0x7f7e663e3ac2 in start_thread nptl/pthread_create.c:442:8 apache#40 0x7f7e6647584f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /root/doris/be/src/vec/common/string_ref.h:271:54 in ```
wyxxxcat
pushed a commit
that referenced
this pull request
Aug 13, 2024
…he#39237) ## Proposed changes which led to accessing uninitialized data. Moved the profile initialization into the init method. ``` /mnt/disk2/yanxuecheng/doris/be/src/pipeline/local_exchange/local_exchanger.cpp:100:9: runtime error: member call on null pointer of type 'doris::RuntimeProfile::Counter' #0 0x563f0fb8b6c9 in doris::pipeline::Exchanger<std::shared_ptr<doris::pipeline::BlockWrapper>>::_dequeue_data(doris::pipeline::LocalExchangeSourceLocalState&, std::shared_ptr<doris::pipeline::BlockWrapper>&, bool*, doris::vectorized::Block*, int) /mnt/disk2/yanxuecheng/doris/be/src/pipeline/local_exchange/local_exchanger.cpp:100:9 #1 0x563f0fb78a45 in doris::pipeline::LocalMergeSortExchanger::finalize(doris::pipeline::LocalExchangeSourceLocalState&) /mnt/disk2/yanxuecheng/doris/be/src/pipeline/local_exchange/local_exchanger.cpp:325:16 #2 0x563f09610366 in doris::pipeline::LocalExchangeSharedState::sub_running_source_operators(doris::pipeline::LocalExchangeSourceLocalState&) /mnt/disk2/yanxuecheng/doris/be/src/pipeline/dependency.cpp:196:20 #3 0x563f0fb63709 in doris::pipeline::LocalExchangeSourceLocalState::close(doris::RuntimeState*) /mnt/disk2/yanxuecheng/doris/be/src/pipeline/local_exchange/local_exchange_source_operator.cpp:59:24 #4 0x563f09728e80 in doris::pipeline::OperatorXBase::close(doris::RuntimeState*) /mnt/disk2/yanxuecheng/doris/be/src/pipeline/exec/operator.cpp:245:28 apache#5 0x563f0fd3fb58 in doris::pipeline::PipelineTask::close(doris::Status) /mnt/disk2/yanxuecheng/doris/be/src/pipeline/pipeline_task.cpp:459:28 apache#6 0x563f0fdb5315 in doris::pipeline::_close_task(doris::pipeline::PipelineTask*, doris::Status) /mnt/disk2/yanxuecheng/doris/be/src/pipeline/task_scheduler.cpp:91:27 apache#7 0x563f0fdb6573 in doris::pipeline::TaskScheduler::_do_work(unsigned long) /mnt/disk2/yanxuecheng/doris/be/src/pipeline/task_scheduler.cpp:125:13 apache#8 0x563f0fdb9d6a in doris::pipeline::TaskScheduler::start()::$_0::operator()() const /mnt/disk2/yanxuecheng/doris/be/src/pipeline/task_scheduler.cpp:64:9 apache#9 0x563f0fdb9cee in void std::__invoke_impl<void, doris::pipeline::TaskScheduler::start()::$_0&>(std::__invoke_other, doris::pipeline::TaskScheduler::start()::$_0&) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14 apache#10 0x563f0fdb9c4e in std::enable_if<is_invocable_r_v<void, doris::pipeline::TaskScheduler::start()::$_0&>, void>::type std::__invoke_r<void, doris::pipeline::TaskScheduler::start()::$_0&>(doris::pipeline::TaskScheduler::start()::$_0&) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:111:2 apache#11 0x563f0fdb99d5 in std::_Function_handler<void (), doris::pipeline::TaskScheduler::start()::$_0>::_M_invoke(std::_Any_data const&) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290:9 apache#12 0x563ebdddc8cf in std::function<void ()>::operator()() const /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591:9 apache#13 0x563ec4bd6db4 in doris::FunctionRunnable::run() /mnt/disk2/yanxuecheng/doris/be/src/util/threadpool.cpp:48:27 apache#14 0x563ec4bbc1b5 in doris::ThreadPool::dispatch_thread() /mnt/disk2/yanxuecheng/doris/be/src/util/threadpool.cpp:543:24 apache#15 0x563ec4bf9a53 in void std::__invoke_impl<void, void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(std::__invoke_memfun_deref, void (doris::ThreadPool::*&)(), doris::ThreadPool*&) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:74:14 apache#16 0x563ec4bf9858 in std::__invoke_result<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>::type std::__invoke<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(void (doris::ThreadPool::*&)(), doris::ThreadPool*&) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:96:14 apache#17 0x563ec4bf9790 in void std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>::__call<void, 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/functional:506:11 apache#18 0x563ec4bf9585 in void std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>::operator()<void>() /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/functional:591:17 apache#19 0x563ec4bf947e in void std::__invoke_impl<void, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&>(std::__invoke_other, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14 apache#20 0x563ec4bf93be in std::enable_if<is_invocable_r_v<void, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&>, void>::type std::__invoke_r<void, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&>(std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:111:2 apache#21 0x563ec4bf8e55 in std::_Function_handler<void (), std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>>::_M_invoke(std::_Any_data const&) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290:9 apache#22 0x563ebdddc8cf in std::function<void ()>::operator()() const /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591:9 apache#23 0x563ec4b78d91 in doris::Thread::supervise_thread(void*) /mnt/disk2/yanxuecheng/doris/be/src/util/thread.cpp:498:5 apache#24 0x563ebdb2fe0a in asan_thread_start(void*) crtstuff.c apache#25 0x7feeac9e21c9 in start_thread (/lib64/libpthread.so.0+0x81c9) (BuildId: 823fccea3475e5870a4167dfe47df20e53222db0) apache#26 0x7feead3d1e72 in clone (/lib64/libc.so.6+0x39e72) (BuildId: ec3d7025354f1f1985831ff08ef0eb3b50aefbce) SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /mnt/disk2/yanxuecheng/doris/be/src/pipeline/local_exchange/local_exchanger.cpp:100:9 in *** Query id: ea174401bc134452-bd8a35522726a96a *** *** is nereids: 1 *** *** tablet id: 0 *** *** Aborted at 1723455006 (unix time) try "date -d @1723455006" if you are using GNU date *** *** Current BE git commitID: c47399c *** *** SIGSEGV address not mapped to object (@0x0) received by PID 3055435 (TID 3060341 OR 0x7fe320acf700) from PID 0; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /mnt/disk2/yanxuecheng/doris/be/src/common/signal_handler.h:421 1# 0x00007FEEAD3E6B50 in /lib64/libc.so.6 2# doris::pipeline::Exchanger<std::shared_ptr<doris::pipeline::BlockWrapper> >::_dequeue_data(doris::pipeline::LocalExchangeSourceLocalState&, std::shared_ptr<doris::pipeline::BlockWrapper>&, bool*, doris::vectorized::Block*, int) at /mnt/disk2/yanxuecheng/doris/be/src/pipeline/local_exchange/local_exchanger.cpp:100 3# doris::pipeline::LocalMergeSortExchanger::finalize(doris::pipeline::LocalExchangeSourceLocalState&) at /mnt/disk2/yanxuecheng/doris/be/src/pipeline/local_exchange/local_exchanger.cpp:325 4# doris::pipeline::LocalExchangeSharedState::sub_running_source_operators(doris::pipeline::LocalExchangeSourceLocalState&) at /mnt/disk2/yanxuecheng/doris/be/src/pipeline/dependency.cpp:196 5# doris::pipeline::LocalExchangeSourceLocalState::close(doris::RuntimeState*) in /mnt/disk2/yanxuecheng/doris/output/be/lib/doris_be 6# doris::pipeline::OperatorXBase::close(doris::RuntimeState*) at /mnt/disk2/yanxuecheng/doris/be/src/pipeline/exec/operator.cpp:245 7# doris::pipeline::PipelineTask::close(doris::Status) at /mnt/disk2/yanxuecheng/doris/be/src/pipeline/pipeline_task.cpp:459 8# doris::pipeline::_close_task(doris::pipeline::PipelineTask*, doris::Status) at /mnt/disk2/yanxuecheng/doris/be/src/pipeline/task_scheduler.cpp:91 9# doris::pipeline::TaskScheduler::_do_work(unsigned long) at /mnt/disk2/yanxuecheng/doris/be/src/pipeline/task_scheduler.cpp:125 10# doris::pipeline::TaskScheduler::start()::$_0::operator()() const at /mnt/disk2/yanxuecheng/doris/be/src/pipeline/task_scheduler.cpp:64 11# void std::__invoke_impl<void, doris::pipeline::TaskScheduler::start()::$_0&>(std::__invoke_other, doris::pipeline::TaskScheduler::start()::$_0&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61 12# std::enable_if<is_invocable_r_v<void, doris::pipeline::TaskScheduler::start()::$_0&>, void>::type std::__invoke_r<void, doris::pipeline::TaskScheduler::start()::$_0&>(doris::pipeline::TaskScheduler::start()::$_0&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:117 13# std::_Function_handler<void (), doris::pipeline::TaskScheduler::start()::$_0>::_M_invoke(std::_Any_data const&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290 14# std::function<void ()>::operator()() const at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591 15# doris::FunctionRunnable::run() at /mnt/disk2/yanxuecheng/doris/be/src/util/threadpool.cpp:48 16# doris::ThreadPool::dispatch_thread() at /mnt/disk2/yanxuecheng/doris/be/src/util/threadpool.cpp:543 17# void std::__invoke_impl<void, void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(std::__invoke_memfun_deref, void (doris::ThreadPool::*&)(), doris::ThreadPool*&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:74 18# std::__invoke_result<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>::type std::__invoke<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(void (doris::ThreadPool::*&)(), doris::ThreadPool*&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:96 19# void std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/functional:506 20# void std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>::operator()<, void>() at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/functional:591 21# void std::__invoke_impl<void, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&>(std::__invoke_other, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61 22# std::enable_if<is_invocable_r_v<void, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&>, void>::type std::__invoke_r<void, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&>(std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:117 23# std::_Function_handler<void (), std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()> >::_M_invoke(std::_Any_data const&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290 24# std::function<void ()>::operator()() const at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591 25# doris::Thread::supervise_thread(void*) at /mnt/disk2/yanxuecheng/doris/be/src/util/thread.cpp:498 26# asan_thread_start(void*) in /mnt/disk2/yanxuecheng/doris/output/be/lib/doris_be 27# start_thread in /lib64/libpthread.so.0 28# __clone in /lib64/libc.so.6 ``` <!--Describe your changes.-->
wyxxxcat
pushed a commit
that referenced
this pull request
Aug 14, 2024
…pache#38935) ``` 1 __pthread_kill_internal (signo=6, threadid=140314099000896) at ./nptl/pthread_kill.c:78 #2 __GI___pthread_kill (threadid=140314099000896, signo=signo@entry=6) at ./nptl/pthread_kill.c:89 #3 0x00007fa1e2e8e476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26 #4 0x00007fa1e2e747f3 in __GI_abort () at ./stdlib/abort.c:79 apache#5 0x000055f77d0e71ea in __gnu_cxx::__verbose_terminate_handler () at ../../../../libstdc++-v3/libsupc++/vterminate.cc:95 apache#6 0x000055f77d0e5956 in __cxxabiv1::__terminate (handler=<optimized out>) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48 apache#7 0x000055f77d0e59c1 in std::terminate () at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:58 apache#8 0x000055f76ffc5fbe in __clang_call_terminate () apache#9 0x000055f77667ef44 in doris::Defer<doris::vectorized::FunctionVariantElement::get_element_column(doris::vectorized::ColumnObject const&, COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn> const&, COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>*)::{lambda()#1}>::~Defer() (this=0x7f9d6c06a900) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/util/defer_op.h:37 ```
wyxxxcat
pushed a commit
that referenced
this pull request
Aug 14, 2024
## Proposed changes Undefined behavior occurs if there is a null value in the list. ``` /root/doris/be/src/vec/common/string_ref.h:271:54: runtime error: null pointer passed as argument 2, which is declared to never be null /var/local/ldb-toolchain/bin/../usr/include/string.h:64:33: note: nonnull attribute specified here #0 0x5616d072245d in doris::StringRef::eq(doris::StringRef const&) const /root/doris/be/src/vec/common/string_ref.h:271:41 #1 0x5616d072245d in doris::StringRef::operator==(doris::StringRef const&) const /root/doris/be/src/vec/common/string_ref.h:274:60 #2 0x5616d072245d in doris::FixedContainer::find(doris::StringRef const&) const /root/doris/be/src/exprs/hybrid_set.h:76:36 #3 0x5616d072245d in void doris::StringValueSet>::_find_batch(doris::vectorized::IColumn const&, unsigned long, doris::vectorized::PODArray, 16ul, 15ul> const*, doris::vectorized::PODArray, 16ul, 15ul>&) /root/doris/be/src/exprs/hybrid_set.h:688:63 #4 0x5616d0747857 in doris::vectorized::FunctionIn::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/in.h:170:21 apache#5 0x5616c741fa3a in doris::vectorized::DefaultExecutable::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/function.h:462:26 apache#6 0x5616cbb5b650 in doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp apache#7 0x5616cbb4e14e in doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:244:12 apache#8 0x5616cbb4e3c2 in doris::vectorized::PreparedFunctionImpl::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:250:12 apache#9 0x5616c741cd68 in doris::vectorized::IFunctionBase::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.h:190:19 apache#10 0x5616c74cf712 in doris::vectorized::VInPredicate::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vin_predicate.cpp:130:5 apache#11 0x5616c740d5c0 in doris::vectorized::VectorizedFnCall::_do_execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*, std::vector>&) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:183:9 apache#12 0x5616c740ecf5 in doris::vectorized::VectorizedFnCall::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:215:12 apache#13 0x5616c7462e24 in doris::vectorized::VCompoundPred::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vcompound_pred.h:127:38 apache#14 0x5616c74bccec in doris::vectorized::VExprContext::execute(doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vexpr_context.cpp:54:5 apache#15 0x5616c74c1dcc in doris::vectorized::VExprContext::execute_conjuncts(std::vector, std::allocator>> const&, std::vector, 16ul, 15ul>, std::allocator, 16ul, 15ul>>> const*, bool, doris::vectorized::Block*, doris::vectorized::PODArray, 16ul, 15ul>, bool) /root/doris/be/src/vec/exprs/vexpr_context.cpp:169:9 apache#16 0x5616c74c5108 in doris::vectorized::VExprContext::execute_conjuncts_and_filter_block(std::vector, std::allocator>> const&, doris::vectorized::Block*, std::vector>&, int, doris::vectorized::PODArray, 16ul, 15ul>&) /root/doris/be/src/vec/exprs/vexpr_context.cpp:322:5 apache#17 0x5616ad8a7f1a in doris::segment_v2::SegmentIterator::_execute_common_expr(unsigned short*, unsigned short&, doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2680:5 apache#18 0x5616ad89e86e in doris::segment_v2::SegmentIterator::_next_batch_internal(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2582:25 apache#19 0x5616ad892f5c in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*)::$_0::operator()() const /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2315:9 apache#20 0x5616ad892f5c in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2314:19 apache#21 0x5616ad6dd9cc in doris::segment_v2::LazyInitSegmentIterator::next_batch(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/lazy_init_segment_iterator.h:44:33 apache#22 0x5616ad269d67 in doris::BetaRowsetReader::next_block(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/beta_rowset_reader.cpp:380:29 apache#23 0x5616de6de110 in doris::vectorized::VCollectIterator::Level0Iterator::_refresh() /root/doris/be/src/vec/olap/vcollect_iterator.h apache#24 0x5616de6c967f in doris::vectorized::VCollectIterator::Level0Iterator::refresh_current_row() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:514:24 apache#25 0x5616de6ca8a6 in doris::vectorized::VCollectIterator::Level0Iterator::ensure_first_row_ref() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:493:14 apache#26 0x5616de6d7008 in doris::vectorized::VCollectIterator::Level1Iterator::ensure_first_row_ref() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:692:27 apache#27 0x5616de6bd200 in doris::vectorized::VCollectIterator::build_heap(std::vector, std::allocator>>&) /root/doris/be/src/vec/olap/vcollect_iterator.cpp:186:9 apache#28 0x5616de651b6c in doris::vectorized::BlockReader::_init_collect_iter(doris::TabletReader::ReaderParams const&) /root/doris/be/src/vec/olap/block_reader.cpp:157:5 apache#29 0x5616de65526f in doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams const&) /root/doris/be/src/vec/olap/block_reader.cpp:229:19 apache#30 0x5616e175a0f9 in doris::vectorized::NewOlapScanner::open(doris::RuntimeState*) /root/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:237:32 apache#31 0x5616c736ad34 in doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr, std::shared_ptr) /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:236:5 apache#32 0x5616c736f05e in doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:176:21 apache#33 0x5616c736f05e in doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:175:31 apache#34 0x5616c736f05e in void std::_invoke_impl, std::shared_ptr)::$_1::operator()() const::'lambda'()&>(std::_invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 apache#35 0x5616c736f05e in std::enable_if, std::shared_ptr)::$1::operator()() const::'lambda'()&>, void>::type std::_invoke_r, std::shared_ptr)::$_1::operator()() const::'lambda'()&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 apache#36 0x5616c736f05e in std::_Function_handler, std::shared_ptr)::$_1::operator()() const::'lambda'()>::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 apache#37 0x5616aeed6a3b in doris::ThreadPool::dispatch_thread() /root/doris/be/src/util/threadpool.cpp:543:24 apache#38 0x5616aeeae4f7 in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:498:5 apache#39 0x7f7e663e3ac2 in start_thread nptl/pthread_create.c:442:8 apache#40 0x7f7e6647584f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /root/doris/be/src/vec/common/string_ref.h:271:54 in ```
wyxxxcat
pushed a commit
that referenced
this pull request
Aug 14, 2024
…he#39237) ## Proposed changes which led to accessing uninitialized data. Moved the profile initialization into the init method. ``` /mnt/disk2/yanxuecheng/doris/be/src/pipeline/local_exchange/local_exchanger.cpp:100:9: runtime error: member call on null pointer of type 'doris::RuntimeProfile::Counter' #0 0x563f0fb8b6c9 in doris::pipeline::Exchanger<std::shared_ptr<doris::pipeline::BlockWrapper>>::_dequeue_data(doris::pipeline::LocalExchangeSourceLocalState&, std::shared_ptr<doris::pipeline::BlockWrapper>&, bool*, doris::vectorized::Block*, int) /mnt/disk2/yanxuecheng/doris/be/src/pipeline/local_exchange/local_exchanger.cpp:100:9 #1 0x563f0fb78a45 in doris::pipeline::LocalMergeSortExchanger::finalize(doris::pipeline::LocalExchangeSourceLocalState&) /mnt/disk2/yanxuecheng/doris/be/src/pipeline/local_exchange/local_exchanger.cpp:325:16 #2 0x563f09610366 in doris::pipeline::LocalExchangeSharedState::sub_running_source_operators(doris::pipeline::LocalExchangeSourceLocalState&) /mnt/disk2/yanxuecheng/doris/be/src/pipeline/dependency.cpp:196:20 #3 0x563f0fb63709 in doris::pipeline::LocalExchangeSourceLocalState::close(doris::RuntimeState*) /mnt/disk2/yanxuecheng/doris/be/src/pipeline/local_exchange/local_exchange_source_operator.cpp:59:24 #4 0x563f09728e80 in doris::pipeline::OperatorXBase::close(doris::RuntimeState*) /mnt/disk2/yanxuecheng/doris/be/src/pipeline/exec/operator.cpp:245:28 apache#5 0x563f0fd3fb58 in doris::pipeline::PipelineTask::close(doris::Status) /mnt/disk2/yanxuecheng/doris/be/src/pipeline/pipeline_task.cpp:459:28 apache#6 0x563f0fdb5315 in doris::pipeline::_close_task(doris::pipeline::PipelineTask*, doris::Status) /mnt/disk2/yanxuecheng/doris/be/src/pipeline/task_scheduler.cpp:91:27 apache#7 0x563f0fdb6573 in doris::pipeline::TaskScheduler::_do_work(unsigned long) /mnt/disk2/yanxuecheng/doris/be/src/pipeline/task_scheduler.cpp:125:13 apache#8 0x563f0fdb9d6a in doris::pipeline::TaskScheduler::start()::$_0::operator()() const /mnt/disk2/yanxuecheng/doris/be/src/pipeline/task_scheduler.cpp:64:9 apache#9 0x563f0fdb9cee in void std::__invoke_impl<void, doris::pipeline::TaskScheduler::start()::$_0&>(std::__invoke_other, doris::pipeline::TaskScheduler::start()::$_0&) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14 apache#10 0x563f0fdb9c4e in std::enable_if<is_invocable_r_v<void, doris::pipeline::TaskScheduler::start()::$_0&>, void>::type std::__invoke_r<void, doris::pipeline::TaskScheduler::start()::$_0&>(doris::pipeline::TaskScheduler::start()::$_0&) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:111:2 apache#11 0x563f0fdb99d5 in std::_Function_handler<void (), doris::pipeline::TaskScheduler::start()::$_0>::_M_invoke(std::_Any_data const&) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290:9 apache#12 0x563ebdddc8cf in std::function<void ()>::operator()() const /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591:9 apache#13 0x563ec4bd6db4 in doris::FunctionRunnable::run() /mnt/disk2/yanxuecheng/doris/be/src/util/threadpool.cpp:48:27 apache#14 0x563ec4bbc1b5 in doris::ThreadPool::dispatch_thread() /mnt/disk2/yanxuecheng/doris/be/src/util/threadpool.cpp:543:24 apache#15 0x563ec4bf9a53 in void std::__invoke_impl<void, void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(std::__invoke_memfun_deref, void (doris::ThreadPool::*&)(), doris::ThreadPool*&) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:74:14 apache#16 0x563ec4bf9858 in std::__invoke_result<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>::type std::__invoke<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(void (doris::ThreadPool::*&)(), doris::ThreadPool*&) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:96:14 apache#17 0x563ec4bf9790 in void std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>::__call<void, 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/functional:506:11 apache#18 0x563ec4bf9585 in void std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>::operator()<void>() /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/functional:591:17 apache#19 0x563ec4bf947e in void std::__invoke_impl<void, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&>(std::__invoke_other, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14 apache#20 0x563ec4bf93be in std::enable_if<is_invocable_r_v<void, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&>, void>::type std::__invoke_r<void, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&>(std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:111:2 apache#21 0x563ec4bf8e55 in std::_Function_handler<void (), std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>>::_M_invoke(std::_Any_data const&) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290:9 apache#22 0x563ebdddc8cf in std::function<void ()>::operator()() const /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591:9 apache#23 0x563ec4b78d91 in doris::Thread::supervise_thread(void*) /mnt/disk2/yanxuecheng/doris/be/src/util/thread.cpp:498:5 apache#24 0x563ebdb2fe0a in asan_thread_start(void*) crtstuff.c apache#25 0x7feeac9e21c9 in start_thread (/lib64/libpthread.so.0+0x81c9) (BuildId: 823fccea3475e5870a4167dfe47df20e53222db0) apache#26 0x7feead3d1e72 in clone (/lib64/libc.so.6+0x39e72) (BuildId: ec3d7025354f1f1985831ff08ef0eb3b50aefbce) SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /mnt/disk2/yanxuecheng/doris/be/src/pipeline/local_exchange/local_exchanger.cpp:100:9 in *** Query id: ea174401bc134452-bd8a35522726a96a *** *** is nereids: 1 *** *** tablet id: 0 *** *** Aborted at 1723455006 (unix time) try "date -d @1723455006" if you are using GNU date *** *** Current BE git commitID: c47399c *** *** SIGSEGV address not mapped to object (@0x0) received by PID 3055435 (TID 3060341 OR 0x7fe320acf700) from PID 0; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /mnt/disk2/yanxuecheng/doris/be/src/common/signal_handler.h:421 1# 0x00007FEEAD3E6B50 in /lib64/libc.so.6 2# doris::pipeline::Exchanger<std::shared_ptr<doris::pipeline::BlockWrapper> >::_dequeue_data(doris::pipeline::LocalExchangeSourceLocalState&, std::shared_ptr<doris::pipeline::BlockWrapper>&, bool*, doris::vectorized::Block*, int) at /mnt/disk2/yanxuecheng/doris/be/src/pipeline/local_exchange/local_exchanger.cpp:100 3# doris::pipeline::LocalMergeSortExchanger::finalize(doris::pipeline::LocalExchangeSourceLocalState&) at /mnt/disk2/yanxuecheng/doris/be/src/pipeline/local_exchange/local_exchanger.cpp:325 4# doris::pipeline::LocalExchangeSharedState::sub_running_source_operators(doris::pipeline::LocalExchangeSourceLocalState&) at /mnt/disk2/yanxuecheng/doris/be/src/pipeline/dependency.cpp:196 5# doris::pipeline::LocalExchangeSourceLocalState::close(doris::RuntimeState*) in /mnt/disk2/yanxuecheng/doris/output/be/lib/doris_be 6# doris::pipeline::OperatorXBase::close(doris::RuntimeState*) at /mnt/disk2/yanxuecheng/doris/be/src/pipeline/exec/operator.cpp:245 7# doris::pipeline::PipelineTask::close(doris::Status) at /mnt/disk2/yanxuecheng/doris/be/src/pipeline/pipeline_task.cpp:459 8# doris::pipeline::_close_task(doris::pipeline::PipelineTask*, doris::Status) at /mnt/disk2/yanxuecheng/doris/be/src/pipeline/task_scheduler.cpp:91 9# doris::pipeline::TaskScheduler::_do_work(unsigned long) at /mnt/disk2/yanxuecheng/doris/be/src/pipeline/task_scheduler.cpp:125 10# doris::pipeline::TaskScheduler::start()::$_0::operator()() const at /mnt/disk2/yanxuecheng/doris/be/src/pipeline/task_scheduler.cpp:64 11# void std::__invoke_impl<void, doris::pipeline::TaskScheduler::start()::$_0&>(std::__invoke_other, doris::pipeline::TaskScheduler::start()::$_0&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61 12# std::enable_if<is_invocable_r_v<void, doris::pipeline::TaskScheduler::start()::$_0&>, void>::type std::__invoke_r<void, doris::pipeline::TaskScheduler::start()::$_0&>(doris::pipeline::TaskScheduler::start()::$_0&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:117 13# std::_Function_handler<void (), doris::pipeline::TaskScheduler::start()::$_0>::_M_invoke(std::_Any_data const&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290 14# std::function<void ()>::operator()() const at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591 15# doris::FunctionRunnable::run() at /mnt/disk2/yanxuecheng/doris/be/src/util/threadpool.cpp:48 16# doris::ThreadPool::dispatch_thread() at /mnt/disk2/yanxuecheng/doris/be/src/util/threadpool.cpp:543 17# void std::__invoke_impl<void, void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(std::__invoke_memfun_deref, void (doris::ThreadPool::*&)(), doris::ThreadPool*&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:74 18# std::__invoke_result<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>::type std::__invoke<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(void (doris::ThreadPool::*&)(), doris::ThreadPool*&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:96 19# void std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/functional:506 20# void std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>::operator()<, void>() at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/functional:591 21# void std::__invoke_impl<void, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&>(std::__invoke_other, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61 22# std::enable_if<is_invocable_r_v<void, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&>, void>::type std::__invoke_r<void, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&>(std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:117 23# std::_Function_handler<void (), std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()> >::_M_invoke(std::_Any_data const&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290 24# std::function<void ()>::operator()() const at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591 25# doris::Thread::supervise_thread(void*) at /mnt/disk2/yanxuecheng/doris/be/src/util/thread.cpp:498 26# asan_thread_start(void*) in /mnt/disk2/yanxuecheng/doris/output/be/lib/doris_be 27# start_thread in /lib64/libpthread.so.0 28# __clone in /lib64/libc.so.6 ``` <!--Describe your changes.-->
wyxxxcat
pushed a commit
that referenced
this pull request
Aug 22, 2024
…ort (apache#39340) Since some variables are only initialized when enable_java_support is enabled, not adding this check here would result in accessing a nullptr. ``` *** Query id: 0-0 *** *** is nereids: 0 *** *** tablet id: 0 *** *** Aborted at 1723618805 (unix time) try "date -d @1723618805" if you are using GNU date *** *** Current BE git commitID: 8d64cfa563 *** *** SIGSEGV address not mapped to object (@0x0) received by PID 2579243 (TID 2583082 OR 0x7f07ae942700) from PID 0; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /mnt/disk2/yanxuecheng/doris/be/src/common/signal_handler.h:421 1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /mnt/disk2/yanxuecheng/java17/lib/server/libjvm.so 2# JVM_handle_linux_signal in /mnt/disk2/yanxuecheng/java17/lib/server/libjvm.so 3# 0x00007F0DA7188B50 in /lib64/libc.so.6 4# jni_CallVoidMethodV in /mnt/disk2/yanxuecheng/java17/lib/server/libjvm.so 5# JNIEnv_::CallVoidMethod(_jobject*, _jmethodID*, ...) at /mnt/disk2/yanxuecheng/java17/include/jni.h:1059 6# doris::JniUtil::clean_udf_class_load_cache(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) in /mnt/disk2/yanxuecheng/doris/output/be/lib/doris_be 7# doris::clean_udf_cache_callback(doris::TAgentTaskRequest const&) at /mnt/disk2/yanxuecheng/doris/be/src/agent/task_worker_pool.cpp:2072 8# auto doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_19::operator()<doris::TAgentTaskRequest const&>(doris::TAgentTaskRequest const&) const at /mnt/disk2/yanxuecheng/doris/be/src/agent/agent_server.cpp:184 9# void std::__invoke_impl<void, doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_19&, doris::TAgentTaskRequest const&>(std::__invoke_other, doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_19&, doris::TAgentTaskRequest const&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61 10# std::enable_if<is_invocable_r_v<void, doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_19&, doris::TAgentTaskRequest const&>, void>::type std::__invoke_r<void, doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_19&, doris::TAgentTaskRequest const&>(doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_19&, doris::TAgentTaskRequest const&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:117 11# std::_Function_handler<void (doris::TAgentTaskRequest const&), doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_19>::_M_invoke(std::_Any_data const&, doris::TAgentTaskRequest const&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290 12# std::function<void (doris::TAgentTaskRequest const&)>::operator()(doris::TAgentTaskRequest const&) const at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591 13# doris::TaskWorkerPool::submit_task(doris::TAgentTaskRequest const&)::$_0::operator()<doris::TAgentTaskRequest const&>(doris::TAgentTaskRequest const&) const::{lambda()#1}::operator()() const at /mnt/disk2/yanxuecheng/doris/be/src/agent/task_worker_pool.cpp:541 14# void std::__invoke_impl<void, doris::TaskWorkerPool::submit_task(doris::TAgentTaskRequest const&)::$_0::operator()<doris::TAgentTaskRequest const&>(doris::TAgentTaskRequest const&) const::{lambda()#1}&>(std::__invoke_other, doris::TaskWorkerPool::submit_task(doris::TAgentTaskRequest const&)::$_0::operator()<doris::TAgentTaskRequest const&>(doris::TAgentTaskRequest const&) const::{lambda()#1}&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61 15# std::enable_if<is_invocable_r_v<void, doris::TaskWorkerPool::submit_task(doris::TAgentTaskRequest const&)::$_0::operator()<doris::TAgentTaskRequest const&>(doris::TAgentTaskRequest const&) const::{lambda()#1}&>, void>::type std::__invoke_r<void, doris::TaskWorkerPool::submit_task(doris::TAgentTaskRequest const&)::$_0::operator()<doris::TAgentTaskRequest const&>(doris::TAgentTaskRequest const&) const::{lambda()#1}&>(doris::TaskWorkerPool::submit_task(doris::TAgentTaskRequest const&)::$_0::operator()<doris::TAgentTaskRequest const&>(doris::TAgentTaskRequest const&) const::{lambda()#1}&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:117 16# std::_Function_handler<void (), doris::TaskWorkerPool::submit_task(doris::TAgentTaskRequest const&)::$_0::operator()<doris::TAgentTaskRequest const&>(doris::TAgentTaskRequest const&) const::{lambda()#1}>::_M_invoke(std::_Any_data const&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290 17# std::function<void ()>::operator()() const at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591 18# doris::FunctionRunnable::run() at /mnt/disk2/yanxuecheng/doris/be/src/util/threadpool.cpp:48 19# doris::ThreadPool::dispatch_thread() at /mnt/disk2/yanxuecheng/doris/be/src/util/threadpool.cpp:543 20# void std::__invoke_impl<void, void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(std::__invoke_memfun_deref, void (doris::ThreadPool::*&)(), doris::ThreadPool*&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:74 21# std::__invoke_result<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>::type std::__invoke<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(void (doris::ThreadPool::*&)(), doris::ThreadPool*&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:96 22# void std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/functional:506 23# void std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>::operator()<, void>() at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/functional:591 24# void std::__invoke_impl<void, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&>(std::__invoke_other, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61 25# std::enable_if<is_invocable_r_v<void, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&>, void>::type std::__invoke_r<void, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&>(std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:117 26# std::_Function_handler<void (), std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()> >::_M_invoke(std::_Any_data const&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290 27# std::function<void ()>::operator()() const at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591 28# doris::Thread::supervise_thread(void*) at /mnt/disk2/yanxuecheng/doris/be/src/util/thread.cpp:498 29# asan_thread_start(void*) in /mnt/disk2/yanxuecheng/doris/output/be/lib/doris_be 30# start_thread in /lib64/libpthread.so.0 31# __clone in /lib64/libc.so.6 ```
wyxxxcat
pushed a commit
that referenced
this pull request
Aug 22, 2024
in fe we will check the params same with be
in be we throw exception if params is not support
otherwise we will meet coredump like this
```
start BE in local mode
/mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/array/function_array_aggregation.cpp:150:26: runtime error: member call on null pointer of type 'doris::vectorized::IAggregateFunction'
#0 0x55dc04c6b409 in doris::vectorized::ArrayAggregateImpl<(doris::vectorized::AggregateOperation)1>::get_return_type(std::vector<std::shared_ptr<doris::vectorized::IDataType const>, std::allocator<std::shared_ptr<doris::vectorized::IDataType const>>> const&) /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/array/function_array_aggregation.cpp:150:26
#1 0x55dc04c6a41f in doris::vectorized::FunctionArrayMapped<doris::vectorized::ArrayAggregateImpl<(doris::vectorized::AggregateOperation)1>, doris::vectorized::NameArrayMax>::get_return_type_impl(std::vector<std::shared_ptr<doris::vectorized::IDataType const>, std::allocator<std::shared_ptr<doris::vectorized::IDataType const>>> const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/array/function_array_mapped.h:71:16
#2 0x55dc049b06cb in doris::vectorized::FunctionBuilderImpl::get_return_type_impl(std::vector<doris::vectorized::ColumnWithTypeAndName, std::allocator<doris::vectorized::ColumnWithTypeAndName>> const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function.h:339:16
#3 0x55dc049c1385 in doris::vectorized::DefaultFunctionBuilder::get_return_type_impl(std::vector<doris::vectorized::ColumnWithTypeAndName, std::allocator<doris::vectorized::ColumnWithTypeAndName>> const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function.h:579:26
#4 0x55dc0b0a5652 in doris::vectorized::FunctionBuilderImpl::get_return_type_without_low_cardinality(std::vector<doris::vectorized::ColumnWithTypeAndName, std::allocator<doris::vectorized::ColumnWithTypeAndName>> const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function.cpp:283:12
apache#5 0x55dc0b0a5fd4 in doris::vectorized::FunctionBuilderImpl::get_return_type(std::vector<doris::vectorized::ColumnWithTypeAndName, std::allocator<doris::vectorized::ColumnWithTypeAndName>> const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function.cpp:298:17
apache#6 0x55dc04997050 in doris::vectorized::FunctionBuilderImpl::build(std::vector<doris::vectorized::ColumnWithTypeAndName, std::allocator<doris::vectorized::ColumnWithTypeAndName>> const&, std::shared_ptr<doris::vectorized::IDataType const> const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function.h:296:47
apache#7 0x55dbde3d727f in doris::vectorized::SimpleFunctionFactory::get_function(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::vector<doris::vectorized::ColumnWithTypeAndName, std::allocator<doris::vectorized::ColumnWithTypeAndName>> const&, std::shared_ptr<doris::vectorized::IDataType const> const&, int) /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/simple_function_factory.h:193:32
apache#8 0x55dc04984b4d in doris::vectorized::VectorizedFnCall::prepare(doris::RuntimeState*, doris::RowDescriptor const&, doris::vectorized::VExprContext*) /mnt/disk1/wangqiannan/amory/doris/be/src/vec/exprs/vectorized_fn_call.cpp:111:55
apache#9 0x55dc04ab9b7a in doris::vectorized::VExprContext::prepare(doris::RuntimeState*, doris::RowDescriptor const&) /mnt/disk1/wangqiannan/amory/doris/be/src/vec/exprs/vexpr_context.cpp:64:5
apache#10 0x55dbdf625845 in doris::Status doris::FoldConstantExecutor::_prepare_and_open<doris::vectorized::VExprContext>(doris::vectorized::VExprContext*) /mnt/disk1/wangqiannan/amory/doris/be/src/runtime/fold_constant_executor.cpp:185:5
apache#11 0x55dbdf60db76 in doris::FoldConstantExecutor::fold_constant_vexpr(doris::TFoldConstantParams const&, doris::PConstantExprResult*) /mnt/disk1/wangqiannan/amory/doris/be/src/runtime/fold_constant_executor.cpp:94:13
apache#12 0x55dbdf4f58d6 in doris::PInternalService::_fold_constant_expr(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, doris::PConstantExprResult*) /mnt/disk1/wangqiannan/amory/doris/be/src/service/internal_service.cpp:1498:5
apache#13 0x55dbdf544e3e in doris::PInternalService::fold_constant_expr(google::protobuf::RpcController*, doris::PConstantExprRequest const*, doris::PConstantExprResult*, google::protobuf::Closure*)::$_0::operator()() const /mnt/disk1/wangqiannan/amory/doris/be/src/service/internal_service.cpp:1476:14
apache#14 0x55dbdf54431e in void std::__invoke_impl<void, doris::PInternalService::fold_constant_expr(google::protobuf::RpcController*, doris::PConstantExprRequest const*, doris::PConstantExprResult*, google::protobuf::Closure*)::$_0&>(std::__invoke_other, doris::PInternalService::fold_constant_expr(google::protobuf::RpcController*, doris::PConstantExprRequest const*, doris::PConstantExprResult*, google::protobuf::Closure*)::$_0&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
apache#15 0x55dbdf54425e in std::enable_if<is_invocable_r_v<void, doris::PInternalService::fold_constant_expr(google::protobuf::RpcController*, doris::PConstantExprRequest const*, doris::PConstantExprResult*, google::protobuf::Closure*)::$_0&>, void>::type std::__invoke_r<void, doris::PInternalService::fold_constant_expr(google::protobuf::RpcController*, doris::PConstantExprRequest const*, doris::PConstantExprResult*, google::protobuf::Closure*)::$_0&>(doris::PInternalService::fold_constant_expr(google::protobuf::RpcController*, doris::PConstantExprRequest const*, doris::PConstantExprResult*, google::protobuf::Closure*)::$_0&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2
apache#16 0x55dbdf543f35 in std::_Function_handler<void (), doris::PInternalService::fold_constant_expr(google::protobuf::RpcController*, doris::PConstantExprRequest const*, doris::PConstantExprResult*, google::protobuf::Closure*)::$_0>::_M_invoke(std::_Any_data const&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9
apache#17 0x55dbd8cd425a in std::function<void ()>::operator()() const /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9
apache#18 0x55dbdf59ee14 in doris::WorkThreadPool<false>::work_thread(int) /mnt/disk1/wangqiannan/amory/doris/be/src/util/work_thread_pool.hpp:158:17
@@@
```
wyxxxcat
pushed a commit
that referenced
this pull request
Aug 22, 2024
…che#39566) ==17355==ERROR: AddressSanitizer: heap-use-after-free on address 0x61c00388c269 at pc 0x55b47f79e5c4 bp 0x7fde7dd98e30 sp 0x7fde7dd98e28 READ of size 1 at 0x61c00388c269 thread T604 (SpillIOThreadPo) #0 0x55b47f79e5c3 in doris::QueryContext::is_nereids() const /root/doris/be/src/runtime/query_context.h:229:38 #1 0x55b47f79e5c3 in doris::RuntimeState::is_nereids() const /root/doris/be/src/runtime/runtime_state.cpp:542:24 #2 0x55b47f89afed in doris::AttachTask::AttachTask(doris::RuntimeState*) /root/doris/be/src/runtime/thread_context.cpp:48:50 #3 0x55b4b039e7e5 in doris::pipeline::SpillRunnable::run() /root/doris/be/src/pipeline/exec/spill_utils.h:43:9 #4 0x55b47fdf739d in doris::ThreadPool::dispatch_thread() /root/doris/be/src/util/threadpool.cpp:543:24 apache#5 0x55b47fdd05de in std::function<void ()>::operator()() const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9 apache#6 0x55b47fdd05de in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:498:5 apache#7 0x7fe1bcb5e608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477:8 apache#8 0x7fe1bce0b132 in __clone /build/glibc-SzIz7B/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Co-authored-by: yiguolei <[email protected]>
wyxxxcat
pushed a commit
that referenced
this pull request
Sep 2, 2024
… enough rdlock inside tablet internal function calls. (apache#39211) ## Proposed changes There's a recurse rdlock inside get_cooldown；when using libcxx，libcxx‘s shared_mutex dosen't support recursive lock, so causes hangs below: #0 0x00007f788b101a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005636b8ff996f in std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) () #2 0x00005636b8ffad8b in std::__1::__shared_mutex_base::lock_shared() () #3 0x00005636addc524c in doris::Tablet::_has_data_to_cooldown() () #4 0x00005636addc353b in doris::Tablet::pick_cooldown_rowset() () apache#5 0x00005636addc56b9 in doris::Tablet::need_cooldown(long*, unsigned long*) () apache#6 0x00005636adde52f7 in doris::TabletManager::get_cooldown_tablets(std::__1::vector<std::__1::shared_ptr<doris::Tablet>, std::__1::allocator<std::__1::shared_ptr<doris::Tablet> > >*, std::__1::vector<std::__1::shared_ptr<doris::Rowset>, std::__1::allocator<std::__1::shared_ptr<doris::Rowset> > >*, std::__1::function<bool (std::__1::shared_ptr<doris::Tablet> const&)>) () apache#7 0x00005636adb0ed81 in doris::StorageEngine::_cooldown_tasks_producer_callback() () apache#8 0x00005636ae17a253 in doris::Thread::supervise_thread(void*) () apache#9 0x00007f788b0fdea5 in start_thread () from /lib64/libpthread.so.0 apache#10 0x00007f788b916b0d in clone () from /lib64/libc.so.6 review the code, there's no need a rdlock inside TabletManager::get_cooldown_tablets
wyxxxcat
pushed a commit
that referenced
this pull request
Sep 11, 2024
We have dead lock when submit scanner to scheduler failed. pstack looks like ```txt Thread 2012 (Thread 0x7f87363fb700 (LWP 4179707) "Pipe_normal [wo"): #0 0x00007f8b8f3dc82d in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007f8b8f3d5ad9 in pthread_mutex_lock () from /lib64/libpthread.so.0 #2 0x000055b20f333e7a in __gthread_mutex_lock (__mutex=0x7f8733d960a8) at /mnt/disk1/hezhiqiang/toolchains/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/x86_64-linux-gnu/c++/11/bits/gthr-default .h:749 #3 std::mutex::lock (this=0x7f8733d960a8) at /mnt/disk1/hezhiqiang/toolchains/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_mutex.h:100 #4 std::lock_guard<std::mutex>::lock_guard (__m=..., this=<optimized out>) at /mnt/disk1/hezhiqiang/toolchains/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_mutex.h:229 apache#5 doris::vectorized::ScannerContext::append_block_to_queue (this=<optimized out>, scan_task=...) at /mnt/disk1/hezhiqiang/doris/be/src/vec/exec/scan/scanner_context.cpp:234 apache#6 0x000055b20f32c0f9 in doris::vectorized::ScannerScheduler::submit (this=<optimized out>, ctx=..., scan_task=...) at /mnt/disk1/hezhiqiang/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:209 apache#7 0x000055b20f3338fc in doris::vectorized::ScannerContext::submit_scan_task (this=this@entry=0x7f8733d96010, scan_task=...) at /mnt/disk1/hezhiqiang/doris/be/src/vec/exec/scan/scanner_context.cpp:217 apache#8 0x000055b20f3346cd in doris::vectorized::ScannerContext::get_block_from_queue (this=0x7f8733d96010, state=<optimized out>, block=0x7f871f728de0, eos=0x7f871abce470, id=<optimized out>) at /mnt/disk1/hezhiqiang/doris/be/src/vec/exec/scan/scanner_context.cpp:290 apache#9 0x000055b214cb4f13 in doris::pipeline::ScanOperatorX<doris::pipeline::OlapScanLocalState>::get_block (this=<optimized out>, state=0x7f872f0eb400, block=0x7f8b8f3dc82d <__lll_lock_wait+29>, eos=0x7f871abce470) at /mnt/disk1/hezhiqiang/doris/be/src/pipeline/exec/scan_operator.cpp:1292 apache#10 0x000055b2142b5772 in doris::pipeline::ScanOperatorX<doris::pipeline::OlapScanLocalState>::get_block_after_projects (this=0x80, state=0x0, block=0x7f8b8f3dc82d <__lll_lock_wait+29>, eos=0x7f8733d960a8) at /mnt/disk1/hezhiqiang/doris/be/src/pipeline/exec/scan_operator.h:363 apache#11 0x000055b2142e7880 in doris::pipeline::StatefulOperatorX<doris::pipeline::StreamingAggLocalState>::get_block (this=0x7f871f9bee00, state=0x7f872f0eb400, block=0x7f8716d49060, eos=0x7f87363f4937) at /mnt/disk1/hezhiqiang/doris/be/src/pipeline/exec/operator.cpp:587 ``` Deallock happens with following ```cpp Status ScannerContext::get_block_from_queue { std::unique_lock l(_transfer_lock); ... if (scan_task->is_eos()) { ... } else { // resubmit current running scanner to read the next block submit_scan_task(scan_task); } } ScannerContext::submit_scan_task(std::shared_ptr<ScanTask> scan_task) { _scanner_scheduler->submit(shared_from_this(), scan_task); } void ScannerScheduler::submit(std::shared_ptr<ScannerContext> ctx, std::shared_ptr<ScanTask> scan_task) { ... if (auto ret = sumbit_task(); !ret) { scan_task->set_status(Status::InternalError( "Failed to submit scanner to scanner pool reason:" + std::string(ret.msg()) + "|type:" + std::to_string(type))); ctx->append_block_to_queue(scan_task); return; } } void ScannerContext::append_block_to_queue(std::shared_ptr<ScanTask> scan_task) { ... std::lock_guard<std::mutex> l(_transfer_lock); ... } ``` Since mutex in cpp is not re-enterable, so the scanner thread will deadlock with itself. This pr fix the problem by making `ScannerScheduler::submit` return a Status instead of doing append failed task to the ScannerContext. The caller itself will decide where resubmit the scanner or just abort the execution of the query.
wyxxxcat
pushed a commit
that referenced
this pull request
Sep 17, 2024
…che#40273) ## Proposed changes Issue Number: close #xxx cur_reader pointer can be null in VFileScanner. can cause BE crash [ RUN ] VfileScannerExcepTest.failure_case AddressSanitizer:DEADLYSIGNAL ================================================================= ==1892247==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x5cdbe26eb5f8 bp 0x7ffed6728610 sp 0x7ffed67277a0 T0) ==1892247==The signal is caused by a READ memory access. ==1892247==Hint: address points to the zero page. #0 0x5cdbe26eb5f8 in doris::vectorized::VFileScanner::_get_next_reader() /root/doris/workspace/doris/be/src/vec/exec/scan/vfile_scanner.cpp:980:9 #1 0x5cdbe26e3512 in doris::vectorized::VFileScanner::_get_block_wrapped(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/workspace/doris/be/src/vec/exec/scan/vfile_scanner.cpp:286:25 #2 0x5cdbe26e2e46 in doris::vectorized::VFileScanner::_get_block_impl(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/workspace/doris/be/src/vec/exec/scan/vfile_scanner.cpp:252:17 #3 0x5cdbe28e05ce in doris::vectorized::VScanner::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/workspace/doris/be/src/vec/exec/scan/vscanner.cpp:117:17 #4 0x5cdbc405a922 in doris::vectorized::VfileScannerExcepTest_failure_case_Test::TestBody() /root/doris/workspace/doris/be/test/vec/exec/vfile_scanner_excep_test.cpp:309:24 apache#5 0x5cdbff5b191a in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/root/doris/workspace/doris/be/ut_build_ASAN/test/doris_be_test+0x51eb491a) (BuildId: 21d41a2d207823b9) apache#6 0x5cdbff59f989 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/root/doris/workspace/doris/be/ut_build_ASAN/test/doris_be_test+0x51ea2989) (BuildId: 21d41a2d207823b9) apache#7 0x5cdbff57a9c2 in testing::Test::Run() (/root/doris/workspace/doris/be/ut_build_ASAN/test/doris_be_test+0x51e7d9c2) (BuildId: 21d41a2d207823b9) apache#8 0x5cdbff57b708 in testing::TestInfo::Run() (/root/doris/workspace/doris/be/ut_build_ASAN/test/doris_be_test+0x51e7e708) (BuildId: 21d41a2d207823b9) apache#9 0x5cdbff57bec3 in testing::TestSuite::Run() (/root/doris/workspace/doris/be/ut_build_ASAN/test/doris_be_test+0x51e7eec3) (BuildId: 21d41a2d207823b9) After fix: I20240902 09:11:07.722273 1946048 run_all_tests.cpp:67] init config 1 Note: Google Test filter = VfileScannerE* [==========] Running 1 test from 1 test suite. [----------] Global test environment set-up. [----------] 1 test from VfileScannerExceptionTest [ RUN ] VfileScannerExceptionTest.failure_case msg = [INTERNAL_ERROR]cur path: . Failed to create reader for file format: 11 [ OK ] VfileScannerExceptionTest.failure_case (3 ms) [----------] 1 test from VfileScannerExceptionTest (3 ms total) [----------] Global test environment tear-down [==========] 1 test from 1 test suite ran. (3 ms total) [ PASSED ] 1 test. === Finished. Gtest output: /root/doris/workspace/doris/be/ut_build_ASAN/gtest_output
wyxxxcat
pushed a commit
that referenced
this pull request
Sep 17, 2024
…#40643) ## Proposed changes ### Issue ``` *** is nereids: 1 *** tablet id: 4 Abort at 1725864966 (unix time) try "date -d @1725864966" if you are using GNU date *** *** Set a breakpoint in static void __GI_abort() to debug *** PC: @ 0x7f007fb4090a04 *** SIGSEGV (address not mapped to object 0xa0fa868a41d6) received by PID 404737 (TID 274135 OR 0x7ece29df700) from PID 1755584205; stack trace: *** #0 __GI_raise #1 __GI_abort #2 sig_handler #3 _sigaction #4 JVM_handle_linux_signal apache#5 _sigaction apache#6 doris::vectorized::ByteArrayDictDecoder::set_dict(std::unique_ptr<unsigned char[], std::default_delete<unsigned char[]>> &&, int, unsigned long) at /mnt/disk1/yy/git/enterprise-core/be/src/vec/exec/format/parquet/byte_array_dict_decoder.cpp:41 apache#7 doris::vectorized::ColumnChunkReader::_decode_dict_page() at /mnt/disk1/yy/git/enterprise-core/be/src/vec/exec/format/parquet/vparquet_column_chunk_reader.cpp:258 apache#8 doris::vectorized::ColumnChunkReader::next_page() at /mnt/disk1/yy/git/enterprise-core/be/src/vec/exec/format/parquet/vparquet_column_chunk_reader.cpp:105 apache#9 doris::vectorized::ParquetColumnReader::_read_column_data(doris::vectorized::Block*, bool*) at /mnt/disk1/yy/git/enterprise-core/be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:508 apache#10 doris::vectorized::ScalarColumnReader::_next_value(doris::vectorized::ICollumn*, unsigned long, unsigned long*, bool*) at /mnt/disk1/yy/git/enterprise-core/be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:699 apache#11 doris::vectorized::RowGroupReader::_read_column_data(doris::vectorized::Block*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> &, std::vector<doris::vectorized::ColumnSelectVector>*, unsigned long, unsigned long*, bool*) at /mnt/disk1/yy/git/enterprise-core/be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:425 apache#12 doris::vectorized::RowGroupReader::get_next_block(doris::vectorized::Block*, bool*) at /mnt/disk1/yy/git/enterprise-core/be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:311 apache#13 doris::vectorized::ParquetReader::get_next(doris::vectorized::Block*, bool*) at /mnt/disk1/yy/git/enterprise-core/be/src/vec/exec/format/parquet/vparquet_reader.cpp:533 apache#14 doris::vectorized::VFileScanner::_get_next_reader_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /mnt/disk1/yy/git/enterprise-core/be/src/vec/exec/scan/vfile_scanner.cpp:368 apache#15 doris::vectorized::VFileScanner::_get_block_impl(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /mnt/disk1/yy/git/enterprise-core/be/src/vec/exec/scan/vfile_scanner.cpp:411 apache#16 doris::vectorized::VScanner::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /mnt/disk1/yy/git/enterprise-core/be/src/vec/exec/scan/vscanner.cpp:431 apache#17 doris::vectorized::VScanner::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /mnt/disk1/yy/git/enterprise-core/be/src/vec/exec/scan/vscanner.cpp:96 apache#18 doris::vectorized::ScannerScheduler::submit(doris::vectorized::ScannerContext*, std::shared_ptr<doris::vectorized::ScanTask>) at /mnt/disk1/yy/git/enterprise-core/be/src/vec/exec/scan/scanner_context.cpp:96 apache#19 doris::Thread::supervise_thread(void*) at /mnt/disk1/yy/git/enterprise-core/be/src/util/thread.cpp:499 apache#20 start_thread apache#21 clone in /lib64/libc.so.6 ``` ### Solution It is not known why the parquet dictionary page will be null in this case, causing a crash. This PR adds defensive code to prevent the crash.
wyxxxcat
pushed a commit
that referenced
this pull request
Sep 17, 2024
1. Add a new class MemCounter.
2. MemTracker and MemTrackerLimiter no longer have inheritance and
dependencies.
3. MemTrackerLimiter is used to count process memory, the BE web page
`/mem_tracker` also only displays MemTrackerLimiter.
4. MemTracker is used to count the memory of operators and some data
structures. It is not counted in the process memory and is used for
logic control and profile.
In addition, it seems that the crash is caused by memory abnormality,
not sure whether this PR can fix the problem, but this will help locate
the problem and more elegant.
```
==6641==ERROR: AddressSanitizer: heap-use-after-free on address 0x603000980e70 at pc 0x559be21880be bp 0x7fd7cfc75070 sp 0x7fd7cfc75068
READ of size 8 at 0x603000980e70 thread T1453 (memory_maintena)
#0 0x559be21880bd in std::__atomic_base<long>::load(std::memory_order) const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/atomic_base.h:481:9
#1 0x559be21880bd in doris::MemTracker::MemCounter::current_value() const /home/zcp/repo_center/doris_master/doris/be/src/runtime/memory/mem_tracker.h:139:63
#2 0x559be21880bd in doris::MemTracker::consumption() const /home/zcp/repo_center/doris_master/doris/be/src/runtime/memory/mem_tracker.h:165:56
#3 0x559be3985133 in doris::MemTrackerLimiter::refresh_global_counter() /home/zcp/repo_center/doris_master/doris/be/src/runtime/memory/mem_tracker_limiter.cpp:245:59
#4 0x559bdfcd52d5 in doris::Daemon::memory_maintenance_thread() /home/zcp/repo_center/doris_master/doris/be/src/common/daemon.cpp:239:13
apache#5 0x559be40e6c17 in doris::Thread::supervise_thread(void*) /home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:498:5
apache#6 0x7fdf563e3ac2 in start_thread nptl/pthread_create.c:442:8
apache#7 0x7fdf5647584f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
0x603000980e74 is located 0 bytes after 20-byte region [0x603000980e60,0x603000980e74)
04:19:48 freed by thread T1491 (Scan_normal [wo) here:
04:19:48 #0 0x559bdfb0dd9d in operator delete(void*) (/mnt/hdd01/ci/master-deploy/be/lib/doris_be+0x33546d9d) (BuildId: f170e92ad3c55512)
04:19:48 #1 0x559bdfb1d42b in __gnu_cxx::new_allocator<char>::deallocate(char*, unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/ext/new_allocator.h:139:2
04:19:48 #2 0x559bdfb1d42b in std::allocator<char>::deallocate(char*, unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/allocator.h:187:27
04:19:48 #3 0x559bdfb1d42b in std::allocator_traits<std::allocator<char>>::deallocate(std::allocator<char>&, char*, unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:492:13
04:19:48 #4 0x559bdfb1d42b in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_destroy(unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:237:9
04:19:48 apache#5 0x559bdfb1d42b in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_dispose() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:232:4
04:19:48 apache#6 0x559bdfb1d42b in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::~basic_string() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:658:9
04:19:48 apache#7 0x559bdfb22501 in void std::destroy_at<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:88:15
04:19:48 apache#8 0x559bdfb22501 in void std::_Destroy<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:138:7
04:19:48 apache#9 0x559bdfb22501 in void std::_Destroy_aux<false>::__destroy<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:152:6
04:19:48 apache#10 0x559bdfb22501 in void std::_Destroy<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:184:7
04:19:48 apache#11 0x559bdfb22501 in void std::_Destroy<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:746:7
04:19:48 apache#12 0x559bdfb22501 in std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>>::~vector() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:680:2
04:19:48 apache#13 0x559be5515262 in doris::TCondition::~TCondition() /home/zcp/repo_center/doris_master/doris/gensrc/build/gen_cpp/PaloInternalService_types.cpp:9867:1
04:19:48 apache#14 0x559be5515262 in doris::TCondition::~TCondition() /home/zcp/repo_center/doris_master/doris/gensrc/build/gen_cpp/PaloInternalService_types.cpp:9866:36
04:19:48 apache#15 0x559be0047ee6 in doris::Status doris::DeleteHandler::_parse_column_pred<doris::DeleteSubPredicatePB>(std::shared_ptr<doris::TabletSchema>, std::shared_ptr<doris::TabletSchema>, google::protobuf::RepeatedPtrField<doris::DeleteSubPredicatePB> const&, doris::DeleteConditions*) /home/zcp/repo_center/doris_master/doris/be/src/olap/delete_handler.cpp:372:5
04:19:48 apache#16 0x559be003ba59 in doris::DeleteHandler::init(std::shared_ptr<doris::TabletSchema>, std::vector<std::shared_ptr<doris::RowsetMeta>, std::allocator<std::shared_ptr<doris::RowsetMeta>>> const&, long, bool) /home/zcp/repo_center/doris_master/doris/be/src/olap/delete_handler.cpp:404:13
04:19:48 apache#17 0x559be30ed6e3 in doris::TabletReader::_init_delete_condition(doris::TabletReader::ReaderParams const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/tablet_reader.cpp:636:28
04:19:48 apache#18 0x559be30e23d1 in doris::TabletReader::_init_params(doris::TabletReader::ReaderParams const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/tablet_reader.cpp:290:18
04:19:48 apache#19 0x559be30e1592 in doris::TabletReader::init(doris::TabletReader::ReaderParams const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/tablet_reader.cpp:124:18
04:19:48 apache#20 0x559c129840c8 in doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams const&) /home/zcp/repo_center/doris_master/doris/be/src/vec/olap/block_reader.cpp:210:5
04:19:48 apache#21 0x559c15a4ea09 in doris::vectorized::NewOlapScanner::open(doris::RuntimeState*) /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:232:32
04:19:48 apache#22 0x559bfb4acd94 in doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>) /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:236:5
04:19:48 apache#23 0x559bfb4b10be in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:176:21
04:19:48 apache#24 0x559bfb4b10be in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()::operator()() const /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:175:31
04:19:48 apache#25 0x559bfb4b10be in void std::__invoke_impl<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&>(std::__invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
04:19:48 apache#26 0x559bfb4b10be in std::enable_if<is_invocable_r_v<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&>, void>::type std::__invoke_r<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2
04:19:48 apache#27 0x559bfb4b10be in std::_Function_handler<void (), doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()>::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9
04:19:48 apache#28 0x559be410ed1b in doris::ThreadPool::dispatch_thread() /home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:543:24
04:19:48 apache#29 0x559be40e6c17 in doris::Thread::supervise_thread(void*) /home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:498:5
04:19:48 apache#30 0x7fdf563e3ac2 in start_thread nptl/pthread_create.c:442:8
04:19:48
04:19:48 previously allocated by thread T1491 (Scan_normal [wo) here:
04:19:48 #0 0x559bdfb0d53d in operator new(unsigned long) (/mnt/hdd01/ci/master-deploy/be/lib/doris_be+0x3354653d) (BuildId: f170e92ad3c55512)
04:19:48 #1 0x559bdfb33f74 in void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_construct<char*>(char*, char*, std::forward_iterator_tag) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.tcc:219:14
04:19:48 #2 0x559bdfb32705 in void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_construct_aux<char*>(char*, char*, std::__false_type) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:247:11
04:19:48 #3 0x559bdfb32705 in void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_construct<char*>(char*, char*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:266:4
04:19:48 #4 0x559bdfb32705 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::basic_string(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:451:9
04:19:48 apache#5 0x559bdfc326e6 in decltype(::new((void*)(0)) std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>(std::declval<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&>())) std::construct_at<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:97:39
04:19:48 apache#6 0x559bdfc326e6 in void std::allocator_traits<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>>::construct<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&>(std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:514:4
04:19:48 apache#7 0x559bdfc326e6 in void std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>>::_M_realloc_insert<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&>(__gnu_cxx::__normal_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>>>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/vector.tcc:449:4
04:19:48 apache#8 0x559be0032206 in doris::DeleteHandler::parse_condition(doris::DeleteSubPredicatePB const&, doris::TCondition*) /home/zcp/repo_center/doris_master/doris/be/src/olap/delete_handler.cpp:300:33
04:19:48 apache#9 0x559be0047aa3 in doris::Status doris::DeleteHandler::_parse_column_pred<doris::DeleteSubPredicatePB>(std::shared_ptr<doris::TabletSchema>, std::shared_ptr<doris::TabletSchema>, google::protobuf::RepeatedPtrField<doris::DeleteSubPredicatePB> const&, doris::DeleteConditions*) /home/zcp/repo_center/doris_master/doris/be/src/olap/delete_handler.cpp:355:9
04:19:48 apache#10 0x559be003ba59 in doris::DeleteHandler::init(std::shared_ptr<doris::TabletSchema>, std::vector<std::shared_ptr<doris::RowsetMeta>, std::allocator<std::shared_ptr<doris::RowsetMeta>>> const&, long, bool) /home/zcp/repo_center/doris_master/doris/be/src/olap/delete_handler.cpp:404:13
04:19:48 apache#11 0x559be30ed6e3 in doris::TabletReader::_init_delete_condition(doris::TabletReader::ReaderParams const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/tablet_reader.cpp:636:28
04:19:48 apache#12 0x559be30e23d1 in doris::TabletReader::_init_params(doris::TabletReader::ReaderParams const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/tablet_reader.cpp:290:18
04:19:48 apache#13 0x559be30e1592 in doris::TabletReader::init(doris::TabletReader::ReaderParams const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/tablet_reader.cpp:124:18
04:19:48 apache#14 0x559c129840c8 in doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams const&) /home/zcp/repo_center/doris_master/doris/be/src/vec/olap/block_reader.cpp:210:5
04:19:48 apache#15 0x559c15a4ea09 in doris::vectorized::NewOlapScanner::open(doris::RuntimeState*) /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:232:32
04:19:48 apache#16 0x559bfb4acd94 in doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>) /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:236:5
04:19:48 apache#17 0x559bfb4b10be in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:176:21
04:19:48 apache#18 0x559bfb4b10be in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()::operator()() const /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:175:31
04:19:48 apache#19 0x559bfb4b10be in void std::__invoke_impl<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&>(std::__invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
04:19:48 apache#20 0x559bfb4b10be in std::enable_if<is_invocable_r_v<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&>, void>::type std::__invoke_r<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2
04:19:48 apache#21 0x559bfb4b10be in std::_Function_handler<void (), doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()>::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9
04:19:48 apache#22 0x559be410ed1b in doris::ThreadPool::dispatch_thread() /home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:543:24
04:19:48 apache#23 0x559be40e6c17 in doris::Thread::supervise_thread(void*) /home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:498:5
04:19:48 apache#24 0x7fdf563e3ac2 in start_thread nptl/pthread_create.c:442:8
```
wyxxxcat
pushed a commit
that referenced
this pull request
Sep 19, 2024
…0857) ## Proposed changes ### Issue ``` /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1046:9: runtime error: reference binding to null pointer of type 'doris::StringRef' #0 0x55ee63eb0418 in std::vector<doris::StringRef, std::allocator<doris::StringRef>>::operator[](unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1046:2 #1 0x55ee63eb0418 in doris::Status doris::vectorized::OrcReader::_decode_string_non_dict_encoded_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn> const&, orc::TypeKind const&, orc::EncodedStringVectorBatch*, unsigned long) /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1172:39 #2 0x55ee63ea2685 in doris::Status doris::vectorized::OrcReader::_decode_string_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn> const&, orc::TypeKind const&, orc::ColumnVectorBatch*, unsigned long) /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1124:16 #3 0x55ee63e97e7a in doris::Status doris::vectorized::OrcReader::_fill_doris_data_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>&, std::shared_ptr<doris::vectorized::IDataType const> const&, orc::Type const*, orc::ColumnVectorBatch*, unsigned long) /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1365:16 #4 0x55ee63b0e450 in doris::Status doris::vectorized::OrcReader::_orc_column_to_doris_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, std::shared_ptr<doris::vectorized::IDataType const> const&, orc::Type const*, orc::ColumnVectorBatch*, unsigned long) /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1532:5 apache#5 0x55ee63e99622 in doris::Status doris::vectorized::OrcReader::_fill_doris_data_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>&, std::shared_ptr<doris::vectorized::IDataType const> const&, orc::Type const*, orc::ColumnVectorBatch*, unsigned long) /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1410:9 apache#6 0x55ee63b0e450 in doris::Status doris::vectorized::OrcReader::_orc_column_to_doris_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, std::shared_ptr<doris::vectorized::IDataType const> const&, orc::Type const*, orc::ColumnVectorBatch*, unsigned long) /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1532:5 apache#7 0x55ee63ad4f86 in doris::vectorized::OrcReader::get_next_block_impl(doris::vectorized::Block*, unsigned long*, bool*) /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1714:13 apache#8 0x55ee63ad093b in doris::vectorized::OrcReader::get_next_block(doris::vectorized::Block*, unsigned long*, bool*) /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1547:5 ``` ### Solution [Fix] (orc-reader) Fix StringRef nullptr data in orc-reader. When string is empty in orc row batch, the data can point anything, maybe nullptr, StringRef has undefined behavior when data is nullptr. Related with apache#37845.
wyxxxcat
pushed a commit
that referenced
this pull request
Oct 8, 2024
…o null pointer (apache#41486) ## Proposed changes ``` /root/doris/be/src/vec/common/pod_array.h:406:13: runtime error: applying non-zero offset 16 to null pointer #0 0x55827f4bbf0c in void doris::vectorized::PODArray<__int128, 4096ul, Allocator<false, false, false, DefaultMemoryAllocator>, 0ul, 0ul>::push_back<__int128&>(__int128&) /root/doris/be/src/vec/common/pod_array.h:406:13 #1 0x55827f4baa40 in doris::Counts<__int128>::increment(__int128) /root/doris/be/src/util/counts.h:160:36 #2 0x55827f4baa40 in doris::vectorized::PercentileState<__int128>::add(__int128, doris::vectorized::PODArray<double, 4096ul, Allocator<false, false, false, DefaultMemoryAllocator>, 16ul, 15ul> const&, int) /root/doris/be/src/vec/aggregate_functions/aggregate_function_percentile.h:634:27 #3 0x55827f6ca40f in doris::vectorized::AggregateFunctionNullVariadicInline<doris::vectorized::AggregateFunctionPercentileArray<__int128>, false>::add(char*, doris::vectorized::IColumn const**, long, doris::vectorized::Arena*) const /root/doris/be/src/vec/aggregate_functions/aggregate_function_null.h:324:32 #4 0x55827f6cef0e in doris::vectorized::IAggregateFunctionHelper<doris::vectorized::AggregateFunctionNullVariadicInline<doris::vectorized::AggregateFunctionPercentileArray<__int128>, false> >::add_batch(unsigned long, char**, unsigned long, doris::vectorized::IColumn const**, doris::vectorized::Arena*, bool) const /root/doris/be/src/vec/aggregate_functions/aggregate_function.h:273:22 apache#5 0x5582a0c41059 in doris::vectorized::AggFnEvaluator::execute_batch_add(doris::vectorized::Block*, unsigned long, char**, doris::vectorized::Arena*, bool) /root/doris/be/src/vec/exprs/vectorized_agg_fn.cpp:241:16 apache#6 0x5582a0c39037 in doris::Status doris::pipeline::AggSinkLocalState::_execute_with_serialized_key_helper<false>(doris::vectorized::Block*)::'lambda'()::operator()() const /root/doris/be/src/pipeline/exec/aggregation_sink_operator.cpp:481:17 apache#7 0x5582a0b51e1e in doris::Status doris::pipeline::AggSinkLocalState::_execute_with_serialized_key_helper<false>(doris::vectorized::Block*) /root/doris/be/src/pipeline/exec/aggregation_sink_operator.cpp:497:13 apache#8 0x5582a0a014e1 in doris::pipeline::AggSinkLocalState::_execute_with_serialized_key(doris::vectorized::Block*) /root/doris/be/src/pipeline/exec/aggregation_sink_operator.cpp:437:16 apache#9 0x5582a0c37811 in doris::pipeline::AggSinkLocalState::Executor<false, false>::execute(doris::pipeline::AggSinkLocalState*, doris::vectorized::Block*) /root/doris/be/src/pipeline/exec/aggregation_sink_operator.h:61:41 ``` <!--Describe your changes.-->
wyxxxcat
pushed a commit
that referenced
this pull request
Oct 11, 2024
…apache#41495) ## Proposed changes Fix problem "Runtime Error: Null pointer passed to 'StringTypeInvertedIndexReader::query', which requires a non-null argument." ``` Stack Trace: #0 doris::segment_v2::StringTypeInvertedIndexReader::query(...) inverted_index_reader.cpp:473 #1 doris::segment_v2::InvertedIndexIterator::read_from_inverted_index(...) inverted_index_reader.cpp:1237 #2 doris::vectorized::FunctionArraysOverlap::evaluate_inverted_index(...) function_arrays_overlap.h:192 #3 doris::vectorized::DefaultFunction::evaluate_inverted_index(...) function.h:532 #4 doris::vectorized::VExpr::_evaluate_inverted_index(...) vexpr.cpp:708 apache#5 doris::vectorized::VectorizedFnCall::evaluate_inverted_index(...) vectorized_fn_call.cpp:143 apache#6 doris::vectorized::VExprContext::evaluate_inverted_index(...) vexpr_context.cpp:126 ```
wyxxxcat
pushed a commit
that referenced
this pull request
Oct 14, 2024
…#41743) ==40376==ERROR: AddressSanitizer: heap-use-after-free on address 0x616002419ed0 at pc 0x56132d2dc34a bp 0x7fe20150ba70 sp 0x7fe20150ba68 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27557_27557&logView=flowAware) WRITE of size 4 at 0x616002419ed0 thread T1474 (Pipe_normal [wo) [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27558_27558&logView=flowAware) #0 0x56132d2dc349 in doris::pipeline::PipelineTask::update_queue_level(int) /root/doris/be/src/pipeline/pipeline_task.h:174:67 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27559_27559&logView=flowAware) #1 0x56132d2dc349 in doris::pipeline::PriorityTaskQueue::_try_take_unprotected(bool) /root/doris/be/src/pipeline/task_queue.cpp:79:15 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27560_27560&logView=flowAware) #2 0x56132d2dc615 in doris::pipeline::PriorityTaskQueue::try_take(bool) /root/doris/be/src/pipeline/task_queue.cpp:97:12 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27561_27561&logView=flowAware) #3 0x56132d2de69f in doris::pipeline::MultiCoreTaskQueue::take(int) /root/doris/be/src/pipeline/task_queue.cpp:164:50 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27562_27562&logView=flowAware) #4 0x56132d2ee0e3 in doris::pipeline::TaskScheduler::_do_work(unsigned long) /root/doris/be/src/pipeline/task_scheduler.cpp:102:35 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27563_27563&logView=flowAware) apache#5 0x5612fc23029d in doris::ThreadPool::dispatch_thread() /root/doris/be/src/util/threadpool.cpp:543:24 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27564_27564&logView=flowAware) apache#6 0x5612fc20880e in std::function<void ()>::operator()() const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27565_27565&logView=flowAware) apache#7 0x5612fc20880e in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:498:5 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27566_27566&logView=flowAware) apache#8 0x7fe9405ca608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477:8 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27567_27567&logView=flowAware) apache#9 0x7fe940877132 in __clone /build/glibc-SzIz7B/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27568_27568&logView=flowAware) [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27569_27569&logView=flowAware) 0x616002419ed0 is located 80 bytes inside of 632-byte region [0x616002419e80,0x61600241a0f8) [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27570_27570&logView=flowAware) freed by thread T1463 (Pipe_normal [wo) here: [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27571_27571&logView=flowAware) #0 0x5612f7ac280d in operator delete(void*) (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x2ed1180d) (BuildId: 04f75ee9f41a400a) [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27572_27572&logView=flowAware) #1 0x56132d2519d4 in std::default_delete<doris::pipeline::PipelineTask>::operator()(doris::pipeline::PipelineTask*) const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:85:2 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27573_27573&logView=flowAware) #2 0x56132d2519d4 in std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >::~unique_ptr() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:361:4 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27574_27574&logView=flowAware) #3 0x56132d2519d4 in void std::destroy_at<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > >(std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:88:15 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27575_27575&logView=flowAware) #4 0x56132d2519d4 in void std::_Destroy<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > >(std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:138:7 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27576_27576&logView=flowAware) apache#5 0x56132d2519d4 in void std::_Destroy_aux<false>::__destroy<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >*>(std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >*, std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:152:6 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27577_27577&logView=flowAware) apache#6 0x56132d2519d4 in void std::_Destroy<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >*>(std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >*, std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:184:7 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27578_27578&logView=flowAware) apache#7 0x56132d2519d4 in void std::_Destroy<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >*, std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > >(std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >*, std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >*, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > >&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:746:7 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27579_27579&logView=flowAware) apache#8 0x56132d2519d4 in std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >::~vector() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:680:2 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27580_27580&logView=flowAware) apache#9 0x56132d1d717c in void std::destroy_at<std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > > >(std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:88:15 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27581_27581&logView=flowAware) apache#10 0x56132d1d717c in void std::_Destroy<std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > > >(std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:138:7 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27582_27582&logView=flowAware) apache#11 0x56132d1d717c in void std::_Destroy_aux<false>::__destroy<std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >*>(std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >*, std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:152:6 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27583_27583&logView=flowAware) apache#12 0x56132d1d717c in void std::_Destroy<std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >*>(std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >*, std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:184:7 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27584_27584&logView=flowAware) apache#13 0x56132d1d717c in void std::_Destroy<std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >*, std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > > >(std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >*, std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >*, std::allocator<std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > > >&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:746:7 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27585_27585&logView=flowAware) apache#14 0x56132d1d717c in std::vector<std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >, std::allocator<std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > > > >::_M_erase_at_end(std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1796:6 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27586_27586&logView=flowAware) apache#15 0x56132d1d717c in std::vector<std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >, std::allocator<std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > > > >::clear() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1499:9 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27587_27587&logView=flowAware) apache#16 0x56132d1d717c in doris::pipeline::PipelineFragmentContext::~PipelineFragmentContext() /root/doris/be/src/pipeline/pipeline_fragment_context.cpp:143:12 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27588_27588&logView=flowAware) apache#17 0x5612f7aeabdc in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:168:6 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27589_27589&logView=flowAware) apache#18 0x56132d2ed705 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:702:11 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27590_27590&logView=flowAware) apache#19 0x56132d2ed705 in std::__shared_ptr<doris::TaskExecutionContext, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1149:31 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27591_27591&logView=flowAware) apache#20 0x56132d2ed705 in doris::pipeline::_close_task(doris::pipeline::PipelineTask*, doris::Status) /root/doris/be/src/pipeline/task_scheduler.cpp:98:1 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27592_27592&logView=flowAware) apache#21 0x56132d2efce9 in doris::pipeline::TaskScheduler::_do_work(unsigned long) /root/doris/be/src/pipeline/task_scheduler.cpp:181:17 [02:02:28 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/542356?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true&showLog=542356_27593_27593&logView=flowAware) apache#22 0x5612fc23029d in doris::ThreadPool::dispatch_thread() /root/doris/be/src/util/threadpool.cpp:543:24
wyxxxcat
pushed a commit
that referenced
this pull request
Oct 22, 2024
…dir when index file writer open index (apache#42207) ## Proposed changes Fix UAF as below ``` ==16442==ERROR: AddressSanitizer: heap-use-after-free on address 0x60f0008718a0 at pc 0x5586663cea7a bp 0x7f02673d0530 sp 0x7f02673d0528 READ of size 8 at 0x60f0008718a0 thread T490 (CumuCompactionT) #0 0x5586663cea79 in doris::segment_v2::InvertedIndexColumnWriterImpl<(doris::FieldType)5>::finish() /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:549:63 #1 0x5586663360bc in doris::segment_v2::ScalarColumnWriter::write_inverted_index() /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/column_writer.cpp:640:41 #2 0x5586662ce160 in doris::segment_v2::SegmentWriter::_write_inverted_index() /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:1435:9 #3 0x5586662ccb13 in doris::segment_v2::SegmentWriter::finalize_columns_index(unsigned long*) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:1317:5 #4 0x55866653c074 in doris::VerticalBetaRowsetWriter::_flush_columns(doris::segment_v2::SegmentWriter*, bool) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/vertical_beta_rowset_writer.cpp:127:5 apache#5 0x558666534e3f in doris::VerticalBetaRowsetWriter::flush_columns(bool) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/vertical_beta_rowset_writer.cpp:153:5 apache#6 0x558665a8d8b2 in doris::Merger::vertical_compact_one_group(std::shared_ptr, doris::ReaderType, doris::TabletSchema const&, bool, std::vector> const&, doris::vectorized::RowSourcesBuffer*, std::vector, std::allocator>> const&, doris::RowsetWriter*, long, doris::Merger::Statistics*, std::vector>, long, doris::CompactionSampleInfo*) /home/zcp/repo_center/doris_master/doris/be/src/olap/merger.cpp:333:5 apache#7 0x558665a93190 in doris::Merger::vertical_merge_rowsets(std::shared_ptr, doris::ReaderType, doris::TabletSchema const&, std::vector, std::allocator>> const&, doris::RowsetWriter*, long, long, doris::Merger::Statistics*) /home/zcp/repo_center/doris_master/doris/be/src/olap/merger.cpp:471:21 apache#8 0x558665a00fd7 in doris::Compaction::merge_input_rowsets() /home/zcp/repo_center/doris_master/doris/be/src/olap/compaction.cpp:192:19 apache#9 0x558665a329a3 in doris::CloudCompactionMixin::execute_compact_impl(long) /home/zcp/repo_center/doris_master/doris/be/src/olap/compaction.cpp:1158:5 apache#10 0x558665a33381 in doris::CloudCompactionMixin::execute_compact() /home/zcp/repo_center/doris_master/doris/be/src/olap/compaction.cpp:1173:5 apache#11 0x55869a893d89 in doris::CloudCumulativeCompaction::execute_compact() /home/zcp/repo_center/doris_master/doris/be/src/cloud/cloud_cumulative_compaction.cpp:191:38 apache#12 0x55869a861fac in doris::CloudStorageEngine::_submit_cumulative_compaction_task(std::shared_ptr const&)::$_1::operator()() const /home/zcp/repo_center/doris_master/doris/be/src/cloud/cloud_storage_engine.cpp:693:31 apache#13 0x55869a861fac in void std::__invoke_impl const&)::$_1&>(std::__invoke_other, doris::CloudStorageEngine::_submit_cumulative_compaction_task(std::shared_ptr const&)::$_1&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 apache#14 0x55869a861fac in std::enable_if const&)::$_1&>, void>::type std::__invoke_r const&)::$_1&>(doris::CloudStorageEngine::_submit_cumulative_compaction_task(std::shared_ptr const&)::$_1&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 apache#15 0x55869a861fac in std::_Function_handler const&)::$_1>::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 apache#16 0x5586679033fb in doris::ThreadPool::dispatch_thread() /home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:543:24 apache#17 0x5586678db057 in doris::Thread::supervise_thread(void*) /home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:498:5 apache#18 0x7f0639d7bac2 in start_thread nptl/pthread_create.c:442:8 apache#19 0x7f0639e0d84f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 0x60f0008718a0 is located 0 bytes inside of 168-byte region [0x60f0008718a0,0x60f000871948) freed by thread T490 (CumuCompactionT) here: #0 0x558663259d9d in operator delete(void*) (/mnt/hdd01/ci/master-deploy/cluster0/be/lib/doris_be+0x337a1d9d) (BuildId: fa8094411569cb8d) #1 0x558665a655eb in std::default_delete::operator()(lucene::store::Directory*) const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:85:2 #2 0x558665a655eb in std::unique_ptr>::~unique_ptr() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:361:4 #3 0x558665f83dfe in std::pair, std::allocator>> const, std::unique_ptr>>::~pair() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_iterator.h:2379:12 #4 0x558665f83dfe in void std::destroy_at, std::allocator>> const, std::unique_ptr>>>(std::pair, std::allocator>> const, std::unique_ptr>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:88:15 apache#5 0x558665f83dfe in void std::allocator_traits, std::allocator>> const, std::unique_ptr>>>>>::destroy, std::allocator>> const, std::unique_ptr>>>(std::allocator, std::allocator>> const, std::unique_ptr>>>>&, std::pair, std::allocator>> const, std::unique_ptr>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:533:4 apache#6 0x558665f83dfe in std::_Rb_tree, std::allocator>>, std::pair, std::allocator>> const, std::unique_ptr>>, std::_Select1st, std::allocator>> const, std::unique_ptr>>>, std::less, std::allocator>>>, std::allocator, std::allocator>> const, std::unique_ptr>>>>::_M_destroy_node(std::_Rb_tree_node, std::allocator>> const, std::unique_ptr>>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_tree.h:623:2 apache#7 0x558665f83dfe in std::_Rb_tree, std::allocator>>, std::pair, std::allocator>> const, std::unique_ptr>>, std::_Select1st, std::allocator>> const, std::unique_ptr>>>, std::less, std::allocator>>>, std::allocator, std::allocator>> const, std::unique_ptr>>>>::_M_drop_node(std::_Rb_tree_node, std::allocator>> const, std::unique_ptr>>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_tree.h:631:2 apache#8 0x558665f9e9b8 in std::pair, std::allocator>> const, std::unique_ptr>>>, bool> std::_Rb_tree, std::allocator>>, std::pair, std::allocator>> const, std::unique_ptr>>, std::_Select1st, std::allocator>> const, std::unique_ptr>>>, std::less, std::allocator>>>, std::allocator, std::allocator>> const, std::unique_ptr>>>>::_M_emplace_unique, std::allocator>>, std::unique_ptr>>(std::pair, std::allocator>>&&, std::unique_ptr>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_tree.h:2391:6 apache#9 0x558665f86e29 in std::pair, std::allocator>> const, std::unique_ptr>>>, bool> std::map, std::allocator>>, std::unique_ptr>, std::less, std::allocator>>>, std::allocator, std::allocator>> const, std::unique_ptr>>>>::emplace, std::allocator>>, std::unique_ptr>>(std::pair, std::allocator>>&&, std::unique_ptr>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_map.h:577:16 apache#10 0x558665f86e29 in doris::segment_v2::InvertedIndexFileWriter::open(doris::TabletIndex const*) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_file_writer.cpp:60:19 apache#11 0x5586663d0423 in doris::segment_v2::InvertedIndexColumnWriterImpl<(doris::FieldType)5>::open_index_directory() /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:160:16 apache#12 0x5586663d00f7 in doris::segment_v2::InvertedIndexColumnWriterImpl<(doris::FieldType)5>::init_bkd_index() /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:146:16 apache#13 0x5586663cb2d9 in doris::segment_v2::InvertedIndexColumnWriterImpl<(doris::FieldType)5>::init() /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:107:24 apache#14 0x5586663b1201 in doris::segment_v2::InvertedIndexColumnWriter::create(doris::Field const*, std::unique_ptr>*, doris::segment_v2::InvertedIndexFileWriter*, doris::TabletIndex const*) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:697:27 apache#15 0x55866632cf8d in doris::segment_v2::ScalarColumnWriter::init() /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/column_writer.cpp:483:13 apache#16 0x5586662a5e52 in doris::segment_v2::SegmentWriter::_create_column_writer(unsigned int, doris::TabletColumn const&, std::shared_ptr const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:267:5 apache#17 0x5586662a71b4 in doris::segment_v2::SegmentWriter::_create_writers(std::shared_ptr const&, std::vector> const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:316:9 apache#18 0x5586662a2be3 in doris::segment_v2::SegmentWriter::init(std::vector> const&, bool) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:285:5 apache#19 0x5586665333b2 in doris::VerticalBetaRowsetWriter::add_columns(doris::vectorized::Block const*, std::vector> const&, bool, unsigned int) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/vertical_beta_rowset_writer.cpp:96:17 apache#20 0x558665a8cb9b in doris::Merger::vertical_compact_one_group(std::shared_ptr, doris::ReaderType, doris::TabletSchema const&, bool, std::vector> const&, doris::vectorized::RowSourcesBuffer*, std::vector, std::allocator>> const&, doris::RowsetWriter*, long, doris::Merger::Statistics*, std::vector>, long, doris::CompactionSampleInfo*) /home/zcp/repo_center/doris_master/doris/be/src/olap/merger.cpp:309:9 apache#21 0x558665a93190 in doris::Merger::vertical_merge_rowsets(std::shared_ptr, doris::ReaderType, doris::TabletSchema const&, std::vector, std::allocator>> const&, doris::RowsetWriter*, long, long, doris::Merger::Statistics*) /home/zcp/repo_center/doris_master/doris/be/src/olap/merger.cpp:471:21 apache#22 0x558665a00fd7 in doris::Compaction::merge_input_rowsets() /home/zcp/repo_center/doris_master/doris/be/src/olap/compaction.cpp:192:19 apache#23 0x558665a329a3 in doris::CloudCompactionMixin::execute_compact_impl(long) /home/zcp/repo_center/doris_master/doris/be/src/olap/compaction.cpp:1158:5 apache#24 0x558665a33381 in doris::CloudCompactionMixin::execute_compact() /home/zcp/repo_center/doris_master/doris/be/src/olap/compaction.cpp:1173:5 apache#25 0x55869a893d89 in doris::CloudCumulativeCompaction::execute_compact() /home/zcp/repo_center/doris_master/doris/be/src/cloud/cloud_cumulative_compaction.cpp:191:38 apache#26 0x55869a861fac in doris::CloudStorageEngine::_submit_cumulative_compaction_task(std::shared_ptr const&)::$_1::operator()() const /home/zcp/repo_center/doris_master/doris/be/src/cloud/cloud_storage_engine.cpp:693:31 apache#27 0x55869a861fac in void std::__invoke_impl const&)::$_1&>(std::__invoke_other, doris::CloudStorageEngine::_submit_cumulative_compaction_task(std::shared_ptr const&)::$_1&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 apache#28 0x55869a861fac in std::enable_if const&)::$_1&>, void>::type std::__invoke_r const&)::$_1&>(doris::CloudStorageEngine::_submit_cumulative_compaction_task(std::shared_ptr const&)::$_1&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 apache#29 0x55869a861fac in std::_Function_handler const&)::$_1>::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 apache#30 0x5586679033fb in doris::ThreadPool::dispatch_thread() /home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:543:24 apache#31 0x5586678db057 in doris::Thread::supervise_thread(void*) /home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:498:5 apache#32 0x7f0639d7bac2 in start_thread nptl/pthread_create.c:442:8 previously allocated by thread T490 (CumuCompactionT) here: #0 0x55866325953d in operator new(unsigned long) (/mnt/hdd01/ci/master-deploy/cluster0/be/lib/doris_be+0x337a153d) (BuildId: fa8094411569cb8d) #1 0x558665fb7d7b in doris::segment_v2::DorisFSDirectoryFactory::getDirectory(std::shared_ptr const&, char const*, bool, lucene::store::LockFactory*) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_fs_directory.cpp:797:15 #2 0x558665f86b5c in doris::segment_v2::InvertedIndexFileWriter::open(doris::TabletIndex const*) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_file_writer.cpp:58:17 #3 0x5586663d0423 in doris::segment_v2::InvertedIndexColumnWriterImpl<(doris::FieldType)5>::open_index_directory() /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:160:16 #4 0x5586663d00f7 in doris::segment_v2::InvertedIndexColumnWriterImpl<(doris::FieldType)5>::init_bkd_index() /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:146:16 apache#5 0x5586663cb2d9 in doris::segment_v2::InvertedIndexColumnWriterImpl<(doris::FieldType)5>::init() /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:107:24 apache#6 0x5586663b1201 in doris::segment_v2::InvertedIndexColumnWriter::create(doris::Field const*, std::unique_ptr>*, doris::segment_v2::InvertedIndexFileWriter*, doris::TabletIndex const*) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:697:27 apache#7 0x55866632cf8d in doris::segment_v2::ScalarColumnWriter::init() /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/column_writer.cpp:483:13 apache#8 0x5586662a5e52 in doris::segment_v2::SegmentWriter::_create_column_writer(unsigned int, doris::TabletColumn const&, std::shared_ptr const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:267:5 apache#9 0x5586662a71b4 in doris::segment_v2::SegmentWriter::_create_writers(std::shared_ptr const&, std::vector> const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:316:9 apache#10 0x5586662a2be3 in doris::segment_v2::SegmentWriter::init(std::vector> const&, bool) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:285:5 apache#11 0x5586665333b2 in doris::VerticalBetaRowsetWriter::add_columns(doris::vectorized::Block const*, std::vector> const&, bool, unsigned int) /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/vertical_beta_rowset_writer.cpp:96:17 apache#12 0x558665a8cb9b in doris::Merger::vertical_compact_one_group(std::shared_ptr, doris::ReaderType, doris::TabletSchema const&, bool, std::vector> const&, doris::vectorized::RowSourcesBuffer*, std::vector, std::allocator>> const&, doris::RowsetWriter*, long, doris::Merger::Statistics*, std::vector>, long, doris::CompactionSampleInfo*) /home/zcp/repo_center/doris_master/doris/be/src/olap/merger.cpp:309:9 apache#13 0x558665a93190 in doris::Merger::vertical_merge_rowsets(std::shared_ptr, doris::ReaderType, doris::TabletSchema const&, std::vector, std::allocator>> const&, doris::RowsetWriter*, long, long, doris::Merger::Statistics*) /home/zcp/repo_center/doris_master/doris/be/src/olap/merger.cpp:471:21 apache#14 0x558665a00fd7 in doris::Compaction::merge_input_rowsets() /home/zcp/repo_center/doris_master/doris/be/src/olap/compaction.cpp:192:19 apache#15 0x558665a329a3 in doris::CloudCompactionMixin::execute_compact_impl(long) /home/zcp/repo_center/doris_master/doris/be/src/olap/compaction.cpp:1158:5 apache#16 0x558665a33381 in doris::CloudCompactionMixin::execute_compact() /home/zcp/repo_center/doris_master/doris/be/src/olap/compaction.cpp:1173:5 apache#17 0x55869a893d89 in doris::CloudCumulativeCompaction::execute_compact() /home/zcp/repo_center/doris_master/doris/be/src/cloud/cloud_cumulative_compaction.cpp:191:38 apache#18 0x55869a861fac in doris::CloudStorageEngine::_submit_cumulative_compaction_task(std::shared_ptr const&)::$_1::operator()() const /home/zcp/repo_center/doris_master/doris/be/src/cloud/cloud_storage_engine.cpp:693:31 apache#19 0x55869a861fac in void std::__invoke_impl const&)::$_1&>(std::__invoke_other, doris::CloudStorageEngine::_submit_cumulative_compaction_task(std::shared_ptr const&)::$_1&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 apache#20 0x55869a861fac in std::enable_if const&)::$_1&>, void>::type std::__invoke_r const&)::$_1&>(doris::CloudStorageEngine::_submit_cumulative_compaction_task(std::shared_ptr const&)::$_1&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 apache#21 0x55869a861fac in std::_Function_handler const&)::$_1>::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 apache#22 0x5586679033fb in doris::ThreadPool::dispatch_thread() /home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:543:24 apache#23 0x5586678db057 in doris::Thread::supervise_thread(void*) /home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:498:5 apache#24 0x7f0639d7bac2 in start_thread nptl/pthread_create.c:442:8 ```
wyxxxcat
pushed a commit
that referenced
this pull request
Oct 29, 2024
…e#42666) ## Proposed changes Should not modify state of vexpr when executing the expr, since it may be executed in parallel. Issue introduced by apache#42136 ``` ==75397==ERROR: AddressSanitizer: heap-use-after-free on address 0x60700294b060 at pc 0x56507fb8a370 bp 0x7fc861cd7050 sp 0x7fc861cd7048 19:58:49 READ of size 8 at 0x60700294b060 thread T1463 (Pipe_normal [wo) 19:58:49 #0 0x56507fb8a36f in doris::vectorized::FunctionCoalesce::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/function_coalesce.cpp:163:13 19:58:49 #1 0x56507958edfe in doris::vectorized::DefaultExecutable::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/function.h:463:26 19:58:49 #2 0x56507dbcbf5c in doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp 19:58:49 #3 0x56507dbc65e3 in doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:245:12 19:58:49 #4 0x56507dbc683b in doris::vectorized::PreparedFunctionImpl::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:251:12 19:58:49 apache#5 0x56507958c81c in doris::vectorized::IFunctionBase::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.h:194:19 19:58:49 apache#6 0x565079578bee in doris::vectorized::VectorizedFnCall::_do_execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*, std::vector<unsigned long, std::allocator<unsigned long> >&) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:188:5 19:58:49 apache#7 0x56507957a3e4 in doris::vectorized::VectorizedFnCall::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:203:12 19:58:49 apache#8 0x56507964994a in doris::vectorized::VExprContext::execute(doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vexpr_context.cpp:55:5 19:58:49 apache#9 0x56508f8d7496 in doris::pipeline::OperatorXBase::do_projections(doris::RuntimeState*, doris::vectorized::Block*, doris::vectorized::Block*) const /root/doris/be/src/pipeline/exec/operator.cpp:294:13 19:58:49 apache#10 0x56508f8d96ba in doris::pipeline::OperatorXBase::get_block_after_projects(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/be/src/pipeline/exec/operator.cpp:335:18 19:58:49 apache#11 0x565093c2560b in doris::pipeline::PipelineTask::execute(bool*) /root/doris/be/src/pipeline/pipeline_task.cpp:374:13 19:58:49 apache#12 0x565093c6a164 in doris::pipeline::TaskScheduler::_do_work(int) /root/doris/be/src/pipeline/task_scheduler.cpp:138:9 19:58:49 apache#13 0x565062529d1d in doris::ThreadPool::dispatch_thread() /root/doris/be/src/util/threadpool.cpp:543:24 19:58:49 apache#14 0x5650625021ce in std::function<void ()>::operator()() const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9 19:58:49 apache#15 0x5650625021ce in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:498:5 19:58:49 apache#16 0x7fca1844a608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477:8 19:58:49 apache#17 0x7fca186f7132 in __clone /build/glibc-SzIz7B/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95 19:58:49 19:58:49 0x60700294b060 is located 16 bytes inside of 80-byte region [0x60700294b050,0x60700294b0a0) 19:58:49 freed by thread T1462 (Pipe_normal [wo) here: 19:58:49 #0 0x56505dd2f80d in operator delete(void*) (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x2f79e80d) (BuildId: 60cc2a41212f4e75) 19:58:49 #1 0x565079596a6f in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:702:11 19:58:49 #2 0x565079596a6f in std::__shared_ptr<doris::vectorized::IFunctionBase, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1149:31 19:58:49 #3 0x565079596a6f in std::__shared_ptr<doris::vectorized::IFunctionBase, (__gnu_cxx::_Lock_policy)2>::operator=(std::__shared_ptr<doris::vectorized::IFunctionBase, (__gnu_cxx::_Lock_policy)2>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1245:2 19:58:49 #4 0x56507fb89acd in std::shared_ptr<doris::vectorized::IFunctionBase>::operator=(std::shared_ptr<doris::vectorized::IFunctionBase>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr.h:385:27 19:58:49 apache#5 0x56507fb89acd in doris::vectorized::FunctionCoalesce::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/function_coalesce.cpp:89:30 19:58:49 apache#6 0x56507958edfe in doris::vectorized::DefaultExecutable::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/function.h:463:26 19:58:49 apache#7 0x56507dbcbf5c in doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp 19:58:49 apache#8 0x56507dbc65e3 in doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:245:12 19:58:49 apache#9 0x56507dbc683b in doris::vectorized::PreparedFunctionImpl::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:251:12 19:58:49 apache#10 0x56507958c81c in doris::vectorized::IFunctionBase::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.h:194:19 19:58:49 apache#11 0x565079578bee in doris::vectorized::VectorizedFnCall::_do_execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*, std::vector<unsigned long, std::allocator<unsigned long> >&) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:188:5 19:58:49 apache#12 0x56507957a3e4 in doris::vectorized::VectorizedFnCall::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:203:12 19:58:49 apache#13 0x56507964994a in doris::vectorized::VExprContext::execute(doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vexpr_context.cpp:55:5 19:58:49 apache#14 0x56508f8d7496 in doris::pipeline::OperatorXBase::do_projections(doris::RuntimeState*, doris::vectorized::Block*, doris::vectorized::Block*) const /root/doris/be/src/pipeline/exec/operator.cpp:294:13 19:58:49 apache#15 0x56508f8d96ba in doris::pipeline::OperatorXBase::get_block_after_projects(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/be/src/pipeline/exec/operator.cpp:335:18 19:58:49 apache#16 0x565093c2560b in doris::pipeline::PipelineTask::execute(bool*) /root/doris/be/src/pipeline/pipeline_task.cpp:374:13 ```
wyxxxcat
pushed a commit
that referenced
this pull request
Nov 7, 2024
…#42260) ## Proposed changes DeletePredicatePB should be DeleteSubPredicatePB. Test case is too ambiguous to add, since this bug is triggered by a huge random test and failed to find the minimal case. However, this fix is verified under the wild test that it does works. Note that this problem may be triggered by another bug, cuz schema in delete predicate rowset should contain column referred in delete condition. Even if we don't have this fix, this error should never happend. But this error occurred under wild tests, means that schema in delete predicate rowset is not adaptable with delete condition. I think it is under some status that delete operation use BE tablet schema rather than schema from FE, and the former rename operation result in that status. But I failed to add a test case to reproduce, and think that by no way will it happend occurding to the related code. ``` (1105, 'errCode = 2, detailMessage = ([172.20.50.7](http://172.20.50.7/))[INTERNAL_ERROR]failed to initialize storage reader. tablet=78026, res=[INTERNAL_ERROR]column not found, name=loc1, table_id=-1, schema_version=2 \t0# doris::TabletSchema::column(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const at /home/zcp/repo_center/doris_master/doris/be/src/common/status.h:375 \t1# doris::Status doris::DeleteHandler::_parse_column_pred<doris::DeleteSubPredicatePB>(std::shared_ptr<doris::TabletSchema>, std::shared_ptr<doris::TabletSchema>, google::protobuf::RepeatedPtrField<doris::DeleteSubPredicatePB> const&, doris::DeleteConditions*) at /home/zcp/repo_center/doris_master/doris/be/src/util/expected.hpp:1986 \t2# doris::DeleteHandler::init(std::shared_ptr<doris::TabletSchema>, std::vector<std::shared_ptr<doris::RowsetMeta>, std::allocator<std::shared_ptr<doris::RowsetMeta> > > const&, long) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:701 \t3# doris::TabletReader::_init_delete_condition(doris::TabletReader::ReaderParams const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:701 \t4# doris::TabletReader::_init_params(doris::TabletReader::ReaderParams const&) at /home/zcp/repo_center/doris_master/doris/be/src/common/status.h:499 \t5# doris::TabletReader::init(doris::TabletReader::ReaderParams const&) at /home/zcp/repo_center/doris_master/doris/be/src/common/status.h:499 \t6# doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams const&) at /home/zcp/repo_center/doris_master/doris/be/src/common/status.h:499 \t7# doris::vectorized::NewOlapScanner::open(doris::RuntimeState*) at /home/zcp/repo_center/doris_master/doris/be/src/common/status.h:499 \t8# doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>) at /home/zcp/repo_center/doris_master/doris/be/src/common/status.h:388 \t9# std::_Function_handler<void (), doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::{lambda()#1}>::_M_invoke(std::_Any_data const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:701 \t10# doris::ThreadPool::dispatch_thread() at /home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:0 \t11# doris::Thread::supervise_thread(void*) at /var/local/ldb-toolchain/bin/../usr/include/pthread.h:562 \t12# ? \t13# ? , backend=[172.20.50.7](http://172.20.50.7/)') ``` ```cpp auto tablet_schema = std::make_shared<TabletSchema>(); tablet_schema->copy_from(*tablet->tablet_schema()); if (!request.columns_desc.empty() && request.columns_desc[0].col_unique_id >= 0) { tablet_schema->clear_columns(); // TODO(lhy) handle variant for (const auto& column_desc : request.columns_desc) { tablet_schema->append_column(TabletColumn(column_desc)); } } RowsetSharedPtr rowset_to_add; // writes res = _convert_v2(tablet, &rowset_to_add, tablet_schema, push_type); if (!res.ok()) { LOG(WARNING) << "fail to convert tmp file when realtime push. res=" << res << ", failed to process realtime push." << ", tablet=" << tablet->tablet_id() << ", transaction_id=" << request.transaction_id; Status rollback_status = _engine.txn_manager()->rollback_txn(request.partition_id, *tablet, request.transaction_id); // has to check rollback status to ensure not delete a committed rowset if (rollback_status.ok()) { _engine.add_unused_rowset(rowset_to_add); } return res; } // add pending data to tablet if (push_type == PushType::PUSH_FOR_DELETE) { rowset_to_add->rowset_meta()->set_delete_predicate(std::move(del_preds.front())); del_preds.pop(); } ```
wyxxxcat
pushed a commit
that referenced
this pull request
Nov 7, 2024
…#39201) ## Proposed changes pick apache#39191 Undefined behavior occurs if there is a null value in the list. ``` /root/doris/be/src/vec/common/string_ref.h:271:54: runtime error: null pointer passed as argument 2, which is declared to never be null /var/local/ldb-toolchain/bin/../usr/include/string.h:64:33: note: nonnull attribute specified here #0 0x5616d072245d in doris::StringRef::eq(doris::StringRef const&) const /root/doris/be/src/vec/common/string_ref.h:271:41 #1 0x5616d072245d in doris::StringRef::operator==(doris::StringRef const&) const /root/doris/be/src/vec/common/string_ref.h:274:60 #2 0x5616d072245d in doris::FixedContainer::find(doris::StringRef const&) const /root/doris/be/src/exprs/hybrid_set.h:76:36 #3 0x5616d072245d in void doris::StringValueSet>::_find_batch(doris::vectorized::IColumn const&, unsigned long, doris::vectorized::PODArray, 16ul, 15ul> const*, doris::vectorized::PODArray, 16ul, 15ul>&) /root/doris/be/src/exprs/hybrid_set.h:688:63 #4 0x5616d0747857 in doris::vectorized::FunctionIn::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/in.h:170:21 apache#5 0x5616c741fa3a in doris::vectorized::DefaultExecutable::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/function.h:462:26 apache#6 0x5616cbb5b650 in doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp apache#7 0x5616cbb4e14e in doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:244:12 apache#8 0x5616cbb4e3c2 in doris::vectorized::PreparedFunctionImpl::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:250:12 apache#9 0x5616c741cd68 in doris::vectorized::IFunctionBase::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.h:190:19 apache#10 0x5616c74cf712 in doris::vectorized::VInPredicate::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vin_predicate.cpp:130:5 apache#11 0x5616c740d5c0 in doris::vectorized::VectorizedFnCall::_do_execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*, std::vector>&) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:183:9 apache#12 0x5616c740ecf5 in doris::vectorized::VectorizedFnCall::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:215:12 apache#13 0x5616c7462e24 in doris::vectorized::VCompoundPred::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vcompound_pred.h:127:38 apache#14 0x5616c74bccec in doris::vectorized::VExprContext::execute(doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vexpr_context.cpp:54:5 apache#15 0x5616c74c1dcc in doris::vectorized::VExprContext::execute_conjuncts(std::vector, std::allocator>> const&, std::vector, 16ul, 15ul>, std::allocator, 16ul, 15ul>>> const*, bool, doris::vectorized::Block*, doris::vectorized::PODArray, 16ul, 15ul>, bool) /root/doris/be/src/vec/exprs/vexpr_context.cpp:169:9 apache#16 0x5616c74c5108 in doris::vectorized::VExprContext::execute_conjuncts_and_filter_block(std::vector, std::allocator>> const&, doris::vectorized::Block*, std::vector>&, int, doris::vectorized::PODArray, 16ul, 15ul>&) /root/doris/be/src/vec/exprs/vexpr_context.cpp:322:5 apache#17 0x5616ad8a7f1a in doris::segment_v2::SegmentIterator::_execute_common_expr(unsigned short*, unsigned short&, doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2680:5 apache#18 0x5616ad89e86e in doris::segment_v2::SegmentIterator::_next_batch_internal(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2582:25 apache#19 0x5616ad892f5c in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*)::$_0::operator()() const /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2315:9 apache#20 0x5616ad892f5c in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2314:19 apache#21 0x5616ad6dd9cc in doris::segment_v2::LazyInitSegmentIterator::next_batch(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/lazy_init_segment_iterator.h:44:33 apache#22 0x5616ad269d67 in doris::BetaRowsetReader::next_block(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/beta_rowset_reader.cpp:380:29 apache#23 0x5616de6de110 in doris::vectorized::VCollectIterator::Level0Iterator::_refresh() /root/doris/be/src/vec/olap/vcollect_iterator.h apache#24 0x5616de6c967f in doris::vectorized::VCollectIterator::Level0Iterator::refresh_current_row() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:514:24 apache#25 0x5616de6ca8a6 in doris::vectorized::VCollectIterator::Level0Iterator::ensure_first_row_ref() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:493:14 apache#26 0x5616de6d7008 in doris::vectorized::VCollectIterator::Level1Iterator::ensure_first_row_ref() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:692:27 apache#27 0x5616de6bd200 in doris::vectorized::VCollectIterator::build_heap(std::vector, std::allocator>>&) /root/doris/be/src/vec/olap/vcollect_iterator.cpp:186:9 apache#28 0x5616de651b6c in doris::vectorized::BlockReader::_init_collect_iter(doris::TabletReader::ReaderParams const&) /root/doris/be/src/vec/olap/block_reader.cpp:157:5 apache#29 0x5616de65526f in doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams const&) /root/doris/be/src/vec/olap/block_reader.cpp:229:19 apache#30 0x5616e175a0f9 in doris::vectorized::NewOlapScanner::open(doris::RuntimeState*) /root/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:237:32 apache#31 0x5616c736ad34 in doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr, std::shared_ptr) /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:236:5 apache#32 0x5616c736f05e in doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:176:21 apache#33 0x5616c736f05e in doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:175:31 apache#34 0x5616c736f05e in void std::_invoke_impl, std::shared_ptr)::$_1::operator()() const::'lambda'()&>(std::_invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 apache#35 0x5616c736f05e in std::enable_if, std::shared_ptr)::$1::operator()() const::'lambda'()&>, void>::type std::_invoke_r, std::shared_ptr)::$_1::operator()() const::'lambda'()&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 apache#36 0x5616c736f05e in std::_Function_handler, std::shared_ptr)::$_1::operator()() const::'lambda'()>::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 apache#37 0x5616aeed6a3b in doris::ThreadPool::dispatch_thread() /root/doris/be/src/util/threadpool.cpp:543:24 apache#38 0x5616aeeae4f7 in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:498:5 apache#39 0x7f7e663e3ac2 in start_thread nptl/pthread_create.c:442:8 apache#40 0x7f7e6647584f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /root/doris/be/src/vec/common/string_ref.h:271:54 in ``` ## Proposed changes Issue Number: close #xxx <!--Describe your changes.-->
wyxxxcat
pushed a commit
that referenced
this pull request
Nov 7, 2024
…he#39237) (apache#39266) ## Proposed changes apache#39237 which led to accessing uninitialized data. Moved the profile initialization into the init method. ``` /mnt/disk2/yanxuecheng/doris/be/src/pipeline/local_exchange/local_exchanger.cpp:100:9: runtime error: member call on null pointer of type 'doris::RuntimeProfile::Counter' #0 0x563f0fb8b6c9 in doris::pipeline::Exchanger<std::shared_ptr<doris::pipeline::BlockWrapper>>::_dequeue_data(doris::pipeline::LocalExchangeSourceLocalState&, std::shared_ptr<doris::pipeline::BlockWrapper>&, bool*, doris::vectorized::Block*, int) /mnt/disk2/yanxuecheng/doris/be/src/pipeline/local_exchange/local_exchanger.cpp:100:9 #1 0x563f0fb78a45 in doris::pipeline::LocalMergeSortExchanger::finalize(doris::pipeline::LocalExchangeSourceLocalState&) /mnt/disk2/yanxuecheng/doris/be/src/pipeline/local_exchange/local_exchanger.cpp:325:16 #2 0x563f09610366 in doris::pipeline::LocalExchangeSharedState::sub_running_source_operators(doris::pipeline::LocalExchangeSourceLocalState&) /mnt/disk2/yanxuecheng/doris/be/src/pipeline/dependency.cpp:196:20 #3 0x563f0fb63709 in doris::pipeline::LocalExchangeSourceLocalState::close(doris::RuntimeState*) /mnt/disk2/yanxuecheng/doris/be/src/pipeline/local_exchange/local_exchange_source_operator.cpp:59:24 #4 0x563f09728e80 in doris::pipeline::OperatorXBase::close(doris::RuntimeState*) /mnt/disk2/yanxuecheng/doris/be/src/pipeline/exec/operator.cpp:245:28 apache#5 0x563f0fd3fb58 in doris::pipeline::PipelineTask::close(doris::Status) /mnt/disk2/yanxuecheng/doris/be/src/pipeline/pipeline_task.cpp:459:28 apache#6 0x563f0fdb5315 in doris::pipeline::_close_task(doris::pipeline::PipelineTask*, doris::Status) /mnt/disk2/yanxuecheng/doris/be/src/pipeline/task_scheduler.cpp:91:27 apache#7 0x563f0fdb6573 in doris::pipeline::TaskScheduler::_do_work(unsigned long) /mnt/disk2/yanxuecheng/doris/be/src/pipeline/task_scheduler.cpp:125:13 apache#8 0x563f0fdb9d6a in doris::pipeline::TaskScheduler::start()::$_0::operator()() const /mnt/disk2/yanxuecheng/doris/be/src/pipeline/task_scheduler.cpp:64:9 apache#9 0x563f0fdb9cee in void std::__invoke_impl<void, doris::pipeline::TaskScheduler::start()::$_0&>(std::__invoke_other, doris::pipeline::TaskScheduler::start()::$_0&) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14 apache#10 0x563f0fdb9c4e in std::enable_if<is_invocable_r_v<void, doris::pipeline::TaskScheduler::start()::$_0&>, void>::type std::__invoke_r<void, doris::pipeline::TaskScheduler::start()::$_0&>(doris::pipeline::TaskScheduler::start()::$_0&) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:111:2 apache#11 0x563f0fdb99d5 in std::_Function_handler<void (), doris::pipeline::TaskScheduler::start()::$_0>::_M_invoke(std::_Any_data const&) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290:9 apache#12 0x563ebdddc8cf in std::function<void ()>::operator()() const /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591:9 apache#13 0x563ec4bd6db4 in doris::FunctionRunnable::run() /mnt/disk2/yanxuecheng/doris/be/src/util/threadpool.cpp:48:27 apache#14 0x563ec4bbc1b5 in doris::ThreadPool::dispatch_thread() /mnt/disk2/yanxuecheng/doris/be/src/util/threadpool.cpp:543:24 apache#15 0x563ec4bf9a53 in void std::__invoke_impl<void, void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(std::__invoke_memfun_deref, void (doris::ThreadPool::*&)(), doris::ThreadPool*&) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:74:14 apache#16 0x563ec4bf9858 in std::__invoke_result<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>::type std::__invoke<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(void (doris::ThreadPool::*&)(), doris::ThreadPool*&) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:96:14 apache#17 0x563ec4bf9790 in void std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>::__call<void, 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/functional:506:11 apache#18 0x563ec4bf9585 in void std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>::operator()<void>() /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/functional:591:17 apache#19 0x563ec4bf947e in void std::__invoke_impl<void, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&>(std::__invoke_other, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14 apache#20 0x563ec4bf93be in std::enable_if<is_invocable_r_v<void, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&>, void>::type std::__invoke_r<void, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&>(std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:111:2 apache#21 0x563ec4bf8e55 in std::_Function_handler<void (), std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>>::_M_invoke(std::_Any_data const&) /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290:9 apache#22 0x563ebdddc8cf in std::function<void ()>::operator()() const /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591:9 apache#23 0x563ec4b78d91 in doris::Thread::supervise_thread(void*) /mnt/disk2/yanxuecheng/doris/be/src/util/thread.cpp:498:5 apache#24 0x563ebdb2fe0a in asan_thread_start(void*) crtstuff.c apache#25 0x7feeac9e21c9 in start_thread (/lib64/libpthread.so.0+0x81c9) (BuildId: 823fccea3475e5870a4167dfe47df20e53222db0) apache#26 0x7feead3d1e72 in clone (/lib64/libc.so.6+0x39e72) (BuildId: ec3d7025354f1f1985831ff08ef0eb3b50aefbce) SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /mnt/disk2/yanxuecheng/doris/be/src/pipeline/local_exchange/local_exchanger.cpp:100:9 in *** Query id: ea174401bc134452-bd8a35522726a96a *** *** is nereids: 1 *** *** tablet id: 0 *** *** Aborted at 1723455006 (unix time) try "date -d @1723455006" if you are using GNU date *** *** Current BE git commitID: c47399c *** *** SIGSEGV address not mapped to object (@0x0) received by PID 3055435 (TID 3060341 OR 0x7fe320acf700) from PID 0; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /mnt/disk2/yanxuecheng/doris/be/src/common/signal_handler.h:421 1# 0x00007FEEAD3E6B50 in /lib64/libc.so.6 2# doris::pipeline::Exchanger<std::shared_ptr<doris::pipeline::BlockWrapper> >::_dequeue_data(doris::pipeline::LocalExchangeSourceLocalState&, std::shared_ptr<doris::pipeline::BlockWrapper>&, bool*, doris::vectorized::Block*, int) at /mnt/disk2/yanxuecheng/doris/be/src/pipeline/local_exchange/local_exchanger.cpp:100 3# doris::pipeline::LocalMergeSortExchanger::finalize(doris::pipeline::LocalExchangeSourceLocalState&) at /mnt/disk2/yanxuecheng/doris/be/src/pipeline/local_exchange/local_exchanger.cpp:325 4# doris::pipeline::LocalExchangeSharedState::sub_running_source_operators(doris::pipeline::LocalExchangeSourceLocalState&) at /mnt/disk2/yanxuecheng/doris/be/src/pipeline/dependency.cpp:196 5# doris::pipeline::LocalExchangeSourceLocalState::close(doris::RuntimeState*) in /mnt/disk2/yanxuecheng/doris/output/be/lib/doris_be 6# doris::pipeline::OperatorXBase::close(doris::RuntimeState*) at /mnt/disk2/yanxuecheng/doris/be/src/pipeline/exec/operator.cpp:245 7# doris::pipeline::PipelineTask::close(doris::Status) at /mnt/disk2/yanxuecheng/doris/be/src/pipeline/pipeline_task.cpp:459 8# doris::pipeline::_close_task(doris::pipeline::PipelineTask*, doris::Status) at /mnt/disk2/yanxuecheng/doris/be/src/pipeline/task_scheduler.cpp:91 9# doris::pipeline::TaskScheduler::_do_work(unsigned long) at /mnt/disk2/yanxuecheng/doris/be/src/pipeline/task_scheduler.cpp:125 10# doris::pipeline::TaskScheduler::start()::$_0::operator()() const at /mnt/disk2/yanxuecheng/doris/be/src/pipeline/task_scheduler.cpp:64 11# void std::__invoke_impl<void, doris::pipeline::TaskScheduler::start()::$_0&>(std::__invoke_other, doris::pipeline::TaskScheduler::start()::$_0&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61 12# std::enable_if<is_invocable_r_v<void, doris::pipeline::TaskScheduler::start()::$_0&>, void>::type std::__invoke_r<void, doris::pipeline::TaskScheduler::start()::$_0&>(doris::pipeline::TaskScheduler::start()::$_0&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:117 13# std::_Function_handler<void (), doris::pipeline::TaskScheduler::start()::$_0>::_M_invoke(std::_Any_data const&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290 14# std::function<void ()>::operator()() const at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591 15# doris::FunctionRunnable::run() at /mnt/disk2/yanxuecheng/doris/be/src/util/threadpool.cpp:48 16# doris::ThreadPool::dispatch_thread() at /mnt/disk2/yanxuecheng/doris/be/src/util/threadpool.cpp:543 17# void std::__invoke_impl<void, void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(std::__invoke_memfun_deref, void (doris::ThreadPool::*&)(), doris::ThreadPool*&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:74 18# std::__invoke_result<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>::type std::__invoke<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(void (doris::ThreadPool::*&)(), doris::ThreadPool*&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:96 19# void std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/functional:506 20# void std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>::operator()<, void>() at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/functional:591 21# void std::__invoke_impl<void, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&>(std::__invoke_other, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61 22# std::enable_if<is_invocable_r_v<void, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&>, void>::type std::__invoke_r<void, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&>(std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:117 23# std::_Function_handler<void (), std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()> >::_M_invoke(std::_Any_data const&) at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290 24# std::function<void ()>::operator()() const at /mnt/disk2/yanxuecheng/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591 25# doris::Thread::supervise_thread(void*) at /mnt/disk2/yanxuecheng/doris/be/src/util/thread.cpp:498 26# asan_thread_start(void*) in /mnt/disk2/yanxuecheng/doris/output/be/lib/doris_be 27# start_thread in /lib64/libpthread.so.0 28# __clone in /lib64/libc.so.6 ``` <!--Describe your changes.--> ## Proposed changes Issue Number: close #xxx <!--Describe your changes.-->
wyxxxcat
pushed a commit
that referenced
this pull request
Jan 6, 2025
…ache#46296) ### What problem does this PR solve? ```cpp 172.20.50.85 be.out: apache#8 0x56435579ab5e in lucene::store::BufferedIndexInput::readBytes(unsigned char*, int, bool) /home/zcp/repo_center/doris_branch-3.0/doris/be/src/clucene/src/core/CLucene/store/IndexInput.cpp:192:9 apache#9 0x564319f7d95d in doris::segment_v2::CSIndexInput::readInternal(unsigned char*, int) /home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/inverted_index_compound_reader.cpp:107:11 apache#10 0x56435579bb5b in lucene::store::BufferedIndexInput::refill() /home/zcp/repo_center/doris_branch-3.0/doris/be/src/clucene/src/core/CLucene/store/IndexInput.cpp:285:5 apache#11 0x56435579b096 in lucene::store::BufferedIndexInput::readBytes(unsigned char*, int, int, bool) /home/zcp/repo_center/doris_branch-3.0/doris/be/src/clucene/src/core/CLucene/store/IndexInput.cpp:217:9 apache#12 0x56435579a9f6 in lucene::store::BufferedIndexInput::readBytes(unsigned char*, int) /home/zcp/repo_center/doris_branch-3.0/doris/be/src/clucene/src/core/CLucene/store/IndexInput.cpp:186:9 apache#13 0x564319fe8aee in doris::segment_v2::InvertedIndexReader::read_null_bitmap(doris::io::IOContext const*, doris::OlapReaderStatistics*, doris::segment_v2::InvertedIndexQueryCacheHandle*, lucene::store::Directory*) /home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/inverted_index_reader.cpp:144:29 apache#14 0x564319febab4 in doris::segment_v2::InvertedIndexReader::handle_searcher_cache(doris::segment_v2::InvertedIndexCacheHandle*, doris::io::IOContext const*, doris::OlapReaderStatistics*) /home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/inverted_index_reader.cpp:194:27 apache#15 0x564319ff4b0b in doris::segment_v2::StringTypeInvertedIndexReader::query(doris::io::IOContext const*, doris::OlapReaderStatistics*, doris::RuntimeState*, std::__cxx11::basic_string, std::allocator> const&, void const*, doris::segment_v2::InvertedIndexQueryType, std::shared_ptr&) /home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/inverted_index_reader.cpp:407:5 apache#16 0x56431a001f00 in doris::segment_v2::InvertedIndexIterator::read_from_inverted_index(std::__cxx11::basic_string, std::allocator> const&, void const*, doris::segment_v2::InvertedIndexQueryType, unsigned int, std::shared_ptr&, bool) /home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/inverted_index_reader.cpp:1137:5 apache#17 0x5643191f4a52 in doris::ComparisonPredicateBase<(doris::PrimitiveType)23, (doris::PredicateType)1>::evaluate(std::pair, std::allocator>, std::shared_ptr> const&, doris::segment_v2::InvertedIndexIterator*, unsigned int, roaring::Roaring*) const /home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/comparison_predicate.h:107:9 apache#18 0x56431a249ef7 in doris::segment_v2::SegmentIterator::_apply_inverted_index_on_column_predicate(doris::ColumnPredicate*, std::vector>&, bool*) /home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:871:28 apache#19 0x56431a23ddc0 in doris::segment_v2::SegmentIterator::_apply_inverted_index() /home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:951:13 apache#20 0x56431a22f956 in doris::segment_v2::SegmentIterator::_get_row_ranges_by_column_conditions() /home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:512:13 apache#21 0x56431a22b78f in doris::segment_v2::SegmentIterator::_lazy_init() /home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:383:5 apache#22 0x56431a26c667 in doris::segment_v2::SegmentIterator::_next_batch_internal(doris::vectorized::Block*) /home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2008:9 apache#23 0x56431a264fb6 in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*)::$_0::operator()() const /home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:1917:9 apache#24 0x56431a264fb6 in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*) /home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:1916:19 apache#25 0x56431a08315c in doris::segment_v2::LazyInitSegmentIterator::next_batch(doris::vectorized::Block*) /home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/lazy_init_segment_iterator.h:44:33 apache#26 0x564319b95855 in doris::BetaRowsetReader::next_block(doris::vectorized::Block*) /home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/beta_rowset_reader.cpp:366:29 apache#27 0x56434e3a0c10 in doris::vectorized::VCollectIterator::Level0Iterator::_refresh() /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/olap/vcollect_iterator.h apache#28 0x56434e38ba5b in doris::vectorized::VCollectIterator::Level0Iterator::refresh_current_row() /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/olap/vcollect_iterator.cpp:509:24 apache#29 0x56434e38cd76 in doris::vectorized::VCollectIterator::Level0Iterator::ensure_first_row_ref() /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/olap/vcollect_iterator.cpp:482:14 apache#30 0x56434e399745 in doris::vectorized::VCollectIterator::Level1Iterator::ensure_first_row_ref() /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/olap/vcollect_iterator.cpp:696:27 apache#31 0x56434e37f7f9 in doris::vectorized::VCollectIterator::build_heap(std::vector, std::allocator>>&) /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/olap/vcollect_iterator.cpp:186:9 apache#32 0x56434e312fe2 in doris::vectorized::BlockReader::_init_collect_iter(doris::TabletReader::ReaderParams const&) /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/olap/block_reader.cpp:140:5 apache#33 0x56434e3165df in doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams const&) /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/olap/block_reader.cpp:212:19 apache#34 0x5643515aba04 in doris::vectorized::NewOlapScanner::open(doris::RuntimeState*) /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:231:32 apache#35 0x564336c138c6 in doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr, std::shared_ptr) /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:247:5 apache#36 0x564336c18db0 in doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()::operator()() const::'lambda0'()::operator()() const /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:180:21 apache#37 0x564336c18db0 in doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()::operator()() const /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:179:31 apache#38 0x564336c18db0 in void std::__invoke_impl, std::shared_ptr)::$_1::operator()() const::'lambda'()&>(std::__invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 apache#39 0x564336c18db0 in std::enable_if, std::shared_ptr)::$_1::operator()() const::'lambda'()&>, void>::type std::__invoke_r, std::shared_ptr)::$_1::operator()() const::'lambda'()&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 apache#40 0x564336c18db0 in std::_Function_handler, std::shared_ptr)::$_1::operator()() const::'lambda'()>::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 apache#41 0x56431b9d94eb in doris::ThreadPool::dispatch_thread() /home/zcp/repo_center/doris_branch-3.0/doris/be/src/util/threadpool.cpp:543:24 apache#42 0x56431b9b18c7 in doris::Thread::supervise_thread(void*) /home/zcp/repo_center/doris_branch-3.0/doris/be/src/util/thread.cpp:498:5 apache#43 0x7f31d3b87ac2 in start_thread nptl/pthread_create.c:442:8 apache#44 0x7f31d3c1984f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 0x61d000a7f3c8 is located 1352 bytes inside of 2064-byte region [0x61d000a7ee80,0x61d000a7f690) freed by thread T1698 (Pipe_normal [wo) here: #0 0x564317278d9d in operator delete(void*) (/mnt/hdd01/ci/doris-deploy-branch-3.0-cloud/be/lib/doris_be+0x3626bd9d) (BuildId: 9dab40b94b1dc995) #1 0x5643515b192c in std::unique_ptr>::reset(doris::TabletReader*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:456:7 #2 0x5643515b192c in doris::vectorized::NewOlapScanner::close(doris::RuntimeState*) /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:534:20 #3 0x564352457463 in doris::vectorized::ScannerDelegate::~ScannerDelegate() /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/exec/scan/vscan_node.h:35:31 #4 0x5643172a8324 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:168:6 apache#5 0x56434e7d7870 in std::__shared_ptr::~__shared_ptr() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1149:31 apache#6 0x56434e7d7870 in void std::destroy_at>(std::shared_ptr*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:88:15 apache#7 0x56434e7d7870 in void std::allocator_traits>>>::destroy>(std::allocator>>&, std::shared_ptr*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:533:4 apache#8 0x56434e7d7870 in std::__cxx11::_List_base, std::allocator>>::_M_clear() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/list.tcc:77:4 apache#9 0x564351db11d2 in std::__cxx11::_List_base, std::allocator>>::~_List_base() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_list.h:499:9 apache#10 0x564351db11d2 in doris::pipeline::ScanLocalState::close(doris::RuntimeState*) /home/zcp/repo_center/doris_branch-3.0/doris/be/src/pipeline/exec/scan_operator.cpp:1261:5 apache#11 0x56434e6357e3 in doris::pipeline::OperatorXBase::close(doris::RuntimeState*) /home/zcp/repo_center/doris_branch-3.0/doris/be/src/pipeline/exec/operator.cpp:245:28 apache#12 0x564352ad00c1 in doris::pipeline::PipelineTask::close(doris::Status) /home/zcp/repo_center/doris_branch-3.0/doris/be/src/pipeline/pipeline_task.cpp:489:28 apache#13 0x564352b0701f in doris::pipeline::_close_task(doris::pipeline::PipelineTask*, doris::Status) /home/zcp/repo_center/doris_branch-3.0/doris/be/src/pipeline/task_scheduler.cpp:91:27 apache#14 0x564352b095f3 in doris::pipeline::TaskScheduler::_do_work(unsigned long) /home/zcp/repo_center/doris_branch-3.0/doris/be/src/pipeline/task_scheduler.cpp:181:17 apache#15 0x56431b9d94eb in doris::ThreadPool::dispatch_thread() /home/zcp/repo_center/doris_branch-3.0/doris/be/src/util/threadpool.cpp:543:24 apache#16 0x56431b9b18c7 in doris::Thread::supervise_thread(void*) /home/zcp/repo_center/doris_branch-3.0/doris/be/src/util/thread.cpp:498:5 apache#17 0x7f31d3b87ac2 in start_thread nptl/pthread_create.c:442:8 ``` Problem Summary: ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
wyxxxcat
pushed a commit
that referenced
this pull request
Jan 9, 2025
…apache#46541) ### What problem does this PR solve? Problem Summary: In 2.1.7, if a sql parse failed in nereids planner, it will fallback to old planner. and old planner maybe create `SelectNode` with some conjuncts that are not boolean type. Reproduce: Create table sql see apache#46498 query sql1: select b.c_id from dbgr as b left join (select c.c_id from lo where event_date between 20220500 and 20220600 limit 100 )c on c.c_id LIMIT 200; query sql2: select b.c_id from dbgr as b left join (select c.c_id from lo )c on c.c_id LIMIT 0, 200; Because `select c.c_id`, these sqls will fallback to old planner. Because `on c.c_id` is not boolean type, and be will core. A part of query plan is as follows: | 1:VOlapScanNode | | TABLE: test.lo(lo), PREAGGREGATION: ON | | PREDICATES: (`c`.`c_id` AND (`test`.`lo`.`__DORIS_DELETE_SIGN__` = 0)) | | partitions=1/3 (p_202206) | | tablets=2/2, tabletList=89678,89680 | | cardinality=46, avgRowSize=165.54349, numNodes=1 | | pushAggOp=NONE | +-----------------------------------------------------------------------------+ A fatal log is as follows: F20241219 23:13:23.457937 33282 assert_cast.h:58] Bad cast from type:doris::vectorized::ColumnVector<int> to doris::vectorized::ColumnVector<unsigned char> *** Check failure stack trace: *** @ 0x55bfa043b956 google::LogMessageFatal::~LogMessageFatal() @ 0x55bf6f3bc070 assert_cast<>() @ 0x55bf8978d767 doris::vectorized::VExprContext::execute_conjuncts() @ 0x55bf8978c463 doris::vectorized::VExprContext::execute_conjuncts_and_filter_block() @ 0x55bf8978bf72 doris::vectorized::VExprContext::filter_block() @ 0x55bfa035b8e4 doris::pipeline::SelectOperatorX::pull() @ 0x55bf9fee2b62 doris::pipeline::StreamingOperatorX<>::get_block() @ 0x55bf9feab54b doris::pipeline::OperatorXBase::get_block_after_projects() @ 0x55bfa03dd07c doris::pipeline::PipelineXTask::execute() @ 0x55bfa0413e85 doris::pipeline::TaskScheduler::_do_work() @ 0x55bfa0417dcb doris::pipeline::TaskScheduler::start()::$_0::operator()() @ 0x55bfa0417d55 std::__invoke_impl<>() @ 0x55bfa0417d05 _ZSt10__invoke_rIvRZN5doris8pipeline13TaskScheduler5startEvE3$_0JEENSt9enable_ifIX16is_invocable_r_vIT_T0_DpT1_EES6_E4typeEO S7_DpOS8_ @ 0x55bfa0417bcd std::_Function_handler<>::_M_invoke() @ 0x55bf6e6c9b63 std::function<>::operator()() @ 0x55bf7289e209 doris::FunctionRunnable::run() @ 0x55bf728899c0 doris::ThreadPool::dispatch_thread() @ 0x55bf728b0c24 std::__invoke_impl<>() @ 0x55bf728b0afd std::__invoke<>() @ 0x55bf728b0a85 _ZNSt5_BindIFMN5doris10ThreadPoolEFvvEPS1_EE6__callIvJEJLm0EEEET_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE @ 0x55bf728b092e std::_Bind<>::operator()<>() @ 0x55bf728b0845 std::__invoke_impl<>() @ 0x55bf728b07e5 _ZSt10__invoke_rIvRSt5_BindIFMN5doris10ThreadPoolEFvvEPS2_EEJEENSt9enable_ifIX16is_invocable_r_vIT_T0_DpT1_EESA_E4typeEOSB_D pOSC_ @ 0x55bf728b048d std::_Function_handler<>::_M_invoke() @ 0x55bf6e6c9b63 std::function<>::operator()() @ 0x55bf728521fc doris::Thread::supervise_thread() @ 0x7f4260614ea5 start_thread @ 0x7f42610439fd __clone @ (nil) (unknown) And another: F20250108 13:07:05.275424 184257 assert_cast.h:58] Bad cast from type:doris::vectorized::ColumnVector<int> to doris::vectorized::ColumnVector<unsigned char> 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/tc/be/src/common/signal_handler.h:421 1# 0x00007FB73FB31400 in /lib64/libc.so.6 2# __GI_raise in /lib64/libc.so.6 3# abort in /lib64/libc.so.6 4# 0x000055CDAAF0090D in /usr/local/service/doris/lib/be/doris_be 5# google::LogMessage::SendToLog() in /usr/local/service/doris/lib/be/doris_be 6# google::LogMessage::Flush() in /usr/local/service/doris/lib/be/doris_be 7# google::LogMessageFatal::~LogMessageFatal() in /usr/local/service/doris/lib/be/doris_be 8# doris::vectorized::ColumnVector<unsigned char> const& assert_cast<doris::vectorized::ColumnVector<unsigned char> const&, doris::vec torized::IColumn const&>(doris::vectorized::IColumn const&) in /usr/local/service/doris/lib/be/doris_be 9# doris::vectorized::VExprContext::execute_conjuncts(std::vector<std::shared_ptr<doris::vectorized::VExprContext>, std::allocator<std ::shared_ptr<doris::vectorized::VExprContext> > > const&, std::vector<doris::vectorized::PODArray<unsigned char, 4096ul, Allocator<fals e, false, false, DefaultMemoryAllocator>, 16ul, 16ul>*, std::allocator<doris::vectorized::PODArray<unsigned char, 4096ul, Allocator<fal se, false, false, DefaultMemoryAllocator>, 16ul, 16ul>*> > const*, bool, doris::vectorized::Block*, doris::vectorized::PODArray<unsigne d char, 4096ul, Allocator<false, false, false, DefaultMemoryAllocator>, 16ul, 16ul>*, bool*) at /root/tc/be/src/vec/exprs/vexpr_context .cpp:181 10# doris::vectorized::VExprContext::execute_conjuncts_and_filter_block(std::vector<std::shared_ptr<doris::vectorized::VExprContext>, s td::allocator<std::shared_ptr<doris::vectorized::VExprContext> > > const&, doris::vectorized::Block*, std::vector<unsigned int, std::al locator<unsigned int> >&, int, doris::vectorized::PODArray<unsigned char, 4096ul, Allocator<false, false, false, DefaultMemoryAllocator >, 16ul, 16ul>&) at /root/tc/be/src/vec/exprs/vexpr_context.cpp:324 11# doris::segment_v2::SegmentIterator::_execute_common_expr(unsigned short*, unsigned short&, doris::vectorized::Block*) at /root/tc/b e/src/olap/rowset/segment_v2/segment_iterator.cpp:2274 12# doris::segment_v2::SegmentIterator::_next_batch_internal(doris::vectorized::Block*) at /root/tc/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2178 13# doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*)::$_0::operator()() const at /root/tc/be/src/olap/rowset/segment_v2/segment_iterator.cpp:1914 14# doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*) at /root/tc/be/src/olap/rowset/segment_v2/segment_iterator.cpp:1913 15# doris::segment_v2::LazyInitSegmentIterator::next_batch(doris::vectorized::Block*) in /usr/local/service/doris/lib/be/doris_be 16# doris::BetaRowsetReader::next_block(doris::vectorized::Block*) at /root/tc/be/src/olap/rowset/beta_rowset_reader.cpp:348 17# doris::vectorized::VCollectIterator::Level0Iterator::_refresh() in /usr/local/service/doris/lib/be/doris_be 18# doris::vectorized::VCollectIterator::Level0Iterator::refresh_current_row() at /root/tc/be/src/vec/olap/vcollect_iterator.cpp:511 19# doris::vectorized::VCollectIterator::Level0Iterator::ensure_first_row_ref() at /root/tc/be/src/vec/olap/vcollect_iterator.cpp:482 20# doris::vectorized::VCollectIterator::Level1Iterator::ensure_first_row_ref() at /root/tc/be/src/vec/olap/vcollect_iterator.cpp:697 21# doris::vectorized::VCollectIterator::build_heap(std::vector<std::shared_ptr<doris::RowsetReader>, std::allocator<std::shared_ptr<doris::RowsetReader> > >&) at /root/tc/be/src/vec/olap/vcollect_iterator.cpp:186 22# doris::vectorized::BlockReader::_init_collect_iter(doris::TabletReader::ReaderParams const&) at /root/tc/be/src/vec/olap/block_reader.cpp:139 23# doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams const&) at /root/tc/be/src/vec/olap/block_reader.cpp:211 24# doris::vectorized::NewOlapScanner::open(doris::RuntimeState*) at /root/tc/be/src/vec/exec/scan/new_olap_scanner.cpp:227 25# doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>) at /root/tc/be/src/vec/exec/scan/scanner_scheduler.cpp:259 26# doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::{lambda()#1}::operator()() const::{lambda()#2}::operator()() const at /root/tc/be/src/vec/exec/scan/scanner_scheduler.cpp:180 ... --------- Co-authored-by: liutang123 <[email protected]>
wyxxxcat
pushed a commit
that referenced
this pull request
Jan 11, 2025
…obeSinkOperatorX (apache#46706) ### What problem does this PR solve? ``` *** Query id: 80819fcc223e4a45-b46155de6e0c4eee *** *** is nereids: 1 *** *** tablet id: 0 *** *** Aborted at 1736352810 (unix time) try "date -d @1736352810" if you are using GNU date *** *** Current BE git commitID: 08683cb *** *** SIGSEGV address not mapped to object (@0x38) received by PID 8736 (TID 11549 OR 0x7f8dd0922640) from PID 56; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/doris_branch-3.0/doris/be/src/common/signal_handler.h:421 1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so 2# JVM_handle_linux_signal in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so 3# 0x00007F92019CA520 in /lib/x86_64-linux-gnu/libc.so.6 4# auto doris::pipeline::SetProbeSinkOperatorX::_refresh_hash_table(doris::pipeline::SetProbeSinkLocalState&)::{lambda(auto:1&&)#1}::operator(), HashTableNoState>, DefaultHash, HashTableGrower<10ul>, Allocator > >&>(doris::vectorized::MethodSerialized, HashTableNoState>, DefaultHash, HashTableGrower<10ul>, Allocator > >&) const at /root/doris_branch-3.0/doris/be/src/pipeline/exec/set_probe_sink_operator.cpp:213 5# doris::pipeline::SetProbeSinkOperatorX::_finalize_probe(doris::pipeline::SetProbeSinkLocalState&) at /root/doris_branch-3.0/doris/be/src/pipeline/exec/set_probe_sink_operator.cpp:184 6# doris::pipeline::SetProbeSinkOperatorX::sink(doris::RuntimeState*, doris::vectorized::Block*, bool) at /root/doris_branch-3.0/doris/be/src/pipeline/exec/set_probe_sink_operator.cpp:98 7# doris::pipeline::PipelineTask::execute(bool*) at /root/doris_branch-3.0/doris/be/src/pipeline/pipeline_task.cpp:387 8# doris::pipeline::TaskScheduler::_do_work(unsigned long) at /root/doris_branch-3.0/doris/be/src/pipeline/task_scheduler.cpp:138 9# doris::ThreadPool::dispatch_thread() in /mnt/ssd01/doris-branch40preview/NEREIDS_ASAN/be/lib/doris_be 10# doris::Thread::supervise_thread(void*) at /root/doris_branch-3.0/doris/be/src/util/thread.cpp:499 11# start_thread at ./nptl/pthread_create.c:442 12# 0x00007F9201AAE850 at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:83 ```
wyxxxcat
pushed a commit
that referenced
this pull request
Jan 14, 2025
…obeSinkOperatorX (apache#46706) (apache#46832) ``` *** Query id: 80819fcc223e4a45-b46155de6e0c4eee *** *** is nereids: 1 *** *** tablet id: 0 *** *** Aborted at 1736352810 (unix time) try "date -d @1736352810" if you are using GNU date *** *** Current BE git commitID: 08683cb *** *** SIGSEGV address not mapped to object (@0x38) received by PID 8736 (TID 11549 OR 0x7f8dd0922640) from PID 56; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/doris_branch-3.0/doris/be/src/common/signal_handler.h:421 1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so 2# JVM_handle_linux_signal in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so 3# 0x00007F92019CA520 in /lib/x86_64-linux-gnu/libc.so.6 4# auto doris::pipeline::SetProbeSinkOperatorX::_refresh_hash_table(doris::pipeline::SetProbeSinkLocalState&)::{lambda(auto:1&&)#1}::operator(), HashTableNoState>, DefaultHash, HashTableGrower<10ul>, Allocator > >&>(doris::vectorized::MethodSerialized, HashTableNoState>, DefaultHash, HashTableGrower<10ul>, Allocator > >&) const at /root/doris_branch-3.0/doris/be/src/pipeline/exec/set_probe_sink_operator.cpp:213 5# doris::pipeline::SetProbeSinkOperatorX::_finalize_probe(doris::pipeline::SetProbeSinkLocalState&) at /root/doris_branch-3.0/doris/be/src/pipeline/exec/set_probe_sink_operator.cpp:184 6# doris::pipeline::SetProbeSinkOperatorX::sink(doris::RuntimeState*, doris::vectorized::Block*, bool) at /root/doris_branch-3.0/doris/be/src/pipeline/exec/set_probe_sink_operator.cpp:98 7# doris::pipeline::PipelineTask::execute(bool*) at /root/doris_branch-3.0/doris/be/src/pipeline/pipeline_task.cpp:387 8# doris::pipeline::TaskScheduler::_do_work(unsigned long) at /root/doris_branch-3.0/doris/be/src/pipeline/task_scheduler.cpp:138 9# doris::ThreadPool::dispatch_thread() in /mnt/ssd01/doris-branch40preview/NEREIDS_ASAN/be/lib/doris_be 10# doris::Thread::supervise_thread(void*) at /root/doris_branch-3.0/doris/be/src/util/thread.cpp:499 11# start_thread at ./nptl/pthread_create.c:442 12# 0x00007F9201AAE850 at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:83 ```
wyxxxcat
pushed a commit
that referenced
this pull request
Jan 17, 2025
…apache#46997) ### What problem does this PR solve? Related PR: apache#40912 Problem Summary: Do not reset _arena in MemTable::to_block(), because it is still used in ~MemTable() when releasing agg places Fix the following use-after-free Use: ==3628099==ERROR: AddressSanitizer: heap-use-after-free on address 0x52100381be60 at pc 0x5648f30893f8 bp 0x7f8842433310 sp 0x7f8842433308 READ of size 8 at 0x52100381be60 thread T4767 (wg_flush_broker) #0 0x5648f30893f7 in phmap::priv::raw_hash_set<phmap::priv::FlatHashSetPolicy<unsigned long>, phmap::Hash<unsigned long>, phmap::EqualTo<unsigned long>, std::allocator<unsigned long>>::destroy_slots() doris/thirdparty/installed/include/parallel_hashmap/phmap.h:1992:14 #1 0x5648f30936f6 in phmap::priv::raw_hash_set<phmap::priv::FlatHashSetPolicy<unsigned long>, phmap::Hash<unsigned long>, phmap::EqualTo<unsigned long>, std::allocator<unsigned long>>::~raw_hash_set() doris/thirdparty/installed/include/parallel_hashmap/phmap.h:1236:23 #2 0x5648f3089276 in phmap::flat_hash_set<unsigned long, phmap::Hash<unsigned long>, phmap::EqualTo<unsigned long>, std::allocator<unsigned long>>::~flat_hash_set() doris/thirdparty/installed/include/parallel_hashmap/phmap.h:4577:7 #3 0x5648f308922a in doris::BitmapValue::~BitmapValue() doris/be/src/util/bitmap_value.h:824:7 #4 0x56490d319fa6 in doris::vectorized::AggregateFunctionBitmapData<doris::vectorized::AggregateFunctionBitmapUnionOp>::~AggregateFunctionBitmapData() doris/be/src/vec/aggregate_functions/aggregate_function_bitmap.h:127:8 apache#5 0x56490d49636a in doris::vectorized::IAggregateFunctionDataHelper<doris::vectorized::AggregateFunctionBitmapData<doris::vectorized::AggregateFunctionBitmapUnionOp>, doris::vectorized::AggregateFunctionBitmapOp<doris::vectorized::AggregateFunctionBitmapUnionOp>>::destroy(char*) const doris/be/src/vec/aggregate_functions/aggregate_function.h:563:92 apache#6 0x5648f68376e9 in doris::MemTable::~MemTable() doris/be/src/olap/memtable.cpp:159:27 Free: 0x52100381be60 is located 352 bytes inside of 4096-byte region [0x52100381bd00,0x52100381cd00) freed by thread T4767 (wg_flush_broker) here: #0 0x5648f2f3ee46 in free (doris/output/be/lib/doris_be+0x57418e46) (BuildId: 298b9c91a1ec8fe0) #1 0x5648f3080dfc in DefaultMemoryAllocator::free(void*) doris/be/src/vec/common/allocator.h:108:41 #2 0x5648f3080b3f in Allocator<false, false, false, DefaultMemoryAllocator>::free(void*, unsigned long) doris/be/src/vec/common/allocator.h:323:13 #3 0x5648f30b6dee in doris::vectorized::Arena::Chunk::~Chunk() doris/be/src/vec/common/arena.h:77:31 #4 0x5648f30b6d1f in doris::vectorized::Arena::~Arena() doris/be/src/vec/common/arena.h:151:16 apache#5 0x5648f30b695a in std::default_delete<doris::vectorized::Arena>::operator()(doris::vectorized::Arena*) const env/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/unique_ptr.h:99:2 apache#6 0x5648f30b67c8 in std::__uniq_ptr_impl<doris::vectorized::Arena, std::default_delete<doris::vectorized::Arena>>::reset(doris::vectorized::Arena*) env/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/unique_ptr.h:211:4 apache#7 0x5648f30b5d8c in std::unique_ptr<doris::vectorized::Arena, std::default_delete<doris::vectorized::Arena>>::reset(doris::vectorized::Arena*) env/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/unique_ptr.h:509:7 apache#8 0x5648f684253b in doris::MemTable::_to_block(std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block>>*) doris/be/src/olap/memtable.cpp:522:12 apache#9 0x5648f6842ac5 in doris::MemTable::to_block(std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block>>*) doris/be/src/olap/memtable.cpp:528:5 apache#10 0x5648f6907a72 in doris::FlushToken::_do_flush_memtable(doris::MemTable*, int, long*) doris/be/src/olap/memtable_flush_executor.cpp:144:9 apache#11 0x5648f690932c in doris::FlushToken::_flush_memtable(std::shared_ptr<doris::MemTable>, int, long) doris/be/src/olap/memtable_flush_executor.cpp:183:16 apache#12 0x5648f6915d18 in doris::MemtableFlushTask::run() doris/be/src/olap/memtable_flush_executor.cpp:60:20
wyxxxcat
pushed a commit
that referenced
this pull request
Jan 19, 2025
…obeSinkOperatorX (apache#46706) (apache#46831) ``` *** Query id: 80819fcc223e4a45-b46155de6e0c4eee *** *** is nereids: 1 *** *** tablet id: 0 *** *** Aborted at 1736352810 (unix time) try "date -d @1736352810" if you are using GNU date *** *** Current BE git commitID: 08683cb *** *** SIGSEGV address not mapped to object (@0x38) received by PID 8736 (TID 11549 OR 0x7f8dd0922640) from PID 56; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/doris_branch-3.0/doris/be/src/common/signal_handler.h:421 1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so 2# JVM_handle_linux_signal in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so 3# 0x00007F92019CA520 in /lib/x86_64-linux-gnu/libc.so.6 4# auto doris::pipeline::SetProbeSinkOperatorX::_refresh_hash_table(doris::pipeline::SetProbeSinkLocalState&)::{lambda(auto:1&&)#1}::operator(), HashTableNoState>, DefaultHash, HashTableGrower<10ul>, Allocator > >&>(doris::vectorized::MethodSerialized, HashTableNoState>, DefaultHash, HashTableGrower<10ul>, Allocator > >&) const at /root/doris_branch-3.0/doris/be/src/pipeline/exec/set_probe_sink_operator.cpp:213 5# doris::pipeline::SetProbeSinkOperatorX::_finalize_probe(doris::pipeline::SetProbeSinkLocalState&) at /root/doris_branch-3.0/doris/be/src/pipeline/exec/set_probe_sink_operator.cpp:184 6# doris::pipeline::SetProbeSinkOperatorX::sink(doris::RuntimeState*, doris::vectorized::Block*, bool) at /root/doris_branch-3.0/doris/be/src/pipeline/exec/set_probe_sink_operator.cpp:98 7# doris::pipeline::PipelineTask::execute(bool*) at /root/doris_branch-3.0/doris/be/src/pipeline/pipeline_task.cpp:387 8# doris::pipeline::TaskScheduler::_do_work(unsigned long) at /root/doris_branch-3.0/doris/be/src/pipeline/task_scheduler.cpp:138 9# doris::ThreadPool::dispatch_thread() in /mnt/ssd01/doris-branch40preview/NEREIDS_ASAN/be/lib/doris_be 10# doris::Thread::supervise_thread(void*) at /root/doris_branch-3.0/doris/be/src/util/thread.cpp:499 11# start_thread at ./nptl/pthread_create.c:442 12# 0x00007F9201AAE850 at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:83 ```
wyxxxcat
pushed a commit
that referenced
this pull request
Feb 14, 2025
…#47744) *** Query id: b7482b4d3ab94bcf-acdb53e8a8fcff81 *** *** is nereids: 1 *** *** tablet id: 0 *** *** Aborted at 1739024048 (unix time) try "date -d @1739024048" if you are using GNU date *** *** Current BE git commitID: a733a56 *** *** SIGSEGV address not mapped to object (@0x0) received by PID 5943 (TID 7905 OR 0x7fd0b0c80640) from PID 0; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/common/signal_handler.h:421 1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so 2# JVM_handle_linux_signal in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so 3# 0x00007FD4985D8520 in /lib/x86_64-linux-gnu/libc.so.6 4# doris::segment_v2::FileColumnIterator::next_batch(unsigned long*, COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>&, bool*) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/column_reader.cpp:1271 5# doris::segment_v2::SegmentIterator::_read_columns(std::vector<unsigned int, std::allocator<unsigned int> > const&, std::vector<COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>, std::allocator<COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn> > >&, unsigned long) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:1575 6# doris::segment_v2::SegmentIterator::_seek_and_peek(unsigned int) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:1253 7# doris::segment_v2::SegmentIterator::_lookup_ordinal_from_sk_index(doris::RowCursor const&, bool, unsigned int, unsigned int*) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:1148 8# doris::segment_v2::SegmentIterator::_get_row_ranges_by_keys() at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:432 9# doris::segment_v2::SegmentIterator::_lazy_init() at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:382 10# doris::segment_v2::SegmentIterator::_next_batch_internal(doris::vectorized::Block*) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2012 11# doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:1920 12# doris::segment_v2::LazyInitSegmentIterator::next_batch(doris::vectorized::Block*) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/lazy_init_segment_iterator.h:45 13# doris::vectorized::VUnionIterator::next_batch(doris::vectorized::Block*) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/olap/vgeneric_iterators.cpp:405 14# doris::BetaRowsetReader::next_block(doris::vectorized::Block*) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/beta_rowset_reader.cpp:376 15# doris::vectorized::VCollectIterator::Level0Iterator::refresh_current_row() at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/olap/vcollect_iterator.cpp:509 16# doris::vectorized::VCollectIterator::Level0Iterator::init(bool) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/olap/vcollect_iterator.cpp:462 17# doris::vectorized::VCollectIterator::build_heap(std::vector<std::shared_ptr<doris::RowsetReader>, std::allocator<std::shared_ptr<doris::RowsetReader> > >&) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/olap/vcollect_iterator.cpp:126 18# doris::vectorized::BlockReader::_init_collect_iter(doris::TabletReader::ReaderParams const&) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/olap/block_reader.cpp:148 19# doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams const&) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/olap/block_reader.cpp:222 20# doris::vectorized::NewOlapScanner::open(doris::RuntimeState*) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:232 21# doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:221 22# std::_Function_handler<void (), doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::{lambda()#1}>::_M_invoke(std::_Any_data const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291 23# doris::ThreadPool::dispatch_thread() in /mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be 24# doris::Thread::supervise_thread(void*) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/util/thread.cpp:499 25# start_thread at ./nptl/pthread_create.c:442 26# 0x00007FD4986BC850 at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:83
wyxxxcat
pushed a commit
that referenced
this pull request
Feb 19, 2025
### What problem does this PR solve?
In BE_TEST, a query context is not initialized, resulting in a core
dump.
```
==1732796==ERROR: AddressSanitizer: SEGV on unknown address 0x0000000001e0 (pc 0x55c664a74e74 bp 0x7ffcc76e9d20 sp 0x7ffcc76e9cf0 T0)
==1732796==The signal is caused by a READ memory access.
==1732796==Hint: address points to the zero page.
#0 0x55c664a74e74 in std::__shared_ptr<doris::ResourceContext, (__gnu_cxx::_Lock_policy)2>::__shared_ptr(std::__shared_ptr<doris::ResourceContext, (__gnu_cxx::_Lock_policy)2> const&) /mnt/disk6/common/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/shared_ptr_base.h:1522:7
#1 0x55c664a74e0e in std::shared_ptr<doris::ResourceContext>::shared_ptr(std::shared_ptr<doris::ResourceContext> const&) /mnt/disk6/common/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/shared_ptr.h:204:7
#2 0x55c66878ff18 in doris::QueryContext::resource_ctx() /mnt/disk12/yanxuecheng/doris/be/src/runtime/query_context.h:292:62
#3 0x55c69a2f2c19 in doris::vectorized::BlockSerializer::serialize_block(doris::vectorized::Block const*,
```
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [x] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->
### Check List (For Reviewer who merge this PR)
- [x] Confirm the release note
- [x] Confirm test cases
- [x] Confirm document
- [x] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
wyxxxcat
pushed a commit
that referenced
this pull request
Feb 20, 2025
…sMgr (apache#48004) ### What problem does this PR solve? Wait for QueryTaskController to be initialized before register ResourceContext fix ``` /root/doris/be/src/runtime/workload_management/task_controller.h:59:44: runtime error: load of value 3200171710, which is not a valid value for type 'TQueryType::type' #0 0x555ff9336162 in doris::TaskController::query_type() /root/doris/be/src/runtime/workload_management/task_controller.h:59:44 #1 0x555ff9336162 in doris::RuntimeQueryStatisticsMgr::report_runtime_query_statistics() /root/doris/be/src/runtime/runtime_query_statistics_mgr.cpp:346:50 #2 0x555ff50d789d in doris::Daemon::report_runtime_query_statistics_thread() /root/doris/be/src/common/daemon.cpp:487:65 #3 0x555ff9a3efae in std::function<void ()>::operator()() const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9 #4 0x555ff9a3efae in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:498:5 apache#5 0x7f4935d00608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477:8 apache#6 0x7f4935fad132 in __clone /build/glibc-SzIz7B/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95 ```
wyxxxcat
pushed a commit
that referenced
this pull request
Feb 21, 2025
WRITE of size 1 at 0x6160007e86f0 thread T1983 (Pipe_normal [wo) #0 0x55fc8065b975 in std::__atomic_base<bool>::store(bool, std::memory_order) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/atomic_base.h:457:2 #1 0x55fc8065b975 in std::__atomic_base<bool>::operator=(bool) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/atomic_base.h:349:2 #2 0x55fc8065b975 in std::atomic<bool>::operator=(bool) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/atomic:80:22 #3 0x55fc8065b975 in doris::pipeline::PipelineTask::set_running(bool) /root/doris/be/src/pipeline/pipeline_task.h:192:47 #4 0x55fc8065b975 in doris::pipeline::TaskScheduler::_do_work(int)::$_0::operator()() const /root/doris/be/src/pipeline/task_scheduler.cpp:121:23 apache#5 0x55fc8065b975 in doris::Defer<doris::pipeline::TaskScheduler::_do_work(int)::$_0>::~Defer() /root/doris/be/src/util/defer_op.h:37:16 apache#6 0x55fc8065b975 in doris::pipeline::TaskScheduler::_do_work(int) /root/doris/be/src/pipeline/task_scheduler.cpp:162:5 apache#7 0x55fc4c57cd19 in doris::ThreadPool::dispatch_thread() /root/doris/be/src/util/threadpool.cpp:608:24 apache#8 0x55fc4c55395e in std::function<void ()>::operator()() const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9 apache#9 0x55fc4c55395e in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:498:5 apache#10 0x7f9ee3d25608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477:8 apache#11 0x7f9ee3fd2132 in __clone /build/glibc-SzIz7B/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95 0x6160007e86f0 is located 624 bytes inside of 632-byte region [0x6160007e8480,0x6160007e86f8) freed by thread T1981 (Pipe_normal [wo) here: #0 0x55fc47aa680d in operator delete(void*) (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x3376e80d) (BuildId: 865149e62959581e) #1 0x55fc8059db84 in std::default_delete<doris::pipeline::PipelineTask>::operator()(doris::pipeline::PipelineTask*) const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:85:2 #2 0x55fc8059db84 in std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >::~unique_ptr() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:361:4 #3 0x55fc8059db84 in void std::destroy_at<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > >(std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:88:15 #4 0x55fc8059db84 in void std::_Destroy<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > >(std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:138:7 apache#5 0x55fc8059db84 in void std::_Destroy_aux<false>::__destroy<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >*>(std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >*, std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:152:6 apache#6 0x55fc8059db84 in void std::_Destroy<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >*>(std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >*, std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:184:7 apache#7 0x55fc8059db84 in void std::_Destroy<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >*, std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > >(std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >*, std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >*, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > >&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:746:7 apache#8 0x55fc8059db84 in std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >::~vector() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:680:2 apache#9 0x55fc8052571c in void std::destroy_at<std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > > >(std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:88:15 apache#10 0x55fc8052571c in void std::_Destroy<std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > > >(std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:138:7 apache#11 0x55fc8052571c in void std::_Destroy_aux<false>::__destroy<std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >*>(std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >*, std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:152:6 apache#12 0x55fc8052571c in void std::_Destroy<std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >*>(std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >*, std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:184:7 apache#13 0x55fc8052571c in void std::_Destroy<std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >*, std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > > >(std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >*, std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >*, std::allocator<std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > > >&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:746:7 apache#14 0x55fc8052571c in std::vector<std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >, std::allocator<std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > > > >::_M_erase_at_end(std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1796:6 apache#15 0x55fc8052571c in std::vector<std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >, std::allocator<std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > > > >::clear() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1499:9 apache#16 0x55fc8052571c in doris::pipeline::PipelineFragmentContext::~PipelineFragmentContext() /root/doris/be/src/pipeline/pipeline_fragment_context.cpp:142:12 apache#17 0x55fc47ad30cc in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:168:6 apache#18 0x55fc80658d57 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:702:11 apache#19 0x55fc80658d57 in std::__shared_ptr<doris::TaskExecutionContext, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1149:31 apache#20 0x55fc80658d57 in doris::pipeline::close_task(doris::pipeline::PipelineTask*, doris::Status) /root/doris/be/src/pipeline/task_scheduler.cpp:100:1 apache#21 0x55fc8065aa17 in doris::pipeline::TaskScheduler::_do_work(int) /root/doris/be/src/pipeline/task_scheduler.cpp:160:36 apache#22 0x55fc4c57cd19 in doris::ThreadPool::dispatch_thread() /root/doris/be/src/util/threadpool.cpp:608:24 apache#23 0x55fc4c55395e in std::function<void ()>::operator()() const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9 apache#24 0x55fc4c55395e in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:498:5 apache#25 0x7f9ee3d25608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477:8
wyxxxcat
pushed a commit
that referenced
this pull request
Feb 27, 2025
Schema scanner runs on a separate thread which is executed asynchronously. We should make sure all context used not be freed once it is scheduled. ERROR: AddressSanitizer: heap-buffer-overflow on address 0x613002f33eb2 at pc 0x55e085dccbe3 bp 0x7f345c0e1f10 sp 0x7f345c0e1f08 READ of size 1 at 0x613002f33eb2 thread T2776 (FragmentMgrAsyn) #0 0x55e085dccbe2 in std::__atomic_base::load(std::memory_order) const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/atomic_base.h:481:9 #1 0x55e085dccbe2 in std::atomic::operator bool() const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/atomic:87:22 #2 0x55e085dccbe2 in doris::SchemaScanner::get_next_block_async(doris::RuntimeState*)::$_0::operator()() const /home/zcp/repo_center/doris_master/doris/be/src/exec/schema_scanner.cpp:118:5 #3 0x55e085dccbe2 in void std::__invoke_impl(std::__invoke_other, doris::SchemaScanner::get_next_block_async(doris::RuntimeState*)::$_0&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 #4 0x55e085dccbe2 in std::enable_if, void>::type std::__invoke_r(doris::SchemaScanner::get_next_block_async(doris::RuntimeState*)::$_0&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 apache#5 0x55e085dccbe2 in std::_Function_handler::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 apache#6 0x55e050f081ca in doris::ThreadPool::dispatch_thread() /home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:608:24 apache#7 0x55e050ede467 in doris::Thread::supervise_thread(void*) /home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:498:5 apache#8 0x7f376ef5aac2 in start_thread nptl/pthread_create.c:442:8 apache#9 0x7f376efec84f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
wyxxxcat
pushed a commit
that referenced
this pull request
Feb 27, 2025
==14365==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x613002873c70 at pc 0x5559a290ec05 bp 0x7f6c40385e90 sp 0x7f6c40385e88 READ of size 2 at 0x613002873c70 thread T4019 (FragmentMgrAsyn) #0 0x5559a290ec04 in std::__atomic_base::load(std::memory_order) const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/atomic_base.h:481:9 #1 0x5559a290ec04 in doris::AtomicStatus::update(doris::Status const&) /home/zcp/repo_center/doris_master/doris/be/src/common/status.h:601:44 #2 0x5559d8f289c0 in doris::SchemaScanner::get_next_block_async(doris::RuntimeState*)::$_0::operator()() const /home/zcp/repo_center/doris_master/doris/be/src/exec/schema_scanner.cpp:118:5 #3 0x5559d8f289c0 in void std::__invoke_impl(std::__invoke_other, doris::SchemaScanner::get_next_block_async(doris::RuntimeState*)::$_0&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 #4 0x5559d8f289c0 in std::enable_if, void>::type std::__invoke_r(doris::SchemaScanner::get_next_block_async(doris::RuntimeState*)::$_0&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 apache#5 0x5559d8f289c0 in std::_Function_handler::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 apache#6 0x5559a4140e4a in doris::ThreadPool::dispatch_thread() /home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:608:24 apache#7 0x5559a4117877 in doris::Thread::supervise_thread(void*) /home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:498:5 apache#8 0x7f7912f84ac2 in start_thread nptl/pthread_create.c:442:8 apache#9 0x7f791301684f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
wyxxxcat
pushed a commit
that referenced
this pull request
Mar 17, 2025
Fix problem: 01:48:12 ==4739==ERROR: AddressSanitizer: heap-use-after-free on address 0x6170022e9b90 at pc 0x556df319882f bp 0x7f3c7be04200 sp 0x7f3c7be041f8 01:48:12 READ of size 1 at 0x6170022e9b90 thread T2822 (update_wal_dir_) 01:48:31 #0 0x556df319882e in std::__atomic_base<bool>::load(std::memory_order) const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/atomic_base.h:481:9 01:48:31 #1 0x556df319882e in std::atomic<bool>::load(std::memory_order) const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/atomic:112:22 01:48:31 #2 0x556dfdace32a in doris::WalManager::_update_wal_dir_info_thread() /root/doris/be/src/olap/wal/wal_manager.cpp:477:19 01:48:31 #3 0x556dfdad57df in doris::WalManager::_init_wal_dirs_info()::$_0::operator()() const /root/doris/be/src/olap/wal/wal_manager.cpp:162:48 http://43.132.222.7:8111/buildConfiguration/Doris_DorisBeUt_BeUt/642336?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&showLog=642336_18402_305.18541&logFilter=debug&logView=flowAware ### What problem does this PR solve?
wyxxxcat
pushed a commit
that referenced
this pull request
Mar 18, 2025
…che#49081) When replay wal on test regression-test/suites/mv_p0/dis_26495/dis_26495.groovy, be will core, this pr fix it. *** Query id: 764ad7ba18485829-63a6eb98dcea4981 *** *** is nereids: 1 *** *** tablet id: 0 *** *** Aborted at 1740808023 (unix time) try "date -d @1740808023" if you are using GNU date *** *** Current BE git commitID: a386e8b *** *** SIGSEGV address not mapped to object (@0xD9) received by PID 8864 (TID 23505 OR 0x7f6aa893c640) from PID 217; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/common/signal_handler.h:421 1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so 2# JVM_handle_linux_signal in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so 3# 0x00007F76D2AD4520 in /lib/x86_64-linux-gnu/libc.so.6 4# doris::vectorized::ColumnNullable::insert_range_from_not_nullable(doris::vectorized::IColumn const&, unsigned long, unsigned long) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/columns/column_nullable.cpp:340 5# doris::vectorized::VScanner::_do_projections(doris::vectorized::Block*, doris::vectorized::Block*) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/exec/scan/vscanner.cpp:204 6# doris::vectorized::VScanner::get_block_after_projects(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/exec/scan/vscanner.cpp:84 7# doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr, std::shared_ptr) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:221 8# std::_Function_handler, std::shared_ptr)::$_1::operator()() const::{lambda()#1}>::_M_invoke(std::_Any_data const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291 9# doris::ThreadPool::dispatch_thread() in /mnt/hdd01/ci/doris-deploy-branch-3.0-local/be/lib/doris_be 10# doris::Thread::supervise_thread(void*) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/util/thread.cpp:499 11# start_thread at ./nptl/pthread_create.c:442 12# 0x00007F76D2BB8850 at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:83 172.20.48.99 last coredump sql: ### What problem does this PR solve?
wyxxxcat
pushed a commit
that referenced
this pull request
Mar 20, 2025
… nested decimal precision (apache#48964) before this pr , we deduce the precision of the function named_struct containing the decimal type is same with the precision of the decimal type in select . In this case we can not make right named_struct for multiple decimal value like : ``` mysql> create table t01 (a decimal(6,3), d struct<col:bigint, col1:decimal(7,2)>) properties ("replication_num"="1"); Query OK, 0 rows affected (0.14 sec) mysql> insert into t01 values (123.321, named_struct('col', 1, 'col1', 345.24)); Query OK, 1 row affected (0.26 sec) {'label':'label_a2dc895ee7834701_841639ad6d93923f', 'status':'VISIBLE', 'txnId':'9010'} mysql> select named_struct("col_11", a, "col_12", d) from t01; +--------------------------------------------------------+ | named_struct("col_11", a, "col_12", d) | +--------------------------------------------------------+ | {"col_11":123.321, "col_12":{"col":1, "col1":345.240}} | +--------------------------------------------------------+ 1 row in set (0.17 sec) mysql> insert into t01 values (123.321, named_struct('col', 1, 'col1', 12345.24)); Query OK, 1 row affected (0.17 sec) {'label':'label_77ed6047f6a6494d_810c26becfd0535c', 'status':'VISIBLE', 'txnId':'9011'} mysql> select named_struct("col_11", a, "col_12", d) from t01; ERROR 1105 (HY000): errCode = 2, detailMessage = (10.16.10.6)[E-255]Arithmetic overflow when converting value 12345.24 from type Decimal(7, 2) to type Decimal(6, 3) mysql> desc verbose select named_struct("col_11", a, "col_12", d) from t01; +----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Explain String(Nereids Planner) | +----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | PLAN FRAGMENT 0 | | OUTPUT EXPRS: | | named_struct("col_11", a, "col_12", d)[#2] | | PARTITION: RANDOM | | | | HAS_COLO_PLAN_NODE: false | | | | VRESULT SINK | | MYSQL_PROTOCAL | | | | 0:VOlapScanNode(50) | | TABLE: tt.t01(t01), PREAGGREGATION: ON | | partitions=1/1 (t01) | | tablets=10/10, tabletList=1741751856342,1741751856344,1741751856346 ... | | cardinality=2, avgRowSize=2955.0, numNodes=1 | | pushAggOp=NONE | | final projections: named_struct('col_11', a[#0], 'col_12', CAST(d[#1] AS struct<col:bigint,col1:decimalv3(6,3)>)) | | final project output tuple id: 1 | | tuple ids: 0 | | | | Tuples: | | TupleDescriptor{id=0, tbl=t01} | | SlotDescriptor{id=0, col=a, colUniqueId=0, type=decimalv3(6,3), nullable=true, isAutoIncrement=false, subColPath=null} | | SlotDescriptor{id=1, col=d, colUniqueId=1, type=struct<col:bigint,col1:decimalv3(7,2)>, nullable=true, isAutoIncrement=false, subColPath=null} | | | | TupleDescriptor{id=1, tbl=t01} | | SlotDescriptor{id=2, col=null, colUniqueId=null, type=struct<col_11:decimalv3(6,3),col_12:struct<col:bigint,col1:decimalv3(6,3)>>, nullable=false, isAutoIncrement=false, subColPath=null} | | | | | | ```
wyxxxcat
pushed a commit
that referenced
this pull request
Mar 20, 2025
### What problem does this PR solve?
```
/root/doris/be/src/vec/common/pod_array.h:510:29: runtime error: null pointer passed as argument 2, which is declared to never be null
/root/ldb_toolchain/bin/../usr/include/string.h:43:28: note: nonnull attribute specified here
#0 0x55cb2c2cea1e in void doris::vectorized::PODArray<unsigned char, 4096ul, Allocator<false, false, false, DefaultMemoryAllocator>, 16ul, 15ul>::insert_assume_reserved<char const*, char const*>(char const*, char const*) /root/doris/be/src/vec/common/pod_array.h:510:9
#1 0x55cb2c2ce8a7 in void doris::vectorized::PODArray<unsigned char, 4096ul, Allocator<false, false, false, DefaultMemoryAllocator>, 16ul, 15ul>::insert<char const*, char const*>(char const*, char const*) /root/doris/be/src/vec/common/pod_array.h:472:9
#2 0x55cb5a0b0d50 in doris::vectorized::StringOP::push_value_string(std::basic_string_view<char, std::char_traits<char>> const&, unsigned long, doris::vectorized::PODArray<unsigned char, 4096ul, Allocator<false, false, false, DefaultMemoryAllocator>, 16ul, 15ul>&, doris::vectorized::PODArray<unsigned int, 4096ul, Allocator<false, false, false, DefaultMemoryAllocator>, 16ul, 15ul>&) /root/doris/be/src/vec/functions/function_string.h:128:15
#3 0x55cb5d6843a2 in doris::Status doris::vectorized::FunctionStringParseUrl::vector_parse<false, true>(doris::vectorized::ColumnStr<unsigned int> const*, std::vector<doris::UrlParser::UrlPart, std::allocator<doris::UrlParser::UrlPart>>&, int, doris::vectorized::PODArray<unsigned char, 4096ul, Allocator<false, false, false, DefaultMemoryAllocator>, 16ul, 15ul>&, doris::vectorized::PODArray<unsigned char, 4096ul, Allocator<false, false, false, DefaultMemoryAllocator>, 16ul, 15ul>&, doris::vectorized::PODArray<unsigned int, 4096ul, Allocator<false, false, false, DefaultMemoryAllocator>, 16ul, 15ul>&) /root/doris/be/src/vec/functions/function_string.h:2806:17
#4 0x55cb5d683e9d in auto doris::vectorized::FunctionStringParseUrl::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long) const::'lambda'(auto, auto)::operator()<std::integral_constant<bool, false>, std::integral_constant<bool, true>>(auto, auto) const /root/doris/be/src/vec/functions/function_string.h:2783:13
apache#5 0x55cb5d683c39 in auto std::__invoke_impl<doris::Status, doris::vectorized::FunctionStringParseUrl::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long) const::'lambda'(auto, auto), std::integral_constant<bool, false>, std::integral_constant<bool, true>>(std::__invoke_other, auto&&, std::integral_constant<bool, false>&&, std::integral_constant<bool, true>&&) /root/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14
apache#6 0x55cb5d683af5 in std::__invoke_result<auto, std::integral_constant<bool, false>, std::integral_constant<bool, true>>::type std::__invoke<doris::vectorized::FunctionStringParseUrl::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long) const::'lambda'(auto, auto), std::integral_constant<bool, false>, std::integral_constant<bool, true>>(auto&&, std::integral_constant<bool, false>&&, std::integral_constant<bool, true>&&) /root/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:96:14
apache#7 0x55cb5d682dd3 in std::__detail::__variant::__gen_vtable_impl<std::__detail::__variant::_Multi_array<std::__detail::__variant::__deduce_visit_result<doris::Status> (*)(doris::vectorized::FunctionStringParseUrl::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long) const::'lambda'(auto, auto)&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&)>, std::integer_sequence<unsigned long, 0ul, 1ul>>::__visit_invoke(doris::vectorized::FunctionStringParseUrl::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long) const::'lambda'(auto, auto)&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&) /root/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/variant:1032:11
apache#8 0x55cb5d682ad5 in decltype(auto) std::__do_visit<std::__detail::__variant::__deduce_visit_result<doris::Status>, doris::vectorized::FunctionStringParseUrl::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long) const::'lambda'(auto, auto), std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>(auto&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&) /root/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/variant:1758:15
apache#9 0x55cb5d67766b in std::invoke_result<auto, std::__conditional<is_lvalue_reference_v<std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>>::type<std::variant_alternative<0ul, std::remove_reference<decltype(__variant::__as(std::declval<std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>()))>::type>::type&, std::variant_alternative<0ul, std::remove_reference<decltype(__variant::__as(std::declval<std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>()))>::type>::type&&>, std::__conditional<is_lvalue_reference_v<std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>>::type<std::variant_alternative<0ul, std::remove_reference<decltype(__variant::__as(std::declval<std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>()))>::type>::type&, std::variant_alternative<0ul, std::remove_reference<decltype(__variant::__as(std::declval<std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>()))>::type>::type&&>>::type std::visit<doris::vectorized::FunctionStringParseUrl::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long) const::'lambda'(auto, auto), std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>(auto&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&) /root/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/variant:1859:9
apache#10 0x55cb5d676604 in doris::vectorized::FunctionStringParseUrl::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long) const /root/doris/be/src/vec/functions/function_string.h:2783:13
apache#11 0x55cb51a43fd4 in doris::vectorized::DefaultExecutable::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long) const /root/doris/be/src/vec/functions/function.h:434:26
apache#12 0x55cb583f7dff in doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:119:16
apache#13 0x55cb583e6de9 in doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:244:12
apache#14 0x55cb583e5f53 in doris::vectorized::PreparedFunctionImpl::default_implementation_for_nulls(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long, bool, bool*) const /root/doris/be/src/vec/functions/function.cpp:216:9
apache#15 0x55cb583f7939 in doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:110:5
apache#16 0x55cb583e6de9 in doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:244:12
apache#17 0x55cb583e7069 in doris::vectorized::PreparedFunctionImpl::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:250:12
apache#18 0x55cb51a3fd95 in doris::vectorized::IFunctionBase::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long, bool) const /root/doris/be/src/vec/functions/function.h:193:19
apache#19 0x55cb51a2797f in doris::vectorized::VectorizedFnCall::_do_execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*, std::vector<unsigned int, std::allocator<unsigned int>>&) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:187:5
apache#20 0x55cb51a28a77 in doris::vectorized::VectorizedFnCall::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:220:12
apache#21 0x55cb51b5f3b4 in doris::vectorized::VExprContext::execute(doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vexpr_context.cpp:61:5
apache#22 0x55cb519a6732 in doris::vectorized::Scanner::_do_projections(doris::vectorized::Block*, doris::vectorized::Block*) /root/doris/be/src/vec/exec/scan/scanner.cpp:200:9
apache#23 0x55cb519a1b13 in doris::vectorized::Scanner::get_block_after_projects(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/be/src/vec/exec/scan/scanner.cpp:82:16
apache#24 0x55cb5192a59d in doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>) /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:241:5
apache#25 0x55cb51931c38 in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:148:21
apache#26 0x55cb519314ef in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:147:31
apache#27 0x55cb519312fe in void std::__invoke_impl<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&>(std::__invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&) /root/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14
apache#28 0x55cb5193123e in std::enable_if<is_invocable_r_v<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&>, void>::type std::__invoke_r<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&) /root/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:111:2
apache#29 0x55cb51930e45 in std::_Function_handler<void (), doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()>::_M_invoke(std::_Any_data const&) /root/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290:9
apache#30 0x55cb216f8e3f in std::function<void ()>::operator()() const /root/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591:9
apache#31 0x55cb51940ec6 in doris::vectorized::SimplifiedScanScheduler::submit_scan_task(doris::vectorized::SimplifiedScanTask)::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.h:149:65
apache#32 0x55cb51940e7e in void std::__invoke_impl<void, doris::vectorized::SimplifiedScanScheduler::submit_scan_task(doris::vectorized::SimplifiedScanTask)::'lambda'()&>(std::__invoke_other, doris::vectorized::SimplifiedScanScheduler::submit_scan_task(doris::vectorized::SimplifiedScanTask)::'lambda'()&) /root/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14
apache#33 0x55cb51940dbe in std::enable_if<is_invocable_r_v<void, doris::vectorized::SimplifiedScanScheduler::submit_scan_task(doris::vectorized::SimplifiedScanTask)::'lambda'()&>, void>::type std::__invoke_r<void, doris::vectorized::SimplifiedScanScheduler::submit_scan_task(doris::vectorized::SimplifiedScanTask)::'lambda'()&>(doris::vectorized::SimplifiedScanScheduler::submit_scan_task(doris::vectorized::SimplifiedScanTask)::'lambda'()&) /root/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:111:2
apache#34 0x55cb51940745 in std::_Function_handler<void (), doris::vectorized::SimplifiedScanScheduler::submit_scan_task(doris::vectorized::SimplifiedScanTask)::'lambda'()>::_M_invoke(std::_Any_data const&) /root/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290:9
apache#35 0x55cb216f8e3f in std::function<void ()>::operator()() const /root/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591:9
apache#36 0x55cb28bd9844 in doris::FunctionRunnable::run() /root/doris/be/src/util/threadpool.cpp:64:27
apache#37 0x55cb28bb84b9 in doris::ThreadPool::dispatch_thread() /root/doris/be/src/util/threadpool.cpp:616:24
apache#38 0x55cb28bfd263 in void std::__invoke_impl<void, void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(std::__invoke_memfun_deref, void (doris::ThreadPool::*&)(), doris::ThreadPool*&) /root/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:74:14
apache#39 0x55cb28bfd068 in std::__invoke_result<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>::type std::__invoke<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(void (doris::ThreadPool::*&)(), doris::ThreadPool*&) /root/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:96:14
apache#40 0x55cb28bfcfa0 in void std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>::__call<void, 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /root/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/functional:506:11
apache#41 0x55cb28bfcd95 in void std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>::operator()<void>() /root/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/functional:591:17
apache#42 0x55cb28bfcc8e in void std::__invoke_impl<void, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&>(std::__invoke_other, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&) /root/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14
apache#43 0x55cb28bfcbce in std::enable_if<is_invocable_r_v<void, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&>, void>::type std::__invoke_r<void, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&>(std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&) /root/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:111:2
apache#44 0x55cb28bfc665 in std::_Function_handler<void (), std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>>::_M_invoke(std::_Any_data const&) /root/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290:9
apache#45 0x55cb216f8e3f in std::function<void ()>::operator()() const /root/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591:9
apache#46 0x55cb28b74241 in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:498:5
apache#47 0x55cb2142be0a in asan_thread_start(void*) crtstuff.c
apache#48 0x7f17840221c9 in start_thread (/lib64/libpthread.so.0+0x81c9) (BuildId: 7c4add5c7a885e6ff4ce17867d6a2286e4420eec)
apache#49 0x7f1784a118d2 in clone (/lib64/libc.so.6+0x398d2) (BuildId: 4ee3325955e3b55b6805f33959b7cb77745ad625)
wyxxxcat
pushed a commit
that referenced
this pull request
Mar 24, 2025
…ings (apache#49241) ### What problem does this PR solve? ``` #0 0x55f5cd170097 in doris::validate_ascii_fast(char const*, unsigned long) /mnt/disk12/yanxuecheng/tmp-doris/be/src/util/simd/vstring_function.h:60:37 #1 0x55f5cd16e3c0 in doris::simd::VStringFunctions::is_ascii(doris::StringRef const&) /mnt/disk12/yanxuecheng/tmp-doris/be/src/util/simd/vstring_function.h:212:16 #2 0x55f5cd16dcd3 in auto doris::vectorized::SubReplaceImpl::replace_execute(doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long)::'lambda'(auto, auto, auto, auto)::operator()<std::integral_constant<bool, false>, std::integral_constant<bool, false>, std::integral_constant<bool, false>, std::integral_constant<bool, false>>(auto, auto, auto, auto) const /mnt/disk12/yanxuecheng/tmp-doris/be/src/vec/functions/function_string.h:3600:25 #3 0x55f5cd16d926 in auto std::__invoke_impl<void, doris::vectorized::SubReplaceImpl::replace_execute(doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long)::'lambda'(auto, auto, auto, auto), std::integral_constant<bool, false>, std::integral_constant<bool, false>, std::integral_constant<bool, false>, std::integral_constant<bool, false>>(std::__invoke_other, auto&&, std::integral_constant<bool, false>&&, std::integral_constant<bool, false>&&, std::integral_constant<bool, false>&&, std::integral_constant<bool, false>&&) /mnt/disk6/common/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14 #4 0x55f5cd16d8b6 in std::__invoke_result<auto, std::integral_constant<bool, false>, std::integral_constant<bool, false>, std::integral_constant<bool, false>, std::integral_constant<bool, false>>::type std::__invoke<doris::vectorized::SubReplaceImpl::replace_execute(doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long)::'lambda'(auto, auto, auto, auto), std::integral_constant<bool, false>, std::integral_constant<bool, false>, std::integral_constant<bool, false>, std::integral_constant<bool, false>>(auto&&, std::integral_constant<bool, false>&&, std::integral_constant<bool, false>&&, std::integral_constant<bool, false>&&, std::integral_constant<bool, false>&&) /mnt/disk6/common/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:96:14 apache#5 0x55f5cd16cfed in std::__detail::__variant::__gen_vtable_impl<std::__detail::__variant::_Multi_array<std::__detail::__variant::__deduce_visit_result<void> (*)(doris::vectorized::SubReplaceImpl::replace_execute(doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long)::'lambda'(auto, auto, auto, auto)&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&)>, std::integer_sequence<unsigned long, 0ul, 0ul, 0ul, 0ul>>::__visit_invoke(doris::vectorized::SubReplaceImpl::replace_execute(doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long)::'lambda'(auto, auto, auto, auto)&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&) /mnt/disk6/common/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/variant:1032:11 apache#6 0x55f5cd16ce46 in decltype(auto) std::__do_visit<std::__detail::__variant::__deduce_visit_result<void>, doris::vectorized::SubReplaceImpl::replace_execute(doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long)::'lambda'(auto, auto, auto, auto), std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>(auto&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&) /mnt/disk6/common/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/variant:1758:15 apache#7 0x55f5cd16c7ca in std::invoke_result<auto, std::__conditional<is_lvalue_reference_v<std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>>::type<std::variant_alternative<0ul, std::remove_reference<decltype(__variant::__as(std::declval<std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>()))>::type>::type&, std::variant_alternative<0ul, std::remove_reference<decltype(__variant::__as(std::declval<std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>()))>::type>::type&&>, std::__conditional<is_lvalue_reference_v<std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>>::type<std::variant_alternative<0ul, std::remove_reference<decltype(__variant::__as(std::declval<std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>()))>::type>::type&, std::variant_alternative<0ul, std::remove_reference<decltype(__variant::__as(std::declval<std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>()))>::type>::type&&>, std::__conditional<is_lvalue_reference_v<std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>>::type<std::variant_alternative<0ul, std::remove_reference<decltype(__variant::__as(std::declval<std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>()))>::type>::type&, std::variant_alternative<0ul, std::remove_reference<decltype(__variant::__as(std::declval<std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>()))>::type>::type&&>, std::__conditional<is_lvalue_reference_v<std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>>::type<std::variant_alternative<0ul, std::remove_reference<decltype(__variant::__as(std::declval<std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>()))>::type>::type&, std::variant_alternative<0ul, std::remove_reference<decltype(__variant::__as(std::declval<std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>()))>::type>::type&&>>::type std::visit<doris::vectorized::SubReplaceImpl::replace_execute(doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long)::'lambda'(auto, auto, auto, auto), std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>(auto&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&) /mnt/disk6/common/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/variant:1859:9 apache#8 0x55f5cd16bd75 in doris::vectorized::SubReplaceImpl::replace_execute(doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long) /mnt/disk12/yanxuecheng/tmp-doris/be/src/vec/functions/function_string.h:3594:9 ``` Related PR:(apache#40929)
wyxxxcat
pushed a commit
that referenced
this pull request
Mar 28, 2025
…iteral with one more subfield causing BE coredump (apache#49485) Fixed the issue of inserting into a struct type string literal with one more subfield causing BE coredump some situation like this blow will make BE core ``` create table t(a int, b int, s struct<a:int>) PROPERTIES ("replication_allocation" = "tag.location.default: 1"); insert into t values(1,1,'{1,2}'); ``` core info: ``` [WARNING!] /sys/kernel/mm/transparent_hugepage/enabled: [always] madvise never, Doris not recommend turning on THP, which may cause the BE process to use more memory and cannot be freed in time. Turn off THP: `echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/enabled` start BE in local mode ================================================================= ==2818976==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x603000f17a90 at pc 0x55eba7fdd69c bp 0x7f03243dbd30 sp 0x7f03243dbd28 READ of size 8 at 0x603000f17a90 thread T928 (brpc_light) #0 0x55eba7fdd69b in std::__shared_ptr<doris::vectorized::DataTypeSerDe, (__gnu_cxx::_Lock_policy)2>::get() const /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1291:16 #1 0x55eba7fdd649 in std::__shared_ptr_access<doris::vectorized::DataTypeSerDe, (__gnu_cxx::_Lock_policy)2, false, false>::_M_get() const /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:990:66 #2 0x55eba7fda316 in std::__shared_ptr_access<doris::vectorized::DataTypeSerDe, (__gnu_cxx::_Lock_policy)2, false, false>::operator->() const /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:984:9 #3 0x55ebcfc8b19e in doris::vectorized::DataTypeStructSerDe::deserialize_one_cell_from_json(doris::vectorized::IColumn&, doris::Slice&, doris::vectorized::DataTypeSerDe::FormatOptions const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/data_types/serde/data_type_struct_serde.cpp:200:25 #4 0x55ebdac49c8b in doris::vectorized::ConvertImplGenericFromString::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long) /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function_cast.h:618:36 apache#5 0x55ebda2234d0 in doris::Status std::__invoke_impl<doris::Status, doris::Status (*&)(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long), doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long, unsigned long>(std::__invoke_other, doris::Status (*&)(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long), doris::FunctionContext*&&, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long&&, unsigned long&&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 apache#6 0x55ebda22323a in std::enable_if<is_invocable_r_v<doris::Status, doris::Status (*&)(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long), doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long, unsigned long>, doris::Status>::type std::__invoke_r<doris::Status, doris::Status (*&)(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long), doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long, unsigned long>(doris::Status (*&)(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long), doris::FunctionContext*&&, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long&&, unsigned long&&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:114:9 apache#7 0x55ebda222e71 in std::_Function_handler<doris::Status (doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long, unsigned long), doris::Status (*)(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long)>::_M_invoke(std::_Any_data const&, doris::FunctionContext*&&, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long&&, unsigned long&&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 apache#8 0x55ebd9b3f1d2 in std::function<doris::Status (doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long, unsigned long)>::operator()(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long, unsigned long) const /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9 apache#9 0x55ebda1ad7c7 in doris::vectorized::FunctionCast::prepare_remove_nullable(doris::FunctionContext*, std::shared_ptr<doris::vectorized::IDataType const> const&, std::shared_ptr<doris::vectorized::IDataType const> const&, bool) const::'lambda'(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long)::operator()(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function_cast.h:2241:17 apache#10 0x55ebda1acee2 in doris::Status std::__invoke_impl<doris::Status, doris::vectorized::FunctionCast::prepare_remove_nullable(doris::FunctionContext*, std::shared_ptr<doris::vectorized::IDataType const> const&, std::shared_ptr<doris::vectorized::IDataType const> const&, bool) const::'lambda'(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long)&, doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long, unsigned long>(std::__invoke_other, doris::vectorized::FunctionCast::prepare_remove_nullable(doris::FunctionContext*, std::shared_ptr<doris::vectorized::IDataType const> const&, std::shared_ptr<doris::vectorized::IDataType const> const&, bo@@@ ```
wyxxxcat
pushed a commit
that referenced
this pull request
Apr 1, 2025
…he#49700) ### What problem does this PR solve? When a BE is down, global object should shut down following a specific order to ensure BE will not crash. Because all running queries will finish once fragment mgr finished, stream mgr must be released before it because it may wake up some blocked tasks. #0 __GI___pthread_sigmask (how=2, newmask=<optimized out>, oldmask=0x0) at ./nptl/pthread_sigmask.c:43 #1 0x00007f842b2bb71e in PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] () from /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so #2 0x00007f842b2bc206 in JVM_handle_linux_signal () from /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so #3 <signal handler called> #4 ___pthread_mutex_lock (mutex=0x1b8) at ./nptl/pthread_mutex_lock.c:80 apache#5 0x000055c5f03edcca in __gthread_mutex_lock (__mutex=0x1b8) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/x86_64-linux-gnu/c++/11/bits/gthr-default.h:749 apache#6 std::mutex::lock (this=0x1b8) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_mutex.h:100 apache#7 std::lock_guard<std::mutex>::lock_guard (__m=..., this=<optimized out>) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_mutex.h:229 apache#8 doris::RuntimeProfile::add_info_string (this=0x0, key=..., value=...) at /home/zcp/repo_center/doris_master/doris/be/src/util/runtime_profile.cpp:354 apache#9 0x000055c5f9cccaa7 in doris::pipeline::PipelineTask::_state_transition (this=0x7f84015b4000, new_state=doris::pipeline::PipelineTask::State::RUNNABLE) at /home/zcp/repo_center/doris_master/doris/be/src/pipeline/pipeline_task.h:339 apache#10 0x000055c5fa53aec5 in doris::pipeline::PipelineTask::wake_up (this=0x7fffbd28a8e8, dep=<optimized out>) at /home/zcp/repo_center/doris_master/doris/be/src/pipeline/pipeline_task.cpp:748 apache#11 0x000055c5f9cba3f5 in doris::pipeline::Dependency::set_ready (this=0x7f832f848310) at /home/zcp/repo_center/doris_master/doris/be/src/pipeline/dependency.cpp:72 apache#12 0x000055c5f9c53bcd in doris::vectorized::VDataStreamRecvr::SenderQueue::set_source_ready (this=0x7f839d485880) at /home/zcp/repo_center/doris_master/doris/be/src/vec/runtime/vdata_stream_recvr.cpp:142 apache#13 doris::vectorized::VDataStreamRecvr::SenderQueue::close (this=0x7f839d485880) at /home/zcp/repo_center/doris_master/doris/be/src/vec/runtime/vdata_stream_recvr.cpp:301 apache#14 0x000055c5f9c55b39 in doris::vectorized::VDataStreamRecvr::close (this=0x7f83fbe78940) at /home/zcp/repo_center/doris_master/doris/be/src/vec/runtime/vdata_stream_recvr.cpp:488 apache#15 0x000055c5f9c4dc68 in doris::vectorized::VDataStreamMgr::~VDataStreamMgr (this=0x7f83f6d99da0) at /home/zcp/repo_center/doris_master/doris/be/src/vec/runtime/vdata_stream_mgr.cpp:58 apache#16 0x000055c5f00cb651 in doris::ExecEnv::destroy (this=0x55c5fdf7a198 <doris::ExecEnv::GetInstance()::s_exec_env>) at /home/zcp/repo_center/doris_master/doris/be/src/runtime/exec_env_init.cpp:796 apache#17 0x000055c5ef387909 in main (argc=<optimized out>, argv=<optimized out>) at /home/zcp/repo_center/doris_master/doris/be/src/service/doris_main.cpp:635
dataroaring
pushed a commit
that referenced
this pull request
Apr 1, 2025
…49269) ### What problem does this PR solve? 1. fix micro BUILD_FILE_CACHE_MICROBENCH_TOOL not work in file_cache_microbench.cpp 2. fix MircobenchS3FileWriter typo error 3. ./build.sh default not compile file_cache_microbench 4. fix , exit coredump ``` Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f54c898a550 in __pthread_timedjoin_ex () from /lib64/libpthread.so.0 [Current thread is 1 (Thread 0x7f54cafad940 (LWP 2810860))] (gdb) bt #0 0x00007f54c898a550 in __pthread_timedjoin_ex () from /lib64/libpthread.so.0 #1 0x000055eb6a7ac513 in __gthread_join (__value_ptr=0x0, __threadid=<optimized out>) at /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:669 #2 std::thread::join (this=0x55eb6eda1ce0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:112 #3 0x000055eb5cb548bb in ThreadPool::~ThreadPool (this=this@entry=0x7fff65768480) #4 0x000055eb5cb4aebe in JobManager::~JobManager (this=this@entry=0x7fff65768420) apache#5 0x000055eb5cb49a14 in main (argc=1, argv=<optimized out>) ``` Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [x] No need to test or manual test. Explain why: - [x] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [x] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [x] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
wyxxxcat
pushed a commit
that referenced
this pull request
Apr 7, 2025
…r/signal to avoid mult… (apache#49739) …ithread read and write wrapper ### What problem does this PR solve? ```cpp ================================================================= ==28655==ERROR: AddressSanitizer: heap-use-after-free on address 0x61000331f350 at pc 0x5577863cea6c bp 0x7fac1d2a9ef0 sp 0x7fac1d2a9ee8 READ of size 1 at 0x61000331f350 thread T1933 (Pipe_normal [wo) #0 0x5577863cea6b in doris::RuntimeFilterWrapper::debug_string[abi:cxx11]() /root/doris/be/src/runtime_filter/runtime_filter_wrapper.cpp:585:46 #1 0x55778632f8f5 in doris::RuntimeFilter::_debug_string[abi:cxx11]() const /root/doris/be/src/runtime_filter/runtime_filter.cpp:128:61 #2 0x55778638d946 in doris::RuntimeFilterConsumer::debug_string[abi:cxx11]() const /root/doris/be/src/runtime_filter/runtime_filter_consumer.h:61:57 #3 0x55778634ed24 in doris::RuntimeFilterConsumer::_set_state(doris::RuntimeFilterConsumer::State) /root/doris/be/src/runtime_filter/runtime_filter_consumer.h:125:43 #4 0x55778634c6fa in doris::RuntimeFilterConsumer::acquire_expr(std::vector<std::shared_ptr<doris::vectorized::VRuntimeFilterWrapper>, std::allocator<std::shared_ptr<doris::vectorized::VRuntimeFilterWrapper> > >&) /root/doris/be/src/runtime_filter/runtime_filter_consumer.cpp:67:9 apache#5 0x5577a1fbdd1b in doris::RuntimeFilterConsumerHelper::acquire_runtime_filter(std::vector<std::shared_ptr<doris::vectorized::VExprContext>, std::allocator<std::shared_ptr<doris::vectorized::VExprContext> > >&) /root/doris/be/src/runtime_filter/runtime_filter_consumer_helper.cpp:90:9 apache#6 0x5577b9aec26c in doris::pipeline::ScanLocalState<doris::pipeline::OlapScanLocalState>::open(doris::RuntimeState*) /root/doris/be/src/pipeline/exec/scan_operator.cpp:100:5 apache#7 0x5577ba853e05 in doris::pipeline::PipelineTask::_open() /root/doris/be/src/pipeline/pipeline_task.cpp:217:32 apache#8 0x5577ba8608e0 in doris::pipeline::PipelineTask::execute(bool*) /root/doris/be/src/pipeline/pipeline_task.cpp:407:9 apache#9 0x5577ba8b4bb5 in doris::pipeline::TaskScheduler::_do_work(int) /root/doris/be/src/pipeline/task_scheduler.cpp:147:9 apache#10 0x5577869c537c in doris::ThreadPool::dispatch_thread() /root/doris/be/src/util/threadpool.cpp:619:24 apache#11 0x55778699ba1e in std::function<void ()>::operator()() const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9 apache#12 0x55778699ba1e in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:496:5 apache#13 0x7fb57bef3608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477:8 apache#14 0x7fb57c1a0132 in __clone /build/glibc-SzIz7B/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95 0x61000331f350 is located 16 bytes inside of 192-byte region [0x61000331f340,0x61000331f400) freed by thread T817 (brpc_light) here: #0 0x557781e1c80d in operator delete(void*) (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x3395280d) (BuildId: 45c9f8cc52f91b28) #1 0x557781e778b8 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::operator=(std::__shared_count<(__gnu_cxx::_Lock_policy)2> const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:721:15 #2 0x55778634d1d1 in std::__shared_ptr<doris::RuntimeFilterWrapper, (__gnu_cxx::_Lock_policy)2>::operator=(std::__shared_ptr<doris::RuntimeFilterWrapper, (__gnu_cxx::_Lock_policy)2> const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1148:69 #3 0x55778634d1d1 in std::shared_ptr<doris::RuntimeFilterWrapper>::operator=(std::shared_ptr<doris::RuntimeFilterWrapper> const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr.h:359:65 #4 0x55778634d1d1 in doris::RuntimeFilterConsumer::signal(doris::RuntimeFilter*) /root/doris/be/src/runtime_filter/runtime_filter_consumer.cpp:76:14 apache#5 0x557785dd52c3 in auto doris::FragmentMgr::apply_filterv2(doris::PPublishFilterRequestV2 const*, butil::IOBufAsZeroCopyInputStream*)::$_0::operator()<std::shared_ptr<doris::RuntimeFilterConsumer> >(std::shared_ptr<doris::RuntimeFilterConsumer>&) const /root/doris/be/src/runtime/fragment_mgr.cpp:1295:72 apache#6 0x557785dd52c3 in void std::__invoke_impl<void, doris::FragmentMgr::apply_filterv2(doris::PPublishFilterRequestV2 const*, butil::IOBufAsZeroCopyInputStream*)::$_0&, std::shared_ptr<doris::RuntimeFilterConsumer>&>(std::__invoke_other, doris::FragmentMgr::apply_filterv2(doris::PPublishFilterRequestV2 const*, butil::IOBufAsZeroCopyInputStream*)::$_0&, std::shared_ptr<doris::RuntimeFilterConsumer>&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 apache#7 0x557785dd52c3 in std::__invoke_result<doris::FragmentMgr::apply_filterv2(doris::PPublishFilterRequestV2 const*, butil::IOBufAsZeroCopyInputStream*)::$_0&, std::shared_ptr<doris::RuntimeFilterConsumer>&>::type std::__invoke<doris::FragmentMgr::apply_filterv2(doris::PPublishFilterRequestV2 const*, butil::IOBufAsZeroCopyInputStream*)::$_0&, std::shared_ptr<doris::RuntimeFilterConsumer>&>(doris::FragmentMgr::apply_filterv2(doris::PPublishFilterRequestV2 const*, butil::IOBufAsZeroCopyInputStream*)::$_0&, std::shared_ptr<doris::RuntimeFilterConsumer>&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 apache#8 0x557785dd52c3 in std::ranges::in_fun_result<__gnu_cxx::__normal_iterator<std::shared_ptr<doris::RuntimeFilterConsumer>*, std::vector<std::shared_ptr<doris::RuntimeFilterConsumer>, std::allocator<std::shared_ptr<doris::RuntimeFilterConsumer> > > >, doris::FragmentMgr::apply_filterv2(doris::PPublishFilterRequestV2 const*, butil::IOBufAsZeroCopyInputStream*)::$_0> std::ranges::__for_each_fn::operator()<__gnu_cxx::__normal_iterator<std::shared_ptr<doris::RuntimeFilterConsumer>*, std::vector<std::shared_ptr<doris::RuntimeFilterConsumer>, std::allocator<std::shared_ptr<doris::RuntimeFilterConsumer> > > >, __gnu_cxx::__normal_iterator<std::shared_ptr<doris::RuntimeFilterConsumer>*, std::vector<std::shared_ptr<doris::RuntimeFilterConsumer>, std::allocator<std::shared_ptr<doris::RuntimeFilterConsumer> > > >, std::identity, doris::FragmentMgr::apply_filterv2(doris::PPublishFilterRequestV2 const*, butil::IOBufAsZeroCopyInputStream*)::$_0>(__gnu_cxx::__normal_iterator<std::shared_ptr<doris::RuntimeFilterConsumer>*, std::vector<std::shared_ptr<doris::RuntimeFilterConsumer>, std::allocator<std::shared_ptr<doris::RuntimeFilterConsumer> > > >, __gnu_cxx::__normal_iterator<std::shared_ptr<doris::RuntimeFilterConsumer>*, std::vector<std::shared_ptr<doris::RuntimeFilterConsumer>, std::allocator<std::shared_ptr<doris::RuntimeFilterConsumer> > > >, doris::FragmentMgr::apply_filterv2(doris::PPublishFilterRequestV2 const*, butil::IOBufAsZeroCopyInputStream*)::$_0, std::identity) const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/ranges_algo.h:187:4 apache#9 0x557785dd52c3 in _ZNKSt6ranges13__for_each_fnclIRSt6vectorISt10shared_ptrIN5doris21RuntimeFilterConsumerEESaIS6_EESt8identityZNS4_11FragmentMgr14apply_filterv2EPKNS4_23PPublishFilterRequestV2EPN5butil26IOBufAsZeroCopyInputStreamEE3$_0EENS_13in_fun_resultINSt11conditionalIL_ZNS_14borrowed_rangeIT_EEEDTclsr8__detailE14__ranges_beginclsr3stdE7declvalIRSM_EEEENS_8danglingEE4typeET1_EEOSM_SS_T0_ /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/ranges_algo.h:197:9 apache#10 0x557785dd52c3 in doris::FragmentMgr::apply_filterv2(doris::PPublishFilterRequestV2 const*, butil::IOBufAsZeroCopyInputStream*) /root/doris/be/src/runtime/fragment_mgr.cpp:1295:13 apache#11 0x5577865a517e in doris::PInternalService::apply_filterv2(google::protobuf::RpcController*, doris::PPublishFilterRequestV2 const*, doris::PPublishFilterResponse*, google::protobuf::Closure*)::$_0::operator()() const /root/doris/be/src/service/internal_service.cpp:1444:48 apache#12 0x5577865a517e in void std::__invoke_impl<void, doris::PInternalService::apply_filterv2(google::protobuf::RpcController*, doris::PPublishFilterRequestV2 const*, doris::PPublishFilterResponse*, google::protobuf::Closure*)::$_0&>(std::__invoke_other, doris::PInternalService::apply_filterv2(google::protobuf::RpcController*, doris::PPublishFilterRequestV2 const*, doris::PPublishFilterResponse*, google::protobuf::Closure*)::$_0&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 apache#13 0x5577865a517e in std::enable_if<is_invocable_r_v<void, doris::PInternalService::apply_filterv2(google::protobuf::RpcController*, doris::PPublishFilterRequestV2 const*, doris::PPublishFilterResponse*, google::protobuf::Closure*)::$_0&>, void>::type std::__invoke_r<void, doris::PInternalService::apply_filterv2(google::protobuf::RpcController*, doris::PPublishFilterRequestV2 const*, doris::PPublishFilterResponse*, google::protobuf::Closure*)::$_0&>(doris::PInternalService::apply_filterv2(google::protobuf::RpcController*, doris::PPublishFilterRequestV2 const*, doris::PPublishFilterResponse*, google::protobuf::Closure*)::$_0&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 apache#14 0x5577865a517e in std::_Function_handler<void (), doris::PInternalService::apply_filterv2(google::protobuf::RpcController*, doris::PPublishFilterRequestV2 const*, doris::PPublishFilterResponse*, google::protobuf::Closure*)::$_0>::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 apache#15 0x5577865f07c1 in std::function<void ()>::operator()() const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9 apache#16 0x5577865f07c1 in doris::WorkThreadPool<false>::work_thread(int) /root/doris/be/src/util/work_thread_pool.hpp:158:17 apache#17 0x5577bddb233f in execute_native_thread_routine /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/src/c++11/../../../../../libstdc++-v3/src/c++11/thread.cc:82:18 previously allocated by thread T781 (brpc_light) here: #0 0x557781e1bfad in operator new(unsigned long) (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x33951fad) (BuildId: 45c9f8cc52f91b28) #1 0x5577863411ae in __gnu_cxx::new_allocator<std::_Sp_counted_ptr_inplace<doris::RuntimeFilterWrapper, std::allocator<doris::RuntimeFilterWrapper>, (__gnu_cxx::_Lock_policy)2> >::allocate(unsigned long, void const*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/ext/new_allocator.h:121:27 #2 0x5577863411ae in std::allocator<std::_Sp_counted_ptr_inplace<doris::RuntimeFilterWrapper, std::allocator<doris::RuntimeFilterWrapper>, (__gnu_cxx::_Lock_policy)2> >::allocate(unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/allocator.h:173:32 #3 0x5577863411ae in std::allocator_traits<std::allocator<std::_Sp_counted_ptr_inplace<doris::RuntimeFilterWrapper, std::allocator<doris::RuntimeFilterWrapper>, (__gnu_cxx::_Lock_policy)2> > >::allocate(std::allocator<std::_Sp_counted_ptr_inplace<doris::RuntimeFilterWrapper, std::allocator<doris::RuntimeFilterWrapper>, (__gnu_cxx::_Lock_policy)2> >&, unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:460:20 #4 0x5577863411ae in std::__allocated_ptr<std::allocator<std::_Sp_counted_ptr_inplace<doris::RuntimeFilterWrapper, std::allocator<doris::RuntimeFilterWrapper>, (__gnu_cxx::_Lock_policy)2> > > std::__allocate_guarded<std::allocator<std::_Sp_counted_ptr_inplace<doris::RuntimeFilterWrapper, std::allocator<doris::RuntimeFilterWrapper>, (__gnu_cxx::_Lock_policy)2> > >(std::allocator<std::_Sp_counted_ptr_inplace<doris::RuntimeFilterWrapper, std::allocator<doris::RuntimeFilterWrapper>, (__gnu_cxx::_Lock_policy)2> >&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/allocated_ptr.h:97:21 apache#5 0x5577863411ae in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<doris::RuntimeFilterWrapper, std::allocator<doris::RuntimeFilterWrapper>, doris::RuntimeFilterParams*>(doris::RuntimeFilterWrapper*&, std::_Sp_alloc_shared_tag<std::allocator<doris::RuntimeFilterWrapper> >, doris::RuntimeFilterParams*&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:648:19 apache#6 0x55778632e555 in std::__shared_ptr<doris::RuntimeFilterWrapper, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<doris::RuntimeFilterWrapper>, doris::RuntimeFilterParams*>(std::_Sp_alloc_shared_tag<std::allocator<doris::RuntimeFilterWrapper> >, doris::RuntimeFilterParams*&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1337:14 apache#7 0x55778632e555 in std::shared_ptr<doris::RuntimeFilterWrapper>::shared_ptr<std::allocator<doris::RuntimeFilterWrapper>, doris::RuntimeFilterParams*>(std::_Sp_alloc_shared_tag<std::allocator<doris::RuntimeFilterWrapper> >, doris::RuntimeFilterParams*&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr.h:409:4 apache#8 0x55778632e555 in std::shared_ptr<doris::RuntimeFilterWrapper> std::allocate_shared<doris::RuntimeFilterWrapper, std::allocator<doris::RuntimeFilterWrapper>, doris::RuntimeFilterParams*>(std::allocator<doris::RuntimeFilterWrapper> const&, doris::RuntimeFilterParams*&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr.h:860:14 apache#9 0x55778632e555 in std::shared_ptr<doris::RuntimeFilterWrapper> std::make_shared<doris::RuntimeFilterWrapper, doris::RuntimeFilterParams*>(doris::RuntimeFilterParams*&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr.h:876:14 apache#10 0x55778632e555 in doris::RuntimeFilter::_init_with_desc(doris::TRuntimeFilterDesc const*, doris::TQueryOptions const*) /root/doris/be/src/runtime_filter/runtime_filter.cpp:123:16 apache#11 0x5577863794de in doris::RuntimeFilterConsumer::create(doris::RuntimeFilterParamsContext*, doris::TRuntimeFilterDesc const*, int, std::shared_ptr<doris::RuntimeFilterConsumer>*, doris::RuntimeProfile*) /root/doris/be/src/runtime_filter/runtime_filter_consumer.h:46:9 apache#12 0x55778636bb9f in doris::RuntimeFilterMgr::register_consumer_filter(doris::TRuntimeFilterDesc const&, int, std::shared_ptr<doris::RuntimeFilterConsumer>*, doris::RuntimeProfile*) /root/doris/be/src/runtime_filter/runtime_filter_mgr.cpp:80:5 apache#13 0x557786147c10 in doris::RuntimeState::register_consumer_runtime_filter(doris::TRuntimeFilterDesc const&, bool, int, std::shared_ptr<doris::RuntimeFilterConsumer>*, doris::RuntimeProfile*) /root/doris/be/src/runtime/runtime_state.cpp:514:17 apache#14 0x5577a1fbc3e1 in doris::RuntimeFilterConsumerHelper::_register_runtime_filter(bool) /root/doris/be/src/runtime_filter/runtime_filter_consumer_helper.cpp:51:9 ``` ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [x] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [x] Other reason <!-- Add your reason? --> - Behavior changed: - [x] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [x] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
wyxxxcat
pushed a commit
that referenced
this pull request
Apr 17, 2025
### What problem does this PR solve?
Fix 1
```
read.F20250412 04:16:47.972313 198085 thread_context.h:204] Check failed: is_attach_task()
9# google::LogMessageFatal::~LogMessageFatal() in /mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be
10# Allocator<false, false, false, DefaultMemoryAllocator>::sys_memory_check(unsigned long) const at /home/zcp/repo_center/doris_master/doris/be/src/vec/common/allocator.cpp:191
11# Allocator<false, false, false, DefaultMemoryAllocator>::alloc_impl(unsigned long, unsigned long) at /home/zcp/repo_center/doris_master/doris/be/src/vec/common/allocator.h:254
12# doris::io::Memory<Allocator<false, false, false, DefaultMemoryAllocator> >::Memory(unsigned long) at /home/zcp/repo_center/doris_master/doris/be/src/io/fs/s3_file_bufferpool.cpp:48
13# doris::io::FileBuffer::FileBuffer(doris::io::BufferType, std::function<std::unique_ptr<doris::io::FileBlocksHolder, std::default_delete<doris::io::FileBlocksHolder> > ()>, unsigned long, doris::io::OperationState) at /home/zcp/repo_center/doris_master/doris/be/src/io/fs/s3_file_bufferpool.cpp:86
14# doris::io::UploadFileBuffer::UploadFileBuffer(std::function<void (doris::io::UploadFileBuffer&)>, doris::io::OperationState, unsigned long, std::function<std::unique_ptr<doris::io::FileBlocksHolder, std::default_delete<doris::io::FileBlocksHolder> > ()>) at /home/zcp/repo_center/doris_master/doris/be/src/io/fs/s3_file_bufferpool.h:157
15# decltype (::new ((void*)(0)) doris::io::UploadFileBuffer(std::declval<std::function<void (doris::io::UploadFileBuffer&)> >(), std::declval<doris::io::OperationState>(), std::declval<unsigned long&>(), std::declval<std::function<std::unique_ptr<doris::io::FileBlocksHolder, std::default_delete<doris::io::FileBlocksHolder> > ()> >())) std::construct_at<doris::io::UploadFileBuffer, std::function<void (doris::io::UploadFileBuffer&)>, doris::io::OperationState, unsigned long&, std::function<std::unique_ptr<doris::io::FileBlocksHolder, std::default_delete<doris::io::FileBlocksHolder> > ()> >(doris::io::UploadFileBuffer*, std::function<void (doris::io::UploadFileBuffer&)>&&, doris::io::OperationState&&, unsigned long&, std::function<std::unique_ptr<doris::io::FileBlocksHolder, std::default_delete<doris::io::FileBlocksHolder> > ()>&&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:97
16# doris::io::FileBufferBuilder::build(std::shared_ptr<doris::io::FileBuffer>*) at /home/zcp/repo_center/doris_master/doris/be/src/io/fs/s3_file_bufferpool.cpp:247
17# doris::io::S3FileWriter::_build_upload_buffer() at /home/zcp/repo_center/doris_master/doris/be/src/io/fs/s3_file_writer.cpp:199
18# doris::io::S3FileWriter::_close_impl() at /home/zcp/repo_center/doris_master/doris/be/src/io/fs/s3_file_writer.cpp:215
19# doris::io::S3FileWriter::close(bool) at /home/zcp/repo_center/doris_master/doris/be/src/io/fs/s3_file_writer.cpp:156
20# doris::vectorized::ParquetOutputStream::Close() at /home/zcp/repo_center/doris_master/doris/be/src/vec/runtime/vparquet_transformer.cpp:85
21# doris::vectorized::ParquetOutputStream::~ParquetOutputStream() at /home/zcp/repo_center/doris_master/doris/be/src/vec/runtime/vparquet_transformer.cpp:58
22# doris::vectorized::ParquetOutputStream::~ParquetOutputStream() at /home/zcp/repo_center/doris_master/doris/be/src/vec/runtime/vparquet_transformer.cpp:56
23# doris::vectorized::VParquetTransformer::~VParquetTransformer() at /home/zcp/repo_center/doris_master/doris/be/src/vec/runtime/vparquet_transformer.h:110
24# doris::vectorized::VParquetTransformer::~VParquetTransformer() at /home/zcp/repo_center/doris_master/doris/be/src/vec/runtime/vparquet_transformer.h:110
25# doris::vectorized::VFileResultWriter::~VFileResultWriter() at /home/zcp/repo_center/doris_master/doris/be/src/vec/sink/writer/vfile_result_writer.h:56
26# doris::pipeline::AsyncWriterSink<doris::vectorized::VFileResultWriter, doris::pipeline::ResultFileSinkOperatorX>::~AsyncWriterSink() at /home/zcp/repo_center/doris_master/doris/be/src/pipeline/exec/operator.h:1059
27# doris::pipeline::ResultFileSinkLocalState::~ResultFileSinkLocalState() at /home/zcp/repo_center/doris_master/doris/be/src/pipeline/exec/result_file_sink_operator.cpp:57
28# doris::RuntimeState::~RuntimeState() at /home/zcp/repo_center/doris_master/doris/be/src/runtime/runtime_state.cpp:211
29# doris::pipeline::PipelineFragmentContext::~PipelineFragmentContext() at /home/zcp/repo_center/doris_master/doris/be/src/pipeline/pipeline_fragment_context.cpp:149
30# std::_Function_handler<void (), doris::vectorized::AsyncResultWriter::start_writer(doris::RuntimeState*, doris::RuntimeProfile*)::$_0>::_M_invoke(std::_Any_data const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291
```
Fix 2:
```
F20250411 00:00:31.856455 173889 thread_context.h:204] Check failed: is_attach_task()
9# google::LogMessageFatal::~LogMessageFatal() in /mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be
10# Allocator<false, false, false, DefaultMemoryAllocator>::sys_memory_check(unsigned long) const at /home/zcp/repo_center/doris_master/doris/be/src/vec/common/allocator.cpp:191
11# Allocator<false, false, false, DefaultMemoryAllocator>::alloc_impl(unsigned long, unsigned long) at /home/zcp/repo_center/doris_master/doris/be/src/vec/common/allocator.h:254
12# doris::io::Memory<Allocator<false, false, false, DefaultMemoryAllocator> >::Memory(unsigned long) at /home/zcp/repo_center/doris_master/doris/be/src/io/fs/s3_file_bufferpool.cpp:48
13# doris::io::FileBuffer::FileBuffer(doris::io::BufferType, std::function<std::unique_ptr<doris::io::FileBlocksHolder, std::default_delete<doris::io::FileBlocksHolder> > ()>, unsigned long, doris::io::OperationState) at /home/zcp/repo_center/doris_master/doris/be/src/io/fs/s3_file_bufferpool.cpp:86
14# doris::io::UploadFileBuffer::UploadFileBuffer(std::function<void (doris::io::UploadFileBuffer&)>, doris::io::OperationState, unsigned long, std::function<std::unique_ptr<doris::io::FileBlocksHolder, std::default_delete<doris::io::FileBlocksHolder> > ()>) at /home/zcp/repo_center/doris_master/doris/be/src/io/fs/s3_file_bufferpool.h:157
15# decltype (::new ((void*)(0)) doris::io::UploadFileBuffer(std::declval<std::function<void (doris::io::UploadFileBuffer&)> >(), std::declval<doris::io::OperationState>(), std::declval<unsigned long&>(), std::declval<std::function<std::unique_ptr<doris::io::FileBlocksHolder, std::default_delete<doris::io::FileBlocksHolder> > ()> >())) std::construct_at<doris::io::UploadFileBuffer, std::function<void (doris::io::UploadFileBuffer&)>, doris::io::OperationState, unsigned long&, std::function<std::unique_ptr<doris::io::FileBlocksHolder, std::default_delete<doris::io::FileBlocksHolder> > ()> >(doris::io::UploadFileBuffer*, std::function<void (doris::io::UploadFileBuffer&)>&&, doris::io::OperationState&&, unsigned long&, std::function<std::unique_ptr<doris::io::FileBlocksHolder, std::default_delete<doris::io::FileBlocksHolder> > ()>&&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:97
16# doris::io::FileBufferBuilder::build(std::shared_ptr<doris::io::FileBuffer>*) at /home/zcp/repo_center/doris_master/doris/be/src/io/fs/s3_file_bufferpool.cpp:247
17# doris::io::S3FileWriter::_build_upload_buffer() at /home/zcp/repo_center/doris_master/doris/be/src/io/fs/s3_file_writer.cpp:199
18# doris::io::S3FileWriter::appendv(doris::Slice const*, unsigned long) at /home/zcp/repo_center/doris_master/doris/be/src/io/fs/s3_file_writer.cpp:251
19# std::enable_if<is_invocable_r_v<doris::Status, doris::io::S3FileSystem::batch_upload_impl(std::vector<std::filesystem::__cxx11::path, std::allocator<std::filesystem::__cxx11::path> > const&, std::vector<std::filesystem::__cxx11::path, std::allocator<std::filesystem::__cxx11::path> > const&)::$_0&, unsigned long>, doris::Status>::type std::__invoke_r<doris::Status, doris::io::S3FileSystem::batch_upload_impl(std::vector<std::filesystem::__cxx11::path, std::allocator<std::filesystem::__cxx11::path> > const&, std::vector<std::filesystem::__cxx11::path, std::allocator<std::filesystem::__cxx11::path> > const&)::$_0&, unsigned long>(doris::io::S3FileSystem::batch_upload_impl(std::vector<std::filesystem::__cxx11::path, std::allocator<std::filesystem::__cxx11::path> > const&, std::vector<std::filesystem::__cxx11::path, std::allocator<std::filesystem::__cxx11::path> > const&)::$_0&, unsigned long&&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:114
20# std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<doris::Status>, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_state<doris::io::S3FileSystem::batch_upload_impl(std::vector<std::filesystem::__cxx11::path, std::allocator<std::filesystem::__cxx11::path> > const&, std::vector<std::filesystem::__cxx11::path, std::allocator<std::filesystem::__cxx11::path> > const&)::$_0, std::allocator<int>, doris::Status (unsigned long)>::_M_run(unsigned long&&)::{lambda()#1}, doris::Status> >::_M_invoke(std::_Any_data const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291
21# std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/future:575
22# __pthread_once_slow at ./nptl/pthread_once.c:118
23# std::__future_base::_State_baseV2::_M_set_result(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>, bool) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/future:411
24# std::__future_base::_Task_state<doris::io::S3FileSystem::batch_upload_impl(std::vector<std::filesystem::__cxx11::path, std::allocator<std::filesystem::__cxx11::path> > const&, std::vector<std::filesystem::__cxx11::path, std::allocator<std::filesystem::__cxx11::path> > const&)::$_0, std::allocator<int>, doris::Status (unsigned long)>::_M_run(unsigned long&&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/future:1447
25# std::_Function_handler<void (), doris::io::S3FileSystem::batch_upload_impl(std::vector<std::filesystem::__cxx11::path, std::allocator<std::filesystem::__cxx11::path> > const&, std::vector<std::filesystem::__cxx11::path, std::allocator<std::filesystem::__cxx11::path> > const&)::$_1>::_M_invoke(std::_Any_data const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291
```
wyxxxcat
pushed a commit
that referenced
this pull request
Apr 17, 2025
Problem Summary: fix DCHECK error after multiple add/drop rename column when building inverted index ``` F20250408 11:43:45.886974 1066024 index_builder.cpp:423] Check failed: output_rowset_schema->has_inverted_index_with_index_id(index_id) *** Check failure stack trace: *** F20250408 11:43:45.886988 1066022 index_builder.cpp:423] Check failed: output_rowset_schema->has_inverted_index_with_index_id(index_id) *** Check failure stack trace: *** @ 0x590b0f234d26 google::LogMessage::SendToLog() @ 0x590b0f234d26 google::LogMessage::SendToLog() @ 0x590b0f231770 google::LogMessage::Flush() @ 0x590b0f231770 google::LogMessage::Flush() @ 0x590b0f235569 google::LogMessageFatal::~LogMessageFatal() @ 0x590b0f235569 google::LogMessageFatal::~LogMessageFatal() @ 0x590ad85d3459 doris::IndexBuilder::handle_single_rowset() @ 0x590ad85daf25 doris::IndexBuilder::handle_inverted_index_data() @ 0x590ad85d3459 doris::IndexBuilder::handle_single_rowset() @ 0x590ad85ddc31 doris::IndexBuilder::do_build_inverted_index() @ 0x590ad85daf25 doris::IndexBuilder::handle_inverted_index_data() @ 0x590ad711ad86 doris::StorageEngine::_handle_index_change() @ 0x590ad85ddc31 doris::IndexBuilder::do_build_inverted_index() @ 0x590ad711a1da doris::StorageEngine::process_index_change_task() @ 0x590ad711ad86 doris::StorageEngine::_handle_index_change() @ 0x590ad857b89d doris::EngineIndexChangeTask::execute() @ 0x590ad711a1da doris::StorageEngine::process_index_change_task() @ 0x590ad49deea2 doris::alter_inverted_index_callback() @ 0x590ad4a1de9d _ZNSt17_Function_handlerIFvvEZZN5doris14TaskWorkerPool11submit_taskERKNS1_17TAgentTaskRequestEENK3$_0clIS5_EEDaOT_EUlvE_E9_M_invokeERKSt9_Any_data @ 0x590ad857b89d doris::EngineIndexChangeTask::execute() @ 0x590ad936128b doris::ThreadPool::dispatch_thread() @ 0x590ad49deea2 doris::alter_inverted_index_callback() @ 0x590ad4a1de9d _ZNSt17_Function_handlerIFvvEZZN5doris14TaskWorkerPool11submit_taskERKNS1_17TAgentTaskRequestEENK3$_0clIS5_EEDaOT_EUlvE_E9_M_invokeERKSt9_Any_data @ 0x590ad9337828 doris::Thread::supervise_thread() @ 0x740bd7c9caa4 (unknown) @ 0x740bd7d29c3c (unknown) @ (nil) (unknown) *** Query id: 0-0 *** *** is nereids: 0 *** *** tablet id: 0 *** *** Aborted at 1744083830 (unix time) try "date -d @1744083830" if you are using GNU date *** *** Current BE git commitID: 3640f6c *** *** SIGABRT unknown detail explain (@0x10417b) received by PID 1065339 (TID 1066024 OR 0x7408c42006c0) from PID 1065339; stack trace: *** @ 0x590ad936128b doris::ThreadPool::dispatch_thread() @ 0x590ad9337828 doris::Thread::supervise_thread() @ 0x740bd7c9caa4 (unknown) @ 0x740bd7d29c3c (unknown) @ (nil) (unknown) 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_master/doris/be/src/common/signal_handler.h:421 1# 0x0000740BD7C45330 in /lib/x86_64-linux-gnu/libc.so.6 2# pthread_kill in /lib/x86_64-linux-gnu/libc.so.6 3# gsignal in /lib/x86_64-linux-gnu/libc.so.6 4# abort in /lib/x86_64-linux-gnu/libc.so.6 5# 0x0000590B0F23F5FD in /root/wyx/doris/be/lib/doris_be 6# 0x0000590B0F231C3A in /root/wyx/doris/be/lib/doris_be 7# google::LogMessage::SendToLog() in /root/wyx/doris/be/lib/doris_be 8# google::LogMessage::Flush() in /root/wyx/doris/be/lib/doris_be 9# google::LogMessageFatal::~LogMessageFatal() in /root/wyx/doris/be/lib/doris_be 10# doris::IndexBuilder::handle_single_rowset(std::shared_ptr<doris::RowsetMeta>, std::vector<std::shared_ptr<doris::segment_v2::Segment>, std::allocator<std::shared_ptr<doris::segment_v2::Segment> > >&) at /home/zcp/repo_center/doris_master/doris/be/src/olap/task/index_builder.cpp:423 11# doris::IndexBuilder::handle_inverted_index_data() at /home/zcp/repo_center/doris_master/doris/be/src/olap/task/index_builder.cpp:711 12# doris::IndexBuilder::do_build_inverted_index() in /root/wyx/doris/be/lib/doris_be 13# doris::StorageEngine::_handle_index_change(std::shared_ptr<doris::IndexBuilder>) at /home/zcp/repo_center/doris_master/doris/be/src/olap/olap_server.cpp:1213 14# doris::StorageEngine::process_index_change_task(doris::TAlterInvertedIndexReq const&) at /home/zcp/repo_center/doris_master/doris/be/src/olap/olap_server.cpp:1207 15# doris::EngineIndexChangeTask::execute() in /root/wyx/doris/be/lib/doris_be 16# doris::alter_inverted_index_callback(doris::StorageEngine&, doris::TAgentTaskRequest const&) in /root/wyx/doris/be/lib/doris_be 17# std::_Function_handler<void (), doris::TaskWorkerPool::submit_task(doris::TAgentTaskRequest const&)::$_0::operator()<doris::TAgentTaskRequest const&>(doris::TAgentTaskRequest const&) const::{lambda()#1}>::_M_invoke(std::_Any_data const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291 18# doris::ThreadPool::dispatch_thread() in /root/wyx/doris/be/lib/doris_be 19# doris::Thread::supervise_thread(void*) at /home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:497 20# 0x0000740BD7C9CAA4 in /lib/x86_64-linux-gnu/libc.so.6 21# 0x0000740BD7D29C3C in /lib/x86_64-linux-gnu/libc.so.6 ```
wyxxxcat
pushed a commit
that referenced
this pull request
Apr 23, 2025
TaskQueue should be released after all theads stop.
```
=================================================================
==6632==ERROR: AddressSanitizer: heap-use-after-free on address 0x61700518d440 at pc 0x5616fa5c2a87 bp 0x7f7635b709b0 sp 0x7f7635b709a8
READ of size 8 at 0x61700518d440 thread T5188 (Pipe_test_follo)
#0 0x5616fa5c2a86 in std::__atomic_base<unsigned long>::load(std::memory_order) const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/atomic_base.h:481:9
#1 0x5616fa5c2a86 in std::__atomic_base<unsigned long>::operator unsigned long() const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/atomic_base.h:341:16
#2 0x5616fa5c2a86 in doris::pipeline::PriorityTaskQueue::_try_take_unprotected(bool) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/pipeline/task_queue.cpp:58:9
#3 0x5616fa5c2e16 in doris::pipeline::PriorityTaskQueue::take(unsigned int) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/pipeline/task_queue.cpp:110:16
#4 0x5616fa5c4b06 in doris::pipeline::MultiCoreTaskQueue::take(int) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/pipeline/task_queue.cpp:170:51
apache#5 0x5616fa5d1919 in doris::pipeline::TaskScheduler::_do_work(unsigned long) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/pipeline/task_scheduler.cpp:267:35
apache#6 0x5616d1e169f8 in doris::ThreadPool::dispatch_thread() /home/zcp/repo_center/doris_branch-2.1/doris/be/src/util/threadpool.cpp:544:24
apache#7 0x5616d1df3e38 in std::function<void ()>::operator()() const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9
apache#8 0x5616d1df3e38 in doris::Thread::supervise_thread(void*) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/util/thread.cpp:498:5
apache#9 0x7f83a4c35ac2 in start_thread nptl/pthread_create.c:442:8
apache#10 0x7f83a4cc784f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
0x61700518d440 is located 704 bytes inside of 728-byte region [0x61700518d180,0x61700518d458)
freed by thread T2166 here:
#0 0x5616cea38d9d in operator delete(void*) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x16dc0d9d) (BuildId: 09b7b6856fee4f7f)
#1 0x5616fa5cae77 in std::default_delete<doris::pipeline::PriorityTaskQueue>::operator()(doris::pipeline::PriorityTaskQueue*) const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:85:2
#2 0x5616fa5cae77 in std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>::~unique_ptr() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:361:4
#3 0x5616fa5cae77 in void std::destroy_at<std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>>(std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:88:15
#4 0x5616fa5cae77 in void std::_Destroy<std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>>(std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:138:7
apache#5 0x5616fa5cae77 in void std::_Destroy_aux<false>::__destroy<std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>*>(std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>*, std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:152:6
apache#6 0x5616fa5cae77 in void std::_Destroy<std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>*>(std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>*, std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:184:7
apache#7 0x5616fa5cae77 in void std::_Destroy<std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>*, std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>>(std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>*, std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>*, std::allocator<std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>>&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:746:7
apache#8 0x5616fa5cae77 in std::vector<std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>, std::allocator<std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>>>::~vector() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:680:2
apache#9 0x5616fa5c466d in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:168:6
apache#10 0x5616fa5c466d in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:702:11
apache#11 0x5616fa5c466d in std::__shared_ptr<std::vector<std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>, std::allocator<std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>>>, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1149:31
apache#12 0x5616fa5c466d in void std::atomic_store<std::vector<std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>, std::allocator<std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>>>>(std::shared_ptr<std::vector<std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>, std::allocator<std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>>>>*, std::shared_ptr<std::vector<std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>, std::allocator<std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>>>>) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_atomic.h:149:7
apache#13 0x5616fa5c466d in doris::pipeline::MultiCoreTaskQueue::close() /home/zcp/repo_center/doris_branch-2.1/doris/be/src/pipeline/task_queue.cpp:151:5
apache#14 0x5616fa5ce9f4 in doris::pipeline::TaskScheduler::stop() /home/zcp/repo_center/doris_branch-2.1/doris/be/src/pipeline/task_scheduler.cpp:437:26
apache#15 0x5616d19b7d36 in doris::WorkloadGroup::try_stop_schedulers() /home/zcp/repo_center/doris_branch-2.1/doris/be/src/runtime/workload_group/workload_group.cpp:654:22
apache#16 0x5616d19c6042 in doris::WorkloadGroupMgr::delete_workload_group_by_ids(std::set<unsigned long, std::less<unsigned long>, std::allocator<unsigned long>>) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/runtime/workload_group/workload_group_manager.cpp:107:13
apache#17 0x5616d1a7f6a1 in doris::WorkloadGroupListener::handle_topic_info(std::vector<doris::TopicInfo, std::allocator<doris::TopicInfo>> const&) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/agent/workload_group_listener.cpp:82:38
apache#18 0x5616d1a7dd7e in doris::TopicSubscriber::handle_topic_info(doris::TPublishTopicRequest const&) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/agent/topic_subscriber.cpp:45:35
apache#19 0x5616d2002932 in doris::BackendServiceProcessor::process_publish_topic_info(int, apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, void*) /home/zcp/repo_center/doris_branch-2.1/doris/gensrc/build/gen_cpp/BackendService.cpp:6577:13
apache#20 0x5616d1fd6f11 in doris::BackendServiceProcessor::dispatchCall(apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, int, void*) /home/zcp/repo_center/doris_branch-2.1/doris/gensrc/build/gen_cpp/BackendService.cpp:5416:3
apache#21 0x5616d2039186 in apache::thrift::TDispatchProcessor::process(std::shared_ptr<apache::thrift::protocol::TProtocol>, std::shared_ptr<apache::thrift::protocol::TProtocol>, void*) /home/zcp/repo_center/doris_branch-2.1/doris/thirdparty/installed/include/thrift/TDispatchProcessor.h:121:12
apache#22 0x5616faee19f7 in apache::thrift::server::TConnectedClient::run() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x432699f7) (BuildId: 09b7b6856fee4f7f)
apache#23 0x5616faee2cf6 in apache::thrift::server::TThreadedServer::TConnectedClientRunner::run() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4326acf6) (BuildId: 09b7b6856fee4f7f)
apache#24 0x5616faee830d in apache::thrift::concurrency::Thread::threadMain(std::shared_ptr<apache::thrift::concurrency::Thread>) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4327030d) (BuildId: 09b7b6856fee4f7f)
apache#25 0x5616faee81b3 in void std::__invoke_impl<void, void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>(std::__invoke_other, void (*&&)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>&&) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x432701b3) (BuildId: 09b7b6856fee4f7f)
apache#26 0x5616faee8106 in std::__invoke_result<void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>::type std::__invoke<void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>(void (*&&)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>&&) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x43270106) (BuildId: 09b7b6856fee4f7f)
apache#27 0x5616faee8076 in void std::thread::_Invoker<std::tuple<void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>>::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x43270076) (BuildId: 09b7b6856fee4f7f)
apache#28 0x5616faee7ffb in std::thread::_Invoker<std::tuple<void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>>::operator()() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4326fffb) (BuildId: 09b7b6856fee4f7f)
apache#29 0x5616faee7f9f in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>>>::_M_run() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4326ff9f) (BuildId: 09b7b6856fee4f7f)
apache#30 0x5616fd80568f in execute_native_thread_routine /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/src/c++11/../../../../../libstdc++-v3/src/c++11/thread.cc:82:18
previously allocated by thread T3392 here:
#0 0x5616cea3853d in operator new(unsigned long) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x16dc053d) (BuildId: 09b7b6856fee4f7f)
#1 0x5616fa5c3e29 in std::_MakeUniq<doris::pipeline::PriorityTaskQueue>::__single_object std::make_unique<doris::pipeline::PriorityTaskQueue>() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:962:30
#2 0x5616fa5c3e29 in doris::pipeline::MultiCoreTaskQueue::MultiCoreTaskQueue(int) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/pipeline/task_queue.cpp:139:39
#3 0x5616d19b276d in decltype(::new((void*)(0)) doris::pipeline::MultiCoreTaskQueue(std::declval<int&>())) std::construct_at<doris::pipeline::MultiCoreTaskQueue, int&>(doris::pipeline::MultiCoreTaskQueue*, int&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:97:39
#4 0x5616d19b276d in void std::allocator_traits<std::allocator<doris::pipeline::MultiCoreTaskQueue>>::construct<doris::pipeline::MultiCoreTaskQueue, int&>(std::allocator<doris::pipeline::MultiCoreTaskQueue>&, doris::pipeline::MultiCoreTaskQueue*, int&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:514:4
apache#5 0x5616d19b276d in std::_Sp_counted_ptr_inplace<doris::pipeline::MultiCoreTaskQueue, std::allocator<doris::pipeline::MultiCoreTaskQueue>, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<int&>(std::allocator<doris::pipeline::MultiCoreTaskQueue>, int&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:519:4
apache#6 0x5616d19b276d in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<doris::pipeline::MultiCoreTaskQueue, std::allocator<doris::pipeline::MultiCoreTaskQueue>, int&>(doris::pipeline::MultiCoreTaskQueue*&, std::_Sp_alloc_shared_tag<std::allocator<doris::pipeline::MultiCoreTaskQueue>>, int&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:651:6
apache#7 0x5616d19b276d in std::__shared_ptr<doris::pipeline::MultiCoreTaskQueue, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<doris::pipeline::MultiCoreTaskQueue>, int&>(std::_Sp_alloc_shared_tag<std::allocator<doris::pipeline::MultiCoreTaskQueue>>, int&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1337:14
apache#8 0x5616d19b276d in std::shared_ptr<doris::pipeline::MultiCoreTaskQueue>::shared_ptr<std::allocator<doris::pipeline::MultiCoreTaskQueue>, int&>(std::_Sp_alloc_shared_tag<std::allocator<doris::pipeline::MultiCoreTaskQueue>>, int&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr.h:409:4
apache#9 0x5616d19b276d in std::shared_ptr<doris::pipeline::MultiCoreTaskQueue> std::allocate_shared<doris::pipeline::MultiCoreTaskQueue, std::allocator<doris::pipeline::MultiCoreTaskQueue>, int&>(std::allocator<doris::pipeline::MultiCoreTaskQueue> const&, int&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr.h:860:14
apache#10 0x5616d19b276d in std::shared_ptr<doris::pipeline::MultiCoreTaskQueue> std::make_shared<doris::pipeline::MultiCoreTaskQueue, int&>(int&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr.h:876:14
apache#11 0x5616d19b276d in doris::WorkloadGroup::upsert_thread_pool_no_lock(doris::WorkloadGroupInfo*, std::shared_ptr<doris::CgroupCpuCtl>, doris::ExecEnv*) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/runtime/workload_group/workload_group.cpp:462:27
apache#12 0x5616d19b5991 in doris::WorkloadGroup::upsert_task_scheduler(doris::WorkloadGroupInfo*, doris::ExecEnv*) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/runtime/workload_group/workload_group.cpp:572:9
apache#13 0x5616d1a7ecd8 in doris::WorkloadGroupListener::handle_topic_info(std::vector<doris::TopicInfo, std::allocator<doris::TopicInfo>> const&) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/agent/workload_group_listener.cpp:60:13
apache#14 0x5616d1a7dd7e in doris::TopicSubscriber::handle_topic_info(doris::TPublishTopicRequest const&) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/agent/topic_subscriber.cpp:45:35
apache#15 0x5616d2002932 in doris::BackendServiceProcessor::process_publish_topic_info(int, apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, void*) /home/zcp/repo_center/doris_branch-2.1/doris/gensrc/build/gen_cpp/BackendService.cpp:6577:13
apache#16 0x5616d1fd6f11 in doris::BackendServiceProcessor::dispatchCall(apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, int, void*) /home/zcp/repo_center/doris_branch-2.1/doris/gensrc/build/gen_cpp/BackendService.cpp:5416:3
apache#17 0x5616d2039186 in apache::thrift::TDispatchProcessor::process(std::shared_ptr<apache::thrift::protocol::TProtocol>, std::shared_ptr<apache::thrift::protocol::TProtocol>, void*) /home/zcp/repo_center/doris_branch-2.1/doris/thirdparty/installed/include/thrift/TDispatchProcessor.h:121:12
apache#18 0x5616faee19f7 in apache::thrift::server::TConnectedClient::run() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x432699f7) (BuildId: 09b7b6856fee4f7f)
apache#19 0x5616faee2cf6 in apache::thrift::server::TThreadedServer::TConnectedClientRunner::run() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4326acf6) (BuildId: 09b7b6856fee4f7f)
apache#20 0x5616faee830d in apache::thrift::concurrency::Thread::threadMain(std::shared_ptr<apache::thrift::concurrency::Thread>) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4327030d) (BuildId: 09b7b6856fee4f7f)
apache#21 0x5616faee81b3 in void std::__invoke_impl<void, void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>(std::__invoke_other, void (*&&)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>&&) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x432701b3) (BuildId: 09b7b6856fee4f7f)
apache#22 0x5616faee8106 in std::__invoke_result<void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>::type std::__invoke<void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>(void (*&&)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>&&) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x43270106) (BuildId: 09b7b6856fee4f7f)
apache#23 0x5616faee8076 in void std::thread::_Invoker<std::tuple<void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>>::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x43270076) (BuildId: 09b7b6856fee4f7f)
apache#24 0x5616faee7ffb in std::thread::_Invoker<std::tuple<void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>>::operator()() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4326fffb) (BuildId: 09b7b6856fee4f7f)
apache#25 0x5616faee7f9f in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>>>::_M_run() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4326ff9f) (BuildId: 09b7b6856fee4f7f)
apache#26 0x5616fd80568f in execute_native_thread_routine /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/src/c++11/../../../../../libstdc++-v3/src/c++11/thread.cc:82:18
Thread T5188 (Pipe_test_follo) created by T3392 here:
#0 0x5616ce9e5caa in pthread_create (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x16d6dcaa) (BuildId: 09b7b6856fee4f7f)
#1 0x5616d1df2cc9 in doris::Thread::start_thread(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::function<void ()> const&, unsigned long, scoped_refptr<doris::Thread>*) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/util/thread.cpp:449:15
#2 0x5616d1e1d979 in doris::Status doris::Thread::create<void (doris::ThreadPool::*)(), doris::ThreadPool*>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, void (doris::ThreadPool::* const&)(), doris::ThreadPool* const&, scoped_refptr<doris::Thread>*) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/util/thread.h:56:16
#3 0x5616d1e1423d in doris::ThreadPool::create_thread() /home/zcp/repo_center/doris_branch-2.1/doris/be/src/util/threadpool.cpp:612:12
#4 0x5616d1e13e43 in doris::ThreadPool::init() /home/zcp/repo_center/doris_branch-2.1/doris/be/src/util/threadpool.cpp:266:25
apache#5 0x5616cec31535 in doris::Status doris::ThreadPoolBuilder::build<doris::ThreadPool>(std::unique_ptr<doris::ThreadPool, std::default_delete<doris::ThreadPool>>*) const /home/zcp/repo_center/doris_branch-2.1/doris/be/src/util/threadpool.h:121:13
apache#6 0x5616fa5cef22 in doris::pipeline::TaskScheduler::start() /home/zcp/repo_center/doris_branch-2.1/doris/be/src/pipeline/task_scheduler.cpp:208:5
apache#7 0x5616d19b2a9b in doris::WorkloadGroup::upsert_thread_pool_no_lock(doris::WorkloadGroupInfo*, std::shared_ptr<doris::CgroupCpuCtl>, doris::ExecEnv*) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/runtime/workload_group/workload_group.cpp:467:47
apache#8 0x5616d19b5991 in doris::WorkloadGroup::upsert_task_scheduler(doris::WorkloadGroupInfo*, doris::ExecEnv*) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/runtime/workload_group/workload_group.cpp:572:9
apache#9 0x5616d1a7ecd8 in doris::WorkloadGroupListener::handle_topic_info(std::vector<doris::TopicInfo, std::allocator<doris::TopicInfo>> const&) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/agent/workload_group_listener.cpp:60:13
apache#10 0x5616d1a7dd7e in doris::TopicSubscriber::handle_topic_info(doris::TPublishTopicRequest const&) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/agent/topic_subscriber.cpp:45:35
apache#11 0x5616d2002932 in doris::BackendServiceProcessor::process_publish_topic_info(int, apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, void*) /home/zcp/repo_center/doris_branch-2.1/doris/gensrc/build/gen_cpp/BackendService.cpp:6577:13
apache#12 0x5616d1fd6f11 in doris::BackendServiceProcessor::dispatchCall(apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, int, void*) /home/zcp/repo_center/doris_branch-2.1/doris/gensrc/build/gen_cpp/BackendService.cpp:5416:3
apache#13 0x5616d2039186 in apache::thrift::TDispatchProcessor::process(std::shared_ptr<apache::thrift::protocol::TProtocol>, std::shared_ptr<apache::thrift::protocol::TProtocol>, void*) /home/zcp/repo_center/doris_branch-2.1/doris/thirdparty/installed/include/thrift/TDispatchProcessor.h:121:12
apache#14 0x5616faee19f7 in apache::thrift::server::TConnectedClient::run() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x432699f7) (BuildId: 09b7b6856fee4f7f)
apache#15 0x5616faee2cf6 in apache::thrift::server::TThreadedServer::TConnectedClientRunner::run() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4326acf6) (BuildId: 09b7b6856fee4f7f)
apache#16 0x5616faee830d in apache::thrift::concurrency::Thread::threadMain(std::shared_ptr<apache::thrift::concurrency::Thread>) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4327030d) (BuildId: 09b7b6856fee4f7f)
apache#17 0x5616faee81b3 in void std::__invoke_impl<void, void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>(std::__invoke_other, void (*&&)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>&&) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x432701b3) (BuildId: 09b7b6856fee4f7f)
apache#18 0x5616faee8106 in std::__invoke_result<void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>::type std::__invoke<void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>(void (*&&)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>&&) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x43270106) (BuildId: 09b7b6856fee4f7f)
apache#19 0x5616faee8076 in void std::thread::_Invoker<std::tuple<void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>>::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x43270076) (BuildId: 09b7b6856fee4f7f)
apache#20 0x5616faee7ffb in std::thread::_Invoker<std::tuple<void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>>::operator()() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4326fffb) (BuildId: 09b7b6856fee4f7f)
apache#21 0x5616faee7f9f in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>>>::_M_run() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4326ff9f) (BuildId: 09b7b6856fee4f7f)
apache#22 0x5616fd80568f in execute_native_thread_routine /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/src/c++11/../../../../../libstdc++-v3/src/c++11/thread.cc:82:18
Thread T3392 created by T933 here:
#0 0x5616ce9e5caa in pthread_create (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x16d6dcaa) (BuildId: 09b7b6856fee4f7f)
#1 0x5616fd8057b5 in __gthread_create /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:663:35
#2 0x5616fd8057b5 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State>>, void (*)()) /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/src/c++11/../../../../../libstdc++-v3/src/c++11/thread.cc:147:37
#3 0x5616faee667c in apache::thrift::concurrency::Thread::start() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4326e67c) (BuildId: 09b7b6856fee4f7f)
#4 0x5616faee2a26 in apache::thrift::server::TThreadedServer::onClientConnected(std::shared_ptr<apache::thrift::server::TConnectedClient> const&) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4326aa26) (BuildId: 09b7b6856fee4f7f)
apache#5 0x5616faedebc4 in apache::thrift::server::TServerFramework::newlyConnectedClient(std::shared_ptr<apache::thrift::server::TConnectedClient> const&) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x43266bc4) (BuildId: 09b7b6856fee4f7f)
apache#6 0x5616faede25f in apache::thrift::server::TServerFramework::serve() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4326625f) (BuildId: 09b7b6856fee4f7f)
apache#7 0x5616faee2791 in apache::thrift::server::TThreadedServer::serve() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4326a791) (BuildId: 09b7b6856fee4f7f)
apache#8 0x5616d1e31419 in doris::ThriftServer::ThriftServerEventProcessor::supervise() /home/zcp/repo_center/doris_branch-2.1/doris/be/src/util/thrift_server.cpp:206:34
apache#9 0x5616fd80568f in execute_native_thread_routine /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/src/c++11/../../../../../libstdc++-v3/src/c++11/thread.cc:82:18
Thread T933 created by T0 here:
#0 0x5616ce9e5caa in pthread_create (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x16d6dcaa) (BuildId: 09b7b6856fee4f7f)
#1 0x5616fd8057b5 in __gthread_create /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:663:35
#2 0x5616fd8057b5 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State>>, void (*)()) /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/src/c++11/../../../../../libstdc++-v3/src/c++11/thread.cc:147:37
#3 0x5616d1e37788 in doris::ThriftServer::start() /home/zcp/repo_center/doris_branch-2.1/doris/be/src/util/thrift_server.cpp:421:5
#4 0x5616cea3f174 in main /home/zcp/repo_center/doris_branch-2.1/doris/be/src/service/doris_main.cpp:536:25
apache#5 0x7f83a4bcad8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
Thread T2166 created by T933 here:
#0 0x5616ce9e5caa in pthread_create (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x16d6dcaa) (BuildId: 09b7b6856fee4f7f)
#1 0x5616fd8057b5 in __gthread_create /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:663:35
#2 0x5616fd8057b5 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State>>, void (*)()) /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/src/c++11/../../../../../libstdc++-v3/src/c++11/thread.cc:147:37
#3 0x5616faee667c in apache::thrift::concurrency::Thread::start() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4326e67c) (BuildId: 09b7b6856fee4f7f)
#4 0x5616faee2a26 in apache::thrift::server::TThreadedServer::onClientConnected(std::shared_ptr<apache::thrift::server::TConnectedClient> const&) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4326aa26) (BuildId: 09b7b6856fee4f7f)
apache#5 0x5616faedebc4 in apache::thrift::server::TServerFramework::newlyConnectedClient(std::shared_ptr<apache::thrift::server::TConnectedClient> const&) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x43266bc4) (BuildId: 09b7b6856fee4f7f)
apache#6 0x5616faede25f in apache::thrift::server::TServerFramework::serve() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4326625f) (BuildId: 09b7b6856fee4f7f)
apache#7 0x5616faee2791 in apache::thrift::server::TThreadedServer::serve() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4326a791) (BuildId: 09b7b6856fee4f7f)
apache#8 0x5616d1e31419 in doris::ThriftServer::ThriftServerEventProcessor::supervise() /home/zcp/repo_center/doris_branch-2.1/doris/be/src/util/thrift_server.cpp:206:34
apache#9 0x5616fd80568f in execute_native_thread_routine /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/src/c++11/../../../../../libstdc++-v3/src/c++11/thread.cc:82:18
SUMMARY: AddressSanitizer: heap-use-after-free /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/atomic_base.h:481:9 in std::__atomic_base<unsigned long>::load(std::memory_order) const
Shadow bytes around the buggy address:
0x61700518d180: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x61700518d200: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x61700518d280: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x61700518d300: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x61700518d380: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
=>0x61700518d400: fd fd fd fd fd fd fd fd[fd]fd fd fa fa fa fa fa
0x61700518d480: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x61700518d500: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x61700518d580: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x61700518d600: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x61700518d680: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==6632==ABORTING
```
wyxxxcat
pushed a commit
that referenced
this pull request
May 16, 2025
) ### What problem does this PR solve? If there is a mark join condition, then even in a right semi join, the columns from the left table involved in the mark join condition will still appear in the intermediate tuple. This PR also completes the missing handling logic for right semi joins with mark join condition. ``` ==2126730==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x503004ee02e0 at pc 0x55f4a7d3f354 bp 0x7f6b6b587e70 sp 0x7f6b6b587e68 READ of size 8 at 0x503004ee02e0 thread T1296 (brpc_light) #0 0x55f4a7d3f353 in std::__shared_ptr<doris::vectorized::IDataType const, (__gnu_cxx::_Lock_policy)2>::__shared_ptr(std::__shared_ptr<doris::vectorized::IDataType const, (__gnu_cxx::_Lock_policy)2> const&) /root/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/shared_ptr_base.h:1522:7 #1 0x55f4a7d3f01e in std::shared_ptr<doris::vectorized::IDataType const>::shared_ptr(std::shared_ptr<doris::vectorized::IDataType const> const&) /root/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/shared_ptr.h:204:7 #2 0x55f4f8aa231c in doris::pipeline::HashJoinProbeOperatorX::prepare(doris::RuntimeState*) /root/doris/be/src/pipeline/exec/hashjoin_probe_operator.cpp:577:33 #3 0x55f4fb1e4a28 in doris::pipeline::Pipeline::prepare(doris::RuntimeState*) /root/doris/be/src/pipeline/pipeline.cpp:89:5 #4 0x55f4fb089d3f in doris::pipeline::PipelineFragmentContext::prepare(doris::TPipelineFragmentParams const&, doris::ThreadPool*) /root/doris/be/src/pipeline/pipeline_fragment_context.cpp:352:9 apache#5 0x55f4ad615a5c in doris::FragmentMgr::exec_plan_fragment(doris::TPipelineFragmentParams const&, doris::QuerySource, std::function<void (doris::RuntimeState*, doris::Status*)> const&, doris::TPipelineFragmentParamsList const&) /root/doris/be/src/runtime/fragment_mgr.cpp:855:9 apache#6 0x55f4ad613ca6 in doris::FragmentMgr::exec_plan_fragment(doris::TPipelineFragmentParams const&, doris::QuerySource, doris::TPipelineFragmentParamsList const&) /root/doris/be/src/runtime/fragment_mgr.cpp:634:16 apache#7 0x55f4ae2e867c in doris::PInternalService::_exec_plan_fragment_impl(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, doris::PFragmentRequestVersion, bool, std::function<void (doris::RuntimeState*, doris::Status*)> const&) /root/doris/be/src/service/internal_service.cpp:613:17 apache#8 0x55f4ae2e4b67 in doris::PInternalService::_exec_plan_fragment_in_pthread(google::protobuf::RpcController*, doris::PExecPlanFragmentRequest const*, doris::PExecPlanFragmentResult*, google::protobuf::Closure*) /root/doris/be/src/service/internal_service.cpp:343:14 apache#9 0x55f4ae31d4e1 in doris::PInternalService::exec_plan_fragment_prepare(google::protobuf::RpcController*, doris::PExecPlanFragmentRequest const*, doris::PExecPlanFragmentResult*, google::protobuf::Closure*)::$_0::operator()() const /root/doris/be/src/service/internal_service.cpp:367:9 apache#10 0x55f4ae31d33e in void std::__invoke_impl<void, doris::PInternalService::exec_plan_fragment_prepare(google::protobuf::RpcController*, doris::PExecPlanFragmentRequest const*, doris::PExecPlanFragmentResult*, google::protobuf::Closure*)::$_0&>(std::__invoke_other, doris::PInternalService::exec_plan_fragment_prepare(google::protobuf::RpcController*, doris::PExecPlanFragmentRequest const*, doris::PExecPlanFragmentResult*, google::protobuf::Closure*)::$_0&) /root/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14 apache#11 0x55f4ae31d27e in std::enable_if<is_invocable_r_v<void, doris::PInternalService::exec_plan_fragment_prepare(google::protobuf::RpcController*, doris::PExecPlanFragmentRequest const*, doris::PExecPlanFragmentResult*, google::protobuf::Closure*)::$_0&>, void>::type std::__invoke_r<void, doris::PInternalService::exec_plan_fragment_prepare(google::protobuf::RpcController*, doris::PExecPlanFragmentRequest const*, doris::PExecPlanFragmentResult*, google::protobuf::Closure*)::$_0&>(doris::PInternalService::exec_plan_fragment_prepare(google::protobuf::RpcController*, doris::PExecPlanFragmentRequest const*, doris::PExecPlanFragmentResult*, google::protobuf::Closure*)::$_0&) /root/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:111:2 apache#12 0x55f4ae31cf55 in std::_Function_handler<void (), doris::PInternalService::exec_plan_fragment_prepare(google::protobuf::RpcController*, doris::PExecPlanFragmentRequest const*, doris::PExecPlanFragmentResult*, google::protobuf::Closure*)::$_0>::_M_invoke(std::_Any_data const&) /root/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290:9 apache#13 0x55f4a74175af in std::function<void ()>::operator()() const /root/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591:9 apache#14 0x55f4ae3c21c4 in doris::WorkThreadPool<false>::work_thread(int) /root/doris/be/src/util/work_thread_pool.hpp:158:17 apache#15 0x55f4ae3c4cc8 in void std::__invoke_impl<void, void (doris::WorkThreadPool<false>::* const&)(int), doris::WorkThreadPool<false>*&, int&>(std::__invoke_memfun_deref, void (doris::WorkThreadPool<false>::* const&)(int), doris::WorkThreadPool<false>*&, int&) /root/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:74:14 apache#16 0x55f4ae3c4a92 in std::__invoke_result<void (doris::WorkThreadPool<false>::* const&)(int), doris::WorkThreadPool<false>*&, int&>::type std::__invoke<void (doris::WorkThreadPool<false>::* const&)(int), doris::WorkThreadPool<false>*&, int&>(void (doris::WorkThreadPool<false>::* const&)(int), doris::WorkThreadPool<false>*&, int&) /root/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:96:14 apache#17 0x55f4ae3c49f8 in decltype(std::__invoke((*this)._M_pmf, std::forward<doris::WorkThreadPool<false>*&>(fp), std::forward<int&>(fp))) std::_Mem_fn_base<void (doris::WorkThreadPool<false>::*)(int), true>::operator()<doris::WorkThreadPool<false>*&, int&>(doris::WorkThreadPool<false>*&, int&) const /root/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/functional:170:11 apache#18 0x55f4ae3c4942 in void std::__invoke_impl<void, std::_Mem_fn<void (doris::WorkThreadPool<false>::*)(int)>&, doris::WorkThreadPool<false>*&, int&>(std::__invoke_other, std::_Mem_fn<void (doris::WorkThreadPool<false>::*)(int)>&, doris::WorkThreadPool<false>*&, int&) /root/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14 apache#19 0x55f4ae3c4752 in std::enable_if<is_invocable_r_v<void, std::_Mem_fn<void (doris::WorkThreadPool<false>::*)(int)>&, doris::WorkThreadPool<false>*&, int&>, void>::type std::__invoke_r<void, std::_Mem_fn<void (doris::WorkThreadPool<false>::*)(int)>&, doris::WorkThreadPool<false>*&, int&>(std::_Mem_fn<void (doris::WorkThreadPool<false>::*)(int)>&, doris::WorkThreadPool<false>*&, int&) /root/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:111:2 apache#20 0x55f4ae3c4664 in void std::_Bind_result<void, std::_Mem_fn<void (doris::WorkThreadPool<false>::*)(int)> (doris::WorkThreadPool<false>*, int)>::__call<void, 0ul, 1ul>(std::tuple<>&&, std::_Index_tuple<0ul, 1ul>) /root/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/functional:654:11 apache#21 0x55f4ae3c4395 in void std::_Bind_result<void, std::_Mem_fn<void (doris::WorkThreadPool<false>::*)(int)> (doris::WorkThreadPool<false>*, int)>::operator()<>() /root/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/functional:713:17 apache#22 0x55f4ae3c428e in void std::__invoke_impl<void, std::_Bind_result<void, std::_Mem_fn<void (doris::WorkThreadPool<false>::*)(int)> (doris::WorkThreadPool<false>*, int)>>(std::__invoke_other, std::_Bind_result<void, std::_Mem_fn<void (doris::WorkThreadPool<false>::*)(int)> (doris::WorkThreadPool<false>*, int)>&&) /root/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14 apache#23 0x55f4ae3c41ce in std::__invoke_result<std::_Bind_result<void, std::_Mem_fn<void (doris::WorkThreadPool<false>::*)(int)> (doris::WorkThreadPool<false>*, int)>>::type std::__invoke<std::_Bind_result<void, std::_Mem_fn<void (doris::WorkThreadPool<false>::*)(int)> (doris::WorkThreadPool<false>*, int)>>(std::_Bind_result<void, std::_Mem_fn<void (doris::WorkThreadPool<false>::*)(int)> (doris::WorkThreadPool<false>*, int)>&&) /root/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:96:14 apache#24 0x55f4ae3c417b in void std::thread::_Invoker<std::tuple<std::_Bind_result<void, std::_Mem_fn<void (doris::WorkThreadPool<false>::*)(int)> (doris::WorkThreadPool<false>*, int)>>>::_M_invoke<0ul>(std::_Index_tuple<0ul>) /root/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_thread.h:292:13 apache#25 0x55f4ae3c40f6 in std::thread::_Invoker<std::tuple<std::_Bind_result<void, std::_Mem_fn<void (doris::WorkThreadPool<false>::*)(int)> (doris::WorkThreadPool<false>*, int)>>>::operator()() /root/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_thread.h:299:11 apache#26 0x55f4ae3c3f34 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::_Bind_result<void, std::_Mem_fn<void (doris::WorkThreadPool<false>::*)(int)> (doris::WorkThreadPool<false>*, int)>>>>::_M_run() /root/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_thread.h:244:13 apache#27 0x55f4fea05d2e in execute_native_thread_routine pthread_atfork.c apache#28 0x55f4a7162e0a in asan_thread_start(void*) crtstuff.c apache#29 0x7f7149c421c9 in start_thread (/lib64/libpthread.so.0+0x81c9) (BuildId: 7c4add5c7a885e6ff4ce17867d6a2286e4420eec) apache#30 0x7f714a6318d2 in clone (/lib64/libc.so.6+0x398d2) (BuildId: 4ee3325955e3b55b6805f33959b7cb77745ad625) ```
wyxxxcat
pushed a commit
that referenced
this pull request
May 20, 2025
…string to a struct. (apache#50851) ``` select cast('{"a":1,"b":"1","c":"1","d":"1"}' as struct<a:int, b:int>); ==3393513==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x506001118e20 at pc 0x55808f91364a bp 0x7f9ec8816af0 sp 0x7f9ec8816ae8 READ of size 8 at 0x506001118e20 thread T1298 (brpc_light) #0 0x55808f913649 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_data() const /mnt/disk6/common/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/basic_string.h:223:28 #1 0x55808f9386f6 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::data() const /mnt/disk6/common/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/basic_string.h:2597:16 #2 0x55808fc4e57a in doris::Slice::Slice(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /mnt/disk12/yanxuecheng/doris/be/src/util/slice.h:72:40 #3 0x5580bebb7ed9 in doris::vectorized::DataTypeStructSerDe::deserialize_one_cell_from_json(doris::vectorized::IColumn&, doris::Slice&, doris::vectorized::DataTypeSerDe::FormatOptions const&) const /mnt/disk12/yanxuecheng/doris/be/src/vec/data_types/serde/data_type_struct_serde.cpp:141:17 #4 0x5580c90cf235 in doris::vectorized::ConvertImplGenericFromString::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long) /mnt/disk12/yanxuecheng/doris/be/src/vec/functions/function_cast.h:513:36 apache#5 0x5580c89607e6 in doris::Status std::__invoke_impl<doris::Status, doris::Status (*&)(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long), doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long, unsigned long>(std::__invoke_other, doris::Status (*&)(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long), doris::FunctionContext*&&, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long&&, unsigned long&&) /mnt/disk6/common/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14 apache#6 0x5580c89605ca in std::enable_if<is_invocable_r_v<doris::Status, doris::Status (*&)(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long), doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long, unsigned long>, doris::Status>::type std::__invoke_r<doris::Status, doris::Status (*&)(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long), doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long, unsigned long>(doris::Status (*&)(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long), doris::FunctionContext*&&, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long&&, unsigned long&&) /mnt/disk6/common/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:114:9 apache#7 0x5580c8960201 in std::_Function_handler<doris::Status (doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long, unsigned long), doris::Status (*)(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long)>::_M_invoke(std::_Any_data const&, doris::FunctionContext*&&, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long&&, unsigned long&&) /mnt/disk6/common/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290:9 apache#8 0x5580c827ff5c in std::function<doris::Status (doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long, unsigned long)>::operator()(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long, unsigned long) const /mnt/disk6/common/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591:9 apache#9 0x5580c88fb7a7 in doris::vectorized::FunctionCast::prepare_remove_nullable(doris::FunctionContext*, std::shared_ptr<doris::vectorized::IDataType const> const&, std::shared_ptr<doris::vectorized::IDataType const> const&, bool) const::'lambda'(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long)::operator()(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long) const /mnt/disk12/yanxuecheng/doris/be/src/vec/functions/function_cast.h:2148:17 apache#10 0x5580c88fae32 in doris::Status std::__invoke_impl<doris::Status, doris::vectorized::FunctionCast::prepare_remove_nullable(doris::FunctionContext*, std::shared_ptr<doris::vectorized::IDataType const> const&, std::shared_ptr<doris::vectorized::IDataType const> const&, bool) const::'lambda'(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long)&, doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long, unsigned long>(std::__invoke_other, doris::vectorized::FunctionCast::prepare_remove_nullable(doris::FunctionContext*, std::shared_ptr<doris::vectorized::IDataType const> const&, std::shared_ptr<doris::vectorized::IDataType const> const&, bool) const::'lambda'(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long)&, doris::FunctionContext*&&, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long&&, unsigned long&&) /mnt/disk6/common/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14 apache#11 0x5580c88fac0a in std::enable_if<is_invocable_r_v<doris::Status, doris::vectorized::FunctionCast::prepare_remove_nullable(doris::FunctionContext*, std::shared_ptr<doris::vectorized::IDataType const> const&, std::shared_ptr<doris::vectorized::IDataType const> const&, bool) const::'lambda'(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long)&, doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long, unsigned long>, doris::Status>::type std::__invoke_r<doris::Status, doris::vectorized::FunctionCast::prepare_remove_nullable(doris::FunctionContext*, std::shared_ptr<doris::vectorized::IDataType const> const&, std::shared_ptr<doris::vectorized::IDataType const> const&, bool) const::'lambda'(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long)&, doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long, unsigned long>(doris::vectorized::FunctionCast::prepare_remove_nullable(doris::FunctionContext*, std::shared_ptr<doris::vectorized::IDataType const> const&, std::shared_ptr<doris::vectorized::IDataType const> const&, bool) const::'lambda'(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long)&, doris::FunctionContext*&&, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long&&, unsigned long&&) /mnt/disk6/common/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:114:9 apache#12 0x5580c88fa641 in std::_Function_handler<doris::Status (doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long, unsigned long), doris::vectorized::FunctionCast::prepare_remove_nullable(doris::FunctionContext*, std::shared_ptr<doris::vectorized::IDataType const> const&, std::shared_ptr<doris::vectorized::IDataType const> const&, bool) const::'lambda'(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long)>::_M_invoke(std::_Any_data const&, doris::FunctionContext*&&, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long&&, unsigned long&&) /mnt/disk6/common/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290:9 apache#13 0x5580c827ff5c in std::function<doris::Status (doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long, unsigned long)>::operator()(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long, unsigned long) const /mnt/disk6/common/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591:9 ``` now ``` mysql> select cast('{"a":1,"b":"1","c":"1","d":"1"}' as struct<a:int, b:int>); +-----------------------------------------------------------------+ | cast('{"a":1,"b":"1","c":"1","d":"1"}' as struct<a:int, b:int>) | +-----------------------------------------------------------------+ | NULL | +-----------------------------------------------------------------+
wyxxxcat
pushed a commit
that referenced
this pull request
May 23, 2025
apache#50839) ### What problem does this PR solve? apache#50623 Optimize the initialization logic of preExecutionAuthenticator and remove redundant logic from HadoopIcebergCatalog." This change improves the robustness of the code by guaranteeing that the required initialization logic is executed, preventing potential null references or incomplete setup. fe.log ``` 2025-05-13 04:57:21,348 INFO (thrift-server-pool-22|191) [InternalCatalog.createDb():466] createDb dbName = regression_test_external_table_p0_broker_load, id = 1747082008077 2025-05-13 04:57:21,351 WARN (thrift-server-pool-9|191) [FrontendServiceImpl.fetchSchemaTableData():2341] Failed to fetchSchemaTableData java.lang.RuntimeException: NullPointerException: Cannot invoke "org.apache.doris.common.security.authentication.PreExecutionAuthenticator.execute(java.util.concurrent.Callable)" because the return value of "org.apache.doris.datasource.ExternalCatalog.getPreExecutionAuthenticator()" is null at org.apache.doris.datasource.iceberg.IcebergMetadataCache.loadTable(IcebergMetadataCache.java:133) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.datasource.iceberg.IcebergMetadataCache.lambda$new$1(IcebergMetadataCache.java:71) ~[doris-fe.jar:1.2-SNAPSHOT] at com.github.benmanes.caffeine.cache.LocalLoadingCache.lambda$newMappingFunction$2(LocalLoadingCache.java:145) ~[hive-catalog-shade-2.1.4.jar:2.1.4] at com.github.benmanes.caffeine.cache.LocalCache.lambda$statsAware$0(LocalCache.java:139) ~[hive-catalog-shade-2.1.4.jar:2.1.4] at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2406) ~[hive-catalog-shade-2.1.4.jar:2.1.4] at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1916) ~[?:?] at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2404) ~[hive-catalog-shade-2.1.4.jar:2.1.4] at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2387) ~[hive-catalog-shade-2.1.4.jar:2.1.4] at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:108) ~[hive-catalog-shade-2.1.4.jar:2.1.4] at com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:56) ~[hive-catalog-shade-2.1.4.jar:2.1.4] at org.apache.doris.datasource.iceberg.IcebergMetadataCache.getIcebergTable(IcebergMetadataCache.java:94) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.datasource.iceberg.IcebergMetadataCache.loadSnapshots(IcebergMetadataCache.java:113) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.datasource.iceberg.IcebergMetadataCache.lambda$new$0(IcebergMetadataCache.java:63) ~[doris-fe.jar:1.2-SNAPSHOT] at com.github.benmanes.caffeine.cache.LocalLoadingCache.lambda$newMappingFunction$2(LocalLoadingCache.java:145) ~[hive-catalog-shade-2.1.4.jar:2.1.4] at com.github.benmanes.caffeine.cache.LocalCache.lambda$statsAware$0(LocalCache.java:139) ~[hive-catalog-shade-2.1.4.jar:2.1.4] at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2406) ~[hive-catalog-shade-2.1.4.jar:2.1.4] at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1916) ~[?:?] at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2404) ~[hive-catalog-shade-2.1.4.jar:2.1.4] at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2387) ~[hive-catalog-shade-2.1.4.jar:2.1.4] at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:108) ~[hive-catalog-shade-2.1.4.jar:2.1.4] at com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:56) ~[hive-catalog-shade-2.1.4.jar:2.1.4] at org.apache.doris.datasource.iceberg.IcebergMetadataCache.getSnapshotList(IcebergMetadataCache.java:89) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.tablefunction.MetadataGenerator.icebergMetadataResult(MetadataGenerator.java:349) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.tablefunction.MetadataGenerator.getMetadataTable(MetadataGenerator.java:225) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.service.FrontendServiceImpl.fetchSchemaTableData(FrontendServiceImpl.java:2335) ~[doris-fe.jar:1.2-SNAPSHOT] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?] at org.apache.doris.service.FeServer.lambda$start$0(FeServer.java:60) ~[doris-fe.jar:1.2-SNAPSHOT] at jdk.proxy3.$Proxy67.fetchSchemaTableData(Unknown Source) ~[?:?] at org.apache.doris.thrift.FrontendService$Processor$fetchSchemaTableData.getResult(FrontendService.java:4777) ~[fe-common-1.2-SNAPSHOT.jar:1.2-SNAPSHOT] at org.apache.doris.thrift.FrontendService$Processor$fetchSchemaTableData.getResult(FrontendService.java:4757) ~[fe-common-1.2-SNAPSHOT.jar:1.2-SNAPSHOT] at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) ~[libthrift-0.16.0.jar:0.16.0] at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38) ~[libthrift-0.16.0.jar:0.16.0] at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:250) ~[libthrift-0.16.0.jar:0.16.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?] at java.lang.Thread.run(Thread.java:833) ~[?:?] Caused by: java.lang.NullPointerException: Cannot invoke "org.apache.doris.common.security.authentication.PreExecutionAuthenticator.execute(java.util.concurrent.Callable)" because the return value of "org.apache.doris.datasource.ExternalCatalog.getPreExecutionAuthenticator()" is null at org.apache.doris.datasource.iceberg.IcebergMetadataCache.loadTable(IcebergMetadataCache.java:130) ~[doris-fe.jar:1.2-SNAPSHOT] ... 38 more ``` be.log ``` W 2025-05-13 04:57:21,496 14827 status.h:451] meet error status: [INTERNAL_ERROR]TStatus: NullPointerException: Cannot invoke "org.apache.doris.common.security.authentication.PreExecutionAuthenticator.execute(java.util.concurrent.Callable)" because the return value of "org.apache.doris.datasource.ExternalCatalog.getPreExecutionAuthenticator()" is null 0# doris::Status doris::Status::create<true>(doris::TStatus const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:658 1# doris::vectorized::MetaScanner::_fetch_metadata(doris::TMetaScanRange const&) at /home/zcp/repo_center/doris_master/doris/be/src/common/status.h:517 2# doris::vectorized::MetaScanner::open(doris::RuntimeState*) at /home/zcp/repo_center/doris_master/doris/be/src/common/status.h:517 3# doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>) at /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:0 4# std::_Function_handler<void (), doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::{lambda()#1}>::_M_invoke(std::_Any_data const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1149 5# doris::ThreadPool::dispatch_thread() at /home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:0 6# doris::Thread::supervise_thread(void*) at /var/local/ldb-toolchain/bin/../usr/include/pthread.h:563 7# ? 8# ? W 2025-05-13 04:57:21,496 14827 meta_scanner.cpp:295] fetch schema table data from master failed, errmsg=[INTERNAL_ERROR]TStatus: NullPointerException: Cannot invoke "org.apache.doris.common.security.authentication.PreExecutionAuthenticator.execute(java.util.concurrent.Callable)" because the return value of "org.apache.doris.datasource.ExternalCatalog.getPreExecutionAuthenticator()" is null ```
wyxxxcat
pushed a commit
that referenced
this pull request
Jun 6, 2025
### What problem does this PR solve?
```
=================================================================
==8328==ERROR: AddressSanitizer: attempting double-free on 0x60600333cf80 in thread T1682:
#0 0x5631ccdaed9d in operator delete(void*) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x3caa6d9d) (BuildId: dab4df3df11ecdce)
#1 0x5631ccdde467 in std::__uniq_ptr_impl>::operator=(std::__uniq_ptr_impl>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:167:2
#2 0x5631ccdc942d in std::__uniq_ptr_data, true, true>::operator=(std::__uniq_ptr_data, true, true>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:212:61
#3 0x5631ccdc942d in std::unique_ptr>::operator=(std::unique_ptr>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:371:51
#4 0x5631ccdc942d in doris::Status::operator=(doris::Status&&) /home/zcp/repo_center/doris_master/doris/be/src/common/status.h:400:22
apache#5 0x56320bfc33ba in doris::pipeline::MaterializationCallback::call() /home/zcp/repo_center/doris_master/doris/be/src/pipeline/exec/materialization_sink_operator.cpp:88:39
apache#6 0x56320bfc5cdd in doris::AutoReleaseClosure>::Run() /home/zcp/repo_center/doris_master/doris/be/src/util/brpc_closure.h:98:18
apache#7 0x5631d03b7b1b in doris::FailureDetectClosure::Run() /home/zcp/repo_center/doris_master/doris/be/src/util/brpc_client_cache.h:83:20
apache#8 0x56320dce1943 in brpc::Controller::EndRPC(brpc::Controller::CompletionInfo const&) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7d9d9943) (BuildId: dab4df3df11ecdce)
apache#9 0x56320dd0cc4e in brpc::policy::ProcessRpcResponse(brpc::InputMessageBase*) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7da04c4e) (BuildId: dab4df3df11ecdce)
apache#10 0x56320dd03f76 in brpc::ProcessInputMessage(void*) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7d9fbf76) (BuildId: dab4df3df11ecdce)
apache#11 0x56320dd04920 in brpc::InputMessenger::InputMessageClosure::~InputMessageClosure() (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7d9fc920) (BuildId: dab4df3df11ecdce)
apache#12 0x56320dd052ab in brpc::InputMessenger::OnNewMessages(brpc::Socket*) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7d9fd2ab) (BuildId: dab4df3df11ecdce)
apache#13 0x56320de3437d in brpc::Socket::ProcessEvent(void*) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7db2c37d) (BuildId: dab4df3df11ecdce)
apache#14 0x56320dc970a6 in bthread::TaskGroup::task_runner(long) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7d98f0a6) (BuildId: dab4df3df11ecdce)
apache#15 0x56320dc83660 in bthread_make_fcontext (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7d97b660) (BuildId: dab4df3df11ecdce)
0x60600333cf80 is located 0 bytes inside of 64-byte region [0x60600333cf80,0x60600333cfc0)
freed by thread T1791 here:
#0 0x5631ccdaed9d in operator delete(void*) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x3caa6d9d) (BuildId: dab4df3df11ecdce)
previously allocated by thread T1623 here:
#0 0x5631ccdae53d in operator new(unsigned long) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x3caa653d) (BuildId: dab4df3df11ecdce)
Thread T1682 created by T0 here:
#0 0x5631ccd5bcaa in pthread_create (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x3ca53caa) (BuildId: dab4df3df11ecdce)
#1 0x56320dc907be in bthread::TaskControl::add_workers(int) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7d9887be) (BuildId: dab4df3df11ecdce)
#2 0x56320dc7f0ac in bthread_setconcurrency (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7d9770ac) (BuildId: dab4df3df11ecdce)
#3 0x56320ddfcc79 in brpc::Server::StartInternal(butil::EndPoint const&, brpc::PortRange const&, brpc::ServerOptions const*) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7daf4c79) (BuildId: dab4df3df11ecdce)
#4 0x56320ddfecc9 in brpc::Server::Start(butil::EndPoint const&, brpc::ServerOptions const*) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7daf6cc9) (BuildId: dab4df3df11ecdce)
apache#5 0x5631d13708df in doris::BRpcService::start(int, int) /home/zcp/repo_center/doris_master/doris/be/src/service/brpc_service.cpp:94:18
apache#6 0x5631ccdb73e2 in main /home/zcp/repo_center/doris_master/doris/be/src/service/doris_main.cpp:570:28
apache#7 0x7faec2201d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
Thread T1791 created by T0 here:
#0 0x5631ccd5bcaa in pthread_create (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x3ca53caa) (BuildId: dab4df3df11ecdce)
#1 0x56320dc907be in bthread::TaskControl::add_workers(int) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7d9887be) (BuildId: dab4df3df11ecdce)
#2 0x56320dc7f0ac in bthread_setconcurrency (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7d9770ac) (BuildId: dab4df3df11ecdce)
#3 0x56320ddfcc79 in brpc::Server::StartInternal(butil::EndPoint const&, brpc::PortRange const&, brpc::ServerOptions const*) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7daf4c79) (BuildId: dab4df3df11ecdce)
#4 0x56320ddfecc9 in brpc::Server::Start(butil::EndPoint const&, brpc::ServerOptions const*) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7daf6cc9) (BuildId: dab4df3df11ecdce)
apache#5 0x5631d13708df in doris::BRpcService::start(int, int) /home/zcp/repo_center/doris_master/doris/be/src/service/brpc_service.cpp:94:18
apache#6 0x5631ccdb73e2 in main /home/zcp/repo_center/doris_master/doris/be/src/service/doris_main.cpp:570:28
apache#7 0x7faec2201d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
Thread T1623 created by T0 here:
#0 0x5631ccd5bcaa in pthread_create (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x3ca53caa) (BuildId: dab4df3df11ecdce)
#1 0x56320dc907be in bthread::TaskControl::add_workers(int) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7d9887be) (BuildId: dab4df3df11ecdce)
#2 0x56320dc7f0ac in bthread_setconcurrency (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7d9770ac) (BuildId: dab4df3df11ecdce)
#3 0x56320ddfcc79 in brpc::Server::StartInternal(butil::EndPoint const&, brpc::PortRange const&, brpc::ServerOptions const*) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7daf4c79) (BuildId: dab4df3df11ecdce)
#4 0x56320ddfecc9 in brpc::Server::Start(butil::EndPoint const&, brpc::ServerOptions const*) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7daf6cc9) (BuildId: dab4df3df11ecdce)
apache#5 0x5631d13708df in doris::BRpcService::start(int, int) /home/zcp/repo_center/doris_master/doris/be/src/service/brpc_service.cpp:94:18
apache#6 0x5631ccdb73e2 in main /home/zcp/repo_center/doris_master/doris/be/src/service/doris_main.cpp:570:28
apache#7 0x7faec2201d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
SUMMARY: AddressSanitizer: double-free (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x3caa6d9d) (BuildId: dab4df3df11ecdce) in operator delete(void*)
```
Add a lock when need change the materialization status
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [x] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [x] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [x] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
wyxxxcat
pushed a commit
that referenced
this pull request
Jun 17, 2025
…apache#51743) During graceful exit, async result writer will not exit and hang because it will try to pull data block the queue endlessly. AsyncResultWriter thread should exit when fragment mgr wants to exit. Thread 652 (Thread 0x7fd20c1b0700 (LWP 434882) "FragmentMgrAsyn"): #0 futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7fca3442f560, clockid=<optimized out>, expected=0, futex_word=0x61b000c009f8) at ../sysdeps/nptl/futex-internal.h:320 #1 __pthread_cond_wait_common (abstime=0x7fca3442f560, clockid=<optimized out>, mutex=0x61b000c009a8, cond=0x61b000c009d0) at pthread_cond_wait.c:520 #2 __pthread_cond_timedwait (cond=0x61b000c009d0, mutex=0x61b000c009a8, abstime=0x7fca3442f560) at pthread_cond_wait.c:665 #3 0x00005566ce61f350 in __gthread_cond_timedwait (__cond=0x61b000c009d0, __mutex=0x61b000c009a8, __abs_timeout=0x7fca3442f560) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/x86_64-linux-gnu/c++/11/bits/gthr-default.h:872 #4 std::__condvar::wait_until (this=0x61b000c009d0, __m=..., __abs_time=...) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_mutex.h:162 apache#5 std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=<optimized out>, __lock=..., __atime=...) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/condition_variable:222 apache#6 0x00005566ce61f021 in std::condition_variable::wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=<optimized out>, __lock=..., __atime=...) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/condition_variable:135 apache#7 0x000055670b59b0be in std::condition_variable::wait_for<long, std::ratio<1l, 1l> > (this=0x61b000c009d0, __lock=..., __rtime=...) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/condition_variable:163 apache#8 doris::vectorized::AsyncResultWriter::process_block (this=<optimized out>, state=0x61e001b3c080, profile=<optimized out>) at /root/doris/be/src/vec/sink/writer/async_result_writer.cpp:149 apache#9 0x000055670b59eb1f in doris::vectorized::AsyncResultWriter::start_writer(doris::RuntimeState*, doris::RuntimeProfile*)::$_0::operator()() const (this=0x60400146b110) at /root/doris/be/src/vec/sink/writer/async_result_writer.cpp:103 apache#10 std::__invoke_impl<void, doris::vectorized::AsyncResultWriter::start_writer(doris::RuntimeState*, doris::RuntimeProfile*)::$_0&>(std::__invoke_other, doris::vectorized::AsyncResultWriter::start_writer(doris::RuntimeState*, doris::RuntimeProfile*)::$_0&) (__f=...) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61 apache#11 std::__invoke_r<void, doris::vectorized::AsyncResultWriter::start_writer(doris::RuntimeState*, doris::RuntimeProfile*)::$_0&>(doris::vectorized::AsyncResultWriter::start_writer(doris::RuntimeState*, doris::RuntimeProfile*)::$_0&) (__fn=...) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111 apache#12 std::_Function_handler<void (), doris::vectorized::AsyncResultWriter::start_writer(doris::RuntimeState*, doris::RuntimeProfile*)::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291 apache#13 0x00005566d2f2270b in doris::ThreadPool::dispatch_thread (this=0x616000415580) at /root/doris/be/src/util/threadpool.cpp:615 apache#14 0x00005566d2ef973a in doris::Thread::supervise_thread (arg=0x611005af0540) at /root/doris/be/src/util/thread.cpp:468 apache#15 0x00007fd5c2034609 in start_thread (arg=<optimized out>) at pthread_create.c:477 apache#16 0x00007fd5c22e1133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
wyxxxcat
pushed a commit
that referenced
this pull request
Jun 24, 2025
* aws/aws-sdk-cpp#2314 * In the aws-sdk-cpp code, there is a suggestion: "Please call ShutdownAPI from the same thread from which InitAPI() has been called" otherwise it will cause memory leak. ``` ================================================================= ==3271527==ERROR: LeakSanitizer: detected memory leaks Direct leak of 216 byte(s) in 1 object(s) allocated from: #0 0x555f9ecf55ee in malloc (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x370815ee) (BuildId: 22b637395dd039b0) #1 0x555fd8e19bd7 in CRYPTO_zalloc (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x711a5bd7) (BuildId: 22b637395dd039b0) #2 0x555fda7c9d14 in aws_mqtt_library_init (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x72b55d14) (BuildId: 22b637395dd039b0) #3 0x555fda74ed1f in Aws::Crt::ApiHandle::ApiHandle(aws_allocator*) (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x72adad1f) (BuildId: 22b637395dd039b0) #4 0x555fa365f0c9 in doris::S3ClientFactory::instance() /root/doris/be/src/util/s3_util.cpp:188:28 apache#5 0x555f9f13b87b in doris::io::ObjClientHolder::init() /root/doris/be/src/io/fs/s3_file_system.cpp:78:15 apache#6 0x555f9f140eee in doris::io::S3FileSystem::init() /root/doris/be/src/io/fs/s3_file_system.cpp:182:21 ...... ```
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed changes
Issue Number: close #xxx