Repository navigation
add parallel op case - #2
Merged
suxiaogang223 merged 2 commits intoJan 13, 2026
Merged
suxiaogang223 merged 2 commits into
suxiaogang223 merged 2 commits into
Conversation
suxiaogang223
pushed a commit
that referenced
this pull request
Jan 14, 2026
* update * add auth case
suxiaogang223
pushed a commit
that referenced
this pull request
Jan 16, 2026
…pache#59920) ### What problem does this PR solve? When `BaseBetaRowsetWriter` is destructed(this may happen when the load is canceled) before the execution of the task submitted to thread pool, the task may encounter coredump due to use after free. ``` (gdb) bt #0 __GI___pthread_sigmask (how=2, newmask=<optimized out>, oldmask=0x0) at ./nptl/pthread_sigmask.c:43 #1 0x00007fa1d0c1171e in PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] () from /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so #2 0x00007fa1d0c12206 in JVM_handle_linux_signal () from /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so #3 <signal handler called> apache#4 doris::TUniqueId::TUniqueId (this=0x7f99955f2208, other51=...) at /home/zcp/repo_center/doris_branch-4.0/doris/gensrc/build/gen_cpp/Types_types.cpp:2571 apache#5 0x00005653d14008ca in doris::AttachTask::init (rc=..., this=<optimized out>) at /home/zcp/repo_center/doris_branch-4.0/doris/be/src/runtime/thread_context.cpp:29 apache#6 doris::AttachTask::AttachTask (this=<optimized out>, rc=...) at /home/zcp/repo_center/doris_branch-4.0/doris/be/src/runtime/thread_context.cpp:34 apache#7 0x00005653d0d05087 in doris::CalcDeleteBitmapToken::submit_func<doris::BaseBetaRowsetWriter::_generate_delete_bitmap(int)::$_0>(doris::BaseBetaRowsetWriter::_generate_delete_bitmap(int)::$_0&&)::{lambda()#1}::operator()() const (this=0x7f9cdf302500) at /home/zcp/repo_center/doris_branch-4.0/doris/be/src/olap/calc_delete_bitmap_executor.h:74 apache#8 std::__invoke_impl<void, doris::CalcDeleteBitmapToken::submit_func<doris::BaseBetaRowsetWriter::_generate_delete_bitmap(int)::$_0>(doris::BaseBetaRowsetWriter::_generate_delete_bitmap(int)::$_0&&)::{lambda()#1}&>(std::__invoke_other, doris::CalcDeleteBitmapToken::submit_func<doris::BaseBetaRowsetWriter::_generate_delete_bitmap(int)::$_0>(doris::BaseBetaRowsetWriter::_generate_delete_bitmap(int)::$_0&&)::{lambda()#1}&) (__f=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:63 apache#9 std::__invoke_r<void, doris::CalcDeleteBitmapToken::submit_func<doris::BaseBetaRowsetWriter::_generate_delete_bitmap(int)::$_0>(doris::BaseBetaRowsetWriter::_generate_delete_bitmap(int)::$_0&&)::{lambda()#1}&>(doris::CalcDeleteBitmapToken::submit_func<doris::BaseBetaRowsetWriter::_generate_delete_bitmap(int)::$_0>(doris::BaseBetaRowsetWriter::_generate_delete_bitmap(int)::$_0&&)::{lambda()#1}&) (__fn=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:113 apache#10 std::_Function_handler<void (), doris::CalcDeleteBitmapToken::submit_func<doris::BaseBetaRowsetWriter::_generate_delete_bitmap(int)::$_0>(doris::BaseBetaRowsetWriter::_generate_delete_bitmap(int)::$_0&&)::{lambda()#1}>::_M_invoke(std::_Any_data const&) ( __functor=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:292 apache#11 0x00005653d16392e5 in doris::ThreadPool::dispatch_thread (this=0x7fa120d9af00) at /home/zcp/repo_center/doris_branch-4.0/doris/be/src/util/threadpool.cpp:616 apache#12 0x00005653d162e38c in std::function<void ()>::operator()() const (this=0x7f99955f2208) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593 apache#13 doris::Thread::supervise_thread (arg=0x7fa0c0049110) at /home/zcp/repo_center/doris_branch-4.0/doris/be/src/util/thread.cpp:460 apache#14 0x00007fa1cfcacac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 apache#15 0x00007fa1cfd3e850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 ``` ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
suxiaogang223
pushed a commit
that referenced
this pull request
Feb 3, 2026
…lain verbose (apache#60308) ### What problem does this PR solve? Problem Summary: This PR enhances the output of EXPLAIN VERBOSE for File Scan nodes by adding the following metrics: `dataFileNum=xxx, deleteFileNum=xxx, deleteSplitNum=xxx` Especially useful for iceberg/paimon/hive acid These metrics provide more visibility into the underlying file and split layout, helping users better tune parameters and control query performance. Details: `dataFileNum` : The number of distinct data files that need to be read. This is not equivalent to the number of splits, since a single data file can be divided into multiple splits. `deleteFileNum` : The number of distinct delete files that need to be read. `deleteSplitNum` : Added because the relationship between data files and delete files is many-to-many: one data file may be associated with multiple delete files one delete file may apply to multiple data files Using deleteSplitNum / dataSplitNum, users can estimate the average number of delete splits that need to be read per data split. Example: ``` mysql> explain verbose select * from iceberg.format_v3.dv_test_1w; +-----------------------------------------------------------------------------------------------------------------------------------------------+ | Explain String(Nereids Planner) | +-----------------------------------------------------------------------------------------------------------------------------------------------+ | PLAN FRAGMENT 0 | | OUTPUT EXPRS: | | id[#0] | | grp[#1] | | value[#2] | | ts[#3] | | PARTITION: RANDOM | | | | HAS_COLO_PLAN_NODE: false | | | | VRESULT SINK | | MYSQL_PROTOCOL | | | | 0:VICEBERG_SCAN_NODE(32) | | table: iceberg.format_v3.dv_test_1w | | inputSplitNum=220, totalFileSize=720774, scanRanges=220 | | partition=0/0 | | backends: | | 1769590309070 | | s3://warehouse/wh/format_v3/dv_test_1w/data/00004-51-fc462f9a-d42a-404d-adfc-c8d2781c8d04-0-00001.parquet start: 4 length: 2672 | | s3://warehouse/wh/format_v3/dv_test_1w/data/00003-50-fc462f9a-d42a-404d-adfc-c8d2781c8d04-0-00001.parquet start: 4 length: 2852 | | s3://warehouse/wh/format_v3/dv_test_1w/data/00000-47-fc462f9a-d42a-404d-adfc-c8d2781c8d04-0-00001.parquet start: 4 length: 2894 | | ... other 216 files ... | | s3://warehouse/wh/format_v3/dv_test_1w/data/00001-48-fc462f9a-d42a-404d-adfc-c8d2781c8d04-0-00001.parquet start: 58397 length: 13894 | | dataFileNum=10, deleteFileNum=1 deleteSplitNum=220 | | cardinality=33334, numNodes=1 | | pushdown agg=NONE | | tuple ids: 0 | | | | Tuples: | | TupleDescriptor{id=0, tbl=dv_test_1w} | | SlotDescriptor{id=0, col=id, colUniqueId=1, type=bigint, nullable=true, isAutoIncrement=false, subColPath=null, virtualColumn=null} | | SlotDescriptor{id=1, col=grp, colUniqueId=2, type=int, nullable=true, isAutoIncrement=false, subColPath=null, virtualColumn=null} | | SlotDescriptor{id=2, col=value, colUniqueId=3, type=int, nullable=true, isAutoIncrement=false, subColPath=null, virtualColumn=null} | | SlotDescriptor{id=3, col=ts, colUniqueId=4, type=datetimev2(6), nullable=true, isAutoIncrement=false, subColPath=null, virtualColumn=null} | | | | | | | | | | ========== STATISTICS ========== | +-----------------------------------------------------------------------------------------------------------------------------------------------+ ```
suxiaogang223
pushed a commit
that referenced
this pull request
Mar 12, 2026
…n_udf_support` is not set (apache#61196) When `enable_python_udf_support` is not set to true, a null pointer dereference occurs when invoking helper commands (`SHOW PYTHON VERSION/ PACKAGES IN`). ```text /home/zcp/repo_center/doris_master/doris/be/src/udf/python/python_env.cpp:305:38: runtime error: member call on null pointer of type 'doris::PythonEnvScanner' #0 0x559edede162c in doris::PythonVersionManager::env_infos_to_thrift() const /home/zcp/repo_center/doris_master/doris/be/src/udf/python/python_env.cpp:305:38 #1 0x559ede63acd4 in doris::BaseBackendService::get_python_envs(std::vector>&) /home/zcp/repo_center/doris_master/doris/be/src/service/backend_service.cpp:1316:47 #2 0x559edf99796a in doris::BackendServiceProcessor::process_get_python_envs(int, apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, void*) /home/zcp/repo_center/doris_master/doris/gensrc/build/gen_cpp/BackendService.cpp:7789:13 #3 0x559edf94a69c in doris::BackendServiceProcessor::dispatchCall(apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, std::__cxx11::basic_string, std::allocator> const&, int, void*) /home/zcp/repo_center/doris_master/doris/gensrc/build/gen_cpp/BackendService.cpp:6466:3 apache#4 0x559eac6bad06 in apache::thrift::TDispatchProcessor::process(std::shared_ptr, std::shared_ptr, void*) /home/zcp/repo_center/doris_master/doris/thirdparty/installed/include/thrift/TDispatchProcessor.h:121:12 apache#5 0x559ee47c071a in apache::thrift::server::TConnectedClient::run() (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x8a1c871a) apache#6 0x559ee47c1a45 in apache::thrift::server::TThreadedServer::TConnectedClientRunner::run() (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x8a1c9a45) apache#7 0x559ee47c557f in apache::thrift::concurrency::Thread::threadMain(std::shared_ptr) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x8a1cd57f) apache#8 0x559ee47c52a5 in void std::__invoke_impl), std::shared_ptr>(std::__invoke_other, void (*&&)(std::shared_ptr), std::shared_ptr&&) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x8a1cd2a5) apache#9 0x559ee47c521c in std::__invoke_result), std::shared_ptr>::type std::__invoke), std::shared_ptr>(void (*&&)(std::shared_ptr), std::shared_ptr&&) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x8a1cd21c) apache#10 0x559ee47c51f1 in void std::thread::_Invoker), std::shared_ptr>>::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x8a1cd1f1) apache#11 0x559ee47c51b4 in std::thread::_Invoker), std::shared_ptr>>::operator()() (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x8a1cd1b4) apache#12 0x559ee47c4fd8 in std::thread::_State_impl), std::shared_ptr>>>::_M_run() (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x8a1ccfd8) apache#13 0x559ef61b5f3f in execute_native_thread_routine archive64.c apache#14 0x559eac535d26 in asan_thread_start(void*) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x51f3dd26) apache#15 0x7fcba62bfac2 in start_thread nptl/pthread_create.c:442:8 apache#16 0x7fcba635184f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /home/zcp/repo_center/doris_master/doris/be/src/udf/python/python_env.cpp:305:38 ``` now: ```text Doris> show python packages in '3.19.18'; ERROR 1105 (HY000): errCode = 2, detailMessage = Failed to get python packages from any backend: [E42] Set 'python_venv_interpreter_paths' in be.conf to enable PythonUDF feature Doris> show python versions; ERROR 1105 (HY000): errCode = 2, detailMessage = Failed to get python envs from any backend: [E-236] Set 'python_venv_interpreter_paths' in be.conf to enable PythonUDF feature ```
suxiaogang223
pushed a commit
that referenced
this pull request
Mar 19, 2026
… exit (apache#61040) ### What problem does this PR solve? The SyncPoint singleton was being destroyed during program exit while background bthread timers were still accessing it, causing heap-use-after-free. Changed get_instance() to return a never-destructed heap-allocated pointer instead of a stack-allocated static object to prevent destruction order issues. This is a standard pattern for singletons accessed by background threads. ``` 11:47:40 #0 0x5582a963153d in operator new(unsigned long) (/root/doris/cloud/ut_build_ASAN/test/recycler_test+0x122b53d) (BuildId: 1191affa24589f52) 11:47:40 #1 0x5582ab0aa52f in doris::SyncPoint::SyncPoint() /root/doris/common/cpp/sync_point.cpp:77:9 11:47:40 #2 0x5582ab0aa46b in doris::SyncPoint::get_instance() /root/doris/common/cpp/sync_point.cpp:73:20 11:47:40 #3 0x5582aae33ee8 in doris::cloud::memkv::Transaction::get(std::basic_string_view<char, std::char_traits<char>>, std::basic_string_view<char, std::char_traits<char>>, std::unique_ptr<doris::cloud::RangeGetIterator, std::default_delete<doris::cloud::RangeGetIterator>>*, bool, int) /root/doris/cloud/src/meta-store/mem_txn_kv.cpp:276:5 11:47:40 apache#4 0x5582a968f49f in doris::cloud::txn_get(doris::cloud::TxnKv*, std::basic_string_view<char, std::char_traits<char>>, std::basic_string_view<char, std::char_traits<char>>, std::unique_ptr<doris::cloud::RangeGetIterator, std::default_delete<doris::cloud::RangeGetIterator>>&) /root/doris/cloud/src/recycler/recycler.cpp:99:18 11:47:40 apache#5 0x5582a9657090 in doris::cloud::InstanceRecycler::scan_and_recycle(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::basic_string_view<char, std::char_traits<char>>, std::function<int (std::basic_string_view<char, std::char_traits<char>>, std::basic_string_view<char, std::char_traits<char>>)>, std::function<int ()>) /root/doris/cloud/src/recycler/recycler.cpp:2976:23 11:47:40 apache#6 0x5582a9686153 in doris::cloud::InstanceRecycler::recycle_rowsets() /root/doris/cloud/src/recycler/recycler.cpp:2587:15 11:47:40 apache#7 0x5582a96a8d47 in doris::cloud::RecyclerTest_recycle_empty_Test::TestBody() /root/doris/cloud/test/recycler_test.cpp:1136:5 ```
suxiaogang223
pushed a commit
that referenced
this pull request
Mar 24, 2026
…free. (apache#61138) ### What problem does this PR solve? Related PR: apache/doris-thirdparty#383 Problem Summary: fix coredump: ``` #0 0x5561f1d46f12 in operator delete(void*, unsigned long) (/mnt/disk2/tengjianping/doris-master/output/be/lib/doris_be+0x50ae4f12) #1 0x55622c94b454 in std::__new_allocator<std::shared_ptr<orc::ExpressionTree>>::deallocate(std::shared_ptr<orc::ExpressionTree>*, unsigned long) (/mnt/disk2/tengjianping/doris-master/output/be/lib/doris_be+0x8b6e9454) #2 0x55622c94b41e in std::_Deque_base<std::shared_ptr<orc::ExpressionTree>, std::allocator<std::shared_ptr<orc::ExpressionTree>>>::_M_deallocate_node(std::shared_ptr<orc::ExpressionTree>*) (/mnt/disk2/tengjianping/doris-master/output/be/lib/doris_be+0x8b6e941e) #3 0x55622c950681 in std::deque<std::shared_ptr<orc::ExpressionTree>, std::allocator<std::shared_ptr<orc::ExpressionTree>>>::_M_pop_front_aux() (/mnt/disk2/tengjianping/doris-master/output/be/lib/doris_be+0x8b6ee681) apache#4 0x55622c94868e in std::deque<std::shared_ptr<orc::ExpressionTree>, std::allocator<std::shared_ptr<orc::ExpressionTree>>>::pop_front() (/mnt/disk2/tengjianping/doris-master/output/be/lib/doris_be+0x8b6e668e) apache#5 0x55622c947d79 in orc::rewriteLeaves(std::shared_ptr<orc::ExpressionTree>, unsigned long*) SearchArgument.cc apache#6 0x55622c947809 in orc::SearchArgumentBuilderImpl::build() (/mnt/disk2/tengjianping/doris-master/output/be/lib/doris_be+0x8b6e5809) apache#7 0x5562195209a6 in doris::vectorized::OrcReader::_init_search_argument(std::vector<std::shared_ptr<doris::vectorized::VExpr>, std::allocator<std::shared_ptr<doris::vectorized::VExpr>>> const&) /mnt/disk2/tengjianping/doris-master/be/src/vec/exec/format/orc/vorc_reader.cpp:1072:27 apache#8 0x556219527175 in doris::vectorized::OrcReader::set_fill_columns(std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::SlotDescriptor const*>, std::hash<std::__cxx11::basic_string<char, ```
suxiaogang223
pushed a commit
that referenced
this pull request
Mar 24, 2026
```
==1472432==ERROR: AddressSanitizer: heap-use-after-free on address 0x7bc7535fba50 at pc 0x557ec5178739 bp 0x7b1743746d40 sp 0x7b1743746500
WRITE of size 104 at 0x7bc7535fba50 thread T2
#0 0x557ec5178738 in memset ~/workspace/llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors_memintrinsics.inc:87:3
#1 0x7f17566e64fc in fdb_transaction_set_option (libfdb_c.so+0xc014fc) (BuildId: 854ec989d11d3c183aa4963a6d758cfc2b571b07)
#2 0x7f17566e2e47 in fdb_transaction_set_option (libfdb_c.so+0xbfde47) (BuildId: 854ec989d11d3c183aa4963a6d758cfc2b571b07)
#3 0x7f17566e4ec5 in fdb_transaction_set_option (libfdb_c.so+0xbffec5) (BuildId: 854ec989d11d3c183aa4963a6d758cfc2b571b07)
apache#4 0x7f17564f80ee in fdb_transaction_set_option (libfdb_c.so+0xa130ee) (BuildId: 854ec989d11d3c183aa4963a6d758cfc2b571b07)
apache#5 0x7f1755e55b71 in fdb_transaction_set_option (libfdb_c.so+0x370b71) (BuildId: 854ec989d11d3c183aa4963a6d758cfc2b571b07)
apache#6 0x7f1755e2b2e9 in fdb_run_network (libfdb_c.so+0x3462e9) (BuildId: 854ec989d11d3c183aa4963a6d758cfc2b571b07)
apache#7 0x557ec6595a2b in doris::cloud::fdb::Network::init()::::operator()() const ~/workspace/doris/cloud/ut_build_ASAN/../src/meta-store/txn_kv.cpp:468:28
apache#8 0x557ec65958e6 in void std::__invoke_impl<void, doris::cloud::fdb::Network::init()::>(std::__invoke_other, doris::cloud::fdb::Network::init()::&&) /usr/lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14
apache#9 0x557ec6595886 in std::__invoke_result<doris::cloud::fdb::Network::init()::>::type std::__invoke<doris::cloud::fdb::Network::init()::>(doris::cloud::fdb::Network::init()::&&) /usr/lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:96:14
apache#10 0x557ec659584e in void std::thread::_Invoker<std::tuple<doris::cloud::fdb::Network::init()::>>::_M_invoke<0ul>(std::_Index_tuple<0ul>) /usr/lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_thread.h:292:13
apache#11 0x557ec6595816 in std::thread::_Invoker<std::tuple<doris::cloud::fdb::Network::init()::>>::operator()() /usr/lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_thread.h:299:11
apache#12 0x557ec659571a in std::thread::_State_impl<std::thread::_Invoker<std::tuple<doris::cloud::fdb::Network::init()::>>>::_M_run() /usr/lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_thread.h:244:13
apache#13 0x557ec7756e43 in execute_native_thread_routine InstrProfilingUtil.c
apache#14 0x557ec50b1a07 in asan_thread_start(void*) ~/workspace/llvm-project/compiler-rt/lib/asan/asan_interceptors.cpp:239:43
apache#15 0x7f17545caaa3 in start_thread nptl/pthread_create.c:447:8
apache#16 0x7f1754657a63 in clone misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:100
```
suxiaogang223
pushed a commit
that referenced
this pull request
Apr 20, 2026
### What problem does this PR solve? Fix memory corruption in PredicateColumnType::insert_duplicate_fields for string types Fixed a bug in the pointer arithmetic when inserting duplicate string values. The original code used dst += i * str.size() which incorrectly multiplied by the loop index, causing: - Iteration 0: no advance (correct by accident) - Iteration 1: advance by 1×size (correct) - Iteration 2: advance by 2×size (wrong - overwrites beyond allocated memory) This resulted in memory corruption and incorrect string data when inserting more than 2 duplicates. The fix changes to dst += str.size() to correctly advance the pointer by one string length per iteration. Added unit tests covering: - String type with multiple duplicate insertions - Integer type duplicates - LargeInt (Int128) type duplicates Introduced by apache#60530 ==13605==ERROR: AddressSanitizer: use-after-poison on address 0x7d639d26bc30 at pc 0x55e8b71df91e bp 0x7b4e8fc1e510 sp 0x7b4e8fc1dcd0 WRITE of size 6 at 0x7d639d26bc30 thread T620 (ls_normal [work) #0 0x55e8b71df91d in __asan_memcpy (/mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be+0x2205791d) #1 0x55e8cce8b1ab in doris::PredicateColumnType<(doris::PrimitiveType)23>::insert_duplicate_fields(doris::Field const&, unsigned long) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/core/column/predicate_column.h:327:17 #2 0x55e8b7db4e83 in doris::ColumnNullable::insert_duplicate_fields(doris::Field const&, unsigned long) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/core/column/column_nullable.cpp:357:29 #3 0x55e8ccf8ba2c in doris::segment_v2::DefaultValueColumnIterator::_insert_many_default(doris::COW<doris::IColumn>::mutable_ptr<doris::IColumn>&, unsigned long) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/segment/column_reader.cpp:2167:14 apache#4 0x55e8ccf3c705 in doris::segment_v2::DefaultValueColumnIterator::next_batch(unsigned long*, doris::COW<doris::IColumn>::mutable_ptr<doris::IColumn>&, bool*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/segment/column_reader.cpp:2152:5 apache#5 0x55e8cd4f9313 in doris::segment_v2::ColumnIterator::next_batch(unsigned long*, doris::COW<doris::IColumn>::mutable_ptr<doris::IColumn>&) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/segment/column_reader.h:330:16 apache#6 0x55e8cd4f9313 in doris::segment_v2::SegmentIterator::_read_columns_by_index(unsigned int, unsigned short&) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/segment/segment_iterator.cpp:2291:13 apache#7 0x55e8cd50621b in doris::segment_v2::SegmentIterator::_next_batch_internal(doris::Block*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/segment/segment_iterator.cpp:2643:5 apache#8 0x55e8cd4a0b82 in doris::segment_v2::SegmentIterator::next_batch(doris::Block*)::$_0::operator()() const /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/segment/segment_iterator.cpp:2519:9 apache#9 0x55e8cd4a0b82 in doris::segment_v2::SegmentIterator::next_batch(doris::Block*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/segment/segment_iterator.cpp:2518:19 apache#10 0x55e8cd37cca1 in doris::segment_v2::LazyInitSegmentIterator::next_batch(doris::Block*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/segment/lazy_init_segment_iterator.h:46:33 apache#11 0x55e8cccc0b05 in doris::Status doris::BetaRowsetReader::_next_batch<doris::Block>(doris::Block*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/rowset/beta_rowset_reader.h:113:35 apache#12 0x55e8ccc98668 in doris::BetaRowsetReader::next_batch(doris::Block*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/rowset/beta_rowset_reader.h:56:55 apache#13 0x55e8ca62ee8f in doris::VCollectIterator::Level0Iterator::_refresh() /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/iterator/vcollect_iterator.h apache#14 0x55e8ca611036 in doris::VCollectIterator::Level0Iterator::refresh_current_row() /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/iterator/vcollect_iterator.cpp:523:24 apache#15 0x55e8ca601a08 in doris::VCollectIterator::Level0Iterator::ensure_first_row_ref() /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/iterator/vcollect_iterator.cpp:496:14 apache#16 0x55e8ca604ea2 in doris::VCollectIterator::Level1Iterator::ensure_first_row_ref() /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/iterator/vcollect_iterator.cpp:713:27 apache#17 0x55e8ca608b20 in doris::VCollectIterator::build_heap(std::vector<std::shared_ptr<doris::RowsetReader>, std::allocator<std::shared_ptr<doris::RowsetReader>>>&) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/iterator/vcollect_iterator.cpp:189:9 apache#18 0x55e8ca5dc1ae in doris::BlockReader::_init_collect_iter(doris::TabletReader::ReaderParams const&) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/iterator/block_reader.cpp:152:9 apache#19 0x55e8ca5d7b9d in doris::BlockReader::init(doris::TabletReader::ReaderParams const&) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/iterator/block_reader.cpp:226:19 apache#20 0x55e8d02df6d6 in doris::OlapScanner::_open_impl(doris::RuntimeState*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/olap_scanner.cpp:317:32 apache#21 0x55e8d02cd697 in doris::Scanner::open(doris::RuntimeState*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/scanner.h:80:16 apache#22 0x55e8d02b576a in doris::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/scanner_scheduler.cpp:185:5 apache#23 0x55e8d02bf603 in doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/scanner_scheduler.cpp:75:17 apache#24 0x55e8d02bf603 in doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()::operator()() const /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/scanner_scheduler.cpp:74:27 apache#25 0x55e8d02bf603 in bool std::__invoke_impl<bool, doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()&>(std::__invoke_other, doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:63:14 apache#26 0x55e8d02bf603 in std::enable_if<is_invocable_r_v<bool, doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()&>, bool>::type std::__invoke_r<bool, doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()&>(doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:116:9 apache#27 0x55e8d02bf603 in std::_Function_handler<bool (), doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()>::_M_invoke(std::_Any_data const&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:292:9 apache#28 0x55e8d02bed85 in std::function<bool ()>::operator()() const /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593:9 apache#29 0x55e8d02bed85 in doris::ScannerSplitRunner::process_for(std::chrono::duration<long, std::ratio<1l, 1000000000l>>) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/scanner_scheduler.cpp:425:25 apache#30 0x55e8d02aa844 in doris::PrioritizedSplitRunner::process() /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/task_executor/time_sharing/prioritized_split_runner.cpp:102:35 apache#31 0x55e8d0272674 in doris::TimeSharingTaskExecutor::_dispatch_thread() /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/task_executor/time_sharing/time_sharing_task_executor.cpp:556:77 apache#32 0x55e8d3fa8ad1 in std::function<void ()>::operator()() const /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593:9 apache#33 0x55e8d3fa8ad1 in doris::Thread::supervise_thread(void*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/util/thread.cpp:460:5 apache#34 0x55e8b71ddd26 in asan_thread_start(void*) (/mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be+0x22055d26) apache#35 0x7f539b835ac2 in start_thread nptl/pthread_create.c:442:8 apache#36 0x7f539b8c78cf misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 0x7d639d26bc30 is located 816 bytes inside of 4096-byte region [0x7d639d26b900,0x7d639d26c900) allocated by thread T620 (ls_normal [work) here: #0 0x55e8b71e1b34 in malloc (/mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be+0x22059b34) #1 0x55e8b78d2140 in doris::DefaultMemoryAllocator::malloc(unsigned long) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/core/allocator.h:90:55 #2 0x55e8b78d2140 in doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, true>::alloc(unsigned long, unsigned long) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/core/allocator.cpp:337:23 #3 0x55e8b7ccdfaf in doris::Arena::Chunk::Chunk(unsigned long, doris::Arena::Chunk*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/core/arena.h:59:63 apache#4 0x55e8b7ccdad9 in doris::Arena::_init_head_if_needed() /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/core/arena.h:131:24 apache#5 0x55e8b7ccd758 in doris::Arena::alloc(unsigned long) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/core/arena.h:148:9 apache#6 0x55e8cce8b118 in doris::PredicateColumnType<(doris::PrimitiveType)23>::insert_duplicate_fields(doris::Field const&, unsigned long) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/core/column/predicate_column.h:325:32 apache#7 0x55e8b7db4e83 in doris::ColumnNullable::insert_duplicate_fields(doris::Field const&, unsigned long) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/core/column/column_nullable.cpp:357:29 apache#8 0x55e8ccf8ba2c in doris::segment_v2::DefaultValueColumnIterator::_insert_many_default(doris::COW<doris::IColumn>::mutable_ptr<doris::IColumn>&, unsigned long) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/segment/column_reader.cpp:2167:14 apache#9 0x55e8ccf3c705 in doris::segment_v2::DefaultValueColumnIterator::next_batch(unsigned long*, doris::COW<doris::IColumn>::mutable_ptr<doris::IColumn>&, bool*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/segment/column_reader.cpp:2152:5 apache#10 0x55e8cd4f9313 in doris::segment_v2::ColumnIterator::next_batch(unsigned long*, doris::COW<doris::IColumn>::mutable_ptr<doris::IColumn>&) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/segment/column_reader.h:330:16 apache#11 0x55e8cd4f9313 in doris::segment_v2::SegmentIterator::_read_columns_by_index(unsigned int, unsigned short&) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/segment/segment_iterator.cpp:2291:13 apache#12 0x55e8cd50621b in doris::segment_v2::SegmentIterator::_next_batch_internal(doris::Block*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/segment/segment_iterator.cpp:2643:5 apache#13 0x55e8cd4a0b82 in doris::segment_v2::SegmentIterator::next_batch(doris::Block*)::$_0::operator()() const /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/segment/segment_iterator.cpp:2519:9 apache#14 0x55e8cd4a0b82 in doris::segment_v2::SegmentIterator::next_batch(doris::Block*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/segment/segment_iterator.cpp:2518:19 apache#15 0x55e8cd37cca1 in doris::segment_v2::LazyInitSegmentIterator::next_batch(doris::Block*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/segment/lazy_init_segment_iterator.h:46:33 apache#16 0x55e8cccc0b05 in doris::Status doris::BetaRowsetReader::_next_batch<doris::Block>(doris::Block*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/rowset/beta_rowset_reader.h:113:35 apache#17 0x55e8ccc98668 in doris::BetaRowsetReader::next_batch(doris::Block*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/rowset/beta_rowset_reader.h:56:55 apache#18 0x55e8ca62ee8f in doris::VCollectIterator::Level0Iterator::_refresh() /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/iterator/vcollect_iterator.h apache#19 0x55e8ca611036 in doris::VCollectIterator::Level0Iterator::refresh_current_row() /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/iterator/vcollect_iterator.cpp:523:24 apache#20 0x55e8ca601a08 in doris::VCollectIterator::Level0Iterator::ensure_first_row_ref() /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/iterator/vcollect_iterator.cpp:496:14 apache#21 0x55e8ca604ea2 in doris::VCollectIterator::Level1Iterator::ensure_first_row_ref() /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/iterator/vcollect_iterator.cpp:713:27 apache#22 0x55e8ca608b20 in doris::VCollectIterator::build_heap(std::vector<std::shared_ptr<doris::RowsetReader>, std::allocator<std::shared_ptr<doris::RowsetReader>>>&) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/iterator/vcollect_iterator.cpp:189:9 apache#23 0x55e8ca5dc1ae in doris::BlockReader::_init_collect_iter(doris::TabletReader::ReaderParams const&) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/iterator/block_reader.cpp:152:9 apache#24 0x55e8ca5d7b9d in doris::BlockReader::init(doris::TabletReader::ReaderParams const&) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/iterator/block_reader.cpp:226:19 apache#25 0x55e8d02df6d6 in doris::OlapScanner::_open_impl(doris::RuntimeState*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/olap_scanner.cpp:317:32 apache#26 0x55e8d02cd697 in doris::Scanner::open(doris::RuntimeState*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/scanner.h:80:16 apache#27 0x55e8d02b576a in doris::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/scanner_scheduler.cpp:185:5 apache#28 0x55e8d02bf603 in doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/scanner_scheduler.cpp:75:17 apache#29 0x55e8d02bf603 in doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()::operator()() const /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/scanner_scheduler.cpp:74:27 apache#30 0x55e8d02bf603 in bool std::__invoke_impl<bool, doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()&>(std::__invoke_other, doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:63:14 apache#31 0x55e8d02bf603 in std::enable_if<is_invocable_r_v<bool, doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()&>, bool>::type std::__invoke_r<bool, doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()&>(doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:116:9 apache#32 0x55e8d02bf603 in std::_Function_handler<bool (), doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()>::_M_invoke(std::_Any_data const&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:292:9 apache#33 0x55e8d02bed85 in std::function<bool ()>::operator()() const /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593:9 apache#34 0x55e8d02bed85 in doris::ScannerSplitRunner::process_for(std::chrono::duration<long, std::ratio<1l, 1000000000l>>) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/scanner_scheduler.cpp:425:25 apache#35 0x55e8d02aa844 in doris::PrioritizedSplitRunner::process() /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/task_executor/time_sharing/prioritized_split_runner.cpp:102:35 apache#36 0x55e8d0272674 in doris::TimeSharingTaskExecutor::_dispatch_thread() /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/task_executor/time_sharing/time_sharing_task_executor.cpp:556:77 apache#37 0x55e8d3fa8ad1 in std::function<void ()>::operator()() const /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593:9 apache#38 0x55e8d3fa8ad1 in doris::Thread::supervise_thread(void*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/util/thread.cpp:460:5 Thread T620 (ls_normal [work) created by T0 here: #0 0x55e8b71c5e91 in pthread_create (/mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be+0x2203de91) #1 0x55e8d3fa7b8a in doris::Thread::start_thread(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::function<void ()> const&, std::shared_ptr<doris::Thread>*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/util/thread.cpp:412:15 #2 0x55e8d02849fd in doris::Status doris::Thread::create<void (doris::TimeSharingTaskExecutor::*)(), doris::TimeSharingTaskExecutor*>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, void (doris::TimeSharingTaskExecutor::* const&)(), doris::TimeSharingTaskExecutor* const&, std::shared_ptr<doris::Thread>*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/util/thread.h:47:16 #3 0x55e8d026e6b3 in doris::TimeSharingTaskExecutor::_create_thread() /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/task_executor/time_sharing/time_sharing_task_executor.cpp:671:12 apache#4 0x55e8d026b628 in doris::TimeSharingTaskExecutor::_try_create_thread(int, std::lock_guard<std::mutex>&) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/task_executor/time_sharing/time_sharing_task_executor.cpp:360:25 apache#5 0x55e8d026967a in doris::TimeSharingTaskExecutor::init() /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/task_executor/time_sharing/time_sharing_task_executor.cpp:221:27 apache#6 0x55e8d025a85a in doris::TaskExecutorSimplifiedScanScheduler::start(int, int, int, int) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/scanner_scheduler.h:288:9 apache#7 0x55e8d368d649 in doris::WorkloadGroup::upsert_thread_pool_no_lock(doris::WorkloadGroupInfo*, std::shared_ptr<doris::CgroupCpuCtl>) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/runtime/workload_group/workload_group.cpp:571:38 apache#8 0x55e8d368f14d in doris::WorkloadGroup::upsert_task_scheduler(doris::WorkloadGroupInfo*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/runtime/workload_group/workload_group.cpp:667:12 apache#9 0x55e8d36bd223 in doris::WorkloadGroupMgr::create_internal_wg() /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/runtime/workload_group/workload_group_manager.cpp:900:5 apache#10 0x55e8d32bbd3d in doris::ExecEnv::_create_internal_workload_group() /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/runtime/exec_env_init.cpp:475:5 apache#11 0x55e8d32b27fc in doris::ExecEnv::_init(std::vector<doris::StorePath, std::allocator<doris::StorePath>> const&, std::vector<doris::StorePath, std::allocator<doris::StorePath>> const&, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>> const&) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/runtime/exec_env_init.cpp:424:5 apache#12 0x55e8d32ad7ec in doris::ExecEnv::init(doris::ExecEnv*, std::vector<doris::StorePath, std::allocator<doris::StorePath>> const&, std::vector<doris::StorePath, std::allocator<doris::StorePath>> const&, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>> const&) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/runtime/exec_env_init.cpp:199:17 apache#13 0x55e8b722cc8b in main /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/service/doris_main.cpp:531:14 apache#14 0x7f539b7cad8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into --> --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
suxiaogang223
pushed a commit
that referenced
this pull request
Apr 28, 2026
…pache#62759) Related PR: apache#58002 Problem Summary: `arrow::Decimal256Array::Value()` returns raw bytes (`const uint8_t*`), we previously reinterpret_cast it to `Decimal256*` which is a misaligned access . ```text ../src/core/data_type_serde/data_type_decimal_serde.cpp:376:21: runtime error: reference binding to misaligned address 0x1338f3345b12 for type 'const FieldType' (aka 'const Decimal<integer<256, int>>'), which requires 8 byte alignment 0x1338f3345b12: note: pointer points here 00 c2 3e 20 00 00 10 63 2d 5e c7 6b 05 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ #0 0x56203d1eb751 in doris::DataTypeDecimalSerDe<(doris::PrimitiveType)35>::read_column_from_arrow(doris::IColumn&, arrow::Array const*, long, long, cctz::time_zone const&) const be/build_ASAN/../src/core/data_type_serde/data_type_decimal_serde.cpp:375:25 #1 0x56203d9c6d6b in doris::DataTypeNullableSerDe::read_column_from_arrow(doris::IColumn&, arrow::Array const*, long, long, cctz::time_zone const&) const be/build_ASAN/../src/core/data_type_serde/data_type_nullable_serde.cpp:350:26 #2 0x562055dd83b5 in doris::FromRecordBatchToBlockConverter::convert(doris::Block*) be/build_ASAN/../src/format/arrow/arrow_block_convertor.cpp:131:9 #3 0x562055dd9229 in doris::convert_from_arrow_batch(std::shared_ptr<arrow::RecordBatch> const&, std::vector<std::shared_ptr<doris::IDataType const>, ```
suxiaogang223
pushed a commit
that referenced
this pull request
Aug 21, 2026
…66841) ### What problem does this PR solve? support two phase in distributed execution for lance vector_search. 1. lance_scan_node will query nearest with query_vector and prefilter in lance rust, will return _row_id, _distance and some postfilter column from lance scan result, and execute the where postfilter. 2. then local topn and global merge topn will get top_k rows. 3. MaterializeNode will use _row_id to fetch the payload project column from lance rust. ``` mysql> desc SELECT row_id, category, label, _distance FROM vector_search( "table"="test_lance_vector_search_two_phase.doris.vs_ivf_pq_f32", "column"="embedding", "query_vector"="[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15]", "top_k"="3", "metric"="l2", "nprobes"="4", "refine_factor"="10", "use_index"="true") WHERE category = 'odd' ORDER BY _distance; +-----------------------------------------------------------------------------------------------------------------------------------------------------------+ | Explain String(Nereids Planner) | +-----------------------------------------------------------------------------------------------------------------------------------------------------------+ | PLAN FRAGMENT 0 | | OUTPUT EXPRS: | | row_id[apache#17] | | category[apache#18] | | label[apache#19] | | _distance[apache#20] | | PARTITION: UNPARTITIONED | | | | HAS_COLO_PLAN_NODE: false | | | | VRESULT SINK | | MYSQL_PROTOCOL | | | | 4:VSORT(193) | | | order by: _distance[apache#20] ASC | | | algorithm: full sort | | | offset: 0 | | | distribute expr lists: | | | | | 3:VMaterializeNode | | | materialize tuple id:3 | | | output tuple id:4 | | | projectList:[row_id, category, label, _distance] | | | column_descs_lists[[`row_id` bigint NOT NULL, `label` text NOT NULL]] | | | locations: [[2, 3]] | | | column_idxs_lists: [[0, 2]] | | | row_ids: [__DORIS_GLOBAL_ROWID_COL__vector_search] | | | isTopMaterializeNode: true | | | final projections: row_id[apache#11], category[apache#9], label[apache#12], _distance[apache#10] | | | final project output tuple id: 4 | | | | | 2:VMERGING-EXCHANGE | | offset: 0 | | limit: 3 | | distribute expr lists: | | | | PLAN FRAGMENT 1 | | | | PARTITION: RANDOM | | | | HAS_COLO_PLAN_NODE: false | | | | STREAM DATA SINK | | EXCHANGE ID: 02 | | UNPARTITIONED | | IS_MERGE: true | | | | 1:VTOP-N(182) | | | order by: _distance[apache#7] ASC | | | algorithm: heap sort | | | local merge sort | | | merge by exchange | | | offset: 0 | | | limit: 3 | | | distribute expr lists: | | | | | 0:VLANCE_SCAN_NODE(174) | | table: null.null.VectorSearchTableValuedFunction<test_lance_vector_search_two_phase.doris.vs_ivf_pq_f32> | | predicates: (_tvf_vector_search.category[#0] = 'odd') | | inputSplitNum=2, totalFileSize=0, scanRanges=2 | | partition=0/0 | | numNodes=1 | | pushdown agg=NONE | | externalSearchType=VECTOR | | lanceVectorColumn=embedding | | lanceTopK=3 | | lanceOffset=0 | | lanceMetric=l2 | | lanceVersion=3 | | lanceSearchFragments=2 | | final projections: _tvf_vector_search.category[#0], _tvf_vector_search._distance[#1], _tvf_vector_search.__DORIS_GLOBAL_ROWID_COL__vector_search[#2] | | final project output tuple id: 1 | | | | | | | | ========== STATISTICS ========== | +-----------------------------------------------------------------------------------------------------------------------------------------------------------+ 76 rows in set (0.04 sec) mysql> ``` ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [x] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
suxiaogang223
pushed a commit
that referenced
this pull request
Sep 28, 2026
### What problem does this PR solve?
Issue Number: N/A
Problem Summary:
**Context.** `run-be-ut.sh` is the only entry point for the BE unit
tests, and on macOS it is also the only way to run them: the `BE UT
(macOS)` workflow runs `build.sh --be`, which configures with
`-DMAKE_TEST=OFF`, so nothing under `be/test` is compiled there (the
workflow says so in its own comment at
`.github/workflows/be-ut-mac.yml:123`). On a clean worktree the script
therefore fails twice before a single test executes, and then stops
seven more times while compiling `be/test`.
**1. The problem, and what it cost**
*The script never initializes two of the submodules it needs.*
`run-be-ut.sh` initializes three submodules (`contrib/datasketches-cpp`,
`contrib/apache-orc`, `contrib/clucene`); `build.sh` initializes five,
and the two it adds — `contrib/faiss` and `contrib/openblas` — are
required before a single file is compiled. `be/CMakeLists.txt:1038` adds
`be/src/storage/index/ann` unconditionally, and that directory's
`cmake-protect/CMakeLists.txt:48,66` calls `add_subdirectory()` on both.
On a fresh clone or a new git worktree the configure step therefore ends
with
CMake Error at src/storage/index/ann/cmake-protect/CMakeLists.txt:48
(add_subdirectory):
The source directory .../contrib/openblas does not contain a
CMakeLists.txt file.
CMake Error at src/storage/index/ann/cmake-protect/CMakeLists.txt:66
(add_subdirectory):
The source directory .../contrib/faiss does not contain a CMakeLists.txt
file.
Running `build.sh` once first hides it, which is why it goes unnoticed.
*Seven `be/test` translation units do not compile on macOS.* The Linux
BE UT build is green for all seven. `run-be-ut.sh` invokes ninja without
`-k`, so the developer fixes one file, waits out another full compile,
and meets the next one — seven round trips, each of them tens of
minutes, for a task whose whole point is to run a test.
| # | TU | Error | Root cause |
|---|---|---|---|
| 1 | `exprs/function/function_map_inner_product_test.cpp` |
`'faiss/impl/platform_macros.h' file not found` | The `APPLE` branch of
`be/test/CMakeLists.txt:170` links `vector_search_test` as
`-Wl,-force_load,$<TARGET_FILE:vector_search_test>`: a file path, which
carries no usage requirements. Every other platform links
`-Wl,--whole-archive vector_search_test`, which is a target in the link
list and does propagate them. |
| 2 | `exprs/function/function_format_round_test.cpp` | `ambiguous
conversion for functional-style cast from 'long' to 'Decimal64'` (11
sites) | `Decimal64` is `Decimal<int64_t>`, so on Darwin the constructor
parameter is `long long` while these literals are `long`; every integral
and floating constructor is then an equally ranked conversion away. |
| 3 | `storage/index/snii/bench/bkd_native_vs_clucene_bench_test.cpp` |
six `-Werror,-Wformat` | `uint64_t` is `unsigned long long` on Darwin
and `unsigned long` on Linux, so `%lu` / `%ld` is right on one and wrong
on the other. |
| 4 | `storage/index/snii/bench/snii_vs_v3_benchmark_test.cpp` | `no
member named 'posix_fadvise'`, `mincore` argument type | `posix_fadvise`
/ `POSIX_FADV_DONTNEED` do not exist on Darwin; `mincore(2)` takes
`char*` there and `unsigned char*` on Linux. |
| 5 | `storage/index/snii/bkd/bkd_corruption_test.cpp` | `deduced
conflicting types ('size_type' vs 'uint64_t')` | An initializer list
mixing `payload.size()` (size_t) with `uint64_t` literals. |
| 6 | `storage/index/snii/writer/spimi_locality_bench_test.cpp` | libc++
`no matching function for call to object of type 'key_equal'` | The
transparent equality functor defines one argument order only. libc++
calls `key_equal` as `(key, lookup key)`, libstdc++ as `(lookup key,
key)`. |
| 7 | `storage/segment/hierarchical_data_iterator_test.cpp` | three `no
matching function for call to 'min'` | `kRows` is `size_t` and
`current_ordinal` is `ordinal_t` (`uint64_t`), which are distinct types
on Darwin, so `std::min` cannot deduce a common type. |
None of the seven is related to a local change; a clean master
reproduces all of them. #2 arrived on 2026-09-24 with apache#68397 (`[fix](be)
Preserve decimal precision in format_round`, by TengJianPing), which is
the commit that added `function_format_round_test.cpp` — after the last
full macOS sweep. So this class of breakage is still being produced:
"green on Linux, red on macOS" is invisible to every CI job we have, and
the two obstacles above make the one manual path that would find it
expensive enough that nobody walks it.
**2. What this PR does, and why it helps**
- `run-be-ut.sh`: the `update_submodule` list gains `contrib/openblas`
and `contrib/faiss`, with the same archive URLs `build.sh:941` already
uses, plus a comment that the list has to stay in sync with `build.sh`.
- `be/test/CMakeLists.txt`: the `APPLE` branch links `ann_index` in
addition to the `force_load` flag. `ann_index` is the target that
exports the faiss include directories `PUBLIC`, so the APPLE link line
now feeds the same compile and link inputs as every other platform,
instead of being a macOS special case.
- Six test files: the smallest change that removes the platform
difference, each spelled the way the file's neighbours already spell it
— `PRIu64` / `PRId64` (`core/decimal12.h`, `util/histogram.cpp`),
`int64_t(...)` around the literals (`storage/key_coder_test.cpp:392`,
`runtime/jsonb_document_test.cpp:433`), `#if
defined(POSIX_FADV_DONTNEED)` around the call.
Linux behaviour is unchanged: every edit is either inside an
`APPLE`-only path or a strictly more portable spelling of the same
value. On macOS `run-be-ut.sh` becomes a usable test line for the first
time.
One cost worth naming: because the APPLE link line now propagates
include directories, the compile command of every `be/test` TU changes
once, so the first macOS build after this lands rebuilds `be/test`
(about 1230 TUs). Later builds are incremental again.
**Branch pick.** apache#68397 was never picked to `branch-4.1`, so
`function_format_round_test.cpp` does not exist there. Of the seven
files, only `storage/segment/hierarchical_data_iterator_test.cpp` exists
on `branch-4.1`, with the same three `std::min(*rows, ROWS -
current_ordinal)` sites and the same types (`ROWS` is `size_t`,
`ordinal_t` is `uint64_t`), so it fails to compile on macOS there too;
that branch also has no `APPLE` `force_load` branch in
`be/test/CMakeLists.txt`, so the faiss part is master-only. If 4.1 is
meant to run BE UT on macOS, the pick is that one file's hunk.
**3. The classes, and how they call each other**
- `run-be-ut.sh` (changed): `update_submodule()` calls `git submodule
update --init --recursive <path>` and falls back to the branch tarball;
the two added lines are identical in shape to the three that were
already there.
- `be/test/CMakeLists.txt` (changed): the `APPLE` branch of the
`doris_be_test` link step.
- `be/src/storage/index/ann/cmake-protect/CMakeLists.txt` (unchanged,
the reason the submodules are mandatory): `add_subdirectory()` on
`contrib/openblas` and `contrib/faiss`, reached unconditionally from
`be/CMakeLists.txt:1038`.
- `ann_index` (unchanged target in `be/src/storage/index/ann`): links
`faiss` `PUBLIC`, so its `INTERFACE_INCLUDE_DIRECTORIES` carry the faiss
include path.
- `vector_search_test` (unchanged target in
`be/test/storage/index/ann`): links `ann_index` `PUBLIC`; on APPLE it
reaches `doris_be_test` as forced-in objects rather than as a target.
- The seven test TUs: unchanged in behaviour, three of them compiled in
a portable spelling.
```
run-be-ut.sh
|- update_submodule contrib/{datasketches-cpp, apache-orc, clucene} [already there]
'- update_submodule contrib/{openblas, faiss} [added here]
'- git submodule update --init --recursive
be/CMakeLists.txt:1038 add_subdirectory(be/src/storage/index/ann)
'- storage/index/ann/CMakeLists.txt:18 add_subdirectory(cmake-protect)
'- cmake-protect/CMakeLists.txt:48,66 add_subdirectory(contrib/{openblas, faiss})
'- target faiss --(PUBLIC)--> target ann_index --(PUBLIC)--> vector_search_test
|
be/test/CMakeLists.txt:170 if (APPLE) |
|- before: -Wl,-force_load,<path to libvector_search_test.a> -- propagates nothing
'- after: the same flag + ann_index -- propagates the faiss include dirs
|
function_map_inner_product_test.cpp <--'
(and every other be/test TU, which now
compiles with the same inputs as on Linux)
be/test/CMakeLists.txt add_executable(doris_be_test ${UT_FILES}) <- the other six TUs, no link involvement
```
suxiaogang223
pushed a commit
that referenced
this pull request
Oct 13, 2026
…pache#68880) Issue Number: N/A Related PR: N/A Problem Summary: EXCEPT DISTINCT can introduce a streaming distinct aggregation whose grouping expressions repeat an input column, for example (s, d, d). When preaggregation switches to pass-through, both output construction and column reuse call assert_mutable() even though they only transfer column references. Retaining output references can make a repeated key shared, so a subsequent assertion fails with COW::assert_mutable: use_count() > 1: ``` W 2026-10-09 06:31:59,596 13803 exception.cpp:34] meet exception, error code: 6, message: COW::assert_mutable: use_count() > 1 0# doris::Exception::Exception(int, std::basic_string_view<char, std::char_traits<char> > const&) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/basic_string.h:239 1# doris::DistinctStreamingAggLocalState::_distinct_pre_agg_with_serialized_key(doris::Block*, doris::Block*) at ../src/core/cow.h:325 2# doris::DistinctStreamingAggOperatorX::push(doris::RuntimeState*, doris::Block*, bool) const at ../src/common/status.h:519 3# doris::StatefulOperatorX<doris::DistinctStreamingAggLocalState>::get_block_impl(doris::RuntimeState*, doris::Block*, bool*) at ../src/common/status.h:519 4# doris::OperatorXBase::get_block(doris::RuntimeState*, doris::Block*, bool*) at ../src/common/status.h:519 5# doris::OperatorXBase::get_block_after_projects(doris::RuntimeState*, doris::Block*, bool*) at ./be/build_RELEASE/../src/exec/operator/operator.cpp:0 6# doris::PipelineTask::execute(bool*) at ./be/src/exec/pipeline/pipeline_task.cpp:0 7# doris::TaskScheduler::_do_work(int) at ./be/src/exec/pipeline/task_scheduler.cpp:155 ``` Use immutable get_ptr() references in both pass-through paths. The evaluated key pointers preserve the original column when a repeated input position has already been replaced during column reuse, and downstream mutations continue to use copy-on-write. No column data is copied during pass-through. Add unit coverage with three references to one input slot for fresh and reused output blocks, transition from deduplication, retained output ownership, and nullable output. Add regression coverage for repeated keys in EXCEPT DISTINCT, nullable EXCEPT output, INTERSECT DISTINCT, and UNION DISTINCT. Fix queries with repeated projected columns failing with a COW assertion during distinct streaming aggregation. 根因是：**Distinct Streaming Aggregation 在透传列时调用了要求“列对象独占”的 `assert_mutable()`，而重复分组键会让多个输出位置共享同一个列对象，导致后续调用违反这个要求。** 下面结合修复前的代码，分别说明两条透传路径。 **1\. `assert_mutable()` 检查的是列对象的引用计数** COW::assert\_mutable() (be/src/core/cow.h:323) 的实现是： ``` MutablePtr assert_mutable() const { if (this->use_count() > 1) { throw Exception(ErrorCode::INTERNAL_ERROR, "COW::assert_mutable: use_count() > 1"); } return const_cast<COW*>(this)->get_ptr(); } ``` 它要求调用时 `use_count() <= 1`，直接返回可变指针，**不会自动复制共享的列对象**。 如果确实需要修改可能共享的列，应使用 IColumn::mutate() (be/src/core/column/column.h:618)：它通过 `shallow_mutate()` 在引用计数大于 1 时复制列，使修改发生在独立对象上。 这里的引用计数统计的是持有列对象的智能指针数量。例如： ``` in_block.column[1] ─┐ ├── 同一个日期列对象 C output.column[2] ───┘ C.use_count() = 2 ``` **2\. 重复投影如何变成重复输入列引用** 例如： ``` SELECT s, d, d FROM t1 EXCEPT DISTINCT SELECT s, d, d FROM t2; ``` 当优化器为集合操作的输入推导局部去重时，InferSetOperatorDistinct (fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/InferSetOperatorDistinct.java:60) 使用子节点输出构造分组键： ``` new LogicalAggregate<>( ImmutableList.copyOf(child.getOutput()), true, child) ``` 因此，生成的局部聚合可能保留重复槽位，例如： ``` group by: s[#1], d[#2], d[#2] ``` BE 在准备分组列 (be/src/exec/operator/distinct\_streaming\_aggregation\_operator.cpp:177)时，记录表达式执行结果： ``` RETURN_IF_ERROR(_probe_expr_ctxs[i]->execute(in_block, &result_column_id)); // 准备列…… key_columns[i] = in_block->get_by_position(result_column_id).column.get(); result_idxs[i] = result_column_id; ``` 假设输入 Block 只有 `s`、`d` 两列，最终得到： ``` in_block: [Cs, Cd] result_idxs: [ 0, 1, 1] key_columns: [Cs*, Cd*, Cd*] ``` 两个日期键指向同一个列对象。`key_columns` 保存的是裸指针，本身不增加引用计数。 当预聚合因内存限制或去重收益不足设置 `_stop_emplace_flag = true` 后，算子开始透传输入，进入下面的问题路径。 **3\. 首次构造输出：两个重复引用即可报错** 修复前，不复用输出列的透传代码是： ``` for (int i = 0; i < key_size; ++i) { if (_stop_emplace_flag) { columns_with_schema.emplace_back( key_columns[i]->assert_mutable(), /* type, name */); } } ``` 对应当前代码的输出构造位置 (be/src/exec/operator/distinct\_streaming\_aggregation\_operator.cpp:267)。 以 `[s, d, d]` 为例，假设日期列 `Cd` 开始只有输入 Block 持有： | 时刻 | 持有 `Cd` 的位置 | 稳定引用计数 | |---|---|---:| | 处理第一个 `d` 前 | 输入列 | 1 | | 第一个 `d` 加入输出后 | 输入列、第一个日期输出列 | 2 | | 处理第二个 `d` 时 | 再次调用 `Cd->assert_mutable()` | 2，抛异常 | 第一个 `d` 输出时增加了一个列引用，输入引用仍然存在。第二个 `d` 再要求独占，就会失败。 后面的 `in_block->clear()` 原本可以释放输入引用，但异常已经在循环内发生，执行不到那里。 **4\. 复用输出 Block：三个重复引用能展示同样的问题** 修复前，复用输出列的透传位置 (be/src/exec/operator/distinct\_streaming\_aggregation\_operator.cpp:235) 是： ``` for (int i = 0; i < key_size; ++i) { auto output_column = out_block->get_by_position(i).column; out_block->replace_by_position(i, key_columns[i]->assert_mutable()); in_block->replace_by_position(result_idxs[i], output_column); } ``` 这个分支将旧输出列放回输入 Block，释放输入对原数据列的引用。 以 `[s, d, d, d]` 为例： ``` result_idxs = [0, 1, 1, 1] ``` 假设输出列最初为空，且没有其他所有者： | 时刻 | 原日期列 `Cd` 的所有者 | 稳定引用计数 | |---|---|---:| | 第一个 `d` 处理结束 | 第一个日期输出列；输入位置已换成旧输出空列 | 1 | | 第二个 `d` 处理结束 | 第一个、第二个日期输出列 | 2 | | 第三个 `d` 开始处理 | 调用 `Cd->assert_mutable()` | 2，抛异常 | 第二次替换的输入位置已经是空列，无法再释放原日期列的引用。于是两个输出列共同持有 `Cd`，第三次独占断言失败。 因此，**复用分支中只有两个相同槽位不一定立即报错**；已有单测使用 `[0, 1, 0, 0]`，让同一输入槽位出现三次，覆盖这一过程。 **5\. 为什么改为 `get_ptr()` 能解决** 两条透传路径现在都使用： ``` key_columns[i]->get_ptr() ``` `key_columns` 的类型是 `std::vector<const IColumn*>`，因此这里调用的是 const 版本的 `get_ptr()` (be/src/core/cow.h:304)，返回不可变列指针。 透传只需要让输出引用已有列数据，允许多个输出位置共享该列。后续需要修改数据时，仍通过 `IColumn::mutate()` 执行 COW。 复用分支继续从预先记录的 `key_columns` 取指针也很关键：第一次替换后，`in_block[result_idxs[i]]` 已经变成旧输出列，而 `key_columns[i]` 仍指向原数据列，并由前面的输出位置保持其生命周期。 关于 GROUP BY，直接的 `GROUP BY s, d, d` 会在 NormalizeAggregate (fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/NormalizeAggregate.java:167) 中去重。静态分析发现，`GROUP BY + LIMIT` 下推到 `UNION ALL` 分支时，槽位映射可能重新引入重复键；已在回归用例 (regression-test/suites/query\_p0/set\_operations/distinct\_streaming\_repeated\_keys.groovy:101)中补充直接分组对照和该候选场景，并检查计划中同一槽位是否出现三次。提交为 `c8c1f67d3e9`，按你的要求未执行新增用例，因此尚未实测确认该 GROUP BY 场景。 - Test: Unit Test / Regression test / Manual test - ASAN build with ./build.sh --be -j 12 - 11 BE unit tests covering distinct streaming aggregation - Four regression queries; results also match non-streaming execution - Original reproducing SQL on the supplied table data - clang-format 16, format check, build hygiene, and clang-tidy on changed lines - Behavior changed: Yes; repeated grouping keys pass through without an incorrect ownership assertion - Does this need documentation: No ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.