[fix]handle loading iceberg partitions by auth by zhaorongsheng · Pull Request #1 · zhaorongsheng/doris

zhaorongsheng · 2025-11-13T09:28:06Z

Wrap loading of iceberg partitions in try-catch block to handle exceptions and log errors.

What problem does this PR solve?

Issue Number: close apache#57987

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Wrap loading of iceberg partitions in try-catch block to handle exceptions and log errors.

…e#56935) ``` (gdb) bt #0 doris::cloud::FdbTxnKv::get_partition_boundaries (this=0x7f11ac6e3950, boundaries=0x7f1130017f20) at /home/zcp/repo_center/doris_branch-4.0/doris/cloud/src/meta-store/txn_kv.cpp:212 #1 0x000055e494de9276 in doris::cloud::export_fdb_kv_ranges_details (kv=0x7f11ac6e3950) at /home/zcp/repo_center/doris_branch-4.0/doris/cloud/src/common/metric.cpp:358 apache#2 doris::cloud::FdbMetricExporter::export_fdb_metrics (txn_kv=0x7f11ac6e3950) at /home/zcp/repo_center/doris_branch-4.0/doris/cloud/src/common/metric.cpp:393 apache#3 0x000055e494dec77d in doris::cloud::FdbMetricExporter::start()::$_0::operator()() const (this=<optimized out>) at /home/zcp/repo_center/doris_branch-4.0/doris/cloud/src/common/metric.cpp:415 apache#4 std::__invoke_impl<void, doris::cloud::FdbMetricExporter::start()::$_0>(std::__invoke_other, doris::cloud::FdbMetricExporter::start()::$_0&&) (__f=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:63 apache#5 std::__invoke<doris::cloud::FdbMetricExporter::start()::$_0>(doris::cloud::FdbMetricExporter::start()::$_0&&) (__fn=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:98 apache#6 std::thread::_Invoker<std::tuple<doris::cloud::FdbMetricExporter::start()::$_0> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=<optimized out>) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_thread.h:303 apache#7 std::thread::_Invoker<std::tuple<doris::cloud::FdbMetricExporter::start()::$_0> >::operator()() (this=<optimized out>) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_thread.h:310 apache#8 std::thread::_State_impl<std::thread::_Invoker<std::tuple<doris::cloud::FdbMetricExporter::start()::$_0> > >::_M_run() (this=<optimized out>) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_thread.h:255 apache#9 0x000055e495e547b0 in execute_native_thread_routine () apache#10 0x00007f11b1dfeac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 apache#11 0x00007f11b1e90850 in __closefrom_fallback (from=-1266337808, dirfd_fallback=<optimized out>) at ../sysdeps/unix/sysv/linux/closefrom_fallback.c:45 apache#12 0x0000000000000000 in ?? () ```

…56755) ### What problem does this PR solve? ```text #0 __GI___pthread_sigmask (how=2, newmask=<optimized out>, oldmask=0x0) at ./nptl/pthread_sigmask.c:43 #1 0x00007f91fd6c471e in PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] () from /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so apache#2 0x00007f91fd6c5206 in JVM_handle_linux_signal () from /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so apache#3 <signal handler called> apache#4 __gnu_cxx::__exchange_and_add (__mem=0xe, __val=-1) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/ext/atomicity.h:68 apache#5 __gnu_cxx::__exchange_and_add_dispatch (__mem=0xe, __val=-1) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/ext/atomicity.h:103 apache#6 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_weak_release (this=0x2) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:211 apache#7 std::__weak_count<(__gnu_cxx::_Lock_policy)2>::~__weak_count (this=<optimized out>) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:1168 apache#8 std::__weak_ptr<doris::QueryContext, (__gnu_cxx::_Lock_policy)2>::~__weak_ptr (this=<optimized out>) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:2003 apache#9 doris::QueryTaskController::revoke_memory()::$_1::~$_1() (this=<optimized out>) at /root/doris/be/src/runtime/workload_management/query_task_controller.cpp:168 apache#10 std::_Function_base::_Base_manager<doris::QueryTaskController::revoke_memory()::$_1>::_M_destroy(std::_Any_data&, std::integral_constant<bool, false>) (__victim=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:177 apache#11 std::_Function_base::_Base_manager<doris::QueryTaskController::revoke_memory()::$_1>::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) (__dest=..., __op=std::__destroy_functor, __source=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:205 apache#12 std::_Function_handler<void (doris::pipeline::SpillContext*), doris::QueryTaskController::revoke_memory()::$_1>::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) (__dest=..., __source=..., __op=std::__destroy_functor) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:284 apache#13 0x000055f37fba60c5 in std::_Function_base::~_Function_base (this=0x7f8dbb0c9fe0) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:246 apache#14 doris::pipeline::SpillContext::~SpillContext (this=0x7f91536f0150) at /root/doris/be/src/pipeline/exec/spill_utils.h:57 apache#15 0x000055f384259e4b in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x7f91536f0140) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:345 apache#16 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::operator= (this=0x7f8d61302518, __r=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:1088 apache#17 std::__shared_ptr<doris::pipeline::SpillContext, (__gnu_cxx::_Lock_policy)2>::operator= (this=0x7f8d61302510) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:1530 apache#18 std::shared_ptr<doris::pipeline::SpillContext>::operator= (this=0x7f8d61302510) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr.h:413 apache#19 doris::pipeline::PipelineTask::revoke_memory (this=0x7f8d61302490, spill_context=...) at /root/doris/be/src/pipeline/pipeline_task.cpp:812 apache#20 0x000055f37fba3c2d in doris::QueryTaskController::revoke_memory (this=<optimized out>) at /root/doris/be/src/runtime/workload_management/query_task_controller.cpp:185 apache#21 0x000055f37fb99d08 in doris::WorkloadGroupMgr::handle_single_query_ (this=<optimized out>, requestor=..., size_to_reserve=size_to_reserve@entry=1024000, time_in_queue=time_in_queue@entry=27, paused_reason=...) at /root/doris/be/src/runtime/workload_group/workload_group_manager.cpp:820 apache#22 0x000055f37fb981d0 in doris::WorkloadGroupMgr::handle_paused_queries (this=0x7f9141d8a800) at /root/doris/be/src/runtime/workload_group/workload_group_manager.cpp:381 apache#23 0x000055f37eb97297 in doris::Daemon::memory_maintenance_thread (this=0x7ffe63cad730) at /root/doris/be/src/common/daemon.cpp:354 apache#24 0x000055f37fd812fc in std::function<void ()>::operator()() const (this=0x7f8dbb0c9fe0) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593 apache#25 doris::Thread::supervise_thread (arg=0x7f914ba25e10) at /root/doris/be/src/util/thread.cpp:460 apache#26 0x00007f91fc75fac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 apache#27 0x00007f91fc7f1850 in __closefrom_fallback (from=1674236160, dirfd_fallback=<optimized out>) at ../sysdeps/unix/sysv/linux/closefrom_fallback.c:45 apache#28 0x0000000000000000 in ?? () ``` Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test  - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [ ] No. - [ ] Yes.  - Does this need documentation? - [ ] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

…icate (apache#57100) ### What problem does this PR solve? ```text #0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=140649357411904) at ./nptl/pthread_kill.c:44 #1 __pthread_kill_internal (signo=6, threadid=140649357411904) at ./nptl/pthread_kill.c:78 apache#2 __GI___pthread_kill (threadid=140649357411904, signo=signo@entry=6) at ./nptl/pthread_kill.c:89 apache#3 0x00007feca2a42476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26 apache#4 0x00007feca2a287f3 in __GI_abort () at ./stdlib/abort.c:79 apache#5 0x000055cd63341b52 in __gnu_cxx::__verbose_terminate_handler() () apache#6 0x000055cd6333fe26 in __cxxabiv1::__terminate(void (*)()) () apache#7 0x000055cd6333fe79 in std::terminate() () apache#8 0x000055cd6333ffc3 in __cxa_throw () apache#9 0x000055cd6339556a in std::__throw_out_of_range(char const*) () apache#10 0x000055cd5a9a7a68 in __gnu_cxx::__stoa<double, double, char> (__str=0x7fe842726fc0 "9.", '9' <repeats 15 times>, "1e-309", __idx=0x0, __convf=<optimized out>, __name=<optimized out>) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/ext/string_conversions.h:90 apache#11 std::__cxx11::stod (__str=..., __idx=0x0) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/basic_string.h:4490 apache#12 doris::IntegerPredicateCreator<(doris::PrimitiveType)9, (doris::PredicateType)1, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::convert (condition=...) at /root/doris_branch-4.0/doris/be/src/olap/predicate_creator.h:68 apache#13 doris::IntegerPredicateCreator<(doris::PrimitiveType)9, (doris::PredicateType)1, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::create (this=<optimized out>, column=..., index=1, conditions=..., opposite=false, arena=...) at /root/doris_branch-4.0/doris/be/src/olap/predicate_creator.h:59 apache#14 0x000055cd5a3dd63b in doris::create_predicate<(doris::PredicateType)1, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > (column=..., index=1, conditions=..., opposite=false, arena=...) at /root/doris_branch-4.0/doris/be/src/olap/predicate_creator.h:270 apache#15 doris::create_comparison_predicate<(doris::PredicateType)1> (column=..., index=1, condition=..., opposite=false, arena=...) at /root/doris_branch-4.0/doris/be/src/olap/predicate_creator.h:278 apache#16 0x000055cd5a3dd435 in doris::parse_to_predicate (column=..., index=1, condition=..., arena=..., opposite=false) at /root/doris_branch-4.0/doris/be/src/olap/predicate_creator.h:326 apache#17 0x000055cd5afc2759 in doris::TabletReader::_init_conditions_param (this=this@entry=0x7fe842e1a800, read_params=...) at /root/doris_branch-4.0/doris/be/src/olap/tablet_reader.cpp:549 apache#18 0x000055cd5afc1022 in doris::TabletReader::_init_params (this=this@entry=0x7fe842e1a800, read_params=...) at /root/doris_branch-4.0/doris/be/src/olap/tablet_reader.cpp:304 apache#19 0x000055cd5afbfe00 in doris::TabletReader::init (this=0x7fe842e1a800, read_params=...) at /root/doris_branch-4.0/doris/be/src/olap/tablet_reader.cpp:127 apache#20 0x000055cd5f019d0f in doris::vectorized::BlockReader::init (this=0x4c1b, read_params=...) at /root/doris_branch-4.0/doris/be/src/vec/olap/block_reader.cpp:206 apache#21 0x000055cd5f79a3f4 in doris::vectorized::OlapScanner::open (this=0x7febe2c16010, state=<optimized out>) at /root/doris_branch-4.0/doris/be/src/vec/exec/scan/olap_scanner.cpp:278 apache#22 0x000055cd5e05303a in doris::vectorized::ScannerScheduler::_scanner_scan (ctx=..., scan_task=...) at /root/doris_branch-4.0/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:182 apache#23 0x000055cd5e0549e5 in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()#1}::operator()() const::{lambda()#1}::operator()() const (this=<optimized out>) at /root/doris_branch-4.0/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:96 apache#24 doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()#1}::operator()() const (this=0x7febe321dac0) at /root/doris_branch-4.0/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:95 apache#25 std::__invoke_impl<bool, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()#1}&>(std::__invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()#1}&) (__f=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:63 apache#26 std::__invoke_r<bool, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()#1}&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()#1}&) (__fn=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:116 apache#27 std::_Function_handler<bool (), doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()#1}>::_M_invoke(std::_Any_data const&) ( __functor=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:292 apache#28 0x000055cd5b4ab5d5 in doris::ThreadPool::dispatch_thread (this=0x7fec35e01800) at /root/doris_branch-4.0/doris/be/src/util/threadpool.cpp:614 apache#29 0x000055cd5b4a06fc in std::function<void ()>::operator()() const (this=0x4a17) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593 apache#30 doris::Thread::supervise_thread (arg=0x7fec35872110) at /root/doris_branch-4.0/doris/be/src/util/thread.cpp:460 apache#31 0x00007feca2a94ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 apache#32 0x00007feca2b26850 in __closefrom_fallback (from=-823527648, dirfd_fallback=<optimized out>) at ../sysdeps/unix/sysv/linux/closefrom_fallback.c:45 apache#33 0x0000000000000000 in ?? () ``` Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test  - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [ ] No. - [ ] Yes.  - Does this need documentation? - [ ] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

### What problem does this PR solve? ```text #0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=136182904088128) at ./nptl/pthread_kill.c:44 #1 __pthread_kill_internal (signo=6, threadid=136182904088128) at ./nptl/pthread_kill.c:78 apache#2 __GI___pthread_kill (threadid=136182904088128, signo=signo@entry=6) at ./nptl/pthread_kill.c:89 apache#3 0x00007fdee8ffb476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26 apache#4 0x00007fdee8fe17f3 in __GI_abort () at ./stdlib/abort.c:79 apache#5 0x000055cc31fb9bcd in ?? () apache#6 0x000055cc31fadeac in google::LogMessage::SendToLog() () apache#7 0x000055cc31fae538 in google::LogMessage::Flush() () apache#8 0x000055cc31fb2116 in google::LogMessageFatal::~LogMessageFatal() () apache#9 0x000055cc1977f798 in doris::vectorized::PODArray<unsigned int, 4096ul, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, false>, 16ul, 15ul>::operator[] (this=<optimized out>, n=2) at /root/doris/be/src/vec/common/pod_array.h:466 apache#10 0x000055cc19777580 in doris::vectorized::ColumnStr<unsigned int>::size_at (this=0x7c5eea94dda0, i=2) at /root/doris/be/src/vec/columns/column_string.h:96 apache#11 doris::vectorized::ColumnStr<unsigned int>::compare_at (this=0x7c5eea94dda0, n=2, m=1, rhs_=...) at /root/doris/be/src/vec/columns/column_string.h:482 apache#12 0x000055cc2f2f1138 in doris::vectorized::VCollectIterator::_topn_next (this=<optimized out>, block=<optimized out>) at /root/doris/be/src/vec/olap/vcollect_iterator.cpp:343 apache#13 0x000055cc2f2efa08 in doris::vectorized::VCollectIterator::next (this=0x898f, block=0x6) at /root/doris/be/src/vec/olap/vcollect_iterator.cpp:247 apache#14 0x000055cc2f28e3ea in doris::vectorized::BlockReader::_direct_next_block (this=0x7dbeed736880, block=<optimized out>, eof=<optimized out>) at /root/doris/be/src/vec/olap/block_reader.cpp:267 apache#15 0x000055cc2f28778b in doris::vectorized::BlockReader::next_block_with_aggregation (this=0x7dbeed736880, block=<optimized out>, eof=<optimized out>) at /root/doris/be/src/vec/olap/block_reader.cpp:66 apache#16 0x000055cc2b33300e in doris::vectorized::OlapScanner::_get_block_impl (this=0x7daeee690690, state=<optimized out>, block=0x7daeee6907b8, eof=<optimized out>) at /root/doris/be/src/vec/exec/scan/olap_scanner.cpp:584 apache#17 0x000055cc2b2c990d in doris::vectorized::Scanner::get_block (this=<optimized out>, state=<optimized out>, block=0x7daeee6907b8, eof=<optimized out>) at /root/doris/be/src/vec/exec/scan/scanner.cpp:116 apache#18 0x000055cc2b2c8fa0 in doris::vectorized::Scanner::get_block_after_projects (this=0x7daeee690690, state=0x7dcee9e6ec80, block=0x7caee9b74fe0, eos=0x7bdb809a11f0) at /root/doris/be/src/vec/exec/scan/scanner.cpp:82 apache#19 0x000055cc2b2e51b4 in doris::vectorized::ScannerScheduler::_scanner_scan (ctx=..., scan_task=...) at /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:182 apache#20 0x000055cc2b2ed92c in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()#1}::operator()() const::{lambda()#1}::operator()() const (this=<optimized out>) at /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:96 apache#21 doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()#1}::operator()() const (this=0x7c0eeec42540) at /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:95 apache#22 std::__invoke_impl<bool, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()#1}&>(std::__invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()#1}&) (__f=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:63 apache#23 std::__invoke_r<bool, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()#1}&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()#1}&) (__fn=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:116 apache#24 std::_Function_handler<bool (), doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()#1}>::_M_invoke(std::_Any_data const&) ( __functor=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:292 apache#25 0x000055cc1dd09d33 in doris::ThreadPool::dispatch_thread (this=0x7d3ee8680f80) at /root/doris/be/src/util/threadpool.cpp:614 apache#26 0x000055cc1dce6da7 in std::function<void ()>::operator()() const (this=<optimized out>) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593 apache#27 doris::Thread::supervise_thread (arg=<optimized out>) at /root/doris/be/src/util/thread.cpp:460 apache#28 0x000055cc19364d27 in asan_thread_start(void*) () apache#29 0x00007fdee904dac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 apache#30 0x00007fdee90df850 in __closefrom_fallback (from=877593024, dirfd_fallback=<optimized out>) at ../sysdeps/unix/sysv/linux/closefrom_fallback.c:45 apache#31 0x0000000000000000 in ?? () ``` Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test  - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [ ] No. - [ ] Yes.  - Does this need documentation? - [ ] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

…gments from a single rowset during heavy schema change (apache#57191) ### What problem does this PR solve? Problem Summary: ``` *** Query id: 0-0 *** *** is nereids: 0 *** *** tablet id: 1751140494462 *** *** Aborted at 1751208065 (unix time) try "date -d @1751208065" if you are using GNU date *** *** Current BE git commitID: 34d8342 *** *** SIGSEGV invalid permissions for mapped object (@0x556ef4dfc000) received by PID 4975 (TID 5587 OR 0x7f71da3c4640) from PID 18446744073522888704; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/common/signal_handler.h:421 1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so 2# JVM_handle_linux_signal in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so 3# 0x00007F731FC1F520 in /lib/x86_64-linux-gnu/libc.so.6 4# doris::vectorized::VMergeIteratorContext::copy_rows(doris::vectorized::Block*, bool) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/olap/vgeneric_iterators.cpp:148 5# doris::Status doris::vectorized::VMergeIterator::_next_batch<doris::vectorized::Block>(doris::vectorized::Block*) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/olap/vgeneric_iterators.h:242 6# doris::vectorized::VMergeIterator::next_batch(doris::vectorized::Block*) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/olap/vgeneric_iterators.h:201 7# doris::BetaRowsetReader::next_block(doris::vectorized::Block*) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/beta_rowset_reader.cpp:378 8# doris::VSchemaChangeDirectly::_inner_process(std::shared_ptr<doris::RowsetReader>, doris::RowsetWriter*, std::shared_ptr<doris::BaseTablet>, std::shared_ptr<doris::TabletSchema>, std::shared_ptr<doris::TabletSchema>) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/schema_change.cpp:523 9# doris::SchemaChange::process(std::shared_ptr<doris::RowsetReader>, doris::RowsetWriter*, std::shared_ptr<doris::BaseTablet>, std::shared_ptr<doris::BaseTablet>, std::shared_ptr<doris::TabletSchema>, std::shared_ptr<doris::TabletSchema>) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/schema_change.h:120 10# doris::SchemaChangeJob::_convert_historical_rowsets(doris::SchemaChangeParams const&, long*) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/schema_change.cpp:1179 11# doris::SchemaChangeJob::_do_process_alter_tablet(doris::TAlterTabletReqV2 const&) in /mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be 12# doris::SchemaChangeJob::process_alter_tablet(doris::TAlterTabletReqV2 const&) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/schema_change.cpp:759 13# doris::alter_tablet_callback(doris::StorageEngine&, doris::TAgentTaskRequest const&) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/agent/task_worker_pool.cpp:1990 14# std::_Function_handler<void (), doris::TaskWorkerPool::submit_task(doris::TAgentTaskRequest const&)::$_0::operator()<doris::TAgentTaskRequest const&>(doris::TAgentTaskRequest const&) const::{lambda()#1}>::_M_invoke(std::_Any_data const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291 15# doris::ThreadPool::dispatch_thread() at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/util/threadpool.cpp:609 16# doris::Thread::supervise_thread(void*) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/util/thread.cpp:499 17# start_thread at ./nptl/pthread_create.c:442 18# 0x00007F731FD03850 at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:83 ``` Schema of input block from base tablet should merge dropped column.

…ntains window function (apache#55066) ### What problem does this PR solve? Support window function rewrite when materialized view contains window function Such as mv def is as following: CREATE MATERIALIZED VIEW mv1 BUILD IMMEDIATE REFRESH COMPLETE ON MANUAL DISTRIBUTED BY RANDOM BUCKETS 2 PROPERTIES ('replication_num' = '1') AS select * from ( select o_orderkey, FIRST_VALUE(o_custkey) OVER ( PARTITION BY o_orderdate ORDER BY o_totalprice NULLS LAST RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW ) AS first_value, RANK() OVER ( PARTITION BY o_orderdate, o_orderstatus ORDER BY o_totalprice NULLS LAST RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW ) AS rank_value, LAG(l_extendedprice, 1, 0) over (partition by o_orderdate, l_shipdate order by l_quantity) AS lag_value from lineitem2 left join orders2 on l_orderkey = o_orderkey and l_shipdate = o_orderdate ) t where o_orderkey > 1; if query as following, this can use mv to represent query select * from ( select o_orderkey, FIRST_VALUE(o_custkey) OVER ( PARTITION BY o_orderdate ORDER BY o_totalprice NULLS LAST RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW ) AS first_value, RANK() OVER ( PARTITION BY o_orderdate, o_orderstatus ORDER BY o_totalprice NULLS LAST RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW ) AS rank_value, LAG(l_extendedprice, 1, 0) over (partition by o_orderdate, l_shipdate order by l_quantity) AS lag_value from lineitem2 left join orders2 on l_orderkey = o_orderkey and l_shipdate = o_orderdate ) t where o_orderkey > 2; explain result is as follwing: +----------------------------------------------------------------------------------------+ | Explain String(Nereids Planner) | +----------------------------------------------------------------------------------------+ | PLAN FRAGMENT 0 | | OUTPUT EXPRS: | | o_orderkey[apache#4] | | first_value[apache#5] | | rank_value[apache#6] | | lag_value[apache#7] | | PARTITION: RANDOM | | | | HAS_COLO_PLAN_NODE: false | | | | VRESULT SINK | | MYSQL_PROTOCAL | | | | 0:VOlapScanNode(445) | | TABLE: regression_test_nereids_rules_p0_mv_window.mv1(mv1), PREAGGREGATION: ON | | PREDICATES: (o_orderkey[#0] > 2) | | partitions=1/1 (mv1) | | tablets=2/2, tabletList=1755678381149,1755678381151 | | cardinality=1, avgRowSize=0.0, numNodes=1 | | pushAggOp=NONE | | final projections: o_orderkey[#0], first_value[#1], rank_value[apache#2], lag_value[apache#3] | | final project output tuple id: 1 | | | | | | ========== MATERIALIZATIONS ========== | | | | MaterializedView | | MaterializedViewRewriteSuccessAndChose: | | internal.regression_test_nereids_rules_p0_mv_window.mv1 chose | | | | MaterializedViewRewriteSuccessButNotChose: | | | | MaterializedViewRewriteFail: | | | | | | ========== STATISTICS ========== | | planed with unknown column statistics | +----------------------------------------------------------------------------------------+

…ich belongs to an agg materialized view (apache#58038) ### What problem does this PR solve? Issue Number: close apache#58037 Problem Summary: ``` #0 0x00007f9aca4a3f8c in __pthread_kill_implementation () from /lib64/libc.so.6 #1 0x00007f9aca454a26 in raise () from /lib64/libc.so.6 apache#2 0x00007f9aca43d87c in abort () from /lib64/libc.so.6 apache#3 0x0000561dc3d1ea1d in ?? () apache#4 0x0000561dc3d1105a in google::LogMessage::Fail() () apache#5 0x0000561dc3d14146 in google::LogMessage::SendToLog() () apache#6 0x0000561dc3d10b90 in google::LogMessage::Flush() () apache#7 0x0000561dc3d14989 in google::LogMessageFatal::~LogMessageFatal() () apache#8 0x0000561db854c996 in assert_cast<doris::vectorized::ColumnStr<unsigned int> const&, (TypeCheckOnRelease)1, doris::vectorized::IColumn const&>(doris::vectorized::IColumn const&)::{lambda(auto:1&&)#1}::operator()<doris::vectorized::IColumn const&>(doris::vectorized::IColumn const&) const (this=this@entry=0x7f9658ccc1f8, from=...) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/common/assert_cast.h:58 apache#9 0x0000561db854c7d7 in assert_cast<doris::vectorized::ColumnStr<unsigned int> const&, (TypeCheckOnRelease)1, doris::vectorized::IColumn const&> (from=...) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/common/assert_cast.h:73 apache#10 0x0000561db854bb0b in doris::vectorized::ColumnStr<unsigned int>::compare_at (this=0x7f957a14e2c0, n=1159288, m=6, rhs_=...) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/columns/column_string.h:526 apache#11 0x0000561dbe108c6b in doris::vectorized::GenericComparisonImpl<doris::vectorized::EqualsOp<int, int> >::vector_constant (a=..., b=..., c=...) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/functions/functions_comparison.h:112 apache#12 doris::vectorized::FunctionComparison<doris::vectorized::EqualsOp, doris::vectorized::NameEquals>::execute_generic_identical_types ( this=<optimized out>, block=..., result=result@entry=10, c0=0x7f957a14e2c0, c1=<optimized out>) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/functions/functions_comparison.h:506 apache#13 0x0000561dbdf9e97e in doris::vectorized::FunctionComparison<doris::vectorized::EqualsOp, doris::vectorized::NameEquals>::execute_generic ( this=0x7f96d6fb1b90, block=..., result=10, c0=..., c1=...) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/functions/functions_comparison.h:517 apache#14 doris::vectorized::FunctionComparison<doris::vectorized::EqualsOp, doris::vectorized::NameEquals>::execute_impl (this=0x7f96d6fb1b90, context=<optimized out>, block=..., arguments=..., result=10, input_rows_count=104) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/functions/functions_comparison.h:707 apache#15 0x0000561dbdcf1b8f in doris::vectorized::DefaultExecutable::execute_impl (this=<optimized out>, context=0x6, block=..., arguments=..., result=1, input_rows_count=104) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/functions/function.h:472 apache#16 0x0000561dbeea76ae in doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal (this=this@entry=0x7f99f62a65d0, context=context@entry=0x7f99f6442b00, block=..., args=..., result=result@entry=10, input_rows_count=104, dry_run=<optimized out>) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/functions/function.cpp:121 apache#17 0x0000561dbeea4ce8 in doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns (this=0x7f99f62a65d0, context=0x7f99f6442b00, block=..., args=..., result=10, input_rows_count=104, dry_run=<optimized out>) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/functions/function.cpp:246 apache#18 doris::vectorized::PreparedFunctionImpl::default_implementation_for_nulls (this=this@entry=0x7f99f62a65d0, context=context@entry=0x7f99f6442b00, block=..., args=..., result=result@entry=10, input_rows_count=104, dry_run=<optimized out>, executed=0x7f9658ccc666) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/functions/function.cpp:218 apache#19 0x0000561dbeea4e9c in doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal (this=0x7f99f62a65d0, context=0x7f99f6442b00, block=..., args=..., result=10, input_rows_count=<optimized out>, dry_run=<optimized out>) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/functions/function.cpp:112 apache#20 doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns (this=0x7f99f62a65d0, context=0x7f99f6442b00, block=..., args=..., result=10, input_rows_count=104, dry_run=<optimized out>) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/functions/function.cpp:246 apache#21 0x0000561dbeea4f66 in doris::vectorized::PreparedFunctionImpl::execute (this=0x11b078, context=0x6, block=..., args=..., result=1, input_rows_count=104, dry_run=<optimized out>) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/functions/function.cpp:252 apache#22 0x0000561dbdcf1500 in doris::vectorized::IFunctionBase::execute (this=<optimized out>, context=0x7f99f6442b00, block=..., arguments=..., result=10, input_rows_count=104, dry_run=<optimized out>) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/functions/function.h:195 --Type <RET> for more, q to quit, c to continue without paging--c apache#23 0x0000561dbdceccad in doris::vectorized::VectorizedFnCall::_do_execute (this=0x7f96f0fec510, context=0x7f957f09cdf0, block=0x7f957b02a3b0, result_column_id=0x7f9658ccca14, args=...) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/exprs/vectorized_fn_call.cpp:197 apache#24 0x0000561dbdced2c6 in doris::vectorized::VectorizedFnCall::execute (this=0x11b078, context=0x6, block=0x7f9aca4a3f8c <__pthread_kill_implementation+268>, result_column_id=0x7f9658ccbe10) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/exprs/vectorized_fn_call.cpp:212 apache#25 0x0000561dbdd1e51b in doris::vectorized::VExprContext::execute (this=0x7f957f09cdf0, block=0x7f9aca4a3f8c <__pthread_kill_implementation+268>, block@entry=0x7f957b02a3b0, result_column_id=result_column_id@entry=0x7f9658ccca14) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/exprs/vexpr_context.cpp:55 apache#26 0x0000561dbdd1fcb5 in doris::vectorized::VExprContext::execute_conjuncts (ctxs=..., filters=filters@entry=0x0, accept_null=false, block=block@entry=0x7f957b02a3b0, result_filter=result_filter@entry=0x7f9658ccccc0, can_filter_all=0x7f9658cccbc7) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/exprs/vexpr_context.cpp:174 apache#27 0x0000561dbdd2131f in doris::vectorized::VExprContext::execute_conjuncts_and_filter_block (ctxs=..., block=0x7f957b02a3b0, columns_to_filter=..., column_to_keep=6, filter=...) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/exprs/vexpr_context.cpp:354 apache#28 0x0000561db8f49450 in doris::segment_v2::SegmentIterator::_execute_common_expr (this=this@entry=0x7f954e20a000, sel_rowid_idx=0x7f955c0d2000, selected_size=@0x7f9658ccce6e: 104, block=block@entry=0x7f957b02a3b0) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2338 apache#29 0x0000561db8f482f8 in doris::segment_v2::SegmentIterator::_next_batch_internal (this=0x7f954e20a000, block=0x7f957b02a3b0) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2230 apache#30 0x0000561db8f45212 in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*)::$_0::operator()() const ( this=<optimized out>) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:1953 apache#31 doris::segment_v2::SegmentIterator::next_batch (this=0x7f954e20a000, block=0x6) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:1952 apache#32 0x0000561db8ee49bc in doris::segment_v2::LazyInitSegmentIterator::next_batch (this=0x7f953bc71f80, block=0x7f957b02a3b0) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/olap/rowset/segment_v2/lazy_init_segment_iterator.h:44 apache#33 0x0000561db8dab844 in doris::BetaRowsetReader::next_block (this=0x7f9a4e215800, block=0x7f957b02a3b0) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/olap/rowset/beta_rowset_reader.cpp:377 apache#34 0x0000561dc2c9413d in doris::vectorized::VCollectIterator::Level0Iterator::_refresh (this=0x7f953ba137a0) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/olap/vcollect_iterator.h:256 apache#35 doris::vectorized::VCollectIterator::Level0Iterator::refresh_current_row (this=this@entry=0x7f953ba137a0) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/olap/vcollect_iterator.cpp:509 apache#36 0x0000561dc2c93bf4 in doris::vectorized::VCollectIterator::Level0Iterator::init (this=0x7f953ba137a0, get_data_by_ref=<optimized out>) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/olap/vcollect_iterator.cpp:461 apache#37 0x0000561dc2c91002 in doris::vectorized::VCollectIterator::build_heap (this=0x7f957a52bb30, rs_readers=...) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/olap/vcollect_iterator.cpp:125 apache#38 0x0000561dc2c7e1f2 in doris::vectorized::BlockReader::_init_collect_iter (this=this@entry=0x7f957a52b400, read_params=...) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/olap/block_reader.cpp:153 apache#39 0x0000561dc2c7f191 in doris::vectorized::BlockReader::init (this=<optimized out>, read_params=...) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/olap/block_reader.cpp:226 apache#40 0x0000561dc3937869 in doris::vectorized::NewOlapScanner::open (this=0x7f9a56270210, state=<optimized out>) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:252 apache#41 0x0000561dbdcc5413 in doris::vectorized::ScannerScheduler::_scanner_scan (ctx=..., scan_task=...) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:221 apache#42 0x0000561dbdcc62bd in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::{lambda()#1}::operator()() const::{lambda()#1}::operator()() const (this=<optimized out>) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:154 apache#43 doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::{lambda()#1}::operator()() const (this=0x7f954e25d3c0) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:153 apache#44 std::__invoke_impl<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::{lambda()#1}&>(std::__invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::{lambda()#1}&) (__f=...) at /data/home/lambxu/installs/ldb_toolchain_bak/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61 apache#45 std::__invoke_r<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::{lambda()#1}&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::{lambda()#1}&) (__fn=...) at /data/home/lambxu/installs/ldb_toolchain_bak/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111 apache#46 std::_Function_handler<void (), doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::{lambda()#1}>::_M_invoke(std::_Any_data const&) (__functor=...) at /data/home/lambxu/installs/ldb_toolchain_bak/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291 apache#47 0x0000561db962137a in doris::ThreadPool::dispatch_thread (this=0x7f9a50f9d380) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/util/threadpool.cpp:602 apache#48 0x0000561db96159a1 in std::function<void ()>::operator()() const (this=0x11a791) at /data/home/lambxu/installs/ldb_toolchain_bak/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560 apache#49 doris::Thread::supervise_thread (arg=0x7f969f569ce0) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/util/thread.cpp:498 apache#50 0x00007f9aca4a2215 in start_thread () from /lib64/libc.so.6 apache#51 0x00007f9aca524bdc in clone3 () from /lib64/libc.so.6 ``` Assume that 0,1,2,3,4, is key columns of an AGG mv, because the PreAgg is OFF at scan node, the block will contain all key columns to merge data in storage layer. if we select 0,1 column, with 3,4 column in where clause, then the slot ids should be 0,1,3,4, and column ids in conjuncts is the index of slot ids.(which is 2 and 3) But the plan use the key type of base table which is DUP key, treating the AGG mv as a DUP mv, so these conjuncts are pushed down to the scan node which belongs to an AGG mv, these conjuncts will pick the wrong column 2 and 3 (which shoud be 4 and 5) in block to exucute. So we should use the key type of mv but not the key type of base table. ### Release note None ### Check List (For Author) - Test  - [x] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [x] No. - [ ] Yes.  - Does this need documentation? - [x] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

…57888) We previously had a crash. The cause is that we should not access the request after calling add_block(...) because add_block may enqueue a closure that runs on another thread and frees the request ``` ==730145==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x7be1efd803a0 at pc 0x556b38d9d625 bp 0x7b16bf0193f0 sp 0x7b16bf0193e8 READ of size 4 at 0x7be1efd803a0 thread T1559 #0 0x556b38d9d624 in google::protobuf::internal::RepeatedPtrFieldBase::size() const /home/zcp/repo_center/doris_master/doris/thirdparty/installed/include/google/protobuf/repeated_ptr_field.h:185:29 #1 0x556b408ab062 in google::protobuf::RepeatedPtrField<doris::PBlock>::size() const /home/zcp/repo_center/doris_master/doris/thirdparty/installed/include/google/protobuf/repeated_ptr_field.h:1248:32 apache#2 0x556b408aaff4 in doris::PTransmitDataParams::_internal_blocks_size() const /home/zcp/repo_center/doris_master/doris/be/../gensrc/build/gen_cpp/internal_service.pb.h:32149:25 apache#3 0x556b4089731c in doris::PTransmitDataParams::blocks_size() const /home/zcp/repo_center/doris_master/doris/be/../gensrc/build/gen_cpp/internal_service.pb.h:32152:10 apache#4 0x556b60a83c17 in doris::vectorized::VDataStreamMgr::transmit_block(doris::PTransmitDataParams const*, google::protobuf::Closure**, long) /home/zcp/repo_center/doris_master/doris/be/src/vec/runtime/vdata_stream_mgr.cpp:150:38 apache#5 0x556b407f7408 in doris::PInternalService::_transmit_block(google::protobuf::RpcController*, doris::PTransmitDataParams const*, doris::PTransmitDataResult*, google::protobuf::Closure*, doris::Status const&, long) /home/zcp/repo_center/doris_master/doris/be/src/service/internal_service.cpp:1673:40 apache#6 0x556b407f52bb in doris::PInternalService::transmit_block(google::protobuf::RpcController*, doris::PTransmitDataParams const*, doris::PTransmitDataResult*, google::protobuf::Closure*) /home/zcp/repo_center/doris_master/doris/be/src/service/internal_service.cpp:1610:9 apache#7 0x556b43fceba2 in doris::PBackendService::CallMethod(google::protobuf::MethodDescriptor const*, google::protobuf::RpcController*, google::protobuf::Message const*, google::protobuf::Message*, google::protobuf::Closure*) /home/zcp/repo_center/doris_master/doris/gensrc/build/gen_cpp/internal_service.pb.cc:49452:7 apache#8 0x556b6736273e in brpc::policy::ProcessRpcRequest(brpc::InputMessageBase*) (/mnt/hdd01/selectdb-cloud-chaos/cluster0/be/lib/doris_be+0x770bd73e) apache#9 0x556b67357426 in brpc::ProcessInputMessage(void*) (/mnt/hdd01/selectdb-cloud-chaos/cluster0/be/lib/doris_be+0x770b2426) apache#10 0x556b67357f20 in brpc::InputMessenger::InputMessageClosure::~InputMessageClosure() (/mnt/hdd01/selectdb-cloud-chaos/cluster0/be/lib/doris_be+0x770b2f20) apache#11 0x556b673588dd in brpc::InputMessenger::OnNewMessages(brpc::Socket*) (/mnt/hdd01/selectdb-cloud-chaos/cluster0/be/lib/doris_be+0x770b38dd) apache#12 0x556b674a0adc in brpc::Socket::ProcessEvent(void*) (/mnt/hdd01/selectdb-cloud-chaos/cluster0/be/lib/doris_be+0x771fbadc) apache#13 0x556b672e0f76 in bthread::TaskGroup::task_runner(long) (/mnt/hdd01/selectdb-cloud-chaos/cluster0/be/lib/doris_be+0x7703bf76) apache#14 0x556b672cbbe0 in bthread_make_fcontext (/mnt/hdd01/selectdb-cloud-chaos/cluster0/be/lib/doris_be+0x77026be0) ``` ``` Note: The done pointer will be saved in add_block and may be called in another thread via done->Run(). For example, when blocks_size == 1, the process is as follows: transmit_block (i=0) └─> recvr->add_block(..., done, ...) // Pass done └─> SenderQueue::add_block └─> _pending_closures.push(done) // done is saved get_batch() [another thread] └─> closure_pair.first->Run() // ⚠️ done->Run() is called └─> brpc releases request and response transmit_block (i=1) [original thread continues] └─> request->blocks_size() // ⚠️ request has already been released! At this point, a use-after-free issue occurs. TODO: We should consider refactoring this part because add_block may release the request. We should not access the request after calling add_block. ``` apache#50113

…ple calls to the get_string method (apache#58107) ### What problem does this PR solve? <img width="2118" height="1222" alt="image" src="https://github.com/user-attachments/assets/aeefb709-f59d-406e-824f-0c9b250bb5af" /> ```text doris_be: /root/doris/thirdparty/installed/include/simdjson/generic/ondemand/json_iterator-inl.h:359: simdjson_result<std::string_view> simdjson::fallback::ondemand::json_iterator::unescape(raw_json_string, bool): Assertion `!parser->string_buffer_overflow(_string_buf_loc)' failed. *** Query id: 24468c5d0b372cf0-3e6c777580238e96 *** *** is nereids: 1 *** *** tablet id: 0 *** *** Aborted at 1763545679 (unix time) try "date -d @1763545679" if you are using GNU date *** *** Current BE git commitID: cbc3d21 *** *** SIGABRT unknown detail explain (@0x3f8001618b8) received by PID 1448120 (TID 1450358 OR 0x7b0fb9f4e700) from PID 1448120; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/doris/be/src/common/signal_handler.h:420 1# 0x00007F183C476D10 in /lib64/libpthread.so.0 2# __GI_raise in /lib64/libc.so.6 3# __GI_abort in /lib64/libc.so.6 4# _nl_load_domain.cold.0 in /lib64/libc.so.6 5# 0x00007F183BCC8E86 in /lib64/libc.so.6 6# simdjson::fallback::ondemand::json_iterator::unescape(simdjson::fallback::ondemand::raw_json_string, bool) at /root/doris/thirdparty/installed/include/simdjson/generic/ondemand/json_iterator-inl.h:359 7# simdjson::fallback::ondemand::raw_json_string::unescape(simdjson::fallback::ondemand::json_iterator&, bool) const at /root/doris/thirdparty/installed/include/simdjson/generic/ondemand/raw_json_string-inl.h:160 8# simdjson::simdjson_result<simdjson::fallback::ondemand::raw_json_string>::unescape(simdjson::fallback::ondemand::json_iterator&, bool) const in /root/doris/be/output/lib/doris_be 9# simdjson::fallback::ondemand::value_iterator::get_string(bool) at /root/doris/thirdparty/installed/include/simdjson/generic/ondemand/value_iterator-inl.h:514 10# simdjson::fallback::ondemand::value::get_string(bool) at /root/doris/thirdparty/installed/include/simdjson/generic/ondemand/value-inl.h:48 11# doris::Status doris::vectorized::NewJsonReader::_simdjson_write_data_to_column<false>(simdjson::fallback::ondemand::value&, std::shared_ptr<doris::vectorized::IDataType const> const&, doris::vectorized::IColumn*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<doris::vectorized::DataTypeSerDe>, bool*) at /root/doris/be/src/vec/exec/format/json/new_json_reader.cpp:1080 12# doris::vectorized::NewJsonReader::_simdjson_write_columns_by_jsonpath(simdjson::fallback::ondemand::object*, std::vector<doris::SlotDescriptor*, std::allocator<doris::SlotDescriptor*> > const&, doris::vectorized::Block&, bool*) at /root/doris/be/src/vec/exec/format/json/new_json_reader.cpp:1460 13# doris::vectorized::NewJsonReader::_simdjson_handle_flat_array_complex_json_write_columns(doris::vectorized::Block&, std::vector<doris::SlotDescriptor*, std::allocator<doris::SlotDescriptor*> > const&, bool*, bool*) at /root/doris/be/src/vec/exec/format/json/new_json_reader.cpp:806 14# doris::vectorized::NewJsonReader::_simdjson_handle_flat_array_complex_json(doris::RuntimeState*, doris::vectorized::Block&, std::vector<doris::SlotDescriptor*, std::allocator<doris::SlotDescriptor*> > const&, bool*, bool*) at /root/doris/be/src/vec/exec/format/json/new_json_reader.cpp:753 15# doris::vectorized::NewJsonReader::_read_json_column(doris::RuntimeState*, doris::vectorized::Block&, std::vector<doris::SlotDescriptor*, std::allocator<doris::SlotDescriptor*> > const&, bool*, bool*) at /root/doris/be/src/vec/exec/format/json/new_json_reader.cpp:485 16# doris::vectorized::NewJsonReader::get_next_block(doris::vectorized::Block*, unsigned long*, bool*) at /root/doris/be/src/vec/exec/format/json/new_json_reader.cpp:209 17# doris::vectorized::FileScanner::_get_block_wrapped(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /root/doris/be/src/vec/exec/scan/file_scanner.cpp:480 18# doris::vectorized::FileScanner::_get_block_impl(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /root/doris/be/src/vec/exec/scan/file_scanner.cpp:417 19# doris::vectorized::Scanner::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /root/doris/be/src/vec/exec/scan/scanner.cpp:113 20# doris::vectorized::Scanner::get_block_after_projects(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /root/doris/be/src/vec/exec/scan/scanner.cpp:85 21# doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>) at /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:182 22# doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()#1}::operator()() const::{lambda()#1}::operator()() const at /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:96 23# doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()#1}::operator()() const at /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:95 ``` Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test  - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [ ] No. - [ ] Yes.  - Does this need documentation? - [ ] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

… already exists in file. (apache#58579) ### What problem does this PR solve? Problem Summary: fix : ``` LOAD LABEL labelx ( DATA INFILE("s3://bucket/load/k1=10/k3=30/test.parquet") INTO TABLE tableName FORMAT AS "parquet" (k2) COLUMNS FROM PATH AS (k1,k3) ) WITH S3 ( xxx) ``` and k1 or k3 column exists in test.parquet file. error: ``` type:LOAD_RUN_FAIL; msg:errCode = 2, detailMessage = (127.0.0.1)[E-7412] assert cast err:[E-7412] Bad cast from type:doris::vectorized::ColumnVector<(doris::PrimitiveType)5> to doris::vectorized::ColumnStr<unsigned int> 0# doris::Exception::Exception(int, std::basic_string_view<char, std::char_traits<char> > const&, bool) at /mnt/disk1/changyuwei/ldb_toolchain/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/unique_ptr.h:193 1# doris::Exception::Exception(doris::Status const&) at /mnt/disk1/changyuwei/doris/be/src/common/exception.h:39 2# doris::vectorized::ColumnStr<unsigned int>& assert_cast<doris::vectorized::ColumnStr<unsigned int>&, (TypeCheckOnRelease)1, doris::vectorized::IColumn&>(doris::vectorized::IColumn&)::{lambda(auto:1&&)#1}::operator()<doris::vectorized::IColumn&>(doris::vectorized::IColumn&) const at /mnt/disk1/changyuwei/doris/be/src/vec/common/assert_cast.h:0 3# doris::vectorized::ColumnStr<unsigned int>& assert_cast<doris::vectorized::ColumnStr<unsigned int>&, (TypeCheckOnRelease)1, doris::vectorized::IColumn&>(doris::vectorized::IColumn&) at /mnt/disk1/changyuwei/doris/be/src/vec/common/assert_cast.h:72 4# doris::vectorized::DataTypeStringSerDeBase<doris::vectorized::ColumnStr<unsigned int> >::deserialize_one_cell_from_json(doris::vectorized::IColumn&, doris::Slice&, doris::vectorized::DataTypeSerDe::FormatOptions const&) const at /mnt/disk1/changyuwei/doris/be/src/vec/data_types/serde/data_type_string_serde.cpp:108 5# doris::vectorized::DataTypeNullableSerDe::deserialize_one_cell_from_json(doris::vectorized::IColumn&, doris::Slice&, doris::vectorized::DataTypeSerDe::FormatOptions const&) const at /mnt/disk1/changyuwei/doris/be/src/common/status.h:525 6# doris::vectorized::DataTypeNullableSerDe::deserialize_column_from_fixed_json(doris::vectorized::IColumn&, doris::Slice&, unsigned long, unsigned long*, doris::vectorized::DataTypeSerDe::FormatOptions const&) const at /mnt/disk1/changyuwei/doris/be/src/common/status.h:525 7# doris::vectorized::RowGroupReader::_fill_partition_columns(doris::vectorized::Block*, unsigned long, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, doris::SlotDescriptor const*>, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, doris::SlotDescriptor const*> > > > const&) at /mnt/disk1/changyuwei/doris/be/src/common/status.h:567 8# doris::vectorized::RowGroupReader::next_batch(doris::vectorized::Block*, unsigned long, unsigned long*, bool*) at /mnt/disk1/changyuwei/doris/be/src/common/status.h:525 9# doris::vectorized::ParquetReader::get_next_block(doris::vectorized::Block*, unsigned long*, bool*) at /mnt/disk1/changyuwei/doris/be/src/common/status.h:520 10# doris::vectorized::FileScanner::_get_block_wrapped(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /mnt/disk1/changyuwei/doris/be/src/common/status.h:525 11# doris::vectorized::FileScanner::_get_block_impl(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /mnt/disk1/changyuwei/doris/be/src/common/status.h:525 12# doris::vectorized::Scanner::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /mnt/disk1/changyuwei/doris/be/src/common/status.h:525 13# doris::vectorized::Scanner::get_block_after_projects(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /mnt/disk1/changyuwei/doris/be/src/vec/exec/scan/scanner.cpp:87 14# doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>) at /mnt/disk1/changyuwei/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:182 ```

…shard_ptr not initialized (apache#58751) ``` #0 0x000055f11470ff29 in std::__shared_ptr<doris::TabletMeta, (__gnu_cxx::_Lock_policy)2>::get (this=0x148) at /home/work/env/ldb_toolchain_master/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:1673 #1 std::__shared_ptr_access<doris::TabletMeta, (__gnu_cxx::_Lock_policy)2, false, false>::_M_get (this=0x148) at /home/work/env/ldb_toolchain_master/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:1370 apache#2 std::__shared_ptr_access<doris::TabletMeta, (__gnu_cxx::_Lock_policy)2, false, false>::operator-> (this=0x148) at /home/work/env/ldb_toolchain_master/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:1364 apache#3 doris::BaseTablet::tablet_id (this=0x0) at /home/work/doris/be/src/olap/base_tablet.h:73 apache#4 doris::segment_v2::SegmentWriter::finalize (this=0x7fa160bde000, segment_file_size=0x7fa564dcc668, index_size=0x7fa564dcc5b8) at /home/work/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:1044 apache#5 0x000055f11442578c in doris::SegmentFlusher::_flush_segment_writer (this=0x7fa160b8a420, writer=..., flush_size=flush_size@entry=0x0) at /home/work/doris/be/src/olap/rowset/segment_creator.cpp:304 apache#6 0x000055f114426c4e in doris::SegmentFlusher::Writer::flush (this=<optimized out>) at /home/work/doris/be/src/olap/rowset/segment_creator.cpp:376 apache#7 doris::SegmentCreator::flush (this=0x7fa160b8a418) at /home/work/doris/be/src/olap/rowset/segment_creator.cpp:422 apache#8 0x000055f1143d4785 in doris::BaseBetaRowsetWriter::flush (this=<optimized out>) at /home/work/doris/be/src/olap/rowset/beta_rowset_writer.cpp:723 apache#9 0x000055f11436bf47 in doris::Merger::vmerge_rowsets (tablet=..., reader_type=<optimized out>, cur_tablet_schema=..., src_rowset_readers=..., dst_rowset_writer=0x7fa160b8a000, stats_output=0x7fa1df108298) at /home/work/doris/be/src/olap/merger.cpp:159 apache#10 0x000055f114345595 in doris::Compaction::merge_input_rowsets (this=this@entry=0x7fa1df108210) at /home/work/doris/be/src/olap/compaction.cpp:220 apache#11 0x000055f1143550d2 in doris::CloudCompactionMixin::execute_compact_impl (this=this@entry=0x7fa1df108210, permits=permits@entry=6) at /home/work/doris/be/src/olap/compaction.cpp:1491 apache#12 0x000055f114342bc1 in doris::CloudCompactionMixin::execute_compact (this=0x7fa1df108210) at /home/work/doris/be/src/olap/compaction.cpp:1620 apache#13 0x000055f11a22c6d8 in doris::CloudCumulativeCompaction::execute_compact (this=0x7fa1df108210) at /home/work/doris/be/src/cloud/cloud_cumulative_compaction.cpp:203 ```

…e#59098) ### What problem does this PR solve? Introduced by apache#58905 ==2037076==ERROR: AddressSanitizer: heap-use-after-free on address 0x7baaae908730 at pc 0x561b769a1fd0 bp 0x7b3caf4ebdf0 sp 0x7b3caf4ebde8 22:30:08 READ of size 1 at 0x7baaae908730 thread T12303 (rs_normal [work) 22:30:08 #0 0x561b769a1fcf in doris::(anonymous namespace)::string_compare(char const*, long, char const*, long, long) /root/doris/be/src/vec/common/string_ref.h:170:29 22:30:08 #1 0x561b769a1fcf in doris::StringRef::compare(doris::StringRef const&) const /root/doris/be/src/vec/common/string_ref.h:259:30 22:30:08 apache#2 0x561b76f537cd in doris::StringRef::ge(doris::StringRef const&) const /root/doris/be/src/vec/common/string_ref.h:282:52 22:30:08 apache#3 0x561b76f537cd in doris::StringRef::operator>=(doris::StringRef const&) const /root/doris/be/src/vec/common/string_ref.h:292:60 22:30:08 apache#4 0x561b76f537cd in bool doris::Compare::greater_equal<doris::StringRef>(doris::StringRef const&, doris::StringRef const&) /root/doris/be/src/common/compare.h:42:18 22:30:08 apache#5 0x561b76f537cd in doris::ComparisonPredicateBase<(doris::PrimitiveType)23, (doris::PredicateType)6>::camp_field(doris::vectorized::Field const&, doris::vectorized::Field const&) const /root/doris/be/src/olap/comparison_predicate.h:192:20 22:30:08 apache#6 0x561b76f4baa4 in doris::ComparisonPredicateBase<(doris::PrimitiveType)23, (doris::PredicateType)6>::evaluate_and(doris::vectorized::ParquetPredicate::ColumnStat*) const /root/doris/be/src/olap/comparison_predicate.h:207:26 22:30:08 apache#7 0x561b76765284 in doris::AndBlockColumnPredicate::evaluate_and(doris::vectorized::ParquetPredicate::ColumnStat*) const /root/doris/be/src/olap/block_column_predicate.h:251:42 22:30:08 apache#8 0x561b89acd735 in doris::vectorized::ParquetReader::_process_column_stat_filter(tparquet::RowGroup const&, std::vector<std::unique_ptr<doris::MutilColumnBlockPredicate, std::default_delete<doris::MutilColumnBlockPredicate> >, std::allocator<std::unique_ptr<doris::MutilColumnBlockPredicate, std::default_delete<doris::MutilColumnBlockPredicate> > > > const&, bool*, bool*, bool*) /root/doris/be/src/vec/exec/format/parquet/vparquet_reader.cpp:1225:25 22:30:08 apache#9 0x561b89ac8dd7 in doris::vectorized::ParquetReader::_process_min_max_bloom_filter(doris::vectorized::RowGroupReader::RowGroupIndex const&, tparquet::RowGroup const&, std::vector<std::unique_ptr<doris::MutilColumnBlockPredicate, std::default_delete<doris::MutilColumnBlockPredicate> >, std::allocator<std::unique_ptr<doris::MutilColumnBlockPredicate, std::default_delete<doris::MutilColumnBlockPredicate> > > > const&, doris::segment_v2::RowRanges*) /root/doris/be/src/vec/exec/format/parquet/vparquet_reader.cpp:1108:9 22:30:08 apache#10 0x561b89ac3e73 in doris::vectorized::ParquetReader::_next_row_group_reader() /root/doris/be/src/vec/exec/format/parquet/vparquet_reader.cpp:718:9 22:30:08 apache#11 0x561b89ac008f in doris::vectorized::ParquetReader::get_next_block(doris::vectorized::Block*, unsigned long*, bool*) /root/doris/be/src/vec/exec/format/parquet/vparquet_reader.cpp:607:21 22:30:08 apache#12 0x561b8a07c6f7 in doris::vectorized::HiveReader::get_next_block_inner(doris::vectorized::Block*, unsigned long*, bool*) /root/doris/be/src/vec/exec/format/table/hive_reader.cpp:32:5 22:30:08 apache#13 0x561b89fee256 in doris::vectorized::TableFormatReader::get_next_block(doris::vectorized::Block*, unsigned long*, bool*) /root/doris/be/src/vec/exec/format/table/table_format_reader.h:81:16 22:30:08 apache#14 0x561b89f71b97 in doris::vectorized::FileScanner::_get_block_wrapped(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/be/src/vec/exec/scan/file_scanner.cpp:472:13 22:30:08 apache#15 0x561b89f7086f in doris::vectorized::FileScanner::_get_block_impl(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/be/src/vec/exec/scan/file_scanner.cpp:409:17 22:30:08 apache#16 0x561b8a19f86e in doris::vectorized::Scanner::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/be/src/vec/exec/scan/scanner.cpp:109:17 22:30:08 apache#17 0x561b8a19f0a6 in doris::vectorized::Scanner::get_block_after_projects(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/be/src/vec/exec/scan/scanner.cpp:85:16 22:30:08 apache#18 0x561b8a1ccd0f in doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>) /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:173:5 22:30:08 apache#19 0x561b8a1d6875 in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:76:17 22:30:08 apache#20 0x561b8a1d6875 in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:75:27 22:30:08 apache#21 0x561b8a1d6875 in bool std::__invoke_impl<bool, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::'lambda'()&>(std::__invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::'lambda'()&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:63:14 22:30:08 apache#22 0x561b8a1d6875 in std::enable_if<is_invocable_r_v<bool, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::'lambda'()&>, bool>::type std::__invoke_r<bool, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::'lambda'()&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::'lambda'()&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:116:9 22:30:08 apache#23 0x561b8a1d6875 in std::_Function_handler<bool (), doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::'lambda'()>::_M_invoke(std::_Any_data const&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:292:9 22:30:08 apache#24 0x561b8a1d5f07 in std::function<bool ()>::operator()() const /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593:9 22:30:08 apache#25 0x561b8a1d5f07 in doris::vectorized::ScannerSplitRunner::process_for(std::chrono::duration<long, std::ratio<1l, 1000000000l> >) /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:407:25 22:30:08 apache#26 0x561b8a2c56d4 in doris::vectorized::PrioritizedSplitRunner::process() /root/doris/be/src/vec/exec/executor/time_sharing/prioritized_split_runner.cpp:103:35 22:30:08 apache#27 0x561b8a29045c in doris::vectorized::TimeSharingTaskExecutor::_dispatch_thread() /root/doris/be/src/vec/exec/executor/time_sharing/time_sharing_task_executor.cpp:570:77 22:30:08 apache#28 0x561b7b9fecb6 in std::function<void ()>::operator()() const /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593:9 22:30:08 apache#29 0x561b7b9fecb6 in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:460:5 22:30:08 apache#30 0x561b76044d26 in asan_thread_start(void*) (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P1/Cluster0/be/lib/doris_be+0x23962d26) 22:30:08 apache#31 0x7f4aaae68608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477:8 22:30:08 apache#32 0x7f4aaad7b132 in __clone /build/glibc-SzIz7B/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95

support export doris native format file ``` // Doris Native format file-level constants. // // File layout (byte stream): // // +-------------------------------+---------------------------+---------------------------+ ... // | File header | Data block #0 | Data block #1 | ... // +-------------------------------+---------------------------+---------------------------+ ... // // File header (12 bytes total): // - [0..7] : magic bytes "DORISN1\0" (DORIS_NATIVE_MAGIC) // - [8..11] : uint32_t format_version (DORIS_NATIVE_FORMAT_VERSION, little-endian) // // Each data block i: // - uint64_t block_size : length in bytes of serialized PBlock (little-endian) // - uint8_t[block_size] : PBlock protobuf payload produced by Block::serialize() // // NativeReader: // - Detects the optional file header by checking the first 8 bytes against DORIS_NATIVE_MAGIC. // - If the header is present, it skips 12 bytes and then starts reading blocks as // [uint64_t block_size][PBlock bytes]... // - If the header is absent (legacy files), it starts reading blocks from offset 0. // // VNativeTransformer: // - Writes the header once in open(), then appends each block in write() as // [uint64_t block_size][PBlock bytes]... // // These constants are shared between writer, reader and tests to keep the on-disk // format definition in a single place. // Header layout: // [magic bytes "DORISN1\0"][uint32_t format_version] ``` docs: apache/doris-website#3190

…pache#59920) ### What problem does this PR solve? When `BaseBetaRowsetWriter` is destructed(this may happen when the load is canceled) before the execution of the task submitted to thread pool, the task may encounter coredump due to use after free. ``` (gdb) bt #0 __GI___pthread_sigmask (how=2, newmask=<optimized out>, oldmask=0x0) at ./nptl/pthread_sigmask.c:43 #1 0x00007fa1d0c1171e in PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] () from /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so apache#2 0x00007fa1d0c12206 in JVM_handle_linux_signal () from /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so apache#3 <signal handler called> apache#4 doris::TUniqueId::TUniqueId (this=0x7f99955f2208, other51=...) at /home/zcp/repo_center/doris_branch-4.0/doris/gensrc/build/gen_cpp/Types_types.cpp:2571 apache#5 0x00005653d14008ca in doris::AttachTask::init (rc=..., this=<optimized out>) at /home/zcp/repo_center/doris_branch-4.0/doris/be/src/runtime/thread_context.cpp:29 apache#6 doris::AttachTask::AttachTask (this=<optimized out>, rc=...) at /home/zcp/repo_center/doris_branch-4.0/doris/be/src/runtime/thread_context.cpp:34 apache#7 0x00005653d0d05087 in doris::CalcDeleteBitmapToken::submit_func<doris::BaseBetaRowsetWriter::_generate_delete_bitmap(int)::$_0>(doris::BaseBetaRowsetWriter::_generate_delete_bitmap(int)::$_0&&)::{lambda()#1}::operator()() const (this=0x7f9cdf302500) at /home/zcp/repo_center/doris_branch-4.0/doris/be/src/olap/calc_delete_bitmap_executor.h:74 apache#8 std::__invoke_impl<void, doris::CalcDeleteBitmapToken::submit_func<doris::BaseBetaRowsetWriter::_generate_delete_bitmap(int)::$_0>(doris::BaseBetaRowsetWriter::_generate_delete_bitmap(int)::$_0&&)::{lambda()#1}&>(std::__invoke_other, doris::CalcDeleteBitmapToken::submit_func<doris::BaseBetaRowsetWriter::_generate_delete_bitmap(int)::$_0>(doris::BaseBetaRowsetWriter::_generate_delete_bitmap(int)::$_0&&)::{lambda()#1}&) (__f=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:63 apache#9 std::__invoke_r<void, doris::CalcDeleteBitmapToken::submit_func<doris::BaseBetaRowsetWriter::_generate_delete_bitmap(int)::$_0>(doris::BaseBetaRowsetWriter::_generate_delete_bitmap(int)::$_0&&)::{lambda()#1}&>(doris::CalcDeleteBitmapToken::submit_func<doris::BaseBetaRowsetWriter::_generate_delete_bitmap(int)::$_0>(doris::BaseBetaRowsetWriter::_generate_delete_bitmap(int)::$_0&&)::{lambda()#1}&) (__fn=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:113 apache#10 std::_Function_handler<void (), doris::CalcDeleteBitmapToken::submit_func<doris::BaseBetaRowsetWriter::_generate_delete_bitmap(int)::$_0>(doris::BaseBetaRowsetWriter::_generate_delete_bitmap(int)::$_0&&)::{lambda()#1}>::_M_invoke(std::_Any_data const&) ( __functor=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:292 apache#11 0x00005653d16392e5 in doris::ThreadPool::dispatch_thread (this=0x7fa120d9af00) at /home/zcp/repo_center/doris_branch-4.0/doris/be/src/util/threadpool.cpp:616 apache#12 0x00005653d162e38c in std::function<void ()>::operator()() const (this=0x7f99955f2208) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593 apache#13 doris::Thread::supervise_thread (arg=0x7fa0c0049110) at /home/zcp/repo_center/doris_branch-4.0/doris/be/src/util/thread.cpp:460 apache#14 0x00007fa1cfcacac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 apache#15 0x00007fa1cfd3e850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 ``` ### Release note None ### Check List (For Author) - Test  - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [ ] No. - [ ] Yes.  - Does this need documentation? - [ ] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

### What problem does this PR solve? *** SIGSEGV address not mapped to object (@0xfffffffffffffff0) received by PID 12431 (TID 13514 OR 0x7f02981d1640) from PID 18446744073709551600; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_branch-4.0/doris/be/src/common/signal_handler.h:420 1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so 2# JVM_handle_linux_signal in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so 3# 0x00007F04AE099520 in /lib/x86_64-linux-gnu/libc.so.6 4# doris::pipeline::ScanLocalState::_normalize_predicate(doris::vectorized::VExprContext*, std::shared_ptr const&, std::shared_ptr&)::{lambda(auto:1&)#1}::operator() >(doris::ColumnValueRange<(doris::PrimitiveType)25>&) const::{lambda()#1}::operator()() const at /home/zcp/repo_center/doris_branch-4.0/doris/be/src/pipeline/exec/scan_operator.cpp:352 5# doris::Defer::_normalize_predicate(doris::vectorized::VExprContext*, std::shared_ptr const&, std::shared_ptr&)::{lambda(auto:1&)#1}::operator() >(doris::ColumnValueRange<(doris::PrimitiveType)25>&) const::{lambda()#1}>::~Defer() at /home/zcp/repo_center/doris_branch-4.0/doris/be/src/util/defer_op.h:76 6# auto doris::pipeline::ScanLocalState::_normalize_predicate(doris::vectorized::VExprContext*, std::shared_ptr const&, std::shared_ptr&)::{lambda(auto:1&)#1}::operator() >(doris::ColumnValueRange<(doris::PrimitiveType)25>&) const in /mnt/hdd01/ci/doris-deploy-branch-4.0-local/be/lib/doris_be 7# doris::pipeline::ScanLocalState::_normalize_predicate(doris::vectorized::VExprContext*, std::shared_ptr const&, std::shared_ptr&) at /home/zcp/repo_center/doris_branch-4.0/doris/be/src/pipeline/exec/scan_operator.cpp:412 8# doris::pipeline::ScanLocalState::_normalize_conjuncts(doris::RuntimeState*) at /home/zcp/repo_center/doris_branch-4.0/doris/be/src/pipeline/exec/scan_operator.cpp:267 9# doris::pipeline::OlapScanLocalState::_process_conjuncts(doris::RuntimeState*) at /home/zcp/repo_center/doris_branch-4.0/doris/be/src/pipeline/exec/olap_scan_operator.cpp:366 10# doris::pipeline::ScanLocalState::open(doris::RuntimeState*) at /home/zcp/repo_center/doris_branch-4.0/doris/be/src/pipeline/exec/scan_operator.cpp:158 11# doris::pipeline::OlapScanLocalState::open(doris::RuntimeState*) at /home/zcp/repo_center/doris_branch-4.0/doris/be/src/pipeline/exec/olap_scan_operator.cpp:787 12# doris::pipeline::PipelineTask::_open() at /home/zcp/repo_center/doris_branch-4.0/doris/be/src/pipeline/pipeline_task.cpp:269 13# doris::pipeline::PipelineTask::execute(bool*) at /home/zcp/repo_center/doris_branch-4.0/doris/be/src/pipeline/pipeline_task.cpp:467 14# doris::pipeline::TaskScheduler::_do_work(int) at /home/zcp/repo_center/doris_branch-4.0/doris/be/src/pipeline/task_scheduler.cpp:153 15# doris::ThreadPool::dispatch_thread() at /home/zcp/repo_center/doris_branch-4.0/doris/be/src/util/threadpool.cpp:623 16# doris::Thread::supervise_thread(void*) at /home/zcp/repo_center/doris_branch-4.0/doris/be/src/util/thread.cpp:461 17# start_thread at ./nptl/pthread_create.c:442 ### Check List (For Author) - Test  - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [ ] No. - [ ] Yes.  - Does this need documentation? - [ ] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

…lain verbose (apache#60308) ### What problem does this PR solve? Problem Summary: This PR enhances the output of EXPLAIN VERBOSE for File Scan nodes by adding the following metrics: `dataFileNum=xxx, deleteFileNum=xxx, deleteSplitNum=xxx` Especially useful for iceberg/paimon/hive acid These metrics provide more visibility into the underlying file and split layout, helping users better tune parameters and control query performance. Details: `dataFileNum` : The number of distinct data files that need to be read. This is not equivalent to the number of splits, since a single data file can be divided into multiple splits. `deleteFileNum` : The number of distinct delete files that need to be read. `deleteSplitNum` : Added because the relationship between data files and delete files is many-to-many: one data file may be associated with multiple delete files one delete file may apply to multiple data files Using deleteSplitNum / dataSplitNum, users can estimate the average number of delete splits that need to be read per data split. Example: ``` mysql> explain verbose select * from iceberg.format_v3.dv_test_1w; +-----------------------------------------------------------------------------------------------------------------------------------------------+ | Explain String(Nereids Planner) | +-----------------------------------------------------------------------------------------------------------------------------------------------+ | PLAN FRAGMENT 0 | | OUTPUT EXPRS: | | id[#0] | | grp[#1] | | value[apache#2] | | ts[apache#3] | | PARTITION: RANDOM | | | | HAS_COLO_PLAN_NODE: false | | | | VRESULT SINK | | MYSQL_PROTOCOL | | | | 0:VICEBERG_SCAN_NODE(32) | | table: iceberg.format_v3.dv_test_1w | | inputSplitNum=220, totalFileSize=720774, scanRanges=220 | | partition=0/0 | | backends: | | 1769590309070 | | s3://warehouse/wh/format_v3/dv_test_1w/data/00004-51-fc462f9a-d42a-404d-adfc-c8d2781c8d04-0-00001.parquet start: 4 length: 2672 | | s3://warehouse/wh/format_v3/dv_test_1w/data/00003-50-fc462f9a-d42a-404d-adfc-c8d2781c8d04-0-00001.parquet start: 4 length: 2852 | | s3://warehouse/wh/format_v3/dv_test_1w/data/00000-47-fc462f9a-d42a-404d-adfc-c8d2781c8d04-0-00001.parquet start: 4 length: 2894 | | ... other 216 files ... | | s3://warehouse/wh/format_v3/dv_test_1w/data/00001-48-fc462f9a-d42a-404d-adfc-c8d2781c8d04-0-00001.parquet start: 58397 length: 13894 | | dataFileNum=10, deleteFileNum=1 deleteSplitNum=220 | | cardinality=33334, numNodes=1 | | pushdown agg=NONE | | tuple ids: 0 | | | | Tuples: | | TupleDescriptor{id=0, tbl=dv_test_1w} | | SlotDescriptor{id=0, col=id, colUniqueId=1, type=bigint, nullable=true, isAutoIncrement=false, subColPath=null, virtualColumn=null} | | SlotDescriptor{id=1, col=grp, colUniqueId=2, type=int, nullable=true, isAutoIncrement=false, subColPath=null, virtualColumn=null} | | SlotDescriptor{id=2, col=value, colUniqueId=3, type=int, nullable=true, isAutoIncrement=false, subColPath=null, virtualColumn=null} | | SlotDescriptor{id=3, col=ts, colUniqueId=4, type=datetimev2(6), nullable=true, isAutoIncrement=false, subColPath=null, virtualColumn=null} | | | | | | | | | | ========== STATISTICS ========== | +-----------------------------------------------------------------------------------------------------------------------------------------------+ ```

…ay_map are used together. (apache#60693) array_map constructs a temporary block but incorrectly passed the selector. ``` 4# 0x000056376DFC5EC5 in /home/work/unlimit_teamcity/TeamCity/Agents/20260211180011agent_172.17.0.220_1/work/60183217f6ee2a9c/output/be/lib/doris_be 5# 0x000056376DFB777A in /home/work/unlimit_teamcity/TeamCity/Agents/20260211180011agent_172.17.0.220_1/work/60183217f6ee2a9c/output/be/lib/doris_be 6# google::LogMessage::SendToLog() in /home/work/unlimit_teamcity/TeamCity/Agents/20260211180011agent_172.17.0.220_1/work/60183217f6ee2a9c/output/be/lib/doris_be 7# google::LogMessage::Flush() in /home/work/unlimit_teamcity/TeamCity/Agents/20260211180011agent_172.17.0.220_1/work/60183217f6ee2a9c/output/be/lib/doris_be 8# google::LogMessageFatal::~LogMessageFatal() in /home/work/unlimit_teamcity/TeamCity/Agents/20260211180011agent_172.17.0.220_1/work/60183217f6ee2a9c/output/be/lib/doris_be 9# doris::vectorized::VExpr::filter_column_with_selector(doris::COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn> const&, doris::vectorized::PODArray<unsigned int, 4096ul, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, false>, 16ul, 15ul> const*, unsigned long) at ../src/vec/exprs/vexpr.h:269 10# doris::vectorized::VSlotRef::execute_column(doris::vectorized::VExprContext*, doris::vectorized::Block const*, doris::vectorized::PODArray<unsigned int, 4096ul, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, false>, 16ul, 15ul>*, unsigned long, doris::COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&) const at /root/doris/be/build_ASAN/../src/vec/exprs/vslot_ref.cpp:96 11# doris::vectorized::VCastExpr::execute_column(doris::vectorized::VExprContext*, doris::vectorized::Block const*, doris::vectorized::PODArray<unsigned int, 4096ul, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, false>, 16ul, 15ul>*, unsigned long, doris::COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&) const at /root/doris/be/build_ASAN/../src/vec/exprs/vcast_expr.cpp:117 12# doris::vectorized::VectorizedFnCall::_do_execute(doris::vectorized::VExprContext*, doris::vectorized::Block const*, doris::vectorized::PODArray<unsigned int, 4096ul, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, false>, 16ul, 15ul>*, unsigned long, doris::COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, doris::COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>*) const at /root/doris/be/build_ASAN/../src/vec/exprs/vectorized_fn_call.cpp:242 13# doris::vectorized::VectorizedFnCall::execute_column(doris::vectorized::VExprContext*, doris::vectorized::Block const*, doris::vectorized::PODArray<unsigned int, 4096ul, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, false>, 16ul, 15ul>*, unsigned long, doris::COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&) const at /root/doris/be/build_ASAN/../src/vec/exprs/vectorized_fn_call.cpp:302 14# doris::vectorized::VectorizedFnCall::_do_execute(doris::vectorized::VExprContext*, doris::vectorized::Block const*, doris::vectorized::PODArray<unsigned int, 4096ul, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, false>, 16ul, 15ul>*, unsigned long, doris::COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, doris::COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>*) const at /root/doris/be/build_ASAN/../src/vec/exprs/vectorized_fn_call.cpp:242 15# doris::vectorized::VectorizedFnCall::execute_column(doris::vectorized::VExprContext*, doris::vectorized::Block const*, doris::vectorized::PODArray<unsigned int, 4096ul, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, false>, 16ul, 15ul>*, unsigned long, doris::COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&) const at /root/doris/be/build_ASAN/../src/vec/exprs/vectorized_fn_call.cpp:302 16# doris::vectorized::VLambdaFunctionExpr::execute_column(doris::vectorized::VExprContext*, doris::vectorized::Block const*, doris::vectorized::PODArray<unsigned int, 4096ul, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, false>, 16ul, 15ul>*, unsigned long, doris::COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&) const at ../src/vec/exprs/vlambda_function_expr.h:48 17# doris::vectorized::ArrayMapFunction::execute(doris::vectorized::VExprContext*, doris::vectorized::Block const*, doris::vectorized::PODArray<unsigned int, 4096ul, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, false>, 16ul, 15ul>*, unsigned long, doris::COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, std::shared_ptr<doris::vectorized::IDataType const> const&, std::vector<std::shared_ptr<doris::vectorized::VExpr>, std::allocator<std::shared_ptr<doris::vectorized::VExpr> > > const&) const at /root/doris/be/build_ASAN/../src/vec/exprs/lambda_function/varray_map_function.cpp:288 18# doris::vectorized::VLambdaFunctionCallExpr::execute_column(doris::vectorized::VExprContext*, doris::vectorized::Block const*, doris::vectorized::PODArray<unsigned int, 4096ul, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, false>, 16ul, 15ul>*, unsigned long, doris::COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&) const at ../src/vec/exprs/vlambda_function_call_expr.h:69 19# doris::vectorized::VectorizedFnCall::_do_execute(doris::vectorized::VExprContext*, doris::vectorized::Block const*, doris::vectorized::PODArray<unsigned int, 4096ul, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, false>, 16ul, 15ul>*, unsigned long, doris::COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, doris::COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>*) const at /root/doris/be/build_ASAN/../src/vec/exprs/vectorized_fn_call.cpp:242 20# doris::vectorized::VectorizedFnCall::execute_column(doris::vectorized::VExprContext*, doris::vectorized::Block const*, doris::vectorized::PODArray<unsigned int, 4096ul, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, false>, 16ul, 15ul>*, unsigned long, doris::COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&) const at /root/doris/be/build_ASAN/../src/vec/exprs/vectorized_fn_call.cpp:302 21# doris::vectorized::VectorizedFnCall::_do_execute(doris::vectorized::VExprContext*, doris::vectorized::Block const*, doris::vectorized::PODArray<unsigned int, 4096ul, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, false>, 16ul, 15ul>*, unsigned long, doris::COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, doris::COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>*) const at /root/doris/be/build_ASAN/../src/vec/exprs/vectorized_fn_call.cpp:242 22# doris::vectorized::VectorizedFnCall::execute_column(doris::vectorized::VExprContext*, doris::vectorized::Block const*, doris::vectorized::PODArray<unsigned int, 4096ul, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, false>, 16ul, 15ul>*, unsigned long, doris::COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&) const at /root/doris/be/build_ASAN/../src/vec/exprs/vectorized_fn_call.cpp:302 23# doris::vectorized::VExprContext::short_circuit_execute_conjuncts(std::vector<std::shared_ptr<doris::vectorized::VExprContext>, std::allocator<std::shared_ptr<doris::vectorized::VExprContext> > > const&, bool, doris::vectorized::Block const*, doris::vectorized::PODArray<unsigned char, 4096ul, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, false>, 16ul, 15ul>*, bool*) at /root/doris/be/build_ASAN/../src/vec/exprs/vexpr_context.cpp:265 24# doris::vectorized::VExprContext::execute_conjuncts(std::vector<std::shared_ptr<doris::vectorized::VExprContext>, std::allocator<std::shared_ptr<doris::vectorized::VExprContext> > > const&, std::vector<doris::vectorized::PODArray<unsigned char, 4096ul, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, false>, 16ul, 15ul>*, std::allocator<doris::vectorized::PODArray<unsigned char, 4096ul, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, false>, 16ul, 15ul>*> > const*, bool, doris::vectorized::Block const*, doris::vectorized::PODArray<unsigned char, 4096ul, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, false>, 16ul, 15ul>*, bool*) in /home/work/unlimit_teamcity/TeamCity/Agents/20260211180011agent_172.17.0.220_1/work/60183217f6ee2a9c/output/be/lib/doris_be 25# doris::vectorized::VExprContext::execute_conjuncts_and_filter_block(std::vector<std::shared_ptr<doris::vectorized::VExprContext>, std::allocator<std::shared_ptr<doris::vectorized::VExprContext> > > const&, doris::vectorized::Block*, std::vector<unsigned int, std::allocator<unsigned int> >&, int) at /root/doris/be/build_ASAN/../src/vec/exprs/vexpr_context.cpp:410 26# doris::vectorized::VExprContext::filter_block(std::vector<std::shared_ptr<doris::vectorized::VExprContext>, std::allocator<std::shared_ptr<doris::vectorized::VExprContext> > > const&, doris::vectorized::Block*, unsigned long) at /root/doris/be/build_ASAN/../src/vec/exprs/vexpr_context.cpp:217 27# doris::vectorized::Scanner::_filter_output_block(doris::vectorized::Block*) at /root/doris/be/build_ASAN/../src/vec/exec/scan/scanner.cpp:176 28# doris::vectorized::Scanner::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /root/doris/be/build_ASAN/../src/vec/exec/scan/scanner.cpp:153 29# doris::vectorized::Scanner::get_block_after_projects(doris::RuntimeState*, doris::vectorized::Block*, bool*) in /home/work/unlimit_teamcity/TeamCity/Agents/20260211180011agent_172.17.0.220_1/work/60183217f6ee2a9c/output/be/lib/doris_be 30# doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>) at /root/doris/be/build_ASAN/../src/vec/exec/scan/scanner_scheduler.cpp:177 31# std::_Function_handler<bool (), doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()#1}>::_M_invoke(std::_Any_data const&) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:292 32# doris::vectorized::ScannerSplitRunner::process_for(std::chrono::duration<long, std::ratio<1l, 1000000000l> >) at /root/doris/be/build_ASAN/../src/vec/exec/scan/scanner_scheduler.cpp:415 33# doris::vectorized::PrioritizedSplitRunner::process() in /home/work/unlimit_teamcity/TeamCity/Agents/20260211180011agent_172.17.0.220_1/work/60183217f6ee2a9c/output/be/lib/doris_be 34# doris::vectorized::TimeSharingTaskExecutor::_dispatch_thread() at /root/doris/be/build_ASAN/../src/vec/exec/executor/time_sharing/time_sharing_task_executor.cpp:568 35# doris::Thread::supervise_thread(void*) at /root/doris/be/build_ASAN/../src/util/thread.cpp:461 36# asan_thread_start(void*) in /home/work/unlimit_teamcity/TeamCity/Agents/20260211180011agent_172.17.0.220_1/work/60183217f6ee2a9c/output/be/lib/doris_be 37# start_thread at /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:478 38# __clone at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97 ```

…n_udf_support` is not set (apache#61196) When `enable_python_udf_support` is not set to true, a null pointer dereference occurs when invoking helper commands (`SHOW PYTHON VERSION/ PACKAGES IN`). ```text /home/zcp/repo_center/doris_master/doris/be/src/udf/python/python_env.cpp:305:38: runtime error: member call on null pointer of type 'doris::PythonEnvScanner' #0 0x559edede162c in doris::PythonVersionManager::env_infos_to_thrift() const /home/zcp/repo_center/doris_master/doris/be/src/udf/python/python_env.cpp:305:38 #1 0x559ede63acd4 in doris::BaseBackendService::get_python_envs(std::vector>&) /home/zcp/repo_center/doris_master/doris/be/src/service/backend_service.cpp:1316:47 apache#2 0x559edf99796a in doris::BackendServiceProcessor::process_get_python_envs(int, apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, void*) /home/zcp/repo_center/doris_master/doris/gensrc/build/gen_cpp/BackendService.cpp:7789:13 apache#3 0x559edf94a69c in doris::BackendServiceProcessor::dispatchCall(apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, std::__cxx11::basic_string, std::allocator> const&, int, void*) /home/zcp/repo_center/doris_master/doris/gensrc/build/gen_cpp/BackendService.cpp:6466:3 apache#4 0x559eac6bad06 in apache::thrift::TDispatchProcessor::process(std::shared_ptr, std::shared_ptr, void*) /home/zcp/repo_center/doris_master/doris/thirdparty/installed/include/thrift/TDispatchProcessor.h:121:12 apache#5 0x559ee47c071a in apache::thrift::server::TConnectedClient::run() (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x8a1c871a) apache#6 0x559ee47c1a45 in apache::thrift::server::TThreadedServer::TConnectedClientRunner::run() (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x8a1c9a45) apache#7 0x559ee47c557f in apache::thrift::concurrency::Thread::threadMain(std::shared_ptr) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x8a1cd57f) apache#8 0x559ee47c52a5 in void std::__invoke_impl), std::shared_ptr>(std::__invoke_other, void (*&&)(std::shared_ptr), std::shared_ptr&&) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x8a1cd2a5) apache#9 0x559ee47c521c in std::__invoke_result), std::shared_ptr>::type std::__invoke), std::shared_ptr>(void (*&&)(std::shared_ptr), std::shared_ptr&&) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x8a1cd21c) apache#10 0x559ee47c51f1 in void std::thread::_Invoker), std::shared_ptr>>::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x8a1cd1f1) apache#11 0x559ee47c51b4 in std::thread::_Invoker), std::shared_ptr>>::operator()() (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x8a1cd1b4) apache#12 0x559ee47c4fd8 in std::thread::_State_impl), std::shared_ptr>>>::_M_run() (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x8a1ccfd8) apache#13 0x559ef61b5f3f in execute_native_thread_routine archive64.c apache#14 0x559eac535d26 in asan_thread_start(void*) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x51f3dd26) apache#15 0x7fcba62bfac2 in start_thread nptl/pthread_create.c:442:8 apache#16 0x7fcba635184f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /home/zcp/repo_center/doris_master/doris/be/src/udf/python/python_env.cpp:305:38 ``` now: ```text Doris> show python packages in '3.19.18'; ERROR 1105 (HY000): errCode = 2, detailMessage = Failed to get python packages from any backend: [E42] Set 'python_venv_interpreter_paths' in be.conf to enable PythonUDF feature Doris> show python versions; ERROR 1105 (HY000): errCode = 2, detailMessage = Failed to get python envs from any backend: [E-236] Set 'python_venv_interpreter_paths' in be.conf to enable PythonUDF feature ```

… exit (apache#61040) ### What problem does this PR solve? The SyncPoint singleton was being destroyed during program exit while background bthread timers were still accessing it, causing heap-use-after-free. Changed get_instance() to return a never-destructed heap-allocated pointer instead of a stack-allocated static object to prevent destruction order issues. This is a standard pattern for singletons accessed by background threads. ``` 11:47:40 #0 0x5582a963153d in operator new(unsigned long) (/root/doris/cloud/ut_build_ASAN/test/recycler_test+0x122b53d) (BuildId: 1191affa24589f52) 11:47:40 #1 0x5582ab0aa52f in doris::SyncPoint::SyncPoint() /root/doris/common/cpp/sync_point.cpp:77:9 11:47:40 apache#2 0x5582ab0aa46b in doris::SyncPoint::get_instance() /root/doris/common/cpp/sync_point.cpp:73:20 11:47:40 apache#3 0x5582aae33ee8 in doris::cloud::memkv::Transaction::get(std::basic_string_view<char, std::char_traits<char>>, std::basic_string_view<char, std::char_traits<char>>, std::unique_ptr<doris::cloud::RangeGetIterator, std::default_delete<doris::cloud::RangeGetIterator>>*, bool, int) /root/doris/cloud/src/meta-store/mem_txn_kv.cpp:276:5 11:47:40 apache#4 0x5582a968f49f in doris::cloud::txn_get(doris::cloud::TxnKv*, std::basic_string_view<char, std::char_traits<char>>, std::basic_string_view<char, std::char_traits<char>>, std::unique_ptr<doris::cloud::RangeGetIterator, std::default_delete<doris::cloud::RangeGetIterator>>&) /root/doris/cloud/src/recycler/recycler.cpp:99:18 11:47:40 apache#5 0x5582a9657090 in doris::cloud::InstanceRecycler::scan_and_recycle(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::basic_string_view<char, std::char_traits<char>>, std::function<int (std::basic_string_view<char, std::char_traits<char>>, std::basic_string_view<char, std::char_traits<char>>)>, std::function<int ()>) /root/doris/cloud/src/recycler/recycler.cpp:2976:23 11:47:40 apache#6 0x5582a9686153 in doris::cloud::InstanceRecycler::recycle_rowsets() /root/doris/cloud/src/recycler/recycler.cpp:2587:15 11:47:40 apache#7 0x5582a96a8d47 in doris::cloud::RecyclerTest_recycle_empty_Test::TestBody() /root/doris/cloud/test/recycler_test.cpp:1136:5 ```

…tadata (apache#61342) ### What problem does this PR solve? When FE starts with -r (metadata_failure_recovery) or without a checkpoint image, it replays all journal logs from the beginning. Old journal entries may contain PrimaryKeyConstraint objects that were serialized before the fields foreignTables, foreignTableNameStrs, or foreignTableInfos were added to the class. Gson uses Unsafe to instantiate objects during deserialization, bypassing the constructor and its field initializers. If a field is absent from the JSON, Gson leaves it as null rather than using the declared initializer (e.g. `= new HashSet<>()`). Calling .isEmpty() or any method on a null collection then throws a NullPointerException. Fix: remove `final` from the three collection fields (so they can be reassigned in gsonPostProcess) and add null-guards at the top of gsonPostProcess() to initialize them to empty collections when absent. Impact on -r mode: when FE is started with --metadata_failure_recovery it replays every journal from journal #1, encountering the oldest serialized PrimaryKeyConstraint objects. Without this fix, FE always crashes during -r recovery if any PrimaryKeyConstraint was persisted before these fields existed.

…free. (apache#61138) ### What problem does this PR solve? Related PR: apache/doris-thirdparty#383 Problem Summary: fix coredump: ``` #0 0x5561f1d46f12 in operator delete(void*, unsigned long) (/mnt/disk2/tengjianping/doris-master/output/be/lib/doris_be+0x50ae4f12) #1 0x55622c94b454 in std::__new_allocator<std::shared_ptr<orc::ExpressionTree>>::deallocate(std::shared_ptr<orc::ExpressionTree>*, unsigned long) (/mnt/disk2/tengjianping/doris-master/output/be/lib/doris_be+0x8b6e9454) apache#2 0x55622c94b41e in std::_Deque_base<std::shared_ptr<orc::ExpressionTree>, std::allocator<std::shared_ptr<orc::ExpressionTree>>>::_M_deallocate_node(std::shared_ptr<orc::ExpressionTree>*) (/mnt/disk2/tengjianping/doris-master/output/be/lib/doris_be+0x8b6e941e) apache#3 0x55622c950681 in std::deque<std::shared_ptr<orc::ExpressionTree>, std::allocator<std::shared_ptr<orc::ExpressionTree>>>::_M_pop_front_aux() (/mnt/disk2/tengjianping/doris-master/output/be/lib/doris_be+0x8b6ee681) apache#4 0x55622c94868e in std::deque<std::shared_ptr<orc::ExpressionTree>, std::allocator<std::shared_ptr<orc::ExpressionTree>>>::pop_front() (/mnt/disk2/tengjianping/doris-master/output/be/lib/doris_be+0x8b6e668e) apache#5 0x55622c947d79 in orc::rewriteLeaves(std::shared_ptr<orc::ExpressionTree>, unsigned long*) SearchArgument.cc apache#6 0x55622c947809 in orc::SearchArgumentBuilderImpl::build() (/mnt/disk2/tengjianping/doris-master/output/be/lib/doris_be+0x8b6e5809) apache#7 0x5562195209a6 in doris::vectorized::OrcReader::_init_search_argument(std::vector<std::shared_ptr<doris::vectorized::VExpr>, std::allocator<std::shared_ptr<doris::vectorized::VExpr>>> const&) /mnt/disk2/tengjianping/doris-master/be/src/vec/exec/format/orc/vorc_reader.cpp:1072:27 apache#8 0x556219527175 in doris::vectorized::OrcReader::set_fill_columns(std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::SlotDescriptor const*>, std::hash<std::__cxx11::basic_string<char, ```

``` ==1472432==ERROR: AddressSanitizer: heap-use-after-free on address 0x7bc7535fba50 at pc 0x557ec5178739 bp 0x7b1743746d40 sp 0x7b1743746500 WRITE of size 104 at 0x7bc7535fba50 thread T2 #0 0x557ec5178738 in memset ~/workspace/llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors_memintrinsics.inc:87:3 #1 0x7f17566e64fc in fdb_transaction_set_option (libfdb_c.so+0xc014fc) (BuildId: 854ec989d11d3c183aa4963a6d758cfc2b571b07) apache#2 0x7f17566e2e47 in fdb_transaction_set_option (libfdb_c.so+0xbfde47) (BuildId: 854ec989d11d3c183aa4963a6d758cfc2b571b07) apache#3 0x7f17566e4ec5 in fdb_transaction_set_option (libfdb_c.so+0xbffec5) (BuildId: 854ec989d11d3c183aa4963a6d758cfc2b571b07) apache#4 0x7f17564f80ee in fdb_transaction_set_option (libfdb_c.so+0xa130ee) (BuildId: 854ec989d11d3c183aa4963a6d758cfc2b571b07) apache#5 0x7f1755e55b71 in fdb_transaction_set_option (libfdb_c.so+0x370b71) (BuildId: 854ec989d11d3c183aa4963a6d758cfc2b571b07) apache#6 0x7f1755e2b2e9 in fdb_run_network (libfdb_c.so+0x3462e9) (BuildId: 854ec989d11d3c183aa4963a6d758cfc2b571b07) apache#7 0x557ec6595a2b in doris::cloud::fdb::Network::init()::::operator()() const ~/workspace/doris/cloud/ut_build_ASAN/../src/meta-store/txn_kv.cpp:468:28 apache#8 0x557ec65958e6 in void std::__invoke_impl<void, doris::cloud::fdb::Network::init()::>(std::__invoke_other, doris::cloud::fdb::Network::init()::&&) /usr/lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14 apache#9 0x557ec6595886 in std::__invoke_result<doris::cloud::fdb::Network::init()::>::type std::__invoke<doris::cloud::fdb::Network::init()::>(doris::cloud::fdb::Network::init()::&&) /usr/lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:96:14 apache#10 0x557ec659584e in void std::thread::_Invoker<std::tuple<doris::cloud::fdb::Network::init()::>>::_M_invoke<0ul>(std::_Index_tuple<0ul>) /usr/lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_thread.h:292:13 apache#11 0x557ec6595816 in std::thread::_Invoker<std::tuple<doris::cloud::fdb::Network::init()::>>::operator()() /usr/lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_thread.h:299:11 apache#12 0x557ec659571a in std::thread::_State_impl<std::thread::_Invoker<std::tuple<doris::cloud::fdb::Network::init()::>>>::_M_run() /usr/lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_thread.h:244:13 apache#13 0x557ec7756e43 in execute_native_thread_routine InstrProfilingUtil.c apache#14 0x557ec50b1a07 in asan_thread_start(void*) ~/workspace/llvm-project/compiler-rt/lib/asan/asan_interceptors.cpp:239:43 apache#15 0x7f17545caaa3 in start_thread nptl/pthread_create.c:447:8 apache#16 0x7f1754657a63 in clone misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:100 ```

…1510) ### What problem does this PR solve? ```text *** SIGSEGV address not mapped to object (@0x20) received by PID 2309810 (TID 2312493 OR 0x7f39af4bf640) from PID 32; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/gcp-user/selectdb-core/be/src/common/signal_handler.h:421 1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /opt/jdk/lib/server/libjvm.so 2# JVM_handle_linux_signal in /opt/jdk/lib/server/libjvm.so 3# 0x00007F3E7723FC30 in /lib64/libc.so.6 4# std::_Hashtable<doris::StringRef, std::pair<doris::StringRef const, int>, std::allocator<std::pair<doris::StringRef const, int> >, std::__detail::_Select1st, std::equal_to<doris::StringRef>, doris::StringRefHash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::find(doris::StringRef const&) const at /home/gcp-user/tools/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/hashtable.h:1587 5# doris::TabletSchema::field_index(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const at /home/gcp-user/selectdb-core/be/src/olap/tablet_schema.cpp:1304 6# doris::vectorized::NewOlapScanner::_init_tablet_reader_params(std::vector<doris::OlapScanRange*, std::allocator<doris::OlapScanRange*> > const&, std::vector<doris::TCondition, std::allocator<doris::TCondition> > const&, doris::pipeline::FilterPredicates const&, std::vector<doris::FunctionFilter, std::allocator<doris::FunctionFilter> > const&) at /home/gcp-user/selectdb-core/be/src/vec/exec/scan/new_olap_scanner.cpp:413 7# doris::vectorized::NewOlapScanner::init() in /opt/selectdb/4.0.7.2025101510/be/lib/doris_be 8# doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>) at /home/gcp-user/selectdb-core/be/src/vec/exec/scan/scanner_scheduler.cpp:221 9# std::_Function_handler<void (), doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::{lambda()#1}>::_M_invoke(std::_Any_data const&) at /home/gcp-user/tools/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291 10# doris::ThreadPool::dispatch_thread() at /home/gcp-user/selectdb-core/be/src/util/threadpool.cpp:609 11# doris::Thread::supervise_thread(void*) at /home/gcp-user/selectdb-core/be/src/util/thread.cpp:499 12# start_thread in /lib64/libc.so.6 13# __GI___clone3 in /lib64/libc.so.6 ``` Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test  - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [ ] No. - [ ] Yes.  - Does this need documentation? - [ ] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

### What problem does this PR solve? Fix memory corruption in PredicateColumnType::insert_duplicate_fields for string types Fixed a bug in the pointer arithmetic when inserting duplicate string values. The original code used dst += i * str.size() which incorrectly multiplied by the loop index, causing: - Iteration 0: no advance (correct by accident) - Iteration 1: advance by 1×size (correct) - Iteration 2: advance by 2×size (wrong - overwrites beyond allocated memory) This resulted in memory corruption and incorrect string data when inserting more than 2 duplicates. The fix changes to dst += str.size() to correctly advance the pointer by one string length per iteration. Added unit tests covering: - String type with multiple duplicate insertions - Integer type duplicates - LargeInt (Int128) type duplicates Introduced by apache#60530 ==13605==ERROR: AddressSanitizer: use-after-poison on address 0x7d639d26bc30 at pc 0x55e8b71df91e bp 0x7b4e8fc1e510 sp 0x7b4e8fc1dcd0 WRITE of size 6 at 0x7d639d26bc30 thread T620 (ls_normal [work) #0 0x55e8b71df91d in __asan_memcpy (/mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be+0x2205791d) #1 0x55e8cce8b1ab in doris::PredicateColumnType<(doris::PrimitiveType)23>::insert_duplicate_fields(doris::Field const&, unsigned long) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/core/column/predicate_column.h:327:17 apache#2 0x55e8b7db4e83 in doris::ColumnNullable::insert_duplicate_fields(doris::Field const&, unsigned long) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/core/column/column_nullable.cpp:357:29 apache#3 0x55e8ccf8ba2c in doris::segment_v2::DefaultValueColumnIterator::_insert_many_default(doris::COW<doris::IColumn>::mutable_ptr<doris::IColumn>&, unsigned long) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/segment/column_reader.cpp:2167:14 apache#4 0x55e8ccf3c705 in doris::segment_v2::DefaultValueColumnIterator::next_batch(unsigned long*, doris::COW<doris::IColumn>::mutable_ptr<doris::IColumn>&, bool*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/segment/column_reader.cpp:2152:5 apache#5 0x55e8cd4f9313 in doris::segment_v2::ColumnIterator::next_batch(unsigned long*, doris::COW<doris::IColumn>::mutable_ptr<doris::IColumn>&) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/segment/column_reader.h:330:16 apache#6 0x55e8cd4f9313 in doris::segment_v2::SegmentIterator::_read_columns_by_index(unsigned int, unsigned short&) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/segment/segment_iterator.cpp:2291:13 apache#7 0x55e8cd50621b in doris::segment_v2::SegmentIterator::_next_batch_internal(doris::Block*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/segment/segment_iterator.cpp:2643:5 apache#8 0x55e8cd4a0b82 in doris::segment_v2::SegmentIterator::next_batch(doris::Block*)::$_0::operator()() const /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/segment/segment_iterator.cpp:2519:9 apache#9 0x55e8cd4a0b82 in doris::segment_v2::SegmentIterator::next_batch(doris::Block*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/segment/segment_iterator.cpp:2518:19 apache#10 0x55e8cd37cca1 in doris::segment_v2::LazyInitSegmentIterator::next_batch(doris::Block*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/segment/lazy_init_segment_iterator.h:46:33 apache#11 0x55e8cccc0b05 in doris::Status doris::BetaRowsetReader::_next_batch<doris::Block>(doris::Block*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/rowset/beta_rowset_reader.h:113:35 apache#12 0x55e8ccc98668 in doris::BetaRowsetReader::next_batch(doris::Block*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/rowset/beta_rowset_reader.h:56:55 apache#13 0x55e8ca62ee8f in doris::VCollectIterator::Level0Iterator::_refresh() /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/iterator/vcollect_iterator.h apache#14 0x55e8ca611036 in doris::VCollectIterator::Level0Iterator::refresh_current_row() /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/iterator/vcollect_iterator.cpp:523:24 apache#15 0x55e8ca601a08 in doris::VCollectIterator::Level0Iterator::ensure_first_row_ref() /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/iterator/vcollect_iterator.cpp:496:14 apache#16 0x55e8ca604ea2 in doris::VCollectIterator::Level1Iterator::ensure_first_row_ref() /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/iterator/vcollect_iterator.cpp:713:27 apache#17 0x55e8ca608b20 in doris::VCollectIterator::build_heap(std::vector<std::shared_ptr<doris::RowsetReader>, std::allocator<std::shared_ptr<doris::RowsetReader>>>&) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/iterator/vcollect_iterator.cpp:189:9 apache#18 0x55e8ca5dc1ae in doris::BlockReader::_init_collect_iter(doris::TabletReader::ReaderParams const&) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/iterator/block_reader.cpp:152:9 apache#19 0x55e8ca5d7b9d in doris::BlockReader::init(doris::TabletReader::ReaderParams const&) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/iterator/block_reader.cpp:226:19 apache#20 0x55e8d02df6d6 in doris::OlapScanner::_open_impl(doris::RuntimeState*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/olap_scanner.cpp:317:32 apache#21 0x55e8d02cd697 in doris::Scanner::open(doris::RuntimeState*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/scanner.h:80:16 apache#22 0x55e8d02b576a in doris::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/scanner_scheduler.cpp:185:5 apache#23 0x55e8d02bf603 in doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/scanner_scheduler.cpp:75:17 apache#24 0x55e8d02bf603 in doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()::operator()() const /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/scanner_scheduler.cpp:74:27 apache#25 0x55e8d02bf603 in bool std::__invoke_impl<bool, doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()&>(std::__invoke_other, doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:63:14 apache#26 0x55e8d02bf603 in std::enable_if<is_invocable_r_v<bool, doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()&>, bool>::type std::__invoke_r<bool, doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()&>(doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:116:9 apache#27 0x55e8d02bf603 in std::_Function_handler<bool (), doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()>::_M_invoke(std::_Any_data const&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:292:9 apache#28 0x55e8d02bed85 in std::function<bool ()>::operator()() const /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593:9 apache#29 0x55e8d02bed85 in doris::ScannerSplitRunner::process_for(std::chrono::duration<long, std::ratio<1l, 1000000000l>>) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/scanner_scheduler.cpp:425:25 apache#30 0x55e8d02aa844 in doris::PrioritizedSplitRunner::process() /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/task_executor/time_sharing/prioritized_split_runner.cpp:102:35 apache#31 0x55e8d0272674 in doris::TimeSharingTaskExecutor::_dispatch_thread() /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/task_executor/time_sharing/time_sharing_task_executor.cpp:556:77 apache#32 0x55e8d3fa8ad1 in std::function<void ()>::operator()() const /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593:9 apache#33 0x55e8d3fa8ad1 in doris::Thread::supervise_thread(void*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/util/thread.cpp:460:5 apache#34 0x55e8b71ddd26 in asan_thread_start(void*) (/mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be+0x22055d26) apache#35 0x7f539b835ac2 in start_thread nptl/pthread_create.c:442:8 apache#36 0x7f539b8c78cf misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 0x7d639d26bc30 is located 816 bytes inside of 4096-byte region [0x7d639d26b900,0x7d639d26c900) allocated by thread T620 (ls_normal [work) here: #0 0x55e8b71e1b34 in malloc (/mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be+0x22059b34) #1 0x55e8b78d2140 in doris::DefaultMemoryAllocator::malloc(unsigned long) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/core/allocator.h:90:55 apache#2 0x55e8b78d2140 in doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, true>::alloc(unsigned long, unsigned long) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/core/allocator.cpp:337:23 apache#3 0x55e8b7ccdfaf in doris::Arena::Chunk::Chunk(unsigned long, doris::Arena::Chunk*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/core/arena.h:59:63 apache#4 0x55e8b7ccdad9 in doris::Arena::_init_head_if_needed() /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/core/arena.h:131:24 apache#5 0x55e8b7ccd758 in doris::Arena::alloc(unsigned long) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/core/arena.h:148:9 apache#6 0x55e8cce8b118 in doris::PredicateColumnType<(doris::PrimitiveType)23>::insert_duplicate_fields(doris::Field const&, unsigned long) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/core/column/predicate_column.h:325:32 apache#7 0x55e8b7db4e83 in doris::ColumnNullable::insert_duplicate_fields(doris::Field const&, unsigned long) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/core/column/column_nullable.cpp:357:29 apache#8 0x55e8ccf8ba2c in doris::segment_v2::DefaultValueColumnIterator::_insert_many_default(doris::COW<doris::IColumn>::mutable_ptr<doris::IColumn>&, unsigned long) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/segment/column_reader.cpp:2167:14 apache#9 0x55e8ccf3c705 in doris::segment_v2::DefaultValueColumnIterator::next_batch(unsigned long*, doris::COW<doris::IColumn>::mutable_ptr<doris::IColumn>&, bool*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/segment/column_reader.cpp:2152:5 apache#10 0x55e8cd4f9313 in doris::segment_v2::ColumnIterator::next_batch(unsigned long*, doris::COW<doris::IColumn>::mutable_ptr<doris::IColumn>&) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/segment/column_reader.h:330:16 apache#11 0x55e8cd4f9313 in doris::segment_v2::SegmentIterator::_read_columns_by_index(unsigned int, unsigned short&) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/segment/segment_iterator.cpp:2291:13 apache#12 0x55e8cd50621b in doris::segment_v2::SegmentIterator::_next_batch_internal(doris::Block*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/segment/segment_iterator.cpp:2643:5 apache#13 0x55e8cd4a0b82 in doris::segment_v2::SegmentIterator::next_batch(doris::Block*)::$_0::operator()() const /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/segment/segment_iterator.cpp:2519:9 apache#14 0x55e8cd4a0b82 in doris::segment_v2::SegmentIterator::next_batch(doris::Block*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/segment/segment_iterator.cpp:2518:19 apache#15 0x55e8cd37cca1 in doris::segment_v2::LazyInitSegmentIterator::next_batch(doris::Block*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/segment/lazy_init_segment_iterator.h:46:33 apache#16 0x55e8cccc0b05 in doris::Status doris::BetaRowsetReader::_next_batch<doris::Block>(doris::Block*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/rowset/beta_rowset_reader.h:113:35 apache#17 0x55e8ccc98668 in doris::BetaRowsetReader::next_batch(doris::Block*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/rowset/beta_rowset_reader.h:56:55 apache#18 0x55e8ca62ee8f in doris::VCollectIterator::Level0Iterator::_refresh() /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/iterator/vcollect_iterator.h apache#19 0x55e8ca611036 in doris::VCollectIterator::Level0Iterator::refresh_current_row() /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/iterator/vcollect_iterator.cpp:523:24 apache#20 0x55e8ca601a08 in doris::VCollectIterator::Level0Iterator::ensure_first_row_ref() /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/iterator/vcollect_iterator.cpp:496:14 apache#21 0x55e8ca604ea2 in doris::VCollectIterator::Level1Iterator::ensure_first_row_ref() /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/iterator/vcollect_iterator.cpp:713:27 apache#22 0x55e8ca608b20 in doris::VCollectIterator::build_heap(std::vector<std::shared_ptr<doris::RowsetReader>, std::allocator<std::shared_ptr<doris::RowsetReader>>>&) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/iterator/vcollect_iterator.cpp:189:9 apache#23 0x55e8ca5dc1ae in doris::BlockReader::_init_collect_iter(doris::TabletReader::ReaderParams const&) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/iterator/block_reader.cpp:152:9 apache#24 0x55e8ca5d7b9d in doris::BlockReader::init(doris::TabletReader::ReaderParams const&) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/iterator/block_reader.cpp:226:19 apache#25 0x55e8d02df6d6 in doris::OlapScanner::_open_impl(doris::RuntimeState*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/olap_scanner.cpp:317:32 apache#26 0x55e8d02cd697 in doris::Scanner::open(doris::RuntimeState*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/scanner.h:80:16 apache#27 0x55e8d02b576a in doris::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/scanner_scheduler.cpp:185:5 apache#28 0x55e8d02bf603 in doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/scanner_scheduler.cpp:75:17 apache#29 0x55e8d02bf603 in doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()::operator()() const /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/scanner_scheduler.cpp:74:27 apache#30 0x55e8d02bf603 in bool std::__invoke_impl<bool, doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()&>(std::__invoke_other, doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:63:14 apache#31 0x55e8d02bf603 in std::enable_if<is_invocable_r_v<bool, doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()&>, bool>::type std::__invoke_r<bool, doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()&>(doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:116:9 apache#32 0x55e8d02bf603 in std::_Function_handler<bool (), doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()>::_M_invoke(std::_Any_data const&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:292:9 apache#33 0x55e8d02bed85 in std::function<bool ()>::operator()() const /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593:9 apache#34 0x55e8d02bed85 in doris::ScannerSplitRunner::process_for(std::chrono::duration<long, std::ratio<1l, 1000000000l>>) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/scanner_scheduler.cpp:425:25 apache#35 0x55e8d02aa844 in doris::PrioritizedSplitRunner::process() /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/task_executor/time_sharing/prioritized_split_runner.cpp:102:35 apache#36 0x55e8d0272674 in doris::TimeSharingTaskExecutor::_dispatch_thread() /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/task_executor/time_sharing/time_sharing_task_executor.cpp:556:77 apache#37 0x55e8d3fa8ad1 in std::function<void ()>::operator()() const /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593:9 apache#38 0x55e8d3fa8ad1 in doris::Thread::supervise_thread(void*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/util/thread.cpp:460:5 Thread T620 (ls_normal [work) created by T0 here: #0 0x55e8b71c5e91 in pthread_create (/mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be+0x2203de91) #1 0x55e8d3fa7b8a in doris::Thread::start_thread(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::function<void ()> const&, std::shared_ptr<doris::Thread>*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/util/thread.cpp:412:15 apache#2 0x55e8d02849fd in doris::Status doris::Thread::create<void (doris::TimeSharingTaskExecutor::*)(), doris::TimeSharingTaskExecutor*>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, void (doris::TimeSharingTaskExecutor::* const&)(), doris::TimeSharingTaskExecutor* const&, std::shared_ptr<doris::Thread>*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/util/thread.h:47:16 apache#3 0x55e8d026e6b3 in doris::TimeSharingTaskExecutor::_create_thread() /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/task_executor/time_sharing/time_sharing_task_executor.cpp:671:12 apache#4 0x55e8d026b628 in doris::TimeSharingTaskExecutor::_try_create_thread(int, std::lock_guard<std::mutex>&) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/task_executor/time_sharing/time_sharing_task_executor.cpp:360:25 apache#5 0x55e8d026967a in doris::TimeSharingTaskExecutor::init() /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/task_executor/time_sharing/time_sharing_task_executor.cpp:221:27 apache#6 0x55e8d025a85a in doris::TaskExecutorSimplifiedScanScheduler::start(int, int, int, int) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/scanner_scheduler.h:288:9 apache#7 0x55e8d368d649 in doris::WorkloadGroup::upsert_thread_pool_no_lock(doris::WorkloadGroupInfo*, std::shared_ptr<doris::CgroupCpuCtl>) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/runtime/workload_group/workload_group.cpp:571:38 apache#8 0x55e8d368f14d in doris::WorkloadGroup::upsert_task_scheduler(doris::WorkloadGroupInfo*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/runtime/workload_group/workload_group.cpp:667:12 apache#9 0x55e8d36bd223 in doris::WorkloadGroupMgr::create_internal_wg() /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/runtime/workload_group/workload_group_manager.cpp:900:5 apache#10 0x55e8d32bbd3d in doris::ExecEnv::_create_internal_workload_group() /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/runtime/exec_env_init.cpp:475:5 apache#11 0x55e8d32b27fc in doris::ExecEnv::_init(std::vector<doris::StorePath, std::allocator<doris::StorePath>> const&, std::vector<doris::StorePath, std::allocator<doris::StorePath>> const&, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>> const&) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/runtime/exec_env_init.cpp:424:5 apache#12 0x55e8d32ad7ec in doris::ExecEnv::init(doris::ExecEnv*, std::vector<doris::StorePath, std::allocator<doris::StorePath>> const&, std::vector<doris::StorePath, std::allocator<doris::StorePath>> const&, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>> const&) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/runtime/exec_env_init.cpp:199:17 apache#13 0x55e8b722cc8b in main /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/service/doris_main.cpp:531:14 apache#14 0x7f539b7cad8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 ### Check List (For Author) - Test  - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [ ] No. - [ ] Yes.  - Does this need documentation? - [ ] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label  --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

…d ip" log Two follow-up improvements on top of the eviction work in this PR: 1. _resolve_hostname's failure branch re-checks cache.find(hostname) under the unique_lock before ++failure_count[hostname]. The earlier shared_lock read of cached_ip and the later unique_lock are separated by hostname_to_ip(), which can block for seconds on DNS timeout; the refresh thread can _erase() the entry inside that window. The re-check prevents re-introducing a failure_count entry for an already-evicted host, preserving keys(failure_count) is a subset of keys(cache) under all concurrent interleavings. 2. The "no cached ip available" warning is now throttled by an atomic counter driven by dns_cache_log_every_n_failures. Without this, after a host is evicted, every brpc retry path that calls dns_cache->get(host) hits this branch unthrottled, replacing the refresh-thread flood with a caller-side flood. The throttle is a coarse global rate limit (not per-hostname) because maintaining per-hostname state in this branch would break the invariant from fix #1. Added unit test failure_count_not_reintroduced_on_eviction_race that exercises the new re-check by injecting a resolver which calls _erase mid-resolution (mimicking the race window) and verifying failure_count remains 0. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

…Paimon tables (apache#62177) ### What problem does this PR solve? Problem Summary: ``` 3# raise at ../sysdeps/posix/raise.c:27 4# abort at ./stdlib/abort.c:81 5# 0x0000556DDAC000A1 in /mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be 6# std::vector, std::allocator >, std::allocator, std::allocator > > >::operator[](unsigned long) const at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/stl_vector.h:1282 7# doris::FileScanner::_generate_partition_columns() at ./be/build_ASAN/../src/exec/scan/file_scanner.cpp:1653 8# doris::FileScanner::_get_next_reader() at ./be/build_ASAN/../src/exec/scan/file_scanner.cpp:957 9# doris::FileScanner::_get_block_wrapped(doris::RuntimeState*, doris::Block*, bool*) at ./be/build_ASAN/../src/exec/scan/file_scanner.cpp:439 10# doris::FileScanner::_get_block_impl(doris::RuntimeState*, doris::Block*, bool*) at ./be/build_ASAN/../src/exec/scan/file_scanner.cpp:403 11# doris::Scanner::get_block(doris::RuntimeState*, doris::Block*, bool*) at ./be/build_ASAN/../src/exec/scan/scanner.cpp:143 12# doris::Scanner::get_block_after_projects(doris::RuntimeState*, doris::Block*, bool*) at ./be/build_ASAN/../src/exec/scan/scanner.cpp:119 13# doris::ScannerScheduler::_scanner_scan(std::shared_ptr, std::shared_ptr) at ./be/build_ASAN/../src/exec/scan/scanner_scheduler.cpp:179 14# doris::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_0::operator()() const::{lambda()#1}::operator()() const::{lambda()#1}::operator()() const at ./be/build_ASAN/../src/exec/scan/scanner_scheduler.cpp:76 ``` ### Release note BE crashes with SIGABRT (vector out-of-bounds) in `FileScanner::_generate_partition_columns()` when scanning Iceberg or Paimon partitioned tables. Root cause: `_partition_slot_index_map` is built once from the first scan range's `columns_from_path_keys`, but different ranges may have different `columns_from_path` sizes due to: 1. **Iceberg partition evolution**: Tables evolving from non-identity transforms (e.g., `day(ts)`) to identity transforms (`ts`). `IcebergUtils.getPartitionInfoMap()` returns null for non-identity transforms, so `setIcebergParams()` leaves `columnsFromPath` as an empty list for those ranges while populating it for identity-transform ranges. 2. **Paimon mixed reader paths**: Native reader splits (Parquet/ORC) call `setPaimonPartitionValues(partitionInfoMap)`, but JNI scanner splits (for merge-required or unsupported-format data) skip this call entirely, leaving `paimonPartitionValues` as null. The root cause is that `createFileRangeDesc()` unconditionally called `setColumnsFromPath([])` for all ranges, setting the Thrift `__isset` flag to true even with an empty list. When BE sees `__isset=true`, it enters the partition column loop and crashes on out-of-bounds access for ranges that were never populated with actual partition values. Fix: - **FileQueryScanNode.createFileRangeDesc()**: Only call `setColumnsFromPath()` and `setColumnsFromPathKeys()` when `columnsFromPathKeys` is non-empty. This keeps `__isset=false` for Iceberg/Paimon ranges that have no path-derived partition keys, so BE skips partition column generation entirely for those ranges. Ranges that do have partition values (set later by `setIcebergParams`/`setPaimonParams`) work correctly since they explicitly call `setColumnsFromPath()` with actual values. - **PaimonScanNode**: Additionally set partition values on JNI scanner splits (previously missing), enabling runtime filter partition pruning for JNI splits. Fix a BE crash (SIGABRT) that could occur when querying Iceberg tables with partition evolution or Paimon tables with mixed-format data splits.

…pache#62759) Related PR: apache#58002 Problem Summary: `arrow::Decimal256Array::Value()` returns raw bytes (`const uint8_t*`), we previously reinterpret_cast it to `Decimal256*` which is a misaligned access . ```text ../src/core/data_type_serde/data_type_decimal_serde.cpp:376:21: runtime error: reference binding to misaligned address 0x1338f3345b12 for type 'const FieldType' (aka 'const Decimal<integer<256, int>>'), which requires 8 byte alignment 0x1338f3345b12: note: pointer points here 00 c2 3e 20 00 00 10 63 2d 5e c7 6b 05 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ #0 0x56203d1eb751 in doris::DataTypeDecimalSerDe<(doris::PrimitiveType)35>::read_column_from_arrow(doris::IColumn&, arrow::Array const*, long, long, cctz::time_zone const&) const be/build_ASAN/../src/core/data_type_serde/data_type_decimal_serde.cpp:375:25 #1 0x56203d9c6d6b in doris::DataTypeNullableSerDe::read_column_from_arrow(doris::IColumn&, arrow::Array const*, long, long, cctz::time_zone const&) const be/build_ASAN/../src/core/data_type_serde/data_type_nullable_serde.cpp:350:26 apache#2 0x562055dd83b5 in doris::FromRecordBatchToBlockConverter::convert(doris::Block*) be/build_ASAN/../src/format/arrow/arrow_block_convertor.cpp:131:9 apache#3 0x562055dd9229 in doris::convert_from_arrow_batch(std::shared_ptr<arrow::RecordBatch> const&, std::vector<std::shared_ptr<doris::IDataType const>, ```

…ire (apache#62805) ## Proposed changes Fix two related bugs on the event-driven warm-up path. Together they stall BE's heavy work pool when a warm-up job is cancelled or expired while BE still has its id in `_tablet_replica_cache`. ### 1. BE — `CANCELED` vs `CANCELLED` typo `be/src/cloud/cloud_warm_up_manager.cpp` used `st.is<CANCELED>()` (one L). `CANCELED` is not in `ErrorCode`; ADL resolved it to `PCacheStatus::CANCELED = 9` from a proto enum, so the check compared against 9 and was always false. When FE returned `TStatusCode.CANCELLED` (value 1) to tell BE a job was done, BE never pushed the `job_id` into `cancelled_jobs`, leaving a zombie entry in `_tablet_replica_cache` that every subsequent `commit_rowset` re-queried. Fix: use `st.is<ErrorCode::CANCELLED>()`, matching the same namespace-qualified form used elsewhere in the file. ### 2. FE — NPE in `getTabletReplicaInfos` after job removal ```java if (job == null || job.isDone()) { LOG.info("warmup job {} is not running, notify caller BE {} to cancel job", job.getJobId(), clientAddr); // NPE when job == null ... } ``` Once a cancelled job was removed from `cloudWarmUpJobs` past `history_cloud_warm_up_job_keep_max_second`, `job` is null and the log call NPE'd. With bug #1 keeping the stale id in BE cache, BE kept hitting this RPC forever; each failure took the `apache::thrift::TException` branch in `thrift_rpc_helper.cpp`, which sleeps 2s inside `CloudWarmUpManager::_mtx`. That serialised `bthread_fork_join(commit_rowset)`, blocked heavy-pool threads in `CloudTabletsChannel::close`, and backed up the heavy-pool queue — leading to load timeouts and query `Fragment RPC Phase1` latency in the 10s range. Fix: log `request.getWarmUpJobId()` instead; it is guaranteed set by the enclosing `request.isSetWarmUpJobId()` check.

…pache#63191) Related PR: apache#62489 Problem Summary: ```cpp static const std::string no_message = ""; return ok() ? no_message : state_->msg; ``` In the clang-built Doris BE binary, this inline static `std::string` is emitted as a weak/COMDAT object and placed in `.data.rel.ro`. ```bash $ readelf -sW "$BIN" | c++filt | grep 'Status::message.*no_message' 105229: 000000007d6958e0 32 OBJECT WEAK DEFAULT 28 arrow::Status::message[abi:cxx11]() const::no_message[abi:cxx11] $ readelf -SW doris/output/be/lib/doris_be | grep '\[ *28\]' [28] .data.rel.ro PROGBITS 000000007d0f7720 7d0f6720 7e4208 00 WA 0 0 32 ``` After relocation, RELRO makes this section read-only. However, C++ function-local statics are lazily initialized on first execution, so the first call to `Status::message()` tries to construct `no_message` at runtime. The `std::string` constructor writes to the object storage in `.data.rel.ro`, which triggers `SIGSEGV invalid permissions for mapped object` like: *FromStatus(arrow::Status)* ```text *** Query id: 0-0 *** *** is nereids: 0 *** *** tablet id: 0 *** *** Aborted at 1778559900 (unix time) try "date -d @1778559900" if you are using GNU date *** *** Current BE git commitID: f02e9e680c8 *** *** SIGSEGV invalid permissions for mapped object (@0xaaaaf1ecbf28) received by PID 3634450 (TID 3636517 OR 0xfffa35fe97c0) from PID 18446744073473408808; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/selectdb-core/be/src/common/signal_handler.h:421 1# PosixSignals::chained_handler(int, siginfo_t*, void*) [clone .part.0] in /opt/jdk/lib/server/libjvm.so 2# JVM_handle_linux_signal in /opt/jdk/lib/server/libjvm.so 3# 0x0000FFFFAE091830 in linux-vdso.so.1 4# std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string<std::allocator<char> >(char const*, std::allocator<char> const&) at /root/toolchain/ldb-toolchain-v0.21/bin/../lib/gcc/aarch64-linux-gnu/13/../../../../include/c++/13/bits/basic_string.h:632 5# arrow::flight::internal::TransportStatus::FromStatus(arrow::Status const&) in /opt/selectdb/4.1.3.2026042721/be/lib/doris_be 6# arrow::flight::transport::grpc::ToGrpcStatus(arrow::Status const&, grpc::ServerContext*) in /opt/selectdb/4.1.3.2026042721/be/lib/doris_be 7# arrow::flight::transport::grpc::(anonymous namespace)::GrpcServiceHandler::DoGet(grpc::ServerContext*, arrow::flight::protocol::Ticket const*, grpc::ServerWriter<arrow::flight::protocol::FlightData>*) in /opt/selectdb/4.1.3.2026042721/be/lib/doris_be 8# grpc::Status grpc::internal::CatchingFunctionHandler<grpc::internal::ServerStreamingHandler<arrow::flight::protocol::FlightService::Service, arrow::flight::protocol::Ticket, arrow::flight::protocol::FlightData>::RunHandler(grpc::internal::MethodHandler::HandlerParameter const&)::{lambda()#1}>(grpc::internal::ServerStreamingHandler<arrow::flight::protocol::FlightService::Service, arrow::flight::protocol::Ticket, arrow::flight::protocol::FlightData>::RunHandler(grpc::internal::MethodHandler::HandlerParameter const&)::{lambda()#1}&&) in /opt/selectdb/4.1.3.2026042721/be/lib/doris_be 9# grpc::internal::ServerStreamingHandler<arrow::flight::protocol::FlightService::Service, arrow::flight::protocol::Ticket, arrow::flight::protocol::FlightData>::RunHandler(grpc::internal::MethodHandler::HandlerParameter const&) in /opt/selectdb/4.1.3.2026042721/be/lib/doris_be 10# grpc::Server::SyncRequest::ContinueRunAfterInterception() in /opt/selectdb/4.1.3.2026042721/be/lib/doris_be 11# grpc::Server::SyncRequest::Run(std::shared_ptr<grpc::Server::GlobalCallbacks> const&, bool) in /opt/selectdb/4.1.3.2026042721/be/lib/doris_be 12# grpc::ThreadManager::MainWorkLoop() in /opt/selectdb/4.1.3.2026042721/be/lib/doris_be 13# grpc::ThreadManager::WorkerThread::WorkerThread(grpc::ThreadManager*)::$_0::__invoke(void*) in /opt/selectdb/4.1.3.2026042721/be/lib/doris_be 14# grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::{lambda(void*)#1}::__invoke(void*) in /opt/selectdb/4.1.3.2026042721/be/lib/doris_be 15# start_thread in /lib64/libc.so.6 16# thread_start in /lib64/libc.so.6 ``` *ToStatus()* ```text *** SIGSEGV invalid permissions for mapped object (@0x56448ff1ef38) received by PID 66637 (TID 67372 OR 0x7f2687017640) from PID 18446744071829581624; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at ../src/common/signal_handler.h:417 1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so 2# JVM_handle_linux_signal in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so 3# 0x00007F2836CE7520 in /lib/x86_64-linux-gnu/libc.so.6 4# std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string<std::allocator<char> >(char const*, std::allocator<char> const&) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/basic_string.h:707 5# arrow::Status::WithDetail(std::shared_ptr<arrow::StatusDetail>) const in /mnt/disk1/PERFORMANCE_ENV/be/lib/doris_be 6# arrow::flight::internal::TransportStatus::ToStatus() const in /mnt/disk1/PERFORMANCE_ENV/be/lib/doris_be 7# arrow::flight::transport::grpc::FromGrpcStatus(grpc::Status const&, grpc::ClientContext*) in /mnt/disk1/PERFORMANCE_ENV/be/lib/doris_be 8# arrow::flight::transport::grpc::(anonymous namespace)::GrpcClientInterceptorAdapter::Intercept(grpc::experimental::InterceptorBatchMethods*) in /mnt/disk1/PERFORMANCE_ENV/be/lib/doris_be 9# grpc::internal::InterceptorBatchMethodsImpl::RunInterceptors() in /mnt/disk1/PERFORMANCE_ENV/be/lib/doris_be 10# grpc::internal::CallOpSet<grpc::internal::CallOpRecvInitialMetadata, grpc::internal::CallOpClientRecvStatus, grpc::internal::CallNoOp<3>, grpc::internal::CallNoOp<4>, grpc::internal::CallNoOp<5>, grpc::internal::CallNoOp<6> >::FinalizeResult(void**, bool*) in /mnt/disk1/PERFORMANCE_ENV/be/lib/doris_be 11# grpc::ClientReaderWriter<arrow::flight::protocol::FlightData, arrow::flight::protocol::FlightData>::Finish() in /mnt/disk1/PERFORMANCE_ENV/be/lib/doris_be 12# arrow::flight::transport::grpc::(anonymous namespace)::FinishableDataStream<grpc::ClientReaderWriter<arrow::flight::protocol::FlightData, arrow::flight::protocol::FlightData>, arrow::flight::internal::FlightData>::DoFinish() in /mnt/disk1/PERFORMANCE_ENV/be/lib/doris_be 13# arrow::flight::transport::grpc::(anonymous namespace)::WritableDataStream<grpc::ClientReaderWriter<arrow::flight::protocol::FlightData, arrow::flight::protocol::FlightData>, arrow::flight::internal::FlightData>::DoFinish() in /mnt/disk1/PERFORMANCE_ENV/be/lib/doris_be 14# arrow::flight::internal::ClientDataStream::Finish(arrow::Status) in /mnt/disk1/PERFORMANCE_ENV/be/lib/doris_be 15# arrow::flight::ClientStreamReader::Next() in /mnt/disk1/PERFORMANCE_ENV/be/lib/doris_be 16# doris::PythonClient::read_batch(std::shared_ptr<arrow::RecordBatch>*) at ./be/build_RELEASE/../src/udf/python/python_client.cpp:136 17# doris::PythonUDFClient::evaluate(arrow::RecordBatch const&, std::shared_ptr<arrow::RecordBatch>*) at ./be/build_RELEASE/../src/udf/python/python_udf_client.cpp:36 ``` ### fix Move the empty message/detail sentinels out of the header inline path and make non-OK statuses return `state_->msg` directly. This avoids touching the empty OK-message sentinel on error paths and prevents the inline weak/COMDAT `std::string` object from being lazily constructed from a read-only mapping.

…d ip" log Two follow-up improvements on top of the eviction work in this PR: 1. _resolve_hostname's failure branch re-checks cache.find(hostname) under the unique_lock before ++failure_count[hostname]. The earlier shared_lock read of cached_ip and the later unique_lock are separated by hostname_to_ip(), which can block for seconds on DNS timeout; the refresh thread can _erase() the entry inside that window. The re-check prevents re-introducing a failure_count entry for an already-evicted host, preserving keys(failure_count) is a subset of keys(cache) under all concurrent interleavings. 2. The "no cached ip available" warning is now throttled by an atomic counter driven by dns_cache_log_every_n_failures. Without this, after a host is evicted, every brpc retry path that calls dns_cache->get(host) hits this branch unthrottled, replacing the refresh-thread flood with a caller-side flood. The throttle is a coarse global rate limit (not per-hostname) because maintaining per-hostname state in this branch would break the invariant from fix #1. Added unit test failure_count_not_reintroduced_on_eviction_race that exercises the new re-check by injecting a resolver which calls _erase mid-resolution (mimicking the race window) and verifying failure_count remains 0. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

…pache#68880) Issue Number: N/A Related PR: N/A Problem Summary: EXCEPT DISTINCT can introduce a streaming distinct aggregation whose grouping expressions repeat an input column, for example (s, d, d). When preaggregation switches to pass-through, both output construction and column reuse call assert_mutable() even though they only transfer column references. Retaining output references can make a repeated key shared, so a subsequent assertion fails with COW::assert_mutable: use_count() > 1: ``` W 2026-10-09 06:31:59,596 13803 exception.cpp:34] meet exception, error code: 6, message: COW::assert_mutable: use_count() > 1 0# doris::Exception::Exception(int, std::basic_string_view<char, std::char_traits<char> > const&) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/basic_string.h:239 1# doris::DistinctStreamingAggLocalState::_distinct_pre_agg_with_serialized_key(doris::Block*, doris::Block*) at ../src/core/cow.h:325 2# doris::DistinctStreamingAggOperatorX::push(doris::RuntimeState*, doris::Block*, bool) const at ../src/common/status.h:519 3# doris::StatefulOperatorX<doris::DistinctStreamingAggLocalState>::get_block_impl(doris::RuntimeState*, doris::Block*, bool*) at ../src/common/status.h:519 4# doris::OperatorXBase::get_block(doris::RuntimeState*, doris::Block*, bool*) at ../src/common/status.h:519 5# doris::OperatorXBase::get_block_after_projects(doris::RuntimeState*, doris::Block*, bool*) at ./be/build_RELEASE/../src/exec/operator/operator.cpp:0 6# doris::PipelineTask::execute(bool*) at ./be/src/exec/pipeline/pipeline_task.cpp:0 7# doris::TaskScheduler::_do_work(int) at ./be/src/exec/pipeline/task_scheduler.cpp:155 ``` Use immutable get_ptr() references in both pass-through paths. The evaluated key pointers preserve the original column when a repeated input position has already been replaced during column reuse, and downstream mutations continue to use copy-on-write. No column data is copied during pass-through. Add unit coverage with three references to one input slot for fresh and reused output blocks, transition from deduplication, retained output ownership, and nullable output. Add regression coverage for repeated keys in EXCEPT DISTINCT, nullable EXCEPT output, INTERSECT DISTINCT, and UNION DISTINCT. Fix queries with repeated projected columns failing with a COW assertion during distinct streaming aggregation. 根因是：**Distinct Streaming Aggregation 在透传列时调用了要求“列对象独占”的 `assert_mutable()`，而重复分组键会让多个输出位置共享同一个列对象，导致后续调用违反这个要求。** 下面结合修复前的代码，分别说明两条透传路径。 **1\. `assert_mutable()` 检查的是列对象的引用计数** COW::assert\_mutable() (be/src/core/cow.h:323) 的实现是： ``` MutablePtr assert_mutable() const { if (this->use_count() > 1) { throw Exception(ErrorCode::INTERNAL_ERROR, "COW::assert_mutable: use_count() > 1"); } return const_cast<COW*>(this)->get_ptr(); } ``` 它要求调用时 `use_count() <= 1`，直接返回可变指针，**不会自动复制共享的列对象**。如果确实需要修改可能共享的列，应使用 IColumn::mutate() (be/src/core/column/column.h:618)：它通过 `shallow_mutate()` 在引用计数大于 1 时复制列，使修改发生在独立对象上。这里的引用计数统计的是持有列对象的智能指针数量。例如： ``` in_block.column[1] ─┐ ├── 同一个日期列对象 C output.column[2] ───┘ C.use_count() = 2 ``` **2\. 重复投影如何变成重复输入列引用** 例如： ``` SELECT s, d, d FROM t1 EXCEPT DISTINCT SELECT s, d, d FROM t2; ``` 当优化器为集合操作的输入推导局部去重时，InferSetOperatorDistinct (fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/InferSetOperatorDistinct.java:60) 使用子节点输出构造分组键： ``` new LogicalAggregate<>( ImmutableList.copyOf(child.getOutput()), true, child) ``` 因此，生成的局部聚合可能保留重复槽位，例如： ``` group by: s[#1], d[apache#2], d[apache#2] ``` BE 在准备分组列 (be/src/exec/operator/distinct\_streaming\_aggregation\_operator.cpp:177)时，记录表达式执行结果： ``` RETURN_IF_ERROR(_probe_expr_ctxs[i]->execute(in_block, &result_column_id)); // 准备列…… key_columns[i] = in_block->get_by_position(result_column_id).column.get(); result_idxs[i] = result_column_id; ``` 假设输入 Block 只有 `s`、`d` 两列，最终得到： ``` in_block: [Cs, Cd] result_idxs: [ 0, 1, 1] key_columns: [Cs*, Cd*, Cd*] ``` 两个日期键指向同一个列对象。`key_columns` 保存的是裸指针，本身不增加引用计数。当预聚合因内存限制或去重收益不足设置 `_stop_emplace_flag = true` 后，算子开始透传输入，进入下面的问题路径。 **3\. 首次构造输出：两个重复引用即可报错** 修复前，不复用输出列的透传代码是： ``` for (int i = 0; i < key_size; ++i) { if (_stop_emplace_flag) { columns_with_schema.emplace_back( key_columns[i]->assert_mutable(), /* type, name */); } } ``` 对应当前代码的输出构造位置 (be/src/exec/operator/distinct\_streaming\_aggregation\_operator.cpp:267)。以 `[s, d, d]` 为例，假设日期列 `Cd` 开始只有输入 Block 持有： | 时刻 | 持有 `Cd` 的位置 | 稳定引用计数 | |---|---|---:| | 处理第一个 `d` 前 | 输入列 | 1 | | 第一个 `d` 加入输出后 | 输入列、第一个日期输出列 | 2 | | 处理第二个 `d` 时 | 再次调用 `Cd->assert_mutable()` | 2，抛异常 | 第一个 `d` 输出时增加了一个列引用，输入引用仍然存在。第二个 `d` 再要求独占，就会失败。后面的 `in_block->clear()` 原本可以释放输入引用，但异常已经在循环内发生，执行不到那里。 **4\. 复用输出 Block：三个重复引用能展示同样的问题** 修复前，复用输出列的透传位置 (be/src/exec/operator/distinct\_streaming\_aggregation\_operator.cpp:235) 是： ``` for (int i = 0; i < key_size; ++i) { auto output_column = out_block->get_by_position(i).column; out_block->replace_by_position(i, key_columns[i]->assert_mutable()); in_block->replace_by_position(result_idxs[i], output_column); } ``` 这个分支将旧输出列放回输入 Block，释放输入对原数据列的引用。以 `[s, d, d, d]` 为例： ``` result_idxs = [0, 1, 1, 1] ``` 假设输出列最初为空，且没有其他所有者： | 时刻 | 原日期列 `Cd` 的所有者 | 稳定引用计数 | |---|---|---:| | 第一个 `d` 处理结束 | 第一个日期输出列；输入位置已换成旧输出空列 | 1 | | 第二个 `d` 处理结束 | 第一个、第二个日期输出列 | 2 | | 第三个 `d` 开始处理 | 调用 `Cd->assert_mutable()` | 2，抛异常 | 第二次替换的输入位置已经是空列，无法再释放原日期列的引用。于是两个输出列共同持有 `Cd`，第三次独占断言失败。因此，**复用分支中只有两个相同槽位不一定立即报错**；已有单测使用 `[0, 1, 0, 0]`，让同一输入槽位出现三次，覆盖这一过程。 **5\. 为什么改为 `get_ptr()` 能解决** 两条透传路径现在都使用： ``` key_columns[i]->get_ptr() ``` `key_columns` 的类型是 `std::vector<const IColumn*>`，因此这里调用的是 const 版本的 `get_ptr()` (be/src/core/cow.h:304)，返回不可变列指针。透传只需要让输出引用已有列数据，允许多个输出位置共享该列。后续需要修改数据时，仍通过 `IColumn::mutate()` 执行 COW。复用分支继续从预先记录的 `key_columns` 取指针也很关键：第一次替换后，`in_block[result_idxs[i]]` 已经变成旧输出列，而 `key_columns[i]` 仍指向原数据列，并由前面的输出位置保持其生命周期。关于 GROUP BY，直接的 `GROUP BY s, d, d` 会在 NormalizeAggregate (fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/NormalizeAggregate.java:167) 中去重。静态分析发现，`GROUP BY + LIMIT` 下推到 `UNION ALL` 分支时，槽位映射可能重新引入重复键；已在回归用例 (regression-test/suites/query\_p0/set\_operations/distinct\_streaming\_repeated\_keys.groovy:101)中补充直接分组对照和该候选场景，并检查计划中同一槽位是否出现三次。提交为 `c8c1f67d3e9`，按你的要求未执行新增用例，因此尚未实测确认该 GROUP BY 场景。 - Test: Unit Test / Regression test / Manual test - ASAN build with ./build.sh --be -j 12 - 11 BE unit tests covering distinct streaming aggregation - Four regression queries; results also match non-streaming execution - Original reproducing SQL on the supplied table data - clang-format 16, format check, build hygiene, and clang-tidy on changed lines - Behavior changed: Yes; repeated grouping keys pass through without an incorrect ownership assertion - Does this need documentation: No ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test  - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [ ] No. - [ ] Yes.  - Does this need documentation? - [ ] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

…fresh exprId (apache#69111) ### What problem does this PR solve? Problem Summary: When ConstantPropagation replaces an aggregate output by a constant, it materializes the constant in a project above the aggregate, and the project item reused the exprId of the replaced output: LogicalProject ( projects=[..., 1 AS `id`#1, ...] ) +-- LogicalAggregate ( groupByExpr=[id#1], outputExpr=[id#1] ) So one exprId had two definitions, the aggregate output slot and the constant, whose nullable and data type may differ. Rules which compare exprIds only then take them as the same column. For RQG case 11000258898 EliminateUnnecessaryProject removed such a project after the aggregate had been rewritten into `Limit 1`, the plan above fell back to the nullable output slot of the set operation, and the last AdjustNullable pass reported AdjustNullable convert slot 2004-04-28#55 from not-nullable to nullable Fix: Give the constant a fresh exprId only when it really shadows a column, i.e. when the exprId is still produced by the plan below; otherwise keep it: * the constant gets a fresh exprId, and the references above are refreshed with ExprIdRewriter, when the aggregate still outputs the replaced column (its grouping key is kept because all grouping keys are constant) or when a project item re-defines a column which its child still outputs. That is the shadow the old code created, e.g. the RQG case 11000258898 of DORIS-28416. * the constant keeps the exprId when the aggregate removed the constant grouping key from its outputs: the project item is then the only definition of the column, nothing below references the exprId, and consumers which rely on the identity of the column, e.g. the materialized view matching, keep working. The plan of such a query is byte identical to the plan before this change, including the exprIds (verified with the query of `agg_optimize_when_uniform/mv2_0`, for the query side and for the MV side which is generated with constant propagation disabled). The sink also has to follow the fresh exprIds, otherwise it would reference a slot which its child does not output any more; only exprIds are remapped there, the name and the data type declared by the sink are kept (that is why the rule does not replace the slots of the sink by literals). A CTE is rewritten in separate subtree contexts (producer body and consumers) and the consumers, which map their slots to the producer outputs by exprId, are rewritten before the producer. The renamed exprIds are published per CTE (`StatementContext#producerExprIdRemap`), and `RewriteCteChildren` refreshes both maps of every consumer with the last exprId of the chain (`ExprIdRewriter#lastExprId`). Reproduction: ConstantPropagationFreshExprIdTest runs this rule alone, so the project above the aggregate is still visible in the plan, and asserts that it does not reuse the exprId of the aggregate output and that the plan above references the fresh exprIds. ConstantPropagationTest used to pin the reused exprIds in its constant project expectations; it now asserts fresh exprIds. Both fail before this change. ### Release note None ### Check List (For Author) - Test: Unit Test - ConstantPropagationFreshExprIdTest (new): fails before with "the constant project reuses the exprId of the aggregate output". - ConstantPropagationTest: expectations updated, fails before with "the constant project item must not reuse the exprId of the replaced output slot". - 31 related FE unit test classes (constant propagation, constant cte, set operation, column pruning, fd, uniform, sql plan suite, runtime filter, ...) rerun: 385 tests, 0 failures. - Behavior changed: Yes. A constant materialized above an aggregate has its own exprId and the plan above references it, so one exprId no longer describes two different columns. - Does this need documentation: No

zhaorongsheng added 2 commits November 13, 2025 17:25

handle loading iceberg partitions by auth

50cca28

Wrap loading of iceberg partitions in try-catch block to handle exceptions and log errors.

Refactor storage properties retrieval in IcebergScanNode

5661187

zhaorongsheng closed this Nov 13, 2025

Navigation Menu

Repository navigation

[fix]handle loading iceberg partitions by auth - #1

[fix]handle loading iceberg partitions by auth#1
zhaorongsheng wants to merge 2 commits into
masterfrom
bugfix_iceberg_krb5_error

zhaorongsheng commented Nov 13, 2025

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

zhaorongsheng commented Nov 13, 2025

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant