Repository navigation
Conversation
…n cloud (apache#59539) 1. Starting from the getBackendId of the replica, implement a sliding window approach (in TabletAccessStats.java) to count read and write operations over the last hour. The accuracy of this count isn't critical, as the aim is to indicate relative activity levels. 2. Display these statistics in the SHOW TABLETS FROM TABLE and SHOW TABLET {tabletId} commands. ``` show tablet 1767773242851\G ***************************[ 1. row ]*************************** ... AccessCount1H | 1107 LastAccessTime | 1767782384995 ... ``` 3. Integrate with the existing balancing strategy: - For each scheduling operation, identify active databases, tables, and partitions from the active tablets. Schedule operations first on the most active databases, tables, and partitions. - When selecting tablets from the source BE, modify the previous random selection to prioritize non-active tablets for scheduling, keeping active tablets as stable as possible on the source.
fix compile backend on MacOS failed
… is fully populated (apache#60749) Related PR: apache#59990 Problem Summary: Move setTransactionStatus(COMMITTED) to after idToTableCommitInfos population in both commitTransaction methods. This prevents PublishVersionDaemon from seeing COMMITTED status with empty commit info, as it checks status without synchronized(transactionState) lock.
…pache#59747) ### What problem does this PR solve? Issue Number: close #xxx Related PR: apache#59394 Problem Summary: The search DSL should only recognize uppercase `AND`, `OR`, `NOT` as boolean operators in search lucene boolean mode. Previously, lowercase `and`, `or`, `not` were also treated as operators, which does not conform to the specification. This PR makes the boolean operators case-sensitive: - Only uppercase `AND`, `OR`, `NOT` are recognized as operators - Lowercase `and`, `or`, `not` are now treated as regular search terms - Using lowercase operators in DSL will result in a parse error ### Release note Make search DSL boolean operators (AND/OR/NOT) case-sensitive in lucene boolean mode.
apache#60758) Make followed_by_shuffled_operator() virtual and override it in hash join and set operators to correctly report shuffle requirements. For multi-child operators (hash join, set operations), the method now returns true if the operator itself is shuffled (and not colocated), ensuring proper global shuffle behavior in local exchange. Changes: - Make followed_by_shuffled_operator() virtual in base OperatorX class - Override in HashJoinBuildSinkOperatorX and HashJoinProbeOperatorX to return is_shuffled_operator() && !is_colocated_operator() - Override in PartitionedHashJoinProbeOperatorX and PartitionedHashJoinSinkOperatorX to delegate to inner operator - Override in SetProbeSinkOperatorX and SetSinkOperatorX to return !_is_colocate This fixes incorrect data distribution when shuffled join/set operators are used in pipelines with local exchange.
…ception (apache#60754) ### What problem does this PR solve? Issue Number: close apache#60746 Problem Summary: ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [x] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [x] Other reason <!-- Add your reason? --> hard to reproduce - Behavior changed: - [x] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [x] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
…rs (apache#60756) ### What problem does this PR solve? Related apache#60768 Introduce a symmetric JNI writer mechanism that complements the existing JNI scanner framework, enabling custom file writers to be implemented in Java and invoked through JNI from the C++ query executor. Key changes: - Add VJniFormatTransformer for delegating writes to Java via JNI - Extend TTVFTableSink with writer_type and writer_class fields - Add JniWriter base class and LocalFileJniWriter reference implementation - Integrate with file_format_transformer factory routing logic The implementation follows the same patterns as JniScanner for consistency: - Reuses class loading mechanism (Jni::Util::get_jni_scanner_class) - Mirrors constructor signature: (int batchSize, Map<String,String> params) - Supports schema caching and performance instrumentation
### What problem does this PR solve? Issue Number: close #xxx Related PR: apache#60347 Problem Summary: Before: ```text Doris> select now() day_hour; ERROR 1105 (HY000): errCode = 2, detailMessage = extraneous input 'day_hour' expecting {<EOF>, ';'}(line 1, pos 13) ``` After: ```text Doris> select now() day_hour; +---------------------+ | day_hour | +---------------------+ | 2026-02-16 02:31:36 | +---------------------+ ``` ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [x] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [x] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
…sing and fix ES compatibility issues (apache#60654) ### What problem does this PR solve? Problem Summary: The `search()` function's DSL parser had multiple ES compatibility issues and used a two-phase parsing approach (manual pre-parse + ANTLR) that was error-prone. This PR refactors the parser and fixes several bugs: 1. **SearchDslParser refactoring**: Consolidated from two-phase (manual pre-parse + ANTLR) to single-phase ANTLR parsing. The ANTLR grammar now handles all DSL syntax directly, eliminating the fragile manual pre-parse layer. This fixes issues with operator precedence, grouping, and edge cases. 2. **ANTLR grammar improvements**: Updated `SearchLexer.g4` and `SearchParser.g4` to properly handle quoted phrases, field-qualified expressions, prefix/wildcard/regexp patterns, range queries, and boolean operators with correct precedence. 3. **minimum_should_match pipeline**: Added `default_operator` and `minimum_should_match` fields to `TSearchParam` thrift, passing them from FE `SearchPredicate` through to BE `function_search`. When `minimum_should_match > 0`, uses `OccurBooleanQuery` for proper Lucene-style boolean query semantics. 4. **Wildcard/Prefix/Regexp case-sensitivity**: Wildcard and PREFIX patterns are now lowercased when the index has `parser + lower_case=true` (matching ES query_string normalizer behavior). REGEXP patterns are NOT lowercased (matching ES regex behavior where patterns bypass analysis). 5. **MATCH_ALL_DOCS support**: Added `MATCH_ALL_DOCS` clause type for standalone `*` queries and pure NOT query rewrites. Enhanced `AllQuery` with deferred `max_doc` from `context.segment_num_rows` and nullable field support via `NullableScorer`. 6. **BE fixes**: - `regexp_weight._max_expansions`: Changed from 50 to 0 (unlimited) to prevent PREFIX queries from missing documents - `occur_boolean_weight`: Fixed swap→append bug when all SHOULD clauses must match, preserving existing MUST scorers - Variant subcolumn `index_properties` propagation for proper analyzer selection - `lower_case` default handling: inverted index `lower_case` defaults to `"true"` when a parser is configured
…-based indexes (apache#60782) ### What problem does this PR solve? Issue Number: close #xxx Related PR: apache#60654 Problem Summary: Follow-up fix for apache#60654 (SearchDslParser refactoring). When FE resolves a variant subcolumn field pattern to a specific analyzer-based index and sends its `index_properties` via `TSearchFieldBinding`, the BE `FieldReaderResolver` was using `EQUAL_QUERY` for TERM clauses. This caused `select_best_reader` to pick the `STRING_TYPE` reader (untokenized index directory) instead of the `FULLTEXT` reader, so tokenized search terms would never match. **Root cause**: For variant subcolumns with analyzer-based indexes, `EQUAL_QUERY` opens the wrong (untokenized) index directory. The query type needs to be upgraded to `MATCH_ANY_QUERY` so `select_best_reader` picks the correct FULLTEXT reader. **Fix**: In `FieldReaderResolver::resolve()`, when the field is a variant subcolumn and the FE-provided `index_properties` indicate an analyzer-based index (`should_analyzer()` returns true), automatically upgrade `EQUAL_QUERY` to `MATCH_ANY_QUERY` before calling `select_best_reader`. Also reuse the `fb_it` iterator to avoid a redundant map lookup.
…mode (apache#60784) ### What problem does this PR solve? Issue Number: close Problem Summary: When using `search('*', ...)` with multi-field options (`fields` parameter), the query fails with: ``` only inverted index queries are supported ``` The root cause is in `SearchDslParser.java`: the multi-field parsing methods (`parseDslMultiFieldMode` and `parseDslMultiFieldLuceneMode`) collect field bindings by calling `collectFieldNames()` on the expanded AST. When the query is `*` (match all), the AST node is `MATCH_ALL_DOCS` which has no field set — by design it matches all documents regardless of field. This caused `collectFieldNames` to return an empty set, resulting in no field bindings. Without field bindings, `RewriteSearchToSlots` couldn't create slot references, so the search expression was never pushed down to the inverted index path, and BE fell back to `execute_impl()` which returns the error. **Fix**: After `collectFieldNames()`, if the result is empty, fall back to using the original `fields` list as field bindings. This ensures the push-down mechanism works for `MATCH_ALL_DOCS` queries. **Reproducing queries** (from bug report): ```sql select count(*) from wikipedia where search('*', '{"fields":["title", "content"], "type": "best_fields", "default_operator":"AND","mode":"lucene", "minimum_should_match": 0}'); select count(*) from wikipedia where search('*', '{"default_field": "title", "default_operator":"AND","mode":"lucene", "minimum_should_match": 0}'); ```
…pache#60730) ### What problem does this PR solve? followup apache#60296
Issue Number: apache#56005 Co-authored-by: morningman <[email protected]>
…m2) (apache#60786) ### What problem does this PR solve? Issue Number: close #N/A Problem Summary: The `search()` function did not support ES `query_string` field-grouped syntax where all terms inside parentheses inherit the field prefix: ```sql -- Previously failed with syntax error SELECT * FROM t WHERE search('title:(rock OR jazz)', '{"fields":["title","content"]}'); ``` ES semantics: | Input | Expansion | |-------|-----------| | `title:(rock OR jazz)` | `(title:rock OR title:jazz)` | | `title:(rock jazz)` with `default_operator:AND` | `(+title:rock +title:jazz)` | | `title:(rock OR jazz) AND music` with `fields:[title,content]` | `(title:rock OR title:jazz) AND (title:music OR content:music)` | | `title:("rock and roll" OR jazz)` | `(title:"rock and roll" OR title:jazz)` | ### Root cause The ANTLR grammar `SearchParser.g4` defined `fieldQuery : fieldPath COLON searchValue` where `searchValue` only accepts leaf values (TERM, QUOTED, etc.), not a parenthesized sub-clause. So `title:(` caused a syntax error. ### Solution **Grammar** (`SearchParser.g4`): - Add `fieldGroupQuery : fieldPath COLON LPAREN clause RPAREN` rule - Add it as alternative in `atomClause` before `fieldQuery` **Visitor** (`SearchDslParser.java`): - Add `markExplicitFieldRecursive()` helper — marks all leaf nodes in a group as `explicitField=true` to prevent `MultiFieldExpander` from re-expanding them across unintended fields - Modify `visitBareQuery()` in both `QsAstBuilder` and `QsLuceneModeAstBuilder` to use `currentFieldName` as field group context when set - Add `visitFieldGroupQuery()` to both AST builders: sets field context, visits inner clause, marks all leaves explicit - Update `visitAtomClause()` and `collectTermsFromNotClause()` to handle
…pache#60540) ### What problem does this PR solve? Problem Summary: support write iceberg table with sort-order, the write data have been local sorted, and have add lower/upper_bounds metadata. so the iceberg plan could use it to prune datafile. **Notes**: this is only a local sort, not global sort. so if you are more parallel about iceberg writer, you many see overlapping of lower/upper_bounds between files. if you need a global sort, maybe could add order by cluster in the insert SQL. you could create table, and then alter table eg: ``` CREATE TABLE test_table2 ( id INT, name STRING, score DOUBLE, create_time datetime ) ORDER BY ( id ASC NULLS FIRST, score DESC NULLS LAST) PROPERTIES ( 'write-format'='ORC' ); ```
…must be a constant. (apache#60643) The implementation of width_bucket requires the fourth argument to be constant but did not enforce this; a regular column can be passed in, however width_bucket will only use the first row. before ```sql mysql> SELECT k1, v1, v2, v3, width_bucket(v1, date('2023-11-18'), date('2027-11-18'), v4) AS w FROM width_bucket_test ORDER BY k1; +------+------------+-----------+--------+------+ | k1 | v1 | v2 | v3 | w | +------+------------+-----------+--------+------+ | 1 | 2022-11-18 | 290000 | 290000 | 0 | | 2 | 2023-11-18 | 320000 | 320000 | 1 | | 3 | 2024-11-18 | 399999.99 | 399999 | 1 | | 4 | 2025-11-18 | 400000 | 400000 | 1 | | 5 | 2026-11-18 | 470000 | 470000 | 1 | | 6 | 2027-11-18 | 510000 | 510000 | 2 | | 7 | 2028-11-18 | 610000 | 610000 | 2 | | 8 | NULL | NULL | NULL | NULL | +------+------------+-----------+--------+------+ ``` now ```sql mysql> SELECT k1, v1, v2, v3, width_bucket(v1, date('2023-11-18'), date('2027-11-18'), v4) AS w FROM width_bucket_test ORDER BY k1; ERROR 1105 (HY000): errCode = 2, detailMessage = The fourth argument of WidthBucket must be a constant. ```
…ay_map are used together. (apache#60693) array_map constructs a temporary block but incorrectly passed the selector. ``` 4# 0x000056376DFC5EC5 in /home/work/unlimit_teamcity/TeamCity/Agents/20260211180011agent_172.17.0.220_1/work/60183217f6ee2a9c/output/be/lib/doris_be 5# 0x000056376DFB777A in /home/work/unlimit_teamcity/TeamCity/Agents/20260211180011agent_172.17.0.220_1/work/60183217f6ee2a9c/output/be/lib/doris_be 6# google::LogMessage::SendToLog() in /home/work/unlimit_teamcity/TeamCity/Agents/20260211180011agent_172.17.0.220_1/work/60183217f6ee2a9c/output/be/lib/doris_be 7# google::LogMessage::Flush() in /home/work/unlimit_teamcity/TeamCity/Agents/20260211180011agent_172.17.0.220_1/work/60183217f6ee2a9c/output/be/lib/doris_be 8# google::LogMessageFatal::~LogMessageFatal() in /home/work/unlimit_teamcity/TeamCity/Agents/20260211180011agent_172.17.0.220_1/work/60183217f6ee2a9c/output/be/lib/doris_be 9# doris::vectorized::VExpr::filter_column_with_selector(doris::COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn> const&, doris::vectorized::PODArray<unsigned int, 4096ul, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, false>, 16ul, 15ul> const*, unsigned long) at ../src/vec/exprs/vexpr.h:269 10# doris::vectorized::VSlotRef::execute_column(doris::vectorized::VExprContext*, doris::vectorized::Block const*, doris::vectorized::PODArray<unsigned int, 4096ul, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, false>, 16ul, 15ul>*, unsigned long, doris::COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&) const at /root/doris/be/build_ASAN/../src/vec/exprs/vslot_ref.cpp:96 11# doris::vectorized::VCastExpr::execute_column(doris::vectorized::VExprContext*, doris::vectorized::Block const*, doris::vectorized::PODArray<unsigned int, 4096ul, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, false>, 16ul, 15ul>*, unsigned long, doris::COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&) const at /root/doris/be/build_ASAN/../src/vec/exprs/vcast_expr.cpp:117 12# doris::vectorized::VectorizedFnCall::_do_execute(doris::vectorized::VExprContext*, doris::vectorized::Block const*, doris::vectorized::PODArray<unsigned int, 4096ul, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, false>, 16ul, 15ul>*, unsigned long, doris::COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, doris::COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>*) const at /root/doris/be/build_ASAN/../src/vec/exprs/vectorized_fn_call.cpp:242 13# doris::vectorized::VectorizedFnCall::execute_column(doris::vectorized::VExprContext*, doris::vectorized::Block const*, doris::vectorized::PODArray<unsigned int, 4096ul, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, false>, 16ul, 15ul>*, unsigned long, doris::COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&) const at /root/doris/be/build_ASAN/../src/vec/exprs/vectorized_fn_call.cpp:302 14# doris::vectorized::VectorizedFnCall::_do_execute(doris::vectorized::VExprContext*, doris::vectorized::Block const*, doris::vectorized::PODArray<unsigned int, 4096ul, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, false>, 16ul, 15ul>*, unsigned long, doris::COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, doris::COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>*) const at /root/doris/be/build_ASAN/../src/vec/exprs/vectorized_fn_call.cpp:242 15# doris::vectorized::VectorizedFnCall::execute_column(doris::vectorized::VExprContext*, doris::vectorized::Block const*, doris::vectorized::PODArray<unsigned int, 4096ul, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, false>, 16ul, 15ul>*, unsigned long, doris::COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&) const at /root/doris/be/build_ASAN/../src/vec/exprs/vectorized_fn_call.cpp:302 16# doris::vectorized::VLambdaFunctionExpr::execute_column(doris::vectorized::VExprContext*, doris::vectorized::Block const*, doris::vectorized::PODArray<unsigned int, 4096ul, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, false>, 16ul, 15ul>*, unsigned long, doris::COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&) const at ../src/vec/exprs/vlambda_function_expr.h:48 17# doris::vectorized::ArrayMapFunction::execute(doris::vectorized::VExprContext*, doris::vectorized::Block const*, doris::vectorized::PODArray<unsigned int, 4096ul, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, false>, 16ul, 15ul>*, unsigned long, doris::COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, std::shared_ptr<doris::vectorized::IDataType const> const&, std::vector<std::shared_ptr<doris::vectorized::VExpr>, std::allocator<std::shared_ptr<doris::vectorized::VExpr> > > const&) const at /root/doris/be/build_ASAN/../src/vec/exprs/lambda_function/varray_map_function.cpp:288 18# doris::vectorized::VLambdaFunctionCallExpr::execute_column(doris::vectorized::VExprContext*, doris::vectorized::Block const*, doris::vectorized::PODArray<unsigned int, 4096ul, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, false>, 16ul, 15ul>*, unsigned long, doris::COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&) const at ../src/vec/exprs/vlambda_function_call_expr.h:69 19# doris::vectorized::VectorizedFnCall::_do_execute(doris::vectorized::VExprContext*, doris::vectorized::Block const*, doris::vectorized::PODArray<unsigned int, 4096ul, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, false>, 16ul, 15ul>*, unsigned long, doris::COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, doris::COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>*) const at /root/doris/be/build_ASAN/../src/vec/exprs/vectorized_fn_call.cpp:242 20# doris::vectorized::VectorizedFnCall::execute_column(doris::vectorized::VExprContext*, doris::vectorized::Block const*, doris::vectorized::PODArray<unsigned int, 4096ul, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, false>, 16ul, 15ul>*, unsigned long, doris::COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&) const at /root/doris/be/build_ASAN/../src/vec/exprs/vectorized_fn_call.cpp:302 21# doris::vectorized::VectorizedFnCall::_do_execute(doris::vectorized::VExprContext*, doris::vectorized::Block const*, doris::vectorized::PODArray<unsigned int, 4096ul, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, false>, 16ul, 15ul>*, unsigned long, doris::COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, doris::COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>*) const at /root/doris/be/build_ASAN/../src/vec/exprs/vectorized_fn_call.cpp:242 22# doris::vectorized::VectorizedFnCall::execute_column(doris::vectorized::VExprContext*, doris::vectorized::Block const*, doris::vectorized::PODArray<unsigned int, 4096ul, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, false>, 16ul, 15ul>*, unsigned long, doris::COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&) const at /root/doris/be/build_ASAN/../src/vec/exprs/vectorized_fn_call.cpp:302 23# doris::vectorized::VExprContext::short_circuit_execute_conjuncts(std::vector<std::shared_ptr<doris::vectorized::VExprContext>, std::allocator<std::shared_ptr<doris::vectorized::VExprContext> > > const&, bool, doris::vectorized::Block const*, doris::vectorized::PODArray<unsigned char, 4096ul, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, false>, 16ul, 15ul>*, bool*) at /root/doris/be/build_ASAN/../src/vec/exprs/vexpr_context.cpp:265 24# doris::vectorized::VExprContext::execute_conjuncts(std::vector<std::shared_ptr<doris::vectorized::VExprContext>, std::allocator<std::shared_ptr<doris::vectorized::VExprContext> > > const&, std::vector<doris::vectorized::PODArray<unsigned char, 4096ul, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, false>, 16ul, 15ul>*, std::allocator<doris::vectorized::PODArray<unsigned char, 4096ul, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, false>, 16ul, 15ul>*> > const*, bool, doris::vectorized::Block const*, doris::vectorized::PODArray<unsigned char, 4096ul, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, false>, 16ul, 15ul>*, bool*) in /home/work/unlimit_teamcity/TeamCity/Agents/20260211180011agent_172.17.0.220_1/work/60183217f6ee2a9c/output/be/lib/doris_be 25# doris::vectorized::VExprContext::execute_conjuncts_and_filter_block(std::vector<std::shared_ptr<doris::vectorized::VExprContext>, std::allocator<std::shared_ptr<doris::vectorized::VExprContext> > > const&, doris::vectorized::Block*, std::vector<unsigned int, std::allocator<unsigned int> >&, int) at /root/doris/be/build_ASAN/../src/vec/exprs/vexpr_context.cpp:410 26# doris::vectorized::VExprContext::filter_block(std::vector<std::shared_ptr<doris::vectorized::VExprContext>, std::allocator<std::shared_ptr<doris::vectorized::VExprContext> > > const&, doris::vectorized::Block*, unsigned long) at /root/doris/be/build_ASAN/../src/vec/exprs/vexpr_context.cpp:217 27# doris::vectorized::Scanner::_filter_output_block(doris::vectorized::Block*) at /root/doris/be/build_ASAN/../src/vec/exec/scan/scanner.cpp:176 28# doris::vectorized::Scanner::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /root/doris/be/build_ASAN/../src/vec/exec/scan/scanner.cpp:153 29# doris::vectorized::Scanner::get_block_after_projects(doris::RuntimeState*, doris::vectorized::Block*, bool*) in /home/work/unlimit_teamcity/TeamCity/Agents/20260211180011agent_172.17.0.220_1/work/60183217f6ee2a9c/output/be/lib/doris_be 30# doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>) at /root/doris/be/build_ASAN/../src/vec/exec/scan/scanner_scheduler.cpp:177 31# std::_Function_handler<bool (), doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()#1}>::_M_invoke(std::_Any_data const&) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:292 32# doris::vectorized::ScannerSplitRunner::process_for(std::chrono::duration<long, std::ratio<1l, 1000000000l> >) at /root/doris/be/build_ASAN/../src/vec/exec/scan/scanner_scheduler.cpp:415 33# doris::vectorized::PrioritizedSplitRunner::process() in /home/work/unlimit_teamcity/TeamCity/Agents/20260211180011agent_172.17.0.220_1/work/60183217f6ee2a9c/output/be/lib/doris_be 34# doris::vectorized::TimeSharingTaskExecutor::_dispatch_thread() at /root/doris/be/build_ASAN/../src/vec/exec/executor/time_sharing/time_sharing_task_executor.cpp:568 35# doris::Thread::supervise_thread(void*) at /root/doris/be/build_ASAN/../src/util/thread.cpp:461 36# asan_thread_start(void*) in /home/work/unlimit_teamcity/TeamCity/Agents/20260211180011agent_172.17.0.220_1/work/60183217f6ee2a9c/output/be/lib/doris_be 37# start_thread at /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:478 38# __clone at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97 ```
…60764) Refactor require_bucket_distribution to be propagated through the plan tree instead of being accumulated as a class member variable. Problem: The _require_bucket_distribution flag was accumulated globally across the entire plan tree, which caused incorrect partition expression selection for BUCKET_HASH_SHUFFLE in branches that shouldn't require bucket distribution. Solution: - Remove _require_bucket_distribution member variable from PipelineFragmentContext - Add require_bucket_distribution parameter to _create_tree_helper() and _create_operator() - Compute current_require_bucket_distribution per-subtree based on whether the current operator is colocated and uses hash exchange - Simplify _build_operators_for_set_operation_node() by removing unnecessary parameters This ensures bucket distribution requirements are correctly propagated down each branch of the plan tree independently, similar to how followed_by_shuffled_operator is handled.
…ent cluster (apache#60652) Fix: - Add cluster membership check in getCachedBackend() to invalidate cached backends that no longer belong to the expected cluster, which can happen during auto-scaling.
…ache#60522) When running FE in console mode (start_fe.sh --console), the fe.out log file was not generated because stdout/stderr were not redirected to the log file. The root cause is that the log4j console appender was configured with target="SYSTEM_OUT", which sends log4j logs to stdout. This makes it impossible to separate log4j logs from JVM stdout output at the shell level. Fix: 1. Change log4j console appender target from SYSTEM_OUT to SYSTEM_ERR in Log4jConfig.java, so that log4j logs go to stderr (displayed on terminal) instead of stdout. This also matches the env var name DORIS_LOG_TO_STDERR. 2. Redirect stdout to fe.out in start_fe.sh console mode, consistent with daemon mode behavior. The fe.out file now only contains JVM native output, not log4j formatted logs.
…new_union_iterator (apache#60772) This PR ensures merge/union iterators use an explicit output schema projection and copy only the requested columns, preventing column count mismatches when delete-predicate columns are read in addition to return columns. BetaRowsetReader now builds an `output_schema` from return_columns and passes it to merge/union iterators, VMergeIteratorContext copies using the output schema (not the incorrect _iter->schema()) Consider the following table: ```sql CREATE TABLE tbl ( k INT NOT NULL, v1 INT NOT NULL, v2 INT NOT NULL ) DUPLICATE KEY(k) ...; ``` And a delete predicate applied to a non-key column: ```sql DELETE FROM tbl WHERE v1 = 1; ``` When executing ORDER BY k LIMIT n, Doris has a Top-N optimization. Even though the query is SELECT *, the engine initially avoids scanning all columns. It constructs a minimal intermediate schema containing only the sort keys (k) and the internal `__DORIS_ROWID_COL__` to perform the merge and sorting efficiently. (_col_ids = {0, 3}, ==> _num_columns = 2). However, because a delete predicate exists on column v1, the BetaRowsetReader add v1 to this intermediate schema to evaluate and filter out deleted rows during the scan. (_col_ids = {0, 3, 1}, note that column v1 (index=1) is appended to this schema ==> _num_columns = 3) The previous implementation of VMergeIteratorContext::copy_rows used the incorrect _num_columns value, resulting in an array out-of-bounds access and causing BE coredumped. Detailed reproduction steps are follows: 1. modify conf/be.conf ``` write_buffer_size = 8 ``` 2. execute the following sql ```sql CREATE TABLE tbl1 ( k INT NOT NULL, v1 INT NOT NULL, v2 INT NOT NULL ) DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 5 PROPERTIES( "replication_num" = "1" ); CREATE TABLE tbl2 ( k INT NOT NULL, v1 INT NOT NULL, v2 INT NOT NULL ) DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES( "replication_num" = "1" ); INSERT INTO tbl1 VALUES (1, 1, 1),(2, 2, 2),(3, 3, 3),(4, 4, 4),(5, 5, 5); INSERT INTO tbl2 SELECT * FROM tbl1; SELECT * FROM tbl2 ORDER BY k limit 100; -- ok DELETE FROM tbl2 WHERE v1 = 100; SELECT * FROM tbl2 ORDER BY k limit 100; -- coredump ``` Co-authored-by: yiguolei <[email protected]> ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
Add a queue_time_ms column to table __internal_schema.audit_log to track per-query queueing behavior and elapsed time.
…h() function (apache#60790) ### What problem does this PR solve? Problem Summary: This PR adds searcher cache reuse and a DSL result cache for the `search()` function to improve query performance on repeated search queries against the same segments. **Key changes:** 1. **DSL result cache**: Caches the final roaring bitmap per (segment, DSL) pair so repeated identical `search()` queries skip Lucene execution entirely. Uses length-prefix key encoding to avoid hash collisions. 2. **Deep-copy bitmap semantics**: Bitmaps are deep-copied on both cache read and write to prevent `mask_out_null()` from polluting cached entries. 3. **Type-safe cache accessor**: Replaces raw `void*` return with a template `get_value<T>()` that uses `static_assert` to ensure T derives from `LRUCacheValueBase`. 4. **Session-level cache toggle**: Adds `enable_search_function_query_cache` session variable (default: true) to allow disabling the cache per query via `SET_VAR`. 5. **Const-correctness fix**: Removes unsafe `const_cast` in `build_dsl_signature` by copying TSearchParam before Thrift serialization. 6. **Defensive improvements**: Adds null check for `result_bitmap` on cache hit, logging for serialization fallback and cache bypass paths. ### Release note Add DSL result cache for search() function to skip repeated Lucene execution on identical queries.
…y results (apache#60793) ### What problem does this PR solve? Related PR: apache#60782 Problem Summary: When `search()` DSL uses wildcard patterns (e.g. `*ith`, `sm*th`, `sm?th`) on variant subcolumns with analyzer-based indexes (field_pattern), the queries return empty results even though regular TERM search works correctly. **Root cause:** In `FieldReaderResolver::resolve()`, only `EQUAL_QUERY` was upgraded to `MATCH_ANY_QUERY` for variant subcolumns with analyzer-based indexes. `WILDCARD_QUERY` was not upgraded, so `select_best_reader()` picked the `STRING_TYPE` reader instead of `FULLTEXT`. `WildcardWeight` then enumerated terms from the wrong (untokenized) index directory, finding no matches. **Fix:** Extend the query type upgrade condition to also cover `WILDCARD_QUERY`, so wildcard patterns correctly use the FULLTEXT index on variant subcolumns. Also fix a misleading comment in `inverted_index_iterator.cpp` where `is_equal_query()` was described as handling WILDCARD/REGEXP but actually only checks `EQUAL_QUERY`.
## Background `ClusterNamespace` is a legacy artifact from the multi-tenancy era. In earlier versions, Apache Doris implemented multi-tenancy by prefixing database names with a cluster name using `:` as a separator. This functionality has since been completely removed, but the `ClusterNamespace` class and related legacy code remained. ## Changes - Remove all `ClusterNamespace.getNameFromFullName()` calls (~122 call sites), replacing each with its original argument - Remove all `import org.apache.doris.cluster.ClusterNamespace` statements - Delete `ClusterNamespace.java` - Clean up dead code left behind (no-op self-assignments, stale comments like "should be removed in 3.0", empty `removeClusterPrefix()` methods) - Fix import ordering to satisfy checkstyle Co-authored-by: Copilot <[email protected]>
Related PR: apache#61118 shoud only migrate InternalCatalog
…61305) it causes BE UT ccache always miss. now fixed.
…rite (apache#61224) ### What problem does this PR solve? SumLiteralRewrite transforms SUM(expr +/- literal) into SUM(expr) +/- literal * COUNT(expr). When type coercion has introduced an implicit widening cast (e.g. CAST(smallint_col AS INT)), the rewritten SUM/COUNT still operates on the wider type, forcing unnecessary wider data reads. This is redundant because SUM always returns BIGINT for any integer input (TINYINT/SMALLINT/INT/BIGINT). Strip implicit widening integer casts in extractSumLiteral() so the aggregate operates on the original narrow column directly. This benefits ClickBench Q29-style queries where SUM(col), SUM(col+1), SUM(col+2) share a narrow integer column — after stripping the cast, SUM(col+1) and SUM(col+2) reuse the existing SUM(col).
…educe memory (apache#61318) ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: Reduce FE memory by 1. moving top-N table stats filtering from PrometheusMetricVisitor into CloudTabletStatMgr so it's computed once per stat cycle instead of per Prometheus scrape, 2. removing the unused beToTablets field from InfightTask to avoid retaining a large map reference 3. changing InfightTablet.tabletId from Long to long to avoid boxing overhead. ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
…) (apache#61327) Issue Number: close #xxx Related PR: #xxx Problem Summary: None - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into --> ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
…t cases (apache#61333) ### What problem does this PR solve? The result of `array_union` does not guarantee element order, so the outputs of its related test cases are also unstable. Use `array_sort` to make the results deterministic. Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
## What This PR removes several redundant FE-side test dependencies. ## Why A quick dependency audit under `fe/` found a few dependencies that are declared but not actually used: - `hamcrest` in multiple FE modules - JUnit 5 test dependencies in `fe-extension-loader`, which currently has no test sources Keeping these unused dependencies increases dependency surface and maintenance cost without bringing value.
…failures (apache#61334) before it may silently fail. now try it best to return the status if failed.
…le agg to constant (apache#61183) ### What problem does this PR solve? - SimpleAggCacheMgr: singleton async cache backed by internal SQL queries for exact MIN/MAX column stats and row counts. Uses a two-level freshness check — visibleVersionTime (cheap local read) as a fast filter, falling back to getVisibleVersion() (potentially RPC in cloud mode) only when times match — to avoid unnecessary RPCs while still detecting same-millisecond concurrent writes. The cache loader captures version and versionTime both before and after executing the internal SQL to guard against writes that complete during query execution. - RewriteSimpleAggToConstantRule: Nereids rewrite rule that fires on LogicalAggregate over LogicalOlapScan, checks DUP_KEYS model, no GROUP BY, supported agg functions, and replaces with LogicalOneRowRelation containing constant literals. - Bug fix — TRUNCATE TABLE did not reset TableAttributes.visibleVersion: truncateTableInternal() replaced partition data but never called olapTable.resetVisibleVersion(). This left TableAttributes.visibleVersionTime unchanged, so the SimpleAggCacheMgr (and any other version-time-keyed logic) would see a cache hit and return pre-truncate values instead of the correct post-truncate result (e.g. count(*) = 5 instead of 0). The fix adds olapTable.resetVisibleVersion() inside truncateTableInternal(), which bumps both visibleVersion back to TABLE_INIT_VERSION and visibleVersionTime to System.currentTimeMillis(). This applies to both the live execution path and the journal-replay path since both go through truncateTableInternal. - Unit test with mock cache covering 6 positive and 8 negative cases. - Regression tests: -- Cache warmup polling, positive/negative explain checks, correctness assertions, and disable-rule verification. -- truncate_version_reset: verifies that after TRUNCATE TABLE, count(*) returns 0 (not a stale cached value), and that subsequent inserts are counted correctly.
…gregate (apache#60996) Add a new built-in scalar function to_hll(bigint) that constructs an HLL value from a BIGINT directly, bypassing any hash — the bigint is fed as-is into HyperLogLog::update(uint64_t). Changes: - BE (vec/functions/function_hll.cpp): implement struct ToHll with Status-returning execute/vector/vector_nullable methods; register as FunctionAlwaysNotNullable<ToHll, true> so InvalidArgument status is propagated on negative input. - BE test (be/test/vec/function/function_hll_test.cpp): add unit test function_hll_test.function_to_hll_test covering values 0, 1, 2, 100. - FE Nereids (scalar/ToHll.java): new ScalarFunction with signature BIGINT -> HLL. - FE catalog (BuiltinScalarFunctions.java): register to_hll. - FE visitor (ScalarFunctionVisitor.java): add visitToHll visitor hook.
### What problem does this PR solve? Issue Number: apache#61099 Related PR: N/A Problem Summary: - Avoid deadlock when external catalog refresh invalidates cache while another thread initializes the same catalog through a cache loader. - Add a unit test to reproduce the lock inversion and verify both worker threads also exit cleanly after the fix.
… framework (apache#61141) ### What problem does this PR solve? Refactor the JDBC data source scanning architecture to integrate with the unified FileScanner/JniReader framework, replacing the standalone ExternalScanNode-based JDBC scan path. #### Motivation The JDBC scan path was independently implemented with its own operator (JDBCScanLocalState), scanner (JdbcScanner), and JNI connector (JniConnector → BaseJdbcExecutor hierarchy), while other JNI-based data sources (Paimon, Hudi, MaxCompute, TrinoConnector) already use the unified FileScanner → JniReader → JniScanner path. This caused code duplication, maintenance burden, and architectural inconsistency. #### Changes 1. BE: Split JniConnector into two focused classes: - JniReader: JNI lifecycle management (open/read/close), base class for all JNI readers (PaimonJniReader, HudiJniReader, JdbcJniReader, etc.) - JniDataBridge: Stateless data exchange between C++ Blocks and Java shared memory via JNI 2. Java: Introduce Strategy Pattern for database-specific type handling: - JdbcTypeHandler interface with DefaultTypeHandler base implementation - Per-database handlers: MySQLTypeHandler, OracleTypeHandler, PostgreSQLTypeHandler, ClickHouseTypeHandler, SQLServerTypeHandler, DB2TypeHandler, SapHanaTypeHandler, TrinoTypeHandler, GbaseTypeHandler - JdbcTypeHandlerFactory for handler selection - JdbcJniScanner (extends JniScanner) for read path - JdbcJniWriter (extends JniWriter) for write path - Old BaseJdbcExecutor subclasses marked @deprecated but preserved 3. FE: Migrate JdbcScanNode from ExternalScanNode to FileQueryScanNode: - JdbcScanNode now extends FileQueryScanNode - Introduces JdbcSplit (extends FileSplit) to carry JDBC params - Uses TFileScanRange with FORMAT_JNI and table_format_type="jdbc" - Adds jdbc_params field to TTableFormatFileDesc in Thrift - Adapts PhysicalPlanTranslator.visitPhysicalJdbcScan() accordingly 4. BE: Add JdbcJniReader and integrate into FileScanner: - JdbcJniReader handles special types (bitmap, HLL, quantile_state, JSONB) via string-based intermediary and CAST - FileScanner._get_next_reader() adds "jdbc" table_format_type branch - JdbcUtils utility for JDBC driver URL resolution - Existing JdbcScanner preserved as transitional (deprecated) All predicate push-down logic (createJdbcFilters, getJdbcQueryStr, etc.) is preserved from the original JdbcScanNode implementation.
… quorum success (apache#60953) ## Summary When using majority write (quorum success), BE does not distinguish between replicas with continuous versions and replicas with version gaps (`lastFailedVersion >= 0`). This causes inconsistency with FE's commit check, which correctly excludes version-gap replicas from success counting. ## Bad Case Consider 3 replicas on nodes 1, 2, 3 with `load_required_replica_num = 2`: 1. **First write**: nodes 1,2 succeed, node 3 fails → overall success. Node 3 now has a version gap (`lastFailedVersion >= 0`). 2. **Second write**: nodes 1,3 succeed, node 2 fails → - **BE** counts 2 successes (nodes 1,3), considers it quorum success. - **FE** commit only counts node 1 as success (node 3 has version gap), so `successReplicaNum = 1 < 2`, commit fails. - This wastes resources since BE already returned success to the client but FE rejects the transaction. The correct behavior for the second write: - nodes 1,3 succeed → should **FAIL** (node 3 has version gap, only node 1 counts) - nodes 1,2 succeed → should **SUCCEED** (both have continuous versions) ## Solution Pass per-tablet version-gap backend information from FE to BE via a new thrift field `map<tablet_id, list<backend_id>> tablet_version_gap_backends` in `TOlapTablePartition`. On the BE side, when counting successful replicas for majority write in both `VTabletWriter` (V1) and `VTabletWriterV2`, exclude version-gap backends from the `finished_tablets_replica` counter. This makes BE's quorum check consistent with FE's commit check. ### Changes - **Descriptors.thrift**: Add `tablet_version_gap_backends` field to `TOlapTablePartition` - **OlapTable.java**: Add `getPartitionVersionGapBackends()` to compute gap backends per tablet - **OlapTableSink.java**: Populate the new field when building partition info - **tablet_info.h/cpp**: Parse and store gap backends from thrift - **vtablet_writer.cpp**: Exclude gap backends in `_quorum_success` - **vtablet_writer_v2.cpp**: Exclude gap backends in `_quorum_success` and `_create_commit_info`
…y data (apache#61236) ### What problem does this PR solve? When an Elasticsearch field is mapped as keyword/text but the actual data stored is an array (which is valid in ES), Doris was throwing: 'Expected value of type: STRING; but found type: Array' This fix serializes the array to a JSON string representation instead of throwing an error, which is consistent with how other non-string types (Numbers, Objects) are already handled. Fixed two code paths in es_scroll_parser.cpp: 1. fill_columns() - main _source parsing path 2. handle_value() - array element processing path Also added regression test and test data for this scenario.
|
Replace coarse-grained synchronized methods with ReentrantReadWriteLock and refactor erase methods to microbatch pattern to prevent cascading lock blocking when erasePartition runs slowly with many partitions. Key changes: - Change HashMap to ConcurrentHashMap for 4 internal data maps - Add ReentrantReadWriteLock with read/write lock helpers - Convert 8 simple lookup methods to lock-free (ConcurrentHashMap ops) - Convert 4 read-only iteration methods to use read lock - Convert 11 recycle/recover/clear methods to use write lock - Refactor 12 erase methods to microbatch pattern: process one item at a time with all operations (cleanup RPCs + map removal + edit log) inside write lock per item, releasing lock between items - Handle stale IDs in getIdListToEraseByRecycleTime with getOrDefault - Handle concurrent erase in cascade eraseDatabaseInstantly/ eraseTableInstantly with try-catch for DdlException - Add 3 concurrency unit tests validating concurrent reads, concurrent read+write, and microbatch lock release behavior Co-authored-by: Copilot <[email protected]>
|
morrySnow
pushed a commit
that referenced
this pull request
Mar 17, 2026
… exit (apache#61040) ### What problem does this PR solve? The SyncPoint singleton was being destroyed during program exit while background bthread timers were still accessing it, causing heap-use-after-free. Changed get_instance() to return a never-destructed heap-allocated pointer instead of a stack-allocated static object to prevent destruction order issues. This is a standard pattern for singletons accessed by background threads. ``` 11:47:40 #0 0x5582a963153d in operator new(unsigned long) (/root/doris/cloud/ut_build_ASAN/test/recycler_test+0x122b53d) (BuildId: 1191affa24589f52) 11:47:40 #1 0x5582ab0aa52f in doris::SyncPoint::SyncPoint() /root/doris/common/cpp/sync_point.cpp:77:9 11:47:40 #2 0x5582ab0aa46b in doris::SyncPoint::get_instance() /root/doris/common/cpp/sync_point.cpp:73:20 11:47:40 #3 0x5582aae33ee8 in doris::cloud::memkv::Transaction::get(std::basic_string_view<char, std::char_traits<char>>, std::basic_string_view<char, std::char_traits<char>>, std::unique_ptr<doris::cloud::RangeGetIterator, std::default_delete<doris::cloud::RangeGetIterator>>*, bool, int) /root/doris/cloud/src/meta-store/mem_txn_kv.cpp:276:5 11:47:40 #4 0x5582a968f49f in doris::cloud::txn_get(doris::cloud::TxnKv*, std::basic_string_view<char, std::char_traits<char>>, std::basic_string_view<char, std::char_traits<char>>, std::unique_ptr<doris::cloud::RangeGetIterator, std::default_delete<doris::cloud::RangeGetIterator>>&) /root/doris/cloud/src/recycler/recycler.cpp:99:18 11:47:40 apache#5 0x5582a9657090 in doris::cloud::InstanceRecycler::scan_and_recycle(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::basic_string_view<char, std::char_traits<char>>, std::function<int (std::basic_string_view<char, std::char_traits<char>>, std::basic_string_view<char, std::char_traits<char>>)>, std::function<int ()>) /root/doris/cloud/src/recycler/recycler.cpp:2976:23 11:47:40 apache#6 0x5582a9686153 in doris::cloud::InstanceRecycler::recycle_rowsets() /root/doris/cloud/src/recycler/recycler.cpp:2587:15 11:47:40 apache#7 0x5582a96a8d47 in doris::cloud::RecyclerTest_recycle_empty_Test::TestBody() /root/doris/cloud/test/recycler_test.cpp:1136:5 ```
morrySnow
pushed a commit
that referenced
this pull request
Mar 24, 2026
…free. (apache#61138) ### What problem does this PR solve? Related PR: apache/doris-thirdparty#383 Problem Summary: fix coredump: ``` #0 0x5561f1d46f12 in operator delete(void*, unsigned long) (/mnt/disk2/tengjianping/doris-master/output/be/lib/doris_be+0x50ae4f12) #1 0x55622c94b454 in std::__new_allocator<std::shared_ptr<orc::ExpressionTree>>::deallocate(std::shared_ptr<orc::ExpressionTree>*, unsigned long) (/mnt/disk2/tengjianping/doris-master/output/be/lib/doris_be+0x8b6e9454) #2 0x55622c94b41e in std::_Deque_base<std::shared_ptr<orc::ExpressionTree>, std::allocator<std::shared_ptr<orc::ExpressionTree>>>::_M_deallocate_node(std::shared_ptr<orc::ExpressionTree>*) (/mnt/disk2/tengjianping/doris-master/output/be/lib/doris_be+0x8b6e941e) #3 0x55622c950681 in std::deque<std::shared_ptr<orc::ExpressionTree>, std::allocator<std::shared_ptr<orc::ExpressionTree>>>::_M_pop_front_aux() (/mnt/disk2/tengjianping/doris-master/output/be/lib/doris_be+0x8b6ee681) #4 0x55622c94868e in std::deque<std::shared_ptr<orc::ExpressionTree>, std::allocator<std::shared_ptr<orc::ExpressionTree>>>::pop_front() (/mnt/disk2/tengjianping/doris-master/output/be/lib/doris_be+0x8b6e668e) apache#5 0x55622c947d79 in orc::rewriteLeaves(std::shared_ptr<orc::ExpressionTree>, unsigned long*) SearchArgument.cc apache#6 0x55622c947809 in orc::SearchArgumentBuilderImpl::build() (/mnt/disk2/tengjianping/doris-master/output/be/lib/doris_be+0x8b6e5809) apache#7 0x5562195209a6 in doris::vectorized::OrcReader::_init_search_argument(std::vector<std::shared_ptr<doris::vectorized::VExpr>, std::allocator<std::shared_ptr<doris::vectorized::VExpr>>> const&) /mnt/disk2/tengjianping/doris-master/be/src/vec/exec/format/orc/vorc_reader.cpp:1072:27 apache#8 0x556219527175 in doris::vectorized::OrcReader::set_fill_columns(std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::SlotDescriptor const*>, std::hash<std::__cxx11::basic_string<char, ```
morrySnow
pushed a commit
that referenced
this pull request
Mar 24, 2026
```
==1472432==ERROR: AddressSanitizer: heap-use-after-free on address 0x7bc7535fba50 at pc 0x557ec5178739 bp 0x7b1743746d40 sp 0x7b1743746500
WRITE of size 104 at 0x7bc7535fba50 thread T2
#0 0x557ec5178738 in memset ~/workspace/llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors_memintrinsics.inc:87:3
#1 0x7f17566e64fc in fdb_transaction_set_option (libfdb_c.so+0xc014fc) (BuildId: 854ec989d11d3c183aa4963a6d758cfc2b571b07)
#2 0x7f17566e2e47 in fdb_transaction_set_option (libfdb_c.so+0xbfde47) (BuildId: 854ec989d11d3c183aa4963a6d758cfc2b571b07)
#3 0x7f17566e4ec5 in fdb_transaction_set_option (libfdb_c.so+0xbffec5) (BuildId: 854ec989d11d3c183aa4963a6d758cfc2b571b07)
#4 0x7f17564f80ee in fdb_transaction_set_option (libfdb_c.so+0xa130ee) (BuildId: 854ec989d11d3c183aa4963a6d758cfc2b571b07)
apache#5 0x7f1755e55b71 in fdb_transaction_set_option (libfdb_c.so+0x370b71) (BuildId: 854ec989d11d3c183aa4963a6d758cfc2b571b07)
apache#6 0x7f1755e2b2e9 in fdb_run_network (libfdb_c.so+0x3462e9) (BuildId: 854ec989d11d3c183aa4963a6d758cfc2b571b07)
apache#7 0x557ec6595a2b in doris::cloud::fdb::Network::init()::::operator()() const ~/workspace/doris/cloud/ut_build_ASAN/../src/meta-store/txn_kv.cpp:468:28
apache#8 0x557ec65958e6 in void std::__invoke_impl<void, doris::cloud::fdb::Network::init()::>(std::__invoke_other, doris::cloud::fdb::Network::init()::&&) /usr/lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14
apache#9 0x557ec6595886 in std::__invoke_result<doris::cloud::fdb::Network::init()::>::type std::__invoke<doris::cloud::fdb::Network::init()::>(doris::cloud::fdb::Network::init()::&&) /usr/lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:96:14
apache#10 0x557ec659584e in void std::thread::_Invoker<std::tuple<doris::cloud::fdb::Network::init()::>>::_M_invoke<0ul>(std::_Index_tuple<0ul>) /usr/lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_thread.h:292:13
apache#11 0x557ec6595816 in std::thread::_Invoker<std::tuple<doris::cloud::fdb::Network::init()::>>::operator()() /usr/lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_thread.h:299:11
apache#12 0x557ec659571a in std::thread::_State_impl<std::thread::_Invoker<std::tuple<doris::cloud::fdb::Network::init()::>>>::_M_run() /usr/lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_thread.h:244:13
apache#13 0x557ec7756e43 in execute_native_thread_routine InstrProfilingUtil.c
apache#14 0x557ec50b1a07 in asan_thread_start(void*) ~/workspace/llvm-project/compiler-rt/lib/asan/asan_interceptors.cpp:239:43
apache#15 0x7f17545caaa3 in start_thread nptl/pthread_create.c:447:8
apache#16 0x7f1754657a63 in clone misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:100
```
morrySnow
pushed a commit
that referenced
this pull request
Mar 26, 2026
…pache#55500) ### What problem does this PR solve? Related PR: apache#45937 Problem Summary: Fix the error case on ingestion load and the core in parquet reader. ==8898==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x62f0020603fc at pc 0x55f634e64ded bp 0x7fba0d03c410 sp 0x7fba0d03bbd8 READ of size 4 at 0x62f0020603fc thread T768 (PUSH-9699) #0 0x55f634e64dec in __asan_memcpy (/mnt/hdd01/ci/doris-deploy-branch-3.1-local/be/lib/doris_be+0x39a24dec) (BuildId: 9b04e7f7d3075dac) #1 0x55f634eca93f in std::char_traits::copy(char*, char const*, unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/char_traits.h:409:33 #2 0x55f634eca93f in std::__cxx11::basic_string, std::allocator>::_S_copy(char*, char const*, unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:351:4 #3 0x55f634eca93f in std::__cxx11::basic_string, std::allocator>::_S_copy_chars(char*, char const*, char const*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:398:9 #4 0x55f634eca93f in void std::__cxx11::basic_string, std::allocator>::_M_construct(char const*, char const*, std::forward_iterator_tag) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.tcc:225:6 apache#5 0x55f654a4f74d in void std::__cxx11::basic_string, std::allocator>::_M_construct_aux(char const*, char const*, std::__false_type) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:247:11 apache#6 0x55f654a4f74d in void std::__cxx11::basic_string, std::allocator>::_M_construct(char const*, char const*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:266:4 apache#7 0x55f654a4f74d in std::__cxx11::basic_string, std::allocator>::basic_string(char const*, unsigned long, std::allocator const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:513:9 apache#8 0x55f654a4f74d in doris::vectorized::parse_thrift_footer(std::shared_ptr, doris::vectorized::FileMetaData**, unsigned long*, doris::io::IOContext*) /home/zcp/repo_center/doris_branch-3.1/doris/be/src/vec/exec/format/parquet/parquet_thrift_util.h:55:17
morrySnow
pushed a commit
that referenced
this pull request
Mar 26, 2026
…che#5027) pick apache#56854 ``` start BE in cloud mode, cloud_unique_id: ddc_cloud_unique_id_be, meta_service_endpoint: 172.20.56.123:5000 /mnt/disk3/pipeline/repo/selectdb-core_branch-selectdb-doris-3.1/selectdb-core/be/src/vec/sink/writer/vfile_result_writer.cpp:139:71: runtime error: load of value 190, which is not a valid value for type 'bool' #0 0x55b5e9a1e7a0 in doris::vectorized::VFileResultWriter::_create_file_writer(std::__cxx11::basic_string, std::allocator> const&) /mnt/disk3/pipeline/repo/selectdb-core_branch-selectdb-doris-3.1/selectdb-core/be/src/vec/sink/writer/vfile_result_writer.cpp:139:71 #1 0x55b5e9a1ac9f in doris::vectorized::VFileResultWriter::_create_next_file_writer() /mnt/disk3/pipeline/repo/selectdb-core_branch-selectdb-doris-3.1/selectdb-core/be/src/vec/sink/writer/vfile_result_writer.cpp:116:12 #2 0x55b5e9a16445 in doris::vectorized::VFileResultWriter::open(doris::RuntimeState*, doris::RuntimeProfile*) /mnt/disk3/pipeline/repo/selectdb-core_branch-selectdb-doris-3.1/selectdb-core/be/src/vec/sink/writer/vfile_result_writer.cpp:100:12 #3 0x55b5e7677e7c in doris::vectorized::AsyncResultWriter::process_block(doris::RuntimeState*, doris::RuntimeProfile*) /mnt/disk3/pipeline/repo/selectdb-core_branch-selectdb-doris-3.1/selectdb-core/be/src/vec/sink/writer/async_result_writer.cpp:106:23 #4 0x55b5e767ca7a in doris::vectorized::AsyncResultWriter::start_writer(doris::RuntimeState*, doris::RuntimeProfile*)::$_0::operator()() const /mnt/disk3/pipeline/repo/selectdb-core_branch-selectdb-doris-3.1/selectdb-core/be/src/vec/sink/writer/async_result_writer.cpp:93:5 apache#5 0x55b5e767ca7a in void std::__invoke_impl(std::__invoke_other, doris::vectorized::AsyncResultWriter::start_writer(doris::RuntimeState*, doris::RuntimeProfile*)::$_0&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 apache#6 0x55b5e767ca7a in std::enable_if, void>::type std::__invoke_r(doris::vectorized::AsyncResultWriter::start_writer(doris::RuntimeState*, doris::RuntimeProfile*)::$_0&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 apache#7 0x55b5e767ca7a in std::_Function_handler::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 apache#8 0x55b5b50496a7 in doris::ThreadPool::dispatch_thread() /mnt/disk3/pipeline/repo/selectdb-core_branch-selectdb-doris-3.1/selectdb-core/be/src/util/threadpool.cpp:602:24 apache#9 0x55b5b50206de in std::function::operator()() const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9 apache#10 0x55b5b50206de in doris::Thread::supervise_thread(void*) /mnt/disk3/pipeline/repo/selectdb-core_branch-selectdb-doris-3.1/selectdb-core/be/src/util/thread.cpp:498:5 ```
morrySnow
pushed a commit
that referenced
this pull request
Jul 10, 2026
Issue Number: None
Related PR: None
Problem Summary:
UBSAN/ASAN reported a heap-use-after-free during BE shutdown in
TimeSharingTaskExecutor destruction. The scan executor owned pending
split runners, while the split runner function held ScannerContext, and
ScannerContext cleaned up its task by copying the executor shared_ptr
back from the scheduler. When the scheduler was already destroying that
shared_ptr, copying the same shared_ptr member could re-enter the
executor destruction path and release members such as MetricEntity
twice. Keep ScannerContext cleanup tied to a weak reference of the
executor captured when the task is created, and avoid capturing the
scheduler this pointer in split cleanup lambdas.
旧代码的链路大致是:
1. TaskExecutorSimplifiedScanScheduler 析构,成员 _task_executor 是最后阶段要销毁的
std::shared_ptr<TaskExecutor>。
2. 这个 shared_ptr 引用计数降到 0,于是开始执行
TimeSharingTaskExecutor::~TimeSharingTaskExecutor()。
3. TimeSharingTaskExecutor 析构时会释放它持有的 split/task 容器。
4. split 里有 ScannerSplitRunner,它的 std::function 捕获了
std::shared_ptr<ScannerContext>。
5. 释放 split 会释放这个 captured ScannerContext,于是进入
ScannerContext::~ScannerContext()。
6. 旧代码在 ScannerContext::~ScannerContext() 里又执行:
task_executor_scheduler->task_executor()->remove_task(_task_handle)
而 task_executor() 返回的是:
std::shared_ptr<TaskExecutor> task_executor() const { return
_task_executor; }
也就是从 scheduler 的 _task_executor 成员再复制一个 shared_ptr。
问题在于:第 1 步里,scheduler 的 _task_executor 这个 shared_ptr 正在析构,并且它已经触发了
TimeSharingTaskExecutor 的析构。此时再从同一个 member
shared_ptr 复制引用,等价于在其析构过程中重新操作它的 control block。这个 control block 可能已经处于
release-last-use 路径,随后临时复制出来的
shared_ptr 析构又会再次 release 同一个 control block,导致 TimeSharingTaskExecutor
析构链被重入或二次释放其成员。
```
/usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:1069:11: runtime error: member call on address 0x7c63e14891c0 which does not point to an object of type 'std::_Sp_counted_base<>'
0x7c63e14891c0: note: object has invalid vptr
00 00 00 00 e0 74 1d 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 be be be be d0 26 2b e1
^~~~~~~~~~~~~~~~~~~~~~~
invalid vptr
#0 0x55e77069caee in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:1069:11
#1 0x55e77069caee in std::__shared_ptr<doris::MetricEntity, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:1531:31
#2 0x55e77069caee in doris::TimeSharingTaskExecutor::~TimeSharingTaskExecutor() be/build_ASAN/./be/build_ASAN/../src/exec/scan/task_executor/time_sharing/time_sharing_task_executor.cpp:306:1
#3 0x55e75b584397 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use() /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:174:2
#4 0x55e7706486ac in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:1069:11
apache#5 0x55e7706486ac in std::__shared_ptr<doris::TaskExecutor, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:1531:31
apache#6 0x55e7706486ac in doris::ScannerContext::~ScannerContext() be/build_ASAN/./be/build_ASAN/../src/exec/scan/scanner_context.cpp:277:13
apache#7 0x55e75b584397 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use() /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:174:2
apache#8 0x55e7706f288d in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:1069:11
apache#9 0x55e7706f288d in std::__shared_ptr<doris::ScannerContext, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:1531:31
apache#10 0x55e7706f288d in doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()::~() be/build_ASAN/./be/build_ASAN/../src/exec/scan/scanner_scheduler.cpp:77:26
apache#11 0x55e7706f39bb in std::_Function_base::_Base_manager<doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()>::_M_destroy(std::_Any_data&, std::integral_constant<bool, false>) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:177:4
apache#12 0x55e7706f39bb in std::_Function_base::_Base_manager<doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()>::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:205:8
apache#13 0x55e7706f39bb in std::_Function_handler<bool (), doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()>::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:284:6
apache#14 0x55e770702246 in std::_Function_base::~_Function_base() /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:246:2
apache#15 0x55e770702246 in doris::ScannerSplitRunner::~ScannerSplitRunner() be/build_ASAN/../src/exec/scan/scanner_scheduler.h:62:7
apache#16 0x55e75b584397 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use() /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:174:2
apache#17 0x55e7706e11e9 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:1069:11
apache#18 0x55e7706e11e9 in std::__shared_ptr<doris::SplitRunner, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:1531:31
apache#19 0x55e7706e11e9 in doris::PrioritizedSplitRunner::~PrioritizedSplitRunner() be/build_ASAN/../src/exec/scan/task_executor/time_sharing/prioritized_split_runner.h:54:47
apache#20 0x55e75b584397 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use() /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:174:2
apache#21 0x55e7706b8ba4 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:1069:11
apache#22 0x55e7706b8ba4 in std::__shared_ptr<doris::PrioritizedSplitRunner, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:1531:31
apache#23 0x55e7706b8ba4 in void std::destroy_at<std::shared_ptr<doris::PrioritizedSplitRunner> >(std::shared_ptr<doris::PrioritizedSplitRunner>*) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/stl_construct.h:88:15
apache#24 0x55e7706b8ba4 in void std::allocator_traits<std::allocator<std::__detail::_Hash_node<std::shared_ptr<doris::PrioritizedSplitRunner>, false> > >::destroy<std::shared_ptr<doris::PrioritizedSplitRunner> >(std::allocator<std::__detail::_Hash_node<std::shared_ptr<doris::PrioritizedSplitRunner>, false> >&, std::shared_ptr<doris::PrioritizedSplitRunner>*) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/alloc_traits.h:698:4
apache#25 0x55e7706b8ba4 in std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::shared_ptr<doris::PrioritizedSplitRunner>, false> > >::_M_deallocate_node(std::__detail::_Hash_node<std::shared_ptr<doris::PrioritizedSplitRunner>, false>*) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/hashtable_policy.h:1572:7
apache#26 0x55e7706b89ff in std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::shared_ptr<doris::PrioritizedSplitRunner>, false> > >::_M_deallocate_nodes(std::__detail::_Hash_node<std::shared_ptr<doris::PrioritizedSplitRunner>, false>*) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/hashtable_policy.h:1594:4
apache#27 0x55e7706b89ff in std::_Hashtable<std::shared_ptr<doris::PrioritizedSplitRunner>, std::shared_ptr<doris::PrioritizedSplitRunner>, std::allocator<std::shared_ptr<doris::PrioritizedSplitRunner> >, std::__detail::_Identity, std::equal_to<std::shared_ptr<doris::PrioritizedSplitRunner> >, std::hash<std::shared_ptr<doris::PrioritizedSplitRunner> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, true, true> >::~_Hashtable() /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/hashtable.h:1852:13
apache#28 0x55e77069c6ca in std::unordered_set<std::shared_ptr<doris::PrioritizedSplitRunner>, std::hash<std::shared_ptr<doris::PrioritizedSplitRunner> >, std::equal_to<std::shared_ptr<doris::PrioritizedSplitRunner> >, std::allocator<std::shared_ptr<doris::PrioritizedSplitRunner> > >::~unordered_set() /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/unordered_set.h:107:11
apache#29 0x55e77069c6ca in doris::TimeSharingTaskExecutor::~TimeSharingTaskExecutor() be/build_ASAN/./be/build_ASAN/../src/exec/scan/task_executor/time_sharing/time_sharing_task_executor.cpp:306:1
apache#30 0x55e75b583fa1 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:345:8
apache#31 0x55e770688ecf in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:1069:11
apache#32 0x55e770688ecf in std::__shared_ptr<doris::TaskExecutor, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:1531:31
apache#33 0x55e770688ecf in doris::TaskExecutorSimplifiedScanScheduler::~TaskExecutorSimplifiedScanScheduler() be/build_ASAN/../src/exec/scan/scanner_scheduler.h:266:5
apache#34 0x55e7706891ed in doris::TaskExecutorSimplifiedScanScheduler::~TaskExecutorSimplifiedScanScheduler() be/build_ASAN/../src/exec/scan/scanner_scheduler.h:261:53
apache#35 0x55e7744d5afa in std::default_delete<doris::ScannerScheduler>::operator()(doris::ScannerScheduler*) const /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/unique_ptr.h:93:2
apache#36 0x55e7744d5afa in std::__uniq_ptr_impl<doris::ScannerScheduler, std::default_delete<doris::ScannerScheduler> >::reset(doris::ScannerScheduler*) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/unique_ptr.h:205:4
apache#37 0x55e7744d5afa in std::unique_ptr<doris::ScannerScheduler, std::default_delete<doris::ScannerScheduler> >::reset(doris::ScannerScheduler*) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/unique_ptr.h:512:7
apache#38 0x55e7744d5afa in doris::WorkloadGroup::destroy_schedulers() be/build_ASAN/./be/build_ASAN/../src/runtime/workload_group/workload_group.cpp:776:22
apache#39 0x55e774501af0 in doris::WorkloadGroupMgr::destroy_schedulers() be/build_ASAN/./be/build_ASAN/../src/runtime/workload_group/workload_group_manager.cpp:977:23
apache#40 0x55e7740e0ddc in doris::ExecEnv::destroy() be/build_ASAN/./be/build_ASAN/../src/runtime/exec_env_init.cpp:892:34
apache#41 0x55e75b5601f2 in main be/build_ASAN/./be/build_ASAN/../src/service/doris_main.cpp:734:15
apache#42 0x7f53e1e6b082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/../csu/libc-start.c:308:16
apache#43 0x55e75b470029 in _start (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x247b8029)
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:1069:11
```
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
morrySnow
pushed a commit
that referenced
this pull request
Aug 31, 2026
Problem Summary: Streaming `ROWS` window aggregates retain state across
frame evaluations. The eviction path previously considered only whether
buffered blocks had been emitted, so it could erase either the outgoing
row needed by a bounded sliding frame or the next unread row needed by
an `UNBOUNDED PRECEDING ... N PRECEDING` frame. After rebasing, negative
partition and outgoing positions could allow a nullable aggregate to
access its null map out of bounds. Evicting either kind of required row
could also produce incorrect aggregate results.
Root cause: `_remove_unused_rows()` did not account for the earliest row
required by the next ROWS frame evaluation, and
`BoundaryPose::remove_unused_rows()` allowed retained-column coordinates
to become negative.
This change defers block-aligned eviction when the candidate prefix
contains either `frame_start - 1`, the outgoing row required by a
bounded sliding update, or the next unread row required by an `UNBOUNDED
PRECEDING ... N PRECEDING` frame. It also rebases partition and order
boundaries to nonnegative physical-column coordinates. The BE unit
coverage exercises both ROWS executors across eviction boundaries and
verifies boundary rebasing.
Observed ASAN failure before this change (`output/be/log/be.out`):
```text
ERROR: AddressSanitizer: heap-buffer-overflow
READ of size 1
#0 doris::AggregateFunctionNullUnaryInlineV2<...>::execute_function_with_incremental(...)
be/src/exprs/aggregate/aggregate_function_null_v2.h:595
#1 doris::AggFnEvaluator::execute_function_with_incremental(...)
be/src/exprs/vectorized_agg_fn.cpp:334
#2 doris::AnalyticSinkLocalState::_execute_for_function<true>(...)
be/src/exec/operator/analytic_sink_operator.cpp:385
#3 doris::AnalyticSinkLocalState::_get_next_for_sliding_rows(...)
be/src/exec/operator/analytic_sink_operator.cpp:203
#4 doris::AnalyticSinkLocalState::_execute_impl(...)
be/src/exec/operator/analytic_sink_operator.cpp:358
apache#5 doris::AnalyticSinkOperatorX::sink_impl(...)
be/src/exec/operator/analytic_sink_operator.cpp:757
SUMMARY: AddressSanitizer: heap-buffer-overflow in
doris::AggregateFunctionNullUnaryInlineV2<...>::execute_function_with_incremental(...)
```
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
All methods in
CatalogRecycleBin.javausesynchronized(single monitor lock), creating extremely coarse lock granularity. WhenerasePartition()runs slowly with many partitions, othersynchronizedmethods block waiting for the lock. Callers likerecyclePartition()hold TABLE WRITE LOCK while waiting, causing cascading blocking that can bring down the entire Doris metadata service.Solution
Two complementary optimizations:
1. Replace
synchronizedwithReentrantReadWriteLockisRecyclePartition,getRecycleTimeById, etc.)allTabletsInRecycledStatus,getInfo,write, etc.)recycleDatabase/Table/Partition,recover*,clearAll)2. Microbatch Erase Pattern (Critical)
Refactored all 12 erase methods to process items one at a time with lock release between items:
This reduces lock hold time from O(N × T) (all items) to O(T) (one item) per acquisition.
Data Structure Changes
Changed 4 internal maps from
HashMaptoConcurrentHashMapto enable lock-free reads.Bug Fixes (found during self-review)
getIdListToEraseByRecycleTime: UsedgetOrDefaultto handle stale IDs that may be concurrently removed between snapshot and processingeraseDatabaseInstantly/eraseTableInstantlyfor partitions/tables concurrently erased by daemonTesting
testConcurrentReadsDoNotBlock— 10 concurrent reader threadstestConcurrentRecycleAndRead— writer + 5 readers simultaneouslytestMicrobatchEraseReleasesLockBetweenItems— verifies recyclePartition succeeds during erase