Ivm pick create alter validation by seawinde · Pull Request #6 · yujun777/doris

seawinde · 2026-04-15T16:13:36Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

) Issue Number: close #xxx Problem Summary: 1. Remove redundant pre-analysis in IVM analyzeQuery - single analyzeQueryInternal call 2. Unify IVM base table validation error messages (AGG_KEYS vs UNIQUE without MOW) 3. Add excluded_trigger_tables support in IvmNormalizeMtmv (transient row-id for excluded tables) 4. Block ALTER MTMV refresh method to/from INCREMENTAL 5. Validate base table models when ALTER MTMV excluded_trigger_tables 6. Extract MTMVPropertyUtil.parseTableNameInfos utility 7. Add comprehensive unit tests for all validation paths

### What problem does this PR solve? Issue Number: close #xxx Problem Summary: Commit 71a3086 introduced a new StatementContext in analyzeQueryInternal() and restored the original (parser-created) StatementContext in the finally block. This caused ExprId collisions during IVM INCREMENTAL refresh because the parser StatementContext has a much smaller ExprId counter than the analysis StatementContext, and IvmRefreshManager.doRefreshInternal() reads exprIdStart from the ConnectContext's StatementContext after analysis. The fix reverts to the pre-71a3086f pattern: reuse ctx.getStatementContext() (the parser-created StatementContext) directly for analysis instead of creating a new one. This way all ExprId allocations accumulate in the same StatementContext that doRefreshInternal() later reads. ### Release note None ### Check List (For Author) - Test: Regression test (test_ivm_agg_2) - Behavior changed: No - Does this need documentation: No Co-authored-by: Copilot <[email protected]>

…r tables ### What problem does this PR solve? Issue Number: close #xxx Problem Summary: buildRowId() short-circuited to UuidNumeric() for excluded trigger tables, losing the deterministic row-id (hash of unique keys) for MOW and non-MOW UNIQUE_KEYS tables. The fix removes the early return and only uses the isExcludedTriggerTable flag to suppress AnalysisException for unsupported table types (AGG_KEYS etc.), while UNIQUE_KEYS tables always compute buildRowIdHash(keySlots) regardless of exclusion status. ### Release note None ### Check List (For Author) - Test: Unit Test (IvmNormalizeMtmvTest) - Behavior changed: No - Does this need documentation: No Co-authored-by: Copilot <[email protected]>

### What problem does this PR solve? Issue Number: close #xxx Problem Summary: Update testExcludedMowTableUsesTransientRowId to expect deterministic hash-based row-id (Cast expression) instead of UuidNumeric for excluded MOW tables, matching the fix in buildRowId(). ### Release note None ### Check List (For Author) - Test: Unit Test (IvmNormalizeMtmvTest - 27 tests pass) - Behavior changed: No - Does this need documentation: No Co-authored-by: Copilot <[email protected]>

### What problem does this PR solve? Issue Number: close #xxx Problem Summary: The ALTER MTMV refresh method validation was too restrictive (blocking all changes to/from INCREMENTAL) or too permissive in some cases. The new rules: - COMPLETE <-> INCREMENTAL: forbidden (must recreate MV) - COMPLETE/INCREMENTAL -> AUTO: allowed - AUTO -> COMPLETE/INCREMENTAL: forbidden (must recreate MV) - Same method (no-op): allowed ### Release note None ### Check List (For Author) - Test: Regression test - Behavior changed: Yes (COMPLETE/INCREMENTAL to AUTO now allowed) - Does this need documentation: No Co-authored-by: Copilot <[email protected]>

### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: CreateMTMVCommandTest reused the same StatementContext across multiple statements, so table and excluded_trigger_tables state leaked between test cases and caused false incremental MV validation failures. This change resets the statement context for each statement in the test and adds the missing LinkedHashSet import required by StatementContext. ### Release note None ### Check List (For Author) - Test: FE unit test - ./run-fe-ut.sh --run org.apache.doris.mtmv.ivm.IvmAggDeltaStrategyTest,org.apache.doris.mtmv.ivm.IvmDeltaExecutorTest,org.apache.doris.mtmv.ivm.IvmDeltaRewriterTest,org.apache.doris.mtmv.ivm.IvmRefreshManagerTest,org.apache.doris.mtmv.ivm.IvmSimpleScanDeltaStrategyTest,org.apache.doris.mtmv.ivm.IvmUtilTest,org.apache.doris.nereids.rules.rewrite.IvmNormalizeMtmvTest,org.apache.doris.nereids.trees.plans.CreateMTMVCommandTest,org.apache.doris.catalog.ShowCreateMTMVTest - Behavior changed: No - Does this need documentation: No

### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: Rebase onto the latest yujun777/ivm left several FE IVM files importing org.apache.doris.info.TableNameInfo while the current base branch provides the type in org.apache.doris.catalog.info.TableNameInfo. This fixes the stale imports so the rebased branch compiles and the targeted IVM FE tests run successfully again. ### Release note None ### Check List (For Author) - Test: Unit Test - org.apache.doris.mtmv.ivm.IvmAggDeltaStrategyTest - org.apache.doris.mtmv.ivm.IvmDeltaExecutorTest - org.apache.doris.mtmv.ivm.IvmDeltaRewriterTest - org.apache.doris.mtmv.ivm.IvmRefreshManagerTest - org.apache.doris.mtmv.ivm.IvmSimpleScanDeltaStrategyTest - org.apache.doris.mtmv.ivm.IvmUtilTest - org.apache.doris.nereids.rules.rewrite.IvmNormalizeMtmvTest - org.apache.doris.nereids.trees.plans.CreateMTMVCommandTest - org.apache.doris.catalog.ShowCreateMTMVTest - Behavior changed: No - Does this need documentation: No

yujun777 · 2026-04-16T03:05:05Z

+                                    PropertyAnalyzer.PROPERTIES_EXCLUDED_TRIGGER_TABLES)) {
+                        Map<String, String> mergedMvProps = Maps.newHashMap(mtmv.getMvProperties());
+                        mergedMvProps.putAll(alterMTMV.getMvProperties());
+                        MTMVPlanUtil.validateIncrementalBaseTableModels(


alter trigger 时不需要再检查基表类型了，并且trigger 表范围只能扩大，不能缩减，原来在trigger中的表，不能在alter中把它移除

目前trigger中的表，alter中可以移除

ivm exclude的不应该移除，不然binlog不知道从哪里消费

yujun777 · 2026-04-16T03:06:49Z

+        MTMVAnalyzeQueryInfo queryInfo = analyzeQueryInternal(ctx, mvProperties, mvPartitionDefinition, distribution,
+                simpleColumnDefinitions, properties, keys, logicalQuery, true,
+                getExcludedTriggerTables(mvProperties));
+        validateIncrementalBaseTableModels(queryInfo.getRelation(), mvProperties);


这里不用再validate检查了，IvmNormalizeMtmv时已经有检查了

OK，移除掉，因为修改trigger的时候，还要校验，所以这个方法得保留下

yujun777 · 2026-04-16T03:07:50Z

+                mvProperties.get(PropertyAnalyzer.PROPERTIES_EXCLUDED_TRIGGER_TABLES));
+    }
+
+    public static void validateIncrementalBaseTableModels(MTMVRelation relation, Map<String, String> mvProperties) {


前面两个调用的地方都去掉了，这个函数可以删了

yujun777 · 2026-04-16T03:10:26Z

-                + " for table: " + table.getName()
-                + ". Only MOW (UNIQUE_KEYS with merge-on-write) and DUP_KEYS are supported.");
+        if (isExcludedTriggerTable) {
+            return Pair.of(new UuidNumeric(), false);


agg表用rowid = hash(key)

已经修改

yujun777 · 2026-04-16T03:16:01Z

+        // method returns (IvmRefreshManager.doRefreshInternal reads exprIdStart). Creating a
+        // separate StatementContext and restoring the original would lose ExprId allocations.
        try (StatementContext statementContext = ctx.getStatementContext()) {
+            statementContext.setExcludedTriggerTables(excludedTriggerTables);


增量刷新和全量刷新时，不是走analyzeQueryInternal这个函数的。
它两的statement context也要设置上exclude trigger table。

增加一个回归case，基表是AGG表的。并且要手工刷新，测试增量刷新和全量刷新的，并且检查物化表数据有没被更新，现在exclude实际没排除的，所有两种mv表都会输出数据

OK，需要添加下

yujun777 · 2026-04-16T03:23:50Z

-    private Pair<Expression, Boolean> buildRowId(OlapTable table, LogicalOlapScan scan) {
+    private Pair<Expression, Boolean> buildRowId(OlapTable table, LogicalOlapScan scan,
+            boolean isExcludedTriggerTable) {
        KeysType keysType = table.getKeysType();


建议isExcludedTriggerTable 参数从函数签名中移走，放到函数内部逻辑上更好一些，因为row id是 table 类型决定自身就决定好的，

这个是个方法调整，可以调整，可以先合，后续再调下

yujun777 · 2026-04-16T03:26:04Z

+                    .getTableOrMetaException(getMvName().getTbl(), TableIf.TableType.MATERIALIZED_VIEW);
+            RefreshMethod oldMethod = mtmv.getRefreshInfo().getRefreshMethod();
+            RefreshMethod newMethod = refreshInfo.getRefreshMethod();
+            if (newMethod != null && newMethod != oldMethod) {


这里的判断建议简洁一点
if (newMethod == INCREMNTAL) {
throw exception
}
if (mv.isIvm()) {
throw exception
}

COMPLETE or INCREMENTAL 是不是可以改成auto 如果可以的话，简化逻辑会报错

### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: IVM normalization treated excluded AGG_KEYS base tables as transient-row-id scans and generated uuid_numeric() row ids. This changed excluded AGG_KEYS row identity across refreshes and did not follow the base table aggregate keys. Fix buildRowId to hash base-schema key columns for excluded AGG_KEYS tables, and add focused tests to verify the row id is deterministic and excludes non-key value columns. ### Release note None ### Check List (For Author) - Test: Unit Test - org.apache.doris.mtmv.ivm.IvmAggDeltaStrategyTest - org.apache.doris.mtmv.ivm.IvmDeltaExecutorTest - org.apache.doris.mtmv.ivm.IvmDeltaRewriterTest - org.apache.doris.mtmv.ivm.IvmRefreshManagerTest - org.apache.doris.mtmv.ivm.IvmSimpleScanDeltaStrategyTest - org.apache.doris.mtmv.ivm.IvmUtilTest - org.apache.doris.nereids.rules.rewrite.IvmNormalizeMtmvTest - org.apache.doris.nereids.trees.plans.CreateMTMVCommandTest - org.apache.doris.catalog.ShowCreateMTMVTest - Behavior changed: Yes (excluded AGG_KEYS row-id generation is now deterministic on agg keys) - Does this need documentation: No

abcxyz333

LGTM

github-actions · 2026-04-16T10:39:42Z

PR approved by anyone and no changes requested.

### What problem does this PR solve? Fix memory corruption in PredicateColumnType::insert_duplicate_fields for string types Fixed a bug in the pointer arithmetic when inserting duplicate string values. The original code used dst += i * str.size() which incorrectly multiplied by the loop index, causing: - Iteration 0: no advance (correct by accident) - Iteration 1: advance by 1×size (correct) - Iteration 2: advance by 2×size (wrong - overwrites beyond allocated memory) This resulted in memory corruption and incorrect string data when inserting more than 2 duplicates. The fix changes to dst += str.size() to correctly advance the pointer by one string length per iteration. Added unit tests covering: - String type with multiple duplicate insertions - Integer type duplicates - LargeInt (Int128) type duplicates Introduced by apache#60530 ==13605==ERROR: AddressSanitizer: use-after-poison on address 0x7d639d26bc30 at pc 0x55e8b71df91e bp 0x7b4e8fc1e510 sp 0x7b4e8fc1dcd0 WRITE of size 6 at 0x7d639d26bc30 thread T620 (ls_normal [work) #0 0x55e8b71df91d in __asan_memcpy (/mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be+0x2205791d) #1 0x55e8cce8b1ab in doris::PredicateColumnType<(doris::PrimitiveType)23>::insert_duplicate_fields(doris::Field const&, unsigned long) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/core/column/predicate_column.h:327:17 #2 0x55e8b7db4e83 in doris::ColumnNullable::insert_duplicate_fields(doris::Field const&, unsigned long) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/core/column/column_nullable.cpp:357:29 #3 0x55e8ccf8ba2c in doris::segment_v2::DefaultValueColumnIterator::_insert_many_default(doris::COW<doris::IColumn>::mutable_ptr<doris::IColumn>&, unsigned long) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/segment/column_reader.cpp:2167:14 #4 0x55e8ccf3c705 in doris::segment_v2::DefaultValueColumnIterator::next_batch(unsigned long*, doris::COW<doris::IColumn>::mutable_ptr<doris::IColumn>&, bool*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/segment/column_reader.cpp:2152:5 #5 0x55e8cd4f9313 in doris::segment_v2::ColumnIterator::next_batch(unsigned long*, doris::COW<doris::IColumn>::mutable_ptr<doris::IColumn>&) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/segment/column_reader.h:330:16 #6 0x55e8cd4f9313 in doris::segment_v2::SegmentIterator::_read_columns_by_index(unsigned int, unsigned short&) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/segment/segment_iterator.cpp:2291:13 #7 0x55e8cd50621b in doris::segment_v2::SegmentIterator::_next_batch_internal(doris::Block*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/segment/segment_iterator.cpp:2643:5 #8 0x55e8cd4a0b82 in doris::segment_v2::SegmentIterator::next_batch(doris::Block*)::$_0::operator()() const /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/segment/segment_iterator.cpp:2519:9 #9 0x55e8cd4a0b82 in doris::segment_v2::SegmentIterator::next_batch(doris::Block*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/segment/segment_iterator.cpp:2518:19 #10 0x55e8cd37cca1 in doris::segment_v2::LazyInitSegmentIterator::next_batch(doris::Block*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/segment/lazy_init_segment_iterator.h:46:33 #11 0x55e8cccc0b05 in doris::Status doris::BetaRowsetReader::_next_batch<doris::Block>(doris::Block*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/rowset/beta_rowset_reader.h:113:35 #12 0x55e8ccc98668 in doris::BetaRowsetReader::next_batch(doris::Block*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/rowset/beta_rowset_reader.h:56:55 #13 0x55e8ca62ee8f in doris::VCollectIterator::Level0Iterator::_refresh() /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/iterator/vcollect_iterator.h #14 0x55e8ca611036 in doris::VCollectIterator::Level0Iterator::refresh_current_row() /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/iterator/vcollect_iterator.cpp:523:24 #15 0x55e8ca601a08 in doris::VCollectIterator::Level0Iterator::ensure_first_row_ref() /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/iterator/vcollect_iterator.cpp:496:14 #16 0x55e8ca604ea2 in doris::VCollectIterator::Level1Iterator::ensure_first_row_ref() /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/iterator/vcollect_iterator.cpp:713:27 #17 0x55e8ca608b20 in doris::VCollectIterator::build_heap(std::vector<std::shared_ptr<doris::RowsetReader>, std::allocator<std::shared_ptr<doris::RowsetReader>>>&) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/iterator/vcollect_iterator.cpp:189:9 #18 0x55e8ca5dc1ae in doris::BlockReader::_init_collect_iter(doris::TabletReader::ReaderParams const&) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/iterator/block_reader.cpp:152:9 #19 0x55e8ca5d7b9d in doris::BlockReader::init(doris::TabletReader::ReaderParams const&) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/iterator/block_reader.cpp:226:19 #20 0x55e8d02df6d6 in doris::OlapScanner::_open_impl(doris::RuntimeState*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/olap_scanner.cpp:317:32 #21 0x55e8d02cd697 in doris::Scanner::open(doris::RuntimeState*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/scanner.h:80:16 #22 0x55e8d02b576a in doris::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/scanner_scheduler.cpp:185:5 #23 0x55e8d02bf603 in doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/scanner_scheduler.cpp:75:17 #24 0x55e8d02bf603 in doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()::operator()() const /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/scanner_scheduler.cpp:74:27 #25 0x55e8d02bf603 in bool std::__invoke_impl<bool, doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()&>(std::__invoke_other, doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:63:14 #26 0x55e8d02bf603 in std::enable_if<is_invocable_r_v<bool, doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()&>, bool>::type std::__invoke_r<bool, doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()&>(doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:116:9 #27 0x55e8d02bf603 in std::_Function_handler<bool (), doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()>::_M_invoke(std::_Any_data const&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:292:9 #28 0x55e8d02bed85 in std::function<bool ()>::operator()() const /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593:9 #29 0x55e8d02bed85 in doris::ScannerSplitRunner::process_for(std::chrono::duration<long, std::ratio<1l, 1000000000l>>) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/scanner_scheduler.cpp:425:25 #30 0x55e8d02aa844 in doris::PrioritizedSplitRunner::process() /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/task_executor/time_sharing/prioritized_split_runner.cpp:102:35 #31 0x55e8d0272674 in doris::TimeSharingTaskExecutor::_dispatch_thread() /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/task_executor/time_sharing/time_sharing_task_executor.cpp:556:77 #32 0x55e8d3fa8ad1 in std::function<void ()>::operator()() const /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593:9 #33 0x55e8d3fa8ad1 in doris::Thread::supervise_thread(void*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/util/thread.cpp:460:5 #34 0x55e8b71ddd26 in asan_thread_start(void*) (/mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be+0x22055d26) apache#35 0x7f539b835ac2 in start_thread nptl/pthread_create.c:442:8 apache#36 0x7f539b8c78cf misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 0x7d639d26bc30 is located 816 bytes inside of 4096-byte region [0x7d639d26b900,0x7d639d26c900) allocated by thread T620 (ls_normal [work) here: #0 0x55e8b71e1b34 in malloc (/mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be+0x22059b34) #1 0x55e8b78d2140 in doris::DefaultMemoryAllocator::malloc(unsigned long) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/core/allocator.h:90:55 #2 0x55e8b78d2140 in doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, true>::alloc(unsigned long, unsigned long) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/core/allocator.cpp:337:23 #3 0x55e8b7ccdfaf in doris::Arena::Chunk::Chunk(unsigned long, doris::Arena::Chunk*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/core/arena.h:59:63 #4 0x55e8b7ccdad9 in doris::Arena::_init_head_if_needed() /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/core/arena.h:131:24 #5 0x55e8b7ccd758 in doris::Arena::alloc(unsigned long) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/core/arena.h:148:9 #6 0x55e8cce8b118 in doris::PredicateColumnType<(doris::PrimitiveType)23>::insert_duplicate_fields(doris::Field const&, unsigned long) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/core/column/predicate_column.h:325:32 #7 0x55e8b7db4e83 in doris::ColumnNullable::insert_duplicate_fields(doris::Field const&, unsigned long) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/core/column/column_nullable.cpp:357:29 #8 0x55e8ccf8ba2c in doris::segment_v2::DefaultValueColumnIterator::_insert_many_default(doris::COW<doris::IColumn>::mutable_ptr<doris::IColumn>&, unsigned long) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/segment/column_reader.cpp:2167:14 #9 0x55e8ccf3c705 in doris::segment_v2::DefaultValueColumnIterator::next_batch(unsigned long*, doris::COW<doris::IColumn>::mutable_ptr<doris::IColumn>&, bool*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/segment/column_reader.cpp:2152:5 #10 0x55e8cd4f9313 in doris::segment_v2::ColumnIterator::next_batch(unsigned long*, doris::COW<doris::IColumn>::mutable_ptr<doris::IColumn>&) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/segment/column_reader.h:330:16 #11 0x55e8cd4f9313 in doris::segment_v2::SegmentIterator::_read_columns_by_index(unsigned int, unsigned short&) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/segment/segment_iterator.cpp:2291:13 #12 0x55e8cd50621b in doris::segment_v2::SegmentIterator::_next_batch_internal(doris::Block*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/segment/segment_iterator.cpp:2643:5 #13 0x55e8cd4a0b82 in doris::segment_v2::SegmentIterator::next_batch(doris::Block*)::$_0::operator()() const /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/segment/segment_iterator.cpp:2519:9 #14 0x55e8cd4a0b82 in doris::segment_v2::SegmentIterator::next_batch(doris::Block*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/segment/segment_iterator.cpp:2518:19 #15 0x55e8cd37cca1 in doris::segment_v2::LazyInitSegmentIterator::next_batch(doris::Block*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/segment/lazy_init_segment_iterator.h:46:33 #16 0x55e8cccc0b05 in doris::Status doris::BetaRowsetReader::_next_batch<doris::Block>(doris::Block*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/rowset/beta_rowset_reader.h:113:35 #17 0x55e8ccc98668 in doris::BetaRowsetReader::next_batch(doris::Block*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/rowset/beta_rowset_reader.h:56:55 #18 0x55e8ca62ee8f in doris::VCollectIterator::Level0Iterator::_refresh() /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/iterator/vcollect_iterator.h #19 0x55e8ca611036 in doris::VCollectIterator::Level0Iterator::refresh_current_row() /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/iterator/vcollect_iterator.cpp:523:24 #20 0x55e8ca601a08 in doris::VCollectIterator::Level0Iterator::ensure_first_row_ref() /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/iterator/vcollect_iterator.cpp:496:14 #21 0x55e8ca604ea2 in doris::VCollectIterator::Level1Iterator::ensure_first_row_ref() /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/iterator/vcollect_iterator.cpp:713:27 #22 0x55e8ca608b20 in doris::VCollectIterator::build_heap(std::vector<std::shared_ptr<doris::RowsetReader>, std::allocator<std::shared_ptr<doris::RowsetReader>>>&) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/iterator/vcollect_iterator.cpp:189:9 #23 0x55e8ca5dc1ae in doris::BlockReader::_init_collect_iter(doris::TabletReader::ReaderParams const&) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/iterator/block_reader.cpp:152:9 #24 0x55e8ca5d7b9d in doris::BlockReader::init(doris::TabletReader::ReaderParams const&) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/storage/iterator/block_reader.cpp:226:19 #25 0x55e8d02df6d6 in doris::OlapScanner::_open_impl(doris::RuntimeState*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/olap_scanner.cpp:317:32 #26 0x55e8d02cd697 in doris::Scanner::open(doris::RuntimeState*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/scanner.h:80:16 #27 0x55e8d02b576a in doris::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/scanner_scheduler.cpp:185:5 #28 0x55e8d02bf603 in doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/scanner_scheduler.cpp:75:17 #29 0x55e8d02bf603 in doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()::operator()() const /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/scanner_scheduler.cpp:74:27 #30 0x55e8d02bf603 in bool std::__invoke_impl<bool, doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()&>(std::__invoke_other, doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:63:14 #31 0x55e8d02bf603 in std::enable_if<is_invocable_r_v<bool, doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()&>, bool>::type std::__invoke_r<bool, doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()&>(doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:116:9 #32 0x55e8d02bf603 in std::_Function_handler<bool (), doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()>::_M_invoke(std::_Any_data const&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:292:9 #33 0x55e8d02bed85 in std::function<bool ()>::operator()() const /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593:9 #34 0x55e8d02bed85 in doris::ScannerSplitRunner::process_for(std::chrono::duration<long, std::ratio<1l, 1000000000l>>) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/scanner_scheduler.cpp:425:25 apache#35 0x55e8d02aa844 in doris::PrioritizedSplitRunner::process() /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/task_executor/time_sharing/prioritized_split_runner.cpp:102:35 apache#36 0x55e8d0272674 in doris::TimeSharingTaskExecutor::_dispatch_thread() /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/task_executor/time_sharing/time_sharing_task_executor.cpp:556:77 apache#37 0x55e8d3fa8ad1 in std::function<void ()>::operator()() const /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593:9 apache#38 0x55e8d3fa8ad1 in doris::Thread::supervise_thread(void*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/util/thread.cpp:460:5 Thread T620 (ls_normal [work) created by T0 here: #0 0x55e8b71c5e91 in pthread_create (/mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be+0x2203de91) #1 0x55e8d3fa7b8a in doris::Thread::start_thread(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::function<void ()> const&, std::shared_ptr<doris::Thread>*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/util/thread.cpp:412:15 #2 0x55e8d02849fd in doris::Status doris::Thread::create<void (doris::TimeSharingTaskExecutor::*)(), doris::TimeSharingTaskExecutor*>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, void (doris::TimeSharingTaskExecutor::* const&)(), doris::TimeSharingTaskExecutor* const&, std::shared_ptr<doris::Thread>*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/util/thread.h:47:16 #3 0x55e8d026e6b3 in doris::TimeSharingTaskExecutor::_create_thread() /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/task_executor/time_sharing/time_sharing_task_executor.cpp:671:12 #4 0x55e8d026b628 in doris::TimeSharingTaskExecutor::_try_create_thread(int, std::lock_guard<std::mutex>&) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/task_executor/time_sharing/time_sharing_task_executor.cpp:360:25 #5 0x55e8d026967a in doris::TimeSharingTaskExecutor::init() /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/task_executor/time_sharing/time_sharing_task_executor.cpp:221:27 #6 0x55e8d025a85a in doris::TaskExecutorSimplifiedScanScheduler::start(int, int, int, int) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/exec/scan/scanner_scheduler.h:288:9 #7 0x55e8d368d649 in doris::WorkloadGroup::upsert_thread_pool_no_lock(doris::WorkloadGroupInfo*, std::shared_ptr<doris::CgroupCpuCtl>) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/runtime/workload_group/workload_group.cpp:571:38 #8 0x55e8d368f14d in doris::WorkloadGroup::upsert_task_scheduler(doris::WorkloadGroupInfo*) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/runtime/workload_group/workload_group.cpp:667:12 #9 0x55e8d36bd223 in doris::WorkloadGroupMgr::create_internal_wg() /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/runtime/workload_group/workload_group_manager.cpp:900:5 #10 0x55e8d32bbd3d in doris::ExecEnv::_create_internal_workload_group() /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/runtime/exec_env_init.cpp:475:5 #11 0x55e8d32b27fc in doris::ExecEnv::_init(std::vector<doris::StorePath, std::allocator<doris::StorePath>> const&, std::vector<doris::StorePath, std::allocator<doris::StorePath>> const&, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>> const&) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/runtime/exec_env_init.cpp:424:5 #12 0x55e8d32ad7ec in doris::ExecEnv::init(doris::ExecEnv*, std::vector<doris::StorePath, std::allocator<doris::StorePath>> const&, std::vector<doris::StorePath, std::allocator<doris::StorePath>> const&, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>> const&) /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/runtime/exec_env_init.cpp:199:17 #13 0x55e8b722cc8b in main /mnt/disk3/pipeline/repo/selectdb-core_master/selectdb-core/be/src/service/doris_main.cpp:531:14 #14 0x7f539b7cad8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 ### Check List (For Author) - Test  - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [ ] No. - [ ] Yes.  - Does this need documentation? - [ ] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label  --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* [improvement](fe) Add IVM create and alter validation for MTMV (#5) Issue Number: close #xxx Problem Summary: 1. Remove redundant pre-analysis in IVM analyzeQuery - single analyzeQueryInternal call 2. Unify IVM base table validation error messages (AGG_KEYS vs UNIQUE without MOW) 3. Add excluded_trigger_tables support in IvmNormalizeMtmv (transient row-id for excluded tables) 4. Block ALTER MTMV refresh method to/from INCREMENTAL 5. Validate base table models when ALTER MTMV excluded_trigger_tables 6. Extract MTMVPropertyUtil.parseTableNameInfos utility 7. Add comprehensive unit tests for all validation paths * [fix](fe) Fix IVM ExprId collision by reusing parser StatementContext ### What problem does this PR solve? Issue Number: close #xxx Problem Summary: Commit 71a3086 introduced a new StatementContext in analyzeQueryInternal() and restored the original (parser-created) StatementContext in the finally block. This caused ExprId collisions during IVM INCREMENTAL refresh because the parser StatementContext has a much smaller ExprId counter than the analysis StatementContext, and IvmRefreshManager.doRefreshInternal() reads exprIdStart from the ConnectContext's StatementContext after analysis. The fix reverts to the pre-71a3086f pattern: reuse ctx.getStatementContext() (the parser-created StatementContext) directly for analysis instead of creating a new one. This way all ExprId allocations accumulate in the same StatementContext that doRefreshInternal() later reads. ### Release note None ### Check List (For Author) - Test: Regression test (test_ivm_agg_2) - Behavior changed: No - Does this need documentation: No Co-authored-by: Copilot <[email protected]> * [fix](fe) Fix buildRowId to compute proper row-id for excluded trigger tables ### What problem does this PR solve? Issue Number: close #xxx Problem Summary: buildRowId() short-circuited to UuidNumeric() for excluded trigger tables, losing the deterministic row-id (hash of unique keys) for MOW and non-MOW UNIQUE_KEYS tables. The fix removes the early return and only uses the isExcludedTriggerTable flag to suppress AnalysisException for unsupported table types (AGG_KEYS etc.), while UNIQUE_KEYS tables always compute buildRowIdHash(keySlots) regardless of exclusion status. ### Release note None ### Check List (For Author) - Test: Unit Test (IvmNormalizeMtmvTest) - Behavior changed: No - Does this need documentation: No Co-authored-by: Copilot <[email protected]> * [test](fe) Update IvmNormalizeMtmvTest for excluded MOW table row-id fix ### What problem does this PR solve? Issue Number: close #xxx Problem Summary: Update testExcludedMowTableUsesTransientRowId to expect deterministic hash-based row-id (Cast expression) instead of UuidNumeric for excluded MOW tables, matching the fix in buildRowId(). ### Release note None ### Check List (For Author) - Test: Unit Test (IvmNormalizeMtmvTest - 27 tests pass) - Behavior changed: No - Does this need documentation: No Co-authored-by: Copilot <[email protected]> * [fix](fe) Refine ALTER MTMV refresh method compatibility rules ### What problem does this PR solve? Issue Number: close #xxx Problem Summary: The ALTER MTMV refresh method validation was too restrictive (blocking all changes to/from INCREMENTAL) or too permissive in some cases. The new rules: - COMPLETE <-> INCREMENTAL: forbidden (must recreate MV) - COMPLETE/INCREMENTAL -> AUTO: allowed - AUTO -> COMPLETE/INCREMENTAL: forbidden (must recreate MV) - Same method (no-op): allowed ### Release note None ### Check List (For Author) - Test: Regression test - Behavior changed: Yes (COMPLETE/INCREMENTAL to AUTO now allowed) - Does this need documentation: No Co-authored-by: Copilot <[email protected]> * opt code * [test](fe) Isolate CreateMTMVCommandTest statement context ### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: CreateMTMVCommandTest reused the same StatementContext across multiple statements, so table and excluded_trigger_tables state leaked between test cases and caused false incremental MV validation failures. This change resets the statement context for each statement in the test and adds the missing LinkedHashSet import required by StatementContext. ### Release note None ### Check List (For Author) - Test: FE unit test - ./run-fe-ut.sh --run org.apache.doris.mtmv.ivm.IvmAggDeltaStrategyTest,org.apache.doris.mtmv.ivm.IvmDeltaExecutorTest,org.apache.doris.mtmv.ivm.IvmDeltaRewriterTest,org.apache.doris.mtmv.ivm.IvmRefreshManagerTest,org.apache.doris.mtmv.ivm.IvmSimpleScanDeltaStrategyTest,org.apache.doris.mtmv.ivm.IvmUtilTest,org.apache.doris.nereids.rules.rewrite.IvmNormalizeMtmvTest,org.apache.doris.nereids.trees.plans.CreateMTMVCommandTest,org.apache.doris.catalog.ShowCreateMTMVTest - Behavior changed: No - Does this need documentation: No * [fix](fe) Fix TableNameInfo imports after rebasing IVM branch ### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: Rebase onto the latest yujun777/ivm left several FE IVM files importing org.apache.doris.info.TableNameInfo while the current base branch provides the type in org.apache.doris.catalog.info.TableNameInfo. This fixes the stale imports so the rebased branch compiles and the targeted IVM FE tests run successfully again. ### Release note None ### Check List (For Author) - Test: Unit Test - org.apache.doris.mtmv.ivm.IvmAggDeltaStrategyTest - org.apache.doris.mtmv.ivm.IvmDeltaExecutorTest - org.apache.doris.mtmv.ivm.IvmDeltaRewriterTest - org.apache.doris.mtmv.ivm.IvmRefreshManagerTest - org.apache.doris.mtmv.ivm.IvmSimpleScanDeltaStrategyTest - org.apache.doris.mtmv.ivm.IvmUtilTest - org.apache.doris.nereids.rules.rewrite.IvmNormalizeMtmvTest - org.apache.doris.nereids.trees.plans.CreateMTMVCommandTest - org.apache.doris.catalog.ShowCreateMTMVTest - Behavior changed: No - Does this need documentation: No * [fix](fe) Use deterministic row id for excluded AGG_KEYS tables ### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: IVM normalization treated excluded AGG_KEYS base tables as transient-row-id scans and generated uuid_numeric() row ids. This changed excluded AGG_KEYS row identity across refreshes and did not follow the base table aggregate keys. Fix buildRowId to hash base-schema key columns for excluded AGG_KEYS tables, and add focused tests to verify the row id is deterministic and excludes non-key value columns. ### Release note None ### Check List (For Author) - Test: Unit Test - org.apache.doris.mtmv.ivm.IvmAggDeltaStrategyTest - org.apache.doris.mtmv.ivm.IvmDeltaExecutorTest - org.apache.doris.mtmv.ivm.IvmDeltaRewriterTest - org.apache.doris.mtmv.ivm.IvmRefreshManagerTest - org.apache.doris.mtmv.ivm.IvmSimpleScanDeltaStrategyTest - org.apache.doris.mtmv.ivm.IvmUtilTest - org.apache.doris.nereids.rules.rewrite.IvmNormalizeMtmvTest - org.apache.doris.nereids.trees.plans.CreateMTMVCommandTest - org.apache.doris.catalog.ShowCreateMTMVTest - Behavior changed: Yes (excluded AGG_KEYS row-id generation is now deterministic on agg keys) - Does this need documentation: No * fix comment * fix comment * opt code --------- Co-authored-by: Copilot <[email protected]>

* [improvement](fe) Add IVM create and alter validation for MTMV (#5) Issue Number: close #xxx Problem Summary: 1. Remove redundant pre-analysis in IVM analyzeQuery - single analyzeQueryInternal call 2. Unify IVM base table validation error messages (AGG_KEYS vs UNIQUE without MOW) 3. Add excluded_trigger_tables support in IvmNormalizeMtmv (transient row-id for excluded tables) 4. Block ALTER MTMV refresh method to/from INCREMENTAL 5. Validate base table models when ALTER MTMV excluded_trigger_tables 6. Extract MTMVPropertyUtil.parseTableNameInfos utility 7. Add comprehensive unit tests for all validation paths * [fix](fe) Fix IVM ExprId collision by reusing parser StatementContext Issue Number: close #xxx Problem Summary: Commit 71a3086 introduced a new StatementContext in analyzeQueryInternal() and restored the original (parser-created) StatementContext in the finally block. This caused ExprId collisions during IVM INCREMENTAL refresh because the parser StatementContext has a much smaller ExprId counter than the analysis StatementContext, and IvmRefreshManager.doRefreshInternal() reads exprIdStart from the ConnectContext's StatementContext after analysis. The fix reverts to the pre-71a3086f pattern: reuse ctx.getStatementContext() (the parser-created StatementContext) directly for analysis instead of creating a new one. This way all ExprId allocations accumulate in the same StatementContext that doRefreshInternal() later reads. None - Test: Regression test (test_ivm_agg_2) - Behavior changed: No - Does this need documentation: No Co-authored-by: Copilot <[email protected]> * [fix](fe) Fix buildRowId to compute proper row-id for excluded trigger tables Issue Number: close #xxx Problem Summary: buildRowId() short-circuited to UuidNumeric() for excluded trigger tables, losing the deterministic row-id (hash of unique keys) for MOW and non-MOW UNIQUE_KEYS tables. The fix removes the early return and only uses the isExcludedTriggerTable flag to suppress AnalysisException for unsupported table types (AGG_KEYS etc.), while UNIQUE_KEYS tables always compute buildRowIdHash(keySlots) regardless of exclusion status. None - Test: Unit Test (IvmNormalizeMtmvTest) - Behavior changed: No - Does this need documentation: No Co-authored-by: Copilot <[email protected]> * [test](fe) Update IvmNormalizeMtmvTest for excluded MOW table row-id fix Issue Number: close #xxx Problem Summary: Update testExcludedMowTableUsesTransientRowId to expect deterministic hash-based row-id (Cast expression) instead of UuidNumeric for excluded MOW tables, matching the fix in buildRowId(). None - Test: Unit Test (IvmNormalizeMtmvTest - 27 tests pass) - Behavior changed: No - Does this need documentation: No Co-authored-by: Copilot <[email protected]> * [fix](fe) Refine ALTER MTMV refresh method compatibility rules Issue Number: close #xxx Problem Summary: The ALTER MTMV refresh method validation was too restrictive (blocking all changes to/from INCREMENTAL) or too permissive in some cases. The new rules: - COMPLETE <-> INCREMENTAL: forbidden (must recreate MV) - COMPLETE/INCREMENTAL -> AUTO: allowed - AUTO -> COMPLETE/INCREMENTAL: forbidden (must recreate MV) - Same method (no-op): allowed None - Test: Regression test - Behavior changed: Yes (COMPLETE/INCREMENTAL to AUTO now allowed) - Does this need documentation: No Co-authored-by: Copilot <[email protected]> * opt code * [test](fe) Isolate CreateMTMVCommandTest statement context Issue Number: None Related PR: None Problem Summary: CreateMTMVCommandTest reused the same StatementContext across multiple statements, so table and excluded_trigger_tables state leaked between test cases and caused false incremental MV validation failures. This change resets the statement context for each statement in the test and adds the missing LinkedHashSet import required by StatementContext. None - Test: FE unit test - ./run-fe-ut.sh --run org.apache.doris.mtmv.ivm.IvmAggDeltaStrategyTest,org.apache.doris.mtmv.ivm.IvmDeltaExecutorTest,org.apache.doris.mtmv.ivm.IvmDeltaRewriterTest,org.apache.doris.mtmv.ivm.IvmRefreshManagerTest,org.apache.doris.mtmv.ivm.IvmSimpleScanDeltaStrategyTest,org.apache.doris.mtmv.ivm.IvmUtilTest,org.apache.doris.nereids.rules.rewrite.IvmNormalizeMtmvTest,org.apache.doris.nereids.trees.plans.CreateMTMVCommandTest,org.apache.doris.catalog.ShowCreateMTMVTest - Behavior changed: No - Does this need documentation: No * [fix](fe) Fix TableNameInfo imports after rebasing IVM branch Issue Number: None Related PR: None Problem Summary: Rebase onto the latest yujun777/ivm left several FE IVM files importing org.apache.doris.info.TableNameInfo while the current base branch provides the type in org.apache.doris.catalog.info.TableNameInfo. This fixes the stale imports so the rebased branch compiles and the targeted IVM FE tests run successfully again. None - Test: Unit Test - org.apache.doris.mtmv.ivm.IvmAggDeltaStrategyTest - org.apache.doris.mtmv.ivm.IvmDeltaExecutorTest - org.apache.doris.mtmv.ivm.IvmDeltaRewriterTest - org.apache.doris.mtmv.ivm.IvmRefreshManagerTest - org.apache.doris.mtmv.ivm.IvmSimpleScanDeltaStrategyTest - org.apache.doris.mtmv.ivm.IvmUtilTest - org.apache.doris.nereids.rules.rewrite.IvmNormalizeMtmvTest - org.apache.doris.nereids.trees.plans.CreateMTMVCommandTest - org.apache.doris.catalog.ShowCreateMTMVTest - Behavior changed: No - Does this need documentation: No * [fix](fe) Use deterministic row id for excluded AGG_KEYS tables Issue Number: None Related PR: None Problem Summary: IVM normalization treated excluded AGG_KEYS base tables as transient-row-id scans and generated uuid_numeric() row ids. This changed excluded AGG_KEYS row identity across refreshes and did not follow the base table aggregate keys. Fix buildRowId to hash base-schema key columns for excluded AGG_KEYS tables, and add focused tests to verify the row id is deterministic and excludes non-key value columns. None - Test: Unit Test - org.apache.doris.mtmv.ivm.IvmAggDeltaStrategyTest - org.apache.doris.mtmv.ivm.IvmDeltaExecutorTest - org.apache.doris.mtmv.ivm.IvmDeltaRewriterTest - org.apache.doris.mtmv.ivm.IvmRefreshManagerTest - org.apache.doris.mtmv.ivm.IvmSimpleScanDeltaStrategyTest - org.apache.doris.mtmv.ivm.IvmUtilTest - org.apache.doris.nereids.rules.rewrite.IvmNormalizeMtmvTest - org.apache.doris.nereids.trees.plans.CreateMTMVCommandTest - org.apache.doris.catalog.ShowCreateMTMVTest - Behavior changed: Yes (excluded AGG_KEYS row-id generation is now deterministic on agg keys) - Does this need documentation: No * fix comment * fix comment * opt code --------- Co-authored-by: Copilot <[email protected]>

…he#63379) ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: PartitionTopN still passed partition key expressions to BE sort info when the same original slot also appeared in window order keys. These keys are constant inside each partition, so keeping them causes redundant comparison work and noisy explain output, especially after slot remapping changes ExprIds. ``` before | 1:VPartitionTopN(42) | | | functions: rank | | | partition by: c_region[#3] | | | order by: c_region[#5] ASC, c_custkey[#6] ASC | | | has global limit: false | | | partition limit: 1 | | | partition topn phase: TWO_PHASE_LOCAL_PTOPN | | | distribute expr lists: c_custkey[#4] ``` after ``` | 1:VPartitionTopN(42) | | | functions: rank | | | partition by: c_region[#3] | | | order by: c_custkey[#6] ASC | | | has global limit: false | | | partition limit: 1 | | | partition topn phase: TWO_PHASE_LOCAL_PTOPN | | | distribute expr lists: c_custkey[#4] | | | ``` This PR also adds a guard for self-join cases where the partition key and order key come from the same original table column, but from different join inputs. Example: ```sql EXPLAIN SELECT * FROM ( SELECT l.c2 AS lc2, r.c2 AS rc2, row_number() OVER ( PARTITION BY l.c2 ORDER BY r.c2 ) AS rn FROM test_global_partition_topn_plan l JOIN test_global_partition_topn_plan r ON l.c1 = r.c1 ) tmp WHERE rn <= 1; ``` Here l.c2 and r.c2 both come from test_global_partition_topn_plan.c2, but they are different slots after the self join. PARTITION BY l.c2 only guarantees that the left-side c2 is constant inside each partition. It does not guarantee that the right-side r.c2 is constant, so ORDER BY r.c2 must not be pruned. The expected EXPLAIN should still keep the order key in VPartitionTopN: ``` 1:VPartitionTopN | functions: row_number | partition by: c2[#0] | order by: c2[#3] ASC | has global limit: false | partition limit: 1 | partition topn phase: TWO_PHASE_LOCAL_PTOPN The different slot ids, c2[#0] and c2[#3], show that the partition key and order key are different join outputs. If the order key were pruned here, the row selected by rn <= 1 could become nondeterministic or incorrect. ```

* [improvement](fe) Add IVM create and alter validation for MTMV (#5) Issue Number: close #xxx Problem Summary: 1. Remove redundant pre-analysis in IVM analyzeQuery - single analyzeQueryInternal call 2. Unify IVM base table validation error messages (AGG_KEYS vs UNIQUE without MOW) 3. Add excluded_trigger_tables support in IvmNormalizeMtmv (transient row-id for excluded tables) 4. Block ALTER MTMV refresh method to/from INCREMENTAL 5. Validate base table models when ALTER MTMV excluded_trigger_tables 6. Extract MTMVPropertyUtil.parseTableNameInfos utility 7. Add comprehensive unit tests for all validation paths * [fix](fe) Fix IVM ExprId collision by reusing parser StatementContext Issue Number: close #xxx Problem Summary: Commit 71a3086 introduced a new StatementContext in analyzeQueryInternal() and restored the original (parser-created) StatementContext in the finally block. This caused ExprId collisions during IVM INCREMENTAL refresh because the parser StatementContext has a much smaller ExprId counter than the analysis StatementContext, and IvmRefreshManager.doRefreshInternal() reads exprIdStart from the ConnectContext's StatementContext after analysis. The fix reverts to the pre-71a3086f pattern: reuse ctx.getStatementContext() (the parser-created StatementContext) directly for analysis instead of creating a new one. This way all ExprId allocations accumulate in the same StatementContext that doRefreshInternal() later reads. None - Test: Regression test (test_ivm_agg_2) - Behavior changed: No - Does this need documentation: No Co-authored-by: Copilot <[email protected]> * [fix](fe) Fix buildRowId to compute proper row-id for excluded trigger tables Issue Number: close #xxx Problem Summary: buildRowId() short-circuited to UuidNumeric() for excluded trigger tables, losing the deterministic row-id (hash of unique keys) for MOW and non-MOW UNIQUE_KEYS tables. The fix removes the early return and only uses the isExcludedTriggerTable flag to suppress AnalysisException for unsupported table types (AGG_KEYS etc.), while UNIQUE_KEYS tables always compute buildRowIdHash(keySlots) regardless of exclusion status. None - Test: Unit Test (IvmNormalizeMtmvTest) - Behavior changed: No - Does this need documentation: No Co-authored-by: Copilot <[email protected]> * [test](fe) Update IvmNormalizeMtmvTest for excluded MOW table row-id fix Issue Number: close #xxx Problem Summary: Update testExcludedMowTableUsesTransientRowId to expect deterministic hash-based row-id (Cast expression) instead of UuidNumeric for excluded MOW tables, matching the fix in buildRowId(). None - Test: Unit Test (IvmNormalizeMtmvTest - 27 tests pass) - Behavior changed: No - Does this need documentation: No Co-authored-by: Copilot <[email protected]> * [fix](fe) Refine ALTER MTMV refresh method compatibility rules Issue Number: close #xxx Problem Summary: The ALTER MTMV refresh method validation was too restrictive (blocking all changes to/from INCREMENTAL) or too permissive in some cases. The new rules: - COMPLETE <-> INCREMENTAL: forbidden (must recreate MV) - COMPLETE/INCREMENTAL -> AUTO: allowed - AUTO -> COMPLETE/INCREMENTAL: forbidden (must recreate MV) - Same method (no-op): allowed None - Test: Regression test - Behavior changed: Yes (COMPLETE/INCREMENTAL to AUTO now allowed) - Does this need documentation: No Co-authored-by: Copilot <[email protected]> * opt code * [test](fe) Isolate CreateMTMVCommandTest statement context Issue Number: None Related PR: None Problem Summary: CreateMTMVCommandTest reused the same StatementContext across multiple statements, so table and excluded_trigger_tables state leaked between test cases and caused false incremental MV validation failures. This change resets the statement context for each statement in the test and adds the missing LinkedHashSet import required by StatementContext. None - Test: FE unit test - ./run-fe-ut.sh --run org.apache.doris.mtmv.ivm.IvmAggDeltaStrategyTest,org.apache.doris.mtmv.ivm.IvmDeltaExecutorTest,org.apache.doris.mtmv.ivm.IvmDeltaRewriterTest,org.apache.doris.mtmv.ivm.IvmRefreshManagerTest,org.apache.doris.mtmv.ivm.IvmSimpleScanDeltaStrategyTest,org.apache.doris.mtmv.ivm.IvmUtilTest,org.apache.doris.nereids.rules.rewrite.IvmNormalizeMtmvTest,org.apache.doris.nereids.trees.plans.CreateMTMVCommandTest,org.apache.doris.catalog.ShowCreateMTMVTest - Behavior changed: No - Does this need documentation: No * [fix](fe) Fix TableNameInfo imports after rebasing IVM branch Issue Number: None Related PR: None Problem Summary: Rebase onto the latest yujun777/ivm left several FE IVM files importing org.apache.doris.info.TableNameInfo while the current base branch provides the type in org.apache.doris.catalog.info.TableNameInfo. This fixes the stale imports so the rebased branch compiles and the targeted IVM FE tests run successfully again. None - Test: Unit Test - org.apache.doris.mtmv.ivm.IvmAggDeltaStrategyTest - org.apache.doris.mtmv.ivm.IvmDeltaExecutorTest - org.apache.doris.mtmv.ivm.IvmDeltaRewriterTest - org.apache.doris.mtmv.ivm.IvmRefreshManagerTest - org.apache.doris.mtmv.ivm.IvmSimpleScanDeltaStrategyTest - org.apache.doris.mtmv.ivm.IvmUtilTest - org.apache.doris.nereids.rules.rewrite.IvmNormalizeMtmvTest - org.apache.doris.nereids.trees.plans.CreateMTMVCommandTest - org.apache.doris.catalog.ShowCreateMTMVTest - Behavior changed: No - Does this need documentation: No * [fix](fe) Use deterministic row id for excluded AGG_KEYS tables Issue Number: None Related PR: None Problem Summary: IVM normalization treated excluded AGG_KEYS base tables as transient-row-id scans and generated uuid_numeric() row ids. This changed excluded AGG_KEYS row identity across refreshes and did not follow the base table aggregate keys. Fix buildRowId to hash base-schema key columns for excluded AGG_KEYS tables, and add focused tests to verify the row id is deterministic and excludes non-key value columns. None - Test: Unit Test - org.apache.doris.mtmv.ivm.IvmAggDeltaStrategyTest - org.apache.doris.mtmv.ivm.IvmDeltaExecutorTest - org.apache.doris.mtmv.ivm.IvmDeltaRewriterTest - org.apache.doris.mtmv.ivm.IvmRefreshManagerTest - org.apache.doris.mtmv.ivm.IvmSimpleScanDeltaStrategyTest - org.apache.doris.mtmv.ivm.IvmUtilTest - org.apache.doris.nereids.rules.rewrite.IvmNormalizeMtmvTest - org.apache.doris.nereids.trees.plans.CreateMTMVCommandTest - org.apache.doris.catalog.ShowCreateMTMVTest - Behavior changed: Yes (excluded AGG_KEYS row-id generation is now deterministic on agg keys) - Does this need documentation: No * fix comment * fix comment * opt code --------- Co-authored-by: Copilot <[email protected]>

Issue Number: None Related PR: None Problem Summary: UBSAN/ASAN reported a heap-use-after-free during BE shutdown in TimeSharingTaskExecutor destruction. The scan executor owned pending split runners, while the split runner function held ScannerContext, and ScannerContext cleaned up its task by copying the executor shared_ptr back from the scheduler. When the scheduler was already destroying that shared_ptr, copying the same shared_ptr member could re-enter the executor destruction path and release members such as MetricEntity twice. Keep ScannerContext cleanup tied to a weak reference of the executor captured when the task is created, and avoid capturing the scheduler this pointer in split cleanup lambdas. 旧代码的链路大致是： 1. TaskExecutorSimplifiedScanScheduler 析构，成员 _task_executor 是最后阶段要销毁的 std::shared_ptr<TaskExecutor>。 2. 这个 shared_ptr 引用计数降到 0，于是开始执行 TimeSharingTaskExecutor::~TimeSharingTaskExecutor()。 3. TimeSharingTaskExecutor 析构时会释放它持有的 split/task 容器。 4. split 里有 ScannerSplitRunner，它的 std::function 捕获了 std::shared_ptr<ScannerContext>。 5. 释放 split 会释放这个 captured ScannerContext，于是进入 ScannerContext::~ScannerContext()。 6. 旧代码在 ScannerContext::~ScannerContext() 里又执行： task_executor_scheduler->task_executor()->remove_task(_task_handle) 而 task_executor() 返回的是： std::shared_ptr<TaskExecutor> task_executor() const { return _task_executor; } 也就是从 scheduler 的 _task_executor 成员再复制一个 shared_ptr。问题在于：第 1 步里，scheduler 的 _task_executor 这个 shared_ptr 正在析构，并且它已经触发了 TimeSharingTaskExecutor 的析构。此时再从同一个 member shared_ptr 复制引用，等价于在其析构过程中重新操作它的 control block。这个 control block 可能已经处于 release-last-use 路径，随后临时复制出来的 shared_ptr 析构又会再次 release 同一个 control block，导致 TimeSharingTaskExecutor 析构链被重入或二次释放其成员。 ``` /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:1069:11: runtime error: member call on address 0x7c63e14891c0 which does not point to an object of type 'std::_Sp_counted_base<>' 0x7c63e14891c0: note: object has invalid vptr 00 00 00 00 e0 74 1d 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 be be be be d0 26 2b e1 ^~~~~~~~~~~~~~~~~~~~~~~ invalid vptr #0 0x55e77069caee in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:1069:11 #1 0x55e77069caee in std::__shared_ptr<doris::MetricEntity, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:1531:31 #2 0x55e77069caee in doris::TimeSharingTaskExecutor::~TimeSharingTaskExecutor() be/build_ASAN/./be/build_ASAN/../src/exec/scan/task_executor/time_sharing/time_sharing_task_executor.cpp:306:1 #3 0x55e75b584397 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use() /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:174:2 #4 0x55e7706486ac in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:1069:11 #5 0x55e7706486ac in std::__shared_ptr<doris::TaskExecutor, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:1531:31 #6 0x55e7706486ac in doris::ScannerContext::~ScannerContext() be/build_ASAN/./be/build_ASAN/../src/exec/scan/scanner_context.cpp:277:13 #7 0x55e75b584397 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use() /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:174:2 #8 0x55e7706f288d in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:1069:11 #9 0x55e7706f288d in std::__shared_ptr<doris::ScannerContext, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:1531:31 #10 0x55e7706f288d in doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()::~() be/build_ASAN/./be/build_ASAN/../src/exec/scan/scanner_scheduler.cpp:77:26 #11 0x55e7706f39bb in std::_Function_base::_Base_manager<doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()>::_M_destroy(std::_Any_data&, std::integral_constant<bool, false>) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:177:4 #12 0x55e7706f39bb in std::_Function_base::_Base_manager<doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()>::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:205:8 #13 0x55e7706f39bb in std::_Function_handler<bool (), doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::'lambda'()>::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:284:6 #14 0x55e770702246 in std::_Function_base::~_Function_base() /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:246:2 #15 0x55e770702246 in doris::ScannerSplitRunner::~ScannerSplitRunner() be/build_ASAN/../src/exec/scan/scanner_scheduler.h:62:7 #16 0x55e75b584397 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use() /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:174:2 #17 0x55e7706e11e9 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:1069:11 #18 0x55e7706e11e9 in std::__shared_ptr<doris::SplitRunner, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:1531:31 #19 0x55e7706e11e9 in doris::PrioritizedSplitRunner::~PrioritizedSplitRunner() be/build_ASAN/../src/exec/scan/task_executor/time_sharing/prioritized_split_runner.h:54:47 #20 0x55e75b584397 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use() /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:174:2 #21 0x55e7706b8ba4 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:1069:11 #22 0x55e7706b8ba4 in std::__shared_ptr<doris::PrioritizedSplitRunner, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:1531:31 #23 0x55e7706b8ba4 in void std::destroy_at<std::shared_ptr<doris::PrioritizedSplitRunner> >(std::shared_ptr<doris::PrioritizedSplitRunner>*) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/stl_construct.h:88:15 #24 0x55e7706b8ba4 in void std::allocator_traits<std::allocator<std::__detail::_Hash_node<std::shared_ptr<doris::PrioritizedSplitRunner>, false> > >::destroy<std::shared_ptr<doris::PrioritizedSplitRunner> >(std::allocator<std::__detail::_Hash_node<std::shared_ptr<doris::PrioritizedSplitRunner>, false> >&, std::shared_ptr<doris::PrioritizedSplitRunner>*) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/alloc_traits.h:698:4 #25 0x55e7706b8ba4 in std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::shared_ptr<doris::PrioritizedSplitRunner>, false> > >::_M_deallocate_node(std::__detail::_Hash_node<std::shared_ptr<doris::PrioritizedSplitRunner>, false>*) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/hashtable_policy.h:1572:7 #26 0x55e7706b89ff in std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::shared_ptr<doris::PrioritizedSplitRunner>, false> > >::_M_deallocate_nodes(std::__detail::_Hash_node<std::shared_ptr<doris::PrioritizedSplitRunner>, false>*) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/hashtable_policy.h:1594:4 #27 0x55e7706b89ff in std::_Hashtable<std::shared_ptr<doris::PrioritizedSplitRunner>, std::shared_ptr<doris::PrioritizedSplitRunner>, std::allocator<std::shared_ptr<doris::PrioritizedSplitRunner> >, std::__detail::_Identity, std::equal_to<std::shared_ptr<doris::PrioritizedSplitRunner> >, std::hash<std::shared_ptr<doris::PrioritizedSplitRunner> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, true, true> >::~_Hashtable() /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/hashtable.h:1852:13 #28 0x55e77069c6ca in std::unordered_set<std::shared_ptr<doris::PrioritizedSplitRunner>, std::hash<std::shared_ptr<doris::PrioritizedSplitRunner> >, std::equal_to<std::shared_ptr<doris::PrioritizedSplitRunner> >, std::allocator<std::shared_ptr<doris::PrioritizedSplitRunner> > >::~unordered_set() /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/unordered_set.h:107:11 #29 0x55e77069c6ca in doris::TimeSharingTaskExecutor::~TimeSharingTaskExecutor() be/build_ASAN/./be/build_ASAN/../src/exec/scan/task_executor/time_sharing/time_sharing_task_executor.cpp:306:1 #30 0x55e75b583fa1 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:345:8 #31 0x55e770688ecf in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:1069:11 #32 0x55e770688ecf in std::__shared_ptr<doris::TaskExecutor, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:1531:31 #33 0x55e770688ecf in doris::TaskExecutorSimplifiedScanScheduler::~TaskExecutorSimplifiedScanScheduler() be/build_ASAN/../src/exec/scan/scanner_scheduler.h:266:5 #34 0x55e7706891ed in doris::TaskExecutorSimplifiedScanScheduler::~TaskExecutorSimplifiedScanScheduler() be/build_ASAN/../src/exec/scan/scanner_scheduler.h:261:53 apache#35 0x55e7744d5afa in std::default_delete<doris::ScannerScheduler>::operator()(doris::ScannerScheduler*) const /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/unique_ptr.h:93:2 apache#36 0x55e7744d5afa in std::__uniq_ptr_impl<doris::ScannerScheduler, std::default_delete<doris::ScannerScheduler> >::reset(doris::ScannerScheduler*) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/unique_ptr.h:205:4 apache#37 0x55e7744d5afa in std::unique_ptr<doris::ScannerScheduler, std::default_delete<doris::ScannerScheduler> >::reset(doris::ScannerScheduler*) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/unique_ptr.h:512:7 apache#38 0x55e7744d5afa in doris::WorkloadGroup::destroy_schedulers() be/build_ASAN/./be/build_ASAN/../src/runtime/workload_group/workload_group.cpp:776:22 apache#39 0x55e774501af0 in doris::WorkloadGroupMgr::destroy_schedulers() be/build_ASAN/./be/build_ASAN/../src/runtime/workload_group/workload_group_manager.cpp:977:23 apache#40 0x55e7740e0ddc in doris::ExecEnv::destroy() be/build_ASAN/./be/build_ASAN/../src/runtime/exec_env_init.cpp:892:34 apache#41 0x55e75b5601f2 in main be/build_ASAN/./be/build_ASAN/../src/service/doris_main.cpp:734:15 apache#42 0x7f53e1e6b082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/../csu/libc-start.c:308:16 apache#43 0x55e75b470029 in _start (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x247b8029) SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:1069:11 ``` ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test  - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [ ] No. - [ ] Yes.  - Does this need documentation? - [ ] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

* [improvement](fe) Add IVM create and alter validation for MTMV (#5) Issue Number: close #xxx Problem Summary: 1. Remove redundant pre-analysis in IVM analyzeQuery - single analyzeQueryInternal call 2. Unify IVM base table validation error messages (AGG_KEYS vs UNIQUE without MOW) 3. Add excluded_trigger_tables support in IvmNormalizeMtmv (transient row-id for excluded tables) 4. Block ALTER MTMV refresh method to/from INCREMENTAL 5. Validate base table models when ALTER MTMV excluded_trigger_tables 6. Extract MTMVPropertyUtil.parseTableNameInfos utility 7. Add comprehensive unit tests for all validation paths * [fix](fe) Fix IVM ExprId collision by reusing parser StatementContext Issue Number: close #xxx Problem Summary: Commit 71a3086 introduced a new StatementContext in analyzeQueryInternal() and restored the original (parser-created) StatementContext in the finally block. This caused ExprId collisions during IVM INCREMENTAL refresh because the parser StatementContext has a much smaller ExprId counter than the analysis StatementContext, and IvmRefreshManager.doRefreshInternal() reads exprIdStart from the ConnectContext's StatementContext after analysis. The fix reverts to the pre-71a3086f pattern: reuse ctx.getStatementContext() (the parser-created StatementContext) directly for analysis instead of creating a new one. This way all ExprId allocations accumulate in the same StatementContext that doRefreshInternal() later reads. None - Test: Regression test (test_ivm_agg_2) - Behavior changed: No - Does this need documentation: No Co-authored-by: Copilot <[email protected]> * [fix](fe) Fix buildRowId to compute proper row-id for excluded trigger tables Issue Number: close #xxx Problem Summary: buildRowId() short-circuited to UuidNumeric() for excluded trigger tables, losing the deterministic row-id (hash of unique keys) for MOW and non-MOW UNIQUE_KEYS tables. The fix removes the early return and only uses the isExcludedTriggerTable flag to suppress AnalysisException for unsupported table types (AGG_KEYS etc.), while UNIQUE_KEYS tables always compute buildRowIdHash(keySlots) regardless of exclusion status. None - Test: Unit Test (IvmNormalizeMtmvTest) - Behavior changed: No - Does this need documentation: No Co-authored-by: Copilot <[email protected]> * [test](fe) Update IvmNormalizeMtmvTest for excluded MOW table row-id fix Issue Number: close #xxx Problem Summary: Update testExcludedMowTableUsesTransientRowId to expect deterministic hash-based row-id (Cast expression) instead of UuidNumeric for excluded MOW tables, matching the fix in buildRowId(). None - Test: Unit Test (IvmNormalizeMtmvTest - 27 tests pass) - Behavior changed: No - Does this need documentation: No Co-authored-by: Copilot <[email protected]> * [fix](fe) Refine ALTER MTMV refresh method compatibility rules Issue Number: close #xxx Problem Summary: The ALTER MTMV refresh method validation was too restrictive (blocking all changes to/from INCREMENTAL) or too permissive in some cases. The new rules: - COMPLETE <-> INCREMENTAL: forbidden (must recreate MV) - COMPLETE/INCREMENTAL -> AUTO: allowed - AUTO -> COMPLETE/INCREMENTAL: forbidden (must recreate MV) - Same method (no-op): allowed None - Test: Regression test - Behavior changed: Yes (COMPLETE/INCREMENTAL to AUTO now allowed) - Does this need documentation: No Co-authored-by: Copilot <[email protected]> * opt code * [test](fe) Isolate CreateMTMVCommandTest statement context Issue Number: None Related PR: None Problem Summary: CreateMTMVCommandTest reused the same StatementContext across multiple statements, so table and excluded_trigger_tables state leaked between test cases and caused false incremental MV validation failures. This change resets the statement context for each statement in the test and adds the missing LinkedHashSet import required by StatementContext. None - Test: FE unit test - ./run-fe-ut.sh --run org.apache.doris.mtmv.ivm.IvmAggDeltaStrategyTest,org.apache.doris.mtmv.ivm.IvmDeltaExecutorTest,org.apache.doris.mtmv.ivm.IvmDeltaRewriterTest,org.apache.doris.mtmv.ivm.IvmRefreshManagerTest,org.apache.doris.mtmv.ivm.IvmSimpleScanDeltaStrategyTest,org.apache.doris.mtmv.ivm.IvmUtilTest,org.apache.doris.nereids.rules.rewrite.IvmNormalizeMtmvTest,org.apache.doris.nereids.trees.plans.CreateMTMVCommandTest,org.apache.doris.catalog.ShowCreateMTMVTest - Behavior changed: No - Does this need documentation: No * [fix](fe) Fix TableNameInfo imports after rebasing IVM branch Issue Number: None Related PR: None Problem Summary: Rebase onto the latest yujun777/ivm left several FE IVM files importing org.apache.doris.info.TableNameInfo while the current base branch provides the type in org.apache.doris.catalog.info.TableNameInfo. This fixes the stale imports so the rebased branch compiles and the targeted IVM FE tests run successfully again. None - Test: Unit Test - org.apache.doris.mtmv.ivm.IvmAggDeltaStrategyTest - org.apache.doris.mtmv.ivm.IvmDeltaExecutorTest - org.apache.doris.mtmv.ivm.IvmDeltaRewriterTest - org.apache.doris.mtmv.ivm.IvmRefreshManagerTest - org.apache.doris.mtmv.ivm.IvmSimpleScanDeltaStrategyTest - org.apache.doris.mtmv.ivm.IvmUtilTest - org.apache.doris.nereids.rules.rewrite.IvmNormalizeMtmvTest - org.apache.doris.nereids.trees.plans.CreateMTMVCommandTest - org.apache.doris.catalog.ShowCreateMTMVTest - Behavior changed: No - Does this need documentation: No * [fix](fe) Use deterministic row id for excluded AGG_KEYS tables Issue Number: None Related PR: None Problem Summary: IVM normalization treated excluded AGG_KEYS base tables as transient-row-id scans and generated uuid_numeric() row ids. This changed excluded AGG_KEYS row identity across refreshes and did not follow the base table aggregate keys. Fix buildRowId to hash base-schema key columns for excluded AGG_KEYS tables, and add focused tests to verify the row id is deterministic and excludes non-key value columns. None - Test: Unit Test - org.apache.doris.mtmv.ivm.IvmAggDeltaStrategyTest - org.apache.doris.mtmv.ivm.IvmDeltaExecutorTest - org.apache.doris.mtmv.ivm.IvmDeltaRewriterTest - org.apache.doris.mtmv.ivm.IvmRefreshManagerTest - org.apache.doris.mtmv.ivm.IvmSimpleScanDeltaStrategyTest - org.apache.doris.mtmv.ivm.IvmUtilTest - org.apache.doris.nereids.rules.rewrite.IvmNormalizeMtmvTest - org.apache.doris.nereids.trees.plans.CreateMTMVCommandTest - org.apache.doris.catalog.ShowCreateMTMVTest - Behavior changed: Yes (excluded AGG_KEYS row-id generation is now deterministic on agg keys) - Does this need documentation: No * fix comment * fix comment * opt code --------- Co-authored-by: Copilot <[email protected]>

yujun777 force-pushed the ivm branch from 9000e16 to 35597b3 Compare April 16, 2026 00:16

seawinde and others added 8 commits April 16, 2026 10:21

opt code

ef1d687

seawinde force-pushed the ivm_pick_create_alter_validation branch from 9b495f8 to af8eadb Compare April 16, 2026 02:31

yujun777 reviewed Apr 16, 2026

View reviewed changes

seawinde added 4 commits April 16, 2026 12:02

fix comment

ba9f329

fix comment

000cfce

opt code

8df6ade

abcxyz333 approved these changes Apr 16, 2026

View reviewed changes

github-actions Bot added the reviewed label Apr 16, 2026

yujun777 merged commit af2a553 into yujun777:ivm Apr 16, 2026
14 of 20 checks passed

Conversation

seawinde commented Apr 15, 2026

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yujun777 Apr 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yujun777 Apr 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

abcxyz333 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Apr 16, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

yujun777 Apr 16, 2026 •

edited

Loading

yujun777 Apr 16, 2026 •

edited

Loading