Repository navigation
[fix](mtmv) Preserve origin statement for internal refresh - #68269
Merged
Merged
Conversation
### What problem does this PR solve? Problem Summary: Internally constructed MTMV refresh commands could lose their origin statement. Complete refreshes created a StatementContext without the MV query, while StmtExecutor copied a null LogicalPlanAdapter origin back into contexts that already had one. Initialize complete-refresh contexts with the MV query and propagate the same OriginStatement to the LogicalPlanAdapter before executor construction. ### Release note Fix internal materialized view refresh planning to retain its originating SQL statement. ### Check List (For Author) - Test: Unit Test - Behavior changed: No - Does this need documentation: No
morrySnow
requested review from
924060929,
englefly,
starocean999 and
yujun777
as code owners
September 20, 2026 09:36
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
Author
|
/review |
Contributor
|
Codex automated review failed and did not complete. Error: Selected model is at capacity. Please try a different model. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
Contributor
TPC-H: Total hot run time: 27605 ms |
Contributor
TPC-DS: Total hot run time: 151814 ms |
Contributor
ClickBench: Total hot run time: 23.88 s |
Contributor
FE UT Coverage ReportIncrement line coverage |
1 similar comment
Contributor
FE UT Coverage ReportIncrement line coverage |
yiguolei
pushed a commit
that referenced
this pull request
Sep 21, 2026
### What problem does this PR solve? Issue Number: None Related PR: #68269 Problem Summary: Internal MTMV refresh tasks could construct a StatementContext and LogicalPlanAdapter without an origin statement. StmtExecutor then copied the null adapter origin back into the context. If CBO candidate costing failed, CostBasedRewriteJob dereferenced that missing origin while logging and raised an unrelated NullPointerException. This change initializes refresh planning with the MV query, propagates the same origin through the adapter, and makes the CBO failure log tolerate internal contexts without an origin statement. ### Release note Fix internal materialized view refresh failures caused by a missing origin statement. ### Check List (For Author) - Test: Unit Test - ./run-fe-ut.sh --run org.apache.doris.mtmv.MTMVTaskTest,org.apache.doris.nereids.jobs.rewrite.CostBasedRewriteJobTest - 17 tests passed - Behavior changed: No - Does this need documentation: No
…tatement # Conflicts: # fe/fe-core/src/main/java/org/apache/doris/job/extensions/mtmv/MTMVTask.java
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 28129 ms |
Contributor
TPC-DS: Total hot run time: 152674 ms |
Contributor
ClickBench: Total hot run time: 23.9 s |
Contributor
Author
|
/review |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Review opinion: Request changes.
The origin-statement propagation itself is focused and internally consistent: complete/partition refreshes now create an origin-bearing StatementContext, the shared executor helper transfers the same OriginStatement into LogicalPlanAdapter, and both production callers satisfy the new non-null contract. One directly related parallel refresh bug remains and is called out inline.
Critical checkpoints:
- Goal and proof: the changed tests verify both the production MV-query origin and object identity through
StmtExecutor. The author reports the two FE unit suites passed; this automated review was static-only and did not run builds or tests, as required by the review prompt. - Scope and clarity: the four-file change is small and clear. The new precondition fails loudly and all in-repository callers were enumerated.
- Concurrency and lifecycle: no new shared state or locking is introduced. Executor publication/clearing, cancellation, retry isolation, query unregistration, statement-context close, and task thread-local cleanup remain unchanged.
- Configuration and compatibility: no configuration, persistence/EditLog, storage format, FE-BE protocol, or rolling-upgrade contract changes.
- Parallel paths and conditions: complete/partition origin handling is correct, and incremental execution already supplies a non-null origin. However, incremental and incremental-dry-run parsing still do not install that context before parsing
SET_VAR, causing the inline failure. Base/head verification shows this ordering predates the branch, but it is a substantiated, uncovered bug in the reviewed parallel call chain. - Data writes and transactions: insert/overwrite planning, transactionality, and visibility behavior are unchanged by origin propagation.
- Observability and performance: the origin improves query/profile and lock diagnostics; explicit refresh audit text remains unchanged. No meaningful performance regression was found.
- User focus: no additional user-provided focus points.
Review completion: two full review rounds converged at head bb43257535eba46e57c20abec68975bdc6b4ab96; all candidates were independently verified, deduplicated, and resolved. One inline P2 issue is submitted.
### What problem does this PR solve? Issue Number: None Related PR: apache#68269 Problem Summary: Incremental refresh and dry run parsed the MV query before installing their new StatementContext on the internal ConnectContext. Parsing a SET_VAR hint reads that context and could fail with a null dereference. Install the origin-bearing context before parsing in both paths, and cover the hinted MV query with FE and regression tests. ### Release note Fix incremental materialized view refresh and dry run for MV queries containing SET_VAR hints. ### Check List (For Author) - Test: Unit Test - Focused FE unit tests and Checkstyle passed. - Hinted dry-run regression case added; not run locally. - Behavior changed: Yes. Hinted incremental refreshes now parse successfully. - Does this need documentation: No
Contributor
Author
|
/review |
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 28270 ms |
Contributor
TPC-DS: Total hot run time: 153217 ms |
Contributor
ClickBench: Total hot run time: 23.9 s |
Contributor
There was a problem hiding this comment.
Static review of exact head 7cb38cef96b1ed820e7a12859518a692e8f72941 against base 6b3ea1eb56dcf57c5f2dd5abb9f170e0fbcf1907.
Requesting changes for two P2 issues:
- Incremental refresh can let a stored
SET_VAR(compute_group=...)override the compute group captured for the manual task, unlike COMPLETE/PARTITIONS, and can report a different group from the one used. - Incremental dry-run cancellation is not durable during setup/planning, so work can start after Ctrl+C/KILL/timeout.
The current head does fix the previously reported missing-StatementContext SET_VAR NPE and preserves the origin statement through the internal executor; that existing thread was not duplicated.
Critical checkpoint conclusions:
- Goal and proof: explicit origins and pre-parse contexts address the stated origin/hint failure, but the two issues above prevent approval.
- Scope: production changes are otherwise narrow and focused.
- Concurrency: no new locks or shared mutable structures; the remaining lifecycle defect is the nested dry-run cancellation handoff.
- Lifecycle: per-attempt contexts, retries, background cleanup, and dry-run caller-context restoration are sound by static tracing apart from that cancellation handoff.
- Configuration: no new configuration was added; the compute-group issue concerns inconsistent precedence of an existing session variable.
- Compatibility: no storage, journal, RPC, function-symbol, or rolling-upgrade format change.
- Parallel paths: COMPLETE/PARTITIONS and INCREMENTAL order hint parsing versus task affinity differently; explain/context paths were also checked.
- Conditions and errors: the non-null origin precondition fails loudly and all production callers satisfy it.
- Tests: added unit tests cover origin/context wiring and generic hinted incremental/dry-run parsing; the regression exercises real hinted incremental refresh and dry run, but there is no conflicting-compute-group or early-cancellation test.
- Test results: the author reports 181 related FE unit tests and Checkstyle passed; the regression suite was not run. This review did not execute tests or builds.
- Observability: origin/audit propagation improves; the compute-group issue can still make task reporting disagree with execution.
- Persistence, transactions, and writes: no metadata format or EditLog change; insert transaction and atomicity paths are unchanged.
- FE-BE propagation: no new variable or protocol field is introduced.
- Performance: no material hot-path cost was found.
- User focus: no additional focus was provided.
starocean999
approved these changes
Sep 24, 2026
morningman
added a commit
that referenced
this pull request
Sep 24, 2026
### What problem does this PR solve? Issue Number: None (branch pick) Related PR: #67490, #68141, #66963, #68269 Problem Summary: Pick the four merged `incremental-computation` PRs that do not have the `incremental-computation-picked` label into `branch-incremental-computation`, in their master merge order: 1. #67490: prevent a local exchange under a serial parent pipeline. 2. #68141: manage MTMV planning caches through a bounded global LRU cache and expose cache statistics. 3. #66963: restrict HLL, QUANTILE_STATE, and AGG_STATE columns to Aggregate Key tables by default, with a temporary compatibility setting. 4. #68269: preserve the origin statement for internal MTMV refreshes and parse `SET_VAR` hints with the refresh context installed. Dependency check: #33262, referenced by #68141, is already an ancestor of this branch. #66664 is an open related row-binlog fix rather than a code prerequisite for #66963. #68272 is the corresponding branch-4.1 fix for #68269, not a master prerequisite. No additional PR needed picking. ### Release note Bring the four fixes and MTMV cache management change above to the incremental computation branch. In particular, non-Aggregate Key tables now reject new HLL, QUANTILE_STATE, and AGG_STATE columns by default; incremental MTMV refreshes and dry runs can parse MV queries with `SET_VAR` hints. ### Check List (For Author) - Test: - `./build.sh --fe` (passed, including Checkstyle with 0 violations) - Focused `./run-fe-ut.sh --run ...` across 13 changed and related test classes (356 tests passed, 0 failures/errors/skips) - SQL regression tests were not run; this worktree has no BE cluster. - Behavior changed: Yes, as described in the source PRs and release note. - Does this need documentation: No; the source PRs do not require a documentation PR. --------- Co-authored-by: Mryange <[email protected]> Co-authored-by: xy720 <[email protected]> Co-authored-by: Luwei <[email protected]> Co-authored-by: morrySnow <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Problem Summary:
Internally constructed MTMV refresh commands could lose their origin statement. Complete refreshes created a
StatementContextwithout the MV query, whileStmtExecutorcopied a nullLogicalPlanAdapterorigin back into contexts that already had one. Complete and partition refreshes now use the MV query as their origin and pass the same origin to the adapter.Incremental refresh and incremental dry run had a related parsing order bug: an MV definition with a
SET_VARhint was parsed before its newStatementContextwas installed on the internalConnectContext. Hint parsing reads that context and could fail with a null dereference. Both paths now install the origin-bearing context before parsing the MV query. The existing dry-run regression case uses a hinted MV definition to cover both refresh paths.Release note
Fix internal materialized view refresh planning to retain its originating SQL statement, and allow incremental refresh and dry run of MV queries containing
SET_VARhints.Check List (For Author)
./run-fe-ut.sh --run org.apache.doris.mtmv.ivm.IvmIncrRefreshManagerTest,org.apache.doris.nereids.trees.plans.commands.RefreshMTMVCommandTest(26 passed)./run-fe-ut.sh --run org.apache.doris.nereids.trees.plans.CreateMTMVCommandTest,org.apache.doris.mtmv.MTMVTaskTest,org.apache.doris.nereids.trees.plans.commands.UpdateMvByPartitionCommandTest(155 passed)mvn -pl fe-core checkstyle:check -DskipTests -Dmaven.build.cache.enabled=false -q(passed)SET_VARhints.