Repository navigation
[Bug] LogicalOlapTableStreamScan can loop forever after partition pruning #65654
Description
Search before asking
- I had searched in the issues and found no similar issues.
Version
master (reproduced on a local master-based development branch)
What's Wrong?
LogicalOlapTableStreamScan can enter an optimizer rewrite loop after partition pruning.
The issue is not specific to MTMV itself. MTMV incremental refresh just made it easy to expose, because its generated plan contains stream scans and goes through the same pruning path. The real problem is in the generic stream-scan builder contract used by the optimizer.
In the bad case, FE repeatedly rewrites the same stream scan, the query/task does not finish, and FE keeps allocating plans and burning CPU.
This issue is related to the Table Stream / row binlog stack tracked by:
- [Tracking][Binlog] Binlog feature umbrella — all sub-issues tracked here #65265
- [Feature] Track incremental computation with row binlog, Table Stream, and MTMV #65418
What You Expected?
Once partition pruning has been applied to a stream scan, the rewritten plan should be marked as already pruned, so the same pruning rule should not keep matching it forever.
How to Reproduce?
One easy reproduction path is a stream-based plan produced during IVM regression. For example, regression-test/suites/mtmv_p0/ivm/test_ivm_partition_unique_key.groovy can trigger the problem before the fix.
The important part is not MTMV semantics themselves, but that the final refresh plan contains LogicalOlapTableStreamScan and partition pruning rewrites it.
Observed symptoms before the fix:
- the task remains
RUNNING - FE stack traces stay in optimizer rewrite
- FE repeatedly hits
PruneOlapScanPartitionon a stream scan
Current Behavior
Observed on FE:
TopDownVisitorRewriteJob.doRewritekeeps revisiting the same subtreePruneOlapScanPartitioncontinues matching the same stream scan- the rewritten
LogicalOlapTableStreamScanis still considered not pruned
Root Cause
LogicalOlapTableStreamScan.withSelectedPartitionIds(List<Long>, boolean) does not follow the parent LogicalOlapScan contract.
In the parent class, the second argument means hasPartitionPredicate, and the rebuilt scan is always marked as partitionPruned=true.
In LogicalOlapTableStreamScan, the override interprets the second argument as isPartitionPruned and passes it into the constructor directly. As a result, when PruneOlapScanPartition calls:
scan.withSelectedPartitionIds(prunedPartitions, hasPartitionPredicate)and hasPartitionPredicate is false, the rebuilt stream scan also gets partitionPruned=false. Then the same pruning rule matches again, rewrites again, and the optimizer loops.
Anything Else?
The local fix is to align LogicalOlapTableStreamScan.withSelectedPartitionIds(List<Long>, boolean) with the parent contract:
- treat the boolean parameter as
hasPartitionPredicate - always rebuild the stream scan with
partitionPruned=true
After applying that fix locally, the previously hanging reproduction path finishes normally.
Activity
zclllyybb commentedon Jul 15, 2026
Breakwater-GitHub-Analysis-Slot: slot_1e4f4948da0f
This content is generated by AI for reference only.
Initial analysis from the current
upstream/mastercode path:This looks like a real FE/Nereids rewrite-loop bug in the stream-scan subclass contract, not an MTMV-only issue.
The confirmed chain is:
PruneOlapScanPartitionmatches only unpruned OLAP scans (!scan.isPartitionPruned()).scan.withSelectedPartitionIds(prunedPartitions, hasPartitionPredicate).LogicalOlapScan, that second argument meanshasPartitionPredicate, and the rebuilt scan is always created withpartitionPruned=true.LogicalOlapTableStreamScanextendsLogicalOlapScan, so it is eligible for the same pruning rule, but its override currently treats the same second argument asisPartitionPrunedand passes it into the stream-scan constructor as thepartitionPrunedvalue.hasPartitionPredicate=false, the rebuiltLogicalOlapTableStreamScanstill haspartitionPruned=false. The pruning rule can keep matching the same logical shape and repeatedly rebuild the plan.That matches the reported symptoms: repeated
PruneOlapScanPartition,TopDownVisitorRewriteJobrevisiting the same subtree, task remainingRUNNING, and FE CPU/allocation pressure.I also found the linked public fix PR: #65657. Its diff aligns with the code-level diagnosis: keep the boolean parameter as
hasPartitionPredicateand passtruefor the rebuilt stream scan'spartitionPrunedflag. That is the right fix direction for this specific loop.Suggested next steps:
./run-regression-test.sh --run -d mtmv_p0 -s test_ivm_partition_unique_keywas run manually; if possible, attach/keep that result or add a deterministic regression assertion so this does not regress silently.LogicalOlapTableStreamScanand the same override. If yes, it should receive the same fix because the bug is in a generic builder contract used by the optimizer.Missing information is not blocking the root-cause assessment, because the current master code already shows the contract mismatch. For impact scoping/backport decisions, the useful extra data would be the exact commit SHA of the reproduced development branch and, if available, one FE stack sample from the hanging task showing the repeated pruning rewrite.
yujun777 commentedon Jul 22, 2026
FIX PR: #65657
[fix](stream) Fix stream scan partition prune state propagation (apac…
[fix](stream) Fix stream scan partition prune state propagation (apac…
[fix](stream) Fix stream scan partition prune state propagation (apac…