Skip to content

Commit 23e21f4

Browse files
Larboratorzhousimin
andauthored
[improve](nereids) filter nereidsPrunedTabletIds per partition in distributionPrune (apache#63851)
### What problem does this PR solve? apache#53403 short-circuited distributionPrune to return the entire nereidsPrunedTabletIds set when running under Nereids. However, the caller computeTabletInfo invokes distributionPrune inside a per-partition loop and then iterates the returned ids, calling MaterializedIndex.getTablet(id) on each. When nereidsPrunedTabletIds contains tablets across many partitions, every per-partition iteration walks the entire global set and does a getTablet hash lookup on ids that belong to other partitions (which are then filtered out by the null check), yielding O(partitionNum * globalPrunedSize) lookups. The short-circuit also copies the full HashSet into a new ArrayList once per partition. Filter the global set down to the current partition's tablet ids (tabletIdsInOrder, already prepared by the caller) before returning. The result is identical to what the caller's null-check would have produced, so behavior is unchanged; only the redundant lookups and copies are eliminated. The non-Nereids path, the sampleTabletIds path and the empty-set fallback are untouched. Issue Number: close apache#63854 Related PR: apache#53403 Problem Summary: Plan time of OlapScan queries with many partitions and many globally pruned tablets degrades quadratically due to redundant per-partition iterations over the global pruned tablet set in OlapScanNode.distributionPrune. Restore per-partition complexity by filtering the global set down to the current partition's tablets before returning. Co-authored-by: zhousimin <[email protected]>
1 parent 14f7cd2 commit 23e21f4

1 file changed

Lines changed: 17 additions & 5 deletions

File tree

‎fe/fe-core/src/main/java/org/apache/doris/planner/OlapScanNode.java‎

Lines changed: 17 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -376,9 +376,20 @@ private Collection<Long> distributionPrune(
376376
DistributionInfo distributionInfo,
377377
boolean pruneTablesByNereids) throws AnalysisException {
378378
if (pruneTablesByNereids) {
379-
return nereidsPrunedTabletIds.isEmpty()
380-
? null
381-
: new ArrayList<>(nereidsPrunedTabletIds);
379+
if (nereidsPrunedTabletIds.isEmpty()) {
380+
return null;
381+
}
382+
// Filter to tablets belonging to this partition. Without this, the caller's
383+
// per-partition loop in computeTabletInfo becomes O(partitionNum * globalPrunedSize)
384+
// getTablet hash lookups (most returning null), which dominates plan time
385+
// when both partition count and pruned tablet count are large.
386+
List<Long> result = new ArrayList<>();
387+
for (Long id : tabletIdsInOrder) {
388+
if (nereidsPrunedTabletIds.contains(id)) {
389+
result.add(id);
390+
}
391+
}
392+
return result;
382393
}
383394
DistributionPruner distributionPruner = null;
384395
switch (distributionInfo.getType()) {
@@ -924,8 +935,9 @@ private void computeTabletInfo() throws UserException {
924935
boolean notExistsSampleAndPrunedTablets = sampleTabletIds.isEmpty() && nereidsPrunedTabletIds.isEmpty();
925936
if (prunedTabletIds != null) {
926937
for (Long id : prunedTabletIds) {
927-
if (selectedTable.getTablet(id) != null) {
928-
tablets.add(selectedTable.getTablet(id));
938+
Tablet tablet = selectedTable.getTablet(id);
939+
if (tablet != null) {
940+
tablets.add(tablet);
929941
scanTabletIds.add(id);
930942
} else if (notExistsSampleAndPrunedTablets) {
931943
// The tabletID specified in query does not exist in this partition, skip scan partition.

0 commit comments

Comments
 (0)