Repository navigation
Commit 23e21f4
[improve](nereids) filter nereidsPrunedTabletIds per partition in distributionPrune (apache#63851)
### What problem does this PR solve?
apache#53403 short-circuited distributionPrune to return the entire
nereidsPrunedTabletIds set when running under Nereids. However, the
caller computeTabletInfo invokes distributionPrune inside a
per-partition loop and then iterates the returned ids, calling
MaterializedIndex.getTablet(id) on each. When nereidsPrunedTabletIds
contains tablets across many
partitions, every per-partition iteration walks the entire global set
and does a getTablet hash lookup on ids that belong to other partitions
(which are then filtered out by the null check), yielding O(partitionNum
* globalPrunedSize) lookups. The short-circuit also copies the full
HashSet into a new ArrayList once per partition.
Filter the global set down to the current partition's tablet ids
(tabletIdsInOrder, already prepared by the caller) before returning. The
result is identical to what the caller's null-check would have produced,
so behavior is unchanged; only the redundant lookups and copies are
eliminated. The non-Nereids path, the sampleTabletIds path and the
empty-set
fallback are untouched.
Issue Number: close apache#63854
Related PR: apache#53403
Problem Summary:
Plan time of OlapScan queries with many partitions and many globally
pruned tablets degrades quadratically due to redundant per-partition
iterations over the global pruned tablet set in
OlapScanNode.distributionPrune. Restore per-partition complexity by
filtering the global set down to the current partition's tablets before
returning.
Co-authored-by: zhousimin <[email protected]>1 parent 14f7cd2 commit 23e21f4
1 file changed
Lines changed: 17 additions & 5 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
376 | 376 | | |
377 | 377 | | |
378 | 378 | | |
379 | | - | |
380 | | - | |
381 | | - | |
| 379 | + | |
| 380 | + | |
| 381 | + | |
| 382 | + | |
| 383 | + | |
| 384 | + | |
| 385 | + | |
| 386 | + | |
| 387 | + | |
| 388 | + | |
| 389 | + | |
| 390 | + | |
| 391 | + | |
| 392 | + | |
382 | 393 | | |
383 | 394 | | |
384 | 395 | | |
| |||
924 | 935 | | |
925 | 936 | | |
926 | 937 | | |
927 | | - | |
928 | | - | |
| 938 | + | |
| 939 | + | |
| 940 | + | |
929 | 941 | | |
930 | 942 | | |
931 | 943 | | |
| |||
0 commit comments