You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 44544cd
Browse filesBrowse the repository at this point in the historyBrowse files
perf: Optimize range partitioning across multi-key and asymmetric joins (#6342)
## What
Coordinates range partitioning across MPP joins, prioritizing large
relations and multi-key bridge tables, enables 1-sided asymmetric range
adaptation, and incorporates upstream DataFusion improvements.
## Why
- Tables declaring multiple partition keys (e.g. `user_id,topic_id`)
could previously be claimed by small dimension joins during bottom-up
traversal, locking them out of 0-shuffle co-partitioning with large fact
tables.
- Joining partitioned tables with unpartitioned ones can keep the larger
table local (0 shuffles) and adapt only the smaller partner (1 shuffle),
rather than forcing a 2-sided hash shuffle.
- Tracked upstream as apache/datafusion#25302
- `JoinSelection` converts partitioned joins to `CollectLeft`
(broadcast) based on row/byte thresholds alone, ignoring that physically
co-partitioned inputs run task-locally with zero network cost.
- Tracked upstream as apache/datafusion#25301
- Round-robin repartitioning added redundant partitions and stages in
our single-threaded task model.
## How
- `RangePartitioningRule`: Ranks candidate join edges globally by data
volume (`min(left_rows, right_rows)`). Stamps asymmetric anchors only
when strictly larger than their partner or when the partner is already
committed to another key.
- `RangeCoPartitionedJoinRule`: Placed before `EnsureRequirements` to
flip `CollectLeft` back to `Partitioned` for range co-partitioned inputs
and peel unnecessary repartition nodes.
- Session config: Disabled `optimizer.enable_round_robin_repartition` in
planner and worker sessions.
## Tests
- Added `pg_search/tests/pg_regress/sql/mpp_range_partitioning.sql`
exercising 2-table co-partitioning, multi-key bridge joins,
aggregations, and asymmetric joins.
- Benchmarks show up to 70% improvement on some range partitioned joins.
---------
Co-authored-by: paradedb-github-app[bot] <282009505+paradedb-github-app[bot]@users.noreply.github.com>
|`paradedb.enable_custom_scan`|`on`| user | Enables ParadeDB's base custom scan, which replaces table scans with ParadeDB index scans. |
34
-
|`paradedb.enable_aggregate_custom_scan`|`on`| user | Enables the aggregate custom scan for eligible `pdb.agg` and `GROUP BY` queries. |
35
-
|`paradedb.enable_join_custom_scan`|`on`| user | Enables the join custom scan, which pushes eligible joins into the ParadeDB executor. |
36
-
|`paradedb.enable_custom_scan_without_operator`|`off`| user | Allows custom scans without a ParadeDB search operator when the whole `WHERE` clause can push down. |
37
-
|`paradedb.enable_filter_pushdown`|`on`| user | Allows non-indexed predicates to use heap filtering after the ParadeDB scan. |
38
-
|`paradedb.enable_range_partitioned_join`|`off`| user | Enables range co-partitioned joins for indexes configured with `partition_by`.|
39
-
|`paradedb.enable_heuristic_selectivity`|`on`| user | Uses cheap planner selectivity estimates for expensive fuzzy, regex, and range query shapes. |
40
-
|`paradedb.per_tuple_cost`|`100000000.0`| user | Planner cost multiplier that discourages using ParadeDB operators outside a supported index plan. |
41
-
|`paradedb.explain_recursive_estimates`|`off`| user | Shows recursive document-count estimates in `EXPLAIN VERBOSE`; intended for debugging. |
|`paradedb.enable_custom_scan`|`on`| user | Enables ParadeDB's base custom scan, which replaces table scans with ParadeDB index scans. |
34
+
|`paradedb.enable_aggregate_custom_scan`|`on`| user | Enables the aggregate custom scan for eligible `pdb.agg` and `GROUP BY` queries. |
35
+
|`paradedb.enable_join_custom_scan`|`on`| user | Enables the join custom scan, which pushes eligible joins into the ParadeDB executor. |
36
+
|`paradedb.enable_custom_scan_without_operator`|`off`| user | Allows custom scans without a ParadeDB search operator when the whole `WHERE` clause can push down. |
37
+
|`paradedb.enable_filter_pushdown`|`on`| user | Allows non-indexed predicates to use heap filtering after the ParadeDB scan. |
38
+
|`paradedb.enable_range_partitioned_join`|`off`| user | Enables range co-partitioned joins and asymmetric range alignment for MPP joins on indexes configured with `partition_by`. |
39
+
|`paradedb.enable_heuristic_selectivity`|`on`| user | Uses cheap planner selectivity estimates for expensive fuzzy, regex, and range query shapes. |
40
+
|`paradedb.per_tuple_cost`|`100000000.0`| user | Planner cost multiplier that discourages using ParadeDB operators outside a supported index plan. |
41
+
|`paradedb.explain_recursive_estimates`|`off`| user | Shows recursive document-count estimates in `EXPLAIN VERBOSE`; intended for debugging. |
0 commit comments