Skip to content

Commit 554f227

Browse files
Dandandanclaude
andcommitted
Raise hash_join_single_partition_threshold to 4MB
`JoinSelection` collects one side of a hash join into a single partition when its estimated size stays under this threshold, rather than repartitioning both inputs. The bound has been 1MB, which is small: a 4MB build side is still cheaper to collect once than to shuffle both sides for, so more joins against a small side can skip the exchange. The tests that assert byte sizes derived from the threshold now derive their expectations from it too, instead of hardcoding the numbers the 1MB default produced. Co-Authored-By: Claude Opus 5 <[email protected]>
1 parent 9213bb5 commit 554f227

5 files changed

Lines changed: 29 additions & 19 deletions

File tree

‎datafusion/common/src/config.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1689,7 +1689,7 @@ config_namespace! {
16891689

16901690
/// The maximum estimated size in bytes for one input side of a HashJoin
16911691
/// will be collected into a single partition
1692-
pub hash_join_single_partition_threshold: usize, default = 1024 * 1024
1692+
pub hash_join_single_partition_threshold: usize, default = 4 * 1024 * 1024
16931693

16941694
/// The maximum estimated size in rows for one input side of a HashJoin
16951695
/// will be collected into a single partition

‎datafusion/core/tests/physical_optimizer/join_selection.rs‎

Lines changed: 24 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,16 @@ fn get_thresholds() -> (usize, usize) {
7575
)
7676
}
7777

78+
/// The byte size [`small_statistics`] derives from the configured threshold.
79+
fn small_byte_size() -> usize {
80+
get_thresholds().1 / 128
81+
}
82+
83+
/// The byte size [`big_statistics`] derives from the configured threshold.
84+
fn big_byte_size() -> usize {
85+
get_thresholds().1 * 2
86+
}
87+
7888
/// Return statistics for small table
7989
fn small_statistics() -> Statistics {
8090
let (threshold_num_rows, threshold_byte_size) = get_thresholds();
@@ -258,14 +268,14 @@ async fn test_join_with_swap() {
258268
.compute(swapped_join.left().as_ref(), &StatisticsArgs::new())
259269
.unwrap()
260270
.total_byte_size,
261-
Precision::Inexact(8192)
271+
Precision::Inexact(small_byte_size())
262272
);
263273
assert_eq!(
264274
StatisticsContext::new()
265275
.compute(swapped_join.right().as_ref(), &StatisticsArgs::new())
266276
.unwrap()
267277
.total_byte_size,
268-
Precision::Inexact(2097152)
278+
Precision::Inexact(big_byte_size())
269279
);
270280
}
271281

@@ -382,14 +392,14 @@ async fn test_left_join_no_swap() {
382392
.compute(swapped_join.left().as_ref(), &StatisticsArgs::new())
383393
.unwrap()
384394
.total_byte_size,
385-
Precision::Inexact(8192)
395+
Precision::Inexact(small_byte_size())
386396
);
387397
assert_eq!(
388398
StatisticsContext::new()
389399
.compute(swapped_join.right().as_ref(), &StatisticsArgs::new())
390400
.unwrap()
391401
.total_byte_size,
392-
Precision::Inexact(2097152)
402+
Precision::Inexact(big_byte_size())
393403
);
394404
}
395405

@@ -431,14 +441,14 @@ async fn test_join_with_swap_semi() {
431441
.compute(swapped_join.left().as_ref(), &StatisticsArgs::new())
432442
.unwrap()
433443
.total_byte_size,
434-
Precision::Inexact(8192)
444+
Precision::Inexact(small_byte_size())
435445
);
436446
assert_eq!(
437447
StatisticsContext::new()
438448
.compute(swapped_join.right().as_ref(), &StatisticsArgs::new())
439449
.unwrap()
440450
.total_byte_size,
441-
Precision::Inexact(2097152)
451+
Precision::Inexact(big_byte_size())
442452
);
443453
assert_eq!(original_schema, swapped_join.schema());
444454
}
@@ -668,14 +678,14 @@ async fn test_join_with_swap_mark() {
668678
.compute(swapped_join.left().as_ref(), &StatisticsArgs::new())
669679
.unwrap()
670680
.total_byte_size,
671-
Precision::Inexact(8192)
681+
Precision::Inexact(small_byte_size())
672682
);
673683
assert_eq!(
674684
StatisticsContext::new()
675685
.compute(swapped_join.right().as_ref(), &StatisticsArgs::new())
676686
.unwrap()
677687
.total_byte_size,
678-
Precision::Inexact(2097152)
688+
Precision::Inexact(big_byte_size())
679689
);
680690
assert_eq!(original_schema, swapped_join.schema());
681691
}
@@ -794,14 +804,14 @@ async fn test_join_no_swap() {
794804
.compute(swapped_join.left().as_ref(), &StatisticsArgs::new())
795805
.unwrap()
796806
.total_byte_size,
797-
Precision::Inexact(8192)
807+
Precision::Inexact(small_byte_size())
798808
);
799809
assert_eq!(
800810
StatisticsContext::new()
801811
.compute(swapped_join.right().as_ref(), &StatisticsArgs::new())
802812
.unwrap()
803813
.total_byte_size,
804-
Precision::Inexact(2097152)
814+
Precision::Inexact(big_byte_size())
805815
);
806816
}
807817

@@ -867,14 +877,14 @@ async fn test_nl_join_with_swap(join_type: JoinType) {
867877
.compute(swapped_join.left().as_ref(), &StatisticsArgs::new())
868878
.unwrap()
869879
.total_byte_size,
870-
Precision::Inexact(8192)
880+
Precision::Inexact(small_byte_size())
871881
);
872882
assert_eq!(
873883
StatisticsContext::new()
874884
.compute(swapped_join.right().as_ref(), &StatisticsArgs::new())
875885
.unwrap()
876886
.total_byte_size,
877-
Precision::Inexact(2097152)
887+
Precision::Inexact(big_byte_size())
878888
);
879889
}
880890

@@ -938,14 +948,14 @@ async fn test_nl_join_with_swap_no_proj(join_type: JoinType) {
938948
.compute(swapped_join.left().as_ref(), &StatisticsArgs::new())
939949
.unwrap()
940950
.total_byte_size,
941-
Precision::Inexact(8192)
951+
Precision::Inexact(small_byte_size())
942952
);
943953
assert_eq!(
944954
StatisticsContext::new()
945955
.compute(swapped_join.right().as_ref(), &StatisticsArgs::new())
946956
.unwrap()
947957
.total_byte_size,
948-
Precision::Inexact(2097152)
958+
Precision::Inexact(big_byte_size())
949959
);
950960
}
951961

‎datafusion/sqllogictest/test_files/information_schema.slt‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -324,7 +324,7 @@ datafusion.optimizer.expand_views_at_output false
324324
datafusion.optimizer.filter_null_join_keys false
325325
datafusion.optimizer.hash_join_inlist_pushdown_max_distinct_values 150
326326
datafusion.optimizer.hash_join_inlist_pushdown_max_size 131072
327-
datafusion.optimizer.hash_join_single_partition_threshold 1048576
327+
datafusion.optimizer.hash_join_single_partition_threshold 4194304
328328
datafusion.optimizer.hash_join_single_partition_threshold_rows 131072
329329
datafusion.optimizer.join_reordering true
330330
datafusion.optimizer.max_passes 3
@@ -484,7 +484,7 @@ datafusion.optimizer.expand_views_at_output false When set to true, if the retur
484484
datafusion.optimizer.filter_null_join_keys false When set to true, the optimizer will insert filters before a join between a nullable and non-nullable column to filter out nulls on the nullable side. This filter can add additional overhead when the file format does not fully support predicate push down.
485485
datafusion.optimizer.hash_join_inlist_pushdown_max_distinct_values 150 Maximum number of distinct values (rows) in the build side of a hash join to be pushed down as an InList expression for dynamic filtering. Build sides with more rows than this will use hash table lookups instead. Set to 0 to always use hash table lookups. This provides an additional limit beyond `hash_join_inlist_pushdown_max_size` to prevent very large IN lists that might not provide much benefit over hash table lookups. This uses the deduplicated row count once the build side has been evaluated. The default is 150 values per partition. This is inspired by Trino's `max-filter-keys-per-column` setting. See: <https://trino.io/docs/current/admin/dynamic-filtering.html#dynamic-filter-collection-thresholds>
486486
datafusion.optimizer.hash_join_inlist_pushdown_max_size 131072 Maximum size in bytes for the build side of a hash join to be pushed down as an InList expression for dynamic filtering. Build sides larger than this will use hash table lookups instead. Set to 0 to always use hash table lookups. InList pushdown can be more efficient for small build sides because it can result in better statistics pruning as well as use any bloom filters present on the scan side. InList expressions are also more transparent and easier to serialize over the network in distributed uses of DataFusion. On the other hand InList pushdown requires making a copy of the data and thus adds some overhead to the build side and uses more memory. This setting is per-partition, so we may end up using `hash_join_inlist_pushdown_max_size` * `target_partitions` memory. The default is 128kB per partition. This should allow point lookup joins (e.g. joining on a unique primary key) to use InList pushdown in most cases but avoids excessive memory usage or overhead for larger joins.
487-
datafusion.optimizer.hash_join_single_partition_threshold 1048576 The maximum estimated size in bytes for one input side of a HashJoin will be collected into a single partition
487+
datafusion.optimizer.hash_join_single_partition_threshold 4194304 The maximum estimated size in bytes for one input side of a HashJoin will be collected into a single partition
488488
datafusion.optimizer.hash_join_single_partition_threshold_rows 131072 The maximum estimated size in rows for one input side of a HashJoin will be collected into a single partition
489489
datafusion.optimizer.join_reordering true When set to true, the physical plan optimizer may swap join inputs based on statistics. When set to false, statistics-driven join input reordering is disabled and the original join order in the query is used.
490490
datafusion.optimizer.max_passes 3 Number of times that the optimizer will attempt to optimize the plan

‎datafusion/sqllogictest/test_files/statistics_registry.slt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -162,7 +162,7 @@ statement ok
162162
set datafusion.optimizer.use_statistics_registry = false;
163163

164164
statement ok
165-
set datafusion.optimizer.hash_join_single_partition_threshold = 1048576;
165+
set datafusion.optimizer.hash_join_single_partition_threshold = 4194304;
166166

167167
statement ok
168168
set datafusion.optimizer.hash_join_single_partition_threshold_rows = 131072;

‎docs/source/user-guide/configs.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -173,7 +173,7 @@ The following configuration settings are available:
173173
| datafusion.optimizer.use_statistics_registry | false | When set to true, the physical plan optimizer uses the pluggable `StatisticsRegistry` for statistics propagation across operators. This enables more accurate cardinality estimates compared to each operator's built-in `partition_statistics`. |
174174
| datafusion.optimizer.prefer_hash_join | true | When set to true, the physical plan optimizer will prefer HashJoin over SortMergeJoin. HashJoin can work more efficiently than SortMergeJoin but consumes more memory |
175175
| datafusion.optimizer.enable_piecewise_merge_join | false | When set to true, piecewise merge join is enabled. PiecewiseMergeJoin is currently experimental. Physical planner will opt for PiecewiseMergeJoin when there is only one range filter. |
176-
| datafusion.optimizer.hash_join_single_partition_threshold | 1048576 | The maximum estimated size in bytes for one input side of a HashJoin will be collected into a single partition |
176+
| datafusion.optimizer.hash_join_single_partition_threshold | 4194304 | The maximum estimated size in bytes for one input side of a HashJoin will be collected into a single partition |
177177
| datafusion.optimizer.hash_join_single_partition_threshold_rows | 131072 | The maximum estimated size in rows for one input side of a HashJoin will be collected into a single partition |
178178
| datafusion.optimizer.hash_join_inlist_pushdown_max_size | 131072 | Maximum size in bytes for the build side of a hash join to be pushed down as an InList expression for dynamic filtering. Build sides larger than this will use hash table lookups instead. Set to 0 to always use hash table lookups. InList pushdown can be more efficient for small build sides because it can result in better statistics pruning as well as use any bloom filters present on the scan side. InList expressions are also more transparent and easier to serialize over the network in distributed uses of DataFusion. On the other hand InList pushdown requires making a copy of the data and thus adds some overhead to the build side and uses more memory. This setting is per-partition, so we may end up using `hash_join_inlist_pushdown_max_size` \* `target_partitions` memory. The default is 128kB per partition. This should allow point lookup joins (e.g. joining on a unique primary key) to use InList pushdown in most cases but avoids excessive memory usage or overhead for larger joins. |
179179
| datafusion.optimizer.hash_join_inlist_pushdown_max_distinct_values | 150 | Maximum number of distinct values (rows) in the build side of a hash join to be pushed down as an InList expression for dynamic filtering. Build sides with more rows than this will use hash table lookups instead. Set to 0 to always use hash table lookups. This provides an additional limit beyond `hash_join_inlist_pushdown_max_size` to prevent very large IN lists that might not provide much benefit over hash table lookups. This uses the deduplicated row count once the build side has been evaluated. The default is 150 values per partition. This is inspired by Trino's `max-filter-keys-per-column` setting. See: <https://trino.io/docs/current/admin/dynamic-filtering.html#dynamic-filter-collection-thresholds> |

0 commit comments

Comments
 (0)