Skip to content

Commit 5f6b84f

Browse files
committed
feat: lower repartition_file_min_size default from 10 MiB to 1 MiB
`repartition_file_min_size` gates how aggressively `repartitioned()` splits file groups by byte range to fan a scan out across `target_partitions` worth of cores. At 10 MiB the default leaves several SF1-sized dimension tables (TPC-H `part` ≈ 24 MiB, TPC-DS `customer_address` ≈ 7 MiB, …) on a single partition, so any CPU-bound per-batch work in the scan (filter eval, dictionary expansion, etc.) is single-threaded even when the cluster has plenty of idle cores. At 1 MiB those same files split cleanly into `target_partitions` byte ranges, e.g. TPC-H Q22 drops from 30 ms → 17 ms (~1.75× faster) on a 12-core SF1 run by parallelising the `part_with_promo` filter. The cost — more `open()` calls, more metadata loads — is small (10 vs 1 extra opens per file in the worst case, each amortised over the row-group / page-index reads) and the existing knob is still available for workloads where it matters. The csv_files.slt reset is switched from `SET ... = 10485760` to `RESET ...` so the test continues to round-trip the configured default regardless of what that default is.
1 parent 50d74a7 commit 5f6b84f

4 files changed

Lines changed: 12 additions & 7 deletions

File tree

‎datafusion/common/src/config.rs‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1151,8 +1151,13 @@ config_namespace! {
11511151
/// in parallel using the provided `target_partitions` level
11521152
pub repartition_aggregations: bool, default = true
11531153

1154-
/// Minimum total files size in bytes to perform file scan repartitioning.
1155-
pub repartition_file_min_size: usize, default = 10 * 1024 * 1024
1154+
/// Minimum total file size in bytes for file-group byte-range
1155+
/// splitting to fire. Files (or merged file groups) smaller than this
1156+
/// stay as one partition. Lower values produce more, smaller
1157+
/// partitions — better at filling `target_partitions` worth of cores
1158+
/// when files are modestly sized, at the cost of slightly more
1159+
/// per-partition open / metadata-load overhead.
1160+
pub repartition_file_min_size: usize, default = 1024 * 1024
11561161

11571162
/// Should DataFusion repartition data using the join keys to execute joins in parallel
11581163
/// using the provided `target_partitions` level

‎datafusion/sqllogictest/test_files/csv_files.slt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -376,7 +376,7 @@ id3 value3
376376

377377
# Reset repartition_file_min_size to default value
378378
statement ok
379-
SET datafusion.optimizer.repartition_file_min_size = 10485760;
379+
RESET datafusion.optimizer.repartition_file_min_size;
380380

381381
statement ok
382382
drop table stored_table_with_cr_terminator;

‎datafusion/sqllogictest/test_files/information_schema.slt‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -325,7 +325,7 @@ datafusion.optimizer.prefer_existing_union false
325325
datafusion.optimizer.prefer_hash_join true
326326
datafusion.optimizer.preserve_file_partitions 0
327327
datafusion.optimizer.repartition_aggregations true
328-
datafusion.optimizer.repartition_file_min_size 10485760
328+
datafusion.optimizer.repartition_file_min_size 1048576
329329
datafusion.optimizer.repartition_file_scans true
330330
datafusion.optimizer.repartition_joins true
331331
datafusion.optimizer.repartition_sorts true
@@ -475,7 +475,7 @@ datafusion.optimizer.prefer_existing_union false When set to true, the optimizer
475475
datafusion.optimizer.prefer_hash_join true When set to true, the physical plan optimizer will prefer HashJoin over SortMergeJoin. HashJoin can work more efficiently than SortMergeJoin but consumes more memory
476476
datafusion.optimizer.preserve_file_partitions 0 Minimum number of distinct partition values required to group files by their Hive partition column values (enabling Hash partitioning declaration). How the option is used: - preserve_file_partitions=0: Disable it. - preserve_file_partitions=1: Always enable it. - preserve_file_partitions=N, actual file partitions=M: Only enable when M >= N. This threshold preserves I/O parallelism when file partitioning is below it. Note: This may reduce parallelism, rooting from the I/O level, if the number of distinct partitions is less than the target_partitions.
477477
datafusion.optimizer.repartition_aggregations true Should DataFusion repartition data using the aggregate keys to execute aggregates in parallel using the provided `target_partitions` level
478-
datafusion.optimizer.repartition_file_min_size 10485760 Minimum total files size in bytes to perform file scan repartitioning.
478+
datafusion.optimizer.repartition_file_min_size 1048576 Minimum total file size in bytes for file-group byte-range splitting to fire. Files (or merged file groups) smaller than this stay as one partition. Lower values produce more, smaller partitions — better at filling `target_partitions` worth of cores when files are modestly sized, at the cost of slightly more per-partition open / metadata-load overhead.
479479
datafusion.optimizer.repartition_file_scans true When set to `true`, datasource partitions will be repartitioned to achieve maximum parallelism. This applies to both in-memory partitions and FileSource's file groups (1 group is 1 partition). For FileSources, only Parquet and CSV formats are currently supported. If set to `true` for a FileSource, all files will be repartitioned evenly (i.e., a single large file might be partitioned into smaller chunks) for parallel scanning. If set to `false` for a FileSource, different files will be read in parallel, but repartitioning won't happen within a single file. If set to `true` for an in-memory source, all memtable's partitions will have their batches repartitioned evenly to the desired number of `target_partitions`. Repartitioning can change the total number of partitions and batches per partition, but does not slice the initial record tables provided to the MemTable on creation.
480480
datafusion.optimizer.repartition_joins true Should DataFusion repartition data using the join keys to execute joins in parallel using the provided `target_partitions` level
481481
datafusion.optimizer.repartition_sorts true Should DataFusion execute sorts in a per-partition fashion and merge afterwards instead of coalescing first and sorting globally. With this flag is enabled, plans in the form below ```text "SortExec: [a@0 ASC]", " CoalescePartitionsExec", " RepartitionExec: partitioning=RoundRobinBatch(8), input_partitions=1", ``` would turn into the plan below which performs better in multithreaded environments ```text "SortPreservingMergeExec: [a@0 ASC]", " SortExec: [a@0 ASC]", " RepartitionExec: partitioning=RoundRobinBatch(8), input_partitions=1", ```
@@ -895,7 +895,7 @@ show functions
895895
statement ok
896896
reset datafusion.catalog.information_schema;
897897

898-
# The SLT runner sets `target_partitions` to 4 instead of using the default, so
898+
# The SLT runner sets `target_partitions` to 4 instead of using the default, so
899899
# reset it explicitly.
900900
statement ok
901901
set datafusion.execution.target_partitions = 4;

‎docs/source/user-guide/configs.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -149,7 +149,7 @@ The following configuration settings are available:
149149
| datafusion.optimizer.enable_dynamic_filter_pushdown | true | When set to true attempts to push down dynamic filters generated by operators (TopK, Join & Aggregate) into the file scan phase. For example, for a query such as `SELECT * FROM t ORDER BY timestamp DESC LIMIT 10`, the optimizer will attempt to push down the current top 10 timestamps that the TopK operator references into the file scans. This means that if we already have 10 timestamps in the year 2025 any files that only have timestamps in the year 2024 can be skipped / pruned at various stages in the scan. The config will suppress `enable_join_dynamic_filter_pushdown`, `enable_topk_dynamic_filter_pushdown` & `enable_aggregate_dynamic_filter_pushdown` So if you disable `enable_topk_dynamic_filter_pushdown`, then enable `enable_dynamic_filter_pushdown`, the `enable_topk_dynamic_filter_pushdown` will be overridden. |
150150
| datafusion.optimizer.filter_null_join_keys | false | When set to true, the optimizer will insert filters before a join between a nullable and non-nullable column to filter out nulls on the nullable side. This filter can add additional overhead when the file format does not fully support predicate push down. |
151151
| datafusion.optimizer.repartition_aggregations | true | Should DataFusion repartition data using the aggregate keys to execute aggregates in parallel using the provided `target_partitions` level |
152-
| datafusion.optimizer.repartition_file_min_size | 10485760 | Minimum total files size in bytes to perform file scan repartitioning. |
152+
| datafusion.optimizer.repartition_file_min_size | 1048576 | Minimum total file size in bytes for file-group byte-range splitting to fire. Files (or merged file groups) smaller than this stay as one partition. Lower values produce more, smaller partitions — better at filling `target_partitions` worth of cores when files are modestly sized, at the cost of slightly more per-partition open / metadata-load overhead. |
153153
| datafusion.optimizer.repartition_joins | true | Should DataFusion repartition data using the join keys to execute joins in parallel using the provided `target_partitions` level |
154154
| datafusion.optimizer.allow_symmetric_joins_without_pruning | true | Should DataFusion allow symmetric hash joins for unbounded data sources even when its inputs do not have any ordering or filtering If the flag is not enabled, the SymmetricHashJoin operator will be unable to prune its internal buffers, resulting in certain join types - such as Full, Left, LeftAnti, LeftSemi, Right, RightAnti, and RightSemi - being produced only at the end of the execution. This is not typical in stream processing. Additionally, without proper design for long runner execution, all types of joins may encounter out-of-memory errors. |
155155
| datafusion.optimizer.repartition_file_scans | true | When set to `true`, datasource partitions will be repartitioned to achieve maximum parallelism. This applies to both in-memory partitions and FileSource's file groups (1 group is 1 partition). For FileSources, only Parquet and CSV formats are currently supported. If set to `true` for a FileSource, all files will be repartitioned evenly (i.e., a single large file might be partitioned into smaller chunks) for parallel scanning. If set to `false` for a FileSource, different files will be read in parallel, but repartitioning won't happen within a single file. If set to `true` for an in-memory source, all memtable's partitions will have their batches repartitioned evenly to the desired number of `target_partitions`. Repartitioning can change the total number of partitions and batches per partition, but does not slice the initial record tables provided to the MemTable on creation. |

0 commit comments

Comments
 (0)