Skip to content

Commit 51c8780

Browse files
Gene Bordegaraygabotechs
andcommitted
Add ListingOptions::output_partitioning and FileScanConfig::output_partitioning for pre-defined file partitioning (apache#22657)
<!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes apache#22645. <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> This follows up on apache#22607 by replacing range-partitioning sqllogictest boilerplate with a general file/listing scan API for declared output partitioning. Related: apache#21992, apache#22607, apache#22607 (comment) <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> - Add declared `output_partitioning` to file scan and listing table configuration. - Preserve declared partition counts during listing-table file grouping. - Serialize scan `output_partitioning` through physical plan proto. - Refactor `range_partitioning.slt` to use a CSV `ListingTable` instead of a custom test-only `TableProvider` / `DataSource`. Contract: - Declared partitioning expressions are written against the full table schema before scan projection. For example, `Range([range_key@0], [(10), (20)], 3)` remains valid if the scan projects `range_key` and falls back to `UnknownPartitioning(3)` if `range_key` is not projected. - Listing tables create one file group per declared output partition (which can exceed `target_partitions`). It is up to the user to plan their partitioning. For example, a 4-partition range declaration creates four scan file groups, adding empty trailing groups when fewer files are present. - File group index is part of the contract: file group `i` must contain rows for declared output partition `i`. DataFusion does not validate row placement, matching other user-declared properties such as sortedness. <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Yes. <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> Yes. This adds public API for declaring file/listing scan output partitioning. No breaking API changes. <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> --------- Co-authored-by: Gabriel <[email protected]>
1 parent 3598943 commit 51c8780

18 files changed

Lines changed: 1268 additions & 379 deletions

File tree

‎Cargo.lock‎

Lines changed: 0 additions & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎datafusion/catalog-listing/src/helpers.rs‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -325,7 +325,7 @@ pub fn evaluate_partition_prefix<'a>(
325325
}
326326
}
327327

328-
fn filter_partitions(
328+
pub fn filter_partitioned_file(
329329
pf: PartitionedFile,
330330
filters: &[Expr],
331331
df_schema: &DFSchema,
@@ -447,7 +447,7 @@ pub async fn pruned_partition_list<'a>(
447447
))
448448
})
449449
.try_filter_map(move |pf| {
450-
futures::future::ready(filter_partitions(pf, filters, &df_schema))
450+
futures::future::ready(filter_partitioned_file(pf, filters, &df_schema))
451451
})
452452
.boxed())
453453
}

‎datafusion/catalog-listing/src/options.rs‎

Lines changed: 53 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ use datafusion_common::plan_err;
2121
use datafusion_datasource::ListingTableUrl;
2222
use datafusion_datasource::file_format::FileFormat;
2323
use datafusion_execution::config::SessionConfig;
24-
use datafusion_expr::SortExpr;
24+
use datafusion_expr::{Partitioning, SortExpr};
2525
use futures::StreamExt;
2626
use futures::TryStreamExt;
2727
use itertools::Itertools;
@@ -61,6 +61,46 @@ pub struct ListingOptions {
6161
/// multiple equivalent orderings, the outer `Vec` will have a
6262
/// single element.
6363
pub file_sort_order: Vec<Vec<SortExpr>>,
64+
/// Declared output partitioning for scans from this table.
65+
///
66+
/// Expressions are logical expressions over the full table schema. When set,
67+
/// [`ListingTable`](crate::ListingTable) creates one file group per
68+
/// declared output partition. When unset, file grouping uses the scan-time
69+
/// [`SessionConfig::target_partitions`](datafusion_execution::config::SessionConfig::target_partitions).
70+
///
71+
/// Files are listed in path order, split into whole-file groups across the
72+
/// declared partition count, and then padded with trailing empty groups when
73+
/// needed. DataFusion does not route files by partition values or validate
74+
/// row placement, so callers must ensure file group `i` contains rows for
75+
/// partition `i`. Layouts that require explicit file-to-partition assignment
76+
/// are not supported.
77+
///
78+
/// For example, range partitioning on column `a` with split points
79+
/// `[10, 20, 30]` declares four output partitions. With three path-ordered
80+
/// files, the trailing partition is preserved as empty:
81+
///
82+
/// ```text
83+
/// files in path order: f0, f1, f2
84+
///
85+
/// file groups:
86+
/// partition 0: [f0]
87+
/// partition 1: [f1]
88+
/// partition 2: [f2]
89+
/// partition 3: []
90+
/// ```
91+
///
92+
/// With five path-ordered files, a partition can contain multiple files:
93+
///
94+
/// ```text
95+
/// files in path order: f0, f1, f2, f3, f4
96+
///
97+
/// file groups:
98+
/// partition 0: [f0, f1]
99+
/// partition 1: [f2, f3]
100+
/// partition 2: [f4]
101+
/// partition 3: []
102+
/// ```
103+
pub output_partitioning: Option<Partitioning>,
64104
}
65105

66106
impl ListingOptions {
@@ -78,6 +118,7 @@ impl ListingOptions {
78118
collect_stat: false,
79119
target_partitions: 1,
80120
file_sort_order: vec![],
121+
output_partitioning: None,
81122
}
82123
}
83124

@@ -136,6 +177,17 @@ impl ListingOptions {
136177
self
137178
}
138179

180+
/// Set declared output partitioning.
181+
///
182+
/// See [`Self::output_partitioning`] for the contract.
183+
pub fn with_output_partitioning(
184+
mut self,
185+
output_partitioning: Option<Partitioning>,
186+
) -> Self {
187+
self.output_partitioning = output_partitioning;
188+
self
189+
}
190+
139191
/// Set `table partition columns` on [`ListingOptions`] and returns self.
140192
///
141193
/// "partition columns," used to support [Hive Partitioning], are

0 commit comments

Comments
 (0)