Skip to content

Commit b08d70e

Browse files
authored
chore(catalog-listing): remove deprecated split_files free fn (Closes apache#23080) (apache#23152)
Fixes apache#23080 (partial) ## Summary Removes `datafusion_catalog_listing::helpers::split_files`, deprecated since 47.0.0 in favor of the `FileGroup::split_files` method on `datafusion-catalog/datasource::file_groups::FileGroup`. Verified workspace-wide: - 0 non-test call sites for the deprecated free function (`grep -rn "split_files" --include="*.rs"`) - The 5 in-tree references in `datafusion/catalog-listing/src/helpers.rs::tests::test_split_files` are calls on `FileGroup` values (`files.clone().split_files(N)`), not the deprecated free function — they exercise `FileGroup::split_files` correctly. - 3 other call sites in `datafusion/catalog-listing/src/table.rs` also call the `FileGroup` method. - `cargo check -p datafusion-catalog-listing` is clean (no warnings after also dropping the now-unused `use std::mem;`). - `cargo test -p datafusion-catalog-listing test_split_files` passes. This is a 36-line pure deletion: 35 lines for the function body plus the `use std::mem;` import. Replaces only — the public method `FileGroup::split_files` already exists at `datafusion/datasource/src/file_groups.rs:454` and is what callers use. ## Test plan - `cargo check -p datafusion-catalog-listing` ✓ (clean) - `cargo test -p datafusion-catalog-listing test_split_files` ✓ 1 passed - `grep -rn "helpers::split_files\|use.*helpers::{.*split_files" datafusion/` → 0 matches after this change `(e of in-tree call sites verified before edit via workspace grep; revisited after edit.)` ## AI assistance This patch was drafted with the help of an AI coding assistant. The diff was reviewed line by line before submission and the change was verified independently (cargo check + a focused cargo test run). Signed-off-by: Dodothereal <[email protected]> Signed-off-by: Dodothereal <[email protected]>
1 parent e2c3e18 commit b08d70e

1 file changed

Lines changed: 0 additions & 36 deletions

File tree

‎datafusion/catalog-listing/src/helpers.rs‎

Lines changed: 0 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,6 @@
1717

1818
//! Helper functions for the table implementation
1919
20-
use std::mem;
2120
use std::sync::Arc;
2221

2322
use datafusion_catalog::Session;
@@ -137,41 +136,6 @@ pub fn expr_applicable_for_cols(col_names: &[&str], expr: &Expr) -> bool {
137136
/// The maximum number of concurrent listing requests
138137
const CONCURRENCY_LIMIT: usize = 100;
139138

140-
/// Partition the list of files into `n` groups
141-
#[deprecated(since = "47.0.0", note = "use `FileGroup::split_files` instead")]
142-
pub fn split_files(
143-
mut partitioned_files: Vec<PartitionedFile>,
144-
n: usize,
145-
) -> Vec<Vec<PartitionedFile>> {
146-
if partitioned_files.is_empty() {
147-
return vec![];
148-
}
149-
150-
// ObjectStore::list does not guarantee any consistent order and for some
151-
// implementations such as LocalFileSystem, it may be inconsistent. Thus
152-
// Sort files by path to ensure consistent plans when run more than once.
153-
partitioned_files.sort_by(|a, b| a.path().cmp(b.path()));
154-
155-
// effectively this is div with rounding up instead of truncating
156-
let chunk_size = partitioned_files.len().div_ceil(n);
157-
let mut chunks = Vec::with_capacity(n);
158-
let mut current_chunk = Vec::with_capacity(chunk_size);
159-
for file in partitioned_files.drain(..) {
160-
current_chunk.push(file);
161-
if current_chunk.len() == chunk_size {
162-
let full_chunk =
163-
mem::replace(&mut current_chunk, Vec::with_capacity(chunk_size));
164-
chunks.push(full_chunk);
165-
}
166-
}
167-
168-
if !current_chunk.is_empty() {
169-
chunks.push(current_chunk)
170-
}
171-
172-
chunks
173-
}
174-
175139
#[derive(Debug)]
176140
pub struct Partition {
177141
/// The path to the partition, including the table prefix

0 commit comments

Comments
 (0)