Skip to content

Commit 50cfb70

Browse files
adriangbclaude
authored andcommitted
refactor(proto): move PartitionedFile / FileGroup serde into datafusion-datasource (apache#24006)
## Which issue does this PR close? - Part of apache#23494. Precursor for apache#23497 / apache#23683 (`DataSource` / `FileSource` proto hooks) and for apache#23752. ## Rationale for this change The protobuf conversions for the file-scan leaf types — `PartitionedFile`, `FileGroup`, `FileRange` — live in `datafusion-proto` as `TryFromProto` impls, because that is historically the only crate that can name both sides (the DataFusion type and the prost message are both foreign to it, hence the `TryFromProto` workaround trait in the first place). That placement means any *other* crate that needs those conversions has to reimplement them. apache#23683 hits exactly this: a `FileSource` serializing its own scan config needs to encode file groups, so the first cut of that PR grew a private second copy of the `PartitionedFile` wire logic inside `datafusion-datasource`, which can then drift from the central serializer. The same will be true of every source migrated under apache#23516–apache#23518. Nothing about these conversions needs `datafusion-proto`: they are plain data, with `ScalarValue` / `Statistics` / `Schema` going through `datafusion-proto-common`. They belong next to the types. ## What changes are included in this PR? - New `datafusion_datasource::proto` module, behind a new `proto` feature on `datafusion-datasource` (off by default; `datafusion-proto` enables it): - `FileRange::try_to_proto` / `try_from_proto` - `PartitionedFile::try_to_proto` / `try_from_proto` - `FileGroup` <-> `protobuf::FileGroup` - `datafusion-proto`'s `TryFromProto` impls for those types become one-line shims delegating to the new impls, so every existing caller keeps working and the two sides cannot disagree. ### Why these are `TryFrom` and not `try_to_proto` hooks `TryFromProto` exists because `datafusion-proto` owns neither side of the conversions it hosts: with both the DataFusion type and the prost message foreign to it, `impl TryFrom<protobuf::X> for X` is rejected by the orphan rule, so a local trait was the only way to say the same thing. Moving a conversion into the crate that owns the DataFusion type removes that constraint, and `&T` is `#[fundamental]`, so both directions are expressible with the standard trait (checked, not assumed): ```rust impl TryFrom<&protobuf::PartitionedFile> for PartitionedFile // ok impl TryFrom<&PartitionedFile> for protobuf::PartitionedFile // ok impl TryFrom<&[PartitionedFile]> for protobuf::FileGroup // E0117 ``` The last one is why `protobuf::FileGroup`'s *slice* conversion stays a `TryFromProto` shim: `&[PartitionedFile]` is not a type this crate owns, while `&FileGroup` is. Callers inside DataFusion go through `FileGroup`. So the rule this PR sets for the rest of apache#23494: **plain data uses `TryFrom`; anything needing an encode/decode context keeps the `try_to_proto(ctx)` / `try_from_proto(node, ctx)` hooks**, because the standard trait cannot carry that second argument. Usefully, none of the ~40 `TryFromProto`/`FromProto` impls needs a context, and nothing that needs one was ever a `TryFromProto` impl — the two categories are already disjoint, so the shape now tells a reader whether a conversion recurses. ### Why now `FromProto` / `TryFromProto` were added in apache#21929, *after* the 54.0.0 release, and 54.1.0 was cut before any of this landed — so they have never shipped in a release. Replacing them with the standard traits, and eventually deleting them, is a no-op for semver **today** and a major breaking change the moment 55.0.0 goes out. The same applies to the six inherent `try_to_proto` / `try_from_proto` methods this PR would otherwise have added: they are new, unreleased API, so choosing their final shape costs nothing right now. The other reason to settle it here rather than in a follow-up: this is the PR that establishes the pattern for the data-source family (apache#23516-apache#23519 and apache#23752 / apache#23781 are all queued behind it). Whichever shape merges first is the one they will copy. Retiring the remaining ~34 impls is still its own follow-up. Two notes for whoever picks it up: the sink and format-option conversions can move next to their types the same way, but the ones for `datafusion-common`-owned types (`JoinType`, `NullEquality`, `TableReference`, `UnnestOptions`, ...) cannot — `datafusion-common` cannot depend on `datafusion-proto-models` (it is underneath it via `datafusion-proto-common`). Their legal home is `proto-models` itself, implementing on the local proto type, which is already how `proto-common` hosts the `ScalarValue` / `Statistics` conversions. ## Are these changes tested? Yes. - New unit tests in `datafusion_datasource::proto` covering the `PartitionedFile` round trip (path, size, mtime, partition values, range, arrow schema, statistics), the `FileGroup` round trip, and the invalid-path error. - The existing `datafusion-proto` tests now exercise the delegating shims, so they also pin the shims themselves. - `datafusion-proto`, all features: 227 passed / 0 failed. - `datafusion-datasource` with `proto`: 180 passed / 0 failed. - Full workspace run: 10347 passed (`cargo test --profile ci --workspace --lib --tests --features avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption`). The only failures are 8 backtrace-symbolization tests in `datafusion-common`, a crate this PR does not touch and which sits below every crate it does; they fail the same way on the base commit on macOS. - `cargo fmt` and `ci/scripts/rust_clippy.sh` clean. ## Are there any user-facing changes? The protobuf wire format is unchanged, and no existing API changes shape. Additive: - New `proto` feature on `datafusion-datasource` (off by default). - New `TryFrom` impls in both directions between `FileRange`, `PartitionedFile`, `FileGroup` and their protobuf messages, under that feature. No new names are added to the crate's API surface: the trait is `core::convert::TryFrom`. Note for reviewers: while writing the round-trip test I found that `PartitionedFile` statistics do not round-trip cleanly on `main` — filed as apache#23998. This PR preserves that behavior exactly rather than changing decode semantics in a refactor; the test documents it. --------- Co-authored-by: Claude Opus 5 <[email protected]>
1 parent 44d8de5 commit 50cfb70

7 files changed

Lines changed: 272 additions & 79 deletions

File tree

‎Cargo.lock‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎datafusion/datasource/Cargo.toml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,10 @@ all-features = true
3434
backtrace = ["datafusion-common/backtrace"]
3535
compression = ["async-compression", "liblzma", "bzip2", "flate2", "zstd", "tokio-util"]
3636
default = ["compression"]
37+
# Enables the protobuf conversions for the file-scan leaf types owned by this
38+
# crate (`FileRange`, `PartitionedFile`, `FileGroup`). Off by default so
39+
# consumers that never serialize plans pay nothing.
40+
proto = ["dep:datafusion-proto-models"]
3741

3842
[dependencies]
3943
arrow = { workspace = true }
@@ -56,6 +60,7 @@ datafusion-physical-expr = { workspace = true }
5660
datafusion-physical-expr-adapter = { workspace = true }
5761
datafusion-physical-expr-common = { workspace = true }
5862
datafusion-physical-plan = { workspace = true }
63+
datafusion-proto-models = { workspace = true, optional = true }
5964
datafusion-session = { workspace = true }
6065
flate2 = { workspace = true, optional = true }
6166
futures = { workspace = true }

‎datafusion/datasource/src/mod.rs‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,10 @@ pub mod file_stream;
4141
pub mod memory;
4242
pub mod morsel;
4343
pub mod projection;
44+
/// Protobuf conversions for [`FileRange`], [`PartitionedFile`] and
45+
/// [`FileGroup`](crate::file_groups::FileGroup), gated on the `proto` feature.
46+
#[cfg(feature = "proto")]
47+
mod proto;
4448
pub mod schema_adapter;
4549
pub mod sink;
4650
pub mod source;

‎datafusion/datasource/src/proto.rs‎

Lines changed: 238 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,238 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
//! Protobuf conversions for the file-scan leaf types owned by this crate:
19+
//! [`FileRange`], [`PartitionedFile`] and [`FileGroup`].
20+
//!
21+
//! These are the single copy of that wire logic. `datafusion-proto`'s
22+
//! `TryFromProto` implementations for the same types are thin shims that
23+
//! delegate here, so the format cannot drift between the central serializer and
24+
//! the per-source `try_to_proto` hooks.
25+
//!
26+
//! None of these conversions need a codec or an encode/decode context: every
27+
//! field is plain data or goes through `datafusion-proto-common`. That is why
28+
//! they are plain [`TryFrom`] impls rather than the `try_to_proto(ctx)` /
29+
//! `try_from_proto(node, ctx)` hooks used for plans, expressions and scan
30+
//! configs: the standard trait can express a conversion that takes nothing but
31+
//! the value, and the orphan rule allows it here because one side of each
32+
//! conversion is a type this crate owns.
33+
34+
use std::sync::Arc;
35+
36+
use chrono::{TimeZone, Utc};
37+
use datafusion_common::{DataFusionError, Result, internal_datafusion_err};
38+
use datafusion_proto_models::protobuf;
39+
use object_store::ObjectMeta;
40+
use object_store::path::Path;
41+
42+
use crate::file_groups::FileGroup;
43+
use crate::{FileRange, PartitionedFile};
44+
45+
impl TryFrom<&FileRange> for protobuf::FileRange {
46+
type Error = DataFusionError;
47+
48+
fn try_from(range: &FileRange) -> Result<Self> {
49+
Ok(protobuf::FileRange {
50+
start: range.start,
51+
end: range.end,
52+
})
53+
}
54+
}
55+
56+
impl TryFrom<&protobuf::FileRange> for FileRange {
57+
type Error = DataFusionError;
58+
59+
fn try_from(range: &protobuf::FileRange) -> Result<Self> {
60+
Ok(FileRange {
61+
start: range.start,
62+
end: range.end,
63+
})
64+
}
65+
}
66+
67+
impl TryFrom<&PartitionedFile> for protobuf::PartitionedFile {
68+
type Error = DataFusionError;
69+
70+
fn try_from(file: &PartitionedFile) -> Result<Self> {
71+
let last_modified = file.object_meta.last_modified;
72+
let last_modified_ns = last_modified.timestamp_nanos_opt().ok_or_else(|| {
73+
DataFusionError::Plan(format!(
74+
"Invalid timestamp on PartitionedFile::ObjectMeta: {last_modified}"
75+
))
76+
})? as u64;
77+
Ok(protobuf::PartitionedFile {
78+
arrow_schema: file
79+
.arrow_schema
80+
.as_ref()
81+
.map(|s| s.as_ref().try_into())
82+
.transpose()?,
83+
path: file.object_meta.location.as_ref().to_owned(),
84+
size: file.object_meta.size,
85+
last_modified_ns,
86+
partition_values: file
87+
.partition_values
88+
.iter()
89+
.map(|v| v.try_into())
90+
.collect::<Result<Vec<_>, _>>()?,
91+
range: file.range.as_ref().map(TryInto::try_into).transpose()?,
92+
statistics: file.statistics.as_ref().map(|s| s.as_ref().into()),
93+
})
94+
}
95+
}
96+
97+
impl TryFrom<&protobuf::PartitionedFile> for PartitionedFile {
98+
type Error = DataFusionError;
99+
100+
fn try_from(file: &protobuf::PartitionedFile) -> Result<Self> {
101+
let mut pf = PartitionedFile::new_from_meta(ObjectMeta {
102+
location: Path::parse(file.path.as_str()).map_err(|e| {
103+
internal_datafusion_err!("Invalid object_store path: {e}")
104+
})?,
105+
last_modified: Utc.timestamp_nanos(file.last_modified_ns as i64),
106+
size: file.size,
107+
e_tag: None,
108+
version: None,
109+
})
110+
.with_partition_values(
111+
file.partition_values
112+
.iter()
113+
.map(|v| v.try_into())
114+
.collect::<Result<Vec<_>, _>>()?,
115+
);
116+
if let Some(proto_schema) = file.arrow_schema.as_ref() {
117+
pf = pf.with_arrow_schema(Arc::new(
118+
proto_schema.try_into().map_err(DataFusionError::from)?,
119+
));
120+
}
121+
if let Some(range) = file.range.as_ref() {
122+
let range = FileRange::try_from(range)?;
123+
pf = pf.with_range(range.start, range.end);
124+
}
125+
if let Some(proto_stats) = file.statistics.as_ref() {
126+
// The wire format carries statistics for the full table schema (file + partition
127+
// columns), so assign directly — `with_statistics` would append the partition
128+
// column stats a second time.
129+
pf.statistics = Some(Arc::new(proto_stats.try_into()?));
130+
}
131+
Ok(pf)
132+
}
133+
}
134+
135+
impl TryFrom<&FileGroup> for protobuf::FileGroup {
136+
type Error = DataFusionError;
137+
138+
fn try_from(group: &FileGroup) -> Result<Self> {
139+
Ok(protobuf::FileGroup {
140+
files: group
141+
.files()
142+
.iter()
143+
.map(TryInto::try_into)
144+
.collect::<Result<Vec<_>>>()?,
145+
})
146+
}
147+
}
148+
149+
impl TryFrom<&protobuf::FileGroup> for FileGroup {
150+
type Error = DataFusionError;
151+
152+
fn try_from(group: &protobuf::FileGroup) -> Result<Self> {
153+
Ok(FileGroup::new(
154+
group
155+
.files
156+
.iter()
157+
.map(TryInto::try_into)
158+
.collect::<Result<Vec<_>>>()?,
159+
))
160+
}
161+
}
162+
163+
#[cfg(test)]
164+
mod tests {
165+
use arrow::datatypes::{DataType, Field, Schema};
166+
use datafusion_common::{ScalarValue, Statistics};
167+
168+
use super::*;
169+
170+
#[test]
171+
fn partitioned_file_roundtrip_preserves_all_fields() -> Result<()> {
172+
let schema = Arc::new(Schema::new(vec![Field::new("a", DataType::Int32, true)]));
173+
let pf = PartitionedFile::new_from_meta(ObjectMeta {
174+
location: Path::parse("foo/bar.parquet")?,
175+
last_modified: Utc.timestamp_nanos(1_000_000_000),
176+
size: 1234,
177+
e_tag: None,
178+
version: None,
179+
})
180+
.with_partition_values(vec![ScalarValue::from("2024-01-01")])
181+
.with_range(10, 20)
182+
.with_arrow_schema(Arc::clone(&schema))
183+
.with_statistics(Arc::new(Statistics::new_unknown(&schema)));
184+
185+
let encoded = protobuf::PartitionedFile::try_from(&pf)?;
186+
let decoded = PartitionedFile::try_from(&encoded)?;
187+
188+
assert_eq!(decoded.object_meta.location, pf.object_meta.location);
189+
assert_eq!(decoded.object_meta.size, pf.object_meta.size);
190+
assert_eq!(
191+
decoded.object_meta.last_modified,
192+
pf.object_meta.last_modified
193+
);
194+
assert_eq!(decoded.partition_values, pf.partition_values);
195+
assert_eq!(decoded.range, pf.range);
196+
assert_eq!(decoded.arrow_schema.as_deref(), Some(schema.as_ref()));
197+
// Statistics span the full table schema (file columns followed by one
198+
// entry per partition column), and survive the round trip intact.
199+
assert_eq!(
200+
pf.statistics.as_ref().unwrap().column_statistics.len(),
201+
schema.fields().len() + pf.partition_values.len()
202+
);
203+
assert_eq!(decoded.statistics, pf.statistics);
204+
Ok(())
205+
}
206+
207+
#[test]
208+
fn partitioned_file_from_proto_rejects_invalid_path() {
209+
let proto = protobuf::PartitionedFile {
210+
path: "foo//bar.parquet".to_string(),
211+
..Default::default()
212+
};
213+
214+
let err = PartitionedFile::try_from(&proto).unwrap_err();
215+
assert!(
216+
err.to_string().contains("Invalid object_store path"),
217+
"unexpected error: {err}"
218+
);
219+
}
220+
221+
#[test]
222+
fn file_group_roundtrip() -> Result<()> {
223+
let group = FileGroup::new(vec![
224+
PartitionedFile::new("a.parquet", 1),
225+
PartitionedFile::new("b.parquet", 2),
226+
]);
227+
228+
let encoded = protobuf::FileGroup::try_from(&group)?;
229+
let decoded = FileGroup::try_from(&encoded)?;
230+
231+
assert_eq!(decoded.len(), 2);
232+
assert_eq!(
233+
decoded.files()[1].object_meta.location,
234+
group.files()[1].object_meta.location
235+
);
236+
Ok(())
237+
}
238+
}

‎datafusion/proto/Cargo.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -54,7 +54,7 @@ chrono = { workspace = true }
5454
datafusion-catalog = { workspace = true }
5555
datafusion-catalog-listing = { workspace = true }
5656
datafusion-common = { workspace = true }
57-
datafusion-datasource = { workspace = true }
57+
datafusion-datasource = { workspace = true, features = ["proto"] }
5858
datafusion-datasource-arrow = { workspace = true }
5959
datafusion-datasource-avro = { workspace = true, optional = true }
6060
datafusion-datasource-csv = { workspace = true }

‎datafusion/proto/src/physical_plan/from_proto.rs‎

Lines changed: 10 additions & 44 deletions
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,6 @@ use arrow::array::RecordBatch;
2323
use arrow::compute::SortOptions;
2424
use arrow::datatypes::{Field, Schema};
2525
use arrow::ipc::reader::StreamReader;
26-
use chrono::{TimeZone, Utc};
2726
use datafusion_common::{
2827
DataFusionError, Result, ScalarValue, internal_datafusion_err, not_impl_err,
2928
};
@@ -56,8 +55,6 @@ use datafusion_physical_plan::{
5655
Partitioning, PhysicalExpr, RangePartitioning, SplitPoint, WindowExpr,
5756
};
5857
use datafusion_proto_common::common::proto_error;
59-
use object_store::ObjectMeta;
60-
use object_store::path::Path;
6158

6259
use super::{
6360
DefaultPhysicalProtoConverter, PhysicalExtensionCodec, PhysicalPlanDecodeContext,
@@ -632,64 +629,30 @@ pub fn parse_record_batches(buf: &[u8]) -> Result<Vec<RecordBatch>> {
632629
Ok(batches)
633630
}
634631

632+
/// Thin shim over `TryFrom<&protobuf::PartitionedFile>`, which owns the wire logic.
635633
impl TryFromProto<&protobuf::PartitionedFile> for PartitionedFile {
636634
type Error = DataFusionError;
637635

638636
fn try_from_proto(val: &protobuf::PartitionedFile) -> Result<Self, Self::Error> {
639-
let mut pf = PartitionedFile::new_from_meta(ObjectMeta {
640-
location: Path::parse(val.path.as_str())
641-
.map_err(|e| proto_error(format!("Invalid object_store path: {e}")))?,
642-
last_modified: Utc.timestamp_nanos(val.last_modified_ns as i64),
643-
size: val.size,
644-
e_tag: None,
645-
version: None,
646-
})
647-
.with_partition_values(
648-
val.partition_values
649-
.iter()
650-
.map(|v| v.try_into())
651-
.collect::<Result<Vec<_>, _>>()?,
652-
);
653-
if let Some(proto_schema) = val.arrow_schema.as_ref() {
654-
pf = pf.with_arrow_schema(Arc::new(
655-
proto_schema.try_into().map_err(DataFusionError::from)?,
656-
));
657-
}
658-
if let Some(range) = val.range.as_ref() {
659-
let file_range = FileRange::try_from_proto(range)?;
660-
pf = pf.with_range(file_range.start, file_range.end);
661-
}
662-
if let Some(proto_stats) = val.statistics.as_ref() {
663-
// The wire format carries statistics for the full table schema (file + partition
664-
// columns), so assign directly — `with_statistics` would append the partition
665-
// column stats a second time.
666-
pf.statistics = Some(Arc::new(proto_stats.try_into()?));
667-
}
668-
Ok(pf)
637+
PartitionedFile::try_from(val)
669638
}
670639
}
671640

641+
/// Thin shim over `TryFrom<&protobuf::FileRange>`, which owns the wire logic.
672642
impl TryFromProto<&protobuf::FileRange> for FileRange {
673643
type Error = DataFusionError;
674644

675645
fn try_from_proto(value: &protobuf::FileRange) -> Result<Self, Self::Error> {
676-
Ok(FileRange {
677-
start: value.start,
678-
end: value.end,
679-
})
646+
FileRange::try_from(value)
680647
}
681648
}
682649

650+
/// Thin shim over `TryFrom<&protobuf::FileGroup>`, which owns the wire logic.
683651
impl TryFromProto<&protobuf::FileGroup> for FileGroup {
684652
type Error = DataFusionError;
685653

686654
fn try_from_proto(val: &protobuf::FileGroup) -> Result<Self, Self::Error> {
687-
let files = val
688-
.files
689-
.iter()
690-
.map(PartitionedFile::try_from_proto)
691-
.collect::<Result<Vec<_>, _>>()?;
692-
Ok(FileGroup::new(files))
655+
FileGroup::try_from(val)
693656
}
694657
}
695658

@@ -734,7 +697,7 @@ impl TryFromProto<&protobuf::FileSinkConfig> for FileSinkConfig {
734697
let file_group = FileGroup::new(
735698
conf.file_groups
736699
.iter()
737-
.map(PartitionedFile::try_from_proto)
700+
.map(TryInto::try_into)
738701
.collect::<Result<Vec<_>>>()?,
739702
);
740703
let table_paths = conf
@@ -812,6 +775,9 @@ impl datafusion_physical_expr_common::physical_expr::proto_decode::PhysicalExprD
812775
mod tests {
813776
use super::*;
814777
use arrow::datatypes::{DataType, Field, Schema};
778+
use chrono::{TimeZone, Utc};
779+
use object_store::ObjectMeta;
780+
use object_store::path::Path;
815781

816782
#[test]
817783
fn partitioned_file_path_roundtrip_percent_encoded() {

0 commit comments

Comments
 (0)