Skip to content

Commit cd27b85

Browse files
Phoenix500526kosiew
authored andcommitted
Proto: migrate file sink serialization (apache#23781)
This is a stacked PR based on apache#23752. Only the commits on top of apache#23752 are part of this review. ## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes apache#123` indicates that this PR will close issue apache#123. --> - Closes apache#23519. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> apache#23752 adds the `DataSink::try_to_proto` hook and moves the shared `FileSinkConfig` protobuf conversion into `datafusion-datasource`. This PR uses that foundation to move CSV, JSON, and Parquet sink serialization out of the central `datafusion-proto` downcast chain. Each concrete sink now owns its format-specific protobuf encoding and decoding logic. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> - Implement `DataSink::try_to_proto` for `CsvSink`, `JsonSink`, and `ParquetSink`. - Add inherent `try_from_proto` methods to reconstruct each sink's `DataSinkExec`. - Repoint the physical-plan decode arms to the sink-owned decoders. - Add feature-gated protobuf dependencies to the three format crates. - Remove the active central `DataSinkExec` serialization dispatch after migrating its final built-in sink. - Retain the old serialization helpers as deprecated compatibility delegates. - Let sinks without a built-in protobuf representation fall through to the physical extension codec. - Preserve the existing protobuf wire representation. The migrations are split into one commit per sink. ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Yes. Existing sink round-trip tests cover the protobuf representation. The following checks passed: - Focused CSV, JSON, and Parquet sink round-trip tests. - All `datafusion-proto` integration tests. - `cargo check -p datafusion-proto --no-default-features`. - `cargo fmt --all`. - `cargo clippy --all-targets --all-features -- -D warnings`. - The required extended workspace test suite, including all 495 SQL logic test files. ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> No functional or wire-format changes are intended. The old compatibility helpers remain available but are deprecated. --------- Signed-off-by: Jiawei Zhao <[email protected]>
1 parent 1b15340 commit cd27b85

12 files changed

Lines changed: 433 additions & 200 deletions

File tree

‎Cargo.lock‎

Lines changed: 3 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎datafusion/datasource-csv/Cargo.toml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,13 @@ version.workspace = true
3030
[package.metadata.docs.rs]
3131
all-features = true
3232

33+
[features]
34+
proto = [
35+
"dep:datafusion-proto-models",
36+
"datafusion-datasource/proto",
37+
"datafusion-physical-plan/proto",
38+
]
39+
3340
[dependencies]
3441
arrow = { workspace = true }
3542
async-trait = { workspace = true }
@@ -41,6 +48,7 @@ datafusion-execution = { workspace = true }
4148
datafusion-expr = { workspace = true }
4249
datafusion-physical-expr-common = { workspace = true }
4350
datafusion-physical-plan = { workspace = true }
51+
datafusion-proto-models = { workspace = true, optional = true }
4452
datafusion-session = { workspace = true }
4553
futures = { workspace = true }
4654
object_store = { workspace = true }

‎datafusion/datasource-csv/src/file_format.rs‎

Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -825,6 +825,105 @@ impl DataSink for CsvSink {
825825
) -> Result<u64> {
826826
FileSink::write_all(self, data, context).await
827827
}
828+
829+
#[cfg(feature = "proto")]
830+
fn try_to_proto(
831+
&self,
832+
exec: &DataSinkExec,
833+
ctx: &datafusion_physical_plan::proto::ExecutionPlanEncodeCtx<'_>,
834+
) -> Result<Option<datafusion_proto_models::protobuf::PhysicalPlanNode>> {
835+
use datafusion_proto_models::protobuf;
836+
use protobuf::physical_plan_node::PhysicalPlanType;
837+
838+
let input = ctx.encode_child(exec.input())?;
839+
let sort_order = exec.encode_sort_order(ctx)?;
840+
let sink = protobuf::CsvSink::try_from(self)?;
841+
let node = protobuf::CsvSinkExecNode {
842+
input: Some(Box::new(input)),
843+
sink: Some(sink),
844+
sink_schema: Some(exec.schema().as_ref().try_into()?),
845+
sort_order,
846+
};
847+
Ok(Some(protobuf::PhysicalPlanNode {
848+
physical_plan_type: Some(PhysicalPlanType::CsvSink(Box::new(node))),
849+
}))
850+
}
851+
}
852+
853+
#[cfg(feature = "proto")]
854+
impl TryFrom<&CsvSink> for datafusion_proto_models::protobuf::CsvSink {
855+
type Error = DataFusionError;
856+
857+
fn try_from(value: &CsvSink) -> Result<Self> {
858+
Ok(Self {
859+
config: Some(value.config().try_into()?),
860+
writer_options: Some(value.writer_options().try_into()?),
861+
})
862+
}
863+
}
864+
865+
#[cfg(feature = "proto")]
866+
impl TryFrom<&datafusion_proto_models::protobuf::CsvSink> for CsvSink {
867+
type Error = DataFusionError;
868+
869+
fn try_from(value: &datafusion_proto_models::protobuf::CsvSink) -> Result<Self> {
870+
let config =
871+
FileSinkConfig::try_from(value.config.as_ref().ok_or_else(|| {
872+
datafusion_common::internal_datafusion_err!(
873+
"CsvSink is missing required field 'config'"
874+
)
875+
})?)?;
876+
let writer_options = value
877+
.writer_options
878+
.as_ref()
879+
.ok_or_else(|| {
880+
datafusion_common::internal_datafusion_err!(
881+
"CsvSink is missing required field 'writer_options'"
882+
)
883+
})?
884+
.try_into()?;
885+
886+
Ok(Self::new(config, writer_options))
887+
}
888+
}
889+
890+
#[cfg(feature = "proto")]
891+
impl CsvSink {
892+
/// Reconstructs a [`DataSinkExec`] containing a `CsvSink` from protobuf.
893+
pub fn try_from_proto(
894+
node: &datafusion_proto_models::protobuf::PhysicalPlanNode,
895+
ctx: &datafusion_physical_plan::proto::ExecutionPlanDecodeCtx<'_>,
896+
) -> Result<Arc<dyn ExecutionPlan>> {
897+
use datafusion_proto_models::protobuf;
898+
899+
let sink_node = datafusion_physical_plan::expect_plan_variant!(
900+
node,
901+
protobuf::physical_plan_node::PhysicalPlanType::CsvSink,
902+
"CsvSink",
903+
);
904+
let input = ctx.decode_required_child(
905+
sink_node.input.as_deref(),
906+
"CsvSinkExecNode",
907+
"input",
908+
)?;
909+
let proto_sink = sink_node.sink.as_ref().ok_or_else(|| {
910+
datafusion_common::internal_datafusion_err!(
911+
"CsvSinkExecNode is missing required field 'sink'"
912+
)
913+
})?;
914+
let data_sink = CsvSink::try_from(proto_sink)?;
915+
let sort_order = DataSinkExec::decode_sort_order(
916+
sink_node.sort_order.as_ref(),
917+
ctx,
918+
input.schema().as_ref(),
919+
)?;
920+
921+
Ok(Arc::new(DataSinkExec::new(
922+
input,
923+
Arc::new(data_sink),
924+
sort_order,
925+
)))
926+
}
828927
}
829928

830929
#[cfg(test)]

‎datafusion/datasource-json/Cargo.toml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,13 @@ version.workspace = true
3030
[package.metadata.docs.rs]
3131
all-features = true
3232

33+
[features]
34+
proto = [
35+
"dep:datafusion-proto-models",
36+
"datafusion-datasource/proto",
37+
"datafusion-physical-plan/proto",
38+
]
39+
3340
[dependencies]
3441
arrow = { workspace = true }
3542
async-trait = { workspace = true }
@@ -41,6 +48,7 @@ datafusion-execution = { workspace = true }
4148
datafusion-expr = { workspace = true }
4249
datafusion-physical-expr-common = { workspace = true }
4350
datafusion-physical-plan = { workspace = true }
51+
datafusion-proto-models = { workspace = true, optional = true }
4452
datafusion-session = { workspace = true }
4553
futures = { workspace = true }
4654
object_store = { workspace = true }

‎datafusion/datasource-json/src/file_format.rs‎

Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -490,6 +490,105 @@ impl DataSink for JsonSink {
490490
) -> Result<u64> {
491491
FileSink::write_all(self, data, context).await
492492
}
493+
494+
#[cfg(feature = "proto")]
495+
fn try_to_proto(
496+
&self,
497+
exec: &DataSinkExec,
498+
ctx: &datafusion_physical_plan::proto::ExecutionPlanEncodeCtx<'_>,
499+
) -> Result<Option<datafusion_proto_models::protobuf::PhysicalPlanNode>> {
500+
use datafusion_proto_models::protobuf;
501+
use protobuf::physical_plan_node::PhysicalPlanType;
502+
503+
let input = ctx.encode_child(exec.input())?;
504+
let sort_order = exec.encode_sort_order(ctx)?;
505+
let sink = protobuf::JsonSink::try_from(self)?;
506+
let node = protobuf::JsonSinkExecNode {
507+
input: Some(Box::new(input)),
508+
sink: Some(sink),
509+
sink_schema: Some(exec.schema().as_ref().try_into()?),
510+
sort_order,
511+
};
512+
Ok(Some(protobuf::PhysicalPlanNode {
513+
physical_plan_type: Some(PhysicalPlanType::JsonSink(Box::new(node))),
514+
}))
515+
}
516+
}
517+
518+
#[cfg(feature = "proto")]
519+
impl TryFrom<&JsonSink> for datafusion_proto_models::protobuf::JsonSink {
520+
type Error = datafusion_common::DataFusionError;
521+
522+
fn try_from(value: &JsonSink) -> Result<Self> {
523+
Ok(Self {
524+
config: Some(value.config().try_into()?),
525+
writer_options: Some(value.writer_options().try_into()?),
526+
})
527+
}
528+
}
529+
530+
#[cfg(feature = "proto")]
531+
impl TryFrom<&datafusion_proto_models::protobuf::JsonSink> for JsonSink {
532+
type Error = datafusion_common::DataFusionError;
533+
534+
fn try_from(value: &datafusion_proto_models::protobuf::JsonSink) -> Result<Self> {
535+
let config =
536+
FileSinkConfig::try_from(value.config.as_ref().ok_or_else(|| {
537+
datafusion_common::internal_datafusion_err!(
538+
"JsonSink is missing required field 'config'"
539+
)
540+
})?)?;
541+
let writer_options = value
542+
.writer_options
543+
.as_ref()
544+
.ok_or_else(|| {
545+
datafusion_common::internal_datafusion_err!(
546+
"JsonSink is missing required field 'writer_options'"
547+
)
548+
})?
549+
.try_into()?;
550+
551+
Ok(Self::new(config, writer_options))
552+
}
553+
}
554+
555+
#[cfg(feature = "proto")]
556+
impl JsonSink {
557+
/// Reconstructs a [`DataSinkExec`] containing a `JsonSink` from protobuf.
558+
pub fn try_from_proto(
559+
node: &datafusion_proto_models::protobuf::PhysicalPlanNode,
560+
ctx: &datafusion_physical_plan::proto::ExecutionPlanDecodeCtx<'_>,
561+
) -> Result<Arc<dyn ExecutionPlan>> {
562+
use datafusion_proto_models::protobuf;
563+
564+
let sink_node = datafusion_physical_plan::expect_plan_variant!(
565+
node,
566+
protobuf::physical_plan_node::PhysicalPlanType::JsonSink,
567+
"JsonSink",
568+
);
569+
let input = ctx.decode_required_child(
570+
sink_node.input.as_deref(),
571+
"JsonSinkExecNode",
572+
"input",
573+
)?;
574+
let proto_sink = sink_node.sink.as_ref().ok_or_else(|| {
575+
datafusion_common::internal_datafusion_err!(
576+
"JsonSinkExecNode is missing required field 'sink'"
577+
)
578+
})?;
579+
let data_sink = JsonSink::try_from(proto_sink)?;
580+
let sort_order = DataSinkExec::decode_sort_order(
581+
sink_node.sort_order.as_ref(),
582+
ctx,
583+
input.schema().as_ref(),
584+
)?;
585+
586+
Ok(Arc::new(DataSinkExec::new(
587+
input,
588+
Arc::new(data_sink),
589+
sort_order,
590+
)))
591+
}
493592
}
494593

495594
#[derive(Debug)]

‎datafusion/datasource-parquet/Cargo.toml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,7 @@ datafusion-physical-expr = { workspace = true }
4646
datafusion-physical-expr-adapter = { workspace = true }
4747
datafusion-physical-expr-common = { workspace = true }
4848
datafusion-physical-plan = { workspace = true }
49+
datafusion-proto-models = { workspace = true, optional = true }
4950
datafusion-pruning = { workspace = true }
5051
datafusion-session = { workspace = true }
5152
futures = { workspace = true }
@@ -74,6 +75,11 @@ name = "datafusion_datasource_parquet"
7475
path = "src/mod.rs"
7576

7677
[features]
78+
proto = [
79+
"dep:datafusion-proto-models",
80+
"datafusion-datasource/proto",
81+
"datafusion-physical-plan/proto",
82+
]
7783
parquet_encryption = [
7884
"parquet/encryption",
7985
"datafusion-common/parquet_encryption",

0 commit comments

Comments
 (0)