Skip to content

Commit 4d95615

Browse files
mattp5657Matthew Patton
authored andcommitted
refactor: Migrate ScalarSubqueryExpr to self-serialization proto pattern (apache#23130)
Move ScalarSubqueryExpr proto encode/decode into the expression itself via the try_to_proto / try_from_proto hooks, removing the centralized downcast branch in to_proto.rs and the inline construction in from_proto.rs. Because the ScalarSubqueryResults container is runtime-only shared state and not part of the wire format, from_proto.rs still fetches it from the decode context and threads it into try_from_proto. Follows the pattern established in apache#21929. ## Which issue does this PR close? - Closes #[22433](apache#22433). ## Rationale for this change datafusion-proto serializes ScalarSubqueryExpr by downcasting in to_proto.rs and rebuilding it inline in from_proto.rs, which forces its internal fields to be exposed as public accessors just for the proto crate. PR #[21929](apache#21929) introduced a self-serialization pattern (try_to_proto / try_from_proto) where each expression owns its own wire format and the central dispatch just delegates. Other expressions like InListExpr and DynamicFilterPhysicalExpr already use it. This change moves ScalarSubqueryExpr to the same pattern, removing its bespoke proto branches and the proto-only public accessors. One wrinkle: ScalarSubqueryExpr's ScalarSubqueryResults is runtime-only shared state, not part of the wire format, so try_from_proto takes it as an extra argument that from_proto.rs threads in from the decode context. ## What changes are included in this PR? - [x] `try_to_proto` in `impl PhysicalExpr for ScalarSubqueryExpr` (not inherent), `#[cfg(feature = "proto")]` - [x] `ScalarSubqueryExpr::try_from_proto` wired into `from_proto.rs` - [x] central `to_proto.rs` + `from_proto.rs` arms deleted in this PR - [x] direct hook test incl. bad-input case; test module at end of file - [x] roundtrip tests pass; fmt + clippy `-D warnings` clean; PR template filled in ## Are these changes tested? - **Unit tests** (`scalar_subquery.rs`, `#[cfg(all(test, feature = "proto"))]` module at end of file) exercise the hooks directly: - `round_trips_through_proto` — encode via `try_to_proto`, decode via `try_from_proto`, asserting `data_type`/`nullable`/`index` and shared-results identity survive. - `rejects_non_scalar_subquery_node` — wrong `ExprType` variant errors cleanly. - `rejects_missing_data_type` — missing required field errors cleanly. - **Round-trip integration tests** (`datafusion-proto`, `--all-features`) cover the full plan path: `roundtrip_scalar_subquery_exec`, `roundtrip_nested_scalar_subquery_exec_scopes_results`, and `roundtrip_scalar_subquery_exec_with_default_converter_executes`. All pass, and `cargo fmt --all` + `cargo clippy --all-features --all-targets -- -D warnings` are clean. ## Are there any user-facing changes? No --------- Co-authored-by: Matthew Patton <[email protected]>
1 parent 89dd04e commit 4d95615

3 files changed

Lines changed: 201 additions & 31 deletions

File tree

‎datafusion/physical-expr/src/scalar_subquery.rs‎

Lines changed: 199 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -59,22 +59,34 @@ impl ScalarSubqueryExpr {
5959
}
6060
}
6161

62+
pub fn results(&self) -> &ScalarSubqueryResults {
63+
&self.results
64+
}
65+
66+
#[deprecated(
67+
since = "55.0.0",
68+
note = "was only used for proto serialization, which no longer needs it; use `return_field` for type/nullability. It will be removed in 61.0.0 or 6 months after 55.0.0 is released, whichever is longer."
69+
)]
6270
pub fn data_type(&self) -> &DataType {
6371
&self.data_type
6472
}
6573

74+
#[deprecated(
75+
since = "55.0.0",
76+
note = "was only used for proto serialization, which no longer needs it; use `return_field` for type/nullability. It will be removed in 61.0.0 or 6 months after 55.0.0 is released, whichever is longer."
77+
)]
6678
pub fn nullable(&self) -> bool {
6779
self.nullable
6880
}
6981

7082
/// Returns the index of this subquery in the shared results container.
83+
#[deprecated(
84+
since = "55.0.0",
85+
note = "was only used for proto serialization, which no longer needs it. It will be removed in 61.0.0 or 6 months after 55.0.0 is released, whichever is longer."
86+
)]
7187
pub fn index(&self) -> SubqueryIndex {
7288
self.index
7389
}
74-
75-
pub fn results(&self) -> &ScalarSubqueryResults {
76-
&self.results
77-
}
7890
}
7991

8092
impl fmt::Display for ScalarSubqueryExpr {
@@ -139,6 +151,69 @@ impl PhysicalExpr for ScalarSubqueryExpr {
139151
fn fmt_sql(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
140152
write!(f, "(scalar subquery)")
141153
}
154+
155+
#[cfg(feature = "proto")]
156+
fn try_to_proto(
157+
&self,
158+
_ctx: &datafusion_physical_expr_common::physical_expr::proto_encode::PhysicalExprEncodeCtx<'_>,
159+
) -> Result<Option<datafusion_proto_models::protobuf::PhysicalExprNode>> {
160+
use datafusion_proto_models::protobuf;
161+
Ok(Some(protobuf::PhysicalExprNode {
162+
expr_id: None,
163+
expr_type: Some(protobuf::physical_expr_node::ExprType::ScalarSubquery(
164+
protobuf::PhysicalScalarSubqueryExprNode {
165+
data_type: Some((&self.data_type).try_into()?),
166+
nullable: self.nullable,
167+
index: u32::try_from(self.index.as_usize()).map_err(|_| {
168+
internal_datafusion_err!(
169+
"scalar subquery index {} does not fit in u32",
170+
self.index.as_usize()
171+
)
172+
})?,
173+
},
174+
)),
175+
}))
176+
}
177+
}
178+
179+
#[cfg(feature = "proto")]
180+
impl ScalarSubqueryExpr {
181+
/// Reconstruct a [`ScalarSubqueryExpr`] from its protobuf representation.
182+
///
183+
/// Unlike other expressions, this takes a third argument: the shared
184+
/// [`ScalarSubqueryResults`] container. That container is a runtime-only
185+
/// `Arc` shared with the surrounding `ScalarSubqueryExec` and is not part of
186+
/// the wire format, so it cannot be reconstructed here or carried on the
187+
/// decode context (which lives in a crate that cannot depend on
188+
/// `datafusion-expr`). The match arm in `from_proto.rs` fetches it from the
189+
/// plan-level decode context and passes it in.
190+
pub fn try_from_proto(
191+
node: &datafusion_proto_models::protobuf::PhysicalExprNode,
192+
_ctx: &datafusion_physical_expr_common::physical_expr::proto_decode::PhysicalExprDecodeCtx<'_>,
193+
results: &ScalarSubqueryResults,
194+
) -> Result<Arc<dyn PhysicalExpr>> {
195+
use datafusion_physical_expr_common::expect_expr_variant;
196+
use datafusion_physical_expr_common::physical_expr::proto_decode::require_proto_field;
197+
use datafusion_proto_models::protobuf;
198+
199+
let sq = expect_expr_variant!(
200+
node,
201+
protobuf::physical_expr_node::ExprType::ScalarSubquery,
202+
"ScalarSubqueryExpr",
203+
);
204+
let data_type = require_proto_field(
205+
sq.data_type.as_ref(),
206+
"ScalarSubqueryExpr",
207+
"data_type",
208+
)?
209+
.try_into()?;
210+
Ok(Arc::new(ScalarSubqueryExpr::new(
211+
data_type,
212+
sq.nullable,
213+
SubqueryIndex::new(sq.index as usize),
214+
results.clone(),
215+
)))
216+
}
142217
}
143218

144219
#[cfg(test)]
@@ -238,3 +313,123 @@ mod tests {
238313
assert_ne!(e1a, e3);
239314
}
240315
}
316+
317+
/// Tests for the `try_to_proto` / `try_from_proto` hooks.
318+
#[cfg(all(test, feature = "proto"))]
319+
mod proto_tests {
320+
use super::*;
321+
use crate::proto_test_util::{StubEncoder, UnreachableDecoder, column_node};
322+
use datafusion_common::DataFusionError;
323+
use datafusion_physical_expr_common::physical_expr::proto_decode::PhysicalExprDecodeCtx;
324+
use datafusion_physical_expr_common::physical_expr::proto_encode::PhysicalExprEncodeCtx;
325+
use datafusion_proto_models::protobuf::{
326+
PhysicalExprNode, PhysicalScalarSubqueryExprNode, physical_expr_node,
327+
};
328+
329+
/// Build a `ScalarSubquery` proto node directly, with control over each
330+
/// field, so the decode error paths can be exercised independently.
331+
fn proto_scalar_subquery_node(
332+
data_type: Option<datafusion_proto_models::datafusion_common::ArrowType>,
333+
nullable: bool,
334+
index: u32,
335+
) -> PhysicalExprNode {
336+
PhysicalExprNode {
337+
expr_id: None,
338+
expr_type: Some(physical_expr_node::ExprType::ScalarSubquery(
339+
PhysicalScalarSubqueryExprNode {
340+
data_type,
341+
nullable,
342+
index,
343+
},
344+
)),
345+
}
346+
}
347+
348+
#[test]
349+
fn round_trips_through_proto() {
350+
// A three-slot results container so index 2 is meaningful.
351+
let results = ScalarSubqueryResults::new(3);
352+
let expr = ScalarSubqueryExpr::new(
353+
DataType::Int32,
354+
true,
355+
SubqueryIndex::new(2),
356+
results.clone(),
357+
);
358+
359+
// Encode: the expression serializes itself via try_to_proto.
360+
let encoder = StubEncoder::ok();
361+
let enc_ctx = PhysicalExprEncodeCtx::new(&encoder);
362+
let node = expr
363+
.try_to_proto(&enc_ctx)
364+
.unwrap()
365+
.expect("ScalarSubqueryExpr should encode to Some(node)");
366+
367+
assert!(node.expr_id.is_none());
368+
let sq = match &node.expr_type {
369+
Some(physical_expr_node::ExprType::ScalarSubquery(sq)) => sq,
370+
other => panic!("expected a ScalarSubquery node, got {other:?}"),
371+
};
372+
assert!(sq.nullable);
373+
assert_eq!(sq.index, 2);
374+
let encoded_type: DataType = sq
375+
.data_type
376+
.as_ref()
377+
.expect("data_type encoded")
378+
.try_into()
379+
.unwrap();
380+
assert_eq!(encoded_type, DataType::Int32);
381+
382+
// Decode: reconstruct from the proto node, threading in the shared
383+
// results container the surrounding exec would provide.
384+
let decoder = UnreachableDecoder;
385+
let schema = Schema::empty();
386+
let dec_ctx = PhysicalExprDecodeCtx::new(&schema, &decoder);
387+
let decoded =
388+
ScalarSubqueryExpr::try_from_proto(&node, &dec_ctx, &results).unwrap();
389+
let decoded = decoded
390+
.downcast_ref::<ScalarSubqueryExpr>()
391+
.expect("decoded expr should be a ScalarSubqueryExpr");
392+
393+
// data_type + nullable survive the round-trip (observed via return_field).
394+
let field = decoded.return_field(&Schema::empty()).unwrap();
395+
assert_eq!(field.data_type(), &DataType::Int32);
396+
assert!(field.is_nullable());
397+
398+
// Same shared container + same index → equal to the original.
399+
assert_eq!(decoded, &expr);
400+
}
401+
402+
#[test]
403+
fn rejects_non_scalar_subquery_node() {
404+
let node = column_node("a");
405+
let results = ScalarSubqueryResults::new(1);
406+
let decoder = UnreachableDecoder;
407+
let schema = Schema::empty();
408+
let dec_ctx = PhysicalExprDecodeCtx::new(&schema, &decoder);
409+
410+
let err =
411+
ScalarSubqueryExpr::try_from_proto(&node, &dec_ctx, &results).unwrap_err();
412+
assert!(matches!(
413+
err,
414+
DataFusionError::Internal(msg)
415+
if msg.contains("PhysicalExprNode is not a ScalarSubqueryExpr")
416+
));
417+
}
418+
419+
#[test]
420+
fn rejects_missing_data_type() {
421+
let node = proto_scalar_subquery_node(None, false, 0);
422+
let results = ScalarSubqueryResults::new(1);
423+
let decoder = UnreachableDecoder;
424+
let schema = Schema::empty();
425+
let dec_ctx = PhysicalExprDecodeCtx::new(&schema, &decoder);
426+
427+
let err =
428+
ScalarSubqueryExpr::try_from_proto(&node, &dec_ctx, &results).unwrap_err();
429+
assert!(matches!(
430+
err,
431+
DataFusionError::Internal(msg)
432+
if msg.contains("ScalarSubqueryExpr is missing required field 'data_type'")
433+
));
434+
}
435+
}

‎datafusion/proto/src/physical_plan/from_proto.rs‎

Lines changed: 2 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,6 @@ use datafusion_execution::object_store::ObjectStoreUrl;
4242
use datafusion_execution::{FunctionRegistry, TaskContext};
4343
use datafusion_expr::WindowFunctionDefinition;
4444
use datafusion_expr::dml::InsertOp;
45-
use datafusion_expr::execution_props::SubqueryIndex;
4645
use datafusion_physical_expr::expressions::{LambdaExpr, LambdaVariable};
4746
use datafusion_physical_expr::projection::{ProjectionExpr, ProjectionExprs};
4847
use datafusion_physical_expr::scalar_subquery::ScalarSubqueryExpr;
@@ -366,26 +365,14 @@ pub fn parse_physical_expr_with_converter(
366365
}
367366
ExprType::LikeExpr(_) => LikeExpr::try_from_proto(proto, &decode_ctx)?,
368367
ExprType::HashExpr(_) => HashExpr::try_from_proto(proto, &decode_ctx)?,
369-
ExprType::ScalarSubquery(sq) => {
370-
let data_type: arrow::datatypes::DataType = sq
371-
.data_type
372-
.as_ref()
373-
.ok_or_else(|| {
374-
proto_error("Missing data_type in PhysicalScalarSubqueryExprNode")
375-
})?
376-
.try_into()?;
368+
ExprType::ScalarSubquery(_) => {
377369
let results = ctx.scalar_subquery_results().ok_or_else(|| {
378370
proto_error(
379371
"ScalarSubqueryExpr can only be deserialized as part \
380372
of a surrounding ScalarSubqueryExec",
381373
)
382374
})?;
383-
Arc::new(ScalarSubqueryExpr::new(
384-
data_type,
385-
sq.nullable,
386-
SubqueryIndex::new(sq.index as usize),
387-
results.clone(),
388-
))
375+
ScalarSubqueryExpr::try_from_proto(proto, &decode_ctx, results)?
389376
}
390377
ExprType::DynamicFilter(_) => {
391378
DynamicFilterPhysicalExpr::try_from_proto(proto, &decode_ctx)?

‎datafusion/proto/src/physical_plan/to_proto.rs‎

Lines changed: 0 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,6 @@ use datafusion_datasource_json::file_format::JsonSink;
3131
#[cfg(feature = "parquet")]
3232
use datafusion_datasource_parquet::file_format::ParquetSink;
3333
use datafusion_expr::WindowFrame;
34-
use datafusion_physical_expr::scalar_subquery::ScalarSubqueryExpr;
3534
use datafusion_physical_expr::window::{SlidingAggregateWindowExpr, StandardWindowExpr};
3635
use datafusion_physical_expr::{HigherOrderFunctionExpr, ScalarFunctionExpr};
3736
use datafusion_physical_expr_common::sort_expr::PhysicalSortExpr;
@@ -331,17 +330,6 @@ pub fn serialize_physical_expr_with_converter(
331330
},
332331
)),
333332
})
334-
} else if let Some(expr) = expr.downcast_ref::<ScalarSubqueryExpr>() {
335-
Ok(protobuf::PhysicalExprNode {
336-
expr_id,
337-
expr_type: Some(protobuf::physical_expr_node::ExprType::ScalarSubquery(
338-
protobuf::PhysicalScalarSubqueryExprNode {
339-
data_type: Some(expr.data_type().try_into()?),
340-
nullable: expr.nullable(),
341-
index: expr.index().as_usize() as u32,
342-
},
343-
)),
344-
})
345333
} else {
346334
let mut buf: Vec<u8> = vec![];
347335
match codec.try_encode_expr(value, &mut buf) {

0 commit comments

Comments
 (0)