Skip to content

Commit 714956b

Browse files
peterxclialamb
andauthored
fix: preserve overflow behavior and exhaustively destructure expression proto hooks (#25034)
## Which issue does this PR close? Closes #24614. ## Rationale for this change This PR fixes serialization losing the setting that makes arithmetic fail on overflow. For example, checked `Int32::MAX + 1` raises an error before serialization but returns `Int32::MIN` after decoding. Preserving this setting ensures that sending an expression through protobuf preserves its arithmetic behavior. ## What changes are included in this PR? The protobuf message now stores `BinaryExpr::fail_on_overflow`, and both supported decoding formats restore it. When the encoder combines nested expressions into a flat list of operands, it requires their operators and overflow settings to match. This preserves the behavior of expressions that mix checked and wrapping arithmetic. The generated Rust and JSON bindings include the new field. All six encoding and decoding hooks for `BinaryExpr`, `LikeExpr`, and `SqlSimilarToPattern` explicitly list every field without a rest pattern. Adding a field to an expression or its protobuf payload will cause a compile error until the corresponding hook handles it. The PR also changes the PostgreSQL SQLLogicTest decimal formatter to borrow its argument, resolving an existing Clippy error that blocked the required checks before committing. ## What is the testing strategy for this PR? The new tests serialize and decode nested additions, then check their evaluated results for all four combinations of checked and wrapping arithmetic. The regression test failed before the fix because an expression that should raise an overflow error returned `Int32(-2147483648)`. Additional tests cover older messages that omit the new field and verify that JSON preserves the overflow setting. All 17 focused expression tests passed. The extended workspace run passed 11,260 Rust tests, with 8 ignored, and completed all 511 SQLLogicTest files. Both conversion tests passed with the PostgreSQL feature enabled. Formatting, Clippy with all targets and features, and the complete `./dev/rust_lint.sh` suite also passed. ## Are there any user-facing changes? Expressions configured to fail on arithmetic overflow now raise the expected error after serialization and decoding, including nested expressions with different overflow settings. --------- Co-authored-by: Andrew Lamb <[email protected]>
1 parent d42cd85 commit 714956b

7 files changed

Lines changed: 213 additions & 38 deletions

File tree

‎datafusion/physical-expr/src/expressions/binary.rs‎

Lines changed: 38 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -984,19 +984,27 @@ impl PhysicalExpr for BinaryExpr {
984984
) -> Result<Option<datafusion_proto_models::protobuf::PhysicalExprNode>> {
985985
use datafusion_proto_models::protobuf;
986986

987-
// Linearize a nested binary expression tree of the same operator
988-
// into a flat vector of operands to avoid deep recursion in proto.
989-
let op = self.op;
990-
let mut operand_refs: Vec<&Arc<dyn PhysicalExpr>> = vec![&self.right];
991-
let mut current_expr: &BinaryExpr = self;
987+
let Self {
988+
left,
989+
op,
990+
right,
991+
fail_on_overflow,
992+
} = self;
993+
994+
// Linearize a nested binary expression tree with the same operator and
995+
// overflow policy into flat operands to avoid deep recursion in proto.
996+
let mut operand_refs: Vec<&Arc<dyn PhysicalExpr>> = vec![right];
997+
let mut current_left = left;
992998
loop {
993-
match current_expr.left.downcast_ref::<BinaryExpr>() {
994-
Some(bin) if bin.op == op => {
999+
match current_left.downcast_ref::<BinaryExpr>() {
1000+
Some(bin)
1001+
if bin.op == *op && bin.fail_on_overflow == *fail_on_overflow =>
1002+
{
9951003
operand_refs.push(&bin.right);
996-
current_expr = bin;
1004+
current_left = &bin.left;
9971005
}
9981006
_ => {
999-
operand_refs.push(&current_expr.left);
1007+
operand_refs.push(current_left);
10001008
break;
10011009
}
10021010
}
@@ -1014,6 +1022,7 @@ impl PhysicalExpr for BinaryExpr {
10141022
r: None,
10151023
op: format!("{op:?}"),
10161024
operands,
1025+
fail_on_overflow: *fail_on_overflow,
10171026
}),
10181027
)),
10191028
}))
@@ -1045,17 +1054,23 @@ impl BinaryExpr {
10451054
protobuf::physical_expr_node::ExprType::BinaryExpr,
10461055
"BinaryExpr",
10471056
);
1048-
let op = Operator::from_proto_name(&node.op).ok_or_else(|| {
1057+
let protobuf::PhysicalBinaryExprNode {
1058+
l,
1059+
r,
1060+
op,
1061+
operands,
1062+
fail_on_overflow,
1063+
} = node.as_ref();
1064+
let op = Operator::from_proto_name(op).ok_or_else(|| {
10491065
datafusion_common::DataFusionError::Internal(format!(
1050-
"Unsupported binary operator '{}'",
1051-
node.op
1066+
"Unsupported binary operator '{op}'"
10521067
))
10531068
})?;
10541069

1055-
if !node.operands.is_empty() {
1070+
if !operands.is_empty() {
10561071
// New linearized format: reduce the flat operands list back into
10571072
// a nested binary expression tree.
1058-
let operands = ctx.decode_children_expressions(&node.operands)?;
1073+
let operands = ctx.decode_children_expressions(operands)?;
10591074

10601075
if operands.len() < 2 {
10611076
return internal_err!(
@@ -1066,16 +1081,21 @@ impl BinaryExpr {
10661081
Ok(operands
10671082
.into_iter()
10681083
.reduce(|left, right| {
1069-
Arc::new(BinaryExpr::new(left, op, right)) as Arc<dyn PhysicalExpr>
1084+
Arc::new(
1085+
BinaryExpr::new(left, op, right)
1086+
.with_fail_on_overflow(*fail_on_overflow),
1087+
) as Arc<dyn PhysicalExpr>
10701088
})
10711089
.expect("Binary expression could not be reduced to a single expression."))
10721090
} else {
10731091
// Legacy format with l/r fields.
10741092
let left =
1075-
ctx.decode_required_expression(node.l.as_deref(), "BinaryExpr", "left")?;
1093+
ctx.decode_required_expression(l.as_deref(), "BinaryExpr", "left")?;
10761094
let right =
1077-
ctx.decode_required_expression(node.r.as_deref(), "BinaryExpr", "right")?;
1078-
Ok(Arc::new(BinaryExpr::new(left, op, right)))
1095+
ctx.decode_required_expression(r.as_deref(), "BinaryExpr", "right")?;
1096+
Ok(Arc::new(
1097+
BinaryExpr::new(left, op, right).with_fail_on_overflow(*fail_on_overflow),
1098+
))
10791099
}
10801100
}
10811101
}

‎datafusion/physical-expr/src/expressions/like.rs‎

Lines changed: 22 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -153,14 +153,21 @@ impl PhysicalExpr for LikeExpr {
153153
) -> Result<Option<datafusion_proto_models::protobuf::PhysicalExprNode>> {
154154
use datafusion_proto_models::protobuf;
155155

156+
let Self {
157+
negated,
158+
case_insensitive,
159+
expr,
160+
pattern,
161+
} = self;
162+
156163
Ok(Some(protobuf::PhysicalExprNode {
157164
expr_id: None,
158165
expr_type: Some(protobuf::physical_expr_node::ExprType::LikeExpr(Box::new(
159166
protobuf::PhysicalLikeExprNode {
160-
negated: self.negated,
161-
case_insensitive: self.case_insensitive,
162-
expr: Some(Box::new(ctx.encode_child(&self.expr)?)),
163-
pattern: Some(Box::new(ctx.encode_child(&self.pattern)?)),
167+
negated: *negated,
168+
case_insensitive: *case_insensitive,
169+
expr: Some(Box::new(ctx.encode_child(expr)?)),
170+
pattern: Some(Box::new(ctx.encode_child(pattern)?)),
164171
},
165172
))),
166173
}))
@@ -189,19 +196,18 @@ impl LikeExpr {
189196
"LikeExpr",
190197
);
191198

199+
let protobuf::PhysicalLikeExprNode {
200+
negated,
201+
case_insensitive,
202+
expr,
203+
pattern,
204+
} = like_expr.as_ref();
205+
192206
Ok(Arc::new(LikeExpr::new(
193-
like_expr.negated,
194-
like_expr.case_insensitive,
195-
ctx.decode_required_expression(
196-
like_expr.expr.as_deref(),
197-
"LikeExpr",
198-
"expr",
199-
)?,
200-
ctx.decode_required_expression(
201-
like_expr.pattern.as_deref(),
202-
"LikeExpr",
203-
"pattern",
204-
)?,
207+
*negated,
208+
*case_insensitive,
209+
ctx.decode_required_expression(expr.as_deref(), "LikeExpr", "expr")?,
210+
ctx.decode_required_expression(pattern.as_deref(), "LikeExpr", "pattern")?,
205211
)))
206212
}
207213
}

‎datafusion/physical-expr/src/expressions/similar_to_pattern.rs‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -136,11 +136,13 @@ impl PhysicalExpr for SqlSimilarToPattern {
136136
) -> Result<Option<datafusion_proto_models::protobuf::PhysicalExprNode>> {
137137
use datafusion_proto_models::protobuf;
138138

139+
let Self { expr } = self;
140+
139141
Ok(Some(protobuf::PhysicalExprNode {
140142
expr_id: None,
141143
expr_type: Some(protobuf::physical_expr_node::ExprType::SqlSimilarToPattern(
142144
Box::new(protobuf::PhysicalSqlSimilarToPatternNode {
143-
expr: Some(Box::new(ctx.encode_child(&self.expr)?)),
145+
expr: Some(Box::new(ctx.encode_child(expr)?)),
144146
}),
145147
)),
146148
}))
@@ -169,9 +171,11 @@ impl SqlSimilarToPattern {
169171
"SqlSimilarToPattern",
170172
);
171173

174+
let protobuf::PhysicalSqlSimilarToPatternNode { expr } = pattern.as_ref();
175+
172176
Ok(Arc::new(SqlSimilarToPattern::new(
173177
ctx.decode_required_expression(
174-
pattern.expr.as_deref(),
178+
expr.as_deref(),
175179
"SqlSimilarToPattern",
176180
"expr",
177181
)?,

‎datafusion/proto-models/proto/datafusion.proto‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1183,9 +1183,10 @@ message PhysicalBinaryExprNode {
11831183
PhysicalExprNode l = 1;
11841184
PhysicalExprNode r = 2;
11851185
string op = 3;
1186-
// Linearized operands for chains of the same operator (e.g. a AND b AND c).
1186+
// Linearized operands for chains of the same operator and overflow policy.
11871187
// When present, `l` and `r` are ignored and `operands` holds the flattened list.
11881188
repeated PhysicalExprNode operands = 4;
1189+
bool fail_on_overflow = 5;
11891190
}
11901191

11911192
message PhysicalDateTimeIntervalExprNode {

‎datafusion/proto-models/src/generated/pbjson.rs‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18672,6 +18672,9 @@ impl serde::Serialize for PhysicalBinaryExprNode {
1867218672
if !self.operands.is_empty() {
1867318673
len += 1;
1867418674
}
18675+
if self.fail_on_overflow {
18676+
len += 1;
18677+
}
1867518678
let mut struct_ser = serializer.serialize_struct("datafusion.PhysicalBinaryExprNode", len)?;
1867618679
if let Some(v) = self.l.as_ref() {
1867718680
struct_ser.serialize_field("l", v)?;
@@ -18685,6 +18688,9 @@ impl serde::Serialize for PhysicalBinaryExprNode {
1868518688
if !self.operands.is_empty() {
1868618689
struct_ser.serialize_field("operands", &self.operands)?;
1868718690
}
18691+
if self.fail_on_overflow {
18692+
struct_ser.serialize_field("failOnOverflow", &self.fail_on_overflow)?;
18693+
}
1868818694
struct_ser.end()
1868918695
}
1869018696
}
@@ -18699,6 +18705,8 @@ impl<'de> serde::Deserialize<'de> for PhysicalBinaryExprNode {
1869918705
"r",
1870018706
"op",
1870118707
"operands",
18708+
"fail_on_overflow",
18709+
"failOnOverflow",
1870218710
];
1870318711

1870418712
#[allow(clippy::enum_variant_names)]
@@ -18707,6 +18715,7 @@ impl<'de> serde::Deserialize<'de> for PhysicalBinaryExprNode {
1870718715
R,
1870818716
Op,
1870918717
Operands,
18718+
FailOnOverflow,
1871018719
}
1871118720
impl<'de> serde::Deserialize<'de> for GeneratedField {
1871218721
fn deserialize<D>(deserializer: D) -> std::result::Result<GeneratedField, D::Error>
@@ -18732,6 +18741,7 @@ impl<'de> serde::Deserialize<'de> for PhysicalBinaryExprNode {
1873218741
"r" => Ok(GeneratedField::R),
1873318742
"op" => Ok(GeneratedField::Op),
1873418743
"operands" => Ok(GeneratedField::Operands),
18744+
"failOnOverflow" | "fail_on_overflow" => Ok(GeneratedField::FailOnOverflow),
1873518745
_ => Err(serde::de::Error::unknown_field(value, FIELDS)),
1873618746
}
1873718747
}
@@ -18755,6 +18765,7 @@ impl<'de> serde::Deserialize<'de> for PhysicalBinaryExprNode {
1875518765
let mut r__ = None;
1875618766
let mut op__ = None;
1875718767
let mut operands__ = None;
18768+
let mut fail_on_overflow__ = None;
1875818769
while let Some(k) = map_.next_key()? {
1875918770
match k {
1876018771
GeneratedField::L => {
@@ -18781,13 +18792,20 @@ impl<'de> serde::Deserialize<'de> for PhysicalBinaryExprNode {
1878118792
}
1878218793
operands__ = Some(map_.next_value()?);
1878318794
}
18795+
GeneratedField::FailOnOverflow => {
18796+
if fail_on_overflow__.is_some() {
18797+
return Err(serde::de::Error::duplicate_field("failOnOverflow"));
18798+
}
18799+
fail_on_overflow__ = Some(map_.next_value()?);
18800+
}
1878418801
}
1878518802
}
1878618803
Ok(PhysicalBinaryExprNode {
1878718804
l: l__,
1878818805
r: r__,
1878918806
op: op__.unwrap_or_default(),
1879018807
operands: operands__.unwrap_or_default(),
18808+
fail_on_overflow: fail_on_overflow__.unwrap_or_default(),
1879118809
})
1879218810
}
1879318811
}

‎datafusion/proto-models/src/generated/prost.rs‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1833,10 +1833,12 @@ pub struct PhysicalBinaryExprNode {
18331833
pub r: ::core::option::Option<::prost::alloc::boxed::Box<PhysicalExprNode>>,
18341834
#[prost(string, tag = "3")]
18351835
pub op: ::prost::alloc::string::String,
1836-
/// Linearized operands for chains of the same operator (e.g. a AND b AND c).
1836+
/// Linearized operands for chains of the same operator and overflow policy.
18371837
/// When present, `l` and `r` are ignored and `operands` holds the flattened list.
18381838
#[prost(message, repeated, tag = "4")]
18391839
pub operands: ::prost::alloc::vec::Vec<PhysicalExprNode>,
1840+
#[prost(bool, tag = "5")]
1841+
pub fail_on_overflow: bool,
18401842
}
18411843
#[derive(Clone, PartialEq, ::prost::Message)]
18421844
pub struct PhysicalDateTimeIntervalExprNode {

0 commit comments

Comments
 (0)