Skip to content

Commit 5b0705c

Browse files
committed
perf: keep CASE WHEN eager when a branch compares floats
`is_infallible` in `case_when.rs` accepted the `BinaryExpr` of normalized operands that a float comparison used to be, but not the `SparkComparison` that replaced it. A CASE WHEN or IF with a float comparison in a later WHEN or in a branch value therefore fell back from eager to lazy evaluation. `SparkComparison::is_infallible` now reports a comparison of two operands of the same type as unable to fail, which covers every flat Float32 or Float64 comparison. The nested comparators reject a dictionary-encoded leaf against a plain one, which the logical type check in `spark_comparison` lets through, so a nested comparison counts only when the types are equal. The `is_infallible` test builds comparisons through `spark_comparison`, so a change in what the planner builds shows up there. The contributor guide describes comparing in place and lists the float comparison kernels.
1 parent 1579aae commit 5b0705c

3 files changed

Lines changed: 57 additions & 10 deletions

File tree

‎docs/source/contributor-guide/floating_point.md‎

Lines changed: 14 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -92,6 +92,7 @@ Its module documentation is the reference. In short:
9292
| Helper | Rule | Use it to |
9393
| ---------------------------------------------------------------- | ---------------------------- | ----------------------------------------------------------------------------------------------------------- |
9494
| `compare_floats`, `float_lt`, `float_gt` | SQL ordering | Compare values in a kernel that returns the original bits, as `array_min` and `greatest` do |
95+
| `compare_float_arrays`, `compare_float_array_scalar` | SQL ordering | Compare a `FLOAT` or `DOUBLE` array with another or with a scalar in place, as `SparkComparison` does |
9596
| `spark_comparator`, `spark_equality` | SQL ordering, at any depth | Compare arrays and structs in place |
9697
| `normalize_float`, `normalize_floats`, `normalize_nested_floats` | `NormalizeNaNAndZero` | Normalize values before an Arrow kernel sorts, row-encodes, hashes or compares them |
9798
| `NormalizeNaNAndZero`, `NormalizeNestedFloats` | `NormalizeNaNAndZero` | Wrap a key or operand expression. `wrap_if_needed` skips other types and keys that Spark already normalized |
@@ -112,15 +113,19 @@ normalized and the output keeps the original values.
112113
sort orders them the Spark way.
113114
- `spark_comparison` in `native/spark-expr/src/array_funcs/nested_comparison.rs` builds every native
114115
`=`, `<>`, `<=>`, `<`, `<=`, `>`, `>=` and `IS DISTINCT FROM`, whichever operator evaluates it. It
115-
normalizes float operands, and folds a literal while the plan is built so that it stays a
116-
literal. Nested `=` and `<>` compare in place with `spark_equality`.
117-
- A scan's pushed-down data filters leave a float column compared with a literal other than NaN
118-
unwrapped (`FloatOperands::Raw`) so that Parquet pruning still recognizes it, but only while the
119-
reader prunes with them without filtering rows. A bloom filter probe hashes the literal's bits, so
120-
`=` against either zero becomes `= -0.0 OR = 0.0`. With
121-
`spark.comet.parquet.rowFilterPushdown.enabled=true` they normalize both sides, because the
122-
reader drops the rows a filter rejects, and a raw column would reject a stored NaN that Spark
123-
matches ([#6702](https://github.com/apache/datafusion-comet/issues/6702)). Spark's own reader
116+
compares the operands as they are, without normalized copies: `FLOAT` and `DOUBLE` with
117+
`compare_float_arrays` and `compare_float_array_scalar`, and arrays and structs with
118+
`spark_comparator`, or with `spark_equality` for `=`, `<>`, `<=>` and `IS DISTINCT FROM`. It
119+
normalizes a literal while the plan is built so that it stays a literal. Operands of other types,
120+
such as dictionary-encoded floats, are normalized and compared with Arrow's kernels.
121+
- A scan's pushed-down data filters keep a float column compared with a literal other than NaN as a
122+
plain `BinaryExpr` (`FloatOperands::Raw`) so that Parquet pruning still recognizes it, but only
123+
while the reader prunes with them without filtering rows. A bloom filter probe hashes the
124+
literal's bits, so `=` against either zero becomes `= -0.0 OR = 0.0`. With
125+
`spark.comet.parquet.rowFilterPushdown.enabled=true` they compare in Spark's ordering like every
126+
other comparison, because the reader drops the rows a filter rejects, and a raw column would
127+
reject a stored NaN that Spark matches
128+
([#6702](https://github.com/apache/datafusion-comet/issues/6702)). Spark's own reader
124129
keeps the two zeros apart in its dictionary and bloom filters, so it can skip a row group that
125130
Comet reads. A test of this pruning writes out the expected rows instead of comparing them with
126131
Spark's.

‎native/spark-expr/src/array_funcs/nested_comparison.rs‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -335,6 +335,20 @@ impl SparkComparison {
335335
pub fn right(&self) -> &Arc<dyn PhysicalExpr> {
336336
&self.right
337337
}
338+
339+
/// Whether the comparison cannot fail for any value: when both operands have the same type.
340+
/// The float kernels and the nested comparators fail only on operands of different types,
341+
/// such as a dictionary-encoded leaf against a plain one, which the logical type check in
342+
/// [`spark_comparison`] lets through for lists and structs.
343+
pub(crate) fn is_infallible(&self, input_schema: &Schema) -> bool {
344+
match (
345+
self.left.data_type(input_schema),
346+
self.right.data_type(input_schema),
347+
) {
348+
(Ok(left), Ok(right)) => left == right,
349+
_ => false,
350+
}
351+
}
338352
}
339353

340354
impl PartialEq for SparkComparison {

‎native/spark-expr/src/conditional_funcs/case_when.rs‎

Lines changed: 29 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@
1515
// specific language governing permissions and limitations
1616
// under the License.
1717

18-
use crate::{Cast, EvalMode, IfExpr, NormalizeNaNAndZero, SparkCastOptions};
18+
use crate::{Cast, EvalMode, IfExpr, NormalizeNaNAndZero, SparkCastOptions, SparkComparison};
1919
use arrow::array::{
2020
downcast_primitive, Array, ArrayRef, AsArray, BooleanArray, GenericByteArray, PrimitiveArray,
2121
};
@@ -396,6 +396,8 @@ fn is_infallible(expr: &Arc<dyn PhysicalExpr>, input_schema: &Schema) -> bool {
396396
}
397397
let node_is_infallible = if let Some(binary) = expr.downcast_ref::<BinaryExpr>() {
398398
binary_is_infallible(binary, input_schema)
399+
} else if let Some(comparison) = expr.downcast_ref::<SparkComparison>() {
400+
comparison.is_infallible(input_schema)
399401
} else if let Some(cast) = expr.downcast_ref::<Cast>() {
400402
cast.is_infallible(input_schema)
401403
} else if let Some(normalize) = expr.downcast_ref::<NormalizeNaNAndZero>() {
@@ -836,6 +838,7 @@ fn merge_nulls(num_rows: usize, branches: &[Branch]) -> Option<NullBuffer> {
836838
#[cfg(test)]
837839
mod tests {
838840
use super::*;
841+
use crate::{spark_comparison, FloatOperands};
839842
use arrow::array::{
840843
BinaryArray, Int32Array, Int64Array, LargeStringArray, StringArray, StructArray,
841844
TimestampMicrosecondArray,
@@ -1137,12 +1140,24 @@ mod tests {
11371140

11381141
#[test]
11391142
fn infallible_expressions() {
1143+
let list_of = |item: DataType| DataType::List(Arc::new(Field::new("item", item, true)));
11401144
let schema = Schema::new(vec![
11411145
Field::new("i", DataType::Int64, true),
11421146
Field::new("j", DataType::Int32, true),
11431147
Field::new("d", DataType::Float64, true),
1148+
Field::new("e", DataType::Float64, true),
11441149
Field::new("s", DataType::Utf8, true),
11451150
Field::new("dec", DataType::Decimal128(10, 2), true),
1151+
Field::new("l", list_of(DataType::Float64), true),
1152+
Field::new("m", list_of(DataType::Float64), true),
1153+
Field::new(
1154+
"dictionary",
1155+
list_of(DataType::Dictionary(
1156+
Box::new(DataType::Int32),
1157+
Box::new(DataType::Float64),
1158+
)),
1159+
true,
1160+
),
11461161
]);
11471162
let c = |name: &str| col(name, &schema).unwrap();
11481163
let cast = |e: Arc<dyn PhysicalExpr>, to: DataType| -> Arc<dyn PhysicalExpr> {
@@ -1155,6 +1170,10 @@ mod tests {
11551170
))
11561171
};
11571172
let infallible = |e: Arc<dyn PhysicalExpr>| is_infallible(&e, &schema);
1173+
// A comparison as the planner builds it, which follows Spark's ordering for floats
1174+
let compare = |left: &str, op: Operator, right: Arc<dyn PhysicalExpr>| {
1175+
spark_comparison(c(left), op, right, &schema, FloatOperands::Normalize).unwrap()
1176+
};
11581177

11591178
assert!(infallible(c("i")));
11601179
assert!(infallible(lit("x")));
@@ -1181,13 +1200,22 @@ mod tests {
11811200
DataType::Float64,
11821201
c("d")
11831202
))));
1203+
for op in [Operator::Lt, Operator::Eq, Operator::IsNotDistinctFrom] {
1204+
assert!(infallible(compare("d", op, lit(1.5))), "d {op} 1.5");
1205+
assert!(infallible(compare("d", op, c("e"))), "d {op} e");
1206+
}
1207+
for op in [Operator::Lt, Operator::IsNotDistinctFrom] {
1208+
assert!(infallible(compare("l", op, c("m"))), "l {op} m");
1209+
}
11841210

11851211
// Only a boolean can be negated, and only a float normalized
11861212
assert!(!infallible(Arc::new(NotExpr::new(c("i")))));
11871213
assert!(!infallible(Arc::new(NormalizeNaNAndZero::new(
11881214
DataType::Int64,
11891215
c("i")
11901216
))));
1217+
// A nested comparison fails on a dictionary-encoded leaf against a plain one
1218+
assert!(!infallible(compare("l", Operator::Lt, c("dictionary"))));
11911219
// Integer division and remainder fail on a zero divisor
11921220
assert!(!infallible(binary(c("i"), Operator::Divide, c("i"))));
11931221
assert!(!infallible(binary(c("i"), Operator::Modulo, c("i"))));

0 commit comments

Comments
 (0)