You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 9e54913
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: docs/source/contributor-guide/adding_a_new_expression.md
+1Lines changed: 1 addition & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -408,6 +408,7 @@ For full documentation on the test file format, including directives like `Confi
408
408
409
409
-**Cover both column references and literals.** Comet often uses different code paths for each. The Comet SQL Tests suite automatically disables constant folding, so all-literal queries are evaluated natively.
410
410
-**Include edge cases** such as `NULL`, empty strings, boundary values, `NaN`, and multibyte UTF-8 characters.
411
+
-**For `FLOAT` and `DOUBLE` input, include `-0.0` and a NaN with the sign bit set.**[Floating-Point Semantics](floating_point.md) explains which Spark rule an expression follows and how to test it.
411
412
-**Keep one file per expression** to make failures easy to locate.
412
413
-**Pin the mechanism when it depends on the input.** If your serde routes some input types natively and others through the JVM codegen dispatcher, say so with `expect_native(...)` and `expect_dispatch(...)`. A plain `query` cannot tell the two apart, so without these a later change that swaps one for the other passes silently. See [Comet SQL Tests](sql-file-tests.md).
Copy file name to clipboardExpand all lines: docs/source/user-guide/latest/compatibility/floating-point.md
+30-7Lines changed: 30 additions & 7 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -23,20 +23,43 @@ Spark normalizes NaN and zero for floating point numbers for several cases. See
23
23
However, one exception is comparison. Spark does not normalize NaN and zero when comparing values
24
24
because they are handled well in Spark (e.g., `SQLOrderingUtil.compareFloats`). But the comparison
25
25
functions of arrow-rs used by DataFusion do not normalize NaN and zero (e.g., [arrow::compute::kernels::cmp::eq](https://docs.rs/arrow/latest/arrow/compute/kernels/cmp/fn.eq.html#)).
26
-
For top-level `FLOAT` and `DOUBLE` comparisons, Comet normalizes both operands before native
27
-
execution, including noncanonical NaN literals. Top-level `IN`, `InSet`, and `NOT IN` membership
28
-
also normalize dynamic candidates and lists containing NaN. When every candidate is a non-NaN
29
-
literal, Comet keeps DataFusion's static filter and pruning path, enumerating both signed-zero
30
-
forms when a list contains zero.
26
+
For `FLOAT` and `DOUBLE` comparisons (`=`, `<>`, `<=>`, `<`, `<=`, `>` and `>=`), Comet
27
+
normalizes both operands before native execution, including noncanonical NaN literals. This
28
+
applies wherever a comparison appears: projections, filters, aggregate arguments and `FILTER`
29
+
clauses, join conditions, sort keys, and generator arguments.
30
+
31
+
A native Parquet scan prunes row groups and pages with its data filters. So that this pruning
32
+
still applies, a `FLOAT` or `DOUBLE` column compared with a constant other than NaN in a data
33
+
filter is compared without normalizing the column, and the filter above the scan evaluates the
34
+
comparison again with Spark's semantics. Every other comparison in a data filter is normalized.
35
+
Bloom filters store `-0.0` and `0.0` separately, so `=` against either zero checks them for both.
36
+
With `spark.comet.parquet.rowFilterPushdown.enabled=true` the scan also drops the rows its data
37
+
filters reject, so every comparison in a data filter is normalized, and a `FLOAT` or `DOUBLE`
38
+
comparison in a data filter does not prune row groups or pages
0 commit comments