You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 070951d
Browse filesBrowse the repository at this point in the historyBrowse files
feat: convert OneRowRelation to Arrow by default (apache#6666)
* feat: convert OneRowRelation to Arrow by default
Turn on spark.comet.convert.oneRowRelation.enabled by default. The row a
query without a FROM clause reads has no columns, so converting it costs
almost nothing, and it lets the operators above it run in Comet. Where
the parent cannot run in Comet, EliminateRedundantTransitions drops the
conversion again, so no extra transition is added.
On Spark 4.2 the TPC-DS q77a golden now records the remaining blocker,
the struct-typed scalar subquery result the projection above each
OneRowRelation reads (apache#5834), instead of the unsupported leaf.
* fix: handle expressions exposed by converting OneRowRelation by default
Folded array literals are UnsafeArrayData and have no backing array, so
serialize them through toArray. might_contain accepts an untyped NULL
for either argument. parse_url raises Spark's INVALID_URL error under
ANSI instead of a generic native error. An Expand with no output columns
falls back to Spark.
* fix: decline codegen dispatch for expressions that contain a subquery
* test: adapt Spark explain tests to the Comet plan for OneRowRelation queries
Copy file name to clipboardExpand all lines: docs/source/user-guide/latest/compatibility/spark-versions.md
+7-5Lines changed: 7 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -89,11 +89,13 @@ Spark 4.2.0 is supported with Java 17 and Scala 2.13.
89
89
-**`FILTER` on window aggregates**: Spark 4.2 accepts a `FILTER (WHERE ...)` clause on an
90
90
aggregate function used as a window function. Comet does not support the clause there, so the
91
91
window operator falls back to Spark.
92
-
-**`OneRowRelation` in `UNION` branches**
93
-
([#4949](https://github.com/apache/datafusion-comet/issues/4949)): Spark 4.2 plans a
94
-
`OneRowRelation` into the `UNION` branches of some queries, for example TPC-DS q77a. Comet does
95
-
not convert `OneRowRelation` by default, so the union and the aggregates above it fall back to
96
-
Spark.
92
+
-**Merged subqueries in `UNION` branches**
93
+
([#4949](https://github.com/apache/datafusion-comet/issues/4949)): Spark 4.2 turns a single-row
94
+
aggregate in some queries, for example TPC-DS q77a, into a scalar subquery that returns a struct,
95
+
and reads its fields in a projection over a `OneRowRelation`. Comet does not support a
96
+
struct-typed scalar subquery result yet
97
+
([#5834](https://github.com/apache/datafusion-comet/issues/5834)), so that projection, and the
98
+
union and the aggregates above it, fall back to Spark.
97
99
-**ANSI arithmetic overflow**
98
100
([#4967](https://github.com/apache/datafusion-comet/issues/4967)): Spark 4.2 changed some
99
101
ANSI-mode arithmetic overflow behavior, and Comet does not match these changes yet. See
0 commit comments