You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit b82f5ea
Browse filesBrowse the repository at this point in the historyBrowse files
feat: add support for comet cartesian join (apache#6938)
* add support for comet cartesian join
* address review: make native cartesian product opt-in (default false)
DataFusion's CrossJoinExec buffers the whole left input and cannot spill, so a
large left can OOM where Spark's spillable CartesianProduct succeeds (sunchao's
P2). Default the flag to false until the native path can spill; add a regression
test that a cross join stays on Spark by default.
Part of apache#199.
* add benchmarks for comet cartesian join
* add benchmarks for comet cartesian join move ests to sql file
* update documentation per review
* update documentation per review
* update documentation abd tests per review
* update documentation per review
-**Typed Dataset operators** (`DeserializeToObjectExec`, `SerializeFromObjectExec`, `MapElementsExec`, `MapPartitionsExec`, `MapGroupsExec`, `CoGroupExec`, `AppendColumnsExec`, and similar): produced by `map`, `mapPartitions`, `groupByKey`, `cogroup`, and other typed `Dataset` transformations. They exist to run user JVM functions on JVM objects, which Comet cannot do natively. The operators above them can still run natively: set `spark.comet.convert.typedDataset.enabled=true` and Comet converts the output of a typed operation to Arrow. This is disabled by default because it can be slower than Spark when the operators above do little work, such as an aggregate over a few groups, which Spark compiles together with the typed operation into one loop.
59
58
@@ -97,12 +96,13 @@ omitted from the tables below and may be reconsidered based on demand:
|`BroadcastHashJoinExec`| ✅ | Falls back to Spark for a `NOT IN` (null-aware anti join) whose streamed side reports an ordering. |
103
-
|`ShuffledHashJoinExec`| ✅ ||
104
-
|`SortMergeJoinExec`| ✅ | Supports `BINARY` join keys. Nested-type (struct, array, map) and collated-string join keys fall back to Spark. |
105
-
|`BroadcastNestedLoopJoinExec`| ✅ | Falls back to Spark when the preserved side is broadcast (for example LEFT OUTER with BROADCAST on the left) ([#4429](https://github.com/apache/datafusion-comet/pull/4429)). |
|`BroadcastHashJoinExec`| ✅ | Falls back to Spark for a `NOT IN` (null-aware anti join) whose streamed side reports an ordering. |
102
+
|`ShuffledHashJoinExec`| ✅ ||
103
+
|`SortMergeJoinExec`| ✅ | Supports `BINARY` join keys. Nested-type (struct, array, map) and collated-string join keys fall back to Spark. |
104
+
|`BroadcastNestedLoopJoinExec`| ✅ | Falls back to Spark when the preserved side is broadcast (for example LEFT OUTER with BROADCAST on the left) ([#4429](https://github.com/apache/datafusion-comet/pull/4429)). |
105
+
|`CartesianProductExec`| ⚠️ | Opt-in (`spark.comet.exec.cartesianProduct.enabled`). Runs natively only without a join condition and when each input reports exactly one partition. Row order can differ from Spark's. |
106
106
107
107
`ExistenceJoin` (produced when `EXISTS` / `IN` is combined with another predicate via `OR`) runs
108
108
natively on `BroadcastHashJoinExec` and `ShuffledHashJoinExec`. Existence sort-merge joins, residual
0 commit comments