You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 0171562
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: docs/source/contributor-guide/expression-audits/generator_funcs.md
+2Lines changed: 2 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -40,6 +40,7 @@
40
40
## inline
41
41
42
42
- Same `CometExplodeExec` path as `explode`, with the unnested struct flattened into one column per field. Spark 3.4.3, 3.5.8, 4.0.1, and 4.1.1 share one implementation: a NULL struct element yields a row of NULLs, which the native flattening reproduces by folding the struct's validity into each field. Every generated column is reported nullable.
43
+
- An array whose element struct has no fields falls back to Spark. Flattening it would produce a batch with no columns, and such an array carries no data, so the fallback costs nothing.
43
44
44
45
## inline_outer
45
46
@@ -48,5 +49,6 @@
48
49
## stack
49
50
50
51
- Planned as `inline` over an array holding one struct per output row, so `stack(n, v1, ..., vk)` evaluates each argument once per input row, as Spark does, and pads the last row with NULLs. Spark 3.4.3, 3.5.8, 4.0.1, and 4.1.1 share one implementation. An argument that cannot run natively falls back with that argument's own reason.
52
+
- A `stack` with more than 50 rows falls back to Spark. This is the bound at which Spark's `Stack.supportCodegen` turns false in every supported version. Above it, the rewrite would build one struct expression per row on the driver and materialize `batch_size * n` structs per input batch before unnesting.
`explode`, `explode_outer`, `posexplode`, `posexplode_outer`, `inline`, `inline_outer`, and
340
340
`stack` are supported via `CometExplodeExec` (operator-level, not expression-level). `explode` and
341
341
`posexplode` support array input; map input falls back to Spark
342
-
([#2837](https://github.com/apache/datafusion-comet/issues/2837)). Enabled by default via
342
+
([#2837](https://github.com/apache/datafusion-comet/issues/2837)). `inline` and `inline_outer`
343
+
over an array of structs with no fields fall back to Spark. `stack` with more than 50 rows falls
344
+
back to Spark, the same limit Spark uses for code generation. Enabled by default via
343
345
`spark.comet.exec.explode.enabled`.
344
346
345
347
| Function | Status | Implementation | Notes |
346
348
| --- | --- | --- | --- |
347
349
|`explode`| ✅ | — | via `CometExplodeExec`|
348
350
|`explode_outer`| ✅ | — | via `CometExplodeExec` ([audit](../../contributor-guide/expression-audits/generator_funcs.md#explode_outer)) |
349
-
|`inline`| ✅ | — | via `CometExplodeExec` ([audit](../../contributor-guide/expression-audits/generator_funcs.md#inline)) |
350
-
|`inline_outer`| ✅ | — | via `CometExplodeExec` ([audit](../../contributor-guide/expression-audits/generator_funcs.md#inline_outer)) |
351
+
|`inline`| ✅ | — | via `CometExplodeExec`; structs with no fields fall back ([audit](../../contributor-guide/expression-audits/generator_funcs.md#inline)) |
352
+
|`inline_outer`| ✅ | — | via `CometExplodeExec`; structs with no fields fall back ([audit](../../contributor-guide/expression-audits/generator_funcs.md#inline_outer)) |
351
353
|`posexplode`| ✅ | — | via `CometExplodeExec`|
352
354
|`posexplode_outer`| ✅ | — | via `CometExplodeExec` ([audit](../../contributor-guide/expression-audits/generator_funcs.md#posexplode_outer)) |
353
-
|`stack`| ✅ | — | via `CometExplodeExec`, planned as `inline` ([audit](../../contributor-guide/expression-audits/generator_funcs.md#stack)) |
355
+
|`stack`| ✅ | — | via `CometExplodeExec`, planned as `inline`; more than 50 rows falls back ([audit](../../contributor-guide/expression-audits/generator_funcs.md#stack)) |
|`GenerateExec`| ✅ | Supports `explode`, `explode_outer`, `posexplode`, `posexplode_outer` over arrays, `inline`, `inline_outer`, and `stack`. Other generators fall back. |
|`GenerateExec`| ✅ | Supports `explode`, `explode_outer`, `posexplode`, `posexplode_outer` over arrays, `inline`, `inline_outer`, and `stack`. `inline` over structs with no fields and `stack` with more than 50 rows fall back. Other generators fall back. |
0 commit comments