Skip to content

Commit 0171562

Browse files
committed
docs: note the stack row limit and zero-field inline fallback
1 parent a7a754e commit 0171562

3 files changed

Lines changed: 14 additions & 10 deletions

File tree

‎docs/source/contributor-guide/expression-audits/generator_funcs.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@
4040
## inline
4141

4242
- Same `CometExplodeExec` path as `explode`, with the unnested struct flattened into one column per field. Spark 3.4.3, 3.5.8, 4.0.1, and 4.1.1 share one implementation: a NULL struct element yields a row of NULLs, which the native flattening reproduces by folding the struct's validity into each field. Every generated column is reported nullable.
43+
- An array whose element struct has no fields falls back to Spark. Flattening it would produce a batch with no columns, and such an array carries no data, so the fallback costs nothing.
4344

4445
## inline_outer
4546

@@ -48,5 +49,6 @@
4849
## stack
4950

5051
- Planned as `inline` over an array holding one struct per output row, so `stack(n, v1, ..., vk)` evaluates each argument once per input row, as Spark does, and pads the last row with NULLs. Spark 3.4.3, 3.5.8, 4.0.1, and 4.1.1 share one implementation. An argument that cannot run natively falls back with that argument's own reason.
52+
- A `stack` with more than 50 rows falls back to Spark. This is the bound at which Spark's `Stack.supportCodegen` turns false in every supported version. Above it, the rewrite would build one struct expression per row on the driver and materialize `batch_size * n` structs per input batch before unnesting.
5153

5254
[Spark Expression Support]: ../../user-guide/latest/expressions.md

‎docs/source/user-guide/latest/expressions.md‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -339,18 +339,20 @@ The type-name conversion functions (`bigint`, `binary`, `boolean`, `date`, `deci
339339
`explode`, `explode_outer`, `posexplode`, `posexplode_outer`, `inline`, `inline_outer`, and
340340
`stack` are supported via `CometExplodeExec` (operator-level, not expression-level). `explode` and
341341
`posexplode` support array input; map input falls back to Spark
342-
([#2837](https://github.com/apache/datafusion-comet/issues/2837)). Enabled by default via
342+
([#2837](https://github.com/apache/datafusion-comet/issues/2837)). `inline` and `inline_outer`
343+
over an array of structs with no fields fall back to Spark. `stack` with more than 50 rows falls
344+
back to Spark, the same limit Spark uses for code generation. Enabled by default via
343345
`spark.comet.exec.explode.enabled`.
344346

345347
| Function | Status | Implementation | Notes |
346348
| --- | --- | --- | --- |
347349
| `explode` | ✅ | — | via `CometExplodeExec` |
348350
| `explode_outer` | ✅ | — | via `CometExplodeExec` ([audit](../../contributor-guide/expression-audits/generator_funcs.md#explode_outer)) |
349-
| `inline` | ✅ | — | via `CometExplodeExec` ([audit](../../contributor-guide/expression-audits/generator_funcs.md#inline)) |
350-
| `inline_outer` | ✅ | — | via `CometExplodeExec` ([audit](../../contributor-guide/expression-audits/generator_funcs.md#inline_outer)) |
351+
| `inline` | ✅ | — | via `CometExplodeExec`; structs with no fields fall back ([audit](../../contributor-guide/expression-audits/generator_funcs.md#inline)) |
352+
| `inline_outer` | ✅ | — | via `CometExplodeExec`; structs with no fields fall back ([audit](../../contributor-guide/expression-audits/generator_funcs.md#inline_outer)) |
351353
| `posexplode` | ✅ | — | via `CometExplodeExec` |
352354
| `posexplode_outer` | ✅ | — | via `CometExplodeExec` ([audit](../../contributor-guide/expression-audits/generator_funcs.md#posexplode_outer)) |
353-
| `stack` | ✅ | — | via `CometExplodeExec`, planned as `inline` ([audit](../../contributor-guide/expression-audits/generator_funcs.md#stack)) |
355+
| `stack` | ✅ | — | via `CometExplodeExec`, planned as `inline`; more than 50 rows falls back ([audit](../../contributor-guide/expression-audits/generator_funcs.md#stack)) |
354356

355357
---
356358

‎docs/source/user-guide/latest/operators.md‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -125,12 +125,12 @@ combined with `OR`, which Spark plans as an existence `BroadcastNestedLoopJoinEx
125125

126126
## Generators and set operations
127127

128-
| Operator | Status | Notes |
129-
| -------------- | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
130-
| `GenerateExec` | ✅ | Supports `explode`, `explode_outer`, `posexplode`, `posexplode_outer` over arrays, `inline`, `inline_outer`, and `stack`. Other generators fall back. |
131-
| `ExpandExec` | ✅ | |
132-
| `UnionExec` | ✅ | |
133-
| `CoalesceExec` | ✅ | |
128+
| Operator | Status | Notes |
129+
| -------------- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
130+
| `GenerateExec` | ✅ | Supports `explode`, `explode_outer`, `posexplode`, `posexplode_outer` over arrays, `inline`, `inline_outer`, and `stack`. `inline` over structs with no fields and `stack` with more than 50 rows fall back. Other generators fall back. |
131+
| `ExpandExec` | ✅ | |
132+
| `UnionExec` | ✅ | |
133+
| `CoalesceExec` | ✅ | |
134134

135135
## Writes
136136

0 commit comments

Comments
 (0)