Repository navigation
Commit e91b23e
committed
feat: add array_exists with lambda expression support
Add native support for `array_exists(arr, x -> predicate(x))` in SQL
and DataFrame API. This is the first general-purpose lambda expression
infrastructure, which can later be extended to support `array_filter`,
`array_transform`, and `array_forall`.
The lambda body is serialized as a regular expression tree where
`NamedLambdaVariable` leaf nodes are serialized as `LambdaVariable`
proto messages. On the Rust side, `ArrayExistsExpr` evaluates the
lambda body vectorized over all elements in a single pass: it flattens
list values, expands the batch with repeat indices, appends elements
as a `__comet_lambda_var` column, evaluates once, and reduces per row
with SQL three-valued logic semantics.
Unsupported lambda bodies (e.g. containing UDFs) fall back to Spark.
Closes apache#31491 parent a3f7f66 commit e91b23e
10 files changed
Lines changed: 876 additions & 1 deletion
File tree
- native
- core/src/execution
- expressions
- planner
- proto/src/proto
- spark-expr/src/array_funcs
- spark/src
- main/scala/org/apache/comet/serde
- test
- resources/sql-tests/expressions/array
- scala/org/apache/comet
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
| 21 | + | |
21 | 22 | | |
22 | 23 | | |
23 | 24 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
103 | 103 | | |
104 | 104 | | |
105 | 105 | | |
| 106 | + | |
| 107 | + | |
106 | 108 | | |
107 | 109 | | |
108 | 110 | | |
| |||
184 | 186 | | |
185 | 187 | | |
186 | 188 | | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
187 | 192 | | |
188 | 193 | | |
189 | 194 | | |
| |||
306 | 311 | | |
307 | 312 | | |
308 | 313 | | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
309 | 326 | | |
310 | 327 | | |
311 | 328 | | |
| |||
370 | 387 | | |
371 | 388 | | |
372 | 389 | | |
| 390 | + | |
| 391 | + | |
373 | 392 | | |
374 | 393 | | |
375 | 394 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
88 | 88 | | |
89 | 89 | | |
90 | 90 | | |
| 91 | + | |
| 92 | + | |
91 | 93 | | |
92 | 94 | | |
93 | 95 | | |
| |||
440 | 442 | | |
441 | 443 | | |
442 | 444 | | |
| 445 | + | |
| 446 | + | |
| 447 | + | |
| 448 | + | |
| 449 | + | |
| 450 | + | |
| 451 | + | |
| 452 | + | |
| 453 | + | |
| 454 | + | |
| 455 | + | |
0 commit comments