Repository navigation
Commit 22518be
fix: preserve the input list's inner field in array_append/prepend/replace (apache#24365)
## Which issue does this PR close?
- Closes apache#24347.
## Rationale for this change
`array_append`, `array_prepend`, `array_replace`, `array_replace_n` and
`array_replace_all` promise the input list type verbatim — inner field
name, nullability and metadata included — but their kernels rebuilt the
output's inner field from scratch with `Field::new_list_field(...,
true)`. On debug builds this trips the return-type assertion from
apache#17515; on release builds it silently yields a batch whose inner field
disagrees with the schema the planner recorded.
This is the other half of apache#24341, whose `array_slice` part was fixed in
apache#24345. The field-name symptom is a 55.0.0 regression from the same
commit (5b22857, apache#20945); the non-nullable symptom is not a regression.
Unlike `array_slice`, threading the input's field through is not
sufficient here: the appended, prepended or replacement element can
itself be null, so a promise cloned from a `List(non-null T)` input is
wrong at the source and arrow rejects the array with `Non-nullable field
of ListArray cannot contain nulls`.
## What changes are included in this PR?
Each of the five functions now implements `return_field_from_args`,
carrying the input field's name and metadata through while widening
`nullable` when the new element's argument is nullable. The kernels
build their output from `args.return_field` instead of deriving a field
of their own, so promise and payload come from a single source.
Nullability is therefore only widened when the result can genuinely
contain a null:
```sql
array_append(List(non-null Int64), 3) -> List(non-null Int64)
array_append(List(non-null Int64), NULL) -> List(Int64)
```
Two paths beyond those listed in the issue turned out to have the same
defect and are fixed too: `array_append` / `array_prepend` with a
**nested** value type (which delegates to `concat_internal`), and the
`LargeList` variants of all five.
Since these five now implement `return_field_from_args`, their
`return_type` becomes unreachable and returns
`internal_err!("return_field_from_args should be used instead")`,
matching the guidance on `ScalarUDFImpl::return_type` and the existing
convention in `remove.rs` and `map_values.rs`.
Two small cleanups while in here:
- The `List`/`LargeList` inner-field extraction added to
`general_array_slice` by apache#24345 is now shared as
`utils::list_inner_field`, used by all three files. Its error text is
unchanged. The `ListView` variant in `general_list_view_array_slice` is
deliberately left alone — folding all four variants into one helper
would let `general_array_slice` silently accept a `ListView` that its
match currently rejects.
- `array_concat` is **not** affected and its behaviour is unchanged: it
derives a fresh return type via `type_union_resolution` rather than
cloning an input's, so it keeps passing `None` to `concat_internal` and
deriving the field from the aligned inputs.
Behaviour for `Null`-typed array arguments is unchanged in all five
functions (`array_replace*` return `Null`, `array_append` /
`array_prepend` return `List(element)`).
## Are these changes tested?
Yes — 14 new SLT tests across `array_append.slt`, `array_prepend.slt`,
`array_replace.slt`, plus two guard cases in `array_concat.slt` pinning
down that it is unaffected.
`arrow_cast` can express a named inner field (`'List(Int64, field:
''element'')'`), so these reproduce the field-name half of the bug
without needing the Spark dialect. Coverage: inner field name preserved,
non-nullable inner field preserved, nullability widened only when the
new element is nullable (both literal `NULL` and a nullable column),
`LargeList`, nested value types, the `max <= 0` short circuit, and a
`NULL` `max`.
Every one of these queries fails on `main` with the return-type
assertion.
Also run: `cargo clippy --all-targets --all-features -- -D warnings`,
the full sqllogictest suite, and the extended workspace test suite (68
test binaries, 0 failures). The `array_replace` and `array_concat`
benchmarks show no regression against `main`.
## Are there any user-facing changes?
The five functions now return the inner field they promise instead of a
rebuilt one, which is the bug fix. As a consequence, appending or
replacing with a nullable element widens the declared inner nullability
of the result (`List(non-null Int64)` -> `List(Int64)`), which is
required for the result to be representable at all.
No breaking changes to public APIs.1 parent a7cebae commit 22518be
8 files changed
Lines changed: 505 additions & 86 deletions
File tree
- datafusion
- functions-nested/src
- sqllogictest/test_files/array
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
23 | | - | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
24 | 27 | | |
25 | 28 | | |
26 | 29 | | |
27 | 30 | | |
28 | 31 | | |
29 | | - | |
| 32 | + | |
30 | 33 | | |
31 | 34 | | |
32 | 35 | | |
33 | 36 | | |
34 | 37 | | |
35 | 38 | | |
36 | | - | |
| 39 | + | |
37 | 40 | | |
38 | 41 | | |
39 | 42 | | |
40 | 43 | | |
41 | | - | |
42 | | - | |
| 44 | + | |
| 45 | + | |
43 | 46 | | |
44 | 47 | | |
45 | 48 | | |
| |||
104 | 107 | | |
105 | 108 | | |
106 | 109 | | |
107 | | - | |
108 | | - | |
109 | | - | |
110 | | - | |
111 | | - | |
112 | | - | |
113 | | - | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
114 | 123 | | |
115 | 124 | | |
116 | 125 | | |
117 | | - | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
118 | 130 | | |
119 | 131 | | |
120 | 132 | | |
| |||
186 | 198 | | |
187 | 199 | | |
188 | 200 | | |
189 | | - | |
190 | | - | |
191 | | - | |
192 | | - | |
193 | | - | |
194 | | - | |
195 | | - | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
196 | 214 | | |
197 | 215 | | |
198 | 216 | | |
199 | | - | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
200 | 221 | | |
201 | 222 | | |
202 | 223 | | |
| |||
375 | 396 | | |
376 | 397 | | |
377 | 398 | | |
378 | | - | |
| 399 | + | |
| 400 | + | |
| 401 | + | |
| 402 | + | |
| 403 | + | |
| 404 | + | |
| 405 | + | |
| 406 | + | |
| 407 | + | |
| 408 | + | |
| 409 | + | |
| 410 | + | |
| 411 | + | |
| 412 | + | |
| 413 | + | |
| 414 | + | |
379 | 415 | | |
380 | | - | |
| 416 | + | |
381 | 417 | | |
382 | 418 | | |
383 | 419 | | |
384 | | - | |
| 420 | + | |
| 421 | + | |
| 422 | + | |
| 423 | + | |
| 424 | + | |
| 425 | + | |
| 426 | + | |
| 427 | + | |
| 428 | + | |
| 429 | + | |
| 430 | + | |
385 | 431 | | |
386 | 432 | | |
387 | 433 | | |
| |||
438 | 484 | | |
439 | 485 | | |
440 | 486 | | |
441 | | - | |
| 487 | + | |
| 488 | + | |
| 489 | + | |
| 490 | + | |
442 | 491 | | |
443 | 492 | | |
444 | 493 | | |
445 | | - | |
| 494 | + | |
446 | 495 | | |
447 | 496 | | |
448 | 497 | | |
| |||
451 | 500 | | |
452 | 501 | | |
453 | 502 | | |
454 | | - | |
| 503 | + | |
455 | 504 | | |
456 | 505 | | |
457 | 506 | | |
458 | | - | |
459 | | - | |
| 507 | + | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
460 | 511 | | |
461 | 512 | | |
462 | 513 | | |
463 | 514 | | |
464 | | - | |
| 515 | + | |
465 | 516 | | |
466 | 517 | | |
467 | 518 | | |
468 | | - | |
469 | | - | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
470 | 523 | | |
471 | 524 | | |
472 | 525 | | |
473 | 526 | | |
474 | 527 | | |
475 | 528 | | |
476 | 529 | | |
| 530 | + | |
477 | 531 | | |
478 | 532 | | |
479 | 533 | | |
| |||
490 | 544 | | |
491 | 545 | | |
492 | 546 | | |
| 547 | + | |
| 548 | + | |
| 549 | + | |
| 550 | + | |
| 551 | + | |
| 552 | + | |
| 553 | + | |
493 | 554 | | |
494 | | - | |
495 | | - | |
496 | | - | |
| 555 | + | |
| 556 | + | |
| 557 | + | |
| 558 | + | |
497 | 559 | | |
498 | 560 | | |
499 | 561 | | |
500 | | - | |
| 562 | + | |
501 | 563 | | |
502 | 564 | | |
503 | 565 | | |
| |||
516 | 578 | | |
517 | 579 | | |
518 | 580 | | |
519 | | - | |
| 581 | + | |
520 | 582 | | |
521 | 583 | | |
522 | 584 | | |
| |||
528 | 590 | | |
529 | 591 | | |
530 | 592 | | |
531 | | - | |
| 593 | + | |
532 | 594 | | |
533 | 595 | | |
534 | 596 | | |
| |||
565 | 627 | | |
566 | 628 | | |
567 | 629 | | |
568 | | - | |
| 630 | + | |
569 | 631 | | |
570 | 632 | | |
571 | 633 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
47 | 47 | | |
48 | 48 | | |
49 | 49 | | |
50 | | - | |
| 50 | + | |
51 | 51 | | |
52 | 52 | | |
53 | 53 | | |
| |||
624 | 624 | | |
625 | 625 | | |
626 | 626 | | |
627 | | - | |
628 | | - | |
629 | | - | |
630 | | - | |
631 | | - | |
632 | | - | |
633 | | - | |
634 | | - | |
| 627 | + | |
635 | 628 | | |
636 | 629 | | |
637 | 630 | | |
| |||
0 commit comments