Repository navigation
Commit 699e5ac
fix: propagate inner-field metadata through composite-type constructors (apache#21982)
`make_array`, `array_agg`, `array_repeat`, `arrays_zip`, `map`, `range` /
`generate_series`, and `struct` only overrode `return_type` and synthesized
fresh inner fields at runtime, so each one silently dropped the input
field's metadata when wrapping it into a `List`/`Struct`/`Map`. In practice
this broke Arrow extension types (`ARROW:extension:name` /
`ARROW:extension:metadata`) round-tripping through SQL list/struct/map
constructors.
Add two helpers in `datafusion-common::utils` — `list_inner_field_from`
and `struct_inner_fields_from` — that wrap a source `Field` into the inner
field of a list/struct while preserving its metadata, and extend
`SingleRowListArrayBuilder::with_field` to copy metadata too. Use these
helpers consistently from each affected function's
`return_field_from_args` / `return_field`, and thread the resulting
metadata-bearing inner field into the runtime construction paths
(including the `array_agg` accumulators, which now carry a `FieldRef`
instead of a bare `DataType`).
Adds Rust unit tests for each affected function plus an end-to-end
`array_metadata_propagation.slt` that asserts metadata survives every
constructor by string-matching the rendered data type.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>1 parent 9d92944 commit 699e5ac
11 files changed
Lines changed: 1030 additions & 194 deletions
File tree
- datafusion
- common/src/utils
- functions-aggregate
- benches
- src
- functions-nested/src
- functions/src/core
- spark/src/function/aggregate
- sqllogictest/test_files
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
40 | 40 | | |
41 | 41 | | |
42 | 42 | | |
43 | | - | |
| 43 | + | |
44 | 44 | | |
45 | 45 | | |
46 | 46 | | |
47 | 47 | | |
48 | 48 | | |
49 | | - | |
| 49 | + | |
50 | 50 | | |
51 | 51 | | |
52 | 52 | | |
| |||
425 | 425 | | |
426 | 426 | | |
427 | 427 | | |
| 428 | + | |
| 429 | + | |
428 | 430 | | |
429 | 431 | | |
430 | 432 | | |
| |||
434 | 436 | | |
435 | 437 | | |
436 | 438 | | |
| 439 | + | |
437 | 440 | | |
438 | 441 | | |
439 | 442 | | |
| |||
449 | 452 | | |
450 | 453 | | |
451 | 454 | | |
452 | | - | |
453 | | - | |
454 | | - | |
455 | | - | |
| 455 | + | |
| 456 | + | |
| 457 | + | |
| 458 | + | |
| 459 | + | |
| 460 | + | |
| 461 | + | |
| 462 | + | |
| 463 | + | |
| 464 | + | |
| 465 | + | |
456 | 466 | | |
457 | 467 | | |
458 | 468 | | |
| |||
524 | 534 | | |
525 | 535 | | |
526 | 536 | | |
| 537 | + | |
527 | 538 | | |
528 | 539 | | |
529 | | - | |
| 540 | + | |
530 | 541 | | |
531 | 542 | | |
532 | 543 | | |
| 544 | + | |
| 545 | + | |
| 546 | + | |
533 | 547 | | |
534 | 548 | | |
535 | 549 | | |
536 | 550 | | |
| 551 | + | |
| 552 | + | |
| 553 | + | |
| 554 | + | |
| 555 | + | |
| 556 | + | |
| 557 | + | |
| 558 | + | |
| 559 | + | |
| 560 | + | |
| 561 | + | |
| 562 | + | |
| 563 | + | |
| 564 | + | |
| 565 | + | |
| 566 | + | |
| 567 | + | |
| 568 | + | |
| 569 | + | |
| 570 | + | |
| 571 | + | |
| 572 | + | |
| 573 | + | |
| 574 | + | |
| 575 | + | |
| 576 | + | |
| 577 | + | |
| 578 | + | |
| 579 | + | |
| 580 | + | |
| 581 | + | |
| 582 | + | |
| 583 | + | |
| 584 | + | |
| 585 | + | |
| 586 | + | |
| 587 | + | |
| 588 | + | |
| 589 | + | |
| 590 | + | |
| 591 | + | |
| 592 | + | |
| 593 | + | |
| 594 | + | |
| 595 | + | |
| 596 | + | |
537 | 597 | | |
538 | 598 | | |
539 | 599 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
24 | | - | |
| 24 | + | |
25 | 25 | | |
26 | 26 | | |
27 | 27 | | |
| |||
41 | 41 | | |
42 | 42 | | |
43 | 43 | | |
| 44 | + | |
44 | 45 | | |
45 | 46 | | |
46 | 47 | | |
47 | 48 | | |
48 | | - | |
| 49 | + | |
49 | 50 | | |
50 | 51 | | |
51 | 52 | | |
| |||
0 commit comments