Skip to content

Commit 22518be

Browse files
timsaucerimtherealnaska
authored andcommitted
fix: preserve the input list's inner field in array_append/prepend/replace (apache#24365)
## Which issue does this PR close? - Closes apache#24347. ## Rationale for this change `array_append`, `array_prepend`, `array_replace`, `array_replace_n` and `array_replace_all` promise the input list type verbatim — inner field name, nullability and metadata included — but their kernels rebuilt the output's inner field from scratch with `Field::new_list_field(..., true)`. On debug builds this trips the return-type assertion from apache#17515; on release builds it silently yields a batch whose inner field disagrees with the schema the planner recorded. This is the other half of apache#24341, whose `array_slice` part was fixed in apache#24345. The field-name symptom is a 55.0.0 regression from the same commit (5b22857, apache#20945); the non-nullable symptom is not a regression. Unlike `array_slice`, threading the input's field through is not sufficient here: the appended, prepended or replacement element can itself be null, so a promise cloned from a `List(non-null T)` input is wrong at the source and arrow rejects the array with `Non-nullable field of ListArray cannot contain nulls`. ## What changes are included in this PR? Each of the five functions now implements `return_field_from_args`, carrying the input field's name and metadata through while widening `nullable` when the new element's argument is nullable. The kernels build their output from `args.return_field` instead of deriving a field of their own, so promise and payload come from a single source. Nullability is therefore only widened when the result can genuinely contain a null: ```sql array_append(List(non-null Int64), 3) -> List(non-null Int64) array_append(List(non-null Int64), NULL) -> List(Int64) ``` Two paths beyond those listed in the issue turned out to have the same defect and are fixed too: `array_append` / `array_prepend` with a **nested** value type (which delegates to `concat_internal`), and the `LargeList` variants of all five. Since these five now implement `return_field_from_args`, their `return_type` becomes unreachable and returns `internal_err!("return_field_from_args should be used instead")`, matching the guidance on `ScalarUDFImpl::return_type` and the existing convention in `remove.rs` and `map_values.rs`. Two small cleanups while in here: - The `List`/`LargeList` inner-field extraction added to `general_array_slice` by apache#24345 is now shared as `utils::list_inner_field`, used by all three files. Its error text is unchanged. The `ListView` variant in `general_list_view_array_slice` is deliberately left alone — folding all four variants into one helper would let `general_array_slice` silently accept a `ListView` that its match currently rejects. - `array_concat` is **not** affected and its behaviour is unchanged: it derives a fresh return type via `type_union_resolution` rather than cloning an input's, so it keeps passing `None` to `concat_internal` and deriving the field from the aligned inputs. Behaviour for `Null`-typed array arguments is unchanged in all five functions (`array_replace*` return `Null`, `array_append` / `array_prepend` return `List(element)`). ## Are these changes tested? Yes — 14 new SLT tests across `array_append.slt`, `array_prepend.slt`, `array_replace.slt`, plus two guard cases in `array_concat.slt` pinning down that it is unaffected. `arrow_cast` can express a named inner field (`'List(Int64, field: ''element'')'`), so these reproduce the field-name half of the bug without needing the Spark dialect. Coverage: inner field name preserved, non-nullable inner field preserved, nullability widened only when the new element is nullable (both literal `NULL` and a nullable column), `LargeList`, nested value types, the `max <= 0` short circuit, and a `NULL` `max`. Every one of these queries fails on `main` with the return-type assertion. Also run: `cargo clippy --all-targets --all-features -- -D warnings`, the full sqllogictest suite, and the extended workspace test suite (68 test binaries, 0 failures). The `array_replace` and `array_concat` benchmarks show no regression against `main`. ## Are there any user-facing changes? The five functions now return the inner field they promise instead of a rebuilt one, which is the bug fix. As a consequence, appending or replacing with a nullable element widens the declared inner nullability of the result (`List(non-null Int64)` -> `List(Int64)`), which is required for the result to be representable at all. No breaking changes to public APIs.
1 parent a7cebae commit 22518be

8 files changed

Lines changed: 505 additions & 86 deletions

File tree

‎datafusion/functions-nested/src/concat.rs‎

Lines changed: 101 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -20,26 +20,29 @@
2020
use std::sync::Arc;
2121

2222
use crate::make_array::make_array_inner;
23-
use crate::utils::{align_array_dimensions, check_datatypes, make_scalar_function};
23+
use crate::utils::{
24+
align_array_dimensions, check_datatypes, list_inner_field, list_type_with_element,
25+
make_scalar_function,
26+
};
2427
use arrow::array::{
2528
Array, ArrayData, ArrayRef, Capacities, GenericListArray, MutableArrayData,
2629
OffsetSizeTrait,
2730
};
2831
use arrow::buffer::{NullBuffer, OffsetBuffer};
29-
use arrow::datatypes::{DataType, Field};
32+
use arrow::datatypes::{DataType, Field, FieldRef};
3033
use datafusion_common::Result;
3134
use datafusion_common::utils::{
3235
ListCoercion, base_type, coerced_type_with_base_type_only,
3336
};
3437
use datafusion_common::{
3538
cast::as_generic_list_array,
36-
exec_err, plan_err,
39+
exec_err, internal_err, plan_err,
3740
utils::{list_ndims, take_function_args},
3841
};
3942
use datafusion_expr::binary::type_union_resolution;
4043
use datafusion_expr::{
41-
ColumnarValue, Documentation, ScalarFunctionArgs, ScalarUDFImpl, Signature,
42-
Volatility,
44+
ColumnarValue, Documentation, ReturnFieldArgs, ScalarFunctionArgs, ScalarUDFImpl,
45+
Signature, Volatility,
4346
};
4447
use datafusion_macros::user_doc;
4548
use itertools::Itertools;
@@ -104,17 +107,26 @@ impl ScalarUDFImpl for ArrayAppend {
104107
&self.signature
105108
}
106109

107-
fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {
108-
let [array_type, element_type] = take_function_args(self.name(), arg_types)?;
109-
if array_type.is_null() {
110-
Ok(DataType::new_list(element_type.clone(), true))
111-
} else {
112-
Ok(array_type.clone())
113-
}
110+
fn return_type(&self, _arg_types: &[DataType]) -> Result<DataType> {
111+
internal_err!("return_field_from_args should be used instead")
112+
}
113+
114+
fn return_field_from_args(&self, args: ReturnFieldArgs) -> Result<FieldRef> {
115+
let [array_field, element_field] =
116+
take_function_args(self.name(), args.arg_fields)?;
117+
let data_type = append_prepend_return_type(
118+
array_field.data_type(),
119+
element_field.data_type(),
120+
element_field.is_nullable(),
121+
);
122+
Ok(Arc::new(Field::new(self.name(), data_type, true)))
114123
}
115124

116125
fn invoke_with_args(&self, args: ScalarFunctionArgs) -> Result<ColumnarValue> {
117-
make_scalar_function(array_append_inner)(&args.args)
126+
let return_type = args.return_field.data_type().clone();
127+
make_scalar_function(|args: &[ArrayRef]| array_append_inner(args, &return_type))(
128+
&args.args,
129+
)
118130
}
119131

120132
fn aliases(&self) -> &[String] {
@@ -186,17 +198,26 @@ impl ScalarUDFImpl for ArrayPrepend {
186198
&self.signature
187199
}
188200

189-
fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {
190-
let [element_type, array_type] = take_function_args(self.name(), arg_types)?;
191-
if array_type.is_null() {
192-
Ok(DataType::new_list(element_type.clone(), true))
193-
} else {
194-
Ok(array_type.clone())
195-
}
201+
fn return_type(&self, _arg_types: &[DataType]) -> Result<DataType> {
202+
internal_err!("return_field_from_args should be used instead")
203+
}
204+
205+
fn return_field_from_args(&self, args: ReturnFieldArgs) -> Result<FieldRef> {
206+
let [element_field, array_field] =
207+
take_function_args(self.name(), args.arg_fields)?;
208+
let data_type = append_prepend_return_type(
209+
array_field.data_type(),
210+
element_field.data_type(),
211+
element_field.is_nullable(),
212+
);
213+
Ok(Arc::new(Field::new(self.name(), data_type, true)))
196214
}
197215

198216
fn invoke_with_args(&self, args: ScalarFunctionArgs) -> Result<ColumnarValue> {
199-
make_scalar_function(array_prepend_inner)(&args.args)
217+
let return_type = args.return_field.data_type().clone();
218+
make_scalar_function(|args: &[ArrayRef]| array_prepend_inner(args, &return_type))(
219+
&args.args,
220+
)
200221
}
201222

202223
fn aliases(&self) -> &[String] {
@@ -375,13 +396,38 @@ pub fn array_concat_inner(args: &[ArrayRef]) -> Result<ArrayRef> {
375396
args[0].len(),
376397
)))
377398
} else if large_list {
378-
concat_internal::<i64>(args)
399+
concat_internal::<i64>(args, None)
400+
} else {
401+
concat_internal::<i32>(args, None)
402+
}
403+
}
404+
405+
/// Return type shared by `array_append` and `array_prepend`: the input list
406+
/// type, except that its inner field is nullable whenever the appended or
407+
/// prepended element may be null.
408+
fn append_prepend_return_type(
409+
array_type: &DataType,
410+
element_type: &DataType,
411+
element_nullable: bool,
412+
) -> DataType {
413+
if array_type.is_null() {
414+
DataType::new_list(element_type.clone(), true)
379415
} else {
380-
concat_internal::<i32>(args)
416+
list_type_with_element(array_type, element_nullable)
381417
}
382418
}
383419

384-
fn concat_internal<O: OffsetSizeTrait>(args: &[ArrayRef]) -> Result<ArrayRef> {
420+
/// Concatenates the list arrays in `args` row-wise.
421+
///
422+
/// `field` is the list field the output must carry. `array_concat` passes `None`
423+
/// because its `return_type` derives a fresh field from the unified element
424+
/// types, which is what deriving the field from the aligned inputs reproduces.
425+
/// `array_append` / `array_prepend` promise their input's field verbatim and so
426+
/// must pass it in explicitly.
427+
fn concat_internal<O: OffsetSizeTrait>(
428+
args: &[ArrayRef],
429+
field: Option<&FieldRef>,
430+
) -> Result<ArrayRef> {
385431
let args = align_array_dimensions::<O>(args.to_vec())?;
386432

387433
let list_arrays = args
@@ -438,11 +484,14 @@ fn concat_internal<O: OffsetSizeTrait>(args: &[ArrayRef]) -> Result<ArrayRef> {
438484
offsets.push(O::usize_as(mutable.len()));
439485
}
440486

441-
let data_type = list_arrays[0].value_type();
487+
let field = match field {
488+
Some(field) => Arc::clone(field),
489+
None => Arc::new(Field::new_list_field(list_arrays[0].value_type(), true)),
490+
};
442491
let data = mutable.freeze();
443492

444493
Ok(Arc::new(GenericListArray::<O>::try_new(
445-
Arc::new(Field::new_list_field(data_type, true)),
494+
field,
446495
OffsetBuffer::new(offsets.into()),
447496
arrow::array::make_array(data),
448497
valid,
@@ -451,29 +500,34 @@ fn concat_internal<O: OffsetSizeTrait>(args: &[ArrayRef]) -> Result<ArrayRef> {
451500

452501
// Kernel functions
453502

454-
fn array_append_inner(args: &[ArrayRef]) -> Result<ArrayRef> {
503+
fn array_append_inner(args: &[ArrayRef], return_type: &DataType) -> Result<ArrayRef> {
455504
let [array, values] = take_function_args("array_append", args)?;
456505
match array.data_type() {
457506
DataType::Null => make_array_inner(&[Arc::clone(values)]),
458-
DataType::List(_) => general_append_and_prepend::<i32>(args, true),
459-
DataType::LargeList(_) => general_append_and_prepend::<i64>(args, true),
507+
DataType::List(_) => general_append_and_prepend::<i32>(args, true, return_type),
508+
DataType::LargeList(_) => {
509+
general_append_and_prepend::<i64>(args, true, return_type)
510+
}
460511
arg_type => exec_err!("array_append does not support type {arg_type}"),
461512
}
462513
}
463514

464-
fn array_prepend_inner(args: &[ArrayRef]) -> Result<ArrayRef> {
515+
fn array_prepend_inner(args: &[ArrayRef], return_type: &DataType) -> Result<ArrayRef> {
465516
let [values, array] = take_function_args("array_prepend", args)?;
466517
match array.data_type() {
467518
DataType::Null => make_array_inner(&[Arc::clone(values)]),
468-
DataType::List(_) => general_append_and_prepend::<i32>(args, false),
469-
DataType::LargeList(_) => general_append_and_prepend::<i64>(args, false),
519+
DataType::List(_) => general_append_and_prepend::<i32>(args, false, return_type),
520+
DataType::LargeList(_) => {
521+
general_append_and_prepend::<i64>(args, false, return_type)
522+
}
470523
arg_type => exec_err!("array_prepend does not support type {arg_type}"),
471524
}
472525
}
473526

474527
fn general_append_and_prepend<O: OffsetSizeTrait>(
475528
args: &[ArrayRef],
476529
is_append: bool,
530+
return_type: &DataType,
477531
) -> Result<ArrayRef>
478532
where
479533
i64: TryInto<O>,
@@ -490,14 +544,22 @@ where
490544
(list_array, element_array)
491545
};
492546

547+
let name = if is_append {
548+
"array_append"
549+
} else {
550+
"array_prepend"
551+
};
552+
let field = list_inner_field(name, return_type)?;
553+
493554
let res = match list_array.value_type() {
494-
DataType::List(_) => concat_internal::<O>(args)?,
495-
DataType::LargeList(_) => concat_internal::<O>(args)?,
496-
data_type => {
555+
DataType::List(_) | DataType::LargeList(_) => {
556+
concat_internal::<O>(args, Some(&field))?
557+
}
558+
_ => {
497559
return generic_append_and_prepend::<O>(
498560
list_array,
499561
element_array,
500-
&data_type,
562+
field,
501563
is_append,
502564
);
503565
}
@@ -516,7 +578,7 @@ where
516578
///
517579
/// * `list_array` - A reference to the ListArray to which elements will be appended/prepended.
518580
/// * `element_array` - A reference to the Array containing elements to be appended/prepended.
519-
/// * `field` - A reference to the Field describing the data type of the arrays.
581+
/// * `field` - The list field the output must carry, taken from the promised return type.
520582
/// * `is_append` - A boolean flag indicating whether to append (`true`) or prepend (`false`) elements.
521583
///
522584
/// # Examples
@@ -528,7 +590,7 @@ where
528590
fn generic_append_and_prepend<O: OffsetSizeTrait>(
529591
list_array: &GenericListArray<O>,
530592
element_array: &ArrayRef,
531-
data_type: &DataType,
593+
field: FieldRef,
532594
is_append: bool,
533595
) -> Result<ArrayRef>
534596
where
@@ -565,7 +627,7 @@ where
565627
let data = mutable.freeze();
566628

567629
Ok(Arc::new(GenericListArray::<O>::try_new(
568-
Arc::new(Field::new_list_field(data_type.to_owned(), true)),
630+
field,
569631
OffsetBuffer::new(offsets.into()),
570632
arrow::array::make_array(data),
571633
None,

‎datafusion/functions-nested/src/extract.rs‎

Lines changed: 2 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,7 @@ use datafusion_expr::{
4747
use datafusion_macros::user_doc;
4848
use std::sync::Arc;
4949

50-
use crate::utils::make_scalar_function;
50+
use crate::utils::{list_inner_field, make_scalar_function};
5151

5252
// Create static instances of ScalarUDFs for each function
5353
make_udf_expr_and_func!(
@@ -624,14 +624,7 @@ where
624624
// Carry the input's list field through to the output so that the returned
625625
// type matches the one promised by `return_type` / `return_field_from_args`,
626626
// including the field name, nullability and metadata.
627-
let field = match array.data_type() {
628-
List(field) | LargeList(field) => Arc::clone(field),
629-
other => {
630-
return internal_err!(
631-
"general_array_slice got unexpected data type: {other}"
632-
);
633-
}
634-
};
627+
let field = list_inner_field("general_array_slice", array.data_type())?;
635628

636629
// `use_nulls` is false because we never call `try_extend_nulls`: null rows are
637630
// emitted as empty slices. Arrow still allocates a validity buffer on its own

0 commit comments

Comments
 (0)