Repository navigation
Panic when casting [] to FixedSizeList(0, Null) #9158
Description
Describe the bug
DataFusion CLI v35.0.0
❯ select arrow_cast(make_array(), 'FixedSizeList(1, Null)');
Arrow error: Cast error: Cannot cast to FixedSizeList(1): value at index 0 has length 0
❯ select arrow_cast([], 'FixedSizeList(1, Null)');
Arrow error: Cast error: Cannot cast to FixedSizeList(1): value at index 0 has length 0
To Reproduce
No response
Expected behavior
No response
Additional context
No response
Activity
r3stl355 commentedon Feb 8, 2024
I wonder if that's meant to work. The following works:
However, if you wanted a zero-sized list then should it be be
However that throws the following error
alamb commentedon Feb 9, 2024
Panic'ing is definitely not good
r3stl355 commentedon Feb 12, 2024
I'll see what I can do
r3stl355 commentedon Feb 12, 2024
take
r3stl355 commentedon Feb 24, 2024
I've done some digging but did not find an easy fix, only few options listed below. Happy to follow up but need a decision on which fix to attempt.
The following works
select arrow_cast([null], 'FixedSizeList(1, Null)');so it's logical to useFixedSizeList(0, Null)when casting an empty array (select arrow_cast([], 'FixedSizeList(0, Null)');). However, that doesn't work because of the following:datafusion_common::ScalarValue::FixedSizeListsays "The array must be a FixedSizeListArray with length 1." (the same applies to other Scalar::List* types) so any length other than 1 would be invalid.https://github.com/r3stl355/arrow-datafusion/blob/3b355c798a3258f118016b33f26c5a55fed36220/datafusion/common/src/scalar/mod.rs#L231
During the cast,
datafusion_common::ScalarValue::FixedSizeList(0, Null)is converted toarrow_schema::datatype::DataType::FixedSizeList(FieldRef, 0)before being passed toarrow::compute::kernels::cast::cast_with_optionsfor evaluation of thearrow_castarrow::compute::kernels::cast::cast_with_optionsreturns a FixedSizeListArray<0> of length 0 when called witharrow_schema::datatype::DataType::FixedSizeList(FieldRef, 0). Note that this is different for any length greater than 0 used inFixedSizeList(i.e. the return value will always be of length 1), e.g. called withFixedSizeList(FieldRef, 2)as cast type,arrow::compute::kernels::cast::cast_with_optionswhich returns aFixedSizeListArray<2>with a length 1.The possible fix options are:
FixedSizeList(0, Null)'))FixedSizeList(FieldRef, 0)toFixedSizeList(FieldRef, 1)before callingcast_with_optionsbut A. this feels really wrong and B. may still not workcast_with_optionsis called withFixedSizeList(FieldRef, 0). I'll do some digging there to see if it's possible, e.g ifFixedSizeListArray<0>[NullArray(0),]would be a valid typeLastly, this error happens when displaying the result but not when applying some other functions to it, e.g. this following works but its the only function I tested it with:
jayzhan211 commentedon Feb 24, 2024
I prefer 1. I think Fixedsizelist with len 0 is the same as an empty list. I don't think there is any useful case that we need to cast an empty list to Fixedsizelist(0, type). Return
exec_errorif casting to Fixedsizelist(0, any type). We just need to avoid panic for this casting.r3stl355 commentedon Feb 24, 2024
@Weijun-H was there any specific reason you were trying to achieve this (i.e.
select arrow_cast(make_array(), 'FixedSizeList(1, Null)');)?Weijun-H commentedon Feb 25, 2024
There are no particular use cases now, I am working on #9108, which reminded me of this case. And also I vote for the first solution, which is more reasonable.
r3stl355 commentedon Mar 22, 2024
I unassigned myself from this issue as I don't have much bandwidth at the moment so maybe someone else is willing to implement the changes. If nobody does then I'll come back to this in 2-3 weeks.
r3stl355 commentedon Apr 30, 2024
Looks like this is still open, happy to resume if noone else is working on it
Jefffrey commentedon Jun 28, 2026
upstream arrow-rs PR to fix issue where cast output a different length array
[-]Failed to cast `[]` to `FixedSizeList(1, Null)`[/-][+]Panic when casting `[]` to `FixedSizeList(0, Null)`[/+]fix: cast `[]` to `FixedSizeList(0, _)` (apache#23381)