Repository navigation
can we reject dtype inference for numpy object arrays #3077
Description
numpy arrays with dtype "O" are ambiguous, in the sense that they could contain values that zarr should store as:
- variable-length strings
- variable-length arrays
- arbitrary python objects
- etc
Unlike the object dtype, every other numpy dtype has a simple mapping to a zarr metadata representation. For these dtypes (e.g., int8, int16, etc), a user can provide a numpy array and we can automatically pick the right zarr data type representation from that array. But for the object dtype, this is not possible. Extra information is needed to resolve a zarr data type for object dtype arrays.
in zarr-python 2, we used an optional object_codec keyword argument to array creation routines. If a user provided dtype=np.dtype('O') or equivalent without a object_codec, then zarr-python 2 would error.
I don't want to use this exact pattern today, because object_codec is not really well-defined, and this extra parameter, used only for numpy object dtypes, would greatly complicate the dtype inference for all the other dtypes. Here is my alternative proposal: we refuse to do any dtype inference for numpy object dtypes. Instead, the user must provide an explicit zarr dtype that is compatible with the numpy object dtype.
e.g.:
create_array(...., dtype=np.dtype('O')) would raise an informative exception, guiding the user to do this instead:
create_array(..., dtype=zarr.dtypes.VariableLengthString()), or create_array(..., dtype='numpy.variable_length_string')
Thoughts on this pattern?
Activity
rabernat commentedon May 21, 2025
👍 to this.
However, we should recognize that a user may still pass
Otype data when writing to an array, and may receive it back after decoding.d-v-b commentedon May 22, 2025
Yes, under my proposal for pre-numpy 2 variable length string data, it would still be numpy
Oin, and numpyOout, once the array has been created. The main change would be zarr becoming more strict about what thedtypeparameter could be at array creation time.I have a second suggestion: we should also remove the alias "str" for "variable length string".
VariableLengthString(), or using the name of that data typenenb commentedon May 27, 2025
@d-v-b Just to be very clear about what you are proposing: in future releases of
zarr-python, when users try to create a v3 Zarr array then they will not be allowed to pass the 'O' dtype (if they do, it will raise an informative error message)? I understand that the 'O' dtype may still be present internally (e.g. ifnumpy<2is installed), but we will not allow users to pass it. Is this correct?If I have understood this correctly, then I am also in favour.
I am (weakly) against this. I think the fact that
zarr-pythonsupports multiple string data types is a legacy ofnumpy. If I look at the Arrow data types, I believe that they only have one string dtype (ignore the difference betweenUtf8andLarge Utf8for nowI will open another issue related to this shortly). The fact that they only have one dtring dtype suggests to me that in practice, this is all that is really required.Strings, UTF-8 etc are notoriously confusing for folks who have no experience with them. As such, I think we should be putting sensible defaults here i.e. if a user puts in 'str', then we should default to variable length string. If we don't default, this can become quite confusing for users (e.g. they then need to understand things like UTF-8), and it can lead to them creating dtypes that they don't actually want (see Arrow again, and what folks seem to want in practice).
I'm not sure if this is correct e.g. the
numpyvoiddtype, and bitpacking things likeint2,int4etc. I don't want to distract from the main message of this issue though and so I will write this up in a separate thread shortly.d-v-b commentedon Jun 26, 2025
this argument resonates with me. #3170 will ensure that
"string"is an alias for theVariableLengthUTF8at array creation time.d-v-b commentedon Jul 10, 2025
closing as completed via #2874 and #3170