Repository navigation
Confusing constructors for new data types #3247
Description
I'm trying to update VirtualiZarr to work with the new data types, and just wanted to note some ways in which I found the new API confusing.
I briefly skimmed the narrative docs page, read the phrase "native dtypes", and then found a the zarr.dtype.ZDType.from_native_dtype() constructor classmethod in the API docs. But that behaved very counterintuitively:
In [1]: from zarr.dtype import ZDType, parse_data_type
In [2]: import numpy as np
In [3]: np.dtype('int32')
Out[3]: dtype('int32')
In [4]: ZDType.from_native_dtype(np.dtype('int32'))This returns None?!
Looking at the docs more closely, apparently I would have to do Int32.from_native_dtype(np.dtype('int32'))? But that's no help to me if I don't already have the zarr dtype.
I'm also confused as to why passing the base class succeeds at all. If Int32 is a subclass of ZDType, and .from_native_dtype is intended to be used on the subclasses, then why isn't it abstract on the base class?
It seems what I actually needed was parse_data_type:
In [6]: parse_data_type(np.dtype('int32'), zarr_format=3)
Out[6]: Int32(endianness='little')which isn't mentioned explicitly in the narrative docs, nor uses the word "native" anywhere, even though it seems to parse native dtypes.
After reading the full docs page, I still don't think it would be clear how to get from a native dtype to a zarr data type. Presumably that information should be part of the section on Data Type Resolution.
Aside: It's also confusing how the docs page says
Zarr V3 data types do not have endianness.
but that Int32(endianness='little') object certainly seems to have endianess...
/ end rant 😅
Activity
d-v-b commentedon Jul 15, 2025
There isn't in general a 1:1 mapping between numpy data types and zarr data types. So you either need to know the zarr data type you want, or you use the dynamic data type resolution routines.
It is abstract, so I'm surprised that you didn't get an error in your example. Maybe we need to re-order the decorators here?
zarr-python/src/zarr/core/dtype/wrapper.py
Lines 99 to 126 in 2420f9e
In32()is not a zarr v3 data type! It's aZDType, which is a python class we invented for zarr python, to model zarr v2 and zarr v3 data types. Zarr v2 data types have endianness, which is whyInt32()has endianness. But that endianness information is not preserved when you convertInt32to the zarr v3 JSON form.d-v-b commentedon Jul 15, 2025
also, i'm pretty careful about language in the dtype docs: I use "Zarr V2" or "Zarr V3" very specifically to refer to the formats, and I use "Zarr Python" very specifically to refer to the python library. I get the impression that this distinction didn't come across. Would it be helpful to put a disclaimer at the top that explains this distinction?
TomNicholas commentedon Jul 15, 2025
Thanks.
Okay fair enough.
The docs should show you the correct way to do this (I'm still unclear if I'm doing it correctly now).
Okay so there's a bug, which that example above is an MCVE for.
That makes sense now you say it.
I think to be fair this is already explained reasonably well here. My complaint is more that I wanted to do something that sounds straightforward (make zarr-python metadata using a given numpy dtype) and that process could be clearer.
d-v-b commentedon Jul 15, 2025
i'll add a section to the docs demonstrating how to parse user input (like a numpy data type) into a zdtype
d-v-b commentedon Jul 15, 2025
and please let me know if anything else could be improved, or is confusing! The NumPy vs Zarr V2 vs Zarr V3 differences are in fact quite confusing, so we need to put extra effort into the docs to elucidate this stuff
d-v-b commentedon Jun 3, 2026
I think this can be closed given the documentation changes. let me know if that's not the case, and I will re-open!