NEP 41: First steps towards improved Datatype Support - #15506

Perhaps the "Motivate the whole DType change" belongs in NEP 40 and the "Decide that we start implementing NEP 42 proposal" belongs in NEP 42?

Perhaps the "Motivate the whole DType change" belongs in NEP 40 and the "Decide that we start implementing NEP 42 proposal" belongs in NEP 42?

Agree with the second point, disagree with the first. Principle for both: keep things self-contained as much as possible. This is the top-level document for the new dtypes, so the motivation belongs here.

Well, I will rephrase the "decide to start NEP 42", I currently do not wish to remove it.

I will be frank, I do not see implementing this as a big bang can work. There will be way too much hesitation. The complete set of technical details are simply too many to actually settle on anything quickly, and for them not to be a somewhat in flux. Then, it may make sense to have a working example for the new public API of e.g. units. But that will be practically impossible without almost everything finished, e.g. including ufuncs and reorganizing casting, both non-trivial endeavors with moving parts.

That means, I want to be able to build up at least a tiny bit of feeling of momentum. And to get there hope that we:

1. Can agree on some of the central points by accepting this NEP. Which does not actually define any new public API, it is only supposed to give a hint at it, to say that it will be opaque and thus extensible.

2. Get a tiny bit of momentum code wise. I think that would be helpful, and starting with the one decision that type(np.dtype("f8") is a subclass of np.dtype is the central piece that is necessary for everything else.

Now I may be bad at pushing it, but IMO unless we do 1. soon, we will just get stuck in the technical mud (again) with no one willing to make a clear decision on anything. I am willing to debate 2., but I honestly do not see a huge downside: Very few users will write type(np.dtype("f8")), so most users will not even notice; so if we change it completely there will not a lot of fallout.

Of course I can move any implementation to the next NEP and reduce that NEP as much as possible, or propose accepting e.g. only a few points of NEP 42 preliminary and then continue adding/fixing/accepting new points as we go. But I need to know what an acceptable strategy is here, deciding here to do a tiny step in the right direction code wise, seemed like a good "ice breaker" so to say.

I will be frank, I do not see implementing this as a big bang can work. There will be way too much hesitation. The complete set of technical details are simply too many to actually settle on anything quickly, and for them not to be a somewhat in flux. Then, it may make sense to have a working example for the new public API of e.g. units.

@seberg I think you are missing my point a little. I am not suggesting a working example. It can be code as it should look and feel without any implementation backing it up. Thing of it as the thing you'd put in a tutorial, and doctesting it is not needed. Finally, you're not making a hard promise that it will turn out exactly like that - an example that's 80-90% right is better than no example at all.

Without doing that, everyone will simply imagine where you're headed, or stop reading half-way through cause it's too dense. Most of those imaginations will be wrong - we're not mind readers:)

@rgommers, probably. But quite frankly, the only "example", I can think of is:

array_of_categorical = np.array(["spam", "eggs"], dtype=Categorical)
array_with_meters = np.array([1., 3.] dtype=Unit("m"))

And it feels like you probably expect more?

@seberg - am sort-of keeping half an eye on this as I am quite interested in how units could be stored in the dtype (I'm definitely of two minds - I'm still not sure unclear it would in fact be better on the dtype), and reading the above, I agree with @rgommers that a few user-facing examples would be very useful - as he notes, if you don't have it, everyone will fill in the blanks, rather than converge on a single plan. So, do put examples like the ones you quoted above:

array_of_categorical = np.array(["spam", "eggs"], dtype=Categorical)
array_with_meters = np.array([1., 3.] dtype=Unit("m"))

But also let those actually be what you think they should be! For the Categorical, I guess what is missing is what the actual possible categories are. It might also not be clear that what is fed into np.array would normally be a string (rather than, say, an Enum).

For the example with units, right now the dtype does not specify what type of float these are meant to be. Perhaps as a placeholder, you could simply use the datetime notation, i.e., something like f8[m/s]. (Still, this is something that makes me wonder about whether units really belong on the dtype - should there really be a dtype instance for every combination of unit and numerical type? While floats are most common, some complex numbers can sensibly have numbers, and so can some integers and even fractions (e.g., for spin or other type of Quanta). In the end, currently the dtype just tells how to interpret bytes as values, while units are about what those values mean. But then again, there is of course the casting/conversion part that is really similar between units and dtypes.

Anyway, all this mostly to second that examples do matter a lot!

For units there's a ton of existing examples in the documentation of packages, e.g.:

• https://unyt.readthedocs.io/en/stable/usage.html#basic-usage
• https://pint.readthedocs.io/en/0.10.1/#user-guide

So just seeing what is typically done there (defining, checking, converting, simplifying, printing units, and more), and rewriting a couple of the examples to use dtypes would be instructive. Looking at all that, I actually suspect that you can only put part of that behavior in a dtype. When you have a plain numpy array instead of a subclass or duck array (pint provides the latter, with support for array protocols), you have fewer places to hook in. So thinking about what goes missing then may show the need for some design changes, or lead to a conclusion that dtypes aren't the best fit after all.

Hmm, right I will add some short usage examples, of course. I do not want to sound overly annoyed/defensive here, but I think I have to talk with Ralf in person about the real doubts. (I am honestly surprised that I may still have to convince anyone seriously about the general idea!)

@mhvk I guess you would like to see some reasoning why array-likes are not a good generic solution? I will give a stab below:

There is a reason why pandas encodes their array-likes by basically calling them dtypes. (In pandas the dtype is associated to a specific array like in practice). I think the dtype encoding that "each value is a meter" is a much easier concept. I think Quantities do this in a similar way, that works. But it does seem better for users if they have one layer less to think of.

The main question is why create another "protocol" for this (it basically is a very fat protocol, but it is also one we already have, albeit in pretty broken way)?

The reason is that dtypes are much more clearly defined and generalize extremely well! Any NumPy dtype can be put into any container that knows about NumPy dtypes: Dask, pandas, xarray, sparse arrays, …. Pandas is not even an quite an array-like!

DTypes compose even better than __array_ufunc__ when it comes to working with other containers. And have no adoption threshold: as soon as they exist, you can put them into your pandas dataframe.

The main point (that I hoped to make a bit clear in the last change, but is maybe short), is that this all can work so well because dtypes encoding all information required to interpret a single elemet means that container concerns are completely separated from element concerns.

Please tell me if I am wrong, but I always had the feeling that Quantities have to chase other developments (e.g. keep up with new functions) and have some quirks. On the other hand, a dtype does not have to concern itself at all with most functions removing 300 possible quirks with all the functions that are wrapped by __array_function__ :). Keeping up with that and all the other array-likes out there is a moving target: lets make the target smaller!

@seberg - thanks! I think your answer just proves the point - it is useful to have this information in the NEP! I also think both the categorical and the units examples are good, as they are quite different: the categorical example shows that with the NEP ideas it will be easier to give new meaning to more or less arbitrary bytes, while the units example shows that one can add a new layer of meaning.

I do not think things are trivial, though. Clearly, somewhere it has to be defined, e.g., that trig ufuncs take arguments in radians, how multiplication and division affect units, etc. Should this all be part of the dtype via the unit class? Or should ufuncs know about what units they take? And angles are particularly tricky since they are really dimensionless and in many situations one would like to treat them as such. Indeed, astropy deals with this via equivalencies, and also can handle logarithmic units (decibel, etc.), which behave differently for addition. Of course, much of this may well scale to dtypes!

Also, as I noted, I do wonder about how instances would actually work - will there be versions of Meter for each float, integer and complex dtype? Or would one have some combo or stacked dtype, of a unit and a bytes-interpreter?

Another question is where one stops. E.g., I have been playing with classes that also store uncertainty on a number, and found that things were substantially easier if this was in between units and dtype - perhaps not illogically, since physically the uncertainty and the value cannot have incompatible units. Would one want to have dtype that encode how uncertainty is stored? The dtype could "know" how uncertainty propagates... Or how about a mask? My sense is that a general mechanism that allows propagation is super-useful, but it is not quite obvious what should be part of the dtype, or how a dtype allows combinations.

Anyway, this is not to say we need to solve all this before proceeding: I very much agree with you that the new class organization makes sense regardless, even if all we end up getting to is categoricals or the ability to define an int24 type (I want an int2... we deal with 2-bit radio VLBI data all the time...).

as they are quite different: the categorical example shows that with the NEP ideas it will be easier to give new meaning to more or less arbitrary bytes, while the units example shows that one can add a new layer of meaning.

The "what are the bytes" bit seems pretty straight forward, but I've been a bit skeptical about implementing units with dtypes, so an example would be good to have. @njsmith Liked the idea of using dtypes to handle units, so I'd like to know what he had in mind for the implementation.

See, so maybe the solution is actually to delete Quantities here. I have thought about this (and generally the array type a bit, just to proof it: https://gist.github.com/seberg/02dc72cc126c365694f7e470504f0c03). But did not quite come to a conclusion yet for units, the end just gets more fuzzy... I still think they should be on the DType, I see two counter points:

1. It can be nice to have "induced" methods for DTypes, we currently have no such notion. I had the idea of arr.elem.method() before, but without a (possibly dynamic) subclass/mixin indicated by the dtype this is not possible (a bit like what pandas does, the dtype provides some functions, but often is only used to find the correct array subclass). The current np.ndarray is arguably one with "induced" methods for Numerical purposes.
2. Units on the array use np.ndarray as their computational kernel. That limits how they can cast datatypes, etc. with other NumPy datatypes (for example datetime64 already has a unit). But, it has the advantage that a unitless Unit can obviously work with every UFunc. If Unit was part of the dtype, this is less obvious. There may be ways to design a protocol to make it work, but for starters the user may have to do something (and know about it), like units.ensure_unitless_loop_exists(ufunc).

In any case: It should not matter for this NEP!, if units or even half of the above dtypes I list are things that might be better as subclasses, I can just delete it from the motivation for now, and try to put it into another NEP (of course that might create even more NEPs :)). It also means that NEP 42, should maybe not use datetime as an example, since that is unit+nullable_int...

We still need all of the things I want to sign of by accepting this NEP, so do you agree that deleting it (or making it into a "maybe" paragraph), would be a way to get this somewhere that we can agree to accepting this?

We still need all of the things I want to sign of by accepting this NEP, so do you agree that deleting it (or making it into a "maybe" paragraph), would be a way to get this somewhere that we can agree to accepting this?

Please add rather than delete. It's clear that there's unknowns, if it makes you feel better to state that explicitly - please do.

Hmm, right I will add some short usage examples, of course. I do not want to sound overly annoyed/defensive here, but I think I have to talk with Ralf in person about the real doubts. (I am honestly surprised that I may still have to convince anyone seriously about the general idea!)

I think you're still missing the point of asking for examples / use cases / requirements. It's not that I have specific doubts about the implementation. It's just that I know from experience that creating a design without concrete use cases in mind inevitably leads to a sub-optimal (best case) or useless/actively-harmful (worst-case) design. As @mhvk said, it is simply useful - both for us and for yourself.

If you look at software design methodologies you will see that whether it's waterfall, agile, model-driven engineering, or whatever - they all start from use cases and/or requirements.

Describing use cases or external/functional requirements typically takes between 1/2 and 1/6-th of the amount of content of describing the design. It's also the content that's most useful in the future:

• design can be changed again since it's mostly internal, use cases will remain
• use cases can also be used to derive documentation from
• finally, they give the future developers context, so when they make more changes they won't go against original intent

It may be worth asking for some help here. E.g. one of the Pandas devs (@TomAugspurger or @jorisvandenbossche any interest, or can you suggest someone?) may be willing to write up one or two examples of where this helps Pandas; @mhvk or @ngoldbaum or @hgrecco may be willing to adapt a units example of astropy.units/unyt/pint where they can see this having value.

@seberg - my sense would be to move the examples into two categories, one that include things like int24 and (I think) Categorical that would become directly possible with a new dtype system, and others (units only?) that may become much easier, but for which we simply do not yet know. I agree with @rgommers about not deleting pieces! Indeed, if at all possible, try to work some of the discussions into the NEP, as it will help avoid future confusion.

Thanks everyone for your feedback and help. I am sure there will be a few fixups necessary (espeically links to the other NEPs), but we have to start somewhere and I would like to send this to the mailing list.

(Been following the discussion about where the (meta)classes and instances are from the sidelines - but moving this to the main thread since it is only somewhat sideways related - though really the rest should perhaps have been in the main thread as well, to avoid loosing it!)

As units came up again: I do wonder more and more whether we should not separate more clearly the different parts of "the dtype" in the three different tasks that it could encode, since those have a clear hierarchy: lowest is byteorder, needed to make the bytes interpretable, next is the real data type, giving (usually numerical) meaning to the bytes, and last is units, giving physical meaning (for numbers at least).

This makes me wonder whether it is useful to think of each of these as their own classes and instances: after all, swapping the byte order from >i4 to <i4 is no different from doing it from >f4 to <f4. In the context of the discussion about where the leaves of the hierarchy are, it might help to think of, e.g., <i4 as actually having two attributes, one of the data type - i4 - and one of the ordering - <4 - each of which could be instances of a respective class. If so, the dtype proper would just be a collection of pieces that describe the data, and would be a class whose primary purpose is to organize order, first feeding bytes to the byte-ordering instance, then on to the data-type instance, and then to a possible unit instance.

Note I don't mean that this would necessarily be exposed to the user! But in this logic, li4 = np.dtype('<i4') would only be an instance of np.dtype itself; it would be its data_type attribute (or whatever) that is an instance of a DTInteger class and a byte_order attribute that would be an instance of ByteOrder.

I guess bottom line is that it may be good to think about the attributes of a dtype as instances of their own classes. It might even be needed/useful, for some weird architecture that, say, stores strings in bunches with a given ordering (revenge of BCDIC?). With a clear division of tasks, using dtypes for different unit systems may also be easier.

Hmm, well there are two things, here. First is defining a specific storage layer (and methods, maybe even C-level) for certain things. If this was C++ or Python this would be more clear, the DType would simply have attributes with their own methods (type), such as an actual ByteOrder or Unit class. In C, I guess this just means having the attribute since you do not have methods in C, you will just have to use functions, such as a byteswap() function we can provide.

That is, of course you can (and should) use "wrapping" logic:

class Unit(np.dtype):
     # Typical np.dtype slots, such as itemsize
     wrapped_dtype_instance = None
     unit = None

(EDIT: note in the above Unit has the job of filling in itemsize from correctly from wrapped_dtype_instance, we cannot "inherit" it)

Now, actual composition, such as you have a dtype, and then you "add" byteorder to it by inheriting from it or so. I do not see how that is going to work well. Of course in C++ or so it may work, so if you define your DType in C++ I guess you could compose it using inheritance. I am not really convinced that we should go there. We can provide things like a fast DType agnostic byteswap function of course.
Actual compisition would have to figure out things like a <U4 to >U6 cast needs to have the byte-order part do byteswapping, and the unicode part elongate the string, while an encoding part may have to change the encoding... I suppose I am just not convinced it is worth giving me headaches ;). We could do things like include unit as a C-level slot, i.e. a protocol to exchange/access DType units, even if they are implemented by completely different people. I do not think any of the design stops us from this? The composition, however, may actually be tricky to tag on later. Also it may require allowing you to subclass at least DTypeMeta, but I do not have any squirms with that as such.

@seberg - your class Unit(np.dtype) is a nice way to think of it. Might there be a place for a similar class dtype(byte_order)? Though that does feel strange - subclasses are not generally meant to add a lot of new functionality, and certainly the byte_order one can hardly be something attached to an array.

p.s. I realize part of the discussion is about C implementation, but it may be good to think it through as far as possible as if it were a python problem, and only then really move to how it is implemented in C. But I admit writing this while never really have written the C classes, and therefore just hoping to stay away from it!

p.s.2 It might also be much more productive to just start looking at the actual implementation you have been working on...

I think I like to see it as an orthogonal problem. It is not really (fully) maybe though. We need a protocol, with the most basic features that a datatype must expose. As soon as we have that we could try to make it easier for users to write new datatypes.
Right now, I do not think NumPy really has many things that make it easier. Byteorder, sure, but it is trivial enough that there is not much to compose.

The main advantage of having composition, would be if we would allow to "extend" existing dtypes. E.g. take an int8 and say: This is also an int8 but with a weird storage pattern. Of course you can already do that by just adding a new datatype.
Composition or not, the main question is about dispatching IMO. If you have a loop for int8 should you allow downstream to say: Any ufunc written for int8 will work for my int8_with_weird_storage (since it can cast). That is something I currently only have half plans for support. I.e. as long as we do not allow subclassing, a multiple-dispatching system cannot pick up the int8 loop for your int8_with_weird_storage dtype. You could argue that is an advantage of type numbers, but I am not sure I would even agree. It is an argument for allowing (very careful!) subclasses.

As for composition, I suppose I just get back to the part where:

1. Fine we can think about it for the Python API
2. You are free to use a language composition support when writing your own custom dtype (e.g. in C++).

Although, yes, we could do that, we could have a Numeric type, which adds an attribute for issigned or a ByteOrderderd which adds a newbyteorder method. Aside form the limitation of static C language, I do not see much of an issue (although not sure I see much of a gain, at least immediately either, but maybe that is just because I am not used to working like that much).

On the subclassing front, I actually realized that I managed to miss a hole. I need subclassing in my model already (internally to NumPy), to represent the current type aliases: np.dtype(np.intp) is not np.dtype(np.longlong) (even if the current type number is the same). That also means I have to be prepared for it in UFunc dispatching. That is not a big issue as such, though. And the subclassing should be trivial and OK, since both are functionally equivalent.\

EDIT: OTOH, maybe it can just be a detail of the instance... Since the character code currently already is (although that would mean there is not a direct scalar class <-> DType class mapping.

. I need subclassing in my model already (internally to NumPy), to represent the current type aliases: np.dtype(np.intp) is not np.dtype(np.longlong)

No, but I think np.dtype(np.intp) is np.dtype(np.long)?

Sorry, you are right of course... We do add loops for both integers after all, so this is probably OK, at least for now.

@eric-wieser gentle ping. I know I have been slow on this as well, but it would be nice if we can continue that instance vs. flat hierarchy discussion. If there is some ground work I can do to help on that front, happy to do that.