# NEP 41 Background
This is an additional document for NEP 41 to explain the decision of what
DTypes are and why they are split as they are.
Eric Wieser has done a nice summary [here](https://gist.github.com/eric-wieser/49c55bcab744b0e782f6c2740603180b#what-this-could-mean-for-dtypes)
detailing some of the properties, and coming to a slightly different
solution.
Partially, it is an extension of [this discussion](https://github.com/numpy/numpy/pull/15506#discussion_r390016298).
With some distance to the original discussion, some parts become clearer,
and this document is now a followup detailing to more clearly describe the
common parts of the decision and the difference between NEP 41 and Eric's
document.
*Disclaimer: I have no formal background in this field, so terms may be misused.*
(This was another very rough thing, thinking a bit about what an array is
https://gist.github.com/seberg/02dc72cc126c365694f7e470504f0c03 — rather than dtypes.)
## Summary
This document describes design considerations for what a `type(np.dtype("int8"))`
should be in NumPy.
It further introduces some background for *parametric* dtypes and how
this affects the design choice taken in NEP 41.
There are several choices for how we can define and most importantly
implement dtypes. A dtype gives meaning to the elements stored in an array,
but *how* it gives meaning to it should be structured is not
trivial.
NEP 41 defines the following structure:
```python
>>> type(np.dtype)
DTypeMeta
>>> f64 = np.dtype("float64")
>>> print(type(f64))
np.dtype[float64]
>>> issubclass(type(f64), np.dtype)
True
>>> print(type(np.dtype(">float64")))
np.dtype[float64]
```
Where the important distinction is:
* `dtype` *describes* the data. One way to think of this are storage variations:
* `>float64` and `<float64` describe the same thing, but store it differently.
* `np.dtype("S5")` and `np.dtype("S10")` share the same implementation,
but are limited to different length strings.
* From a more theoretical point of view the `dtype` instance is part of each
elements instance.
* `type(dtype)` implements most, or even all behaviour. We also write DType class
in NEP 41:
* There is a DType class associated to every scalar class and implementing
the same behaviour as that scalar class.
* From a type theoretical point of view, the DType class *is* the type.
This is convenient, because it aligns with it being a Python `type`.
Which largely means that most functionality created by the user is cannot be specific to
a `dtype` instance.
Note that we will resolve much of this later, be realizing that many functions
require methods which can specialize for various array shapes.
<details>
<summary>
N.B.: "Kernels" (DType Methods) resolve this lack of specificity!
<p>
</summary>
:::warning
**Kernels (DType Methods) resolve the lack of specificity:**
A Python method is itself a class with a special `__get__` method. When the `.` operator fetches a method from a class, it binds it by calling `__get__`. We can write our own "method" class:
```
from functools import partial
def func(*args, **kwargs):
print(args, kwargs)
class Method:
_func = func # function written by user
def __get__(self, instance, owner=None):
return partial(self._func, instance)
# Note that `__get__` is called and Method works:
class MyObject:
meth = Method
bound_method = MyObject().meth # calls __get__
bound_method() # prints self
```
We can (and in casting to some extend do) translate this to
saying that a "kernel" (DType method), also can have
a `__get__` method:
```
class Kernel:
_func1 = function_for_little_endian
_func2 = function_for_big_endian
def __get__(self, dtype):
if dtype.byteorder == "<":
return _func1
else:
return _func2
```
Making it possible to write functions specific to a a certain instance
(just as it would be possible in pure Python even *without* inheritance).
Although, the *organization* of these specific function occurs
one level higher.
This `__get__` method doesn't need to be exposed right away, but it is a clear
extension if "kernels" define at least almost all functionality of array values.
(Which is the clear goal with respect to ufuncs and casting.)
Once we expose the `__get__` function, the DType implementer could even create their own system of inheritance by looking up behaviour on the dtype instance itself (I guess similar to monkey patching a Python instance?).
:::
</details>
If we look inside an Python float, it has the structure:
```
float instance:
type = float # The instance stores its type
value = 4.
```
The `dtype` itself is a Python type (spelling `type(np.dtype("float64"))` as `Float64`):
```
Float64 instance:
type = Float64
byte_order = "<"
```
While the individual array value can be thought of:
```
Float64 value:
dtype = np.dtype("float64") # Stored on the array
value = 4.
```
And expanding the `dtype` (ignoring one indirection):
```
Float64 value:
type = Float64
byte_order = "<"
value = 4.
```
After this expansion `type = float` defines how `value` is interpreted and behaves
for the Python float example.
In the case of the array value, it is here `Float64` which acts as the type.
The below considerations try to formulate why this decision is taken in NumPy.
It is not the only approach and requires the step of "expanding" the
`dtype` above to have access to the `type`.
:::info
This document gives some background of why the above choice was taken in NEP 41.
If you feel this choice is straight forward, you may rather be interested in
implementation details.
:::
## Types and DTypes
### Definition of Types
NumPy data types are types (or data types) as used in Python or computer science
in general and for example defined in the [Wikipedia article](https://en.wikipedia.org/wiki/Data_type#Definition).
Specifically, here, we will define a type by the following two properties:
1. **Value space:** A set of possible values which instances can take
2. **Behaviour:** Functions (methods) that can be applied to those values.
First, for example, the value space of the Python `float` type are all (machine level)
double precision numbers.
This means that the `float` *instance* (`f = float(1)`) represents exactly one
value out of this value space.
Second, the `float` type defines *behaviour* for those instances in the form of
operators or methods, such as:
```
class float:
def __nonzero__(self):
# Written on the C-level
return self != 0
def conjugate(self):
return self
```
In particular it defines the behaviour for all python *operators*
(such as `>=`, `-`, `abs`, `bool`, `repr`, …).
Operators are special in that they define a set of functionality which every
Python `type` supports (although `type`s can choose not to implement all operators).
The way in which Python implements this, is by defining `float` as itself an instance
of `type`:
```
>>> isinstance(1., float)
True
>>> isinstance(float, type) # read PyType_Type for type
True
```
### Implementation of types in Python
Since the use of the word `type` can easily mix up `type` as opposed to an instance of `type`
such as `float` or `int`, we will use `PyType_Type` (the C-API name) instead.
`PyType_Type` is the value space of all possible `type` instances (`float`, `int`, ...).
As an aside, `PyType_Type` does endow certain behaviour to its instances.
First,`PyType_Type` itself is a `type` and, second, subclassing and inheriting is
functionality specific to `type` instances.
In short, the Python system has a three level hierarchy:
1. `PyType_Type`:
* Describes how Python classes (`type` instances) store certain behaviours (operators).
* Endows python classes with behaviour (functions) (e.g. subclassing, but also operators themselves)
2. `type` instances (classes) such as `float`:
* Describe how the class instance stores its values (thus defining its *value space*)
* Endows its instances with behaviour (functions) by defining operators
3. Class instances, such as `float(1)`:
* Represent a single value of the value space defined by the `type`, for example
a single floating point number.
Reversing this hierarchy may make the picture more clear:
* The *instance* is simply a set of bytes (with some additional information).
In C, this means it is a structure:
```
struct PyFloatObject {
/* Some information for garbage collection */
PyTypeObject *tp_type = ;
/* Further storage defined by `float` (PyFloat_Type): */
double value;
}
```
where the struct always contains the `type` and some additional information for garbage collection
and is then followed by arbitrary data.
* The `type` in the above `struct` is `float` (`PyFloat_Type`),
and knows how to interpret the data stored inside `PyFloatObject`.
- The value can be extracted by knowing the value space, which is defined by
`PyFloatObject`.
- Further meaning is endowed by defining operators, in C for example:
```
struct PyFloat_Type {
/* Some information for garbage collection */
PyTypeObject *tp_type = &PyType_Type;
/* Further information defined by `PyType_Type` */
.tp_repr = function_defining_the_representation;
}
```
* `PyType_Type` defines *how* `float` has to store the operators/functionality.
Since Python is dynamically typed, functionality has to be looked up every time when
using an object.
For example, to print an arbitrary Python object the `repr` operator uses roughly
the following code at its core:
```C
/* `PyObject_Repr` shortened to the absolute minimum */
PyObject *
PyObject_Repr(PyObject *v)
{
PyTypeObject *type_of_v = v->tp_type;
/* Use the type to find the correct function to call: */
reprfunc *repr_function_for_v = type_of_v->tp_repr;
/* Call the repr function with `v` as argument */
return *(repr_function_for_v)(v);
}
```
Which starts with the instance (e.g. `float(1)`) and then goes through the above steps
to give meaning and functionality to the instance.
### Data types in NumPy
The Python `type` as a type gives meaning to an arbitrary Python object:
`type(instance)` defines both value space and behaviour of the `instance`
and every Python object is the instance of a type.
NumPy works with `elements` stored in an array. How each element of an array
should be interpreted is stored in the arrays `dtype` which is exposed as
`arr.dtype`.
The `dtype` serves the same purpose for array elements as the Python `type`
does for its instances, by explaining the value space and behaviour
of each element.
However, we will make two distinctions from Python types:
1. Python instance know their type: `((PyObjet *)v)->tp_type`, while in NumPy
the *array* stores the dtype: `arr.dtype`.
To endow meaning and behaviour to the memory stored inside an array, it is
necessary to know the dtype in advance.
This is *not* necessary for Python objects, the `PyObject_Repr` function
can return the representation of any Python object without knowing its type
in advance.
2. NumPy dtypes are *parametric*. These parameters make them more precise
about how the values are stored in the array and what they represent.
When compared to the Python (scalar) types, we additional have:
* *Storage information:* Specifies more precisely how the data is stored.
In particular almost all NumPy dtypes have a byte order.
For example: `<float64` vs. `>float64` represent identical values and
behaviour for most practical purposes.
* *Parameters*: Compared with the Python type, the dtype sometimes often
represents only part of the same *value space*:
* `np.dtype("S4")` represents a subset of `np.dtype("S8")` which is a subset of
what the Python (scalar) type `np.string_` can represent (arbitrary length).
* `np.dtype("datetime64[ms]")` represents the subset of datetimes
with millisecond precision, while the same scalar Python type `np.datetime64`
is used for all possible units.
We will go into more details about the *parametric* property and how it
affects what `dtypes` are after briefly discussing the relevance of
passing the `dtype` explicitly (Point 1).
#### Signature of dtype methods
Unlike a Python object, the the element stored in a NumPy array does not
store its type.
Thus all methods working on a value must be passed the dtype explicitly.
Instead of:
```
class float:
_value : double # defined in C
def __nonzero__(self):
return self._value != 0
```
where `((PyObjct *)self)->tp_type` (or `self.__class__`) is the type,
a method working with an array element would have to be written:
```
dtype Float64:
def __nonzero__(dtype, element):
# Where `element` is raw memory
value = dtype.interpret_memory(element)
return value != 0
```
Sometimes `dtype.interpret_memory` can be hard-coded, but for example
in the case of strings, the length cannot be deduced from the raw memory
alone.
An alternative to passing the `dtype` itself is to pass the array
and lookup the dtype using `arr.dtype` (before NEP 41, this was the typical model).
While practical, there seems to be no reason to pass use the full array information.
Avoiding this makes the `np.ndarray` object replaceable.
NumPy dtypes could be used for alternative array objects (such as sparse arrays).
:::success
Before NEP 41, NumPy almost never passed the dtype explicitly, instead it would pass the
array object and rely on `arr.dtype` to interpret the memory.
The reason for the above choice is to make the dtype independent from the array object as
much as possible.
In summary, a Python object stores its own type while a NumPy array element
does not know its type.
For Python `self` consists of `(type, value)` and it is not necessary to
know `type` (dynamic typing).
But the NumPy array elements require knowledge of the `dtype`.
The Python instance `self` is translated to the knowledge of both
`dtype` and `element`.
*We can say that the instance of an array value is the tuple `(dtype, element)`.*
This tuple replaces the Python `self`.
:::
#### Parametric dtypes represent are Python instances
To consider the implication of parametric dtypes in more details
it helps to look at some operations provided by NumPy:
```
>>> np.promote_types("S4", "S10")
dtype('S10')
>>> np.promote_types("<f8", "S10")
dtype('S32')
>>> np.promote_types("<f8", "S")
dtype('S32')
```
Or casting:
```
>>> np.can_cast("f8", "S")
```
are operations in which dtypes behave more like instances
than the `type` of elements.
There are no array elements involved in these operations.
Consider also a possible `__promote__` method to implement
the above `np.promote_types` operator for a `Float64` dtype:
```
dtype Float64:
def __promote__(self, other):
if self == other:
return self
if isinstance(other, String):
if other.length < 32:
return String(32)
return other
return NotImplemented # defer
```
Note the necessity of the `isinstance` check which is used to handle
parametric dtypes.
A perhaps more concrete example is array coercion:
```
np.array(["short", "much longer string"], dtype="S")
```
Requires that we find the correct dtype for `"short"` and `"much longer string"`
(before using promotion).
If we define this operation on the dtype it seems more naturally
defined as a classmethod:
```
class String:
@classmethod
def __discover_descriptor__(cls, value):
return cls(len(str(value)))
class Unit:
@classmethod
def __discover_descriptor__(cls, value):
# support discovering unit from datetimes:
if isinstance(value, np.datetime64):
return cls(value.unit)
raise TypeError("only supports datetime64")
```
which return `dtype` instances.
Additionally, some functionality is similarly useful on the class
itself, for example a dtype representing Physical units could have:
```
>>> dtype = Float64Unit("km") # kilometers
>>> dtype.to_si() # convert to SI units (here meters)
Float64Unit("m")
```
and all current dtypes have the method:
```
>>> np.dtype("<i8").newbyteorder()
dtype('>i8')
```
:::success
NumPy dtypes are themselves instances with behaviour defined in a "dtype meta"
(the `np.dtype` class, or a subclass of it).
NEP 41, calls these `DType` class (capitalized) vs. dtype instance.
The important take-away is that we have a `type(arr.dtype)` which
is a generally a subclass of `np.dtype`.
Further, the above examples for dtype discovery show how this
`DType` class can be a useful place to define behaviour of the
dtype instance.
Although a `dtype` serves the purpose of a `type` with respect to the
array elements, it also is an instance with respect to operations on
the dtype parameters (string length, unit) in promotion or dtype discovery:
| | dtype operations (parameters) | Array value |
| ------------- | ------------------------------ | --------------------- |
| `DType` class | Type | |
| `dtype` | Instance | Type |
| `(dtype, element)` | | Instance |
:::
While the `dtype` defines the value space and behaviour of an array value,
until now we did not define *how* this behaviour is defined.
### Implementing behaviour on parametric dtypes
Parametric dtypes present a challenge of whether behaviour
is implemented specific to the parameters, or generic over
all parameters.
In Python, one pattern is to store the parameters as part of the type
instance: A string knows its length, and a datetime knows its unit.
In this case behaviour is implemented generic to that parameter:
There is a single `type` object.
The alternative pattern in Python is a class factory, probably together
with an abstract base class (ABC).
For example for a maximum-length string (similar to the string dtype),
we can write:
```
class StringBase:
length = None
def __init__(self, value):
# Strip to the maximum length:
self.value = value[:self.length]
def __nonzero__(self):
# can be specialized as part of subclassing
return len(self.value) > 0
def MaxLengthString(length):
"""Create class specialized to a specific length.
This function could be a MetaClass.
"""
# We could specialize `__nonzer__` here:
return type(f"MaxLength{length}String", (StringBase,),
{"length": length})
```
as opposed to moving it into the parametric instance:
```
class MaxLengtString:
def __init__(self, value, max_length):
self.max_length = max_length # (not really necessary for scalar)
self.value = value[:max_length]
def __nonzero__(self):
# cannot be specialized
return len(self.value) > 0
```
In the above case the ABC `StringBase` *defines* the value space and behaviour fully
(although Python will not enforce this).
While the dynamically created subclasses represent a part of its value space
and provide the implementation of the behaviour (which may be inherited).
The *implementation* of how the value space is represented
and behaviour is implemented may be specific to the parameters.
The two methods use two different approaches:
```
Value Space+Behaviour Parameters Value Storage
Python type: <-------- type ---------> <------------ instance -------------->
Python with ABC: <------ ABC ------><--------- type ----------> <-- instance --->
```
We should review why one may wish to chose the second option:
* The parameters are large compared to the values. (Not relevant for arrays, since we
assume there are many elements and only one `dtype`.)
* Specialize implementations: It provides a logical hierarchy to implement e.g.
faster (specialized) versions, which do not require inspecting the parameters.
For types this mainly allows inheritance of *implementation* as well as specialization.
For dtypes the difference is larger, since it poses the question of where
the *implementation* of the array value behaviour is stored.
#### Two models of parametric dtypes
Translating this to dtypes, there are two potential models.
(It should be noted that we will largely resolve the differences between
these models later.)
1. The dtype is considered an extension of the instance and stores
parameters but does not define behaviour (NEP 41)
* Behaviour is (mainly) defined and implemented on `type(arr.dtype)`
* We have the following mental model:
```
Python scalar NumPy array
------------- -----------
scalar instance <-> array instance + dtype instance
type(scalar) <-> type(array) + type(dtype)
```
* `np.dtype(bool)` is effectively a singleton instance representing
its type. While `type(arr.dtype)` defines value space and behaviour.
2. The dtype corresponds to the Python type, since it is the type of an array element:
* Behaviour is (mainly) defined on `arr.dtype` but may be inherited.
* We have the following mental model:
```
Python scalar NumPy array
------------- -----------
scalar instance <-> array instance with dtype
type(scalar) <-> type(array) + dtype
```
* It is not immediately clear how the abstract `np.dtype("S")` is represented.
Reproducing and extending the above structure, we get:
```
Value Space+Behaviour Parameters Value Storage
Python type: <-------- type ---------> <------------ instance -------------->
Python with ABC: <------ ABC ------><--------- type ----------> <-- instance --->
Option 1: <----- type(dtype) -----> <----- dtype ----> <--- element --->
Option 2: <-- dtype base ---><-------- dtype ---------> <--- element --->
```
In code, when implementing an operator such as setting a single element from
a Python value:
```
def setitem_Option_1(dtype, element, value):
type(dtype).__setitem__(dtype, element, value)
def setitem_Option_2(dtype, element, value):
dtype.__setitem__(dtype, element, value)
```
:::warning
Current, NumPy looks uses `dtype->f->setitem` (on the C level),
which can be interpreted both ways.
It could be like Option 2, if `dtype->f` was specific to each
dtype instance.
However, effectively `dtype->f` is only specific to each type
number, so that it is much like the type lookup `type(dtype)`
in Option 1.
:::
:::info
We have two approaches to define the behaviour of parametric dtypes.
The first is to model "parameters" as part of the Python instance.
In this case the scalar `instance` consists of `parameters+value` and
translates to `dtype+element`.
The `dtype` (instance) represents the parameters while behaviour is
implemented (mainly) on `type(dtype)` corresponding to the python type.
The alternative view is that parametric dtypes should be the main
implementer of behaviour and thus take the role of the `type`.
In this view `dtype` corresponds to the Python `type`.
Note that the difference between these two mental models are fairly subtle.
:::
In my opinion, the difference between the two models does not matter much
from a user perspective (or I could not think of any real difference).
The reason is that in both cases users will usually work with dtype
instances, or with opaque dtype factories.
For example we may want to write physical untis as:
```
arr_in_m = np.arange(10) * Unit("m")
arr_in_km = arr_in_m.astype(Unit("km"))
```
But it is of little importance how `Unit` is implemented or where
behaviour is implemented.
The only relevant observation is probably that Option 1 enforces
that all of the NumPy scalars (and as per NEP 41 all scalars with
dtypes) have a single well defined `DType` class (`type(dtype)`)
which represents the same value space and behaviour.
This `DType` class may be abstract, meaning that more precise,
concrete, subclasses may be defined.
This can, and probably should, also be enforced in Option 2
since it provides a useful abstraction to the user:
each scalar `type` corresponds to a `DType` type/class.
:::success
If we formulate the desire that every scalar type (as currently defined
in NumPy) is represented by a DType class to provide users with a uniform
abstraction of what can be stored in an array and how it is stored:
```
scalar type <-> type(dtype) or DType class
```
In that case, Option 2 *must* implement the same DType classes as
Option 1 (i.e. a DType class for `np.bool_`, `np.int8`, `np.int16`, …)
and the difference between both options becomes largely conceptual
from a DType implementers point of view and non-existing for an
end-user.
:::
The above requirement enforcing DType classes for every scalar type
aligns with enforcing that every value space and behaviour must
have a corresponding DType class.
I.e. `type(np.dtype(np.bool_))` must exist because `np.dtype` can
also represent integers, floats, etc.
Option 1 must align value space and behaviour with the DType class
(`type(np.dtype(...))`), while Option 2 can be forced to do the same.
:::success
From here on, we will use the NEP 41 definitions.
A concrete DType class *represents* a specific value space and behaviour.
We still have a subtle difference between Option 1 and Option 2,
and this difference does not clash:
1. Option 1 *implements* elementwise behaviour on the DType class
2. Option 2 *implements* elementwise behaviour on the dtype instance.
It must be noted that NEP 41 and 42 do not preclude the possibility of moving
definition of element behaviour into the dtype instance!
:::
#### Resolving the difference