Default integer size in numpy and numba on Windows [continuation of #2643 issue]

I apologize for having provided an inadequate example in #2643. In fact, I meant the following, that when function is jitted the type size for all locally created integer arrays is assumed to be 8 byte instead of 4 byte on Windows.

import numpy as np
import numba as nb
arr = np.arange(10)
print("Initial: ", arr.dtype)

def npfun(arr):
    arr = arr + np.array([10])
    return arr

print("numpy  : ", npfun(arr).dtype)

@nb.njit
def jitfun(arr):
    arr = arr + np.array([10])
    return arr

print("njit   : ", jitfun(arr).dtype)

#----------------------------------

Initial:  int32
numpy  :  int32
njit   :  int64

So numba does not preserve the type size. While it is pretty easy to avoid this situation by providing dtype = np.int32 (~~dtype='i4'~~ doesn't work either :-)) for all integer arrays. This behavior is not compliant with numpy on Windows and is rather annoying. FWIW, that in numpy the default size for np.int_ was 8 bytes (where appropriate) and not compiler-dependent. Currently in numpy the size of np.int_ is defined to be the same as C's long ... and in VS long is still 4 byte.

I understand that this special case for numba will add additional costs for the maintenance. But it seems to me that this design choice (8 byte - default for np.int_) should be at least noted in the Docs (maybe some Windows related section?), because it affects the expectations of type checkers and also affects the needed signatures (i8[:](i4[:]) instead of i4[:](i4[:]) by default).

This compilation will fail on Windows:

@nb.njit(['i4[:](i4[:])'])
def jitfun(arr):
    arr = arr + np.array([10])
    return arr

with:

...
TypingError: Failed at nopython (nopython frontend)
No conversion from array(int64, 1d, C) to array(int32, 1d, A) for '$0.9', defined at None

To overcome this you always need to append dtype = np.int32 :

@nb.njit(['i4[:](i4[:])'])
def jitfun(arr):
    arr = arr + np.array([10], dtype=np.int32)
    return arr

This behavior to a certain extent also affects the adoption of numba as a compliant compiler.

p.s.: I am rather unfamiliar with GitHub and don't know if comment on closed issue pings devs, so I start a new one.

The difference in typing behavior you see here is due to the inability of numba to see the runtime value like numpy does. The np.array([val]) will look at the content in its argument. Here, it will inspect the value of val and find the smallest dtype that can contain the given value. However, numba cannot deduce type base on runtime value. In your case, the value is available in compile-time but special casing constant value has limited usage; esp, when this issue can be avoided easily by providing the dtype argument.

The numpy doc for the dtype argument of np.array (https://docs.scipy.org/doc/numpy/reference/generated/numpy.array.html) says :

The desired data-type for the array. If not given, then the type will be determined as the minimum type required to hold the objects in the sequence. This argument can only be used to ‘upcast’ the array. For downcasting, use the .astype(t) method.

So, if you have a larger value for val, it will choose a bigger dtype. For example:

In [1]: import numpy as np

In [2]: np.array([10]).dtype
Out[2]: dtype('int32')

In [3]: np.array([0xffffffff]).dtype
Out[3]: dtype('int64')

The numpy behavior above is different on 32-bit vs 64-bit platform. On 64-bit platform, you get int64 for 10.

Also, there is no guarantee what dtype you get from np.array([10]). There is some heuristic in it to avoid using int8 or int16 for this case. I'm not sure if the behavior is different on other CPU architecture.

The way numpy chooses the dtype can be weird sometimes. For example, when you mix signed and unsigned int types like below, you get float64:

In [9]: a = np.array([10], dtype=np.int64)

In [10]: b = np.array([10], dtype=np.uint64)

In [11]: np.array([a[0], b[0]]).dtype
Out[11]: dtype('float64')

So, the only portable way to guarantee that the array has int32 dtype is by supplying the dtype keyword argument.

The numpy behavior above is different on 32-bit vs 64-bit platform. On 64-bit platform, you get int64 for 10.

This is not always True, and this is precisely the main point of this issue. See numpy/numpy#8310 "why is int 32 bit on my 64 bit system?" And there it was written that np.int_ or int as dtype uses the C type long. The long type is however just 4bytes on Windows (even 64 bit): https://msdn.microsoft.com/en-us/library/9c3yd98k.aspx.
There is also somewhat related question on SO https://stackoverflow.com/questions/384502/what-is-the-bit-size-of-long-on-64-bit-windows.

So, the only portable way to guarantee that the array has int32 dtype is by supplying the dtype
keyword argument.

I agree with this in general. But since the usual workflow with numba is simply put a decorator above your function and performance magic will happen this behavior is rather tricky. Also because now you provide wheels on Windows and numba becomes more available for ordinary users like me this behavior shoots. There are two reasons why:

It is not expected in this simple case.
It is not documented.

That's why I actually opened this question and see two ways to fix:

If it is possible, not too hard to fix, will not greatly complicate the maintenance and some numba developer has the time and the will to fix it -> Fix it.
In other case to document this behavior, rationale for this design choice and advise how to make code portable. -> Document it.

p.s.:

The numpy doc for the dtype argument of np.array (https://docs.scipy.org/doc/numpy/reference/generated/numpy.array.html) says ...

I will ask about this (if this is generally true for integers) on numpy ML, because I thought that it will try to use np.int_ and if it will not suffice numpy will try to use bigger dtype if possible.

This is not always True, and this is precisely the main point of this issue. See numpy/numpy#8310 "why is int 32 bit on my 64 bit system?" And there it was written that np.int_ or int as dtype uses the C type long. The long type is however just 4bytes on Windows (even 64 bit): https://msdn.microsoft.com/en-us/library/9c3yd98k.aspx.
There is also somewhat related question on SO https://stackoverflow.com/questions/384502/what-is-the-bit-size-of-long-on-64-bit-windows.

Good to know. I think that's just more reason for user to always specify the dtype.

I don't think we could fix this issue properly. Making it work with constant may appear simple but it quickly gets complicated in conditional code; such as:

c = 10
if cond:
    c = x
arr = np.array([c])

In such cases, we can no longer rely on the constant info at compile time. If we support typing base on constant value, the occurences of the if block in the example would change the typing.

So, I think we should go for documenting the differences in our typing and suggest user to provide the dtype argument. We should even consider showing a warning when a user construct an array without specifying the dtype.

c = 10
if cond:
    c = x
arr = np.array([c])
In such cases, we can no longer rely on the constant info at compile time. If we support typing base on constant value, the occurences of the if block in the example would change the typing.

I don't know the internals of numba and how it operates under the scene, my apologies (I want to fix this in the near future). But what is the difficulty to infer the type from variable like numpy does?

We should even consider showing a warning when a user construct an array without specifying the dtype.

As for me, a note (maybe in several places) will be enough for this issue. I've looked through questions on SO and on numba issues and it seems that despite the growing popularity of numba this question did not arise or did not bother anyone. In addition, emitting a Warning for a strange choice of Windows on 64-bit systems is rather annoying :-)

For what I wish, and If anyone has a will, to start a discussion on numpy ML about the default integer size on 64-bit systems (to switch from current policy of C's long to the 64-bit variant appropriate to an underlying OS). I've been using numpy regularly for the last two years and never did meet the advantages of this choice, only a few inconveniences. In addition, it seems to me that it causes inconvenience to the upstream projects (like numba) and does not follow least surprise principle. While I don't have any serious reasons for this change, maybe you (I mean numba and others) have. Also you know the internal difficulties to be compliant with this behavior.

For this exapmple:

The way numpy chooses the dtype can be weird sometimes. For example, when you mix signed and unsigned int types like below, you get float64:
In [9]: a = np.array([10], dtype=np.int64)

In [10]: b = np.array([10], dtype=np.uint64)

In [11]: np.array([a[0], b[0]]).dtype
Out[11]: dtype('float64')

As for me I will prefer if it will cast to object dtype although for many it is a dubious choice. I've asked about this on numpy ML type promotion rules for integers but it seems there is no interest in discussing.

I don't know the internals of numba and how it operates under the scene, my apologies (I want to fix this in the near future). But what is the difficulty to infer the type from variable like numpy does?

Unlike Numpy, Numba has to make the typing decision before the function is executed (via static analysis of the function bytecode). At that stage, if x is passed as an argument, all numba may know about it is its type, which may be int64 for example. This will affect the typing of c, since numba doesn't know if the condition will execute or not. It must assume that the type of c at np.array([c]) is the common type between 10 and int64.

NumPy makes typing decision during function execution. At that point, it sees the exact value of c and can adjust typing for it.

In addition, it seems to me that it causes inconvenience to the upstream projects (like numba) and does not follow least surprise principle. While I don't have any serious reasons for this change, maybe you (I mean numba and others) have. Also you know the internal difficulties to be compliant with this behavior.

Numba, being a compiler with re-implementation of NumPy functionality, will have many deviations from NumPy and it will be hard to avoid them. All we can do is to minimize them and document the remaining differences. It's probably unreasonable to make NumPy follow the restrictions that Numba has since they have the flexibility of being a library inside an interpreter. We have talked with the NumPy developers about making NumPy more JIT friendly but no one has enough time to make such a significant rework of NumPy.

Finally for this specific problem, I think NumPy developers would just suggest you use the dtype argument to ensure you get the exact dtype you want.

It seems that there is an ongoing discussion on this topic is numpy/numpy#9464 "Default int type is platform dependent". Just to link it.

Navigation Menu

Repository navigation

Default integer size in numpy and numba on Windows [continuation of #2643 issue] #2729

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Default integer size in numpy and numba on Windows [continuation of #2643 issue] #2729

Description

Activity

sklam commented on Feb 8, 2018

godaygo commented on Feb 9, 2018

sklam commented on Feb 12, 2018

godaygo commented on Feb 13, 2018

sklam commented on Feb 13, 2018

godaygo commented on Mar 21, 2018

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions