You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
BUG: Inconsistent type resolution for 0d arrays #10322
This seems wrong to me, but appears to be deliberate.
It seems the goal here is to interpret python scalars as loosely-typed (in the presence of arrays). This handles thing like:
letting integer literals choose between signed and unsigned
letting number literals have smaller representations than float64 and int_ if they'd fit
However, the method is too greedy, and also (in the presence of arrays) discards type information from:
np.number instances with an explicit type, like np.int64(1) and np.float32(1)
0d ndarray instances, by decaying them to their scalars
The problem is that PyArray_ResultType (and the ufunc type resolver) has lost the information about where it's array arguments came from, and whether their types are explicit or implicit.
Yeah, this stuff is pretty annoying... But I think we need to do a major release (hehe, we can do those anyway ;)) to change it another time.
I suppose we could try to aim for 0d arrays be interpreted as arrays here in principle. In practice you need to also fix the ufunc machinery to actually also signal the scalar when it was one, I am sure.
Also as long as 0d arrays are second class citizens compared to scalars, I do not really see this improving much personally. My personal opinion is still that we should aim for (in the very long run) 0d arrays and scalars (not mutable normally) and things like arr.sum() would be a scalar, while arr.sum((0, 1)) would be a 0d array. I actually believe that it is transparent, though its probably just consistent and that makes me think it is transparent ;).
Hehe, true, but equality with floats is a tricky beast in any case and nobody is suggesting to change it for the python scalar case I believe, possibly for np.equal(np.arange(0., 0.3, 0.1, dtype=np.float32), np.float64(0.1)) or at least for np.equal(np.arange(0., 0.3, 0.1, dtype=np.float32), np.array(0.1)).
Which both are of course a bit annoying since it breaks the "casting everything first with np.array" is the same thing identity of course….
It does seem to me array scalars should always be treated like arrays.
My ideal future would have gotten rid of numpy scalars altogether - use array scalars mostly and python numbers when needed - but perhaps I am missing what they can be useful for.
would actually return True for the second value (but == [0.1] would not and neither would np.equal(arr, 0.1)! (This can also be slightly crazy for integers, more so than currently. It also won't be quite trivial when you look at things like the / operator where np.result_type is not always correct, so that non-trivial promotion is necessary, probably on the ufunc even though the ufunc might never use it itself...)
We had discussed this a bit (two weeks ago). I think we had some consensus, that we should probably just do it (but it will probably need testing).
By now, I am almost convinced that simply changing our "casting/promotion" rules to fix this everywhere (also in normal ufunc calls), might be the best thing. When scalars are involved, that would usually lead to higher precision (less scary), when Python scalars are involved, errors/warning on conversions should notify users of any changes (so this is also unlikely). The worst thing is probably:
np.float32(3) + 3. # the np.float32 could be created by indexing
Which previously would give float64, but with a change would give float32... I tried this a bit actually. SciPy doesn't really notice (in two tests or so it might, aside from the sparse tests comparing the result with NumPy). astropy did not notice, pandas I am not sure, it has its own type resolution, which probably needs to deal with it. (i.e. it seemed noisy, but probably a lot of that is one failure fairly low).
Compared to that, warning on future precision change is pretty darn noisy (largely due to this issue).
For anyone else coming to this issue trying to understand what is happening, the example in the OP is a bit confusing, because it demonstrates the issue indirectly via a function that returns a bool. It's easier to see via a function that directly returns the promoted value, like add.
Just to make it easier to find again, NEP 50 is the current attempt to fix these inconsistencies. (And is fairly far along, although it is still possible to modify details.)
Activity
[-]BUG: Inconsistent type resolution for float32 arrays[/-][+]BUG: Inconsistent type resolution for 0d arrays[/+]seberg commentedon Jan 4, 2018
Yeah, this stuff is pretty annoying... But I think we need to do a major release (hehe, we can do those anyway ;)) to change it another time.
I suppose we could try to aim for 0d arrays be interpreted as arrays here in principle. In practice you need to also fix the ufunc machinery to actually also signal the scalar when it was one, I am sure.
Also as long as 0d arrays are second class citizens compared to scalars, I do not really see this improving much personally. My personal opinion is still that we should aim for (in the very long run) 0d arrays and scalars (not mutable normally) and things like
arr.sum()would be a scalar, whilearr.sum((0, 1))would be a 0d array. I actually believe that it is transparent, though its probably just consistent and that makes me think it is transparent ;).mhvk commentedon Jan 4, 2018
Duplicate of #9194. It is probably worth noting that one can avoid the problem by also passing a
dtypeto the ufunc.It also worth noting that the solution is not that obvious. E.g., the following would, generally be the expected result:
At least, I think we'd get even more bug reports if the following was the default!
eric-wieser commentedon Jan 4, 2018
Right, but I'd argue that's because it uses the
floattype for0.1. I'd also expect:seberg commentedon Jan 4, 2018
Hehe, true, but equality with floats is a tricky beast in any case and nobody is suggesting to change it for the python scalar case I believe, possibly for
np.equal(np.arange(0., 0.3, 0.1, dtype=np.float32), np.float64(0.1))or at least fornp.equal(np.arange(0., 0.3, 0.1, dtype=np.float32), np.array(0.1)).Which both are of course a bit annoying since it breaks the "casting everything first with
np.array" is the same thing identity of course….eric-wieser commentedon Jan 4, 2018
@mhvk: I've leave it to you to decide whether to close this or the old one
mhvk commentedon Jan 4, 2018
Closed #9194, so we can discuss in one place.
It does seem to me array scalars should always be treated like arrays.
My ideal future would have gotten rid of numpy scalars altogether - use array scalars mostly and python numbers when needed - but perhaps I am missing what they can be useful for.
BUG: Fix crashes when using float32 values in uniform histograms
BUG: Fix crashes when using float32 values in uniform histograms
seberg commentedon Mar 6, 2021
So my current though was that all examples here would be "fixed", but the Python operators might stay unmodified:
would actually return
Truefor the second value (but== [0.1]would not and neither wouldnp.equal(arr, 0.1)! (This can also be slightly crazy for integers, more so than currently. It also won't be quite trivial when you look at things like the/operator wherenp.result_typeis not always correct, so that non-trivial promotion is necessary, probably on the ufunc even though the ufunc might never use it itself...)seberg commentedon Mar 24, 2021
We had discussed this a bit (two weeks ago). I think we had some consensus, that we should probably just do it (but it will probably need testing).
By now, I am almost convinced that simply changing our "casting/promotion" rules to fix this everywhere (also in normal ufunc calls), might be the best thing. When scalars are involved, that would usually lead to higher precision (less scary), when Python scalars are involved, errors/warning on conversions should notify users of any changes (so this is also unlikely). The worst thing is probably:
Which previously would give float64, but with a change would give float32... I tried this a bit actually. SciPy doesn't really notice (in two tests or so it might, aside from the
sparsetests comparing the result with NumPy).astropydid not notice,pandasI am not sure, it has its own type resolution, which probably needs to deal with it. (i.e. it seemed noisy, but probably a lot of that is one failure fairly low).Compared to that, warning on future precision change is pretty darn noisy (largely due to this issue).
asmeurer commentedon Apr 14, 2021
For anyone else coming to this issue trying to understand what is happening, the example in the OP is a bit confusing, because it demonstrates the issue indirectly via a function that returns a bool. It's easier to see via a function that directly returns the promoted value, like
add.Also, if the resulting value does not fit in float32, the result is upcast (value-based casting)
Finally, the issue only concerns 0-d arrays, not scalar objects
seberg commentedon Dec 20, 2022
Just to make it easier to find again, NEP 50 is the current attempt to fix these inconsistencies. (And is fairly far along, although it is still possible to modify details.)
seberg commentedon Jan 20, 2024
Closing, weak promotion/NEP 50 should make these consistent.