You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
It is maybe not what you'd guess at first blush, but it is deliberate, at least, because the floating point dtype is the only type that contains any possible np.uint64 value plus any possible int value.
In your example, I probably would have preferred an overflow than an upcast to floating point, but I'm sure there are arguments for the upcast behavior.
It's not quite as simple as that. For example a uint64+uint64 operation results in a uint64 (with an overflow warning) although if we applied @matthew-brett's reasoning we should be returning a float64 as well.
Rather, it's a confluence of several factors. The implementation of the numpy.add() ufunc that underlies these routines only has type-specific implementations where the two arguments are of the same type. So the ufunc system needs to cast the arguments to a common type. One of the arguments is signed, so both arguments need to be casted to a signed type. The smallest signed type that can represent values up through the maximum range of uint64 is float64 (Note! Not all uint64values can be represented as float64 floats! Precision is lost! But it's better than int64 where the whole upper half of the values are lost.). We can't really look at the value to determine the output type because the same logic needs to apply to arrays as well. This is an unfortunate result, but the tower of integer types needs to top out somewhere.
This behaviour also buged my program. Why it couldn't simply overrun leaving responsibility for user. Casting silently to float64 is bad... If this is "normal" behaviour, it should be at least documented in official documentation.
Reacted by Neil Girdhar, Eleftherios Zisis, Hoàng-Giang Bùi and Alph4rd
When numpy integral types are added to each other, they wrap around but preserve the output type, but when you add a Python integer to them, the return type changes. So there is no consistency between numpy integers and Python integers.
The smallest signed type that can represent values up through the maximum range of uint64 is float64…
To be consistent with this type-promotion logic, adding two float32s should cast to float64 since you don't know whether the mantissa of a float32 can represent the whole result. Similarly, dividing a float32 by 2 should cast it to float64 since you don't know the exponent can hold the result.
This is an unfortunate result, but the tower of integer types needs to top out somewhere.
That's true, but I don't feel like it should be topping out on a non-integer type.
To be consistent with this type-promotion logic, adding two float32s should cast to float64 since you don't know whether the mantissa of a float32 can represent the whole result. Similarly, dividing a float32 by 2 should cast it to float64 since you don't know the exponent can hold the result.
You misread me. The ability of the dtype to represent the result never factors in. The logic is based solely on the ability of the dtype to represent both of the inputs.
I mean, you can continue extending the integer hierarchy ad infinitum if you are willing to write software implementations of all of the ufunc operations (i.e. in such a way that they don't rely on C supporting them as native datatypes). There is no existing integer dtype candidate.
That doesn't really resolve the problem, just hide one instance of it in a special case. The root question is what does uint64 + int64 (and vice versa) do?
I'm beginning to wonder if the answer is just "error out". At least it
would make everyone equally unhappy?
I guess this might be an easier pill to swallow if we also started treating
Python scalars as having indeterminate width/sign that they copy from
whatever the first type they encounter is.
I agree with at least "warning" out the same way you warn out when you run into an overflow. It's really important here to have a clear message that states:
that the arguments were converted to a common type and that the common type of int and uint64 is whatever, and that the whatever can result in loss of information, and
that the message can be suppressed by casting the arguments.
I want to add that using float64 as the top of the integer tower essentially prioritizes protecting 52-ish higher-order bits whereas using int64 prioritizes 63-ish lower-order bits. Is there any reason to believe that higher-order bits are more important? If anything, most computations with integers are done around zero. So, in my opinion, the lower order bits are the most important.
If you end up going with the rule that: int64 + int returns int64 and uint64 + int returns uint64, then the warning above could be selectively emitted when there is actual loss of information, just like overflow warnings are selectively emitted — they aren't emitted whenever the arguments theoretically could have overflowed.
I don't think it's useful to keep talking about the combination of int64 + int (or uint64 + int). The ufunc dtype logic never sees int. The int first gets converted to a numpy dtype (without knowing the context that it is in an operation with another operand), then the two numpy dtypes interact to come to a common dtype. The question is what you want uint64 + int64 to do.
It is maybe not what you'd guess at first blush, but it is deliberate, at least, because the floating point dtype is the only type that contains any possible np.uint64 value plus any possible int value.
Activity
matthew-brett commentedon Jan 27, 2016
It is maybe not what you'd guess at first blush, but it is deliberate, at least, because the floating point dtype is the only type that contains any possible
np.uint64value plus any possible int value.For example, what would you expect for this?:
dalexander commentedon Jan 27, 2016
OK, that's fair.
In your example, I probably would have preferred an overflow than an upcast to floating point, but I'm sure there are arguments for the upcast behavior.
rkern commentedon Jan 27, 2016
It's not quite as simple as that. For example a
uint64+uint64operation results in auint64(with an overflow warning) although if we applied @matthew-brett's reasoning we should be returning afloat64as well.Rather, it's a confluence of several factors. The implementation of the
numpy.add()ufunc that underlies these routines only has type-specific implementations where the two arguments are of the same type. So the ufunc system needs to cast the arguments to a common type. One of the arguments is signed, so both arguments need to be casted to a signed type. The smallest signed type that can represent values up through the maximum range ofuint64isfloat64(Note! Not alluint64values can be represented asfloat64floats! Precision is lost! But it's better thanint64where the whole upper half of the values are lost.). We can't really look at the value to determine the output type because the same logic needs to apply to arrays as well. This is an unfortunate result, but the tower of integer types needs to top out somewhere.sem-geologist commentedon Jan 31, 2016
This behaviour also buged my program. Why it couldn't simply overrun leaving responsibility for user. Casting silently to float64 is bad... If this is "normal" behaviour, it should be at least documented in official documentation.
NeilGirdhar commentedon Jan 24, 2017
When numpy integral types are added to each other, they wrap around but preserve the output type, but when you add a Python integer to them, the return type changes. So there is no consistency between numpy integers and Python integers.
To be consistent with this type-promotion logic, adding two
float32s should cast tofloat64since you don't know whether the mantissa of afloat32can represent the whole result. Similarly, dividing afloat32by 2 should cast it to float64 since you don't know the exponent can hold the result.That's true, but I don't feel like it should be topping out on a non-integer type.
rkern commentedon Jan 25, 2017
You misread me. The ability of the dtype to represent the result never factors in. The logic is based solely on the ability of the dtype to represent both of the inputs.
NeilGirdhar commentedon Jan 25, 2017
Okay, that makes sense. But as you say, even
float64can't represent all of the inputs. There has got to be a better (integer) candidate?rkern commentedon Jan 25, 2017
I mean, you can continue extending the integer hierarchy ad infinitum if you are willing to write software implementations of all of the ufunc operations (i.e. in such a way that they don't rely on C supporting them as native datatypes). There is no existing integer dtype candidate.
NeilGirdhar commentedon Jan 25, 2017
Two other solutions:
int64 + intreturnint64and similarlyuint64 + intreturnuint64. Wrap and warn on overflow just like it does when adding homogenous types.int64 + intreturn a dtype ofobjectand use Python'sint.I prefer the first solution.
rkern commentedon Jan 25, 2017
That doesn't really resolve the problem, just hide one instance of it in a special case. The root question is what does
uint64 + int64(and vice versa) do?njsmith commentedon Jan 25, 2017
NeilGirdhar commentedon Jan 25, 2017
I agree with at least "warning" out the same way you warn out when you run into an overflow. It's really important here to have a clear message that states:
intanduint64is whatever, and that the whatever can result in loss of information, andI want to add that using
float64as the top of the integer tower essentially prioritizes protecting 52-ish higher-order bits whereas usingint64prioritizes 63-ish lower-order bits. Is there any reason to believe that higher-order bits are more important? If anything, most computations with integers are done around zero. So, in my opinion, the lower order bits are the most important.If you end up going with the rule that:
int64 + intreturnsint64anduint64 + intreturnsuint64, then the warning above could be selectively emitted when there is actual loss of information, just like overflow warnings are selectively emitted — they aren't emitted whenever the arguments theoretically could have overflowed.rkern commentedon Jan 25, 2017
I don't think it's useful to keep talking about the combination of
int64 + int(oruint64 + int). The ufunc dtype logic never seesint. Theintfirst gets converted to a numpy dtype (without knowing the context that it is in an operation with another operand), then the two numpy dtypes interact to come to a common dtype. The question is what you wantuint64 + int64to do.NeilGirdhar commentedon Jan 25, 2017
Okay since we're blind to context, then this is my proposal:
intbecomesint64and warns on overflow, anduint64 + int64givesint64and warns on overflow.eric-wieser commentedon Apr 25, 2017
This duplicates #5745, which is older
atbenmurray commentedon Jul 1, 2020
I would expect overflow, personally
Change dtype for `splice` from uint64 to int64