Repository navigation
Shift operator cycles for int32 and int64 (Trac #1856) #2449
Description
Original ticket http://projects.scipy.org/numpy/ticket/1856 on 2011-06-02 by trac user klasj, assigned to unknown.
The bitshift operations, both right and left cycles when they are shifted more bits than
the length of the datatype. This happens for both int32 and int64 but not for int16,
unsigned integers works as expected. In the example below version 1.6.1 is used but
this is also present in at least 1.5.0.
In [1]: import numpy as np
In [2]: np.right_shift(np.int32(8),32)
Out[2]: 8
In [3]: np.right_shift(np.int64(8),64)
Out[3]: 8
In [4]: np.right_shift(np.int16(8),16)
Out[4]: 0
In [5]: np.left_shift(np.int32(8),32)
Out[5]: 8
In [6]: np.left_shift(np.int64(8),64)
Out[6]: 8
In [7]: np.left_shift(np.int16(8),16)
Out[7]: 524288
In [8]: np.version.full_version
Out[8]: '1.6.1.dev-08953ea'
Activity
numpy-gitbot commentedon Oct 19, 2012
@charris wrote on 2011-06-03
Looks like a compiler thing. On my fedora amd64 system
gives
I suspect the problem is that 64 is equal to the register width and so a cyclic shift is used; the problem doesn't occur for a shift of 63. There is probably a compiler flag that can be used to change this...
numpy-gitbot commentedon Oct 19, 2012
@charris wrote on 2011-06-03
See [http://tinyurl.com/3vqv7qz this]. Apparently the Intel hardware masks the shift bits. On 32 bit machines 5 bits are used, max shift 31, and it seems on 64 bit architectures 6 bits are used.
We should be able to fix this, but the c shift operators are not precisely defined anyway. They can be cyclic or not, arithmetic or logical, or at least that used to be the case. I don't think numpy has a spec for this.
numpy-gitbot commentedon Oct 19, 2012
trac user gregsmith_tor wrote on 2012-06-20
I would suggest rename to 'numpy shift operators behave as underlying C operator', since that's the issue here.
The C shift operator is well defined, at least, it has well-defined areas of doubt and uncertainty. The shift amount can range from 0, to one less than the width of the integer type being shifted. So, to get a consistent behaviour you need to check (in the numpy C implementation) the value of the shift and produce a zero (or -1 for >> of negative) if out of range. I don't think it's appropriate to put the burden of that on the numpy user; even worse, to potentially induce users to rely (perhaps accidentally) on the cyclic behaviour or any other result which happens to occur on a specific machine.
There's another issue with shifts and numpy scalars:
I ran into this issue when a simple scalar value (which had been read from a numpy array) did not produce the expected result - it was something like x << 62, where x was 7, and the result was -0x4000000000000000 instead of the expected 0x1C000000000000000. The reason was that 'x' was a numpy.int64 scalar value, which doesn't behave as a python int does.
This is an issue I've run into before with numpy - since numpy started returning 'numpy scalar' objects from array indexing, instead of python types, we have a situation where these types act very much like the builtin types, and thus it's common that they are passed far into generic python code (code which is not aware of numpy, for instance), and these occasional cases where they don't act like the builtin types are problematic. Some of the problems have been due to python bugs, where the python core code was neglecting to attempt coercion of certain things to integers or longs.
I fixed my shift bug by explicitly converting to int certain values that I was reading from numpy arrays.
I suggest that
(1) when shift operators are applied to numpy arrays, the code should check for out-of-range shifts and handle appropriately (i.e. the result should not depend on the compiler/machine); result may overflow on left shifts, but otherwise is the same as for python shifts.
(2) when shift operators are applied to numpy scalar integers, that the result should be as for (1), but if a left shift would overflow, the return value should be a numpy int or long type with the proper value. So, operations on numpy scalars would be consistent with operations on python types.
P.S. @charris as I recall right shift of signed types in C is not guaranteed to sign extend; this is considered platform dependent, but I don't know of any machine where it doesn't, and it's probably reasonable to decline full numpy support of such machines.
charris commentedon Feb 20, 2014
Still a problem. I think we should fix this in the code with a size check on the shift amount.
RosterIn commentedon Jul 14, 2019
What is the status of this?
eric-wieser commentedon Jul 14, 2019
gh-13739 is the most recent proposed fix