Image inversion of uint64

Description

There is a new problem with image inversion. As far as I can tell, it is not related to issue #3028 nor to PR #3030, as it concerns unsigned integer inversion. It is pretty enigmatic to me ... here it is:

In [1]: import numpy as np

In [2]: from skimage.util import invert

In [3]: imin = np.array([0, 1], np.uint64)

In [4]: imout = invert(imin)

In [5]: imtest = invert(imout)
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-5-ab8c45624279> in <module>()
----> 1 imtest = invert(imout)

/Users/twalter/pycharm/scikit-image/skimage/util/_invert.pyc in invert(image, signed_float)
     63         inverted = ~image
     64     elif np.issubdtype(image.dtype, np.unsignedinteger):
---> 65         max_val = dtype_limits(image, clip_negative=False)[1]
     66         inverted = max_val - image
     67     elif np.issubdtype(image.dtype, np.signedinteger):

/Users/twalter/pycharm/scikit-image/skimage/util/dtype.pyc in dtype_limits(image, clip_negative)
     48         warn('The default of `clip_negative` in `skimage.util.dtype_limits` '
     49              'will change to `False` in version 0.15.')
---> 50     imin, imax = dtype_range[image.dtype.type]
     51     if clip_negative:
     52         imin = 0

KeyError: <type 'numpy.uint64'>

My observations:

Among the unsigned integers, the problem only appears for np.uint64.
It is a key error coming from skimage.util.dtype_range. Indeed, we get the following:

In [7]: from skimage.util.dtype import dtype_range

In [8]: imin.dtype.type in dtype_range
Out[8]: True

In [9]: imout.dtype.type in dtype_range
Out[9]: False

And this one I love:

In [10]: imin.dtype.type
Out[10]: numpy.uint64

In [11]: imout.dtype.type
Out[11]: numpy.uint64

In [12]: imin.dtype.type is imout.dtype.type
Out[12]: False

Way to reproduce

[If reporting a bug, please include the following important information:]

Code example
Relevant images (if any)
Operating system and version: MacOS High Sierra, Version 10.13.3
Python version: Python 2.7.14 :: Anaconda custom (64-bit)
scikit-image version (run skimage.__version__): 0.14dev

Sorry for the noise.

OK, I think this is the cause:

In [24]: for k, _ in dtype_range.items():
    ...:     print('{}: {}'.format(k, hex(id(k))))
    ...:     
    ...: 
<class 'numpy.bool_'>: 0x7fec8449a5e0
<class 'numpy.uint8'>: 0x7fec84499740
<class 'numpy.uint16'>: 0x7fec844995a0
<class 'numpy.uint32'>: 0x7fec84499400
<class 'numpy.uint64'>: 0x7fec84499260  <-----------------
<class 'numpy.int8'>: 0x7fec84499f60
<class 'numpy.int16'>: 0x7fec84499dc0
<class 'numpy.int32'>: 0x7fec84499c20
<class 'numpy.int64'>: 0x7fec84499a80
<class 'numpy.float16'>: 0x7fec84498f20
<class 'numpy.float32'>: 0x7fec84498d80
<class 'numpy.float64'>: 0x7fec84498be0

In [25]: hex(id(imin.dtype.type))
Out[25]: '0x7fec84499260'

In [26]: hex(id(imout.dtype.type))
Out[26]: '0x7fec844990c0'

Changing
inverted = max_val - image
to
inverted = image.dtype.type(max_val) - image
solves the problem on my PC.

Although, it seems that a part of the issue is on numpy side.

@soupault

+1 @soupault Would you mind opening a ticket there?

@sciunto

@sciunto sure, see numpy/numpy#11020.

This is a bug in scikit-image, although admittedly one we (numpy) led you into.

This returns false on all platforms

all(t in skimage.util.dtype.dtype_range for t in [np.intc, np.int_, np.longlong])

@eric-wieser

I see:

In [25]: imout.dtype.type is np.uintp
Out[25]: True

I guess, a bugfix will be a bit more complex :).

@eric-wieser Could you point me at the definition code for [np.intc, np.intp, np.int_, np.longlong] and their unsigned analogues? It appears that some of these types are defined as aliases to the np.[u]int{32, ...}, while others are completely different objects (like np.uintp, in this case).

@soupault

@soupault: The following list is all the unique integer types in numpy

np.byte
np.short
np.intc
np.int_
np.longlong
np.ubyte
np.ushort
np.uintc
np.uint
np.ulonglong

These map directly to the underlying C types, char, short, int, long, long long.

Every other type is just an alias to one of these (ie, np.t1 is np.t2).

Unfortunately to make it more confusing, some of these types have the same __name__ - but this shouldn't matter in your implementation.

The definition for these names is spread between the C code and python code. You can see a glimpse of the source in numpy/numpy#10151, which touches this file

Well this is nice and messy. On the plus side, as I've mentioned before, I don't think functions like invert really make sense for uint64 or int64 images — the max value is enormous! My suggested fix would be to catch the keyerror and issue a friendlier error message: "image inversion is only supported for the following dtypes:".

@jni

@jni: That doesn't make any sense. You can't justify supporting long (64 bits) but not long long (64 bits) because "the max value is enormous" for long long - clearly the same argument could be applied to long.

Also, on windows this manifests itself in 32-bit integers anyway (where int and long overlap)

That doesn't make any sense.

I agree :).
Ok, so basically, np.XintYY abstraction is leaky, and we have to switch to the types above in dtype_range. I'm taking a look if this would affect anything.

See #3050 for the fix.

Navigation Menu

Repository navigation

Image inversion of uint64 #3043

Description

Way to reproduce

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Image inversion of uint64 #3043

Description

Description

Way to reproduce

Activity

soupault commented on Apr 30, 2018

soupault commented on Apr 30, 2018

soupault commented on Apr 30, 2018

sciunto commented on Apr 30, 2018

soupault commented on Apr 30, 2018

eric-wieser commented on Apr 30, 2018

soupault commented on May 1, 2018

eric-wieser commented on May 1, 2018

jni commented on May 1, 2018

eric-wieser commented on May 1, 2018

soupault commented on May 1, 2018

soupault commented on May 2, 2018

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions