Skip to content

PyPy3 TypeError: 'numpy.float64' objects are unhashable #8887

New issue
New issue
Closed
Closed
PyPy3 TypeError: 'numpy.float64' objects are unhashable#8887

Description

@peterjc
peterjc
opened on Apr 3, 2017

Cross reference PyPy issue: https://bitbucket.org/pypy/pypy/issues/2479/numpyfloat64-objects-are-unhashable

I'm logging this here for reference, and on the off chance that someone on the NumPy team could say why numpy.float64 might behave so differently from the other size floats under PyPy3.

Expected behaviour, showing Python 3.5.0 in 64-bit Linux:

$ python3
Python 3.5.0 (default, Sep 28 2015, 11:25:31) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-16)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> print(numpy.__version__)
1.12.1
>>> f = 123.456
>>> for t in [np.float, np.float16, np.float32, np.float64, np.float128]:
...     new = t(f)
...     print(hash(new), hash(new)==hash(f), t.__name__)
... 
1051464412201451643 True float
1008806316530991227 False float16
1051467367688700027 False float32
1051464412201451643 True float64
1051464412201451643 True float128
>>> quit()

Observed behaviour using PyPy3.5 v5.7 beta from https://bitbucket.org/squeaky/portable-pypy/downloads/pypy3.5-5.7-beta-linux_x86_64-portable.tar.bz2

$ ~/Downloads/pypy3.5-5.7-beta-linux_x86_64-portable/bin/pypy
Python 3.5.3 (b16a4363e930f6401bceb499b9520955504c6cb0, Mar 21 2017, 12:36:24)
[PyPy 5.7.0-beta0 with GCC 6.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
And now for something completely different: ``the world doesn't want us to
know''
>>>> import numpy as np
>>>> print(np.__version__)
1.12.1
>>>> f = 123.456
>>>> for t in [np.float, np.float16, np.float32, np.float128, np.float64]:
....     new = t(f)
....     print(hash(new), hash(new)==hash(f), t.__name__)
....     
1051464412201451643 True float
1008806316530991227 False float16
1051467367688700027 False float32
1051464412201451643 True float128
Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
TypeError: 'numpy.float64' objects are unhashable
>>>> quit()

The other float types seem to hash fine, but numpy.float64 does not under PyPy3.5 v5.7 beta.

Activity

  1. eric-wieser commented on Apr 3, 2017

    @eric-wieser
    eric-wieser
    on Apr 3, 2017
    Member

    I think the difference is:

    >>> print(np.float16.__hash__)
<slot wrapper '__hash__' of 'numpy.float16' objects>
>>> print(np.float32.__hash__)
<slot wrapper '__hash__' of 'numpy.float32' objects>
>>> print(np.float64.__hash__)
None

    Because np.float64.__bases__ contains float, so this defers to float.__hash__.

    Could this be a bug to do with hashing and multiple inheritance in PyPy?

  2. njsmith commented on Apr 3, 2017

    @njsmith
    njsmith
    on Apr 3, 2017
    Member

    This is almost certainly related somehow to the fact that float64 is a subclass of float (the built-in type), and the other floatXX classes aren't.

  3. eric-wieser commented on Apr 3, 2017

    @eric-wieser
    eric-wieser
    on Apr 3, 2017
    Member

    I'm super confused here though - why is it legal to call hash(float64)? Test case:

    >>> class MyFloat(float):
	__hash__ = None

	
>>> mf = MyFloat(1)
>>> nf = np.float64(1)
>>> mf.__hash__
>>> hash(mf)
Traceback (most recent call last):
  File "<pyshell#31>", line 1, in <module>
    hash(mf)
TypeError: unhashable type: 'MyFloat'
>>> nf.__hash__
>>> hash(nf)
1

    I thought that __hash__ == None always made the type unhashable? How does numpy pull of having __hash__ == None and still being hashable?

  4. rlamy commented on Apr 3, 2017

    @rlamy
    rlamy
    on Apr 3, 2017
    Contributor

    I tried debugging this in PyPy, but got bogged down trying to understand how exactly numpy initialises the types.

    The situation on CPython is quite dodgy, since float64 has both a tp_hash and a descriptor for __hash__ that returns None - which causes the above behaviour: hash(nf) looks up the slot tp_hash, not the descriptor.

  5. njsmith commented on Apr 3, 2017

    @njsmith
    njsmith
    on Apr 3, 2017
    Member

    @eric-wieser: I'm not sure what's going on with float64.__hash__ returning None, but special methods are internally way more complicated than that. Most methods are natively entries in a python dict; special methods are natively slots in a C struct, and there's a whole elaborate layer of code to map back and forth between the python and C representations and try to make them look the same. Part of this is that when defining a class from Python, the type object constructor has a special case where it checks for __hash__ == None and then initializes the tp_hash slot in a special way. But float64 is defined in C, so it doesn't go through that code path at all. And then of course pypy adds a whole additional layer of complexity...

    @peterjc: you should also check your intXX objects -- I bet either int32 or int64 has the same bug (depending on how large your platform's long is).

  6. rlamy commented on Apr 3, 2017

    @rlamy
    rlamy
    on Apr 3, 2017
    Contributor

    @njsmith Nope, the issue exists only with np.float64 and np.complex128 (on a 64-bit Linux).

  7. njsmith commented on Apr 3, 2017

    @njsmith
    njsmith
    on Apr 3, 2017
    Member

    descriptor for __hash__ that returns None

    wut

  8. rlamy commented on Apr 3, 2017

    @rlamy
    rlamy
    on Apr 3, 2017
    Contributor

    @njsmith Sorry, it's not a descriptor, it's just a None value in the class dict (which is the conventional way of indicating "not hashable").

  9. eric-wieser commented on Apr 3, 2017

    @eric-wieser
    eric-wieser
    on Apr 3, 2017
    Member

    Found the bug, patch incoming...

  10. rlamy commented on Apr 3, 2017

    @rlamy
    rlamy
    on Apr 3, 2017
    Contributor

    The culprit is the DUAL_INHERIT macro in multiarraymodule.c::setup_scalartypes(). ATM, it calls PyType_Ready, and then sets tp_hash. But since tp_hash is NULL when PyType_Ready is called, it sets __dict__['__hash__'] = None and tp_hash to PyObject_HashNotImplemented, which is later overwritten.

    It seems that the fix is just to set tp_hash first.

  11. eric-wieser commented on Apr 3, 2017

    @eric-wieser
    eric-wieser
    on Apr 3, 2017
    Member

    @rlamy: Indeed, that's the patch I have

  12. juliantaylor
    closed this as completedin #8888on Apr 3, 2017

  13. eric-wieser commented on Apr 3, 2017

    @eric-wieser
    eric-wieser
    on Apr 3, 2017
    Member

    @peterjc: Does this fix PyPy?

  14. rlamy commented on Apr 3, 2017

    @rlamy
    rlamy
    on Apr 3, 2017
    Contributor

    @eric-wieser Yes, it fixes it.

  15. peterjc commented on Apr 3, 2017

    @peterjc
    peterjc
    on Apr 3, 2017
    ContributorAuthor

    Excellent work everyone - thank you. I'll look forward to this working in the next NumPy release :)

  16. peterjc
    added a commit that references this issue on Apr 4, 2017

    Skip NaiveBayes test if numpy.float64 is unhashable.

    c696fb8
  17. MarkusPiotrowski
    added a commit that references this issue on Oct 31, 2017

    Skip NaiveBayes test if numpy.float64 is unhashable.

    e719856
  18. dlsun
    added a commit that references this issue on Sep 5, 2018

    Change is_hashable()

    e5da427
  19. dlsun
    added 2 commits that reference this issue on Oct 8, 2018

    beginning overhaul of random processes (#67)

    43e7dad

    Merging Kevin's changes (#76)

    c948e36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions