You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hello @hajimes thank you so much for providing this murmur3 implementation in Python and for all the work you do in open source; we really appreciate being able to use your library!
I was recently investigating some compatibility issues with the output produced by mmh3 and a Go library we were using internally. I asked the question on Stackoverflow and got a response back:
It looks like the order of the two uint64s returned by the 128-bit algorithm is reversed between the two libraries; but it's simple enough to modify the returned results in either Go or Python to produce compatible hashes.
I was wondering; would you like me to open a PR to update the README with the compatibility information? Is there any other docs I should update in the PR?
Additionally, if there is any way to reverse the order order of the uint64s returned by murmur3 (e.g. with an argument to hash128 or hash_bytes) I'd be happy to open a PR for that as well. Let me know how you'd like to proceed!
Thank you very much for giving me a lot of examples (I checked your Stack Overflow question), which provided important clues for me to locate the problem.
Short answer
On little-endian machines, murmur3 in Go returns a 128-bit byte sequence that is incompatible with the original MurmurHash3 code and Python mmh3.
This is the first time I ever wrote a Go code, so feel free to modify it so that it suits your code style!
Long answer
The original MurmurHash3 code written by Austin Appleby is endian-sensitive.
The MurmurHash3_x64_128 function in C++ calculates two unsigned 64-bit integers and store them on memory.
Let's assume the result was [0o00112233, 0o44556677] (0o is an octal notation).
On little-endian machines, least significant bits (LSB) are written first, so their actual representations on memory is
<-- low address .. high address -->
3322110077665544
This is what MurmurHash3_x64_128 (in C++) and mmh3.hash_bytes (in Python) returns (on little-endian machines).
(Note that mmh3.hash128 returns 0o4455667700112233 for the above case)
However, Sum in Go murmur3 stores two unsigned 64-bit integers in big-endian.
So given two integers [0o00112233, 0o44556677], it returns the byte sequence
<-- low address .. high address -->
0011223344556677
which differs from the result in the original C++ and Python on little-endian platforms.
My gist in the above is basically the little-endian version of Go murmur3 Sum.
Edit: forgive me for "Short answer" is not that short haha
Thank you so much for taking the time to respond to my question. Your Go snippet is well written; thank you very much! I will contact the authors of the Go Library to see their thoughts on updating their package with your suggestion. Thank you also for posting the temporary notice about compatibility -- I think that will be very helpful to other Go/Python users!
I also wanted to add for future readers of this issue that the Python workaround similarly flips the low and high address:
Activity
hajimes commentedon Apr 4, 2023
Hello @bbengfort!
Thank you very much for giving me a lot of examples (I checked your Stack Overflow question), which provided important clues for me to locate the problem.
Short answer
On little-endian machines, murmur3 in Go returns a 128-bit byte sequence that is incompatible with the original MurmurHash3 code and Python mmh3.
Specifically, the
Sumfunction in https://github.com/spaolacci/murmur3/blob/master/murmur128.go seems to be the source of incompatibility.To solve the issue, I wrote a Go gist here, whose input is the result of
Sum128(orSum128WithSeed) and output is a 128-bit byte sequence that should be compatible with the original and mmh3.https://gist.github.com/hajimes/b174e3cd7b0d0c14be97aa39010f2932
This is the first time I ever wrote a Go code, so feel free to modify it so that it suits your code style!
Long answer
The original MurmurHash3 code written by Austin Appleby is endian-sensitive.
The
MurmurHash3_x64_128function in C++ calculates two unsigned 64-bit integers and store them on memory.Let's assume the result was
[0o00112233, 0o44556677](0ois an octal notation).On little-endian machines, least significant bits (LSB) are written first, so their actual representations on memory is
This is what
MurmurHash3_x64_128(in C++) andmmh3.hash_bytes(in Python) returns (on little-endian machines).(Note that
mmh3.hash128returns 0o4455667700112233 for the above case)However,
Sumin Go murmur3 stores two unsigned 64-bit integers in big-endian.So given two integers
[0o00112233, 0o44556677], it returns the byte sequencewhich differs from the result in the original C++ and Python on little-endian platforms.
My gist in the above is basically the little-endian version of Go murmur3
Sum.Edit: forgive me for "Short answer" is not that short haha
Add a temporary notice for compatibility with murmur3 in Go (#46)
bbengfort commentedon Apr 4, 2023
Thank you so much for taking the time to respond to my question. Your Go snippet is well written; thank you very much! I will contact the authors of the Go Library to see their thoughts on updating their package with your suggestion. Thank you also for posting the temporary notice about compatibility -- I think that will be very helpful to other Go/Python users!
I also wanted to add for future readers of this issue that the Python workaround similarly flips the low and high address:
They can use the Go snippet or the Python snippet depending on which direction they want their hash compatibility to go.