Repository navigation
[Python] pyarrow large integer conversion #32664
Description
I have a json document that looks like this.
{"number": 123451234512345123451234512}
I then run the below code.
from pyarrow.json import read_json
pyarrow_table = read_json('pyarrow_test.json')
pyarrow_table['number'][0].as_py().as_integer_ratio()
(123451234512345125900779520, 1)
notice the float that I get looks like it has been rounded or modified in some way.
Am I reading this file incorrectly or is there an issue with the conversion of this number?
Reporter: Donald Freeman
Note: This issue was originally created as ARROW-17393. Please see the migration documentation for further details.
Activity
asfimport commentedon Aug 16, 2022
Joris Van den Bossche / @jorisvandenbossche:
The integer number you have here is too large to fit in the int64 range:
So that means that this cannot be parsed in a pyarrow column of type int64. Explicitly asking for this would also give an error:
As a result, pyarrow falls back to parsing the number as float. But also here there are inherent limitations for what a float64 can represent, so it is expected that you cannot store such a number faithfully.
adriangb commentedon Nov 30, 2023
Given that other formats can represent larger ints (in my case ClickHouse supports 256-bit integers) and that other implementations support larger ints can this be turned into a feature request to add 128 and 256 bit integer support to this implementation?