Repository navigation
[Python] pa.array() doesn't support pa.lib.TimestampScalar objects #26862
Description
I encountered this edge case bug in PyArrow v2.0.0. For some reason, pa.array() does not know how to support pa.lib.TimestampScalar objects. This bug completely blocks my specific use case, although I do recognize that this edge case seems kind of wonky. Nonetheless, I don't see any reason why PyArrow would not understand one of it's own object types.
Stacktrace:
ArrowInvalid: Could not convert 2020-11-04 22:50:16.276892 with type pyarrow.lib.TimestampScalar: did not recognize Python value type when inferring an Arrow data type
Reproducible Code:
import pandas as pd
import pyarrow as pa
pa.array([pa.scalar(pd.to_datetime('2020-11-04 22:50:16.276892000'))])Environment: Windows 10, Python 3.7.4, PyArrow 2.0.0
Reporter: slatebit
Related issues:
- [Python] accept pyarrow values / scalars in constructor functions ? (is duplicated by)
Note: This issue was originally created as ARROW-10935. Please see the migration documentation for further details.
Activity
asfimport commentedon Dec 16, 2020
Joris Van den Bossche / @jorisvandenbossche:
Indeed, our own scalars are at the moment not supported. See ARROW-5295 for a general issue about this (so will close this issue as a duplicate of that).
Can you explain a bit how it completely blocks you? I understand it can be annoying, but normally you should always be able to convert to pyarrow scalars to python objects before passing them to
pa.array([..])?asfimport commentedon Dec 16, 2020
slatebit:
Hi Joris,
Thanks for the reply. Apologies on my end for not finding that duplicate report. Regarding it blocking my use case, I'm using the Python library Vaex. Under the hood, it relies on PyArrow as one of it's supported backends. One of its key features is lazy evaluation in that columns/features are saved as expressions and only evaluated when necessary, such as printing out the dataframe.
Currently, Vaex is built in such a way that it will if a column is a PyArrow array or chunked array, it does not do any type conversions and assumes that PyArrow will handle everything. This assumption led to this bug report. However, given your proposed solution, I'll bring this up with the Vaex team via a GitHub issue. Appreciate your help!