[Python] accept pyarrow values / scalars in constructor functions ?

@jorisvandenbossche

Currently, functions like `pyarrow.array` don't accept pyarrow Arrays, or also not scalars of it:

In [42]: arr = pa.array([1, 2, 3])

In [43]: pa.array(arr)
...
ArrowInvalid: Could not convert 1 with type pyarrow.lib.Int64Value: did not recognize Python value type when inferring an Arrow data type

In [44]: pa.array(list(arr))
...
ArrowInvalid: Could not convert 1 with type pyarrow.lib.Int64Value: did not recognize Python value type when inferring an Arrow data type

Do we want to allow those / recognize those here? (the first case could even have a fastpath, as we don't need to do it element by element).

Also scalars are not supported:

In [46]: type(arr.sum())
Out[46]: pyarrow.lib.Int64Scalar

In [47]: pa.array([arr.sum()])
...
ArrowInvalid: Could not convert 6 with type pyarrow.lib.Int64Scalar: did not recognize Python value type when inferring an Arrow data type

And also in other functions we don't accept arrow scalars / values:

In [48]: string = pa.array(['a'])[0]

In [49]: type(string)
Out[49]: pyarrow.lib.StringValue

In [50]: pa.field(string, pa.int64())
...
TypeError: expected bytes, pyarrow.lib.StringValue found

Reporter: Joris Van den Bossche / @jorisvandenbossche

Related issues:

[Python] pa.array() doesn't support pa.lib.TimestampScalar objects (duplicates)
[Python] Array constructor doesn't support arrow scalars. (is duplicated by)
[Python] Scalar constructors should be callable (relates to)
[Python] Allow pa.array/pa.chunked_array to infer pa.NA when in a non pyarrow container (relates to)

_{Note: This issue was originally created as ARROW-5295. Please see the migration documentation for further details.}

@pitrou

Antoine Pitrou / @pitrou:
I think allowing Arrays is fine. I'm not sure we want to make Scalars first-class citizens like Numpy scalars, though. At least from an implementation POV, it may add a lot of code all over the place...

@jorisvandenbossche

Joris Van den Bossche / @jorisvandenbossche:
Additional case (from ARROW-6222): pyarrow Arrays are also not recognized as list-like when converting/inferring a list array:

In [43]: pa.array([np.array([1, 1]), np.array([2, 2, 2])])                                                                                                                                                         
Out[43]: 
<pyarrow.lib.ListArray object at 0x7f258fa9d0a0>
[
  [
    1,
    1
  ],
  [
    2,
    2,
    2
  ]
]

In [44]: pa.array([pa.array([1, 1]), pa.array([2, 2, 2])])                                                                                                                                                         
---------------------------------------------------------------------------
ArrowInvalid                              Traceback (most recent call last)
<ipython-input-44-a22e0a500750> in <module>
----> 1 pa.array([pa.array([1, 1]), pa.array([2, 2, 2])])

~/scipy/repos/arrow/python/pyarrow/array.pxi in pyarrow.lib.array()

~/scipy/repos/arrow/python/pyarrow/array.pxi in pyarrow.lib._sequence_to_array()

~/scipy/repos/arrow/python/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowInvalid: Could not convert [
  1,
  1
] with type pyarrow.lib.Int64Array: did not recognize Python value type when inferring an Arrow data type

So list (or array) of numpy arrays works, but list of pyarrow arrays not. Again, not the most typical use case of pyarrow Arrays, so not sure we should add this capability.

(although we might want to find a general solution for array-like objects (eg pytorch.Tensors, see ARROW-6222), and a solution for that (somehow trying to coerce to a numpy array?) might also solve the case of a list of arrow arrays)

@wesm

Wes McKinney / @wesm:
A "simple" workaround would be to invoke Scalars' as_py method if they're passed in. That would add perf overhead, though, since we'd need to do isinstance checks.

Another option is to "sanitize" inputs (using a helper function) only in the case of failure on the initial try. So the normal use case won't be affected

The part about accepting an Array object is solved now with #34275. Leaving this issue open for the Scalar case.

Merge branch 'main' into apachegh-21761-array-accepting-scalars

GH-21761: [Python] accept pyarrow scalars in array constructor (#36162)

Navigation Menu

Repository navigation

[Python] accept pyarrow values / scalars in constructor functions ? #21761

Related issues:

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Python] accept pyarrow values / scalars in constructor functions ? #21761

Description

Related issues:

Activity

asfimport commented on May 9, 2019

asfimport commented on Aug 13, 2019

asfimport commented on Aug 22, 2019

jorisvandenbossche commented on Mar 2, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions