[Python] Converting from NumPy to large_string or large_binary returns not implemented

Describe the bug, including details regarding any error messages, version, and platform.


arr = np.array(["a", "b"])
pa.array(arr, type=pa.large_string())

This returns

    pa.array(arr, type=pa.large_string())
  File "pyarrow/array.pxi", line 316, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 83, in pyarrow.lib._ndarray_to_array
  File "pyarrow/error.pxi", line 121, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: NumPyConverter doesn't implement <large_string> conversion.

I think this should work?

Component(s)

Python

Yes, there is indeed no specific reason for this to not work (apart from someone implementing it).

Looking at the code, this is default fallback because the NumPyConverter only is implemented for the non-large StringType. We have this:

arrow/python/pyarrow/src/arrow/python/numpy_to_arrow.cc

Lines 661 to 666 in c38c778

    
           Status NumPyConverter::Visit(const StringType& type) { 
        
             util::InitializeUTF8(); 
        
             ::arrow::internal::ChunkedStringBuilder builder(kBinaryChunksize, pool_); 
        
             auto data = reinterpret_cast<const uint8_t*>(PyArray_DATA(arr_));

But no equivalent Visit(const LargeStringType& type). The implementation for StringType is based on ChunkedStringBuilder, which is a chunked version of StringBuilder. We already have LargeStringBuilder, so it should certainly be possible to add a ChunkedLargeStringBuilder as well, so we can template the NumPyConverter to work with both builders.

[SPARK-39979][SQL][FOLLOW-UP] Make `spark.sql.execution.arrow.useLarg…

apacheGH-35289: [Python] Support large variable width types in numpy …

Navigation Menu

Repository navigation

[Python] Converting from NumPy to large_string or large_binary returns not implemented #35289

Describe the bug, including details regarding any error messages, version, and platform.

Component(s)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	Status NumPyConverter::Visit(const StringType& type) {
	util::InitializeUTF8();

	::arrow::internal::ChunkedStringBuilder builder(kBinaryChunksize, pool_);

	auto data = reinterpret_cast<const uint8_t*>(PyArray_DATA(arr_));

[Python] Converting from NumPy to large_string or large_binary returns not implemented #35289

Description

Describe the bug, including details regarding any error messages, version, and platform.

Component(s)

Activity

jorisvandenbossche commented on Apr 26, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions