Skip to content

Commit 3b5cbeb

Browse files
committed
[SPARK-46776][PYTHON][FOLLOWUP] Cast large_string/large_binary to the requested type on pyarrow < 19
### What changes were proposed in this pull request? This is a follow-up to #56157 (commit `2f4ed64204e`, `[SPARK-46776][PYTHON] Support pa.ChunkedArray columns in createDataFrame from pandas`), which introduced a regression in the minimum-dependencies CI job (`pyarrow==18.0.0`). In `create_arrow_array_from_pandas` (`python/pyspark/sql/pandas/conversion.py`), after `pa.Array.from_pandas(...)`, cast the result back to the requested `arrow_type` only when pyarrow is older than 19.0.0 and the produced type is the `large_string`/`large_binary` counterpart of the requested `string`/`binary` type. Both guards are required: the version check avoids touching pyarrow >= 19.0.0 (which already honors the requested type), and the exact type check ensures only this offset-width mismatch is corrected, leaving any other type mismatch to surface as before. ### Why are the changes needed? Since pandas 2.2.0, the `string[pyarrow]` extension dtype is backed by `large_string` (64-bit offsets). When such a series is converted via the `__arrow_array__` protocol, pyarrow < 19.0.0 ignores the requested `type` argument, so `pa.Array.from_pandas(series, type=pa.string())` returns a `large_string` array even though `string` was requested. The JVM then reads the 64-bit offset buffers against the 32-bit schema, silently corrupting the data. pyarrow 19.0.0 fixed this in the protocol path (apache/arrow#44195) by casting the result to the requested type. This change replicates that behavior for older pyarrow so the minimum-dependencies build (`pyarrow==18.0.0`) produces correct data instead of corrupt rows. ### Does this PR introduce _any_ user-facing change? Yes. On pyarrow < 19.0.0, `createDataFrame` from a pandas object containing `string[pyarrow]`/`large_string`-backed columns (and the `binary` equivalent) previously produced silently corrupted data; it now produces correct data. On pyarrow >= 19.0.0 there is no behavior change. ### How was this patch tested? Existing `pyspark.sql.tests.test_conversion` suite passes locally. The regression environment was validated by manually dispatching the "Build / Python-only (Minimum dependencies of PySpark)" workflow on my fork against this branch with `pyarrow==18.0.0`; the `pyspark-sql`, `pyspark-pandas`, and related modules all passed: https://github.com/Yicong-Huang/spark/actions/runs/27244934810 ### Was this patch authored or co-authored using generative AI tooling? No. Closes #56380 from Yicong-Huang/SPARK-46776-followup/fix/cast-large-string-pyarrow18. Authored-by: Yicong Huang <[email protected]> Signed-off-by: Yicong-Huang <[email protected]>
1 parent 1f53674 commit 3b5cbeb

1 file changed

Lines changed: 18 additions & 1 deletion

File tree

‎python/pyspark/sql/pandas/conversion.py‎

Lines changed: 18 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -88,6 +88,7 @@ def create_arrow_array_from_pandas(
8888
"""
8989
import pyarrow as pa
9090
import pandas as pd
91+
from pyspark.loose_version import LooseVersion
9192
from pyspark.sql.pandas.types import to_arrow_type, _create_converter_from_pandas
9293

9394
if isinstance(series.dtype, pd.CategoricalDtype):
@@ -113,7 +114,23 @@ def create_arrow_array_from_pandas(
113114
else:
114115
mask = series.isnull()
115116
try:
116-
return pa.Array.from_pandas(series, mask=mask, type=arrow_type, safe=safecheck)
117+
result = pa.Array.from_pandas(series, mask=mask, type=arrow_type, safe=safecheck)
118+
# SPARK-46776: pyarrow < 19.0.0 ignores the requested ``type`` in the
119+
# ``__arrow_array__`` protocol used by pyarrow-backed extension dtypes, so a
120+
# ``string[pyarrow]`` series (backed by ``large_string`` since pandas 2.2.0) can
121+
# come back as ``large_string`` even when ``string`` was requested, silently
122+
# corrupting data on the JVM side. Cast back only for this exact (large_)string /
123+
# (large_)binary offset-width mismatch; pyarrow >= 19.0.0 already honors the type.
124+
if (
125+
arrow_type is not None
126+
and LooseVersion(pa.__version__) < LooseVersion("19.0.0")
127+
and (
128+
(pa.types.is_large_string(result.type) and pa.types.is_string(arrow_type))
129+
or (pa.types.is_large_binary(result.type) and pa.types.is_binary(arrow_type))
130+
)
131+
):
132+
result = result.cast(arrow_type)
133+
return result
117134
except TypeError as e:
118135
error_msg = (
119136
"Exception thrown when converting pandas.Series (%s) "

0 commit comments

Comments
 (0)