Skip to content

Commit 0eb1e18

Browse files
tritsystemclaude
andcommitted
apacheGH-39914: [Python] Skip unparseable complex dtype metadata instead of guessing ArrowDtype
Address review from @jorisvandenbossche: a parse failure on the 'numpy_type' metadata string shouldn't be read as "the user wants an ArrowDtype" -- it can happen for reasons other than a complex/nested Arrow type. Replace the schema-field-type fallback with `continue`, leaving the column out of the metadata-driven extension-dtype handling entirely, same as the existing KeyError case right above it. Also broadened the except clause from `TypeError` to `(TypeError, ValueError)`. Traced why: np.dtype() is tried before the ArrowDtype parser, and for a 'numpy_type' string containing a comma (struct, dictionary, map -- anything but a single-field list) it misreads the comma as its own legacy multi-field format syntax and raises ValueError, never reaching the TypeError the list case gets. Verified directly against pandas_dtype() and np.dtype() on pandas 2.3.3: the previous TypeError-only fix left struct/dictionary/map still crashing. Updated the regression test to match: it now asserts the column lands as object (list/struct) or category (dictionary) dtype with no crash, rather than an exact ArrowDtype round-trip -- which this fix no longer attempts, per the review above. Added a dictionary-dtype case alongside the existing list/struct ones since it exercises the same ValueError path. Co-Authored-By: Claude Sonnet 5 <[email protected]>
1 parent b141fb3 commit 0eb1e18

2 files changed

Lines changed: 36 additions & 16 deletions

File tree

‎python/pyarrow/pandas_compat.py‎

Lines changed: 10 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -904,19 +904,20 @@ def _get_extension_dtypes(table, columns_metadata, types_mapper, options, catego
904904
# that are certainly numpy dtypes
905905
try:
906906
pandas_dtype = _pandas_api.pandas_dtype(dtype)
907-
except TypeError:
907+
except (TypeError, ValueError):
908908
# Complex/nested Arrow types (list, struct, dictionary, ...)
909909
# serialize to a 'numpy_type' string (e.g.
910910
# "list<item: string>[pyarrow]") that pandas_dtype() cannot
911-
# parse back. Fall back to building the ArrowDtype directly
912-
# from the schema's actual field type instead of giving up
913-
# on round-tripping the dtype entirely.
911+
# parse back; np.dtype() is tried first and, for strings
912+
# with a comma (struct, dictionary, map), misreads it as
913+
# its legacy multi-field format syntax and raises
914+
# ValueError before the ArrowDtype parser is ever reached,
915+
# rather than the TypeError a comma-free string like list
916+
# gets. Either way, don't guess that the user wants an
917+
# ArrowDtype for a dtype string we can't actually parse;
918+
# leave this column out so it isn't treated specially.
914919
# See GH-39914 / pandas-dev/pandas#53011.
915-
try:
916-
field = table.schema.field(name)
917-
except KeyError:
918-
continue
919-
pandas_dtype = _pandas_api.pd.ArrowDtype(field.type)
920+
continue
920921
if isinstance(pandas_dtype, _pandas_api.extension_dtype):
921922
if isinstance(pandas_dtype, _pandas_api.pd.StringDtype):
922923
# when the metadata indicate to use the string dtype,

‎python/pyarrow/tests/test_pandas.py‎

Lines changed: 26 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -4570,18 +4570,27 @@ def test_to_pandas_extension_dtypes_mapping_complex_type_no_types_mapper():
45704570
# GH-39914: without an explicit types_mapper, the pandas metadata
45714571
# embedded by from_pandas() stores a 'numpy_type' string like
45724572
# "list<item: string>[pyarrow]" for complex/nested ArrowDtype columns.
4573-
# pandas_dtype() cannot parse that string back, so to_pandas() (and, by
4574-
# extension, pd.read_parquet() without dtype_backend="pyarrow") used to
4575-
# raise a TypeError instead of falling back to the schema's actual field
4576-
# type. See pandas-dev/pandas#53011.
4573+
# pandas_dtype() cannot parse that string back -- with TypeError for a
4574+
# comma-free string like the list case, or ValueError for a string
4575+
# with a comma (struct, dictionary, map), which np.dtype() misreads as
4576+
# its legacy multi-field format syntax before ever reaching the
4577+
# ArrowDtype parser. Either way, to_pandas() (and, by extension,
4578+
# pd.read_parquet() without dtype_backend="pyarrow") used to crash
4579+
# outright instead of leaving the column out of the metadata-driven
4580+
# extension-dtype handling. See pandas-dev/pandas#53011.
4581+
#
4582+
# The ArrowDtype is not expected to round-trip here: an unparseable
4583+
# 'numpy_type' string isn't necessarily an ArrowDtype the caller still
4584+
# wants, so the column just falls back to its normal (non-metadata)
4585+
# conversion.
45774586
list_type = pd.ArrowDtype(pa.list_(pa.string()))
45784587
df0 = pd.DataFrame({
45794588
"a": pd.Series([["x"], ["x", "y"]], dtype=list_type),
45804589
})
4581-
45824590
table = pa.Table.from_pandas(df0)
45834591
df1 = table.to_pandas()
4584-
pd.testing.assert_frame_equal(df0, df1)
4592+
assert df1["a"].dtype == np.dtype("object")
4593+
assert [list(v) for v in df1["a"]] == [["x"], ["x", "y"]]
45854594

45864595
struct_type = pd.ArrowDtype(
45874596
pa.struct([pa.field("bar", pa.bool_()), pa.field("baz", pa.float32())])
@@ -4594,7 +4603,17 @@ def test_to_pandas_extension_dtypes_mapping_complex_type_no_types_mapper():
45944603
})
45954604
table2 = pa.Table.from_pandas(df2)
45964605
df3 = table2.to_pandas()
4597-
pd.testing.assert_frame_equal(df2, df3)
4606+
assert df3["a"].dtype == np.dtype("object")
4607+
assert df3["a"].tolist() == [
4608+
{"bar": True, "baz": 1.0}, {"bar": False, "baz": None}
4609+
]
4610+
4611+
dict_type = pd.ArrowDtype(pa.dictionary(pa.int32(), pa.string()))
4612+
df4 = pd.DataFrame({"a": pd.Series(["x", "y", "x"], dtype=dict_type)})
4613+
table4 = pa.Table.from_pandas(df4)
4614+
df5 = table4.to_pandas()
4615+
assert isinstance(df5["a"].dtype, pd.CategoricalDtype)
4616+
assert df5["a"].tolist() == ["x", "y", "x"]
45984617

45994618

46004619
def test_array_to_pandas():

0 commit comments

Comments
 (0)