GH-43683: [Python] Use pandas StringDtype when enabled (pandas 3+) by jorisvandenbossche · Pull Request #44195 · apache/arrow

jorisvandenbossche · 2024-09-23T15:14:45Z

Rationale for this change

With pandas' PDEP-14 proposal, pandas is planning to introduce a default string dtype in pandas 3.0 (instead of the current object dtype).

This will become the default in pandas 3.0, and can be enabled with an option in the upcoming pandas 2.3 (pd.options.future.infer_string = True). To prepare for that, we should start using that string dtype in to_pandas() conversions when that option is enabled.

What changes are included in this PR?

If pandas >= 3.0 is used or the pandas option is enabled, ensure that to_pandas() calls use the default string dtype of pandas for string-like columns (string, large_string, string_view)

Are these changes tested?

It is tested in the pandas-nightly crossbow build.

There is still one failure that is because of a bug on the pandas side (pandas-dev/pandas#59879)

Are there any user-facing changes?

This PR includes breaking changes to public APIs. Depending on the version of pandas, to_pandas() will change to use pandas' string dtype instead of object dtype. This is a breaking user-facing change, but essentially just following the equivalent change in default dtype on the pandas side.

GitHub Issue: [Python] Support pandas future default string dtype #43683

…as-string-dtype

jorisvandenbossche · 2024-11-08T14:43:32Z

@github-actions crossbow submit test-conda-python-3.11-pandas-nightly-numpy-nightly

github-actions · 2024-11-08T14:45:48Z

Revision: 56b61f2

Submitted crossbow builds: ursacomputing/crossbow @ actions-c85b742ef7

Task	Status
test-conda-python-3.11-pandas-nightly-numpy-nightly

WillAyd · 2024-11-08T15:01:35Z

+    e1 = pd.DataFrame(
+        {'a': a_values},
+        index=pd.RangeIndex(0, 8, step=2, name='qux'),
+        columns=pd.Index(['a'], dtype=object)


Does the column type created with the dict argument differ from this?

This test is specifically using old metadata that specifies the dtype of the columns is object dtype, and then pyarrow tries to restore it that way.

It's the question if we should do that though .. Because every file written from a pandas DataFrame before pandas 3.0 will have that, so maybe we should specifically ignore object dtype here if the inferred type is that it contains all strings, so users consistently get a columns Index object using str dtype

Hmm that's tricky but I think going with the str data type as you suggested is better; I would expect that is a better UX in over 99% of instances

OK, changed this to ensure we actually use str dtype columns Index object, even if the pandas metadata of the pyarrow table says that the original table was using object dtype.

This ensures that all existing files will use (with pandas>= 3) the default str dtype for the columns, but that also has the trade-off that if you explicitly want to use object dtype with strings, that this will no longer roundtrip in pandas->pyarrow/parquet->pandas)

jorisvandenbossche · 2024-11-13T09:05:07Z

@github-actions crossbow submit test-conda-python-3.11-pandas-nightly-numpy-nightly

github-actions · 2024-11-13T09:07:22Z

Revision: 84b8234

Submitted crossbow builds: ursacomputing/crossbow @ actions-ac3103d3ba

Task	Status
test-conda-python-3.11-pandas-nightly-numpy-nightly

jorisvandenbossche · 2024-11-13T09:43:53Z

@github-actions crossbow submit test-conda-python-3.11-pandas-nightly-numpy-nightly

github-actions · 2024-11-13T09:46:09Z

Revision: e5db09f

Submitted crossbow builds: ursacomputing/crossbow @ actions-3c389cd49e

Task	Status
test-conda-python-3.11-pandas-nightly-numpy-nightly

jorisvandenbossche · 2025-01-03T11:10:35Z

@github-actions crossbow submit -g python

github-actions · 2025-01-03T11:13:20Z

Revision: 940b64d

Submitted crossbow builds: ursacomputing/crossbow @ actions-c96266afd8

Task	Status
example-python-minimal-build-fedora-conda
example-python-minimal-build-ubuntu-venv
test-conda-python-3.10
test-conda-python-3.10-cython2
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1
test-conda-python-3.10-pandas-latest-numpy-latest
test-conda-python-3.10-substrait
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-latest-numpy-1.26
test-conda-python-3.11-pandas-latest-numpy-latest
test-conda-python-3.11-pandas-nightly-numpy-nightly
test-conda-python-3.11-pandas-upstream_devel-numpy-nightly
test-conda-python-3.11-spark-master
test-conda-python-3.12
test-conda-python-3.12-cpython-debug
test-conda-python-3.13
test-conda-python-3.9
test-conda-python-3.9-pandas-1.1.3-numpy-1.19.5
test-conda-python-emscripten
test-cuda-python-ubuntu-22.04-cuda-11.7.1
test-debian-12-python-3-amd64
test-debian-12-python-3-i386
test-fedora-39-python-3
test-ubuntu-22.04-python-3
test-ubuntu-22.04-python-313-freethreading
test-ubuntu-24.04-python-3

WillAyd

Sorry @raulcd for missing your ping! I am also not very familiar with this part of the code so I have a few questions, although I generally trust Joris knows what he is doing here :-)

WillAyd · 2025-01-03T14:23:42Z

  if (options.strings_to_categorical) {
    for (int i = 0; i < static_cast<int>(arrays->size()); i++) {
-      if (is_base_binary_like((*arrays)[i]->type()->id())) {
+      if (is_base_binary_like((*arrays)[i]->type()->id()) ||


The binary_view changes are tangential to pandas 3.x right? I wonder if they shouldn't be done in their own PR

Somewhat tangential yes, as in that it is also a useful change to do regardless of the other changes here. I am also not entirely sure there is a very specific test for this.

Happy to move out to a separate PR, although I then would like both to get merged for 19.0

Yea I would prefer to separate out so we can analyze test coverage too. I suppose that would also be helpful if these changes end up going through to different releases (although that's not the aim)

OK. it turns out to not actually affect the tests here, so good to move that to a separate PR with actual test coverage: #45176

I've merged the related issue and marked as 19.0.0

WillAyd · 2025-01-03T14:27:22Z

-    if name is not None and not isinstance(name, str):
+    if (
+        name is not None
+        and not (isinstance(name, float) and np.isnan(name))


Does this mean that np.nan is now a valid column name for the string data type?

Yes, since this was essentially to support some missing values in the column names (but restricted to None), also allowing np.nan keeps somewhat the same behaviour when switching from object dtype to string dtype

…as-string-dtype

…44195) ### Rationale for this change With pandas' [PDEP-14](https://pandas.pydata.org/pdeps/0014-string-dtype.html) proposal, pandas is planning to introduce a default string dtype in pandas 3.0 (instead of the current object dtype). This will become the default in pandas 3.0, and can be enabled with an option in the upcoming pandas 2.3 (`pd.options.future.infer_string = True`). To prepare for that, we should start using that string dtype in `to_pandas()` conversions when that option is enabled. ### What changes are included in this PR? - If pandas >= 3.0 is used or the pandas option is enabled, ensure that `to_pandas()` calls use the default string dtype of pandas for string-like columns (string, large_string, string_view) ### Are these changes tested? It is tested in the pandas-nightly crossbow build. There is still one failure that is because of a bug on the pandas side (pandas-dev/pandas#59879) ### Are there any user-facing changes? **This PR includes breaking changes to public APIs.** Depending on the version of pandas, `to_pandas()` will change to use pandas' string dtype instead of object dtype. This is a breaking user-facing change, but essentially just following the equivalent change in default dtype on the pandas side. * GitHub Issue: #43683 Lead-authored-by: Joris Van den Bossche <[email protected]> Co-authored-by: Raúl Cumplido <[email protected]> Signed-off-by: Joris Van den Bossche <[email protected]>

conbench-apache-arrow · 2025-01-15T11:52:55Z

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 5181c24.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 13 possible false positives for unstable benchmarks that are known to sometimes produce them.

…at (#44963) ### Rationale for this change Small part of apache/arrow#44195 factored out into its own PR because this change is just a small refactor making apache/arrow#44195 easier to do, but in itself not changing any logic. We currently both store `name` and `field_name` in the pandas metadata. `field_name` is guaranteed to be a string, and is always exactly the name used in the arrow schema. `name` can also be None if the original pandas DataFrame used None as the column label or if it was coming from an index level without name. Right now we had several places where we used `name` but then checked for it being None. With this PR I made it more consistently use `field_name` in the cases it needs the string version, by more consistently passing through both names a field_names. ### Are these changes tested? Existing tests should cover this ### Are there any user-facing changes? No * Issue: #44962 * GitHub Issue: #44962 Authored-by: Joris Van den Bossche <[email protected]> Signed-off-by: Raúl Cumplido <[email protected]>

… requested type on pyarrow < 19 ### What changes were proposed in this pull request? This is a follow-up to #56157 (commit `2f4ed64204e`, `[SPARK-46776][PYTHON] Support pa.ChunkedArray columns in createDataFrame from pandas`), which introduced a regression in the minimum-dependencies CI job (`pyarrow==18.0.0`). In `create_arrow_array_from_pandas` (`python/pyspark/sql/pandas/conversion.py`), after `pa.Array.from_pandas(...)`, cast the result back to the requested `arrow_type` only when pyarrow is older than 19.0.0 and the produced type is the `large_string`/`large_binary` counterpart of the requested `string`/`binary` type. Both guards are required: the version check avoids touching pyarrow >= 19.0.0 (which already honors the requested type), and the exact type check ensures only this offset-width mismatch is corrected, leaving any other type mismatch to surface as before. ### Why are the changes needed? Since pandas 2.2.0, the `string[pyarrow]` extension dtype is backed by `large_string` (64-bit offsets). When such a series is converted via the `__arrow_array__` protocol, pyarrow < 19.0.0 ignores the requested `type` argument, so `pa.Array.from_pandas(series, type=pa.string())` returns a `large_string` array even though `string` was requested. The JVM then reads the 64-bit offset buffers against the 32-bit schema, silently corrupting the data. pyarrow 19.0.0 fixed this in the protocol path (apache/arrow#44195) by casting the result to the requested type. This change replicates that behavior for older pyarrow so the minimum-dependencies build (`pyarrow==18.0.0`) produces correct data instead of corrupt rows. ### Does this PR introduce _any_ user-facing change? Yes. On pyarrow < 19.0.0, `createDataFrame` from a pandas object containing `string[pyarrow]`/`large_string`-backed columns (and the `binary` equivalent) previously produced silently corrupted data; it now produces correct data. On pyarrow >= 19.0.0 there is no behavior change. ### How was this patch tested? Existing `pyspark.sql.tests.test_conversion` suite passes locally. The regression environment was validated by manually dispatching the "Build / Python-only (Minimum dependencies of PySpark)" workflow on my fork against this branch with `pyarrow==18.0.0`; the `pyspark-sql`, `pyspark-pandas`, and related modules all passed: https://github.com/Yicong-Huang/spark/actions/runs/27244934810 ### Was this patch authored or co-authored using generative AI tooling? No. Closes #56380 from Yicong-Huang/SPARK-46776-followup/fix/cast-large-string-pyarrow18. Authored-by: Yicong Huang <[email protected]> Signed-off-by: Yicong-Huang <[email protected]>

… requested type on pyarrow < 19 ### What changes were proposed in this pull request? This is a follow-up to #56157 (commit `2f4ed64204e`, `[SPARK-46776][PYTHON] Support pa.ChunkedArray columns in createDataFrame from pandas`), which introduced a regression in the minimum-dependencies CI job (`pyarrow==18.0.0`). In `create_arrow_array_from_pandas` (`python/pyspark/sql/pandas/conversion.py`), after `pa.Array.from_pandas(...)`, cast the result back to the requested `arrow_type` only when pyarrow is older than 19.0.0 and the produced type is the `large_string`/`large_binary` counterpart of the requested `string`/`binary` type. Both guards are required: the version check avoids touching pyarrow >= 19.0.0 (which already honors the requested type), and the exact type check ensures only this offset-width mismatch is corrected, leaving any other type mismatch to surface as before. ### Why are the changes needed? Since pandas 2.2.0, the `string[pyarrow]` extension dtype is backed by `large_string` (64-bit offsets). When such a series is converted via the `__arrow_array__` protocol, pyarrow < 19.0.0 ignores the requested `type` argument, so `pa.Array.from_pandas(series, type=pa.string())` returns a `large_string` array even though `string` was requested. The JVM then reads the 64-bit offset buffers against the 32-bit schema, silently corrupting the data. pyarrow 19.0.0 fixed this in the protocol path (apache/arrow#44195) by casting the result to the requested type. This change replicates that behavior for older pyarrow so the minimum-dependencies build (`pyarrow==18.0.0`) produces correct data instead of corrupt rows. ### Does this PR introduce _any_ user-facing change? Yes. On pyarrow < 19.0.0, `createDataFrame` from a pandas object containing `string[pyarrow]`/`large_string`-backed columns (and the `binary` equivalent) previously produced silently corrupted data; it now produces correct data. On pyarrow >= 19.0.0 there is no behavior change. ### How was this patch tested? Existing `pyspark.sql.tests.test_conversion` suite passes locally. The regression environment was validated by manually dispatching the "Build / Python-only (Minimum dependencies of PySpark)" workflow on my fork against this branch with `pyarrow==18.0.0`; the `pyspark-sql`, `pyspark-pandas`, and related modules all passed: https://github.com/Yicong-Huang/spark/actions/runs/27244934810 ### Was this patch authored or co-authored using generative AI tooling? No. Closes #56380 from Yicong-Huang/SPARK-46776-followup/fix/cast-large-string-pyarrow18. Authored-by: Yicong Huang <[email protected]> Signed-off-by: Yicong-Huang <[email protected]> (cherry picked from commit 3b5cbeb) Signed-off-by: Yicong-Huang <[email protected]>

… requested type on pyarrow < 19 ### What changes were proposed in this pull request? This is a follow-up to apache#56157 (commit `2f4ed64204e`, `[SPARK-46776][PYTHON] Support pa.ChunkedArray columns in createDataFrame from pandas`), which introduced a regression in the minimum-dependencies CI job (`pyarrow==18.0.0`). In `create_arrow_array_from_pandas` (`python/pyspark/sql/pandas/conversion.py`), after `pa.Array.from_pandas(...)`, cast the result back to the requested `arrow_type` only when pyarrow is older than 19.0.0 and the produced type is the `large_string`/`large_binary` counterpart of the requested `string`/`binary` type. Both guards are required: the version check avoids touching pyarrow >= 19.0.0 (which already honors the requested type), and the exact type check ensures only this offset-width mismatch is corrected, leaving any other type mismatch to surface as before. ### Why are the changes needed? Since pandas 2.2.0, the `string[pyarrow]` extension dtype is backed by `large_string` (64-bit offsets). When such a series is converted via the `__arrow_array__` protocol, pyarrow < 19.0.0 ignores the requested `type` argument, so `pa.Array.from_pandas(series, type=pa.string())` returns a `large_string` array even though `string` was requested. The JVM then reads the 64-bit offset buffers against the 32-bit schema, silently corrupting the data. pyarrow 19.0.0 fixed this in the protocol path (apache/arrow#44195) by casting the result to the requested type. This change replicates that behavior for older pyarrow so the minimum-dependencies build (`pyarrow==18.0.0`) produces correct data instead of corrupt rows. ### Does this PR introduce _any_ user-facing change? Yes. On pyarrow < 19.0.0, `createDataFrame` from a pandas object containing `string[pyarrow]`/`large_string`-backed columns (and the `binary` equivalent) previously produced silently corrupted data; it now produces correct data. On pyarrow >= 19.0.0 there is no behavior change. ### How was this patch tested? Existing `pyspark.sql.tests.test_conversion` suite passes locally. The regression environment was validated by manually dispatching the "Build / Python-only (Minimum dependencies of PySpark)" workflow on my fork against this branch with `pyarrow==18.0.0`; the `pyspark-sql`, `pyspark-pandas`, and related modules all passed: https://github.com/Yicong-Huang/spark/actions/runs/27244934810 ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#56380 from Yicong-Huang/SPARK-46776-followup/fix/cast-large-string-pyarrow18. Authored-by: Yicong Huang <[email protected]> Signed-off-by: Yicong-Huang <[email protected]>

apacheGH-43683: [Python] Use pandas StringDtype when enabled (pandas 3+)

ea76574

github-actions Bot added Component: Python awaiting committer review labels Sep 23, 2024

jorisvandenbossche added 5 commits November 6, 2024 17:53

Merge remote-tracking branch 'upstream/main' into apachegh-43683-pand…

3e17983

…as-string-dtype

test on CI

e0b2958

honor strings_to_categorical

8a6d6c3

more test fixes

11d2691

honor categories keyword

56b61f2

WillAyd reviewed Nov 8, 2024

View reviewed changes

github-actions Bot added awaiting changes and removed awaiting committer review labels Nov 8, 2024

jorisvandenbossche added 2 commits November 9, 2024 17:52

propagate env variable in docker image

fdd6af3

ignore pandas_metadata for string dtype in case of dictionary column

84b8234

github-actions Bot added awaiting change review and removed awaiting changes labels Nov 13, 2024

jorisvandenbossche added 2 commits November 13, 2024 10:27

keep columns Index as string dtype even if metadata says object

136b091

fix compute / feather tests

e5db09f

jorisvandenbossche marked this pull request as ready for review November 13, 2024 09:43

jorisvandenbossche requested review from assignUser, jonkeane, kou and raulcd as code owners November 13, 2024 09:43

reformat to avoid diff

ec750bd

github-actions Bot removed the awaiting change review label Nov 13, 2024

github-actions Bot added awaiting change review and removed awaiting changes labels Jan 3, 2025

github-actions Bot added awaiting changes and removed awaiting change review labels Jan 3, 2025

WillAyd reviewed Jan 3, 2025

View reviewed changes

jorisvandenbossche added 2 commits January 5, 2025 12:05

remove strings_to_categorical for string_view changes

70a2c3c

Merge remote-tracking branch 'upstream/main' into apachegh-43683-pand…

ea4cbf4

…as-string-dtype

github-actions Bot added awaiting change review and removed awaiting changes labels Jan 5, 2025

jorisvandenbossche mentioned this pull request Jan 5, 2025

API (string dtype): comparisons between different string classes pandas-dev/pandas#60639

Closed

Merge remote-tracking branch 'upstream/main' into apachegh-43683-pand…

a59a2a2

…as-string-dtype

github-actions Bot added awaiting change review and removed awaiting changes labels Jan 9, 2025

amoeba mentioned this pull request Jan 9, 2025

WIP: Testing-only PR to check maint-19.0.0 status #45194

Closed

jorisvandenbossche merged commit 5181c24 into apache:main Jan 9, 2025

jorisvandenbossche removed the awaiting change review label Jan 9, 2025

jorisvandenbossche mentioned this pull request Jan 9, 2025

[Python] Support pandas future default string dtype #43683

Closed

jorisvandenbossche deleted the gh-43683-pandas-string-dtype branch January 9, 2025 19:22

Yicong-Huang mentioned this pull request Jun 9, 2026

[SPARK-46776][PYTHON][FOLLOWUP] Cast large_string/large_binary to the requested type on pyarrow < 19 apache/spark#56380

Closed

Conversation

jorisvandenbossche commented Sep 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

jorisvandenbossche commented Nov 8, 2024

Uh oh!

github-actions Bot commented Nov 8, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jorisvandenbossche commented Nov 13, 2024

Uh oh!

github-actions Bot commented Nov 13, 2024

Uh oh!

jorisvandenbossche commented Nov 13, 2024

Uh oh!

github-actions Bot commented Nov 13, 2024

Uh oh!

jorisvandenbossche commented Jan 3, 2025

Uh oh!

github-actions Bot commented Jan 3, 2025

Uh oh!

WillAyd left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

conbench-apache-arrow Bot commented Jan 15, 2025

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jorisvandenbossche commented Sep 23, 2024 •

edited

Loading