Skip to content

Preserve extension dtype in str.split(expand=True) (fixes #11884) - #12482

Open
kratos0718 wants to merge 1 commit into
dask:mainfrom
kratos0718:fix/str-split-expand-preserve-dtype
Open

kratos0718 wants to merge 1 commit into
dask:mainfrom
kratos0718:fix/str-split-expand-preserve-dtype

Conversation

@kratos0718

Copy link
Copy Markdown

What

Series.str.split(..., expand=True) on a string / string[pyarrow] column returns object columns instead of preserving the input dtype, which is inconsistent with pandas.

Fixes #11884.

Why

SplitMap._meta rebuilds a one-row sample so the split produces the right number of columns, but constructs it with the default dtype:

meta = self.frame._meta._constructor(
    [delimiter.join(["a"] * (self.n + 1))],
    index=meta.iloc[:1].index,
)

_constructor([...]) defaults to object, so meta.str.split(expand=True) runs on an object series and the resulting meta — and every partition's output — becomes object.

Fix

Pass the source dtype when building the sample so the split preserves it:

             index=meta.iloc[:1].index,
+            dtype=self.frame._meta.dtype,

Test

Added test_str_split_expand_preserves_dtype: splits a dtype="string" column with expand=True and asserts (via assert_eq, which checks dtypes) that the result matches pandas. It fails on main (object) and passes with this change. Uses the pandas string dtype, so no pyarrow dependency is required.

SplitMap._meta rebuilt a one-row sample to get the right number of output
columns but constructed it with the default (object) dtype, so
str.split(expand=True) on a string/string[pyarrow] column returned object
columns instead of preserving the dtype (inconsistent with pandas). Pass the
source dtype when building the sample. Adds a regression test.

Fixes dask#11884
@github-actions

github-actions Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

     25 files  ± 0       25 suites  ±0   7h 8m 7s ⏱️ + 7m 56s
 19 915 tests + 1   18 482 ✅ + 1   1 433 💤 ±0  0 ❌ ±0 
379 373 runs  +20  332 105 ✅ +19  47 268 💤 +1  0 ❌ ±0 

Results for commit 0c8db01. ± Comparison against base commit caaeb48.

@github-actions github-actions Bot added the needs attention It's been a while since this was pushed on. Needs attention from the owner or a maintainer. label Aug 3, 2026
@kratos0718

Copy link
Copy Markdown
Author

Friendly ping — open since 1 July, all checks green and MERGEABLE with no conflicts.

Re-checked against current main: dask/dataframe/dask_expr/_str_accessor.py still constructs the meta for str.split(expand=True) in a way that drops the extension dtype, so a column backed by string[pyarrow] comes back as object after the split and stops matching pandas.

Small, contained change with a test. Happy to rebase if it has drifted.

@github-actions github-actions Bot removed the needs attention It's been a while since this was pushed on. Needs attention from the owner or a maintainer. label Aug 10, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

.str.split coerces dtype to object

1 participant