Repository navigation
Preserve extension dtype in str.split(expand=True) (fixes #11884) - #12482
kratos0718 wants to merge 1 commit into
Conversation
SplitMap._meta rebuilt a one-row sample to get the right number of output columns but constructed it with the default (object) dtype, so str.split(expand=True) on a string/string[pyarrow] column returned object columns instead of preserving the dtype (inconsistent with pandas). Pass the source dtype when building the sample. Adds a regression test. Fixes dask#11884
Unit Test ResultsSee test report for an extended history of previous test failures. This is useful for diagnosing flaky tests. 25 files ± 0 25 suites ±0 7h 8m 7s ⏱️ + 7m 56s Results for commit 0c8db01. ± Comparison against base commit caaeb48. |
|
Friendly ping — open since 1 July, all checks green and Re-checked against current Small, contained change with a test. Happy to rebase if it has drifted. |
What
Series.str.split(..., expand=True)on astring/string[pyarrow]column returnsobjectcolumns instead of preserving the input dtype, which is inconsistent with pandas.Fixes #11884.
Why
SplitMap._metarebuilds a one-row sample so the split produces the right number of columns, but constructs it with the default dtype:_constructor([...])defaults toobject, someta.str.split(expand=True)runs on an object series and the resulting meta — and every partition's output — becomesobject.Fix
Pass the source dtype when building the sample so the split preserves it:
index=meta.iloc[:1].index, + dtype=self.frame._meta.dtype,Test
Added
test_str_split_expand_preserves_dtype: splits adtype="string"column withexpand=Trueand asserts (viaassert_eq, which checks dtypes) that the result matches pandas. It fails onmain(object) and passes with this change. Uses the pandasstringdtype, so no pyarrow dependency is required.