Ensure map_partitions returns Series object if function returns scalar
#10739
Conversation
fjetter
changed the title
map_partitions returns Series object if function returns scalar
|
Looks like there are a bunch of genuine typing issues since I set the dtype on the created series explicitly. @phofl if you want to, you can pick this up after the holidays but if not I'll handle it once I'm back |
|
I'll take a look after the holidays, the dtype issue is most likely because of running this on an empty meta object, that's the situation where I've seen us ending up with NaN there |
| df = func(*args, **kwargs) | ||
|
|
||
| if is_scalar(df) and is_series_like(meta): | ||
| df = type(meta)([df]) |
There was a problem hiding this comment.
I ended up removing the explicit dtype since it caused issues and was quite unnecessary considering that df is already the final result data. The casting/coercing logic of the Series takes care of compat stuff with nulls, etc.
| token=name1, | ||
| meta=self, | ||
| **chunk_kwargs, | ||
| enforce_metadata=False, |
There was a problem hiding this comment.
When using map_partitions internally we sometimes rely on the data not being cast to a Series. The cumulative aggregation logic becomes quite cumbersome otherwise. I hope this is not introducing any other weird artefacts 🤞
fjetter
force-pushed
the
ensure_map_partitions_returns_series
branch
from
b21d4f2 to
71c29e0
Compare
phofl
left a comment
phofl
left a comment
There was a problem hiding this comment.
the logic looks fine, I think the test failures are something that we can fix by adjusting the test
fjetter
force-pushed
the
ensure_map_partitions_returns_series
branch
from
7d4fbd7 to
fc5cd91
Compare
Unit Test ResultsSee test report for an extended history of previous test failures. This is useful for diagnosing flaky tests. 15 files ± 0 15 suites ±0 3h 35m 57s ⏱️ + 13m 1s For more details on these failures and errors, see this check. Results for commit c60269b. ± Comparison against base commit 1ecf464. ♻️ This comment has been updated with latest results. |
fjetter
force-pushed
the
ensure_map_partitions_returns_series
branch
from
fc5cd91 to
a7f08aa
Compare
fjetter
force-pushed
the
ensure_map_partitions_returns_series
branch
from
2e95b60 to
c60269b
Compare
2 participants
pre-commit run --all-files