Repository navigation
fix(pyarrow): shift preserves length when n exceeds the series length - #3920
Merged
Merged
Conversation
Shifting a pyarrow-backed series by more than its length padded the result with `n` nulls regardless of the series length, so the output grew instead of staying the same length (e.g. `shift(3)` on a two-row series returned three rows). This also desynced the array lengths that the rolling functions rely on: `rolling_sum`, `rolling_mean`, `rolling_var` and `rolling_std` all shift by `window_size` internally, so a window larger than the series raised `ArrowInvalid: Array arguments must all be the same length` instead of returning nulls. Cap the null padding at the series length so a shifted series keeps its original length (all null), matching polars and pandas.
FBruzzesi
reviewed
Sep 6, 2026
FBruzzesi
left a comment
Member
There was a problem hiding this comment.
Thanks @winklemad for spotting the issue and coming up with a fix! Left a question and possible suggestion 🙏🏼
When the shift moves every value out (abs(n) >= length), return the all-null array directly and skip the slice and concat. The null padding is capped at the series length so the result keeps its original length.
shift preserves length when n exceeds the series length
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Description
ArrowSeries.shift(n)padded the result withnnulls without cappingnat the series length, so shifting by more than the length grew the series instead of returning an all-null series of the same length. For example,shift(3)on a two-row pyarrow series returned three rows.The same series is used by the rolling functions —
rolling_sum,rolling_mean,rolling_varandrolling_stdall shift bywindow_sizeinternally — so a window larger than the series desynced the chunk lengths and raisedArrowInvalid: Array arguments must all be the same lengthinstead of returning nulls:polars and pandas both return an all-null series of the original length here, so this brings the pyarrow backend in line with them.
The fix caps the null padding at the series length (
min(n, length)), which keeps the existing behaviour for every in-range shift untouched. Added a regression test forshiftbeyond the length across the eager backends, and one forrolling_sumwith a window longer than the series.What type of PR is this? (check all applicable)
Related issues
AI assistance
Checklist