Skip to content

fix(pyarrow): shift preserves length when n exceeds the series length - #3920

Merged
FBruzzesi merged 2 commits into
narwhals-dev:mainfrom
winklemad:fix-pyarrow-shift-length
Sep 6, 2026
Merged

FBruzzesi merged 2 commits into
narwhals-dev:mainfrom
winklemad:fix-pyarrow-shift-length

Conversation

@winklemad

Copy link
Copy Markdown
Contributor

Description

ArrowSeries.shift(n) padded the result with n nulls without capping n at the series length, so shifting by more than the length grew the series instead of returning an all-null series of the same length. For example, shift(3) on a two-row pyarrow series returned three rows.

The same series is used by the rolling functions — rolling_sum, rolling_mean, rolling_var and rolling_std all shift by window_size internally — so a window larger than the series desynced the chunk lengths and raised ArrowInvalid: Array arguments must all be the same length instead of returning nulls:

import narwhals as nw
import pyarrow as pa

s = nw.from_native(pa.chunked_array([[1]]), series_only=True)

s.rolling_sum(window_size=2)   # main: ArrowInvalid  ->  this PR: [None]
s.shift(3).to_list()           # main: [None, None, None]  ->  this PR: [None]

polars and pandas both return an all-null series of the original length here, so this brings the pyarrow backend in line with them.

The fix caps the null padding at the series length (min(n, length)), which keeps the existing behaviour for every in-range shift untouched. Added a regression test for shift beyond the length across the eager backends, and one for rolling_sum with a window longer than the series.

What type of PR is this? (check all applicable)

  • 💾 Refactor
  • ✨ Feature
  • 🐛 Bug Fix
  • 🔧 Optimization
  • 📝 Documentation
  • ✅ Test
  • 🐳 Other

Related issues

  • Found while testing the pyarrow backend; no existing issue.

AI assistance

  • No AI tools were used for this PR.
  • AI tools were used.

Checklist

  • Code follows style guide (ruff)
  • Tests added
  • Documented the changes (no public API change; behaviour now matches polars/pandas)

Shifting a pyarrow-backed series by more than its length padded the
result with `n` nulls regardless of the series length, so the output
grew instead of staying the same length (e.g. `shift(3)` on a two-row
series returned three rows).

This also desynced the array lengths that the rolling functions rely
on: `rolling_sum`, `rolling_mean`, `rolling_var` and `rolling_std` all
shift by `window_size` internally, so a window larger than the series
raised `ArrowInvalid: Array arguments must all be the same length`
instead of returning nulls.

Cap the null padding at the series length so a shifted series keeps its
original length (all null), matching polars and pandas.

@FBruzzesi FBruzzesi left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @winklemad for spotting the issue and coming up with a fix! Left a question and possible suggestion 🙏🏼

Comment thread src/narwhals/_arrow/series.py Resolved
When the shift moves every value out (abs(n) >= length), return the
all-null array directly and skip the slice and concat. The null padding
is capped at the series length so the result keeps its original length.
@FBruzzesi FBruzzesi added fix pyarrow Issue is related to pyarrow backend labels Sep 6, 2026
@FBruzzesi FBruzzesi changed the title fix(pyarrow): shift preserves length when n exceeds the series length fix(pyarrow): shift preserves length when n exceeds the series length Sep 6, 2026

@FBruzzesi FBruzzesi left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks again @winklemad

@FBruzzesi
FBruzzesi merged commit 6bb4f7c into narwhals-dev:main Sep 6, 2026
39 of 41 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fix pyarrow Issue is related to pyarrow backend

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants