Repository navigation
fix(pyarrow): keep nulls through clip instead of replacing them with a bound - #3906
Conversation
…a bound pyarrow's min_element_wise / max_element_wise skip nulls by default, so clip, clip_lower and clip_upper on the pyarrow backend replaced a null value with the clip bound (e.g. [1, None, 3].clip(0, 2) returned [1, 2, 3]) instead of preserving it. Polars and pandas keep the null. Pass skip_nulls=False so a null stays null, matching the other backends.
FBruzzesi
left a comment
There was a problem hiding this comment.
Thanks @winklemad for spotting and fixing the issue. Left a comment to extend the test to lazy backends as well. Once we do that, it's ready to merge
Testing clip null-preservation against every backend (not just the eager ones) showed the SQL backends had the same bug: `least` / `greatest` skip nulls, so clip, clip_lower and clip_upper replaced a null with the bound. Guard each so a null input stays null, and widen the regression test to `constructor` so lazy backends are covered.
|
Good call — widening the test to |
MarcoGorelli
left a comment
There was a problem hiding this comment.
thanks @winklemad ! and thanks @FBruzzesi for review
|
Thanks @FBruzzesi @MarcoGorelli 🙏 — and appreciate the nudge to widen the test to |

Description
On the pyarrow backend,
clip(andclip_lower/clip_upper) replaced a null value in the data with a clip bound instead of preserving it:The cause is that pyarrow's
min_element_wise/max_element_wiseskip nulls by default, somin_element_wise([null], 2)returns2. Passingskip_nulls=Falsekeeps a null as null, matching Polars and pandas.This is distinct from #2889, which was about
Nonebounds (unbounded clip); here the null is in the data. The existingcliptests use no nulls in the data, so this was uncovered.What type of PR is this? (check all applicable)
Related issues
AI assistance
I used Claude Code to help locate the divergence (a differential comparison of pyarrow vs pandas/polars) and draft the fix and test. I reviewed every line, reproduced the bug and the fix locally, and take full responsibility for the change.
Checklist
clip,clip_lowerandclip_upper(the whole class) in the pyarrow backendtest_clip_preserves_nulladded; fails onmainfor pyarrow, passes after; fullclip_test.pygreenruff checkandruff formatclean