Skip to content

fix(pyarrow): keep nulls through clip instead of replacing them with a bound - #3906

Merged
FBruzzesi merged 3 commits into
narwhals-dev:mainfrom
winklemad:fix/arrow-clip-preserve-null
Sep 6, 2026
Merged

FBruzzesi merged 3 commits into
narwhals-dev:mainfrom
winklemad:fix/arrow-clip-preserve-null

Conversation

@winklemad

Copy link
Copy Markdown
Contributor

Description

On the pyarrow backend, clip (and clip_lower / clip_upper) replaced a null value in the data with a clip bound instead of preserving it:

import narwhals as nw, pyarrow as pa
s = nw.from_native(pa.chunked_array([[1, None, 3]]), series_only=True)
s.clip(0, 2).to_list()
# main:   [1, 2, 3]     <- the null became the upper bound
# polars: [1, None, 2]  <- null preserved

The cause is that pyarrow's min_element_wise / max_element_wise skip nulls by default, so min_element_wise([null], 2) returns 2. Passing skip_nulls=False keeps a null as null, matching Polars and pandas.

This is distinct from #2889, which was about None bounds (unbounded clip); here the null is in the data. The existing clip tests use no nulls in the data, so this was uncovered.

What type of PR is this? (check all applicable)

  • 🐛 Bug Fix

Related issues

  • None

AI assistance

  • AI tools were used.

I used Claude Code to help locate the divergence (a differential comparison of pyarrow vs pandas/polars) and draft the fix and test. I reviewed every line, reproduced the bug and the fix locally, and take full responsibility for the change.

Checklist

  • Fix applied to clip, clip_lower and clip_upper (the whole class) in the pyarrow backend
  • Regression test test_clip_preserves_null added; fails on main for pyarrow, passes after; full clip_test.py green
  • ruff check and ruff format clean

…a bound

pyarrow's min_element_wise / max_element_wise skip nulls by default, so
clip, clip_lower and clip_upper on the pyarrow backend replaced a null
value with the clip bound (e.g. [1, None, 3].clip(0, 2) returned
[1, 2, 3]) instead of preserving it. Polars and pandas keep the null.

Pass skip_nulls=False so a null stays null, matching the other backends.

@FBruzzesi FBruzzesi left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @winklemad for spotting and fixing the issue. Left a comment to extend the test to lazy backends as well. Once we do that, it's ready to merge

Comment thread tests/expr_and_series/clip_test.py Resolved
@FBruzzesi FBruzzesi added fix pyarrow Issue is related to pyarrow backend labels Sep 5, 2026
Testing clip null-preservation against every backend (not just the eager
ones) showed the SQL backends had the same bug: `least` / `greatest`
skip nulls, so clip, clip_lower and clip_upper replaced a null with the
bound. Guard each so a null input stays null, and widen the regression
test to `constructor` so lazy backends are covered.
@winklemad

Copy link
Copy Markdown
Contributor Author

Good call — widening the test to constructor caught that the SQL backends (duckdb, and any other least/greatest-based one) had the same bug: those functions skip nulls too, so clip was replacing a null with the bound there as well. Guarded clip, clip_lower and clip_upper in _sql/expr.py with a null check, and the test now runs across every backend (dask was already fine). Full clip suite is green.

@MarcoGorelli MarcoGorelli left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @winklemad ! and thanks @FBruzzesi for review

Comment thread src/narwhals/_sql/expr.py Resolved
@FBruzzesi FBruzzesi removed the pyarrow Issue is related to pyarrow backend label Sep 6, 2026
@FBruzzesi
FBruzzesi merged commit d2d83fa into narwhals-dev:main Sep 6, 2026
42 of 44 checks passed
@winklemad

Copy link
Copy Markdown
Contributor Author

Thanks @FBruzzesi @MarcoGorelli 🙏 — and appreciate the nudge to widen the test to constructor; that's what surfaced the same null-dropping in the least/greatest SQL path, so it ended up a better fix for it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants