You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit df71d2a
Browse filesBrowse the repository at this point in the historyBrowse files
authored
Fix duplicate filtering path in Arrow task batches (apache#3448)
Closesapache#3272
## What this changes
This PR updates the Arrow scan path in `_task_to_record_batches` to
avoid redundant filtering when there are no positional deletes.
- Keeps predicate pushdown in `Scanner.from_fragment` as the only filter
path when `positional_deletes` is absent.
- Applies `current_batch.filter(pyarrow_filter)` only in the
positional-delete path, after deletes are applied.
- Preserves empty-batch handling after both delete application and
conditional filtering.
## Why
The previous flow could perform an extra table-level refilter even when
the scanner already applied the predicate. This change removes that
stale workaround path while keeping correct behavior for positional
delete scenarios.
## Tests
Added regression coverage in `tests/io/test_pyarrow.py`:
-
`test_task_to_record_batches_filter_without_positional_deletes_avoids_table_refilter`
-
`test_task_to_record_batches_filter_with_positional_deletes_handles_empty_batch`
Validated locally:
- `python -m pytest tests/io/test_pyarrow.py -q -k
"task_to_record_batches_nanos or
filter_without_positional_deletes_avoids_table_refilter or
filter_with_positional_deletes_handles_empty_batch"`
- `make lint`
---------
Co-authored-by: Gayathri Srividya Rajavarapu <[email protected]>
Co-authored-by: Claude Sonnet 4.6 <[email protected]>
Co-authored-by: Fokko Driesprong <[email protected]>
# Temporary fix until PyArrow 21 is released ( https://github.com/apache/arrow/pull/46057 )
"""Regression: filter after positional deletes must not raise even when the result is empty.
3361
+
3362
+
PyArrow < 21 raises IndexError from RecordBatch.filter(Expression) when the result has
3363
+
zero rows (fixed in https://github.com/apache/arrow/pull/46057). This test ensures the
3364
+
positional-delete path handles that case gracefully and yields no batches.
0 commit comments