Skip to content

[Python] Filter on __row_index #35301

Closed as not planned
Closed as not planned

Description

@Fokko

Describe the enhancement requested

PyIceberg uses positional deletes to indicate which rows should be omitted. In the normal table.filter(..) a boolean mask is allowed, but not when constructing a scanner. Therefore we first have to read the data into a table, and then filter it. It would simplify the code (and also speed up) if this can be done directly when reading the data.

Having a __row_index where we can filter would greatly simplify this logic.

Component(s)

Python

Activity

  1. jorisvandenbossche commented on Apr 26, 2023

    @jorisvandenbossche
    Member

    Given you want a positional delete, would this rather be a "take" operation than a "filter"? I know this is essentially the same (under the hood, filtering an array also does a "take" of the required values), but conceptually for a Dataset this might be different. A filter can be defined with an expression, but a "take" is always with actual materialized values. And so we already have a Dataset.take() method that does that.

    So even if you start with a boolean filter, you should be able to already use take() by converting the boolean mask to indices with pyarrow.compute.indices_nonzero right now (and we could still consider adding that as a convenience to Dataset.filter to do that for you)

    I am not fully sure how Scanner::TakeRows works given that positional indices depend on the order that data is scanned. I assume it follows the order of the actual vector of fragments.

  2. westonpace commented on May 4, 2023

    @westonpace
    Member

    The scanner assumes that inputs are extremely large (or potentially infinite) streams. How would a mask be provided in that case?

    I think it would be easier to create an order-dependent node that takes in a list of indices and skips those indices. Would that work? I don't think it would be much effort to create.

  3. jorisvandenbossche commented on May 5, 2023

    @jorisvandenbossche
    Member

    I think it would be easier to create an order-dependent node that takes in a list of indices and skips those indices.

    So like the current FetchNode, but then using an array of indices instead of a slice as input argument? (and omitting vs selecting)

  4. Fokko commented on May 5, 2023

    @Fokko
    ContributorAuthor

    Based on @jorisvandenbossche his suggestion, I've updated the PR: https://github.com/apache/iceberg/pull/6775/files#diff-49144c27eab7e03926b0310aa8b513dbeb9c8d2a0d33bacf8dedbd88b4680aacR782-R787.

    The .take() operation works perfectly to remove the deleted rows. The next step would be to first apply the deletes filter, and then a user filter (an additional filter, after the mask has been applied), and push all of this down to Arrow in a single filter operation. This would avoid materializing the table, and then filtering it even further down.

  5. westonpace commented on May 8, 2023

    @westonpace
    Member

    So like the current FetchNode, but then using an array of indices instead of a slice as input argument? (and omitting vs selecting)

    Yes, it would need a sequencing queue like the fetch node and the process function would be something like...

    row_start = current_offset;
    row_end = row_start + batch.length;
    deleted_indices = get_deleted_indices_in_range(row_start, row_end)
    if deleted_indices:
      normalized = deleted_indices - row_start
      kept_indices = range(0, batch.length) - normalized
      batch = batch.take(kept_indices)
    output.input_received(batch)
    
  6. wjones127 commented on May 9, 2023

    @wjones127
    Member

    Have you thought about how you want to handle these masks in conjunction with filter pushdown? I wonder if the best approach would be to expose a new hidden column __row_index in the scanner. Then you can pass down a filter (which can prune row groups or even data pages), and then the data returned with have the original row indices in that __row_index column. From there you could filter the results for where the __filename column matches the filename and __row_index matches doesn't match any of the deleted rows in the mask. What do you think of that?

  7. westonpace commented on May 10, 2023

    @westonpace
    Member

    I was thinking of using the exec batch's index. However, I like the __row_index idea better since that wouldn't require any sequencing of batches so it should be more performant.

  8. rdblue commented on May 29, 2023

    @rdblue
    Contributor

    The __row_index idea is what we do in the Java Iceberg implementation. That works well.

  9. changed the title [-][Python] Allow passing in a mask when creating a scanner[/-] [+][Python] Filter on `__row_index`[/+] on Jul 19, 2023
  10. added
    Status: stale-warningIssues and PRs flagged as stale which are due to be closed if no indication otherwise
    on Feb 18, 2026
  11. thisisnic commented on Feb 18, 2026

    @thisisnic
    Member

    This issue has been marked as stale because it has had no activity in the past 2 years. Please remove the stale label or comment below, or this issue will be closed in 14 days.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions