Skip to content

Preserve primitive Iceberg pruning beside unsupported complex null conjuncts #5883

Open

Description

@ErikBPF

What is the problem?

Iceberg residual serialization currently requires both sides of an AND to convert. Collection null checks cannot bind in the pinned iceberg-rust version, so a query such as WHERE l IS NOT NULL AND id > 5 drops the entire residual and loses native row-group pruning on id.

The post-scan filter still enforces the complete predicate, so this is a performance limitation, not incorrect results. PR #5732 allows collection-null scans to remain native. Its review also requests skipping unbindable complex-column null predicates before serialization; the conservative whole-conjunction behavior should remain until safe partial pushdown is implemented.

Proposed improvement

Preserve representable conjuncts when safely weakening a residual in positive polarity. Keep the exact post-scan filter. Do not simply change every AND to keep whichever side converted: beneath NOT, that would strengthen the predicate and can incorrectly prune rows. Likewise, dropping an unsupported OR branch is unsafe.

Validation

  • A list/map null predicate combined with a primitive predicate retains primitive row-group pruning and matches Spark results.
  • Nested NOT, OR, and mixed conjunctions preserve all qualifying rows.
  • Missing or unsupported predicates continue to disable pushdown whenever safe weakening cannot be established.
  • Cover null values and files containing both matching and nonmatching primitive values; verify actual pruning metrics, not only serialized predicate shape.

Requested in the review of #5732: #5732 (review)

Activity

  1. rich7420 commented on Oct 2, 2026

    @rich7420
    Contributor

    take

  2. added 2 commits that reference this issue on Oct 10, 2026
    89d073b
    03cb063
  3. added 2 commits that reference this issue on Oct 22, 2026
    622d7e9
    e0292d3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:scanParquet scan / data readingenhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions