Skip to content

[python] Keep projected nested ROW predicates on native reads - #10637

Merged
JingsongLi merged 3 commits into
apache:masterfrom
JingsongLi:codex/native-read-write-gaps
Oct 18, 2026
Merged

JingsongLi merged 3 commits into
apache:masterfrom
JingsongLi:codex/native-read-write-gaps

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Keep existing PyPaimon nested ROW projection filters on Native scans and reads, rather than falling back when the projected leaf name is not a top-level table field.

Depends on apache/paimon-rust#1153. This PR remains draft until that core support is merged. Native CI continues building Rust main; no fallback or revision pin is introduced to bypass the dependency.

The existing Python API is unchanged: project profile.detail.score, then filter profile_detail_score using the builder's predicate builder.

Changes

  • Retain each list projection's physical source path on predicates, including top-level columns renamed by collision suffixes. Preserve paths through index/literal rewrites and serialization.
  • Pass physical roots and descendant paths to the Rust core predicate builder; Rust owns path validation, typed literals, exact evaluation and schema evolution.
  • Align Python with the current Java NestedFieldTransform behavior: nested leaves cannot prune using an unrelated parent/scalar column's stats, partition/bucket keys or indexes.
  • Fix a.b / real a_b collisions in Python exact filtering and Native transport. Preserve correct AND/OR pruning, row tracking and explicit partition-filter validation.

Tests

  • Reproduced a real baseline Native fallback and guarded new Native tests against Python scan/read fallback.
  • New coverage spans append/PK tables, batch/stream reads, NULL ancestors, comparisons/string predicates/ranges/membership, projection changes and LIMIT, field promotion/rename/drop-readd, empty/null-containing IN and NOT IN, reversed BETWEEN bounds, optimized LIKE, pickle, and collision cases with full statistics, separate files, partitioning, 8 buckets and AND/OR.
  • Combined Python/Native regression: 998 passed, 2 skipped, 35 subtests passed, with 1021 Native plans and 1183 Native reads exercised.
  • Additional review regressions: 96 collision cases, 26 partition tests and 354 existing predicate/projection/auth/index/bucket tests passed (groups overlap).
  • Flake8 and git diff --check passed. Native tests ran against the built companion extension; extension rebuilds were completed before starting Native test runs.

Review follow-up

  • Rebind filters by their physical paths after projection reordering, including collisions between a.b and a real a_b column. Reject a removed nested path rather than silently filtering the unrelated physical column.
  • Keep top-level filter-only inputs in the Python adapter with aliases assigned together with output fields, preserving complete mixed AND/OR filters. Support ordinary physical predicates without a stored projection path. Native flattening reads only final output dependencies.
  • Reject nested aliases in partition-only chunk-shuffle filters.
  • Expand collision coverage to 192 cases and add 74 filter-only/row-ID cases covering append/PK, Native/Python, Arrow/batch-reader/iterator, duplicate projections, whole ROWs, NULL ancestors and overlapping PK commits.
  • Final combined regression result: 1637 passed, 2 skipped, 39 subtests passed across 23 related Python suites, with 1355 Native plans and 1595 Native reads exercised. Extension compilation completed before Native tests started; no rebuild or install ran during the tests.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JingsongLi
JingsongLi marked this pull request as ready for review October 18, 2026 04:54
@JingsongLi
JingsongLi merged commit 4eec129 into apache:master Oct 18, 2026
14 of 15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants