Repository navigation
branch-4.2: [improvement](zonemap) Support column-vs-column expression pruning (#67774) - #69659
Merged
Merged
Conversation
…runing ### What problem does this PR solve? Issue Number: None Related PR: apache#67774 Problem Summary: Backport column-to-column expression zone map pruning to branch-4.2. Share the six interval non-match rules with literal comparisons, retain conservative handling for NULL, NaN, truncated strings and unsupported types, and carry the Parquet v1 correctness guards and unit/regression coverage. Preserve the branch-4.2 BloomFilterProbe interfaces. Unlike master, 4.2 substitutes the hidden version column after reading a single-version rowset, so exclude its on-disk placeholder statistics from segment and page expression pruning. ### Release note Support column-to-column comparisons in expression zone map pruning for native segments and Parquet V2 row groups. ### Check List (For Author) - Test: clang-format 16.0.6, check-format and git diff --check passed. All 43 regression golden labels match; the generated golden file is unchanged from the original PR. BE build and focused unit-test execution were attempted; no local runtime test has completed with the incomplete third-party package. - Behavior changed: Yes. More predicates can prune; unsupported statistics keep candidates, and hidden-version placeholders are excluded on branch-4.2. - Does this need documentation: No. Backport of an existing feature. (cherry picked from commit 349cd15)
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
|
/review |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Static review of #69659 at 601c18a found no actionable defects. All 11 changed files and affected native/Parquet callers, metadata conversion, capability gates, residual evaluation, and tests were inspected. No existing inline findings require carrying forward. No additional user focus was supplied.
Critical checkpoint conclusions:
- Goal and behavior: the six interval rules conservatively support same-domain column comparisons in native segments and Parquet V2 row groups. The slot/literal refactor preserves the prior rules. Focused unit tests and native regressions cover the feature; execution was not performed.
- Scope and clarity: changes remain focused on comparison capability, metadata safeguards, branch-specific hidden-version handling, and tests.
- Concurrency and locks: no new thread, lock, atomic, or shared mutable scan state. Evaluation contexts/counters are scoped to each segment, group, or page; scanner/segment expression cloning remains intact.
- Lifecycle and static initialization: slot types and immutable zone maps retain shared ownership through evaluation. No new global initialization, ownership cycle, or cleanup transition.
- Configuration: no new configuration. Native/legacy consumers retain the existing expression-zone-map switch; V2 retains its existing fixed metadata-pruning behavior.
- Compatibility: no persisted format or public protocol change. Old floating metadata, truncated native strings, unsafe file/table type conversions, missing metadata, and unknown Parquet NaN presence retain candidates. Compact external bounds were checked against the Parquet statistics contract. Legacy reader edits are conservative pruning safeguards, with no change to the V2 decode boundary.
- Thrift/protobuf: no definitions changed; cross-release field-ID audit is inapplicable.
- Parallel paths: native segment/page, Parquet V1 row-group/page, V2 row-group/page, self-comparisons, and compound AND/OR paths were traced. Distinct-column page alignment remains conservative; dictionary and Bloom gates are unchanged.
- Special conditions: casts and incompatible/unsupported types are excluded; NULL-only checks precede bound access; NaN/Inf and string-bound safeguards prevent unsafe proofs. Native hidden-version exclusions match the single-version substitution condition at both pruning levels. Planner/scanner invariants preserve schema slot order.
- Test coverage: added tests cover all operators, touching/overlapping ranges, point inequality, NULL/missing statistics, unsupported types, strings, decimals, floating uncertainty, variant paths, pruning counters, and hidden versions. There is no new external-writer Parquet end-to-end fixture for the newly reachable V2 pair path or legacy decimal/NaN safeguards.
- Changed results: the added golden counts and ordered IDs match the fixture data by inspection; profile polling uses the existing COMPLETE-aware API. Static inspection cannot independently prove output-generation provenance or runtime passes.
- Observability: filtered-segment/page and unusable-evaluation counters remain connected. No new per-row timer or logging burden.
- Transactions and persistence: no EditLog, version-publication, durability, or failover mutation changes.
- Writes and atomicity: production changes only read/prune metadata; no write protocol or crash-atomicity change. Optional unavailable metadata falls back; existing metadata-read failures propagate.
- FE/BE variables: no new transmitted variables or send paths.
- Performance: interval proof work is per metadata unit, with no new data I/O or per-row allocation. Legacy fallback may conservatively reduce pruning; no actionable performance defect was established, and no benchmark was run.
- Other concerns and limits: no unresolved correctness candidate or uncovered changed-file scope remains. This was static inspection only: no builds, tests, benchmarks, source edits, or submodule initialization. Runtime and interoperability validation remain CI coverage limits.
Proposed finding set: no new inline comments and no existing blocking IDs.
Review converged after 2 round(s).
### What problem does this PR solve? Issue Number: None Related PR: apache#67774, apache#69659 Problem Summary: The backported VARIANT-path fixture inserts parse_to_variant values with max-subcolumns 0 into a table whose fuzzy defaults can differ. Branch-4.2 retains legacy VARIANT execution, which rejects that conversion, so the fixture fails before its path-comparison checks. Defer its setup, two queries and their existing golden blocks until Variant V2 is integrated. Retain the production VARIANT exclusion and the hidden-version regression. ### Release note None ### Check List (For Author) - Test: Manual test: git diff --check; verified all 41 remaining query labels match golden blocks, all remaining golden bytes are unchanged, and the hidden-version test block is unchanged. Runtime tests not run locally; run buildall will be requested on the updated PR. - Behavior changed: No - Does this need documentation: No
Contributor
Author
|
run buildall |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
|
skip buildall |
yiguolei
approved these changes
Oct 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Related PR: #67774
Problem Summary:
Backport #67774 to
branch-4.2. Comparisons such asa != banda < bcan use both columns' zone-map ranges to discard native segments and Parquet V2 row groups that cannot match. The existing slot-vs-literal path shares the same six interval rules; dictionary and Bloom-filter gates remain unchanged.The backport preserves branch-4.2's
BloomFilterProbeinterfaces instead of importing the unrelated master metadata-probe/array-contains refactor. It also retains the original NULL, NaN, string-bound, unsupported-type and legacy Parquet safeguards, plus the unit tests and remaining regression coverage.Branch-specific adaptation: master already reads hidden version columns through a constant reader with substituted zone maps. Branch-4.2 still substitutes
__DORIS_VERSION_COL__after reading a single-version rowset, so this backport excludes that column's on-disk placeholder from both segment- and page-level expression zone-map pruning. The original enabled/disabled hidden-version regression is retained.Deferred VARIANT regression: omit the VARIANT-path fixture and its two result blocks until Variant V2 is integrated into branch-4.2. On this branch, the legacy VARIANT parser returns max-subcolumns 0 while fuzzy table defaults can differ, so INSERT fails before the path-comparison checks. The VARIANT exclusion in the production zone-map gate remains intact. The hidden-version regression is retained.
Release note
Support column-to-column comparisons in expression zone-map pruning for native segments and Parquet V2 row groups.
Check List (For Author)
build-support/clang-format.shandbuild-support/check-format.shwith clang-format 16.0.6;git diff --check.qt_labels match the golden file. Only the two deferred VARIANT result blocks were removed from the runner-generated source PR output; all other result blocks are unchanged.BUILD_TYPE=ASAN ./build.sh --be -j48. The cached third-party package lacks Paimon-Rust headers/archive; stopped the automatic full dependency rebuild.BUILD_TYPE=ASAN ./run-be-ut.sh -j48 --run --filter='ExprZonemapFilterTest.*'. Configuration stopped because the checkout lacks thecontrib/openblasandcontrib/faissCMake sources. No local unit-test or regression runtime pass is claimed; CI validation is required.No FE/BE protocol, storage-format, configuration, transaction or data-write changes.
Check List (For Reviewer who merge this PR)