Repository navigation
Commit ec886f3
[fix](nereids) Disambiguate NULL/OFFSET metadata from same-named nested fields (#65805)
### What problem does this PR solve?
Issue Number: None
Related PR: #65591, #66380
Problem Summary:
Nested-column pruning (feature series #59263 / #61888 / #64535) makes
Doris read only the referenced parts of a struct/map/array column. Two
kinds of special *meta* access paths serve that purpose: a terminal
`NULL` component means the query only needs the null flag (`IS [NOT]
NULL` on a column or nested field), and a terminal `OFFSET` component
means only offsets are needed (`length`/`cardinality`-style functions).
The BE satisfies them in `NULL_MAP_ONLY` / `OFFSET_ONLY` meta-read modes
that skip the payload data entirely. While the optimization itself is
sound, the way meta paths were expressed on the wire and handled on both
sides had several correctness holes:
* **Wire ambiguity.** Every access path was sent as a `DATA`-typed list
of string components, and `NULL`/`OFFSET` lived in the same list as real
child names. There was no way to tell a metadata selector apart from a
struct field literally named `OFFSET` or `NULL`, so such a field could
be swallowed as metadata and its data silently pruned away (wrong query
results), and there was no FE/BE version contract to evolve the encoding
during rolling upgrades.
* **FE null-map correctness.** A slot can be nullable for two different
reasons: its physical column has a null bitmap, or an outer join merely
made the output slot nullable. The old logic keyed on expression/slot
nullability, so a physical `NOT NULL` dimension column of a `LEFT JOIN`
received a `[col, NULL]` path and the BE aborted trying to read a null
map from a `NOT NULL` column. Meta paths were also generated for
non-nullable struct fields / non-null structs and for variant
sub-columns, which do not support meta-only reads.
* **FE pruning bookkeeping.** `NULL`/`OFFSET` were detected by string
suffix, so real fields with those names collided inside the pruning
logic too; and predicate meta paths were stripped or broadened
(map-star) whenever regular data paths existed, losing metadata-only
reads or over-reading payload data.
* **BE routing/read-mode logic.** A terminal `NULL`/`OFFSET` was treated
as current-level metadata at every container, meta-only modes could be
entered even when a sibling `DATA` path still required payload reads,
sibling `META` paths were dropped when unrelated data projections
existed, and Map descendant routing depended on physical key/value child
column names while silently ignoring unknown selectors.
What this PR does:
* **Versioned, type-selected access paths.** Adds an optional `version`
to `TColumnAccessPath` (thrift + protobuf + FE/BE descriptor conversion;
`TCOLUMN_ACCESS_PATH_VERSION_LEGACY = 0`,
`TCOLUMN_ACCESS_PATH_VERSION_TYPED = 1`). In the typed format the path
type is authoritative: `DATA` selects `data_access_path`, `META` selects
`meta_access_path`, and `NULL`/`OFFSET` are only ever emitted as typed
`META` paths. New BEs still decode the legacy all-`DATA` encoding from
old FEs, so the supported rolling upgrade is: upgrade all BEs first,
then let FEs send typed paths.
* **FE** (`AccessPathExpressionCollector`, `NestedColumnPruning`,
`AccessPathPlanCollector`, `DescriptorToThriftConverter`): marks
`NULL`/`OFFSET` collector contexts as `META`; checks the physical
column's nullability (`getOriginalColumn().isAllowNull`) instead of slot
nullability before emitting null-map paths, so outer-join-nullable `NOT
NULL` columns and non-nullable fields/structs no longer get `NULL`
paths; emits the parent-struct `NULL` path for `element_at(...) IS NULL`
only when actually needed; falls back to plain data reads for variant
sub-columns; keeps the exact meta path when a sibling data path is also
read; preserves fields referenced by unchanged predicates; and keys
pruning on the path type instead of string suffixes, rejecting
unsupported `META` paths.
* **BE** (`ColumnIterator` / `column_reader.cpp`): validates path
versions and type-selected payloads; partitions current data / current
metadata / descendant routing with explicit per-container ownership
(Struct owns the null map; Map/Array and string-like scalar columns own
both null map and offsets); derives `NULL_MAP_ONLY` / `OFFSET_ONLY` only
when no current or predicate data path requires payload, and marks data
children `SKIP` consistently; keeps sibling `META` paths explicit so an
unrelated `DATA` projection cannot disable a metadata-only predicate
read; routes Map children by the logical `KEYS`/`VALUES` selectors
(wildcard expansion, arbitrary nesting) independent of physical child
column names; rejects unrecognized Map selectors instead of silently
pruning everything; and documents the full routing flow.
* **Tests & docs.** BE `ColumnReaderTest` coverage for typed-vs-legacy
encoding, metadata-vs-same-named-data disambiguation, meta-read-mode
legality, sibling-data-path handling and Map selector validation;
`SlotDescriptor` protobuf round-trip of versions and payloads; FE
`PruneNestedColumnTest` / `DescriptorToThriftConverterTest` cases; the
`nereids_rules_p0/column_pruning` regression suites updated to assert
the typed `META` paths, plus the new `left_join_not_null_column` suite
guarding the outer-join null-map crash.
### Release note
None
### Check List (For Author)
- Test:
- FE unit test: `./run-fe-ut.sh --run
'org.apache.doris.nereids.rules.rewrite.PruneNestedColumnTest,org.apache.doris.analysis.DescriptorToThriftConverterTest'`
(74 passed on the final rebased FE tree)
- BE unit test: `./run-be-ut.sh --run
--filter='ColumnReaderTest.*:SlotDescriptorTest.AccessPathsPreservedThroughProtobuf'`
(50 passed before the final rebase; the final rebuild was blocked before
GTest by the missing master-added
`thirdparty/installed/lib64/liblance_c.a` in this environment)
- Regression test: `nereids_rules_p0/column_pruning` —
`string_length_column_pruning`, `null_column_pruning`,
`nested_container_offset_pruning`, `lambda_null_pruning`,
`left_join_not_null_column` (passed before the final rebase; the final
rerun was blocked by the same missing dependency)
- Format checks: `build-support/check-format.sh`, `git diff --check`
- Behavior changed: Yes. Typed `META` access paths with an explicit
version replace the legacy all-`DATA` encoding between new FEs and BEs
(upgrade BEs before FEs); struct fields literally named `NULL`/`OFFSET`
are now read correctly instead of being pruned as metadata; `IS NULL` on
outer-join-nullable `NOT NULL` columns no longer crashes the BE; invalid
Map descendant selectors now return an internal error instead of being
silently ignored.
- Does this need documentation: No
---------
Co-authored-by: Hu Shenggang <[email protected]>1 parent 62327d3 commit ec886f3
17 files changed
Lines changed: 2056 additions & 687 deletions
File tree
- be
- src
- runtime
- storage/segment
- test
- runtime
- storage/segment
- fe/fe-core/src
- main/java/org/apache/doris
- analysis
- nereids/rules/rewrite
- test/java/org/apache/doris
- analysis
- nereids/rules/rewrite
- gensrc
- proto
- thrift
- regression-test
- data/nereids_rules_p0/column_pruning
- suites/nereids_rules_p0/column_pruning
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
104 | 104 | | |
105 | 105 | | |
106 | 106 | | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
107 | 110 | | |
108 | 111 | | |
109 | 112 | | |
| |||
160 | 163 | | |
161 | 164 | | |
162 | 165 | | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
163 | 169 | | |
164 | 170 | | |
165 | 171 | | |
| |||
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
| 27 | + | |
27 | 28 | | |
28 | 29 | | |
29 | 30 | | |
| |||
500 | 501 | | |
501 | 502 | | |
502 | 503 | | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
| 507 | + | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
503 | 524 | | |
504 | 525 | | |
505 | 526 | | |
506 | 527 | | |
507 | | - | |
508 | | - | |
509 | | - | |
510 | | - | |
511 | | - | |
512 | | - | |
513 | | - | |
514 | | - | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
| 531 | + | |
| 532 | + | |
| 533 | + | |
| 534 | + | |
| 535 | + | |
| 536 | + | |
| 537 | + | |
| 538 | + | |
| 539 | + | |
| 540 | + | |
| 541 | + | |
| 542 | + | |
| 543 | + | |
| 544 | + | |
| 545 | + | |
515 | 546 | | |
516 | 547 | | |
517 | 548 | | |
| |||
542 | 573 | | |
543 | 574 | | |
544 | 575 | | |
| 576 | + | |
| 577 | + | |
| 578 | + | |
545 | 579 | | |
546 | 580 | | |
547 | 581 | | |
| |||
618 | 652 | | |
619 | 653 | | |
620 | 654 | | |
621 | | - | |
622 | | - | |
623 | | - | |
624 | | - | |
| 655 | + | |
| 656 | + | |
| 657 | + | |
| 658 | + | |
625 | 659 | | |
626 | 660 | | |
627 | 661 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
15 | 15 | | |
16 | 16 | | |
17 | 17 | | |
| 18 | + | |
18 | 19 | | |
19 | 20 | | |
| 21 | + | |
20 | 22 | | |
21 | 23 | | |
22 | 24 | | |
| |||
214 | 216 | | |
215 | 217 | | |
216 | 218 | | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
217 | 274 | | |
0 commit comments