…t> (apache#62631)

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: apache#62205

Problem Summary:

PR apache#62205 introduced the `length(str)` OFFSET optimization for nested
column pruning. When `length()` is applied to a string-like nested
field, the pruner marks the access path with an OFFSET suffix so BE can
read only the offset array instead of full element data.

However, the OFFSET path dedup logic operates at **slot granularity**:
if ANY non-OFFSET path exists for the same slot, ALL OFFSET paths are
stripped. This is incorrect for `array<struct<...>>` columns where
different struct fields have independent access patterns.

**Bug 1 (crash):** Given a query like:
```sql
SELECT array_match_all(x -> length(struct_element(x, 'str_field')) > 0, arr),
       struct_element(element_at(arr, 1), 'int_field')
FROM t
```
The `int_field` non-OFFSET path causes the `str_field` OFFSET path to be
stripped. BE never reads `str_field` data → crash or wrong results.

**Fix:** Replace slot-level OFFSET dedup with per-field bidirectional
prefix matching. An OFFSET path `P+["OFFSET"]` is only stripped when a
non-OFFSET path Q shares a prefix relationship with P (covering the same
container or ancestor), not when Q accesses a sibling struct field.

**Bug 2 (wrong pruning):** When `isStringOffsetOnly=true` AND
`accessPartialChild=true` (e.g., `cardinality(arr) + arr[*].f1`),
`pruneDataType()` returns the full type instead of pruning unused
fields.

**Fix:** Add `!accessPartialChild` guard to the `isStringOffsetOnly`
check.

### Release note

Fix a BE crash caused by nested column pruning incorrectly stripping
OFFSET access paths for array<struct> columns when different struct
fields have independent access patterns (e.g., length() on one field +
direct access on another).

Co-authored-by: Copilot <[email protected]>

…t> (apache#62631)

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: apache#62205

Problem Summary:

PR apache#62205 introduced the `length(str)` OFFSET optimization for nested
column pruning. When `length()` is applied to a string-like nested
field, the pruner marks the access path with an OFFSET suffix so BE can
read only the offset array instead of full element data.

However, the OFFSET path dedup logic operates at **slot granularity**:
if ANY non-OFFSET path exists for the same slot, ALL OFFSET paths are
stripped. This is incorrect for `array<struct<...>>` columns where
different struct fields have independent access patterns.

**Bug 1 (crash):** Given a query like:
```sql
SELECT array_match_all(x -> length(struct_element(x, 'str_field')) > 0, arr),
       struct_element(element_at(arr, 1), 'int_field')
FROM t
```
The `int_field` non-OFFSET path causes the `str_field` OFFSET path to be
stripped. BE never reads `str_field` data → crash or wrong results.

**Fix:** Replace slot-level OFFSET dedup with per-field bidirectional
prefix matching. An OFFSET path `P+["OFFSET"]` is only stripped when a
non-OFFSET path Q shares a prefix relationship with P (covering the same
container or ancestor), not when Q accesses a sibling struct field.

**Bug 2 (wrong pruning):** When `isStringOffsetOnly=true` AND
`accessPartialChild=true` (e.g., `cardinality(arr) + arr[*].f1`),
`pruneDataType()` returns the full type instead of pruning unused
fields.

**Fix:** Add `!accessPartialChild` guard to the `isStringOffsetOnly`
check.

### Release note

Fix a BE crash caused by nested column pruning incorrectly stripping
OFFSET access paths for array<struct> columns when different struct
fields have independent access patterns (e.g., length() on one field +
direct access on another).

…array<struct> (apache#62631)

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: apache#62205

Problem Summary:

PR apache#62205 introduced the `length(str)` OFFSET optimization for nested
column pruning. When `length()` is applied to a string-like nested
field, the pruner marks the access path with an OFFSET suffix so BE can
read only the offset array instead of full element data.

However, the OFFSET path dedup logic operates at **slot granularity**:
if ANY non-OFFSET path exists for the same slot, ALL OFFSET paths are
stripped. This is incorrect for `array<struct<...>>` columns where
different struct fields have independent access patterns.

**Bug 1 (crash):** Given a query like:
```sql
SELECT array_match_all(x -> length(struct_element(x, 'str_field')) > 0, arr),
       struct_element(element_at(arr, 1), 'int_field')
FROM t
```
The `int_field` non-OFFSET path causes the `str_field` OFFSET path to be
stripped. BE never reads `str_field` data → crash or wrong results.

**Fix:** Replace slot-level OFFSET dedup with per-field bidirectional
prefix matching. An OFFSET path `P+["OFFSET"]` is only stripped when a
non-OFFSET path Q shares a prefix relationship with P (covering the same
container or ancestor), not when Q accesses a sibling struct field.

**Bug 2 (wrong pruning):** When `isStringOffsetOnly=true` AND
`accessPartialChild=true` (e.g., `cardinality(arr) + arr[*].f1`),
`pruneDataType()` returns the full type instead of pruning unused
fields.

**Fix:** Add `!accessPartialChild` guard to the `isStringOffsetOnly`
check.

### Release note

Fix a BE crash caused by nested column pruning incorrectly stripping
OFFSET access paths for array<struct> columns when different struct
fields have independent access patterns (e.g., length() on one field +
direct access on another).

… path series (#68214)

Cherry-pick of #62205, #62315, #62631, #62304, #63229, #63736, #64486 to
branch-4.2 (the meta path series, applied in merge order).

Adaptations for branch-4.2:
* `AccessPathInfo` already lives in `org.apache.doris.analysis` on this
branch; the new `ACCESS_STRING_OFFSET` constant was ported there
(aliasing the existing `ACCESS_OFFSET`).
* master-only types renamed to this branch's thrift types
(`ColumnAccessPath* -> TColumnAccessPath`, `ColumnAccessPathType ->
TAccessPathType`); `NestedColumnPruning` keeps branch-4.2's
`comparePathSegments`/variant branch alongside upstream's new logic.
* `LazyMaterializeTopN` merges upstream's restructure while preserving
branch-4.2's Lance lazy-materialization gate;
`LogicalJoin.getRightConditionSlot()` (a master-only helper) had to be
added for the ported `PushDownTopNThroughJoin`.
* `StringEmptyToLengthRuleTest` uses `ElementAt` (branch-4.2 folded
`StructElement` into `ElementAt`); the regression suite expectation for
the same reason.

REQUIRED EXTRA COMMIT: the FE now emits offset-only / null-only nested
access paths, which branch-4.2's BE could not read —
`null_column_pruning.groovy` aborted the BE with
`std::vector<...ColumnIterator>::operator[]: Assertion '__n <
this->size()' failed`. This PR therefore also cherry-picks the BE-side
prerequisite #61888 "[Exec](be) Support offset prue column and null
column in BE" (adapted to branch-4.2's `IColumn::mutate`/Defer COW shape
and `ColumnNullable::get_null_map_column_ptr()` signature). Please
review that extra commit carefully.

Testing on branch-4.2 (FE + BE rebuilt from this branch):
* FE unit tests: PruneNestedColumnTest (57),
PullUpProjectExprUnderTopNTest (28), OperativeColumnDeriveTest (4),
StringEmptyToLengthRuleTest (11), MaterializeProbeVisitorTest (8) — 108
tests, 0 failures.
* Regression: nereids_rules_p0/column_pruning (7 suites incl. the new
string_length/null/nested_container_offset/topn_expr_pullup/topn_lazy_nested
suites) 0 failures; shape_check
clickbench/tpcds_sf100/tpcds_sf1000/tpcds_sf10t_orc/tpcds_sf1000_constraints
suites touched by the series regenerated with -forceGenOut and green;
nereids_rules_p0 push_filter_through / null_un_safe_equals /
lazy_materialize_topn green.
* Known deviations: #62446 (struct field slot type for OFFSET-only
access) is not part of the requested series and was not included; the
BE-side `MapAccessAllWithOffsetDoesNotPropagateOffsetToKey` unit test
from #61888 was kept byte-identical to upstream although it may not hold
(test binary not built with MAKE_TEST=ON).

---------

Co-authored-by: Copilot <[email protected]>
Co-authored-by: Claude Opus 4.7 <[email protected]>
Co-authored-by: HappenLee <[email protected]>

…-named nested fields (#65805) (#68314)

### What problem does this PR solve?

Cherry-pick of #65805 (`[fix](nereids) Disambiguate NULL/OFFSET metadata
from same-named nested fields`) to `branch-4.2`.

The port also carries the two prerequisite commits that #65805 is
written on top of, because branch-4.2 stopped at the state just before
them:

| commit | PR | why it is needed |
| --- | --- | --- |
| `branch-4.2 [fix](NestedColumnPruning) Rewrite the NestedColumnPruning
meta-path handling ...` | #64535 | branch-4.2's FE pruning code was
exactly master-before-#64535; #65805's FE hunks are written against the
rewritten meta-path handling. |
| `branch-4.2 [feat](olap) Support lazy reading mode for pruned complex
columns` | #59263 | on master #59263 replaced #64535's `MetaPathStriper`
approach again (predicate metadata paths are kept,
`normalizePredicateMetaPathForAllAccessPath` /
`addPredicatePathsToFinalAllAccessPaths` / `expandMapStarPaths`), and
that is the code #65805 patches. |
| `branch-4.2 [fix](nereids) Disambiguate NULL/OFFSET metadata from
same-named nested fields` | #65805 | the requested pick. |

Master order for reference: `#61888 → #62205 → #62315 → #62631 → #62304
→ #63229 → #63736 → #64486 → #64535 → #59263 → … → #65805`; branch-4.2
already had everything up to #64486 (ported by #68214).

### What #65805 fixes

* **Versioned, type-selected access paths.** `TColumnAccessPath` gets an
optional `version` (`TCOLUMN_ACCESS_PATH_VERSION_LEGACY = 0`,
`TCOLUMN_ACCESS_PATH_VERSION_TYPED = 1`) that is carried through thrift,
protobuf and both FE/BE descriptor conversions. In the typed format the
path type is authoritative: `DATA` selects `data_access_path`, `META`
selects `meta_access_path`, and `NULL`/`OFFSET` are only ever emitted as
typed `META` paths. New BEs still decode the legacy all-`DATA` encoding,
so the supported rolling upgrade is: upgrade all BEs first, then let FEs
send typed paths.
* **FE.** Marks `NULL`/`OFFSET` collector contexts as `META`; checks the
physical column's nullability instead of slot nullability, so an
outer-join-nullable `NOT NULL` column no longer gets a `[col, NULL]`
path; falls back to plain data reads for variant sub-columns; keeps the
exact meta path when a sibling data path is also read; and keys pruning
on the path type instead of string suffixes.
* **BE.** Validates path versions and type-selected payloads; partitions
current data / current metadata / descendant routing with explicit
per-container ownership; derives `NULL_MAP_ONLY` / `OFFSET_ONLY` only
when no current or predicate data path requires payload; keeps sibling
`META` paths explicit; routes Map children by the logical
`KEYS`/`VALUES` selectors instead of physical child column names; and
rejects unrecognized Map selectors instead of silently pruning
everything.

### branch-4.2 adaptations

* branch-4.2 keeps the FE access-path model as thrift types instead of
master's `ColumnAccessPath` / `DescriptorToThriftConverter`, so the
typed version is stamped in `NestedColumnPruning.buildColumnAccessPaths`
(the branch's single producer) and preserved by
`SlotTypeReplacer.replaceIcebergAccessPathToId`.
* The master-only `DescriptorToThriftConverter` /
`DescriptorToThriftConverterTest` files do not exist on this branch and
were not added; the equivalent coverage lives in
`PruneNestedColumnTest`, whose `path()` / `metaPath()` helpers (and
`VariantPruningLogicTest`'s) now carry the typed version.
* Ported FE code was mapped onto this branch's names (`TAccessPathType`,
`TColumnAccessPath`, `isFunctionNullCheckPath`) and keeps the
segment-wise access-path comparator introduced by the branch's
nested-column-pruning port.
* `gensrc/thrift/PaloInternalService.thrift`: master uses field id `226`
for `enable_prune_nested_column`, but branch-4.2 already uses
`226`-`228`, so the option was moved to the next free id (`229`).
* `segment_iterator.cpp` keeps branch-4.2's `_non_predicate_column_ids`
naming and drops the master-only `_update_lsn_col_if_needed` /
`_update_tso_col_if_needed` calls from the ported hunk; branch-4.2 keeps
its own `common/compile_check_begin.h` include in front of the new
`ScopedColumnIteratorReadPhase` helper. `_read_lazy_pruned_columns` uses
the branch's `_schema_block_id_map` because master's
`Schema::column_index()` does not exist here, and
`FileColumnIterator::get_reader()` was added as in master.
* Regression suites `lambda_null_pruning` and
`map_contains_arg_pruning`, and the unit tests for lambda access paths /
Map lazy-read-by-rowids, only exist on master (they come from PRs
outside this chain) and were not added.

### Test

* FE: `./build.sh --fe` OK. `./run-fe-ut.sh --run
'org.apache.doris.nereids.rules.rewrite.PruneNestedColumnTest'` → 63
passed; `VariantPruningLogicTest` 14, `IcebergScanNodeTest` 95,
`MaterializeProbeVisitorTest` 8, `StringEmptyToLengthRuleTest`,
`PullUpProjectExprUnderTopNTest` — all green.
* BE: `./build.sh --be` OK (the two ported BE test sources were
additionally checked with the unit-test flags, `-DBE_TEST
-fno-access-control`).
* Regression: `./run-regression-test.sh --run -d
nereids_rules_p0/column_pruning` → 8 suites, 0 failed, including the new
`left_join_not_null_column` suite that guards the outer-join null-map
crash.

### Behavior changed

Yes. Typed `META` access paths with an explicit version replace the
legacy all-`DATA` encoding between new FEs and BEs (upgrade BEs before
FEs); struct fields literally named `NULL`/`OFFSET` are now read
correctly instead of being pruned as metadata; `IS NULL` on an
outer-join-nullable `NOT NULL` column no longer crashes the BE; invalid
Map descendant selectors now return an internal error instead of being
silently ignored. The port also brings #59263, which defers reading
non-predicate pruned complex columns until after predicate evaluation,
reducing nested-column I/O.

### Release note

None

### Rebased onto branch-4.2 (2026-09-23)

The branch was rebased onto the current `branch-4.2` (`c0cbcc3e49c`) and
is now a single commit `e044385682f`. The two prerequisite ports listed
above are folded into it: branch-4.2 has since gained its own
nested-column-pruning series (`#68214`) and the `#68331` predicate-path
fix, and `#59263` had already replaced the `#64535` `MetaPathStriper`
design inside this pick, so replaying the intermediate states is no
longer meaningful. See the comment below for the conflict-resolution
details and the verification results.