Skip to content

Commit ec886f3

Browse files
engleflymrhhsg
andauthored
[fix](nereids) Disambiguate NULL/OFFSET metadata from same-named nested fields (#65805)
### What problem does this PR solve? Issue Number: None Related PR: #65591, #66380 Problem Summary: Nested-column pruning (feature series #59263 / #61888 / #64535) makes Doris read only the referenced parts of a struct/map/array column. Two kinds of special *meta* access paths serve that purpose: a terminal `NULL` component means the query only needs the null flag (`IS [NOT] NULL` on a column or nested field), and a terminal `OFFSET` component means only offsets are needed (`length`/`cardinality`-style functions). The BE satisfies them in `NULL_MAP_ONLY` / `OFFSET_ONLY` meta-read modes that skip the payload data entirely. While the optimization itself is sound, the way meta paths were expressed on the wire and handled on both sides had several correctness holes: * **Wire ambiguity.** Every access path was sent as a `DATA`-typed list of string components, and `NULL`/`OFFSET` lived in the same list as real child names. There was no way to tell a metadata selector apart from a struct field literally named `OFFSET` or `NULL`, so such a field could be swallowed as metadata and its data silently pruned away (wrong query results), and there was no FE/BE version contract to evolve the encoding during rolling upgrades. * **FE null-map correctness.** A slot can be nullable for two different reasons: its physical column has a null bitmap, or an outer join merely made the output slot nullable. The old logic keyed on expression/slot nullability, so a physical `NOT NULL` dimension column of a `LEFT JOIN` received a `[col, NULL]` path and the BE aborted trying to read a null map from a `NOT NULL` column. Meta paths were also generated for non-nullable struct fields / non-null structs and for variant sub-columns, which do not support meta-only reads. * **FE pruning bookkeeping.** `NULL`/`OFFSET` were detected by string suffix, so real fields with those names collided inside the pruning logic too; and predicate meta paths were stripped or broadened (map-star) whenever regular data paths existed, losing metadata-only reads or over-reading payload data. * **BE routing/read-mode logic.** A terminal `NULL`/`OFFSET` was treated as current-level metadata at every container, meta-only modes could be entered even when a sibling `DATA` path still required payload reads, sibling `META` paths were dropped when unrelated data projections existed, and Map descendant routing depended on physical key/value child column names while silently ignoring unknown selectors. What this PR does: * **Versioned, type-selected access paths.** Adds an optional `version` to `TColumnAccessPath` (thrift + protobuf + FE/BE descriptor conversion; `TCOLUMN_ACCESS_PATH_VERSION_LEGACY = 0`, `TCOLUMN_ACCESS_PATH_VERSION_TYPED = 1`). In the typed format the path type is authoritative: `DATA` selects `data_access_path`, `META` selects `meta_access_path`, and `NULL`/`OFFSET` are only ever emitted as typed `META` paths. New BEs still decode the legacy all-`DATA` encoding from old FEs, so the supported rolling upgrade is: upgrade all BEs first, then let FEs send typed paths. * **FE** (`AccessPathExpressionCollector`, `NestedColumnPruning`, `AccessPathPlanCollector`, `DescriptorToThriftConverter`): marks `NULL`/`OFFSET` collector contexts as `META`; checks the physical column's nullability (`getOriginalColumn().isAllowNull`) instead of slot nullability before emitting null-map paths, so outer-join-nullable `NOT NULL` columns and non-nullable fields/structs no longer get `NULL` paths; emits the parent-struct `NULL` path for `element_at(...) IS NULL` only when actually needed; falls back to plain data reads for variant sub-columns; keeps the exact meta path when a sibling data path is also read; preserves fields referenced by unchanged predicates; and keys pruning on the path type instead of string suffixes, rejecting unsupported `META` paths. * **BE** (`ColumnIterator` / `column_reader.cpp`): validates path versions and type-selected payloads; partitions current data / current metadata / descendant routing with explicit per-container ownership (Struct owns the null map; Map/Array and string-like scalar columns own both null map and offsets); derives `NULL_MAP_ONLY` / `OFFSET_ONLY` only when no current or predicate data path requires payload, and marks data children `SKIP` consistently; keeps sibling `META` paths explicit so an unrelated `DATA` projection cannot disable a metadata-only predicate read; routes Map children by the logical `KEYS`/`VALUES` selectors (wildcard expansion, arbitrary nesting) independent of physical child column names; rejects unrecognized Map selectors instead of silently pruning everything; and documents the full routing flow. * **Tests & docs.** BE `ColumnReaderTest` coverage for typed-vs-legacy encoding, metadata-vs-same-named-data disambiguation, meta-read-mode legality, sibling-data-path handling and Map selector validation; `SlotDescriptor` protobuf round-trip of versions and payloads; FE `PruneNestedColumnTest` / `DescriptorToThriftConverterTest` cases; the `nereids_rules_p0/column_pruning` regression suites updated to assert the typed `META` paths, plus the new `left_join_not_null_column` suite guarding the outer-join null-map crash. ### Release note None ### Check List (For Author) - Test: - FE unit test: `./run-fe-ut.sh --run 'org.apache.doris.nereids.rules.rewrite.PruneNestedColumnTest,org.apache.doris.analysis.DescriptorToThriftConverterTest'` (74 passed on the final rebased FE tree) - BE unit test: `./run-be-ut.sh --run --filter='ColumnReaderTest.*:SlotDescriptorTest.AccessPathsPreservedThroughProtobuf'` (50 passed before the final rebase; the final rebuild was blocked before GTest by the missing master-added `thirdparty/installed/lib64/liblance_c.a` in this environment) - Regression test: `nereids_rules_p0/column_pruning` — `string_length_column_pruning`, `null_column_pruning`, `nested_container_offset_pruning`, `lambda_null_pruning`, `left_join_not_null_column` (passed before the final rebase; the final rerun was blocked by the same missing dependency) - Format checks: `build-support/check-format.sh`, `git diff --check` - Behavior changed: Yes. Typed `META` access paths with an explicit version replace the legacy all-`DATA` encoding between new FEs and BEs (upgrade BEs before FEs); struct fields literally named `NULL`/`OFFSET` are now read correctly instead of being pruned as metadata; `IS NULL` on outer-join-nullable `NOT NULL` columns no longer crashes the BE; invalid Map descendant selectors now return an internal error instead of being silently ignored. - Does this need documentation: No --------- Co-authored-by: Hu Shenggang <[email protected]>
1 parent 62327d3 commit ec886f3

17 files changed

Lines changed: 2056 additions & 687 deletions

File tree

‎be/src/runtime/descriptors.cpp‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -104,6 +104,9 @@ SlotDescriptor::SlotDescriptor(const PSlotDescriptor& pdesc)
104104
auto convert_to_thrift_column_access_path = [](const PColumnAccessPath& pb_path) {
105105
TColumnAccessPath thrift_path;
106106
thrift_path.type = (TAccessPathType::type)pb_path.type();
107+
if (pb_path.has_version()) {
108+
thrift_path.__set_version(pb_path.version());
109+
}
107110
if (pb_path.has_data_access_path()) {
108111
thrift_path.__isset.data_access_path = true;
109112
for (int i = 0; i < pb_path.data_access_path().path_size(); ++i) {
@@ -160,6 +163,9 @@ void SlotDescriptor::to_protobuf(PSlotDescriptor* pslot) const {
160163
doris::PColumnAccessPath* pb_path) {
161164
pb_path->Clear();
162165
pb_path->set_type((PAccessPathType)thrift_path.type); // 使用 reinterpret_cast 进行类型转换
166+
if (thrift_path.__isset.version) {
167+
pb_path->set_version(thrift_path.version);
168+
}
163169
if (thrift_path.__isset.data_access_path) {
164170
auto* pb_data = pb_path->mutable_data_access_path();
165171
pb_data->Clear();

‎be/src/storage/segment/column_reader.cpp‎

Lines changed: 443 additions & 344 deletions
Large diffs are not rendered by default.

‎be/src/storage/segment/column_reader.h‎

Lines changed: 46 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@
2424

2525
#include <cstddef> // for size_t
2626
#include <cstdint> // for uint32_t
27+
#include <functional>
2728
#include <map>
2829
#include <memory> // for unique_ptr
2930
#include <optional>
@@ -500,18 +501,48 @@ class ColumnIterator {
500501
}
501502

502503
protected:
504+
struct AccessPathSplit {
505+
TColumnAccessPaths descendant_paths;
506+
bool reads_current_data = false;
507+
MetaReadMode current_meta_mode = MetaReadMode::DEFAULT;
508+
509+
bool has_descendant_paths() const { return !descendant_paths.empty(); }
510+
};
511+
512+
// Nested columns share the same current-level access-path planning, while their data-child
513+
// topology and descendant routing remain container-specific.
514+
struct NestedAccessPathPlan {
515+
AccessPathSplit all;
516+
AccessPathSplit predicate;
517+
bool skip_data_descendants = false;
518+
};
519+
520+
// At their current level, Struct supports null-map metadata. Map and Array additionally
521+
// support offsets.
522+
enum class NestedMetaSupport { NULL_MAP, NULL_MAP_AND_OFFSET };
523+
503524
void _convert_to_place_holder_column(MutableColumnPtr& dst, size_t count);
504525

505526
void _recovery_from_place_holder_column(MutableColumnPtr& dst);
506527

507-
// Derive current-level meta-only read mode from access paths. Meta-only is valid only when
508-
// this iterator had no data-read requirement before applying the current paths, and every
509-
// visible path at this level is NULL/OFFSET metadata.
510-
void _check_and_set_meta_read_mode(ReadRequirement requirement_before_access_path,
511-
const TColumnAccessPaths& sub_all_access_paths);
512-
513-
Result<TColumnAccessPaths> _get_sub_access_paths(TColumnAccessPaths access_paths,
514-
bool is_predicate = false);
528+
// Derive current-level meta-only read mode from an explicit access-path split. Meta-only is
529+
// valid only when this iterator had no data-read requirement before applying the current paths,
530+
// no current DATA path exists, and no path must be routed to a descendant iterator.
531+
Status _check_and_set_meta_read_mode(ReadRequirement requirement_before_access_path,
532+
const AccessPathSplit& all_access_paths);
533+
534+
// Apply the common current-level access-path state transitions and select a supported
535+
// parent-owned meta-only mode. When that mode skips data descendants, synchronously invoke the
536+
// callback once with SKIP before returning the routing plan. The callback is never retained.
537+
Result<NestedAccessPathPlan> _prepare_nested_access_paths(
538+
const TColumnAccessPaths& all_access_paths,
539+
const TColumnAccessPaths& predicate_access_paths, NestedMetaSupport meta_support,
540+
const std::function<void(ReadRequirement)>& set_all_data_descendants_read_requirement);
541+
542+
// Normalize the wire encoding, strip this iterator's column name, and explicitly partition
543+
// paths consumed by this iterator from paths that must be routed to descendants. This helper is
544+
// intentionally side-effect free; callers apply DATA/predicate read requirements explicitly.
545+
Result<AccessPathSplit> _split_access_paths(TColumnAccessPaths access_paths) const;
515546
ColumnIteratorOptions _opts;
516547

517548
ReadRequirement _read_requirement {ReadRequirement::NORMAL};
@@ -542,6 +573,9 @@ class FileColumnIterator : public ColumnIterator {
542573
Status read_by_rowids(const rowid_t* rowids, const size_t count,
543574
MutableColumnPtr& dst) override;
544575

576+
Status set_access_paths(const TColumnAccessPaths& all_access_paths,
577+
const TColumnAccessPaths& predicate_access_paths) override;
578+
545579
ordinal_t get_current_ordinal() const override { return _current_ordinal; }
546580

547581
// get row ranges by zone map
@@ -618,10 +652,10 @@ class EmptyFileColumnIterator final : public ColumnIterator {
618652
ordinal_t get_current_ordinal() const override { return 0; }
619653
};
620654

621-
// StringFileColumnIterator extends FileColumnIterator with meta-only reading
622-
// support for string/binary column types. When the OFFSET path is detected in
623-
// set_access_paths, it sets only_read_offsets on the ColumnIteratorOptions so
624-
// that the BinaryPlainPageDecoder skips chars memcpy and only fills offsets.
655+
// StringFileColumnIterator extends FileColumnIterator's NULL metadata support with OFFSET-only
656+
// reading for string/binary column types. When the OFFSET path is detected in set_access_paths, it
657+
// sets only_read_offsets on the ColumnIteratorOptions so that the BinaryPlainPageDecoder skips
658+
// chars memcpy and only fills offsets.
625659
class StringFileColumnIterator final : public FileColumnIterator {
626660
public:
627661
explicit StringFileColumnIterator(std::shared_ptr<ColumnReader> reader);

‎be/test/runtime/descriptor_test.cpp‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,8 +15,10 @@
1515
// specific language governing permissions and limitations
1616
// under the License.
1717

18+
#include <gen_cpp/Descriptors_constants.h>
1819
#include <gen_cpp/Exprs_types.h>
1920
#include <gen_cpp/Types_types.h>
21+
#include <gen_cpp/descriptors.pb.h>
2022
#include <gtest/gtest.h>
2123

2224
#include <memory>
@@ -214,4 +216,59 @@ TEST(TupleDescriptorTest, GetColumnId) {
214216
EXPECT_EQ(tuple_desc->get_column_id(100), -1);
215217
}
216218

219+
TEST_F(SlotDescriptorTest, AccessPathsPreservedThroughProtobuf) {
220+
TColumnAccessPath data_path;
221+
data_path.__set_version(g_Descriptors_constants.TCOLUMN_ACCESS_PATH_VERSION_TYPED);
222+
data_path.__set_type(TAccessPathType::DATA);
223+
TDataAccessPath data_payload;
224+
data_payload.__set_path({"s", "field"});
225+
data_path.__set_data_access_path(data_payload);
226+
227+
TColumnAccessPath meta_path;
228+
meta_path.__set_version(g_Descriptors_constants.TCOLUMN_ACCESS_PATH_VERSION_TYPED);
229+
meta_path.__set_type(TAccessPathType::META);
230+
TMetaAccessPath meta_payload;
231+
meta_payload.__set_path({"s", "field", "NULL"});
232+
meta_path.__set_meta_access_path(meta_payload);
233+
234+
TColumnAccessPath legacy_path;
235+
legacy_path.__set_type(TAccessPathType::DATA);
236+
data_payload.__set_path({"s", "legacy", "NULL"});
237+
legacy_path.__set_data_access_path(data_payload);
238+
239+
TColumnAccessPath explicit_legacy_path;
240+
explicit_legacy_path.__set_version(g_Descriptors_constants.TCOLUMN_ACCESS_PATH_VERSION_LEGACY);
241+
explicit_legacy_path.__set_type(TAccessPathType::DATA);
242+
data_payload.__set_path({"s", "legacy", "VALUES"});
243+
explicit_legacy_path.__set_data_access_path(data_payload);
244+
245+
TSlotDescriptor thrift_descriptor = create_basic_slot_descriptor();
246+
thrift_descriptor.__set_all_access_paths(
247+
{data_path, meta_path, legacy_path, explicit_legacy_path});
248+
thrift_descriptor.__set_predicate_access_paths({meta_path});
249+
250+
SlotDescriptor original(thrift_descriptor);
251+
PSlotDescriptor protobuf_descriptor;
252+
original.to_protobuf(&protobuf_descriptor);
253+
ASSERT_EQ(protobuf_descriptor.all_access_paths_size(), 4);
254+
EXPECT_TRUE(protobuf_descriptor.all_access_paths(0).has_version());
255+
EXPECT_EQ(protobuf_descriptor.all_access_paths(0).version(),
256+
g_Descriptors_constants.TCOLUMN_ACCESS_PATH_VERSION_TYPED);
257+
EXPECT_TRUE(protobuf_descriptor.all_access_paths(1).has_version());
258+
EXPECT_FALSE(protobuf_descriptor.all_access_paths(2).has_version());
259+
EXPECT_TRUE(protobuf_descriptor.all_access_paths(3).has_version());
260+
EXPECT_EQ(protobuf_descriptor.all_access_paths(3).version(),
261+
g_Descriptors_constants.TCOLUMN_ACCESS_PATH_VERSION_LEGACY);
262+
SlotDescriptor round_trip(protobuf_descriptor);
263+
264+
EXPECT_EQ(round_trip.all_access_paths(), thrift_descriptor.all_access_paths);
265+
EXPECT_EQ(round_trip.predicate_access_paths(), thrift_descriptor.predicate_access_paths);
266+
EXPECT_FALSE(round_trip.all_access_paths()[1].__isset.data_access_path);
267+
EXPECT_FALSE(round_trip.predicate_access_paths()[0].__isset.data_access_path);
268+
EXPECT_FALSE(round_trip.all_access_paths()[2].__isset.version);
269+
EXPECT_TRUE(round_trip.all_access_paths()[3].__isset.version);
270+
EXPECT_EQ(round_trip.all_access_paths()[3].version,
271+
g_Descriptors_constants.TCOLUMN_ACCESS_PATH_VERSION_LEGACY);
272+
}
273+
217274
} // namespace doris

0 commit comments

Comments
 (0)