Skip to content

Commit 3c43f4c

Browse files
csun5285claude
andcommitted
[fix](storage) align pruned complex column types with sub-column iterators on merge/agg read paths
When a query prunes nested sub-columns of a complex column (EXPLAIN shows "pruned type"), the scan's struct sub-column iterators are filtered down to the pruned subset (set_access_paths + remove_pruned_sub_iterators), while the FE-pruned data type is recorded only in TabletSchema::_pruned_columns_data_type and honored only by TabletSchema::create_block*. Read paths that materialize column shapes from other sources still used the FULL storage type, so the positional pairing in StructFileColumnIterator::next_batch worked on two different ordinal spaces: 1. VMergeIteratorContext::block_reset built per-segment merge blocks from the storage Schema (full type). A topn ordered read over a rowset with multiple OVERLAPPING segments then decoded pruned children into the wrong dst columns: - "[E-3110] Method insert_many_dict_data is not supported for TINYINT" (dict-encoded text child decoded into a tinyint child) - "... is not supported for INT" (single-child pruning variant) - BE SIGSEGV when the shifted child types happen to match and the loop runs past the pruned iterator vector (same family as DORIS-26595) This is the root cause of DORIS-26889: the CI's s3 insert naturally produces overlapping multi-segment rowsets, while single-segment local loads never take the merge path. On master the default plan usually shields this path via TopN lazy materialization (apache#63736 + apache#64242), but e.g. raising topn_opt_limit_threshold, disabling topn_lazy_materialization_threshold, or light_schema_change=false tables expose it; branch-4.1 hits it by default. 2. BlockReader::_init_agg_state built the reader_replace argument type from the raw TabletColumn (full type) while the stored block uses the pruned type. On AGG tables any pruned read that goes through the aggregate merge path (>= 2 data rowsets, no special session vars needed) failed with "Aggregate function reader_replace argument 0 type check failed: ColumnStruct has 2 elements, but DataTypeStruct has 3 elements". Fix: make the shape-materializing read paths use the same FE-pruned type as the sub-column iterators, so dst blocks and iterators stay in one ordinal space by construction. - VMergeIteratorContext::init reads the FE-pruned complex types from StorageReadOptions::tablet_schema (TabletSchema::pruned_columns_data_type(), already set on the query read path), and block_reset then builds each merge block with the pruned type for pruned columns and the full storage type for the rest, so the block's struct children align with the pruned sub-column iterators. No new plumbing through the merge iterator or the storage Schema. - TabletColumn::get_aggregate_function now takes the column's actual data type explicitly. BlockReader::_init_agg_state passes the stored block's column type (already the pruned type for pruned complex columns), while the load/compaction/schema-change paths pass the full column type. The new regression case constructs an OVERLAPPING multi-segment rowset deterministically (MemTable.need_flush debug point + NULL keys in every batch, segment count kept below the segcompaction threshold), self-checks the layout via tablet meta, and covers all variants: the dict-error and SIGSEGV topn shapes on the inline path, the AGG default-session shape across two rowsets, and the VMaterializeNode default path as a guard. All variants fail before this fix and pass after it. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
1 parent 81708fa commit 3c43f4c

11 files changed

Lines changed: 457 additions & 8 deletions

File tree

‎be/src/load/memtable/memtable.cpp‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@
2929
#include "bvar/bvar.h"
3030
#include "common/config.h"
3131
#include "core/column/column.h"
32+
#include "core/data_type/data_type_factory.hpp"
3233
#include "exprs/aggregate/aggregate_function_reader.h"
3334
#include "exprs/aggregate/aggregate_function_simple_factory.h"
3435
#include "load/memtable/memtable_memory_limiter.h"
@@ -123,6 +124,7 @@ void MemTable::_init_agg_functions(const Block* block) {
123124
}
124125
} else {
125126
function = _tablet_schema->column(cid).get_aggregate_function(
127+
DataTypeFactory::instance().create_data_type(_tablet_schema->column(cid)),
126128
AGG_LOAD_SUFFIX, _tablet_schema->column(cid).get_be_exec_version());
127129
if (function == nullptr) {
128130
LOG(WARNING) << "column get aggregate function failed, column="

‎be/src/storage/iterator/block_reader.cpp‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -501,8 +501,11 @@ Status BlockReader::_init_agg_state(const ReaderParams& read_params) {
501501
for (auto idx : _agg_columns_idx) {
502502
auto column = tablet_schema.column(
503503
read_params.origin_return_columns->at(_return_columns_loc[idx]));
504+
// Use the stored block's column type (already the FE-pruned type for a pruned
505+
// complex column) so the aggregate function's argument type matches.
504506
AggregateFunctionPtr function =
505-
column.get_aggregate_function(AGG_READER_SUFFIX, read_params.get_be_exec_version());
507+
column.get_aggregate_function(_next_row.block->get_by_position(idx).type,
508+
AGG_READER_SUFFIX, read_params.get_be_exec_version());
506509

507510
// to avoid coredump when something goes wrong(i.e. column missmatch)
508511
if (!function) {

‎be/src/storage/iterator/vertical_block_reader.cpp‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@
2828
#include "core/block/column_with_type_and_name.h"
2929
#include "core/column/column_nullable.h"
3030
#include "core/column/column_vector.h"
31+
#include "core/data_type/data_type_factory.hpp"
3132
#include "core/data_type/data_type_number.h"
3233
#include "exprs/aggregate/aggregate_function_reader.h"
3334
#include "storage/compaction/compaction.h"
@@ -194,10 +195,10 @@ void VerticalBlockReader::_init_agg_state(const ReaderParams& read_params) {
194195

195196
auto& tablet_schema = *_tablet_schema;
196197
for (size_t idx = 0; idx < _return_columns.size(); ++idx) {
198+
const auto& column = tablet_schema.column(_return_columns.at(idx));
197199
AggregateFunctionPtr function =
198-
tablet_schema.column(_return_columns.at(idx))
199-
.get_aggregate_function(AGG_READER_SUFFIX,
200-
read_params.get_be_exec_version());
200+
column.get_aggregate_function(DataTypeFactory::instance().create_data_type(column),
201+
AGG_READER_SUFFIX, read_params.get_be_exec_version());
201202
DCHECK(function != nullptr);
202203
const auto* column_ptr = _stored_data_columns[idx].get();
203204
const IColumn* columns[] = {column_ptr};

‎be/src/storage/iterator/vgeneric_iterators.cpp‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -107,7 +107,7 @@ Status VMergeIteratorContext::block_reset(const std::shared_ptr<Block>& block) {
107107
const auto& column_ids = _output_schema->column_ids();
108108
for (size_t i = 0; i < _output_schema->num_column_ids(); ++i) {
109109
auto column_desc = _output_schema->column(column_ids[i]);
110-
auto data_type = Schema::get_data_type_ptr(*column_desc);
110+
auto data_type = _data_type_maybe_pruned(*column_desc);
111111
if (data_type == nullptr) {
112112
return Status::RuntimeError("invalid data type");
113113
}
@@ -294,6 +294,11 @@ Status VAutoIncrementIterator::init(const StorageReadOptions& opts) {
294294
Status VMergeIteratorContext::init(const StorageReadOptions& opts) {
295295
_block_row_max = opts.block_row_max;
296296
_record_rowids = opts.record_rowids;
297+
// For query reads that prune nested sub-columns, block_reset() must build the
298+
// merge block with the FE-pruned complex types recorded on the tablet schema.
299+
if (opts.tablet_schema != nullptr && opts.tablet_schema->has_pruned_columns()) {
300+
_pruned_columns_data_type = &opts.tablet_schema->pruned_columns_data_type();
301+
}
297302
RETURN_IF_ERROR(_load_next_block());
298303
if (valid()) {
299304
RETURN_IF_ERROR(advance());

‎be/src/storage/iterator/vgeneric_iterators.h‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -185,6 +185,20 @@ class VMergeIteratorContext {
185185
// Load next block into _block
186186
Status _load_next_block();
187187

188+
// Data type used to materialize `column` in the merge block: the FE-pruned
189+
// type when the query pruned this complex column's sub-columns (so the block's
190+
// struct children line up with the pruned sub-column iterators), otherwise the
191+
// full storage type.
192+
DataTypePtr _data_type_maybe_pruned(const TabletColumn& column) const {
193+
if (_pruned_columns_data_type) {
194+
auto it = _pruned_columns_data_type->find(column.unique_id());
195+
if (it != _pruned_columns_data_type->end()) {
196+
return it->second;
197+
}
198+
}
199+
return Schema::get_data_type_ptr(column);
200+
}
201+
188202
RowwiseIteratorUPtr _iter;
189203

190204
int _sequence_id_idx = -1;
@@ -209,6 +223,10 @@ class VMergeIteratorContext {
209223
// block_reset() uses _output_schema to build _block, and copy_rows() iterates over
210224
// _output_schema->num_column_ids() columns to copy from _block to the destination.
211225
const SchemaSPtr _output_schema;
226+
// col unique id -> FE-pruned type for pruned complex columns; null otherwise. Set in
227+
// init() from opts.tablet_schema (outlives this context); block_reset() uses it so the
228+
// merge block's struct children line up with the pruned sub-column iterators.
229+
const std::map<int32_t, DataTypePtr>* _pruned_columns_data_type = nullptr;
212230
int _num_key_columns;
213231
std::vector<uint32_t>* _compare_columns;
214232
std::vector<RowLocation> _block_row_locations;

‎be/src/storage/schema_change/schema_change.cpp‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,7 @@
4444
#include "core/block/column_with_type_and_name.h"
4545
#include "core/column/column.h"
4646
#include "core/column/column_nullable.h"
47+
#include "core/data_type/data_type_factory.hpp"
4748
#include "exec/common/util.hpp"
4849
#include "exec/common/variant_util.h"
4950
#include "exprs/aggregate/aggregate_function.h"
@@ -174,6 +175,7 @@ class MultiBlockMerger {
174175
for (int i = key_number; i < columns; i++) {
175176
try {
176177
AggregateFunctionPtr function = tablet_schema->column(i).get_aggregate_function(
178+
DataTypeFactory::instance().create_data_type(tablet_schema->column(i)),
177179
AGG_LOAD_SUFFIX, tablet_schema->column(i).get_be_exec_version());
178180
if (!function) {
179181
return Status::InternalError(

‎be/src/storage/tablet/tablet_schema.cpp‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -788,11 +788,10 @@ AggregateFunctionPtr TabletColumn::get_aggregate_function_union(DataTypePtr type
788788
return AggregateStateUnion::create(state_type->get_nested_function(), {type}, type);
789789
}
790790

791-
AggregateFunctionPtr TabletColumn::get_aggregate_function(std::string suffix,
791+
AggregateFunctionPtr TabletColumn::get_aggregate_function(DataTypePtr type, std::string suffix,
792792
int current_be_exec_version) const {
793793
AggregateFunctionPtr function = nullptr;
794794

795-
auto type = DataTypeFactory::instance().create_data_type(*this);
796795
if (type && type->get_primitive_type() == PrimitiveType::TYPE_AGG_STATE) {
797796
function = get_aggregate_function_union(type, current_be_exec_version);
798797
} else {

‎be/src/storage/tablet/tablet_schema.h‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -153,7 +153,9 @@ class TabletColumn : public MetadataAdder<TabletColumn> {
153153
FieldAggregationMethod aggregation() const { return _aggregation; }
154154
AggregateFunctionPtr get_aggregate_function_union(DataTypePtr type,
155155
int current_be_exec_version) const;
156-
AggregateFunctionPtr get_aggregate_function(std::string suffix,
156+
// `type` is the data type of the column as actually read (the FE-pruned type
157+
// when the query prunes nested sub-columns, otherwise the full column type).
158+
AggregateFunctionPtr get_aggregate_function(DataTypePtr type, std::string suffix,
157159
int current_be_exec_version) const;
158160
int precision() const { return _precision; }
159161
int frac() const { return _frac; }
@@ -753,6 +755,10 @@ class TabletSchema : public MetadataAdder<TabletSchema> {
753755

754756
bool has_pruned_columns() const { return !_pruned_columns_data_type.empty(); }
755757

758+
const std::map<int32_t, DataTypePtr>& pruned_columns_data_type() const {
759+
return _pruned_columns_data_type;
760+
}
761+
756762
TabletStorageFormatPB storage_format() const { return _storage_format; }
757763
void set_storage_format(TabletStorageFormatPB v) { _storage_format = v; }
758764

‎be/test/exec/scan/vgeneric_iterators_test.cpp‎

Lines changed: 150 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -359,6 +359,156 @@ TEST(VGenericIteratorsTest, MergeWithSeqColumn) {
359359
EXPECT_EQ(seg_iter_num - 1, actual_value);
360360
}
361361

362+
// Schema with a smallint key c1 and a struct column c2 = struct<f1:int, f2:int, f3:int>.
363+
static Schema create_struct_schema(int32_t struct_unique_id) {
364+
std::vector<TabletColumnPtr> col_schemas;
365+
auto c1 = std::make_shared<TabletColumn>(FieldAggregationMethod::OLAP_FIELD_AGGREGATION_NONE,
366+
FieldType::OLAP_FIELD_TYPE_SMALLINT, true);
367+
c1->set_is_key(true);
368+
col_schemas.emplace_back(c1);
369+
370+
// The (agg, type, nullable) constructor does not support complex types, so
371+
// build the struct column with the default constructor plus setters.
372+
auto c2 = std::make_shared<TabletColumn>();
373+
c2->set_type(FieldType::OLAP_FIELD_TYPE_STRUCT);
374+
c2->set_aggregation_method(FieldAggregationMethod::OLAP_FIELD_AGGREGATION_NONE);
375+
c2->set_is_nullable(true);
376+
c2->set_name("c2");
377+
c2->set_unique_id(struct_unique_id);
378+
for (const auto* sub_name : {"f1", "f2", "f3"}) {
379+
TabletColumn sub(FieldAggregationMethod::OLAP_FIELD_AGGREGATION_NONE,
380+
FieldType::OLAP_FIELD_TYPE_INT, true);
381+
sub.set_name(sub_name);
382+
c2->add_sub_column(sub);
383+
}
384+
col_schemas.emplace_back(c2);
385+
386+
std::vector<ColumnId> column_ids(col_schemas.size());
387+
for (uint32_t cid = 0; cid < column_ids.size(); ++cid) {
388+
column_ids[cid] = cid;
389+
}
390+
391+
Schema schema(col_schemas, column_ids);
392+
return schema;
393+
}
394+
395+
// Fills whatever block it is given: ascending keys (key_start + i * key_stride) so
396+
// several inputs interleave under merge, and default values for the struct column.
397+
// Also checks that the merge context built its block with the expected struct type.
398+
class PrunedStructUtIterator : public RowwiseIterator {
399+
public:
400+
PrunedStructUtIterator(const Schema& schema, size_t num_rows, int16_t key_start,
401+
int16_t key_stride, DataTypePtr expected_struct_type)
402+
: _schema(schema),
403+
_num_rows(num_rows),
404+
_key(key_start),
405+
_key_stride(key_stride),
406+
_expected_struct_type(std::move(expected_struct_type)) {}
407+
~PrunedStructUtIterator() override = default;
408+
409+
Status init(const StorageReadOptions& opts) override { return Status::OK(); }
410+
411+
Status next_batch(Block* block) override {
412+
if (_rows_returned >= _num_rows) {
413+
return Status::EndOfFile("End of PrunedStructUtIterator");
414+
}
415+
// The merge block must carry the FE-pruned struct type (not the full storage
416+
// type) so that its struct children line up with the pruned sub-column readers.
417+
EXPECT_TRUE(block->get_by_position(1).type->equals(*_expected_struct_type));
418+
while (_rows_returned < _num_rows) {
419+
ColumnWithTypeAndName& key_col = block->get_by_position(0);
420+
((IColumn&)(*key_col.column)).insert_data((const char*)&_key, sizeof(_key));
421+
ColumnWithTypeAndName& struct_col = block->get_by_position(1);
422+
((IColumn&)(*struct_col.column)).insert_default();
423+
_key = (int16_t)(_key + _key_stride);
424+
++_rows_returned;
425+
}
426+
return Status::OK();
427+
}
428+
429+
const Schema& schema() const override { return _schema; }
430+
431+
private:
432+
const Schema& _schema;
433+
size_t _num_rows;
434+
size_t _rows_returned = 0;
435+
int16_t _key;
436+
int16_t _key_stride;
437+
DataTypePtr _expected_struct_type;
438+
};
439+
440+
// When the query prunes struct sub-columns, StorageReadOptions carries a tablet schema
441+
// whose pruned-columns map records the FE-pruned type for the column's unique id. The
442+
// merge iterator must build its internal blocks with that pruned type (see
443+
// VMergeIteratorContext::_data_type_maybe_pruned); otherwise the block's full struct
444+
// children would be paired with the pruned sub-column iterators by bare index.
445+
TEST(VGenericIteratorsTest, MergeWithPrunedStructColumn) {
446+
constexpr int32_t struct_unique_id = 10;
447+
auto schema = create_struct_schema(struct_unique_id);
448+
auto output_schema = std::make_shared<Schema>(schema);
449+
450+
// The FE-pruned type of c2 keeps only {f1, f3} out of {f1, f2, f3}.
451+
TabletColumn pruned_c2;
452+
pruned_c2.set_type(FieldType::OLAP_FIELD_TYPE_STRUCT);
453+
pruned_c2.set_aggregation_method(FieldAggregationMethod::OLAP_FIELD_AGGREGATION_NONE);
454+
pruned_c2.set_is_nullable(true);
455+
pruned_c2.set_name("c2");
456+
pruned_c2.set_unique_id(struct_unique_id);
457+
for (const auto* sub_name : {"f1", "f3"}) {
458+
TabletColumn sub(FieldAggregationMethod::OLAP_FIELD_AGGREGATION_NONE,
459+
FieldType::OLAP_FIELD_TYPE_INT, true);
460+
sub.set_name(sub_name);
461+
pruned_c2.add_sub_column(sub);
462+
}
463+
auto pruned_type = Schema::get_data_type_ptr(pruned_c2);
464+
auto full_type = Schema::get_data_type_ptr(*schema.column(1));
465+
EXPECT_FALSE(pruned_type->equals(*full_type));
466+
467+
auto tablet_schema = std::make_shared<TabletSchema>();
468+
tablet_schema->add_pruned_columns_data_type(struct_unique_id, pruned_type);
469+
EXPECT_TRUE(tablet_schema->has_pruned_columns());
470+
EXPECT_EQ(tablet_schema->pruned_columns_data_type().size(), 1);
471+
EXPECT_TRUE(
472+
tablet_schema->pruned_columns_data_type().at(struct_unique_id)->equals(*pruned_type));
473+
474+
std::vector<RowwiseIteratorUPtr> inputs;
475+
inputs.push_back(std::make_unique<PrunedStructUtIterator>(schema, 100, 0, 2, pruned_type));
476+
inputs.push_back(std::make_unique<PrunedStructUtIterator>(schema, 100, 1, 2, pruned_type));
477+
478+
auto iter = new_merge_iterator(std::move(inputs), -1, false, false, nullptr, output_schema);
479+
StorageReadOptions opts;
480+
opts.tablet_schema = tablet_schema;
481+
auto st = iter->init(opts);
482+
EXPECT_TRUE(st.ok());
483+
484+
// The destination block mirrors a query-level block: the struct column is created
485+
// with the pruned type, matching what the merge context copies into it.
486+
Block block;
487+
auto key_type = Schema::get_data_type_ptr(*schema.column(0));
488+
auto key_column = key_type->create_column();
489+
block.insert(ColumnWithTypeAndName(std::move(key_column), key_type, "c1"));
490+
auto struct_column = pruned_type->create_column();
491+
block.insert(ColumnWithTypeAndName(std::move(struct_column), pruned_type, "c2"));
492+
std::vector<bool> row_is_same;
493+
BlockWithSameBit block_with_same_bit {.block = &block, .same_bit = row_is_same};
494+
495+
do {
496+
st = iter->next_batch(&block_with_same_bit);
497+
} while (st.ok());
498+
499+
EXPECT_TRUE(st.is<END_OF_FILE>());
500+
EXPECT_EQ(block.rows(), 200);
501+
502+
// Keys 0..199 in order: the two inputs (0,2,4,... and 1,3,5,...) interleaved.
503+
auto c0 = block.get_by_position(0).column;
504+
for (size_t i = 0; i < block.rows(); ++i) {
505+
EXPECT_EQ(i, (*c0)[i].get<TYPE_SMALLINT>());
506+
}
507+
// The struct column kept the pruned 2-child layout end to end.
508+
EXPECT_TRUE(block.get_by_position(1).type->equals(*pruned_type));
509+
EXPECT_EQ(block.get_by_position(1).column->size(), 200);
510+
}
511+
362512
// Mirror of MergeWithSeqColumn but with small_seq_first=true.
363513
// Same key across all segments, seq values are 0..seg_iter_num-1; the merge
364514
// iterator should keep exactly one row whose seq value is the smallest (0).
Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
-- This file is automatically generated. You should know what you did if you want to edit this
2+
-- !dict_default_session --
3+
1 s1
4+
2 s2
5+
3 s3
6+
4 s4
7+
5 s5
8+
6 s6
9+
7 s7
10+
8 s8
11+
9 s9
12+
10 s10
13+
14+
-- !agg_default_session --
15+
1 s1
16+
2 s2
17+
3 s3
18+
4 s4
19+
5 s5
20+
6 s6
21+
7 s7
22+
8 s8
23+
9 s9
24+
10 s10
25+
26+
-- !agg_second_rowset --
27+
s1501
28+
s1502
29+
s1503
30+
s1504
31+
s1505
32+
s1506
33+
s1507
34+
s1508
35+
s1509
36+
s1510
37+
38+
-- !dict_inline_topn --
39+
1 s1
40+
2 s2
41+
3 s3
42+
4 s4
43+
5 s5
44+
6 s6
45+
7 s7
46+
8 s8
47+
9 s9
48+
10 s10
49+
50+
-- !dict_inline_topn_single_child --
51+
s1
52+
s2
53+
s3
54+
s4
55+
s5
56+
s6
57+
s7
58+
s8
59+
s9
60+
s10
61+
62+
-- !int_inline_topn_last_child --
63+
3000001
64+
3000002
65+
3000003
66+
3000004
67+
3000005
68+
3000006
69+
3000007
70+
3000008
71+
3000009
72+
3000010
73+

0 commit comments

Comments
 (0)