Skip to content

Commit b86393c

Browse files
csun5285claude
andcommitted
[refactor](be) Refuse INVALID_TYPE AGG_STATE as master does and tidy the write path
### What problem does this PR solve? Related PR: #69070 Problem Summary: Review follow-ups on the IColumn segment write refactor. - An AGG_STATE column whose function serialises to INVALID_TYPE is refused when its writer is created, with master's "OLAP_FIELD_TYPE_AGG_STATE meet unsupported type" error. Master accepted it in `create_agg_state_writer`, but the convertor it created right after the writer threw that error; the refactor had stored such states through AGG_STATE's fixed-length layout instead. - `ArrayColumnWriter::estimate_buffer_size` is master's again. It no longer adds the capacity of the array index's scratch buffers, which is never written to the segment, and `StorageBuffers::allocated_bytes` goes away. The buffers keep their capacity for reuse; their memory is charged to the thread memory tracker when it is allocated. - An array batch without elements hands the index writers the data() of the empty bytes buffer instead of resizing it first. Nothing is read through it, and a PaddedPODArray's data() is never null, which is all the SNII BKD writer checks; the test now asserts that pointer instead of the buffer size. - `ScalarColumnWriter::prepare_cells`, which had one caller, is inlined into `append_values`. - Comments that still named the removed OlapBlockDataConvertor now name the column writer and `rebase_offsets`. ### Release note None ### Check List (For Author) - Test: Unit Test - ASAN BE build; BE UT: 825 tests from 84 suites (writer, storage layout, CLucene, SNII and ANN index, BUILD INDEX, variant and segment suites, plus every suite of the test files changed in this PR): 818 passed, 0 failed, 7 skipped (two SNII corpus tests need an external corpus, four need the NestedGroup module, one runs only in Release) - clang-format 16, build hygiene, clang-tidy on the changed lines - Behavior changed: No; an INVALID_TYPE AGG_STATE column fails as on master - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent dc51f2d commit b86393c

8 files changed

Lines changed: 29 additions & 32 deletions

File tree

‎be/src/storage/index/inverted/clucene/inverted_index_writer.cpp‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -424,7 +424,7 @@ Status InvertedIndexColumnWriter<field_type>::add_array_values(size_t field_size
424424
size_t start_off = 0;
425425
std::vector<ReaderPtr> keep_readers;
426426
for (size_t i = 0; i < count; ++i) {
427-
// nullmap & value ptr-array may not from offsets[i] because olap_convertor make offsets accumulate from _base_offset which may not is 0, but nullmap & value in this segment is from 0, we only need
427+
// nullmap & value ptr-array may not from offsets[i] because rebase_offsets makes offsets start from the segment's item ordinal which may not be 0, but nullmap & value in this segment is from 0, we only need
428428
// every single array row element size to go through the nullmap & value ptr-array, and also can go through the every row in array to keep with _rid++
429429
auto array_elem_size = offsets[i + 1] - offsets[i];
430430
// TODO(Amory).later we use object pool to avoid field creation

‎be/src/storage/segment/array_index_input.cpp‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -55,9 +55,9 @@ Status feed_array_index(IndexColumnWriter* writer, FieldType item_type, const IC
5555
// read them.
5656
const uint8_t* values = nullptr;
5757
if (num_items == 0) {
58-
// Empty arrays still advance document IDs. The raw BKD interface requires
59-
// a non-null value pointer even when none of these rows has an element.
60-
buffers.bytes.resize(field_type_size(item_type));
58+
// Empty arrays still advance document IDs, so the rows are fed even without elements.
59+
// Nothing is read through the value pointer, but the SNII BKD writer checks that it is
60+
// not null; a PaddedPODArray's data() never is, even when empty.
6161
values = buffers.bytes.data();
6262
} else if (item_type == FieldType::OLAP_FIELD_TYPE_CHAR) {
6363
DCHECK_GT(char_length, 0);

‎be/src/storage/segment/column_writer.cpp‎

Lines changed: 15 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -316,8 +316,10 @@ Status ColumnWriter::create_agg_state_writer(const ColumnWriterOptions& opts,
316316
auto data_type = DataTypeFactory::instance().create_data_type(*column);
317317
const auto* agg_state_type = assert_cast<const DataTypeAggState*>(data_type.get());
318318
auto type = agg_state_type->get_serialized_type()->get_primitive_type();
319-
if (type == PrimitiveType::TYPE_STRING || type == PrimitiveType::INVALID_TYPE ||
320-
type == PrimitiveType::TYPE_FIXED_LENGTH_OBJECT || type == PrimitiveType::TYPE_BITMAP) {
319+
// INVALID_TYPE has no storage conversion and is refused below with the other unsupported
320+
// serialised types.
321+
if (type == PrimitiveType::TYPE_STRING || type == PrimitiveType::TYPE_FIXED_LENGTH_OBJECT ||
322+
type == PrimitiveType::TYPE_BITMAP) {
321323
// The states are stored as the function serialises them: strings as VARCHAR, bitmaps
322324
// as BITMAP, fixed-length objects through AGG_STATE's own layout.
323325
FieldType storage_type = FieldType::OLAP_FIELD_TYPE_AGG_STATE;
@@ -553,16 +555,6 @@ Status ScalarColumnWriter::append_nulls(size_t num_rows) {
553555
return Status::OK();
554556
}
555557

556-
const uint8_t* ScalarColumnWriter::prepare_cells(const IColumn& values, size_t row_pos,
557-
size_t num_rows) {
558-
if (_storage_type == FieldType::OLAP_FIELD_TYPE_CHAR) {
559-
// Padded to the column's length, which is what the segment stores.
560-
return StorageLayout<FieldType::OLAP_FIELD_TYPE_CHAR>::column_to_storage(
561-
values, row_pos, num_rows, get_column()->length(), /*null_map=*/nullptr, _buffers);
562-
}
563-
return column_to_storage(_storage_type, values, row_pos, num_rows, _buffers);
564-
}
565-
566558
void ScalarColumnWriter::_reserve_null_bitmap(const uint8_t* null_map, size_t num_rows) {
567559
if (config::enable_rle_batch_put_optimization && _null_bitmap_builder != nullptr) {
568560
// Pre-allocate buffer based on estimated size
@@ -614,7 +606,16 @@ Status ScalarColumnWriter::append(const IColumn& column, size_t row_pos, size_t
614606

615607
Status ScalarColumnWriter::append_values(const IColumn& values, size_t row_pos, size_t num_rows) {
616608
const uint8_t* cells = nullptr;
617-
RETURN_IF_CATCH_EXCEPTION({ cells = prepare_cells(values, row_pos, num_rows); });
609+
RETURN_IF_CATCH_EXCEPTION({
610+
if (_storage_type == FieldType::OLAP_FIELD_TYPE_CHAR) {
611+
// Padded to the column's length, which is what the segment stores.
612+
cells = StorageLayout<FieldType::OLAP_FIELD_TYPE_CHAR>::column_to_storage(
613+
values, row_pos, num_rows, get_column()->length(), /*null_map=*/nullptr,
614+
_buffers);
615+
} else {
616+
cells = column_to_storage(_storage_type, values, row_pos, num_rows, _buffers);
617+
}
618+
});
618619
return append_data(cells, num_rows);
619620
}
620621

@@ -1049,7 +1050,7 @@ Status ArrayColumnWriter::append(const IColumn& column, size_t row_pos, size_t n
10491050
uint64_t ArrayColumnWriter::estimate_buffer_size() {
10501051
return _offset_writer->estimate_buffer_size() +
10511052
(is_nullable() ? _null_writer->estimate_buffer_size() : 0) +
1052-
_item_writer->estimate_buffer_size() + _index_buffers.allocated_bytes();
1053+
_item_writer->estimate_buffer_size();
10531054
}
10541055

10551056
Status ArrayColumnWriter::finish() {

‎be/src/storage/segment/column_writer.h‎

Lines changed: 6 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -217,7 +217,7 @@ class FlushPageCallback {
217217

218218
// Writes scalar values and the offset/null streams of the composite writers. append cuts a
219219
// batch into its NULL runs, written to the null bitmap (append_nulls), and its value runs,
220-
// each converted to its storage cells (prepare_cells) and written to the data pages and
220+
// each converted to its storage cells (append_values) and written to the data pages and
221221
// every index (append_data).
222222
class ScalarColumnWriter : public ColumnWriter {
223223
public:
@@ -282,20 +282,17 @@ class ScalarColumnWriter : public ColumnWriter {
282282
private:
283283
Status _internal_append_data_in_current_page(const uint8_t* ptr, size_t* num_written);
284284

285-
// Converts a run of rows without NULLs to its storage cells and writes them.
285+
// Converts rows [row_pos, row_pos + num_rows) of `values`, a run without NULL rows, to
286+
// num_rows cells of cell_size() bytes back to back through the StorageLayout of
287+
// _storage_type, borrowed from the column or held in _buffers until the next run, and
288+
// writes them.
286289
Status append_values(const IColumn& values, size_t row_pos, size_t num_rows);
287290

288-
// The storage cells of rows [row_pos, row_pos + num_rows) of `values`, a run without NULL
289-
// rows: num_rows cells of cell_size() bytes back to back, through the StorageLayout of
290-
// _storage_type, borrowed from the column or held in _buffers until the next run is
291-
// prepared.
292-
const uint8_t* prepare_cells(const IColumn& values, size_t row_pos, size_t num_rows);
293-
294291
void _reserve_null_bitmap(const uint8_t* null_map, size_t num_rows);
295292

296293
// The StorageLayout the values are stored through.
297294
const FieldType _storage_type;
298-
// Where prepare_cells puts what it cannot borrow from the column.
295+
// Where append_values puts what it cannot borrow from the column.
299296
StorageBuffers _buffers;
300297

301298
std::unique_ptr<PageBuilder> _page_builder;

‎be/src/storage/segment/encoding_info.cpp‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -482,7 +482,7 @@ EncodingInfo::EncodingInfo(TraitsClass traits)
482482
} else if (_encoding == PLAIN_ENCODING_V2) {
483483
// Only binary types (Slice) need the predecoder for PLAIN_ENCODING_V2 — it converts
484484
// varint-encoded lengths to an offset-array format that downstream Slice decoders expect.
485-
// CHAR pages additionally strip trailing '\0' padding written by the convertor; other
485+
// CHAR pages additionally strip trailing '\0' padding written by the column writer; other
486486
// Slice types use the non-CHAR specialization. All current (type, PLAIN_ENCODING_V2)
487487
// registrations are Slice (CHAR/VARCHAR/STRING/JSONB/VARIANT/HLL/BITMAP/QUANTILE_STATE/
488488
// AGG_STATE per storage/types.h). The else throws at construction time to fail loudly

‎be/src/storage/storage_layout.h‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -73,8 +73,6 @@ struct StorageLayout;
7373
struct StorageBuffers {
7474
PaddedPODArray<uint8_t> bytes;
7575
PaddedPODArray<Slice> cells;
76-
77-
size_t allocated_bytes() const { return bytes.allocated_bytes() + cells.allocated_bytes(); }
7876
};
7977

8078
// The FieldTypes stored fixed width, for callers that only know their FieldType at run

‎be/test/storage/index/index_builder_test.cpp‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3492,7 +3492,8 @@ TEST(ArrayIndexInputTest, EmptyRowsStillAdvanceDocumentsAndNullPositions) {
34923492
EXPECT_EQ(writer.array_batch_rows, (std::vector<size_t> {rows}));
34933493
EXPECT_EQ(writer.array_sizes, std::vector<size_t>(rows, 0));
34943494
EXPECT_EQ(writer.array_nulls, nulls);
3495-
EXPECT_FALSE(buffer.bytes.empty());
3495+
// No element is read, but the SNII BKD writer requires a non-null value pointer.
3496+
EXPECT_NE(writer.last_values, nullptr);
34963497
}
34973498

34983499
TEST(ArrayIndexInputTest, BorrowsUnchangedFixedWidthItems) {

‎be/test/storage/segment/binary_plain_page_v3_test.cpp‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -370,7 +370,7 @@ TEST_F(BinaryPlainPageV3Test, TestEncodeDecodeAggState) {
370370
// is stripped on read by BinaryPlainPageV3PreDecoder<true>.
371371
namespace {
372372
// Build a fixed-length CHAR slice payload: each logical value padded with '\0' to
373-
// padded_len, exactly as the convertor hands it to the page builder.
373+
// padded_len, exactly as the column writer hands it to the page builder.
374374
std::vector<std::string> make_padded_char_backing(const std::vector<std::string>& logical,
375375
size_t padded_len) {
376376
std::vector<std::string> padded;

0 commit comments

Comments
 (0)