You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit b86393c
Browse filesBrowse the repository at this point in the historyBrowse files
[refactor](be) Refuse INVALID_TYPE AGG_STATE as master does and tidy the write path
### What problem does this PR solve?
Related PR: #69070
Problem Summary: Review follow-ups on the IColumn segment write refactor.
- An AGG_STATE column whose function serialises to INVALID_TYPE is refused
when its writer is created, with master's "OLAP_FIELD_TYPE_AGG_STATE meet
unsupported type" error. Master accepted it in `create_agg_state_writer`,
but the convertor it created right after the writer threw that error;
the refactor had stored such states through AGG_STATE's fixed-length
layout instead.
- `ArrayColumnWriter::estimate_buffer_size` is master's again. It no longer
adds the capacity of the array index's scratch buffers, which is never
written to the segment, and `StorageBuffers::allocated_bytes` goes away.
The buffers keep their capacity for reuse; their memory is charged to the
thread memory tracker when it is allocated.
- An array batch without elements hands the index writers the data() of the
empty bytes buffer instead of resizing it first. Nothing is read through
it, and a PaddedPODArray's data() is never null, which is all the SNII BKD
writer checks; the test now asserts that pointer instead of the buffer size.
- `ScalarColumnWriter::prepare_cells`, which had one caller, is inlined into
`append_values`.
- Comments that still named the removed OlapBlockDataConvertor now name the
column writer and `rebase_offsets`.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- ASAN BE build; BE UT: 825 tests from 84 suites (writer, storage
layout, CLucene, SNII and ANN index, BUILD INDEX, variant and segment
suites, plus every suite of the test files changed in this PR): 818
passed, 0 failed, 7 skipped (two SNII corpus tests need an external
corpus, four need the NestedGroup module, one runs only in Release)
- clang-format 16, build hygiene, clang-tidy on the changed lines
- Behavior changed: No; an INVALID_TYPE AGG_STATE column fails as on master
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Copy file name to clipboardExpand all lines: be/src/storage/index/inverted/clucene/inverted_index_writer.cpp
+1-1Lines changed: 1 addition & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -424,7 +424,7 @@ Status InvertedIndexColumnWriter<field_type>::add_array_values(size_t field_size
424
424
size_t start_off = 0;
425
425
std::vector<ReaderPtr> keep_readers;
426
426
for (size_t i = 0; i < count; ++i) {
427
-
// nullmap & value ptr-array may not from offsets[i] because olap_convertor make offsets accumulate from _base_offset which may not is 0, but nullmap & value in this segment is from 0, we only need
427
+
// nullmap & value ptr-array may not from offsets[i] because rebase_offsets makes offsets start from the segment's item ordinal which may not be 0, but nullmap & value in this segment is from 0, we only need
428
428
// every single array row element size to go through the nullmap & value ptr-array, and also can go through the every row in array to keep with _rid++
429
429
auto array_elem_size = offsets[i + 1] - offsets[i];
430
430
// TODO(Amory).later we use object pool to avoid field creation
0 commit comments