Skip to content

Commit 012fb52

Browse files
committed
[fix](be) Preserve padding for complex array_agg states
### What problem does this PR solve? Issue Number: close apache#65976 Related PR: apache#66601 Problem Summary: Native DataType deserialization may read STREAMVBYTE_PADDING bytes past the logical compressed payload. Copying the framed complex array_agg state into an exact-size std::string discarded the padded ColumnString backing buffer and also imposed the generic 1 GiB string limit. Deserialize directly from the framed padded buffer, validate the logical payload length, and advance the reader by that length. ### Release note Fix complex array_agg state deserialization for StreamVByte-compressed payloads. ### Check List (For Author) - Test: Unit Test - Unit Test: GLIBC_COMPATIBILITY=OFF ./run-be-ut.sh -j 48 --run --filter=AggregateFunctionArrayAggTest.* - Behavior changed: Yes. StreamVByte-compressed complex array_agg states now retain the required readable padding during deserialization. - Does this need documentation: No
1 parent fb66708 commit 012fb52

2 files changed

Lines changed: 14 additions & 5 deletions

File tree

‎be/src/exprs/aggregate/aggregate_function_array_agg.h‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -280,11 +280,12 @@ struct AggregateFunctionArrayAggData<T> {
280280

281281
void read(BufferReadable& buf) {
282282
DORIS_CHECK(column_data->empty());
283-
std::string serialized_buffer;
284-
buf.read_binary(serialized_buffer);
285-
const auto* end =
286-
column_type->deserialize(serialized_buffer.data(), &column_data, be_exec_version);
287-
DORIS_CHECK_EQ(end, serialized_buffer.data() + serialized_buffer.size());
283+
UInt64 serialized_bytes = 0;
284+
buf.read_var_uint(serialized_bytes);
285+
const auto* serialized_data = buf.data();
286+
const auto* end = column_type->deserialize(serialized_data, &column_data, be_exec_version);
287+
DORIS_CHECK_EQ(end, serialized_data + serialized_bytes);
288+
buf.add_offset(serialized_bytes);
288289
}
289290

290291
void merge(const Self& rhs) {

‎be/test/exprs/aggregate/agg_array_agg_test.cpp‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -275,6 +275,14 @@ TEST_F(AggregateFunctionArrayAggTest, complex_type_state_serialize_deserialize_a
275275
auto nullable_int = make_nullable(std::make_shared<DataTypeInt32>());
276276
auto nullable_string = make_nullable(std::make_shared<DataTypeString>());
277277

278+
Array streamvbyte_values;
279+
for (int32_t value = 0; value < 65; ++value) {
280+
streamvbyte_values.emplace_back(Field::create_field<TYPE_INT>(value));
281+
}
282+
check_complex_array_agg_state(std::make_shared<DataTypeArray>(nullable_int),
283+
{Field::create_field<TYPE_ARRAY>(std::move(streamvbyte_values))},
284+
{});
285+
278286
check_complex_array_agg_state(
279287
std::make_shared<DataTypeArray>(nullable_int),
280288
{array_field({Field::create_field<TYPE_INT>(1), Field()}), Field()},

0 commit comments

Comments
 (0)