Skip to content

Commit 8b4c9cb

Browse files
morningmanclaude
andcommitted
[fix](be) Bound the Variant metadata that Iceberg and Paimon writes copy into each row
### What problem does this PR solve? Issue Number: None Related PR: #68883, #69194 Problem Summary: parse_json() encodes a block with one metadata dictionary of all the keys of its rows, which every row of the column shares. The Iceberg writer (write_column_to_iceberg_arrow passes compact_metadata = false) and the Paimon writer (write_binary_variant_arrow) copy that whole dictionary into the metadata of every row, and the Paimon writer also validates it once per row. A batch therefore costs rows x dictionary bytes of memory, and on Paimon as much validation work. With many distinct keys, such as 4064 rows that each have a distinct 250-byte key (a 1 MiB dictionary), the metadata column needs about 4 GiB, past the 2 GiB an Arrow binary array holds, and the INSERT fails or runs out of memory. Nothing is written wrongly. The Flight path already gives every row a dictionary of its own keys. Table writers now keep the dictionary a row shares with its batch unless it is more than 1 KiB larger than the row's value; then the row is re-encoded with a dictionary of only its keys, as Flight does. The metadata a write copies is then bounded by the bytes of the values plus 1 KiB a row, the re-encoding costs no more than copying that dictionary would, and the Paimon writer validates the row's own dictionary. A small dictionary is still written as the batch shares it, so the rows keep metadata that Parquet dictionary-encodes. Flight keeps giving every row its own keys. ### Release note Fix INSERT into Iceberg and Paimon VARIANT columns failing with an Arrow capacity error, or running out of memory, when a batch of JSON has many distinct keys. ### Check List (For Author) - Test: Regression test (test_iceberg_write_variant_many_keys and test_paimon_write_variant_many_keys write 8192 rows with distinct 248-byte keys and read them back through Doris and Spark; with master's serde the Iceberg INSERT aborts an ASAN BE, since the Arrow capacity error is a FatalError, and the Paimon INSERT fails with it; the existing Iceberg and Paimon Variant suites pass) and Unit Test (data_type_variant_v2_serde_output_test; each of five mutations of the rule turns it red) - Behavior changed: No (a row with a large shared dictionary is written with an equivalent dictionary of its own keys) - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent 811de72 commit 8b4c9cb

6 files changed

Lines changed: 354 additions & 20 deletions

File tree

‎be/src/core/data_type_serde/data_type_variant_v2_serde.cpp‎

Lines changed: 58 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -102,6 +102,49 @@ Status append_flight_variant_value(VariantRef value, VariantBatchBuilder::Row& o
102102
return Status::OK();
103103
}
104104

105+
// The dictionary that a writer gives each Variant row; see with_row_metadata().
106+
enum class VariantRowMetadata {
107+
// Flight sends every row with a dictionary of only the keys it uses.
108+
OWN_KEYS,
109+
// Table writers keep the dictionary a row shares with its batch unless it is more than
110+
// SHARED_METADATA_SLACK bytes larger than the row's value.
111+
BOUNDED,
112+
};
113+
114+
// Rows encoded together, such as the rows of one parse_json() call, share one dictionary of all
115+
// their keys, and a table writer copies the dictionary into every row. 4064 rows with a distinct
116+
// 250-byte key each share a 1 MiB dictionary, so copying it costs 4 GiB, past the 2 GiB that an
117+
// Arrow binary array holds. A row whose dictionary exceeds its value by more than this slack gets
118+
// its own keys instead, which bounds the copied metadata by the value bytes plus 1 KiB a row. A
119+
// smaller dictionary is kept: the rows of a batch then carry the same metadata, which Parquet
120+
// dictionary-encodes.
121+
constexpr size_t SHARED_METADATA_SLACK = 1024;
122+
123+
// Re-encodes `value` with a dictionary of only the keys it uses, kept alive by `owner`, when
124+
// `policy` asks for that and the dictionary holds keys the row does not use. For example, under
125+
// BOUNDED, {"a": 1} keeps a shared dictionary [a, b] but gets [a] when the dictionary also holds
126+
// 4000 other keys.
127+
Status with_row_metadata(VariantRef& value, VariantRowMetadata policy,
128+
std::optional<VariantBatchBuilder>& owner) {
129+
const auto keys = value.metadata.dict_size();
130+
// An empty dictionary or an object using every key already has row-local metadata.
131+
if (keys == 0 ||
132+
(value.basic_type() == VariantBasicType::OBJECT && value.num_elements() == keys)) {
133+
return Status::OK();
134+
}
135+
if (policy == VariantRowMetadata::BOUNDED &&
136+
value.metadata.size <= value.value.size + SHARED_METADATA_SLACK) {
137+
return Status::OK();
138+
}
139+
VariantBatchBuilder encoder;
140+
auto row = encoder.begin_row();
141+
RETURN_IF_ERROR(append_flight_variant_value(value, row));
142+
row.finish();
143+
owner.emplace(encoder.finish_batch());
144+
value = owner->value_at(0);
145+
return Status::OK();
146+
}
147+
105148
using MetaIdsColumn = ColumnVector<TYPE_UINT32>;
106149

107150
std::span<const NullMap::value_type> forced_nulls(const NullMap* null_map) {
@@ -336,7 +379,11 @@ Status write_binary_variant_arrow(const IColumn& column, const NullMap* null_map
336379
column, start, end, outer_nulls,
337380
[&](size_t) { require_variant_arrow_status(builder.AppendNull()); },
338381
[&](size_t row, VariantRef value) {
382+
std::optional<VariantBatchBuilder> own_keys;
339383
try {
384+
// Check and validate the row as it is written: a row given its own keys is
385+
// validated against them instead of the dictionary its batch shares.
386+
THROW_IF_ERROR(with_row_metadata(value, VariantRowMetadata::BOUNDED, own_keys));
340387
constexpr size_t BINARY_VARIANT_SIZE_LIMIT = 128 * 1024 * 1024;
341388
if (value.value.size > BINARY_VARIANT_SIZE_LIMIT ||
342389
value.metadata.size > BINARY_VARIANT_SIZE_LIMIT) {
@@ -784,7 +831,7 @@ Status write_arrow(const IColumn& column, const NullMap* null_map, Builder& buil
784831

785832
Status write_parquet_variant_arrow(const IColumn& column, const NullMap* null_map,
786833
arrow::ArrayBuilder* array_builder, int64_t start, int64_t end,
787-
bool compact_metadata) {
834+
VariantRowMetadata row_metadata) {
788835
if (start < 0 || end < start) {
789836
return Status::InvalidArgument("Invalid Variant Arrow row range [{}, {})", start, end);
790837
}
@@ -815,22 +862,10 @@ Status write_parquet_variant_arrow(const IColumn& column, const NullMap* null_ma
815862
if (!status.ok()) {
816863
return;
817864
}
818-
std::optional<VariantBatchBuilder> compacted;
819-
if (compact_metadata) {
820-
const auto keys = value.metadata.dict_size();
821-
// An empty dictionary or an object using every key already has row-local metadata.
822-
if (keys != 0 && (value.basic_type() != VariantBasicType::OBJECT ||
823-
value.num_elements() != keys)) {
824-
VariantBatchBuilder encoder;
825-
auto row = encoder.begin_row();
826-
status = append_flight_variant_value(value, row);
827-
if (!status.ok()) {
828-
return;
829-
}
830-
row.finish();
831-
compacted.emplace(encoder.finish_batch());
832-
value = compacted->value_at(0);
833-
}
865+
std::optional<VariantBatchBuilder> own_keys;
866+
status = with_row_metadata(value, row_metadata, own_keys);
867+
if (!status.ok()) {
868+
return;
834869
}
835870
if (value.metadata.size > std::numeric_limits<int32_t>::max() ||
836871
value.value.size > std::numeric_limits<int32_t>::max()) {
@@ -909,7 +944,8 @@ Status DataTypeVariantV2SerDe::write_column_to_arrow(const IColumn& column, cons
909944
const size_t first = checked_row(start);
910945
const size_t last = checked_row(end);
911946
if (is_native_arrow_type(*array_builder->type())) {
912-
return write_parquet_variant_arrow(column, null_map, array_builder, start, end, true);
947+
return write_parquet_variant_arrow(column, null_map, array_builder, start, end,
948+
VariantRowMetadata::OWN_KEYS);
913949
}
914950
if (array_builder->type()->id() == arrow::Type::STRING) {
915951
return write_arrow(column, null_map, assert_cast<arrow::StringBuilder&>(*array_builder),
@@ -952,9 +988,11 @@ Status DataTypeVariantV2SerDe::write_column_to_iceberg_arrow(
952988
const std::shared_ptr<const IDataType>&, const IColumn& column, const NullMap* null_map,
953989
const std::shared_ptr<arrow::Field>&, arrow::ArrayBuilder* array_builder, int64_t start,
954990
int64_t end, const cctz::time_zone&) const {
955-
// Table writers retain their validated dictionary; Flight compacts each transmitted row.
991+
// A row keeps a small dictionary shared with its batch and gets its own keys instead of a
992+
// large one; Flight compacts each transmitted row.
956993
RETURN_IF_CATCH_EXCEPTION({
957-
return write_parquet_variant_arrow(column, null_map, array_builder, start, end, false);
994+
return write_parquet_variant_arrow(column, null_map, array_builder, start, end,
995+
VariantRowMetadata::BOUNDED);
958996
});
959997
return Status::OK();
960998
}

‎be/test/core/data_type_serde/data_type_variant_v2_serde_output_test.cpp‎

Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@
2828
#include <span>
2929
#include <string>
3030
#include <string_view>
31+
#include <utility>
3132
#include <vector>
3233

3334
#include "core/arena.h"
@@ -216,6 +217,49 @@ std::unique_ptr<arrow::StructBuilder> binary_variant_arrow_builder() {
216217
std::make_shared<arrow::BinaryBuilder>(arrow::default_memory_pool())});
217218
}
218219

220+
// The struct<metadata, value> storage that Iceberg writes into Parquet.
221+
std::unique_ptr<arrow::StructBuilder> parquet_variant_arrow_builder() {
222+
return std::make_unique<arrow::StructBuilder>(
223+
arrow::struct_({arrow::field("metadata", arrow::binary(), false),
224+
arrow::field("value", arrow::binary(), false)}),
225+
arrow::default_memory_pool(),
226+
std::vector<std::shared_ptr<arrow::ArrayBuilder>> {
227+
std::make_shared<arrow::BinaryBuilder>(arrow::default_memory_pool()),
228+
std::make_shared<arrow::BinaryBuilder>(arrow::default_memory_pool())});
229+
}
230+
231+
// The Variant rows that the Iceberg and the Paimon writer produce for `column`, as they write them.
232+
std::vector<std::vector<std::pair<std::string, std::string>>> table_variant_rows(
233+
const IColumn& column) {
234+
DataTypeVariantV2SerDe serde;
235+
auto iceberg = parquet_variant_arrow_builder();
236+
Status status =
237+
serde.write_column_to_iceberg_arrow(nullptr, column, nullptr, nullptr, iceberg.get(), 0,
238+
column.size(), cctz::utc_time_zone());
239+
EXPECT_TRUE(status.ok()) << status;
240+
auto paimon = binary_variant_arrow_builder();
241+
status = serde.write_column_to_paimon_arrow(nullptr, column, nullptr, nullptr, paimon.get(), 0,
242+
column.size(), cctz::utc_time_zone());
243+
EXPECT_TRUE(status.ok()) << status;
244+
245+
std::vector<std::vector<std::pair<std::string, std::string>>> writers;
246+
const std::array<std::pair<arrow::StructBuilder*, int>, 2> builders {
247+
{{iceberg.get(), 0}, {paimon.get(), 1}}};
248+
for (const auto& [builder, metadata_field] : builders) {
249+
std::shared_ptr<arrow::Array> output;
250+
EXPECT_TRUE(builder->Finish(&output).ok());
251+
const auto& array = assert_cast<const arrow::StructArray&>(*output);
252+
const auto& metadata = assert_cast<const arrow::BinaryArray&>(*array.field(metadata_field));
253+
const auto& values =
254+
assert_cast<const arrow::BinaryArray&>(*array.field(1 - metadata_field));
255+
auto& rows = writers.emplace_back();
256+
for (int64_t row = 0; row < array.length(); ++row) {
257+
rows.emplace_back(std::string(metadata.GetView(row)), std::string(values.GetView(row)));
258+
}
259+
}
260+
return writers;
261+
}
262+
219263
void expect_binary_variant_bytes(const DataTypeVariantV2SerDe& serde, const IColumn& column,
220264
const ColumnVariantV2& encoded,
221265
const NullMap* null_map = nullptr) {
@@ -509,6 +553,71 @@ TEST(DataTypeVariantV2SerdeOutputTest, PaimonKeepsItsOwnBinaryLayout) {
509553
EXPECT_EQ(binary.GetView(0), std::string_view(expected.data, expected.size));
510554
}
511555

556+
TEST(DataTypeVariantV2SerdeOutputTest, TableWritersGiveRowsTheirOwnKeysInsteadOfALargeDictionary) {
557+
// parse_json() encodes a batch with one dictionary of all its keys: 256 rows with a distinct
558+
// 200-byte key share a 52 KiB dictionary. Copied into every row it would cost 13 MiB for 2 KiB of
559+
// values, and 4064 such rows overflow an Arrow binary array, so the Iceberg and the Paimon writer
560+
// give each row a dictionary of its own key; the row still decodes to its key and value. Row 0
561+
// holds a 64 KiB string, which the shared dictionary does not exceed by more than 1 KiB, so it
562+
// keeps that dictionary. MUTATION: keeping every shared dictionary, or giving row 0 its own key
563+
// -> red.
564+
constexpr size_t ROWS = 256;
565+
JsonStringToVariantEncoder encoder(
566+
{.max_json_key_length = 1024, .throw_on_invalid_json = true});
567+
std::vector<std::string> keys;
568+
for (size_t row = 0; row < ROWS; ++row) {
569+
keys.push_back("k" + std::to_string(row) + std::string(200, 'x'));
570+
const std::string value =
571+
row == 0 ? "\"" + std::string(64 * 1024, 'v') + "\"" : std::to_string(row);
572+
const std::string document = "{\"" + keys.back() + "\":" + value + "}";
573+
encoder.add_json({document.data(), document.size()});
574+
}
575+
VariantBatchBuilder batch = encoder.finish_batch();
576+
auto column = ColumnVariantV2::create();
577+
column->insert_encoded_batch(batch);
578+
const VariantRef shared = column->get_value_ref(1);
579+
ASSERT_EQ(shared.metadata.dict_size(), ROWS);
580+
581+
for (const auto& rows : table_variant_rows(*column)) {
582+
ASSERT_EQ(rows.size(), ROWS);
583+
EXPECT_EQ(rows[0].first, std::string_view(shared.metadata.data, shared.metadata.size));
584+
for (size_t row = 0; row < ROWS; ++row) {
585+
const VariantRef value {
586+
.metadata = {.data = rows[row].first.data(), .size = rows[row].first.size()},
587+
.value = {rows[row].second.data(), rows[row].second.size()}};
588+
if (row != 0) {
589+
EXPECT_EQ(value.metadata.dict_size(), 1U) << row;
590+
}
591+
ASSERT_EQ(value.basic_type(), VariantBasicType::OBJECT) << row;
592+
ASSERT_EQ(value.num_elements(), 1U) << row;
593+
uint32_t field_id = 0;
594+
const VariantRef child = value.object_view().value_at(0, &field_id);
595+
EXPECT_EQ(value.metadata.key_at(field_id).to_string(), keys[row]);
596+
if (row == 0) {
597+
EXPECT_EQ(child.get_string().size, size_t {64 * 1024});
598+
} else {
599+
EXPECT_EQ(child.get_int(), static_cast<int64_t>(row));
600+
}
601+
}
602+
}
603+
}
604+
605+
TEST(DataTypeVariantV2SerdeOutputTest, TableWritersKeepASmallSharedDictionary) {
606+
// A dictionary that exceeds a row's value by at most 1 KiB is written as the batch shares it:
607+
// Parquet dictionary-encodes the metadata the rows then have in common, and the row is not
608+
// re-encoded. MUTATION: giving every row its own keys, as Flight does -> red.
609+
auto column = encoded_json({R"({"a":1})", R"({"b":[true,"x"]})", R"({"a":{"b":2}})"});
610+
for (const auto& rows : table_variant_rows(*column)) {
611+
ASSERT_EQ(rows.size(), column->size());
612+
for (size_t row = 0; row < column->size(); ++row) {
613+
const VariantRef expected = column->get_value_ref(row);
614+
EXPECT_EQ(rows[row].first,
615+
std::string_view(expected.metadata.data, expected.metadata.size));
616+
EXPECT_EQ(rows[row].second, std::string_view(expected.value.data, expected.value.size));
617+
}
618+
}
619+
}
620+
512621
TEST(DataTypeVariantV2SerdeOutputTest, BinaryStructRejectsUnsupportedPrimitive) {
513622
DataTypeVariantV2SerDe serde;
514623
VariantBatchBuilder builder(VariantBatchBuilder::ReserveHint {.rows = 1});
Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
-- This file is automatically generated. You should know what you did if you want to edit this
2+
-- !many_keys_read_back --
3+
8192 8192
4+
Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
-- This file is automatically generated. You should know what you did if you want to edit this
2+
-- !many_keys_read_back --
3+
8192 8192
4+
Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
// PARSE_TO_VARIANT encodes a block with one metadata dictionary of all the keys of its rows. Here every row has
19+
// its own 248-byte key, so the dictionary of a 4064-row block is 1 MiB, and copying it into the metadata of every
20+
// row would take 4 GiB, past the 2 GiB that an Arrow binary array holds. The writer gives such a row a dictionary
21+
// of its own key instead, so the INSERT succeeds, and Doris and Spark read every row back as it was written.
22+
suite("test_iceberg_write_variant_many_keys",
23+
"p0,external,iceberg,external_docker,external_docker_iceberg") {
24+
String enabled = context.config.otherConfigs.get("enableIcebergTest")
25+
if (enabled == null || !enabled.equalsIgnoreCase("true")) {
26+
logger.info("disable iceberg test")
27+
return
28+
}
29+
30+
String restPort = context.config.otherConfigs.get("iceberg_rest_uri_port")
31+
String minioPort = context.config.otherConfigs.get("iceberg_minio_port")
32+
String externalEnvIp = context.config.otherConfigs.get("externalEnvIp")
33+
String restUri = context.config.otherConfigs.get("iceberg_rest_uri")
34+
if (restUri == null) {
35+
restUri = "http://${externalEnvIp}:${restPort}"
36+
}
37+
String catalogName = "test_iceberg_write_variant_many_keys"
38+
String dbName = "iceberg_write_variant_many_keys_db"
39+
40+
spark_iceberg_multi """
41+
CREATE NAMESPACE IF NOT EXISTS demo.${dbName};
42+
DROP TABLE IF EXISTS demo.${dbName}.variant_many_keys;
43+
CREATE TABLE demo.${dbName}.variant_many_keys (id BIGINT) USING iceberg
44+
TBLPROPERTIES ('format-version'='3', 'write.format.default'='parquet');
45+
ALTER TABLE demo.${dbName}.variant_many_keys ADD COLUMN payload VARIANT;
46+
"""
47+
48+
sql """DROP CATALOG IF EXISTS ${catalogName}"""
49+
sql """
50+
CREATE CATALOG ${catalogName} PROPERTIES (
51+
'type'='iceberg',
52+
'iceberg.catalog.type'='rest',
53+
'uri'='${restUri}',
54+
's3.access_key'='admin',
55+
's3.secret_key'='password',
56+
's3.endpoint'='http://${externalEnvIp}:${minioPort}',
57+
's3.region'='us-east-1',
58+
'meta.cache.iceberg.table.ttl-second'='0',
59+
'meta.cache.iceberg.schema.ttl-second'='0'
60+
)
61+
"""
62+
sql """SWITCH ${catalogName}"""
63+
sql """USE ${dbName}"""
64+
sql """SET enable_file_scanner_v2=true"""
65+
66+
sql """
67+
INSERT INTO variant_many_keys
68+
SELECT number,
69+
PARSE_TO_VARIANT(CONCAT('{"', REPEAT('k', 240), LPAD(CAST(number AS STRING), 8, '0'), '":',
70+
CAST(number AS STRING), '}'))
71+
FROM numbers("number" = "8192")
72+
"""
73+
order_qt_many_keys_read_back """
74+
SELECT COUNT(*),
75+
SUM(CASE WHEN CAST(payload AS STRING) = CONCAT('{"', REPEAT('k', 240),
76+
LPAD(CAST(id AS STRING), 8, '0'), '":', CAST(id AS STRING), '}') THEN 1 ELSE 0 END)
77+
FROM variant_many_keys
78+
"""
79+
80+
spark_iceberg """REFRESH TABLE demo.${dbName}.variant_many_keys"""
81+
List<List<Object>> sparkRows = spark_iceberg """
82+
SELECT COUNT(*),
83+
SUM(CASE WHEN to_json(payload) = CONCAT('{"', REPEAT('k', 240), LPAD(CAST(id AS STRING), 8, '0'),
84+
'":', CAST(id AS STRING), '}') THEN 1 ELSE 0 END)
85+
FROM demo.${dbName}.variant_many_keys
86+
"""
87+
assertEquals(1, sparkRows.size())
88+
assertEquals("8192", sparkRows[0][0].toString())
89+
assertEquals("8192", sparkRows[0][1].toString())
90+
}

0 commit comments

Comments
 (0)