Skip to content

Commit ed28e47

Browse files
morningmanhubgeterclaude
committed
[feature](iceberg) Write Iceberg VARIANT columns and require NULL VARIANT defaults
### What problem does this PR solve? Issue Number: None Related PR: #66620, #67323 Problem Summary: Port branch-4.1 #66620 (Iceberg Variant V2 writes) and #67323 (NULL defaults for VARIANT) to master. Master read Iceberg VARIANT columns but rejected every write to a table holding one ("Iceberg VARIANT columns are read-only and cannot be written") and every VARIANT in Iceberg DDL ("Unsupported type for Iceberg: VARIANT"), and BE could neither parse "variant" in the Iceberg schema JSON nor map it to Arrow. A connector write whose source mixed a Variant with a number in UNION, IF, CASE, a recursive CTE or LATERAL VIEW got a DECIMAL common type, so object and array Variants were silently written as NULL (Paimon today). BE: the Iceberg type system and Arrow schema handle VARIANT (arrow.parquet.variant over struct<metadata, value>); the Iceberg Parquet writer counts the values and nulls of a Variant on its metadata leaf, under the Variant's field id, and records no size or bounds for it; the format v2 Iceberg reader rejects a non-NULL VARIANT initial default. FE: the Iceberg connector maps a top-level VARIANT in CREATE TABLE, CTAS and ADD COLUMN and rejects a nested one; requires format version 3 and Parquet data files for them and for writes of data files, and rejects write.parquet.shred-variants; keeps MODIFY of a VARIANT column to its comment, nullability and position, and rejects type changes to or from VARIANT; collects column metrics for Variant fields; requires NULL VARIANT defaults. VariantWritePlanValidator rejects an implicit Variant-to-non- Variant cast in the lineage of a Variant target column of any connector sink: INSERT, INSERT OVERWRITE, CTAS, rewrite, Paimon UPDATE / MERGE and Iceberg UPDATE / MERGE, whose merge sink now casts its output columns by position, so a data column named `operation` is no longer left uncast. Unlike branch-4.x, the validator does not inspect the condition of an IF or a CASE branch, which only chooses the written value, its message names no engine, and it also covers Paimon UPDATE / MERGE. ### Release note Doris can write VARIANT columns of Iceberg format-version 3 tables with Parquet data files (INSERT, INSERT OVERWRITE, UPDATE, MERGE and rewrite_data_files) and create or add top-level VARIANT columns. A write into a VARIANT column of an Iceberg or Paimon table now fails when its source implicitly casts a VARIANT to another type, for example IF(cond, payload, 1) or a UNION of a Variant and a number, instead of storing object and array values as NULL; write an explicit CAST to keep such a conversion. ### Check List (For Author) - Test: Regression test / Unit Test - BE UT: Iceberg Variant type, schema parsing, Arrow schema, Parquet writer metrics and NULL initial defaults (474 cases of the related suites pass under ASAN). - FE UT: VariantWritePlanValidatorTest and the merge sink binding in fe-core (131 cases of 12 classes), and all 1580 cases of fe-connector-iceberg. - Regression: test_iceberg_variant_read, the new test_iceberg_variant_defaults and test_paimon_write_variant_errors; all Paimon and Iceberg write suites and the Iceberg row-lineage DML suites pass locally, except test_paimon_write_thread_lifecycle, which needs Linux /proc. - Behavior changed: Yes (Iceberg VARIANT columns become writable; lossy implicit casts into VARIANT targets are rejected) - Does this need documentation: Yes (Iceberg VARIANT writes and DDL) Co-authored-by: daidai <[email protected]> Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent 28c9f8d commit ed28e47

37 files changed

Lines changed: 2951 additions & 203 deletions

File tree

‎be/src/exec/sink/viceberg_merge_sink.cpp‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -168,8 +168,9 @@ Status VIcebergMergeSink::write(RuntimeState* state, Block& block) {
168168
_delete_row_count += delete_rows;
169169
_insert_row_count += insert_rows;
170170

171-
// A delete-only plan deliberately omits the data writer so Variant target schemas never enter
172-
// the unsupported Iceberg data-write path. Reject a mismatched FE plan before dereferencing it.
171+
// A delete-only plan omits the data writer, which it does not need and which a BE before Variant
172+
// writes could not open for a Variant target schema. Reject a mismatched FE plan before
173+
// dereferencing it.
173174
if (has_insert && !_writes_data_files) {
174175
return Status::InternalError(
175176
"Iceberg delete-only merge sink received a data insert operation");

‎be/src/format/table/iceberg/arrow_schema_util.cpp‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@
1717

1818
#include "format/table/iceberg/arrow_schema_util.h"
1919

20+
#include <arrow/extension/parquet_variant.h>
2021
#include <arrow/extension/uuid.h>
2122
#include <arrow/type.h>
2223
#include <arrow/util/key_value_metadata.h>
@@ -29,7 +30,8 @@ const char* IcebergArrowSchemaConvertor::MAP_TYPE_VALUE = "mapType";
2930
const char* IcebergArrowSchemaConvertor::UUID_TYPE_VALUE = "uuid";
3031

3132
Status IcebergArrowSchemaConvertor::get_arrow_schema(std::shared_ptr<arrow::Schema>* result) const {
32-
// Field IDs, UUID annotations and timestamp bindings come from the pinned table schema.
33+
// Field IDs, UUID annotations, Variant storage and timestamp bindings come from the pinned
34+
// table schema.
3335
std::vector<std::shared_ptr<arrow::Field>> fields;
3436
RETURN_IF_ERROR(convert_fields(fields));
3537
auto schema = arrow::schema(std::move(fields));
@@ -108,6 +110,13 @@ Status IcebergArrowSchemaConvertor::convert_to_arrow_field(
108110
arrow_type = arrow::binary();
109111
break;
110112

113+
case iceberg::TypeID::VARIANT:
114+
arrow_type = arrow::extension::variant(arrow::struct_({
115+
arrow::field("metadata", arrow::binary(), false),
116+
arrow::field("value", arrow::binary(), false),
117+
}));
118+
break;
119+
111120
case iceberg::TypeID::UUID:
112121
metadata[ORIGINAL_TYPE] = UUID_TYPE_VALUE;
113122
arrow_type = _use_uuid_extension ? arrow::extension::uuid() : arrow::fixed_size_binary(16);

‎be/src/format/table/iceberg/types.cpp‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -174,6 +174,8 @@ std::unique_ptr<PrimitiveType> Types::from_primitive_string(const std::string& t
174174
return std::make_unique<UUIDType>();
175175
} else if (lower_type_string == "binary") {
176176
return std::make_unique<BinaryType>();
177+
} else if (lower_type_string == "variant") {
178+
return std::make_unique<VariantType>();
177179
} else {
178180
std::regex fixed(R"(fixed\[\s*(\d+)\s*\])");
179181
std::regex decimal(R"(decimal\(\s*(\d+)\s*,\s*(\d+)\s*\))");

‎be/src/format/table/iceberg/types.h‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,7 @@ enum TypeID {
4848
UUID,
4949
FIXED,
5050
BINARY,
51+
VARIANT,
5152
DECIMAL,
5253
STRUCT,
5354
LIST,
@@ -279,6 +280,15 @@ class BinaryType : public PrimitiveType {
279280
std::string to_string() const override { return "binary"; }
280281
};
281282

283+
class VariantType : public PrimitiveType {
284+
public:
285+
~VariantType() override = default;
286+
287+
TypeID type_id() const override { return TypeID::VARIANT; }
288+
289+
std::string to_string() const override { return "variant"; }
290+
};
291+
282292
class FixedType : public PrimitiveType {
283293
public:
284294
~FixedType() override = default;

‎be/src/format/transformer/viceberg_parquet_writer.cpp‎

Lines changed: 44 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,8 @@
2121
#include <parquet/schema.h>
2222

2323
#include <algorithm>
24+
#include <optional>
25+
#include <unordered_set>
2426

2527
#include "format/table/iceberg/iceberg_arrow_block_convertor.h"
2628
#include "format/table/parquet_utils.h"
@@ -51,6 +53,33 @@ static std::string encode_iceberg_bound(const ::parquet::Statistics& stats, std:
5153
encoded.erase(0, start);
5254
return encoded;
5355
}
56+
57+
namespace {
58+
59+
// The Iceberg field whose metrics a Parquet leaf column feeds.
60+
struct IcebergMetricColumn {
61+
int field_id;
62+
bool is_variant_metadata;
63+
};
64+
65+
// The leaves of a Variant carry no Iceberg field id. Its metadata leaf counts the values and nulls
66+
// of the Variant field, and its value leaf feeds no metric: its byte statistics are no logical
67+
// Variant statistics.
68+
std::optional<IcebergMetricColumn> iceberg_metric_column(const ::parquet::schema::Node& leaf) {
69+
const auto* parent = leaf.parent();
70+
const bool is_variant_leaf = parent != nullptr && parent->logical_type() != nullptr &&
71+
parent->logical_type()->is_variant();
72+
if (is_variant_leaf && leaf.name() != "metadata") {
73+
return std::nullopt;
74+
}
75+
const int field_id = is_variant_leaf ? parent->field_id() : leaf.field_id();
76+
// Every field of the Iceberg schema the file is written with carries its id.
77+
DORIS_CHECK_GE(field_id, 0);
78+
return IcebergMetricColumn {field_id, is_variant_leaf};
79+
}
80+
81+
} // namespace
82+
5483
VIcebergParquetWriter::VIcebergParquetWriter(RuntimeState* state, io::FileWriter* file_writer,
5584
const VExprContextSPtrs& output_vexpr_ctxs,
5685
std::vector<std::string> column_names,
@@ -97,16 +126,28 @@ Status VIcebergParquetWriter::collect_file_statistics_after_close(TIcebergColumn
97126
std::map<int, std::string> lower_bounds;
98127
std::map<int, std::string> upper_bounds;
99128
std::map<int, std::shared_ptr<::parquet::Statistics>> merged_column_stats;
129+
// Variant fields get value and null counts, but no column size and no bounds.
130+
std::unordered_set<int> variant_field_ids;
100131

101132
const int num_row_groups = file_metadata->num_row_groups();
102133
const int num_columns = file_metadata->num_columns();
103134
for (int col_idx = 0; col_idx < num_columns; ++col_idx) {
104-
auto field_id = file_metadata->schema()->Column(col_idx)->schema_node()->field_id();
135+
const auto metric_column =
136+
iceberg_metric_column(*file_metadata->schema()->Column(col_idx)->schema_node());
137+
if (!metric_column.has_value()) {
138+
continue;
139+
}
140+
const auto [field_id, is_variant_metadata] = *metric_column;
141+
if (is_variant_metadata) {
142+
variant_field_ids.insert(field_id);
143+
}
105144

106145
for (int rg_idx = 0; rg_idx < num_row_groups; ++rg_idx) {
107146
auto row_group = file_metadata->RowGroup(rg_idx);
108147
auto column_chunk = row_group->ColumnChunk(col_idx);
109-
column_sizes[field_id] += column_chunk->total_compressed_size();
148+
if (!is_variant_metadata) {
149+
column_sizes[field_id] += column_chunk->total_compressed_size();
150+
}
110151

111152
if (column_chunk->is_stats_set()) {
112153
auto column_stat = column_chunk->statistics();
@@ -129,7 +170,7 @@ Status VIcebergParquetWriter::collect_file_statistics_after_close(TIcebergColumn
129170
null_value_counts[field_id] = null_count;
130171
value_counts[field_id] += null_count;
131172
}
132-
if (column_stat->HasMinMax()) {
173+
if (!variant_field_ids.contains(field_id) && column_stat->HasMinMax()) {
133174
has_any_min_max = true;
134175
lower_bounds[field_id] = encode_iceberg_bound(*column_stat, column_stat->EncodeMin());
135176
upper_bounds[field_id] = encode_iceberg_bound(*column_stat, column_stat->EncodeMax());

‎be/src/format_v2/table/iceberg_reader.cpp‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -524,6 +524,10 @@ static Status build_v2_json_scalar_default(const format::ColumnDefinition& field
524524
const rapidjson::Value& json_value,
525525
std::deque<std::string>* binary_storage, Field* result) {
526526
const auto primitive_type = value_type->get_primitive_type();
527+
if (primitive_type == TYPE_VARIANT) {
528+
return Status::InvalidArgument(
529+
"Iceberg VARIANT initial-default for field '{}' must be NULL", field.name);
530+
}
527531
std::string serialized_value = iceberg_json_scalar_text(json_value);
528532
const bool binary_like =
529533
field.initial_default_value_is_base64 || primitive_type == TYPE_VARBINARY;
@@ -606,6 +610,10 @@ static Status build_v2_initial_default_field(const format::ColumnDefinition& fie
606610

607611
const auto value_type = remove_nullable(data_type);
608612
const auto primitive_type = value_type->get_primitive_type();
613+
if (primitive_type == TYPE_VARIANT) {
614+
return Status::InvalidArgument(
615+
"Iceberg VARIANT initial-default for field '{}' must be NULL", field.name);
616+
}
609617
if (is_complex_type(primitive_type)) {
610618
rapidjson::Document document;
611619
document.Parse(field.initial_default_value->data(), field.initial_default_value->size());

‎be/test/format/arrow/arrow_block_convertor_test.cpp‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
#include "format/arrow/arrow_block_convertor.h"
1919

2020
#include <arrow/api.h>
21+
#include <arrow/extension_type.h>
2122
#include <arrow/io/api.h>
2223
#include <arrow/ipc/api.h>
2324
#include <gtest/gtest.h>
@@ -79,13 +80,16 @@ TEST_F(ArrowBlockConvertorTest, ParquetRejectsMismatchedColumnNames) {
7980

8081
TEST_F(ArrowBlockConvertorTest, IcebergBuildsItsOwnSchemaAndMetadata) {
8182
const std::string json =
82-
R"({"type":"struct","fields":[{"id":7,"name":"payload","required":false,"type":"string"}]})";
83+
R"({"type":"struct","fields":[{"id":7,"name":"payload","required":false,"type":"variant"}]})";
8384
auto schema = iceberg::SchemaParser::from_json(json);
8485
iceberg::IcebergArrowBlockConvertor converter(*schema, &json, "UTC", cctz::utc_time_zone());
8586
ASSERT_TRUE(converter.init().ok());
8687
const auto& arrow_schema = converter.arrow_schema();
8788
ASSERT_NE(nullptr, arrow_schema);
88-
EXPECT_EQ(arrow::Type::STRING, arrow_schema->field(0)->type()->id());
89+
ASSERT_EQ(arrow::Type::EXTENSION, arrow_schema->field(0)->type()->id());
90+
const auto& variant = static_cast<const arrow::ExtensionType&>(*arrow_schema->field(0)->type());
91+
EXPECT_EQ(arrow::Type::STRUCT, variant.storage_type()->id());
92+
EXPECT_EQ("arrow.parquet.variant", variant.extension_name());
8993
ASSERT_NE(nullptr, arrow_schema->metadata());
9094
EXPECT_EQ(json, arrow_schema->metadata()->Get("iceberg.schema").ValueOrDie());
9195
EXPECT_EQ("7", arrow_schema->field(0)->metadata()->Get("PARQUET:field_id").ValueOrDie());

‎be/test/format/table/iceberg/arrow_schema_util_test.cpp‎

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
#include "format/table/iceberg/arrow_schema_util.h"
1919

2020
#include <arrow/api.h>
21+
#include <arrow/extension/parquet_variant.h>
2122
#include <arrow/io/api.h>
2223
#include <arrow/status.h>
2324
#include <arrow/type.h>
@@ -28,6 +29,7 @@
2829
#include <parquet/arrow/writer.h>
2930
#include <parquet/schema.h>
3031

32+
#include "core/assert_cast.h"
3133
#include "core/block/block.h"
3234
#include "core/column/column_array.h"
3335
#include "core/column/column_nullable.h"
@@ -358,6 +360,101 @@ TEST(ArrowSchemaUtilTest, test_binary_field_types) {
358360
EXPECT_EQ("uuid", fields[3]->metadata()->Get("originalType").ValueUnsafe());
359361
}
360362

363+
TEST(ArrowSchemaUtilTest, test_variant_field) {
364+
std::vector<NestedField> nested_fields;
365+
nested_fields.emplace_back(true, 21, "payload", std::make_unique<VariantType>(), std::nullopt);
366+
Schema schema(1, std::move(nested_fields));
367+
368+
std::vector<std::shared_ptr<arrow::Field>> fields;
369+
Status st = IcebergArrowSchemaConvertor(schema, "utc").convert_fields(fields);
370+
ASSERT_TRUE(st.ok()) << st;
371+
ASSERT_EQ(1, fields.size());
372+
EXPECT_EQ("21", fields[0]->metadata()->Get(pfid).ValueUnsafe());
373+
ASSERT_EQ(arrow::Type::EXTENSION, fields[0]->type()->id());
374+
375+
const auto& extension = static_cast<const arrow::ExtensionType&>(*fields[0]->type());
376+
EXPECT_EQ("arrow.parquet.variant", extension.extension_name());
377+
ASSERT_EQ(arrow::Type::STRUCT, extension.storage_type()->id());
378+
ASSERT_EQ(2, extension.storage_type()->num_fields());
379+
EXPECT_EQ("metadata", extension.storage_type()->field(0)->name());
380+
EXPECT_EQ(arrow::Type::BINARY, extension.storage_type()->field(0)->type()->id());
381+
EXPECT_FALSE(extension.storage_type()->field(0)->nullable());
382+
EXPECT_FALSE(extension.storage_type()->field(0)->HasMetadata());
383+
EXPECT_EQ("value", extension.storage_type()->field(1)->name());
384+
EXPECT_EQ(arrow::Type::BINARY, extension.storage_type()->field(1)->type()->id());
385+
EXPECT_FALSE(extension.storage_type()->field(1)->nullable());
386+
EXPECT_FALSE(extension.storage_type()->field(1)->HasMetadata());
387+
}
388+
389+
TEST(ArrowSchemaUtilTest, test_nested_variant_fields) {
390+
const std::string schema_json = R"({
391+
"type": "struct",
392+
"fields": [
393+
{
394+
"id": 10,
395+
"name": "info",
396+
"required": false,
397+
"type": {
398+
"type": "struct",
399+
"fields": [
400+
{"id": 11, "name": "payload", "required": false, "type": "variant"}
401+
]
402+
}
403+
},
404+
{
405+
"id": 20,
406+
"name": "events",
407+
"required": false,
408+
"type": {
409+
"type": "list",
410+
"element-id": 21,
411+
"element": "variant",
412+
"element-required": false
413+
}
414+
},
415+
{
416+
"id": 30,
417+
"name": "attrs",
418+
"required": false,
419+
"type": {
420+
"type": "map",
421+
"key-id": 31,
422+
"key": "string",
423+
"value-id": 32,
424+
"value": "variant",
425+
"value-required": false
426+
}
427+
}
428+
]
429+
})";
430+
std::unique_ptr<Schema> schema = SchemaParser::from_json(schema_json);
431+
432+
std::vector<std::shared_ptr<arrow::Field>> fields;
433+
Status status = IcebergArrowSchemaConvertor(*schema, "UTC").convert_fields(fields);
434+
ASSERT_TRUE(status.ok()) << status;
435+
ASSERT_EQ(3, fields.size());
436+
437+
const auto& info = assert_cast<const arrow::StructType&>(*fields[0]->type());
438+
ASSERT_EQ(arrow::Type::EXTENSION, info.field(0)->type()->id());
439+
EXPECT_EQ("11", info.field(0)->metadata()->Get(pfid).ValueUnsafe());
440+
441+
const auto& events = assert_cast<const arrow::ListType&>(*fields[1]->type());
442+
ASSERT_EQ(arrow::Type::EXTENSION, events.value_type()->id());
443+
EXPECT_EQ("21", events.value_field()->metadata()->Get(pfid).ValueUnsafe());
444+
445+
const auto& attrs = assert_cast<const arrow::MapType&>(*fields[2]->type());
446+
ASSERT_EQ(arrow::Type::EXTENSION, attrs.item_type()->id());
447+
EXPECT_EQ("32", attrs.item_field()->metadata()->Get(pfid).ValueUnsafe());
448+
449+
for (const auto& variant_type :
450+
{info.field(0)->type(), events.value_type(), attrs.item_type()}) {
451+
const auto& extension = static_cast<const arrow::ExtensionType&>(*variant_type);
452+
EXPECT_EQ("arrow.parquet.variant", extension.extension_name());
453+
EXPECT_EQ("struct<metadata: binary not null, value: binary not null>",
454+
extension.storage_type()->ToString());
455+
}
456+
}
457+
361458
TEST(ArrowSchemaUtilTest, test_parquet_filed_id) {
362459
std::string test_dir = "ut_dir/test_parquet_filed_id";
363460
Status st;

‎be/test/format/table/iceberg/schema_parser_test.cpp‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,18 @@ const std::string valid_struct_json = R"({
4949
]
5050
})";
5151

52+
const std::string variant_struct_json = R"({
53+
"type": "struct",
54+
"fields": [
55+
{
56+
"id": 30,
57+
"name": "payload",
58+
"type": "variant",
59+
"required": false
60+
}
61+
]
62+
})";
63+
5264
const std::string invalid_json = R"({
5365
"type": "struct",
5466
"fields": [
@@ -202,6 +214,13 @@ TEST(SchemaParserTest, parse_valid_struct) {
202214
EXPECT_EQ(age_field->field_type()->as_primitive_type()->to_string(), "int");
203215
}
204216

217+
TEST(SchemaParserTest, parse_variant) {
218+
std::unique_ptr<Schema> schema = SchemaParser::from_json(variant_struct_json);
219+
ASSERT_NE(schema, nullptr);
220+
ASSERT_NE(schema->find_field(30), nullptr);
221+
EXPECT_EQ(schema->find_field(30)->field_type()->type_id(), TypeID::VARIANT);
222+
}
223+
205224
//TEST(SchemaParserTest, parse_invalid_json) {
206225
// EXPECT_THROW(SchemaParser::from_json(invalid_json), doris::Exception);
207226
//}

‎be/test/format/table/iceberg/type_test.cpp‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,12 @@ TEST(BinaryTypeTest, to_string) {
4040
EXPECT_EQ(binary_type.type_id(), TypeID::BINARY);
4141
}
4242

43+
TEST(VariantTypeTest, to_string) {
44+
VariantType variant_type;
45+
EXPECT_EQ(variant_type.to_string(), "variant");
46+
EXPECT_EQ(variant_type.type_id(), TypeID::VARIANT);
47+
}
48+
4349
TEST(FixedTypeTest, to_string) {
4450
FixedType fixed_type(16);
4551
EXPECT_EQ(fixed_type.to_string(), "fixed[16]");
@@ -146,6 +152,7 @@ TEST(PrimitiveTypesTest, from_primitive_string) {
146152
EXPECT_EQ(Types::from_primitive_string("string")->type_id(), TypeID::STRING);
147153
EXPECT_EQ(Types::from_primitive_string("uuid")->type_id(), TypeID::UUID);
148154
EXPECT_EQ(Types::from_primitive_string("binary")->type_id(), TypeID::BINARY);
155+
EXPECT_EQ(Types::from_primitive_string("variant")->type_id(), TypeID::VARIANT);
149156
EXPECT_EQ(Types::from_primitive_string("fixed[16]")->type_id(), TypeID::FIXED);
150157
EXPECT_EQ(Types::from_primitive_string("decimal(10, 2)")->type_id(), TypeID::DECIMAL);
151158
}

0 commit comments

Comments
 (0)