Skip to content

Commit a2a6965

Browse files
morningmanclaude
andcommitted
[fix](iceberg) Write v1 tables whose partition spec has void fields
### What problem does this PR solve? Issue Number: None Related PR: #69229 Problem Summary: Dropping a partition field from a format-v1 Iceberg table leaves a void field in the spec, and the field's source column may be dropped later. Spark writes such tables (it only reads a partitioned spec whose void field lost its column), but Doris rejected every write before any file was written: - When every field of the current spec is void, bindPartitionSpec pinned an unpartitioned stand-in (isPartitioned() is false for an all-void spec) whose JSON never matched the table's spec, so the commit-time check failed with "pinned spec N is not available with the same definition; retry the statement". - When the source column of a void field was dropped, validateWriterMetadataSources failed with "partition field ... references source field ... outside pinned schema", and BE VIcebergTableWriter would look up the missing column as well. This commit binds every spec that has fields as it is, skips void fields in the source check, in the INSERT routing and in the MERGE insert partitioning, and makes BE fill NULL for a void field without resolving its source. An all-void spec is still written as unpartitioned and committed under the table's spec, as Spark does. The same tables can also take DELETE, UPDATE and MERGE after an upgrade to format v2. ### Release note Doris can write format-v1 Iceberg tables whose partition spec contains void fields left by dropped partition fields, also after their source columns were dropped. ### Check List (For Author) - Test: Unit Test (IcebergWritePlanProviderTest, IcebergConnectorTransactionTest, VIcebergTableWriterTest); Regression test (test_iceberg_write_v1_void_partition); Manual test (INSERT, INSERT OVERWRITE, and DELETE / UPDATE / MERGE after a v2 upgrade on a local Iceberg catalog) - Behavior changed: No - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent 07887d2 commit a2a6965

8 files changed

Lines changed: 438 additions & 1 deletion

File tree

‎be/src/exec/sink/writer/iceberg/viceberg_table_writer.cpp‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@
2929
#include "core/column/column_struct.h"
3030
#include "core/column/column_vector.h"
3131
#include "core/data_type/data_type_nullable.h"
32+
#include "core/data_type/data_type_string.h"
3233
#include "core/data_type/data_type_struct.h"
3334
#include "core/data_type_serde/data_type_serde.h"
3435
#include "exec/sink/writer/external_file_writer_compatibility.h"
@@ -132,6 +133,14 @@ VIcebergTableWriter::_to_iceberg_partition_columns() {
132133
std::vector<IcebergPartitionColumn> partition_columns;
133134

134135
for (const auto& partition_field : _partition_spec->fields()) {
136+
if (partition_field.transform() == "void") {
137+
// A void field is NULL for every row. A v1 table keeps one for each dropped partition
138+
// field and may drop its source column later, so the source is not resolved.
139+
partition_columns.emplace_back(partition_field, TYPE_NULL, -1, std::vector<size_t> {},
140+
std::make_unique<VoidPartitionColumnTransform>(
141+
std::make_shared<DataTypeString>()));
142+
continue;
143+
}
135144
const auto* field_path = _schema->find_field_path(partition_field.source_id());
136145
if (field_path == nullptr || field_path->empty()) {
137146
throw Exception(
@@ -526,6 +535,15 @@ Status VIcebergTableWriter::_write_prepared_block(Block& output_block) {
526535
auto col = ColumnConst::create(std::move(data_col), output_block.rows());
527536
transformed_block.insert(
528537
{std::move(col), result_type, iceberg_partition_columns.field().name()});
538+
} else if (iceberg_partition_columns.source_idx() < 0) {
539+
// A void field has no source column; materialize its NULLs, since partition values
540+
// are read row by row.
541+
auto result_type = make_nullable(
542+
iceberg_partition_columns.partition_column_transform().get_result_type());
543+
auto data_col = result_type->create_column();
544+
data_col->insert_many_defaults(output_block.rows());
545+
transformed_block.insert({std::move(data_col), result_type,
546+
iceberg_partition_columns.field().name()});
529547
} else {
530548
Block source_block;
531549
source_block.insert(

‎be/test/exec/sink/writer/iceberg/viceberg_table_writer_test.cpp‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -284,6 +284,50 @@ TEST(VIcebergTableWriterTest, RejectMissingPartitionSource) {
284284
}
285285
}
286286

287+
TEST(VIcebergTableWriterTest, VoidPartitionFieldNeedsNoSourceColumn) {
288+
// A v1 table keeps void(region) for its dropped partition field after region is dropped too.
289+
std::vector<iceberg::NestedField> columns;
290+
columns.emplace_back(true, 1, "note", std::make_unique<iceberg::IntegerType>(), std::nullopt);
291+
columns.emplace_back(true, 3, "category", std::make_unique<iceberg::IntegerType>(),
292+
std::nullopt);
293+
auto schema = std::make_shared<iceberg::Schema>(std::move(columns));
294+
const std::string spec_json =
295+
R"({"spec-id":1,"fields":[{"name":"region","transform":"void","source-id":2,)"
296+
R"("field-id":1000},{"name":"category","transform":"identity","source-id":3,)"
297+
R"("field-id":1001}]})";
298+
auto int_type = make_nullable(std::make_shared<DataTypeInt32>());
299+
VExprContextSPtrs output_exprs {
300+
VExprContext::create_shared(VSlotRef::create_shared(0, 0, -1, int_type, "note")),
301+
VExprContext::create_shared(VSlotRef::create_shared(1, 1, -1, int_type, "category"))};
302+
303+
TIcebergTableSink iceberg_sink;
304+
TDataSink data_sink;
305+
data_sink.__set_iceberg_table_sink(iceberg_sink);
306+
VIcebergTableWriter writer(data_sink, output_exprs, nullptr, nullptr);
307+
writer._schema = schema;
308+
writer._partition_spec = iceberg::PartitionSpecParser::from_json(schema, spec_json);
309+
writer._iceberg_partition_columns = writer._to_iceberg_partition_columns();
310+
ASSERT_EQ(writer._iceberg_partition_columns.size(), 2);
311+
EXPECT_EQ(writer._iceberg_partition_columns[0].source_idx(), -1);
312+
EXPECT_EQ(writer._iceberg_partition_columns[1].source_idx(), 1);
313+
314+
IcebergPartitionData partition({std::any(), Int32(7)});
315+
EXPECT_EQ("region=null/category=7", writer._partition_to_path(partition));
316+
EXPECT_EQ(std::vector<std::string>({"null", "7"}), writer._partition_values(partition));
317+
auto recording_writer = std::make_shared<RecordingPartitionWriter>();
318+
writer._partitions_to_writers[writer._partition_to_writer_key(partition)] = recording_writer;
319+
auto note = int_type->create_column();
320+
note->insert_many_defaults(2);
321+
auto category = int_type->create_column();
322+
category->insert(Field::create_field<TYPE_INT>(7));
323+
category->insert(Field::create_field<TYPE_INT>(7));
324+
Block block({ColumnWithTypeAndName(note->get_ptr(), int_type, "note"),
325+
ColumnWithTypeAndName(category->get_ptr(), int_type, "category")});
326+
// Both rows must reach the writer of the partition whose void value is NULL.
327+
ASSERT_TRUE(writer._write_prepared_block(block).ok());
328+
EXPECT_EQ(2, recording_writer->rows);
329+
}
330+
287331
TEST(VIcebergTableWriterTest, ResolvesNestedPartitionSource) {
288332
std::vector<iceberg::NestedField> children;
289333
children.emplace_back(true, 2, "part", std::make_unique<iceberg::IntegerType>(), std::nullopt);

‎fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWritePlanProvider.java‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -621,6 +621,10 @@ public ConnectorWritePartitionSpec getWritePartitioning(ConnectorSession session
621621
? active.get().getSchema() : table.schema();
622622
List<ConnectorWritePartitionField> fields = new ArrayList<>();
623623
for (PartitionField field : spec.fields()) {
624+
if (field.transform().isVoid()) {
625+
// NULL for every row, and a v1 table may have dropped its source column.
626+
continue;
627+
}
624628
// Bind a nested source to its top-level slot. fe-core recovers child indexes from the stable Iceberg
625629
// field ids already carried by the Doris Column tree, so the public connector SPI stays unchanged.
626630
String sourceColumnName = findPartitionSourceColumnName(schema, field.sourceId());
@@ -664,6 +668,10 @@ public ConnectorWriteDistribution getWriteDistribution(ConnectorSession session,
664668
List<String> routeColumns = new ArrayList<>();
665669
List<String> transforms = new ArrayList<>();
666670
for (PartitionField field : spec.fields()) {
671+
if (field.transform().isVoid()) {
672+
// NULL for every row, and a v1 table may have dropped its source column.
673+
continue;
674+
}
667675
String sourceColumnName = findTopLevelColumnName(schema, field.sourceId());
668676
if (sourceColumnName == null) {
669677
return null;

‎fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWriteSchemaContext.java‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -222,7 +222,9 @@ private static ConnectorType stringWriteType(Type type, ConnectorType mapped) {
222222

223223
private static PartitionSpec bindPartitionSpec(
224224
PartitionSpec partitionSpec, Schema schema, String tableName) {
225-
if (!partitionSpec.isPartitioned()) {
225+
// A v1 table replaces each dropped partition field with a void one, so a spec that is not partitioned can
226+
// still have fields. Bind those as well: the pinned definition must stay equal to the table's spec.
227+
if (partitionSpec.fields().isEmpty()) {
226228
return PartitionSpec.builderFor(schema).withSpecId(partitionSpec.specId()).build();
227229
}
228230
try {
@@ -250,6 +252,10 @@ private static SortOrder bindSortOrder(SortOrder sortOrder, Schema schema, Strin
250252
private static void validateWriterMetadataSources(
251253
Schema schema, PartitionSpec partitionSpec, SortOrder sortOrder, String tableName) {
252254
for (PartitionField field : partitionSpec.fields()) {
255+
// A void field writes NULL without reading its source, which a v1 table may drop after the field.
256+
if (field.transform().isVoid()) {
257+
continue;
258+
}
253259
// Iceberg permits a nested primitive field as a partition source; field IDs are schema-wide.
254260
if (schema.findField(field.sourceId()) == null) {
255261
throw new DorisConnectorException("Iceberg partition field " + field.fieldId()

‎fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorTransactionTest.java‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -941,6 +941,48 @@ public void overwriteEmptyUnpartitionedRejectsDataCommittedBetweenScanAndBegin()
941941
Assertions.assertThrows(DorisConnectorException.class, txn::commit);
942942
}
943943

944+
@Test
945+
public void insertCommitsNullForV1VoidPartitionFields() {
946+
// A v1 table replaces a dropped partition field with void(region) and may drop region afterwards; Spark
947+
// keeps writing it. MUTATION: pinning an unpartitioned stand-in for the void-only spec -> commit rejects the
948+
// pinned spec; checking the source of the void field -> create rejects the dropped region -> red.
949+
for (boolean identityCategory : new boolean[] {false, true}) {
950+
InMemoryCatalog catalog = freshCatalog();
951+
TableIdentifier id = TableIdentifier.of("db1", "t1");
952+
Schema schema = new Schema(
953+
Types.NestedField.required(1, "id", Types.IntegerType.get()),
954+
Types.NestedField.optional(2, "region", Types.StringType.get()),
955+
Types.NestedField.optional(3, "category", Types.StringType.get()));
956+
PartitionSpec.Builder spec = PartitionSpec.builderFor(schema).identity("region");
957+
if (identityCategory) {
958+
spec.identity("category");
959+
}
960+
Table table = catalog.createTable(id, schema, spec.build(), props("format-version", "1"));
961+
table.updateSpec().removeField("region").commit();
962+
table.updateSchema().deleteColumn("region").commit();
963+
964+
IcebergWriteSchemaContext schemaContext =
965+
IcebergWriteSchemaContext.create(table, table.name(), Optional.empty(), false, false);
966+
IcebergConnectorTransaction txn = txnFor(opsReturning(table), new RecordingConnectorContext());
967+
txn.beginWrite(SESSION, "db1", "t1", new IcebergWriteContext(
968+
WriteOperation.INSERT, false, Collections.emptyMap(), Optional.empty(), -1L, schemaContext));
969+
// BE writes a void-only spec as unpartitioned and reports NULL for a void field of a partitioned one.
970+
txn.addCommitData(commitBytes(identityCategory
971+
? dataFileItem("s3://b/db1/t1/region=null/category=x/f1.parquet", 2L, 1024L,
972+
Arrays.asList("null", "x"))
973+
: dataFileItem("s3://b/db1/t1/f1.parquet", 2L, 1024L)));
974+
txn.commit();
975+
976+
List<DataFile> files = currentDataFiles(catalog.loadTable(id));
977+
Assertions.assertEquals(1, files.size());
978+
Assertions.assertEquals(table.spec().specId(), files.get(0).specId());
979+
Assertions.assertNull(files.get(0).partition().get(0, Object.class));
980+
if (identityCategory) {
981+
Assertions.assertEquals("x", files.get(0).partition().get(1, String.class));
982+
}
983+
}
984+
}
985+
944986
@Test
945987
public void overwriteStaticPartitionUsesRowFilter() {
946988
InMemoryCatalog catalog = freshCatalog();

‎fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergWritePlanProviderTest.java‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -504,6 +504,68 @@ public void writeSchemaAllowsNestedPrimitivePartitionSource() {
504504
table, "db1.nested_partition", Optional.empty(), false, false));
505505
}
506506

507+
/** A v1 table whose region partition field was dropped, which Iceberg keeps as void(region). */
508+
private static Table v1TableWithVoidRegion(InMemoryCatalog catalog, String name, boolean identityCategory,
509+
boolean dropRegion) {
510+
Schema schema = new Schema(
511+
Types.NestedField.required(1, "id", Types.IntegerType.get()),
512+
Types.NestedField.optional(2, "region", Types.StringType.get()),
513+
Types.NestedField.optional(3, "category", Types.StringType.get()));
514+
PartitionSpec.Builder spec = PartitionSpec.builderFor(schema).identity("region");
515+
if (identityCategory) {
516+
spec.identity("category");
517+
}
518+
Table table = catalog.createTable(TableIdentifier.of("db1", name), schema, spec.build(),
519+
Collections.singletonMap("format-version", "1"));
520+
table.updateSpec().removeField("region").commit();
521+
if (dropRegion) {
522+
table.updateSchema().deleteColumn("region").commit();
523+
}
524+
return table;
525+
}
526+
527+
@Test
528+
public void writeSchemaPinsV1VoidPartitionFields() {
529+
// Spark keeps writing a v1 table after a partition field becomes void, also once its source is dropped.
530+
// MUTATION: pinning an unpartitioned stand-in for a void-only spec -> the revalidation rejects it; checking
531+
// the source column of a void field -> create rejects the dropped region -> red.
532+
InMemoryCatalog catalog = freshCatalog();
533+
List<Table> tables = Arrays.asList(
534+
v1TableWithVoidRegion(catalog, "void_only", false, false),
535+
v1TableWithVoidRegion(catalog, "void_only_dropped", false, true),
536+
v1TableWithVoidRegion(catalog, "void_and_identity_dropped", true, true));
537+
for (Table table : tables) {
538+
IcebergWriteSchemaContext writeContext = IcebergWriteSchemaContext.create(
539+
table, table.name(), Optional.empty(), false, false);
540+
541+
Assertions.assertEquals(PartitionSpecParser.toJson(table.spec()), writeContext.getPartitionSpecJson(),
542+
table.name());
543+
Assertions.assertDoesNotThrow(() -> writeContext.validateCurrentSchema(table, true), table.name());
544+
}
545+
}
546+
547+
@Test
548+
public void writeRoutingSkipsV1VoidPartitionFields() {
549+
// A void field is NULL for every row, so routing by the other fields still keeps one partition on one
550+
// writer, and the void field's source may be gone. MUTATION: routing the void field -> no route column for
551+
// the dropped region -> random write -> red.
552+
Table table = v1TableWithVoidRegion(freshCatalog(), "void_and_identity", true, true);
553+
IcebergTableHandle handle = new IcebergTableHandle("db1", "void_and_identity");
554+
555+
ConnectorWriteDistribution distribution = providerFor(table, contextWithStorage())
556+
.getWriteDistribution(sessionFor(table, contextWithStorage()), handle);
557+
Assertions.assertNotNull(distribution);
558+
Assertions.assertEquals(Collections.singletonList("category"), distribution.getRouteColumns());
559+
Assertions.assertEquals(Collections.singletonMap("transforms", "identity"),
560+
distribution.getPartitionFunctionOptions());
561+
562+
ConnectorWritePartitionSpec partitioning = providerFor(table, contextWithStorage())
563+
.getWritePartitioning(sessionFor(table, contextWithStorage()), handle);
564+
Assertions.assertEquals(table.spec().specId(), partitioning.getSpecId());
565+
Assertions.assertEquals(1, partitioning.getFields().size());
566+
Assertions.assertEquals("category", partitioning.getFields().get(0).getSourceColumnName());
567+
}
568+
507569
@Test
508570
public void branchWriteRejectsCurrentRequiredFieldAbsentWithoutDefault() {
509571
InMemoryCatalog catalog = freshCatalog();
Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
-- This file is automatically generated. You should know what you did if you want to edit this
2+
-- !void_only_insert --
3+
1 a 10
4+
2 b 20
5+
3 a 30
6+
4 c 40
7+
5 d 50
8+
9+
-- !void_only_overwrite --
10+
1 a 11
11+
2 b 21
12+
3 a 31
13+
4 c 41
14+
5 d 51
15+
16+
-- !void_only_overwrite_specs --
17+
1
18+
19+
-- !void_only_dml --
20+
2 b 21
21+
3 a 131
22+
4 c 400
23+
5 d 51
24+
6 f 60
25+
26+
-- !void_only_delete_specs --
27+
1
28+
29+
-- !void_only_dropped_source_insert --
30+
1 10
31+
2 20
32+
3 30
33+
4 40
34+
35+
-- !void_only_dropped_source_overwrite --
36+
1 11
37+
2 21
38+
3 31
39+
4 41
40+
41+
-- !void_only_dropped_source_dml --
42+
2 21
43+
3 131
44+
4 400
45+
6 60
46+
47+
-- !void_and_identity_insert --
48+
1 x 10
49+
2 y 20
50+
3 x 30
51+
4 z 40
52+
5 z 50
53+
54+
-- !void_and_identity_overwrite --
55+
1 x 11
56+
2 y 21
57+
3 x 31
58+
4 z 41
59+
5 z 51
60+
61+
-- !void_and_identity_dml --
62+
2 y 200
63+
3 x 31
64+
4 z 141
65+
5 z 151
66+
6 w 60
67+
68+
-- !void_and_identity_delete_specs --
69+
1
70+

0 commit comments

Comments
 (0)