You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit a2a6965
Browse filesBrowse the repository at this point in the historyBrowse files
[fix](iceberg) Write v1 tables whose partition spec has void fields
### What problem does this PR solve?
Issue Number: None
Related PR: #69229
Problem Summary:
Dropping a partition field from a format-v1 Iceberg table leaves a void
field in the spec, and the field's source column may be dropped later.
Spark writes such tables (it only reads a partitioned spec whose void
field lost its column), but Doris rejected every write before any file
was written:
- When every field of the current spec is void, bindPartitionSpec pinned
an unpartitioned stand-in (isPartitioned() is false for an all-void
spec) whose JSON never matched the table's spec, so the commit-time
check failed with "pinned spec N is not available with the same
definition; retry the statement".
- When the source column of a void field was dropped,
validateWriterMetadataSources failed with "partition field ...
references source field ... outside pinned schema", and BE
VIcebergTableWriter would look up the missing column as well.
This commit binds every spec that has fields as it is, skips void fields
in the source check, in the INSERT routing and in the MERGE insert
partitioning, and makes BE fill NULL for a void field without resolving
its source. An all-void spec is still written as unpartitioned and
committed under the table's spec, as Spark does. The same tables can
also take DELETE, UPDATE and MERGE after an upgrade to format v2.
### Release note
Doris can write format-v1 Iceberg tables whose partition spec contains
void fields left by dropped partition fields, also after their source
columns were dropped.
### Check List (For Author)
- Test: Unit Test (IcebergWritePlanProviderTest,
IcebergConnectorTransactionTest, VIcebergTableWriterTest); Regression
test (test_iceberg_write_v1_void_partition); Manual test (INSERT,
INSERT OVERWRITE, and DELETE / UPDATE / MERGE after a v2 upgrade on a
local Iceberg catalog)
- Behavior changed: No
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWritePlanProvider.java
+8Lines changed: 8 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -621,6 +621,10 @@ public ConnectorWritePartitionSpec getWritePartitioning(ConnectorSession session
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWriteSchemaContext.java
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorTransactionTest.java
+42Lines changed: 42 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -941,6 +941,48 @@ public void overwriteEmptyUnpartitionedRejectsDataCommittedBetweenScanAndBegin()
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergWritePlanProviderTest.java
+62Lines changed: 62 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -504,6 +504,68 @@ public void writeSchemaAllowsNestedPrimitivePartitionSource() {
0 commit comments