You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 2937f34
Browse filesBrowse the repository at this point in the historyBrowse files
[fix](fe) Bind Iceberg branch writes to the table's current schema
### What problem does this PR solve?
Issue Number: None
Related PR: #66538, #65851
Problem Summary: A write to an Iceberg branch (INSERT [OVERWRITE] INTO
t@branch(b)) pins the schema of the branch head snapshot as its writer
schema, while a read of the branch uses the table-current schema. Iceberg
validates a branch commit against the table-current schema and stamps
that schema on the new snapshot, Spark binds branch writes to the current
schema, and the Iceberg docs ("Schema selection with branches and tags")
say writes to a branch use the table schema. So once the table schema
evolves, a write to an older branch rejects the current column names and
accepts the old ones, writes the branch head's write-default instead of
the current one, and fails outright when the current partition spec has a
source column the branch head schema does not have. Master also needs an
extra check (validateBranchWriterSchema) because the files it writes carry
a different schema than the one Iceberg records for the new snapshot.
branch-4.1 and branch-4.2 (#66538) bind branch writes to table.schema().
This change pins table.schema() for branch writes, keeps the check that
the target is an existing branch and not a tag, and fences the commit on
the table-current schema id for branch writes as for writes to main. The
branch head checks are removed.
### Release note
Fix: Writes to an Iceberg branch now use the table's current schema
(column names, write-defaults and partition spec), as Spark does, instead
of the schema of the branch's head snapshot.
### Check List (For Author)
- Test: Regression test / Unit Test
- test_iceberg_initial_defaults (a branch write gets the current
write-default 36), write/test_iceberg_write_evolution_refs,
branch_tag/iceberg_branch_tag_schema_change_extended and
test_iceberg_schema_ref_actions_matrix (an older branch takes the
current column names and rejects the old ones) now expect what
branch-4.2 expects. They pass locally; without this change each of
them fails at the new expectation
- IcebergWritePlanProviderTest: a branch created before a rename writes
the new name; the branch write uses the current write-defaults, binds
a current required column without a default, and binds the current
partition spec whose source column the branch head lacks; a schema
change after pinning fails the commit; a tag or a missing ref is
rejected
- Behavior changed: Yes. See the release note.
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWritePlanProvider.java
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWriteSchemaContext.java
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergWritePlanProviderTest.java
+87-31Lines changed: 87 additions & 31 deletions
Original file line number
Diff line number
Diff line change
@@ -101,6 +101,7 @@
101
101
importjava.util.Optional;
102
102
importjava.util.Set;
103
103
importjava.util.UUID;
104
+
importjava.util.stream.Collectors;
104
105
105
106
/**
106
107
* Pins {@link IcebergWritePlanProvider#planWrite} for INSERT/OVERWRITE against legacy
@@ -559,7 +560,9 @@ public void variantWriteDefaultsMustBeNull() {
559
560
}
560
561
561
562
@Test
562
-
publicvoidbranchWritePinsBranchHeadSchema() {
563
+
publicvoidbranchWriteUsesTableCurrentSchema() {
564
+
// A branch only selects the parent snapshot. Iceberg stamps a branch commit with the table-current
565
+
// schema and Spark binds branch writes to it, so a branch created before a rename writes the new name.
563
566
InMemoryCatalogcatalog = freshCatalog();
564
567
Tabletable = unpartitionedUnsortedTable(catalog);
565
568
table.newAppend().commit();
@@ -569,14 +572,14 @@ public void branchWritePinsBranchHeadSchema() {
Copy file name to clipboardExpand all lines: regression-test/suites/external_table_p0/iceberg/branch_tag/iceberg_branch_tag_schema_change_extended.groovy
0 commit comments