Skip to content

Commit 2faf819

Browse files
authored
[test](iceberg) Add partition evolution schema coverage (#65870)
### What problem does this PR solve? Related PR: #65502 Problem Summary: #65502 already fixed the Iceberg scan correctness issue by sending the complete current table schema to readers. Although that change was introduced for equality-delete dependencies, the same all-column schema invariant is also required for partition evolution: a column classified as an identity partition column by a newer partition spec can still be stored physically in files written by an older spec. This follow-up does not change scan behavior. It documents the partition-evolution invariant, extracts the existing schema initialization into a testable helper, and adds a focused FE unit test verifying that a non-projected evolved partition column remains in the reader schema. ### Release note None ### Check List (For Author) - Test - [ ] Regression test - [x] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason - Behavior changed: - [x] No. - [ ] Yes. - Does this need documentation? - [x] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
1 parent 0127314 commit 2faf819

2 files changed

Lines changed: 39 additions & 1 deletion

File tree

‎fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/source/IcebergScanNode.java‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -508,11 +508,16 @@ public void createScanRangeLocations() throws UserException {
508508
super.createScanRangeLocations();
509509
enableCurrentIcebergScanSemantics();
510510
// Extract name mapping from Iceberg table properties
511-
Optional<Map<Integer, List<String>>> nameMapping = extractNameMapping();
511+
initializeIcebergSchemaInfo(extractNameMapping());
512+
}
512513

514+
@VisibleForTesting
515+
void initializeIcebergSchemaInfo(Optional<Map<Integer, List<String>>> nameMapping) throws UserException {
513516
// Equality-delete keys are hidden scan dependencies and need not appear in the query
514517
// projection. Both scanners need the complete current schema to resolve field ids,
515518
// historical names, types, and initial defaults when an old data file lacks such a key.
519+
// An identity partition column can also be a physical field in files written by an older
520+
// partition spec, so preserving the complete schema is required for partition evolution.
516521
ExternalUtil.initSchemaInfoForAllColumn(params, -1L, source.getTargetTable().getColumns(),
517522
nameMapping.orElse(Collections.emptyMap()), nameMapping.isPresent(),
518523
getBase64EncodedInitialDefaultsForScan());

‎fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/source/IcebergScanNodeTest.java‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -138,6 +138,12 @@ int enableAndGetIcebergScanSemanticsVersion() {
138138
enableCurrentIcebergScanSemantics();
139139
return params.getIcebergScanSemanticsVersion();
140140
}
141+
142+
TFileScanRangeParams initializeAndGetIcebergSchemaInfo() throws UserException {
143+
params = new TFileScanRangeParams();
144+
initializeIcebergSchemaInfo(Optional.empty());
145+
return params;
146+
}
141147
}
142148

143149
@Test
@@ -148,6 +154,33 @@ public void testEmitsCurrentIcebergScanSemanticsCapability() {
148154
node.enableAndGetIcebergScanSemanticsVersion());
149155
}
150156

157+
@Test
158+
public void testPartitionEvolutionKeepsNonFileSlotInReaderSchema() throws Exception {
159+
Column evolvedIdentityColumn = new Column("int_col", Type.BIGINT, true);
160+
evolvedIdentityColumn.setUniqueId(1);
161+
Column projectedColumn = new Column("payload", Type.STRING, true);
162+
projectedColumn.setUniqueId(2);
163+
164+
IcebergExternalTable targetTable = Mockito.mock(IcebergExternalTable.class);
165+
Mockito.when(targetTable.getColumns()).thenReturn(
166+
List.of(evolvedIdentityColumn, projectedColumn));
167+
IcebergSource source = Mockito.mock(IcebergSource.class);
168+
Mockito.when(source.getTargetTable()).thenReturn(targetTable);
169+
170+
TestIcebergScanNode node = Mockito.spy(new TestIcebergScanNode(new SessionVariable()));
171+
node.addSlot(1, projectedColumn);
172+
setIcebergSource(node, source);
173+
Mockito.doReturn(Collections.emptyMap()).when(node).getBase64EncodedInitialDefaultsForScan();
174+
175+
TFileScanRangeParams scanParams = node.initializeAndGetIcebergSchemaInfo();
176+
177+
Assert.assertEquals(2, scanParams.getHistorySchemaInfo().get(0).getRootField().getFieldsSize());
178+
Assert.assertEquals("int_col", scanParams.getHistorySchemaInfo().get(0).getRootField()
179+
.getFields().get(0).getFieldPtr().getName());
180+
Assert.assertEquals("payload", scanParams.getHistorySchemaInfo().get(0).getRootField()
181+
.getFields().get(1).getFieldPtr().getName());
182+
}
183+
151184
@Test
152185
public void testExtractNameMappingDistinguishesAbsentAndEmpty() throws Exception {
153186
TestIcebergScanNode node = new TestIcebergScanNode(new SessionVariable());

0 commit comments

Comments
 (0)