Skip to content

Commit cbf21fb

Browse files
morningmanclaude
andcommitted
[fix](iceberg) Relax a dropped equality-delete key only along retained compatible changes
### What problem does this PR solve? Issue Number: None Related PR: #69536 Problem Summary: A dropped Iceberg equality-delete key takes its definition from the scan's snapshot lineage, falling back to the whole schema list. The previous commit then replaced the key's type with the widest type it has in any retained schema, comparing types with TypeUtil.isPromotionAllowed. In Iceberg 1.12.0 that helper knows only INT to LONG, FLOAT to DOUBLE and decimal precision, so for a v3 table whose key was promoted from DATE to an unzoned TIMESTAMP the first retained type, DATE, won and replaced the TIMESTAMP definition the lineage had found. FileScannerV2 compares equality keys in the carrier's type, so it cast a noon TIMESTAMP tombstone to DATE and deleted the midnight row of that day. A schema-only DATE to TIMESTAMP_NS promotion of a key dropped before its delete committed kept DATE as well. The lineage also lost a schema-only change of a required key to optional: a pending delete and an append without the key both commit with the post-drop schema, so the lineage reaches the original REQUIRED definition, and both readers reject the appended file with a missing required field, where the lookup before this PR found OPTIONAL. Each field of the selected definition now starts from its own type and optionality and takes, from every retained schema, only a compatible relaxation: a promotion recognized by TypeUtil.isPromotionAllowed or the v3 promotion of DATE to an unzoned TIMESTAMP or TIMESTAMP_NS, and optional when any retained schema has the field optional. A key cannot be narrowed or switched to an unrelated type, and a key required in every retained schema stays required. ### Release note None ### Check List (For Author) - Test: Unit Test - IcebergScanPlanProviderTest: a v3 DATE key promoted to TIMESTAMP keeps the lineage's TIMESTAMP; a DATE key promoted to TIMESTAMP_NS without a snapshot and dropped before its delete commits widens to TIMESTAMP_NS; a key made optional without a snapshot and dropped before its delete commits stays optional; a key required in every schema stays required - fe-connector-iceberg module tests - Behavior changed: No (fixes a regression of this PR's previous commits) - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent 71f51ed commit cbf21fb

2 files changed

Lines changed: 154 additions & 30 deletions

File tree

‎fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java‎

Lines changed: 45 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -2284,11 +2284,12 @@ private Set<Integer> cachedApplicableEqualityDeleteFieldIds(Table table, TableSc
22842284
* schema, so a schema's position in the metadata list does not tell when it was last active, and the list may
22852285
* also hold schemas committed after a time-travel or branch target. The scan snapshot's parent chain is
22862286
* therefore searched first, so the definition active on that lineage wins; the whole metadata list, newest
2287-
* first, is the fallback for schema-only changes and expired ancestors. A dropped key nevertheless takes the
2288-
* widest type it has in any retained schema: a delete committed after a schema-only promotion holds values of
2289-
* the promoted type even when no snapshot records it. Current fields remain first, allowing a dropped/re-added
2290-
* name to resolve the projected field by name while a live historical equality key resolves by its stable
2291-
* field ID.</p>
2287+
* first, is the fallback for schema-only changes and expired ancestors. The selected definition keeps its name
2288+
* and initial default but takes the relaxations its field has in any retained schema
2289+
* ({@link #withRetainedRelaxations}): a delete committed after a schema-only promotion or required-to-optional
2290+
* change follows the relaxed definition even when no snapshot records it. Current fields remain first,
2291+
* allowing a dropped/re-added name to resolve the projected field by name while a live historical equality key
2292+
* resolves by its stable field ID.</p>
22922293
*/
22932294
@VisibleForTesting
22942295
static List<NestedField> schemaForPotentialEqualityDeletes(
@@ -2308,7 +2309,6 @@ static List<NestedField> schemaForPotentialEqualityDeletes(
23082309

23092310
List<NestedField> fields = new ArrayList<>(scanSchema.columns());
23102311
List<Schema> metadataSchemas = metadataSchemaHistory(table);
2311-
Map<Integer, Type> widestTypes = widestPromotedTypes(missing, metadataSchemas);
23122312
// Snapshots on one lineage mostly share a schema, so each schema is indexed at most once.
23132313
Set<Integer> lineageSchemaIds = new HashSet<>();
23142314
for (Snapshot snapshot = scanSnapshot; snapshot != null && !missing.isEmpty();
@@ -2322,10 +2322,10 @@ static List<NestedField> schemaForPotentialEqualityDeletes(
23222322
throw new IllegalStateException(
23232323
"Iceberg snapshot schema " + schemaId + " is absent from table metadata");
23242324
}
2325-
addHistoricalEqualityFields(fields, missing, lineageSchema, widestTypes);
2325+
addHistoricalEqualityFields(fields, missing, lineageSchema, metadataSchemas);
23262326
}
23272327
for (int index = metadataSchemas.size() - 1; index >= 0 && !missing.isEmpty(); index--) {
2328-
addHistoricalEqualityFields(fields, missing, metadataSchemas.get(index), widestTypes);
2328+
addHistoricalEqualityFields(fields, missing, metadataSchemas.get(index), metadataSchemas);
23292329
}
23302330
if (!missing.isEmpty()) {
23312331
throw new IllegalStateException(
@@ -2342,39 +2342,54 @@ private static List<Schema> metadataSchemaHistory(Table table) {
23422342
}
23432343

23442344
/**
2345-
* Field IDs are never reused and a field's type only changes by promotion, which is monotonic, so the widest
2346-
* type a field has in any retained schema covers every file written for it.
2345+
* A writer commits with the schema that is current at commit, so a key dropped before its delete commits loses
2346+
* the schema-only changes made before the drop from every snapshot. Field IDs are never reused and a field
2347+
* only relaxes compatibly, by promotion or from required to optional, so each field of the selected definition
2348+
* takes the widest promotion of its type and the optionality it has in any retained schema; neither depends on
2349+
* the order of the schemas. A field required in every retained schema stays required.
23472350
*/
2348-
private static Map<Integer, Type> widestPromotedTypes(Set<Integer> fieldIds, List<Schema> schemas) {
2349-
Map<Integer, Type> widestTypes = new HashMap<>();
2350-
for (Schema schema : schemas) {
2351-
for (Integer fieldId : fieldIds) {
2352-
Type type = schema.findType(fieldId);
2353-
if (type == null) {
2351+
private static List<NestedField> withRetainedRelaxations(List<NestedField> fields, List<Schema> schemas) {
2352+
List<NestedField> relaxed = new ArrayList<>(fields.size());
2353+
for (NestedField field : fields) {
2354+
Type type = field.type();
2355+
boolean optional = field.isOptional();
2356+
for (Schema schema : schemas) {
2357+
NestedField retained = schema.findField(field.fieldId());
2358+
if (retained == null) {
23542359
continue;
23552360
}
2356-
Type widest = widestTypes.get(fieldId);
2357-
if (widest == null || TypeUtil.isPromotionAllowed(widest, type.asPrimitiveType())) {
2358-
widestTypes.put(fieldId, type);
2361+
optional |= retained.isOptional();
2362+
if (type.isPrimitiveType() && retained.type().isPrimitiveType()
2363+
&& isWideningPromotion(type.asPrimitiveType(), retained.type().asPrimitiveType())) {
2364+
type = retained.type();
23592365
}
23602366
}
2367+
if (type.isStructType()) {
2368+
type = Types.StructType.of(withRetainedRelaxations(type.asStructType().fields(), schemas));
2369+
}
2370+
relaxed.add(type.equals(field.type()) && optional == field.isOptional()
2371+
? field : Types.NestedField.from(field).ofType(type).isOptional(optional).build());
23612372
}
2362-
return widestTypes;
2373+
return relaxed;
23632374
}
23642375

2365-
private static List<NestedField> withWidestTypes(List<NestedField> fields, Map<Integer, Type> widestTypes) {
2366-
List<NestedField> widened = new ArrayList<>(fields.size());
2367-
for (NestedField field : fields) {
2368-
Type type = field.type().isStructType()
2369-
? Types.StructType.of(withWidestTypes(field.type().asStructType().fields(), widestTypes))
2370-
: widestTypes.getOrDefault(field.fieldId(), field.type());
2371-
widened.add(type.equals(field.type()) ? field : Types.NestedField.from(field).ofType(type).build());
2376+
/**
2377+
* {@link TypeUtil#isPromotionAllowed} plus the v3 promotion of DATE to an unzoned TIMESTAMP or TIMESTAMP_NS,
2378+
* which the SDK's schema update does not implement yet but the format allows.
2379+
*/
2380+
private static boolean isWideningPromotion(Type.PrimitiveType from, Type.PrimitiveType to) {
2381+
if (TypeUtil.isPromotionAllowed(from, to)) {
2382+
return true;
2383+
}
2384+
if (from.typeId() != Type.TypeID.DATE) {
2385+
return false;
23722386
}
2373-
return widened;
2387+
return to.typeId() == Type.TypeID.TIMESTAMP && !((Types.TimestampType) to).shouldAdjustToUTC()
2388+
|| to.typeId() == Type.TypeID.TIMESTAMP_NANO && !((Types.TimestampNanoType) to).shouldAdjustToUTC();
23742389
}
23752390

23762391
private static void addHistoricalEqualityFields(List<NestedField> fields, Set<Integer> missingFieldIds,
2377-
Schema historicalSchema, Map<Integer, Type> widestTypes) {
2392+
Schema historicalSchema, List<Schema> retainedSchemas) {
23782393
Map<Integer, NestedField> historicalFields = TypeUtil.indexById(historicalSchema.asStruct());
23792394
Set<Integer> selectedFieldIds = new HashSet<>();
23802395
for (Integer fieldId : missingFieldIds) {
@@ -2391,7 +2406,7 @@ private static void addHistoricalEqualityFields(List<NestedField> fields, Set<In
23912406
return;
23922407
}
23932408
Schema selectedSchema = TypeUtil.select(historicalSchema, selectedFieldIds);
2394-
mergeHistoricalEqualityFields(fields, withWidestTypes(selectedSchema.columns(), widestTypes));
2409+
mergeHistoricalEqualityFields(fields, withRetainedRelaxations(selectedSchema.columns(), retainedSchemas));
23952410
missingFieldIds.removeAll(selectedFieldIds);
23962411
}
23972412

‎fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java‎

Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2143,6 +2143,115 @@ public void equalityCarrierKeepsPromotedTypeOfKeyDroppedBeforeItsDeleteCommits()
21432143
Assertions.assertEquals(Types.LongType.get(), carrier.get(1).type());
21442144
}
21452145

2146+
@Test
2147+
public void equalityCarrierKeepsLineageTimestampOfPromotedDateKey() throws Exception {
2148+
// v3 promotes DATE to an unzoned TIMESTAMP, which TypeUtil.isPromotionAllowed does not know. The delete
2149+
// snapshot records the TIMESTAMP schema; a DATE carrier would truncate a noon tombstone to its date and
2150+
// delete the midnight row.
2151+
Table table = createTable("promoted_date_lineage_eq_key", new Schema(
2152+
Types.NestedField.required(1, "id", Types.IntegerType.get()),
2153+
Types.NestedField.optional(2, "k", Types.DateType.get())), PartitionSpec.unpartitioned(),
2154+
Collections.singletonMap(TableProperties.FORMAT_VERSION, "3"));
2155+
table.newAppend().appendFile(dataFile(table.spec(),
2156+
"s3://b/db/promoted_date_lineage_eq_key/f1.parquet", 1024, null, null)).commit();
2157+
publishSchema(table, new Schema(Types.NestedField.required(1, "id", Types.IntegerType.get()),
2158+
Types.NestedField.optional(2, "k", Types.TimestampType.withoutZone())), 2);
2159+
table.newRowDelta().addDeletes(equalityDeleteFile(
2160+
"s3://b/db/promoted_date_lineage_eq_key/eq.parquet", FileFormat.PARQUET, 2)).commit();
2161+
table.updateSchema().deleteColumn("k").commit();
2162+
Assertions.assertEquals(Types.TimestampType.withoutZone(),
2163+
table.schemas().get(table.currentSnapshot().schemaId()).findType(2));
2164+
2165+
List<NestedField> carrier = IcebergScanPlanProvider.schemaForPotentialEqualityDeletes(
2166+
table, table.newScan(), table.schema());
2167+
Assertions.assertEquals(2, carrier.size(), "the dropped equality key must stay resolvable: " + carrier);
2168+
Assertions.assertEquals(Types.TimestampType.withoutZone(), carrier.get(1).type());
2169+
}
2170+
2171+
@Test
2172+
public void equalityCarrierWidensDateKeyPromotedWithoutASnapshot() throws Exception {
2173+
// A writer started after a schema-only v3 promotion of k to TIMESTAMP_NS writes TIMESTAMP_NS tombstones,
2174+
// and k is dropped before its delete commits, so only the metadata list knows the promotion.
2175+
Table table = createTable("promoted_date_dropped_eq_key", new Schema(
2176+
Types.NestedField.required(1, "id", Types.IntegerType.get()),
2177+
Types.NestedField.optional(2, "k", Types.DateType.get())), PartitionSpec.unpartitioned(),
2178+
Collections.singletonMap(TableProperties.FORMAT_VERSION, "3"));
2179+
table.newAppend().appendFile(dataFile(table.spec(),
2180+
"s3://b/db/promoted_date_dropped_eq_key/f1.parquet", 1024, null, null)).commit();
2181+
publishSchema(table, new Schema(Types.NestedField.required(1, "id", Types.IntegerType.get()),
2182+
Types.NestedField.optional(2, "k", Types.TimestampNanoType.withoutZone())), 2);
2183+
RowDelta pendingDelete = table.newRowDelta().addDeletes(equalityDeleteFile(
2184+
"s3://b/db/promoted_date_dropped_eq_key/eq.parquet", FileFormat.PARQUET, 2));
2185+
table.updateSchema().deleteColumn("k").commit();
2186+
pendingDelete.commit();
2187+
Assertions.assertEquals(Types.DateType.get(), table.schemas()
2188+
.get(table.snapshot(table.currentSnapshot().parentId()).schemaId()).findType(2));
2189+
2190+
List<NestedField> carrier = IcebergScanPlanProvider.schemaForPotentialEqualityDeletes(
2191+
table, table.newScan(), table.schema());
2192+
Assertions.assertEquals(2, carrier.size(), "the dropped equality key must stay resolvable: " + carrier);
2193+
Assertions.assertEquals(Types.TimestampNanoType.withoutZone(), carrier.get(1).type());
2194+
}
2195+
2196+
@Test
2197+
public void equalityCarrierKeepsSchemaOnlyOptionalityOfDroppedKey() throws Exception {
2198+
// k becomes optional without a snapshot, and both a pending delete and an append without k commit after k
2199+
// is dropped, so every snapshot on the lineage records either no k or the required one. A required carrier
2200+
// would make both readers reject the appended file for a missing required field.
2201+
Table table = droppedRequiredKeyTable("optional_dropped_eq_key", true);
2202+
Assertions.assertTrue(table.schemas().get(table.snapshot(table.snapshot(
2203+
table.currentSnapshot().parentId()).parentId()).schemaId()).findField(2).isRequired());
2204+
2205+
List<NestedField> carrier = IcebergScanPlanProvider.schemaForPotentialEqualityDeletes(
2206+
table, table.newScan(), table.schema());
2207+
Assertions.assertEquals(2, carrier.size(), "the dropped equality key must stay resolvable: " + carrier);
2208+
Assertions.assertTrue(carrier.get(1).isOptional(), carrier.toString());
2209+
Assertions.assertEquals(Types.IntegerType.get(), carrier.get(1).type());
2210+
}
2211+
2212+
@Test
2213+
public void equalityCarrierKeepsAlwaysRequiredDroppedKeyRequired() throws Exception {
2214+
Table table = droppedRequiredKeyTable("required_dropped_eq_key", false);
2215+
2216+
List<NestedField> carrier = IcebergScanPlanProvider.schemaForPotentialEqualityDeletes(
2217+
table, table.newScan(), table.schema());
2218+
Assertions.assertEquals(2, carrier.size(), "the dropped equality key must stay resolvable: " + carrier);
2219+
Assertions.assertTrue(carrier.get(1).isRequired(), carrier.toString());
2220+
}
2221+
2222+
/**
2223+
* Appends a file with a required k, optionally makes k optional without a snapshot, then drops k while an
2224+
* equality delete on it is pending and commits an append without k before the delete.
2225+
*/
2226+
private static Table droppedRequiredKeyTable(String name, boolean makeOptional) {
2227+
Table table = createTable(name, new Schema(
2228+
Types.NestedField.required(1, "id", Types.IntegerType.get()),
2229+
Types.NestedField.required(2, "k", Types.IntegerType.get())), PartitionSpec.unpartitioned(),
2230+
Collections.singletonMap(TableProperties.FORMAT_VERSION, "2"));
2231+
table.newAppend().appendFile(dataFile(table.spec(), "s3://b/db/" + name + "/f1.parquet", 1024, null, null))
2232+
.commit();
2233+
if (makeOptional) {
2234+
table.updateSchema().makeColumnOptional("k").commit();
2235+
}
2236+
RowDelta pendingDelete = table.newRowDelta().addDeletes(equalityDeleteFile(
2237+
"s3://b/db/" + name + "/eq.parquet", FileFormat.PARQUET, 2));
2238+
table.updateSchema().deleteColumn("k").commit();
2239+
table.newAppend().appendFile(dataFile(table.spec(), "s3://b/db/" + name + "/f2.parquet", 1024, null, null))
2240+
.commit();
2241+
pendingDelete.commit();
2242+
Assertions.assertNull(table.schemas().get(table.currentSnapshot().schemaId()).findField(2));
2243+
return table;
2244+
}
2245+
2246+
/** Publishes a schema directly: SchemaUpdate in Iceberg 1.12.0 rejects the v3 DATE promotions. */
2247+
private static void publishSchema(Table table, Schema schema, int lastColumnId) {
2248+
org.apache.iceberg.TableOperations operations = ((org.apache.iceberg.HasTableOperations) table).operations();
2249+
org.apache.iceberg.TableMetadata base = operations.current();
2250+
operations.commit(base, org.apache.iceberg.TableMetadata.buildFrom(base)
2251+
.setCurrentSchema(schema, lastColumnId).build());
2252+
table.refresh();
2253+
}
2254+
21462255
@Test
21472256
public void equalityCarrierResolvesReusedSchemaIdAfterParentSnapshotsExpire() throws Exception {
21482257
Schema schema = new Schema(Types.NestedField.required(1, "id", Types.IntegerType.get()));

0 commit comments

Comments
 (0)