Skip to content

Commit 3cb3d01

Browse files
morningmanclaude
andcommitted
[refactor](fe-core) source the iceberg row-level DML row-id column identity from the connector
The position-delete synthesized row-id column (__DORIS_ICEBERG_ROWID_COL__, a STRUCT{file_path,row_position,partition_spec_id,partition_data}) was defined in two fe-core classes -- IcebergRowId and IcebergMetadataColumn -- duplicating the identical column the iceberg connector already declares as a synthetic write column (IcebergWritePlanProvider.getSyntheticWriteColumns). Delete both classes and source the identity from the connector, aligning with Trino (the connector supplies the row-id column; the engine holds the generic row-level DML machine): - RowLevelDmlRowIdUtils.getRowIdColumn resolves the row-id column from the table's full schema (already connector-appended during row-level DML), falling back to a new ungated PluginDrivenExternalTable.getSyntheticWriteColumns() accessor, and fails loud if the connector declares none (was: reconstruct the STRUCT in fe-core). - IcebergMergeCommand's not-matched INSERT-branch NULL-literal type comes from that column's type instead of the deleted IcebergRowId.getRowIdType(). - IcebergRowLevelDmlTransform's write-constraint exclusion uses an inline constant set of the four $-prefixed position-delete metadata names. The FE<->BE wire name __DORIS_ICEBERG_ROWID_COL__ (resolved by name in the BE) and the STRUCT shape are unchanged, so DML behavior and the BE contract stay byte-identical. Net fe-core source: +40/-194. Tests: strengthen the ConnectorColumnConverter contract pin to assert the exact STRUCT type; add getRowIdColumn branch coverage (full-schema / connector-fallback / throw) and the getSyntheticWriteColumns accessor coverage; drop the tests of the deleted classes. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3
1 parent 39b0a33 commit 3cb3d01

15 files changed

Lines changed: 148 additions & 364 deletions

File tree

‎fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWritePlanProvider.java‎

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -107,15 +107,15 @@ public class IcebergWritePlanProvider implements ConnectorWritePlanProvider {
107107
private static final String SPARK_SQL_COMPRESSION_CODEC = "spark.sql.iceberg.compression-codec";
108108

109109
// Connector-local literal copy of fe-core's Column.ICEBERG_ROWID_COL (connectors must not import
110-
// fe-core). The fe-core ConnectorColumnConverterTest contract pin asserts that converting this exact
111-
// declared shape yields the legacy IcebergRowId.createHiddenColumn() (name / STRUCT / invisible /
112-
// not-null), so a drift on either side turns one of the two tests red.
110+
// fe-core). The connector is the sole source of this row-id column identity; the fe-core
111+
// ConnectorColumnConverterTest contract pin asserts that converting this exact declared shape yields the
112+
// Doris hidden row-id column fe-core's getFullSchema appends (name / STRUCT / invisible / not-null), so a
113+
// drift on either side turns one of the two tests red.
113114
private static final String DORIS_ICEBERG_ROWID_COL = "__DORIS_ICEBERG_ROWID_COL__";
114115

115116
// The single request-scoped synthetic write column iceberg declares: the row-id STRUCT carrying the
116117
// per-row write metadata (file_path / row_position / partition_spec_id / partition_data). Same for
117-
// every iceberg table regardless of format/partitioning (mirrors legacy IcebergRowId), so it is a
118-
// shared immutable instance.
118+
// every iceberg table regardless of format/partitioning, so it is a shared immutable instance.
119119
private static final List<ConnectorColumn> SYNTHETIC_WRITE_COLUMNS =
120120
Collections.singletonList(buildRowIdColumn());
121121

@@ -310,10 +310,10 @@ public ConnectorWritePartitionSpec getWritePartitioning(ConnectorSession session
310310
public List<ConnectorColumn> getSyntheticWriteColumns(ConnectorSession session,
311311
ConnectorTableHandle tableHandle) {
312312
// The iceberg row-id hidden column is the same for every iceberg table regardless of
313-
// format/partitioning — it mirrors legacy IcebergExternalTable.getFullSchema appending
314-
// IcebergRowId.createHiddenColumn() whenever a DML (or show-hidden) is in flight. fe-core gates the
315-
// actual injection request-side (show-hidden / synthetic-write-column ctx flag); here we only
316-
// declare it, so neither the session nor the table handle is consulted.
313+
// format/partitioning. It is the sole source of the row-id column identity: fe-core's getFullSchema
314+
// appends the converted column whenever a DML (or show-hidden) is in flight, and gates the actual
315+
// injection request-side (show-hidden / synthetic-write-column ctx flag); here we only declare it, so
316+
// neither the session nor the table handle is consulted.
317317
return SYNTHETIC_WRITE_COLUMNS;
318318
}
319319

‎fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergWritePlanProviderTest.java‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -697,9 +697,9 @@ public void getWritePartitioningBucketTransformCarriesParamAndDistinctNames() {
697697
// WHY: post-flip the iceberg DML hidden column __DORIS_ICEBERG_ROWID_COL__ that legacy
698698
// IcebergExternalTable.getFullSchema injected is unreachable — a PluginDrivenExternalTable carries no
699699
// iceberg knowledge. The connector therefore declares it as an engine-neutral invisible ConnectorColumn;
700-
// fe-core converts + appends it (gated request-side) while a DML over the table is in flight. This pins
701-
// the carried STRUCT shape (name / 4 fields / types / invisible / not-null) against the legacy
702-
// fe-core IcebergRowId.createHiddenColumn() (its mirror is the fe-core ConnectorColumnConverter contract pin).
700+
// fe-core converts + appends it (gated request-side) while a DML over the table is in flight. The connector
701+
// is the sole source of this identity; this pins the carried STRUCT shape (name / 4 fields / types /
702+
// invisible / not-null), mirrored by the fe-core ConnectorColumnConverter contract pin.
703703

704704
@Test
705705
public void getSyntheticWriteColumnsDeclaresRowIdStruct() {
@@ -712,7 +712,7 @@ public void getSyntheticWriteColumnsDeclaresRowIdStruct() {
712712
ConnectorColumn rowId = cols.get(0);
713713
Assertions.assertEquals("__DORIS_ICEBERG_ROWID_COL__", rowId.getName());
714714
Assertions.assertFalse(rowId.isVisible(), "the row-id column must be hidden (invisible)");
715-
Assertions.assertFalse(rowId.isNullable(), "matches legacy IcebergRowId not-null");
715+
Assertions.assertFalse(rowId.isNullable(), "the row-id column must be not-null");
716716

717717
ConnectorType type = rowId.getType();
718718
Assertions.assertEquals("STRUCT", type.getTypeName());

‎fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalTable.java‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -725,6 +725,17 @@ private List<ConnectorColumn> fetchSyntheticWriteColumns() {
725725
return writePlanProvider.getSyntheticWriteColumns(session, handleOpt.get());
726726
}
727727

728+
/**
729+
* The connector-declared synthetic write columns (e.g. the row-level DML row-id STRUCT), converted to
730+
* engine {@link Column}s. Unlike {@link #getFullSchema()}, this is NOT gated on show-hidden / in-flight
731+
* DML — it always asks the connector. Row-level DML uses it to source the row-id column identity from the
732+
* connector instead of reconstructing the STRUCT in fe-core. Empty on any miss (mirrors
733+
* {@link #fetchSyntheticWriteColumns()}).
734+
*/
735+
public List<Column> getSyntheticWriteColumns() {
736+
return ConnectorColumnConverter.convertColumns(fetchSyntheticWriteColumns());
737+
}
738+
728739
/** The raw connector-emitted table-property map (including FE-internal / render-hint keys). */
729740
private Map<String, String> rawTableProperties() {
730741
makeSureInitialized();

‎fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/IcebergMergeCommand.java‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -359,7 +359,8 @@ private LogicalPlan buildMergeProjectPlan(ConnectContext ctx, ExternalTable iceb
359359
}
360360
}
361361

362-
DataType rowIdType = DataType.fromCatalogType(IcebergRowId.getRowIdType());
362+
DataType rowIdType = DataType.fromCatalogType(
363+
RowLevelDmlRowIdUtils.getRowIdColumn(icebergTable).getType());
363364
for (MergeNotMatchedClause clause : notMatchedClauses) {
364365
finalProjections.add(buildInsertProjection(clause, columns, ctx, rowIdType));
365366
}

‎fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/IcebergMetadataColumn.java‎

Lines changed: 0 additions & 122 deletions
This file was deleted.

‎fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/IcebergRowId.java‎

Lines changed: 0 additions & 68 deletions
This file was deleted.

‎fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/IcebergRowLevelDmlTransform.java‎

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,8 @@
4444
import org.apache.doris.planner.PlanFragment;
4545
import org.apache.doris.qe.ConnectContext;
4646

47+
import com.google.common.collect.ImmutableSet;
48+
4749
import java.util.Optional;
4850
import java.util.Set;
4951
import java.util.function.Predicate;
@@ -61,14 +63,23 @@
6163
*/
6264
public class IcebergRowLevelDmlTransform implements RowLevelDmlTransform {
6365

66+
/**
67+
* Position-delete metadata column names ({@code $file_path}/{@code $row_position}/{@code $partition_spec_id}/
68+
* {@code $partition_data}): the connector-declared row-id STRUCT field names, {@code $}-prefixed. Kept as
69+
* FE-side synthetic-column name constants (the same category as {@link Column#ICEBERG_ROWID_COL}); matched
70+
* case-sensitively ({@code equals}), unlike the rowid ({@code equalsIgnoreCase}).
71+
*/
72+
private static final Set<String> ICEBERG_METADATA_COLUMN_NAMES = ImmutableSet.of(
73+
"$file_path", "$row_position", "$partition_spec_id", "$partition_data");
74+
6475
/**
6576
* Slots excluded from the O5-2 target-only write constraint: the synthetic {@code $row_id} column and
6677
* iceberg metadata columns. Mirrors legacy {@code IcebergConflictDetectionFilterUtils.isTargetOnlyPredicate}
6778
* exactly — keep the {@code equalsIgnoreCase} (rowid) vs {@code equals} (metadata) asymmetry.
6879
*/
6980
private static final Predicate<SlotReference> ICEBERG_EXCLUSION =
7081
slot -> Column.ICEBERG_ROWID_COL.equalsIgnoreCase(slot.getName())
71-
|| IcebergMetadataColumn.isMetadataColumn(slot.getName());
82+
|| ICEBERG_METADATA_COLUMN_NAMES.contains(slot.getName());
7283

7384
@Override
7485
public boolean handles(TableIf table) {

‎fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/RowLevelDmlRowIdUtils.java‎

Lines changed: 15 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@
2222
import org.apache.doris.datasource.ExternalTable;
2323
import org.apache.doris.datasource.plugin.PluginDrivenExternalTable;
2424
import org.apache.doris.nereids.analyzer.Unbound;
25+
import org.apache.doris.nereids.exceptions.AnalysisException;
2526
import org.apache.doris.nereids.trees.expressions.NamedExpression;
2627
import org.apache.doris.nereids.trees.expressions.Slot;
2728
import org.apache.doris.nereids.trees.expressions.SlotReference;
@@ -95,7 +96,11 @@ public static boolean hasRowIdProject(List<NamedExpression> projects) {
9596
return false;
9697
}
9798

98-
/** Resolve the row-id Column definition from the table's full schema. */
99+
/**
100+
* Resolve the row-id Column definition. The identity is owned by the connector, which declares it as a
101+
* synthetic write column; prefer the copy already appended to the table's full schema, and fall back to
102+
* asking the connector directly when the gated append has not run — never reconstruct the STRUCT in fe-core.
103+
*/
99104
public static Column getRowIdColumn(ExternalTable table) {
100105
List<Column> fullSchema = table.getFullSchema();
101106
if (fullSchema != null) {
@@ -105,7 +110,15 @@ public static Column getRowIdColumn(ExternalTable table) {
105110
}
106111
}
107112
}
108-
return IcebergRowId.createHiddenColumn();
113+
if (table instanceof PluginDrivenExternalTable) {
114+
for (Column column : ((PluginDrivenExternalTable) table).getSyntheticWriteColumns()) {
115+
if (Column.ICEBERG_ROWID_COL.equalsIgnoreCase(column.getName())) {
116+
return column;
117+
}
118+
}
119+
}
120+
throw new AnalysisException("Row-id column " + Column.ICEBERG_ROWID_COL
121+
+ " is not declared by the connector for table " + table.getName());
109122
}
110123

111124
/**

0 commit comments

Comments
 (0)