Skip to content

Commit ed17894

Browse files
morningmansuxiaogang223claude
committed
[test](external) Cover connector INSERT OVERWRITE of a source the planner proves empty
### What problem does this PR solve? Issue Number: None Related PR: #69010, #62344 Problem Summary: When the planner folds the source of an INSERT OVERWRITE to an empty relation (LIMIT 0, WHERE false) and no exchange separates the sink from it, the insert is an empty insert. Before #69010 an empty insert skipped its transaction, so such an overwrite returned OK and the target kept all its rows. branch-4.1 #62344 runs that transaction only for an Iceberg static partition overwrite; master #69010 runs it for every connector overwrite (PluginDrivenInsertExecutor.requiresTransaction). Master behaves correctly, but no test covers it: - PluginDrivenInsertExecutorTest has no case for requiresTransaction. - The planner-proven empty overwrite in test_iceberg_partition_overwrite_mode writes a partitioned table, which master routes through a hash exchange, so the writers run and the case passes without the #69010 override. - The LIMIT 0 overwrite in test_paimon_write_transaction writes an append-only table, which goes through an exchange as well. The sink sits directly on the empty relation for a Paimon primary-key table without a fixed bucket count (Paimon's default, written by a single writer), and for an unpartitioned Iceberg table when enable_strict_consistency_dml is false (the default in cloud mode). On a local cluster with the #69010 override removed, a LIMIT 0 overwrite of such a Paimon table, of a static partition of one, and of such an Iceberg table returned OK and left every row in place. On master all three clear their target. This adds: - PluginDrivenInsertExecutorTest: an empty overwrite, of the whole table or of a static partition, requires its transaction; an empty INSERT does not. - PaimonConnectorTransactionTest: an overwrite that receives no commit data replaces its static partition, or the whole table, with nothing. - IcebergConnectorTransactionTest: the same for an Iceberg static partition, ported from the branch-4.1 test of #62344. - test_paimon_write_transaction: LIMIT 0 overwrites of a dynamic-bucket primary-key table and of a static partition of one, with the plan that puts the sink on the empty relation. The comment of the append-only case now says that it goes through an exchange. - test_iceberg_insert_overwrite_with_empty_table: a LIMIT 0 overwrite of an unpartitioned table with strict DML consistency off. ### Release note None ### Check List (For Author) - Test: Unit Test, Regression test - PluginDrivenInsertExecutorTest (run-fe-ut.sh), PaimonConnectorTransactionTest and IcebergConnectorTransactionTest pass; each new case fails on a mutation of the code it covers. - test_paimon_write_transaction and test_iceberg_insert_overwrite_with_empty_table pass on a local cluster against the iceberg docker environment, and fail with the #69010 override removed. - Behavior changed: No - Does this need documentation: No Co-authored-by: Socrates <[email protected]> Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent 61179b6 commit ed17894

7 files changed

Lines changed: 256 additions & 2 deletions

File tree

‎fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorTransactionTest.java‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -961,6 +961,30 @@ public void overwriteStaticPartitionUsesRowFilter() {
961961
Assertions.assertEquals("1", snap.summary().get("added-data-files"));
962962
}
963963

964+
@Test
965+
public void overwriteEmptyStaticPartitionClearsOnlyThatPartition() {
966+
InMemoryCatalog catalog = freshCatalog();
967+
TableIdentifier id = TableIdentifier.of("db1", "t1");
968+
PartitionSpec spec = PartitionSpec.builderFor(PART_SCHEMA).identity("region").build();
969+
Table table = catalog.createTable(id, PART_SCHEMA, spec, props("write.format.default", "parquet"));
970+
table.newAppend()
971+
.appendFile(partitionedDataFile(spec, "s3://b/db1/t1/region=us/seed.parquet", 1L, "region=us"))
972+
.appendFile(partitionedDataFile(spec, "s3://b/db1/t1/region=eu/seed.parquet", 1L, "region=eu"))
973+
.commit();
974+
975+
// INSERT OVERWRITE ... PARTITION(region='us') of a source with no rows: no commit data, and the 'us'
976+
// partition is still replaced with nothing while the 'eu' file stays.
977+
IcebergConnectorTransaction txn = txnFor(opsReturning(catalog.loadTable(id)), new RecordingConnectorContext());
978+
txn.beginWrite(SESSION, "db1", "t1",
979+
overwriteStaticCtx(catalog.loadTable(id), Collections.singletonMap("region", "us")));
980+
txn.commit();
981+
982+
List<DataFile> files = currentDataFiles(catalog.loadTable(id));
983+
Assertions.assertEquals(1, files.size(), files.toString());
984+
Assertions.assertTrue(files.get(0).path().toString().contains("region=eu"),
985+
files.get(0).path().toString());
986+
}
987+
964988
@Test
965989
public void promotedDateStaticOverwriteRejectsUnsafeRetainedBounds() {
966990
InMemoryCatalog catalog = freshCatalog();

‎fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonConnectorTransactionTest.java‎

Lines changed: 107 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,22 +24,31 @@
2424
import org.apache.doris.thrift.TPaimonCommitMessage;
2525

2626
import org.apache.paimon.CoreOptions;
27+
import org.apache.paimon.Snapshot;
2728
import org.apache.paimon.consumer.ConsumerManager;
2829
import org.apache.paimon.data.BinaryRow;
30+
import org.apache.paimon.data.BinaryString;
31+
import org.apache.paimon.data.GenericRow;
32+
import org.apache.paimon.data.InternalRow;
2933
import org.apache.paimon.fs.local.LocalFileIO;
3034
import org.apache.paimon.io.CompactIncrement;
3135
import org.apache.paimon.io.DataIncrement;
3236
import org.apache.paimon.io.DataOutputViewStreamWrapper;
37+
import org.apache.paimon.reader.RecordReader;
3338
import org.apache.paimon.schema.Schema;
3439
import org.apache.paimon.schema.SchemaManager;
3540
import org.apache.paimon.schema.TableSchema;
3641
import org.apache.paimon.table.DelegatedFileStoreTable;
3742
import org.apache.paimon.table.FileStoreTable;
3843
import org.apache.paimon.table.FileStoreTableFactory;
44+
import org.apache.paimon.table.sink.BatchTableCommit;
45+
import org.apache.paimon.table.sink.BatchTableWrite;
46+
import org.apache.paimon.table.sink.BatchWriteBuilder;
3947
import org.apache.paimon.table.sink.CommitMessage;
4048
import org.apache.paimon.table.sink.CommitMessageImpl;
4149
import org.apache.paimon.table.sink.CommitMessageSerializer;
4250
import org.apache.paimon.table.sink.TableCommitImpl;
51+
import org.apache.paimon.table.source.ReadBuilder;
4352
import org.apache.paimon.types.DataField;
4453
import org.apache.paimon.types.DataTypes;
4554
import org.apache.thrift.TSerializer;
@@ -64,7 +73,8 @@
6473
* {@code PaimonTransactionTest}: a commit whose outcome was lost is reconciled against the snapshots
6574
* Paimon actually published, a transaction whose outcome stays unknown is never aborted, a failure
6675
* before the commit started stays abortable, and the commit payloads the BEs report are
67-
* de-duplicated by their exact content.
76+
* de-duplicated by their exact content. An overwrite whose writers report no commit data still
77+
* commits: it replaces its static partition, or the whole table, with nothing.
6878
*
6979
* <p>No mocking framework (this module has none by convention). The table is a real on-disk Paimon
7080
* table, so reconciliation reads real snapshot files; only the committer it hands out is scripted,
@@ -188,6 +198,35 @@ public void reReportedPayloadsCommitOnce() throws Exception {
188198
Assertions.assertEquals(2, target.committedMessages.size());
189199
}
190200

201+
@Test
202+
public void emptyStaticPartitionOverwriteReplacesThePartitionWithNothing() throws Exception {
203+
// INSERT OVERWRITE ... PARTITION (p = 'a') of a source with no rows: no writer reports commit data, and
204+
// the commit still replaces partition 'a' with nothing. (Mutation: return early on empty commit data for
205+
// an overwrite too -> no snapshot is committed and both rows stay.)
206+
FileStoreTable partitioned = partitionedTableWithRows();
207+
PaimonConnectorTransaction transaction =
208+
overwriteTransaction(partitioned, Collections.singletonMap("p", "a"));
209+
210+
transaction.commit();
211+
212+
Assertions.assertEquals(Snapshot.CommitKind.OVERWRITE,
213+
partitioned.snapshotManager().latestSnapshot().commitKind());
214+
Assertions.assertEquals(Collections.singletonList("2:b"), rows(partitioned),
215+
"only the overwritten partition is cleared");
216+
}
217+
218+
@Test
219+
public void emptyOverwriteReplacesTheTableWithNothing() throws Exception {
220+
FileStoreTable partitioned = partitionedTableWithRows();
221+
PaimonConnectorTransaction transaction = overwriteTransaction(partitioned, Collections.emptyMap());
222+
223+
transaction.commit();
224+
225+
Assertions.assertEquals(Snapshot.CommitKind.OVERWRITE,
226+
partitioned.snapshotManager().latestSnapshot().commitKind());
227+
Assertions.assertEquals(Collections.emptyList(), rows(partitioned));
228+
}
229+
191230
@Test
192231
public void commitUserIsNamespacedByDorisCluster() {
193232
Assertions.assertEquals(
@@ -211,6 +250,73 @@ private static PaimonConnectorTransaction boundTransaction(FileStoreTable target
211250
return transaction;
212251
}
213252

253+
/** A table {@code (id INT, p STRING)} partitioned by {@code p}, holding the rows (1, 'a') and (2, 'b'). */
254+
private FileStoreTable partitionedTableWithRows() throws Exception {
255+
org.apache.paimon.fs.Path location =
256+
new org.apache.paimon.fs.Path("file://" + warehouse + "/db.db/partitioned");
257+
LocalFileIO fileIO = LocalFileIO.create();
258+
new SchemaManager(fileIO, location).createTable(new Schema(
259+
Arrays.asList(ID, new DataField(1, "p", DataTypes.STRING())), Collections.singletonList("p"),
260+
Collections.emptyList(), Collections.emptyMap(), ""));
261+
FileStoreTable partitioned = FileStoreTableFactory.create(fileIO, location);
262+
BatchWriteBuilder writeBuilder = partitioned.newBatchWriteBuilder();
263+
try (BatchTableWrite write = writeBuilder.newWrite()) {
264+
write.write(GenericRow.of(1, BinaryString.fromString("a")));
265+
write.write(GenericRow.of(2, BinaryString.fromString("b")));
266+
List<CommitMessage> messages = write.prepareCommit();
267+
try (BatchTableCommit commit = writeBuilder.newCommit()) {
268+
commit.commit(messages);
269+
}
270+
}
271+
return partitioned;
272+
}
273+
274+
private static PaimonConnectorTransaction overwriteTransaction(FileStoreTable target,
275+
Map<String, String> staticPartition) {
276+
PaimonConnectorTransaction transaction =
277+
new PaimonConnectorTransaction(TRANSACTION_ID, new RecordingConnectorContext());
278+
transaction.bind(PaimonWriteBinding.create(
279+
new PaimonTableHandle("db", "partitioned", target.partitionKeys(), Collections.emptyList()),
280+
target, Collections.emptyMap(), overwrite(staticPartition), true));
281+
return transaction;
282+
}
283+
284+
/** The rows of the table as {@code id:p}, ordered by id. */
285+
private static List<String> rows(FileStoreTable target) throws Exception {
286+
ReadBuilder readBuilder = target.newReadBuilder();
287+
List<String> rows = new ArrayList<>();
288+
try (RecordReader<InternalRow> reader = readBuilder.newRead().createReader(readBuilder.newScan().plan())) {
289+
reader.forEachRemaining(row -> rows.add(row.getInt(0) + ":" + row.getString(1)));
290+
}
291+
rows.sort(null);
292+
return rows;
293+
}
294+
295+
/** An INSERT OVERWRITE, of the given static partition or, when it is empty, of the whole table. */
296+
private static ConnectorWriteHandle overwrite(Map<String, String> staticPartition) {
297+
return new ConnectorWriteHandle() {
298+
@Override
299+
public ConnectorTableHandle getTableHandle() {
300+
return null;
301+
}
302+
303+
@Override
304+
public List<ConnectorColumn> getColumns() {
305+
return Collections.emptyList();
306+
}
307+
308+
@Override
309+
public boolean isOverwrite() {
310+
return true;
311+
}
312+
313+
@Override
314+
public Map<String, String> getStaticPartitionSpec() {
315+
return staticPartition;
316+
}
317+
};
318+
}
319+
214320
/** An INSERT without a PARTITION clause. */
215321
private static ConnectorWriteHandle plainInsert() {
216322
return new ConnectorWriteHandle() {

‎fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/insert/PluginDrivenInsertExecutorTest.java‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@
3232
import org.apache.doris.connector.spi.write.ConnectorSinkPlan;
3333
import org.apache.doris.connector.spi.write.ConnectorWritePlanProvider;
3434
import org.apache.doris.datasource.plugin.PluginDrivenExternalTable;
35+
import org.apache.doris.nereids.trees.expressions.literal.StringLiteral;
3536
import org.apache.doris.planner.PluginDrivenTableSink;
3637
import org.apache.doris.thrift.TDataSink;
3738
import org.apache.doris.transaction.PluginDrivenTransactionManager;
@@ -188,6 +189,33 @@ public void transactionTypeIsUnknownWhenNoTransaction() {
188189
"with no connector transaction (empty insert), transactionType is UNKNOWN");
189190
}
190191

192+
@Test
193+
public void overwriteOfASourceProvenEmptyStillRunsItsTransaction() {
194+
// A sink planned directly on an empty relation (a single-writer plan of a source folded away, such as
195+
// LIMIT 0) makes the write an empty insert. An overwrite of nothing still replaces the table or the static
196+
// partition with nothing, so it must reach the connector's commit instead of skipping it and reporting
197+
// success. (Mutation: drop the requiresTransaction override -> both overwrites skip their transaction.)
198+
PluginDrivenInsertCommandContext wholeTable = new PluginDrivenInsertCommandContext();
199+
wholeTable.setOverwrite(true);
200+
Assertions.assertTrue(emptyInsertExecutor(Optional.of(wholeTable)).requiresTransaction(),
201+
"an empty overwrite of the whole table must still clear the table");
202+
203+
PluginDrivenInsertCommandContext staticPartition = new PluginDrivenInsertCommandContext();
204+
staticPartition.setOverwrite(true);
205+
staticPartition.setStaticPartitionSpecFromExpressions(
206+
Collections.singletonMap("p", new StringLiteral("a")));
207+
Assertions.assertTrue(emptyInsertExecutor(Optional.of(staticPartition)).requiresTransaction(),
208+
"an empty overwrite of a static partition must still clear the partition");
209+
}
210+
211+
@Test
212+
public void insertOfASourceProvenEmptySkipsItsTransaction() {
213+
// An INSERT of no rows has nothing to commit.
214+
Assertions.assertFalse(emptyInsertExecutor(Optional.of(new PluginDrivenInsertCommandContext()))
215+
.requiresTransaction());
216+
Assertions.assertFalse(emptyInsertExecutor(Optional.empty()).requiresTransaction());
217+
}
218+
191219
@Test
192220
public void doBeforeCommitBackfillsLoadedRowsWhenTransactionReportsCount() throws UserException {
193221
// Transaction model with a real row count (e.g. maxcompute): BE reports rows only through
@@ -239,6 +267,16 @@ private static PluginDrivenInsertExecutor newUnconstructedExecutor() {
239267
return Mockito.mock(PluginDrivenInsertExecutor.class, Mockito.CALLS_REAL_METHODS);
240268
}
241269

270+
/** An executor whose sink sits directly on an empty relation, as initPlan builds it for such a plan. */
271+
private static PluginDrivenInsertExecutor emptyInsertExecutor(Optional<InsertCommandContext> insertCtx) {
272+
PluginDrivenInsertExecutor exec = newUnconstructedExecutor();
273+
Deencapsulation.setField(exec, "emptyInsert", true);
274+
// Field initializers do not run without the constructor.
275+
Deencapsulation.setField(exec, "streamUpdateInfos", Collections.emptyList());
276+
Deencapsulation.setField(exec, "insertCtx", insertCtx);
277+
return exec;
278+
}
279+
242280
/** Write ops that hand back a fixed connector transaction and record the handle the executor threads in. */
243281
private static final class FakeTxnWriteOps implements ConnectorWriteOps {
244282
private final ConnectorTransaction txn;

‎regression-test/data/external_table_p0/iceberg/write/test_iceberg_insert_overwrite_with_empty_table.out‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,3 +16,9 @@
1616

1717
-- !q3 --
1818

19+
-- !q4_plan --
20+
PhysicalConnectorTableSink
21+
--PhysicalEmptyRelation
22+
23+
-- !q4 --
24+

‎regression-test/data/external_table_p0/paimon/write/test_paimon_write_transaction.out‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,28 @@
3737
-- !txn_empty_overwrite_limit_zero --
3838
0
3939

40+
-- !txn_dynamic_bucket_empty_overwrite_plan --
41+
PhysicalConnectorTableSink
42+
--PhysicalEmptyRelation
43+
44+
-- !txn_dynamic_bucket_empty_overwrite --
45+
0
46+
47+
-- !txn_dynamic_bucket_empty_overwrite_snapshots --
48+
1 APPEND
49+
2 OVERWRITE
50+
51+
-- !txn_dynamic_bucket_empty_static_overwrite_plan --
52+
PhysicalConnectorTableSink
53+
--PhysicalEmptyRelation
54+
55+
-- !txn_dynamic_bucket_empty_static_overwrite --
56+
2 west_old west
57+
58+
-- !txn_dynamic_bucket_empty_static_overwrite_snapshots --
59+
1 APPEND
60+
2 OVERWRITE
61+
4062
-- !txn_static_partition --
4163
10 east_new east
4264
2 west_old west

‎regression-test/suites/external_table_p0/iceberg/write/test_iceberg_insert_overwrite_with_empty_table.groovy‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,4 +69,14 @@ suite("test_iceberg_insert_overwrite_with_empty_table", "p0,external") {
6969
order_qt_q2 """ select * from ${tb1} """ // should have 3 records
7070
order_qt_q3 """ select * from ${tb2} """ // should have no records
7171

72+
// With strict DML consistency off (the cloud default), the sink of an unpartitioned table sits directly on
73+
// the empty relation of a source the planner proves empty, and no writer runs. Such an insert of nothing
74+
// skips its transaction, but an overwrite still clears the table.
75+
sql """ insert into ${tb2} values (1,1), (1,2), (1,3) """
76+
sql """ set enable_strict_consistency_dml = false """
77+
qt_q4_plan """ explain shape plan insert overwrite table ${tb2} select * from ${tb2} limit 0 """
78+
sql """ insert overwrite table ${tb2} select * from ${tb2} limit 0 """
79+
sql """ set enable_strict_consistency_dml = true """
80+
order_qt_q4 """ select * from ${tb2} """ // should have no records
81+
7282
}

‎regression-test/suites/external_table_p0/paimon/write/test_paimon_write_transaction.groovy‎

Lines changed: 49 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,23 @@ suite("test_paimon_write_transaction", "p0,external,paimon") {
4747
id INT, name STRING
4848
) USING paimon;
4949
50+
DROP TABLE IF EXISTS paimon.${dbName}.t_overwrite_dynamic_bucket;
51+
CREATE TABLE paimon.${dbName}.t_overwrite_dynamic_bucket (
52+
id INT, name STRING
53+
) USING paimon
54+
TBLPROPERTIES (
55+
'primary-key' = 'id'
56+
);
57+
58+
DROP TABLE IF EXISTS paimon.${dbName}.t_overwrite_dynamic_bucket_part;
59+
CREATE TABLE paimon.${dbName}.t_overwrite_dynamic_bucket_part (
60+
id INT, name STRING, region STRING
61+
) USING paimon
62+
PARTITIONED BY (region)
63+
TBLPROPERTIES (
64+
'primary-key' = 'id,region'
65+
);
66+
5067
DROP TABLE IF EXISTS paimon.${dbName}.t_overwrite_part;
5168
CREATE TABLE paimon.${dbName}.t_overwrite_part (
5269
id INT, name STRING, region STRING
@@ -208,12 +225,43 @@ suite("test_paimon_write_transaction", "p0,external,paimon") {
208225
qt_txn_empty_overwrite """SELECT COUNT(*) FROM t_overwrite"""
209226
assertTableEquals("t_overwrite", "ORDER BY id")
210227

211-
// A direct PhysicalEmptyRelation must still publish the empty overwrite snapshot.
228+
// A source that LIMIT 0 folds to an empty relation must still publish the empty overwrite snapshot. An
229+
// append-only table is written through an exchange, so the writers run and report nothing.
212230
sql """INSERT INTO t_overwrite VALUES (30, 'old_for_limit_zero')"""
213231
sql """INSERT OVERWRITE TABLE t_overwrite SELECT 1, 'unused' LIMIT 0"""
214232
qt_txn_empty_overwrite_limit_zero """SELECT COUNT(*) FROM t_overwrite"""
215233
assertTableEquals("t_overwrite", "ORDER BY id")
216234

235+
// A primary-key table without a fixed bucket count has a single writer, so the sink of an overwrite
236+
// whose source the planner proves empty sits directly on the empty relation, and no writer runs. Such an
237+
// insert of nothing skips its transaction, but an overwrite still replaces the table, or the static
238+
// partition, with nothing.
239+
sql """INSERT INTO t_overwrite_dynamic_bucket VALUES (1, 'old1'), (2, 'old2')"""
240+
qt_txn_dynamic_bucket_empty_overwrite_plan """
241+
EXPLAIN SHAPE PLAN INSERT OVERWRITE TABLE t_overwrite_dynamic_bucket SELECT 1, 'unused' LIMIT 0
242+
"""
243+
sql """INSERT OVERWRITE TABLE t_overwrite_dynamic_bucket SELECT 1, 'unused' LIMIT 0"""
244+
qt_txn_dynamic_bucket_empty_overwrite """SELECT COUNT(*) FROM t_overwrite_dynamic_bucket"""
245+
order_qt_txn_dynamic_bucket_empty_overwrite_snapshots """
246+
SELECT snapshot_id, commit_kind FROM t_overwrite_dynamic_bucket\$snapshots
247+
"""
248+
assertTableEquals("t_overwrite_dynamic_bucket", "ORDER BY id")
249+
250+
sql """INSERT INTO t_overwrite_dynamic_bucket_part VALUES
251+
(1, 'east_old', 'east'), (2, 'west_old', 'west')"""
252+
qt_txn_dynamic_bucket_empty_static_overwrite_plan """
253+
EXPLAIN SHAPE PLAN INSERT OVERWRITE TABLE t_overwrite_dynamic_bucket_part
254+
PARTITION (region = 'east') SELECT 1, 'unused' LIMIT 0
255+
"""
256+
sql """INSERT OVERWRITE TABLE t_overwrite_dynamic_bucket_part
257+
PARTITION (region = 'east') SELECT 1, 'unused' LIMIT 0"""
258+
order_qt_txn_dynamic_bucket_empty_static_overwrite """SELECT id, name, region
259+
FROM t_overwrite_dynamic_bucket_part ORDER BY id"""
260+
order_qt_txn_dynamic_bucket_empty_static_overwrite_snapshots """
261+
SELECT snapshot_id, commit_kind FROM t_overwrite_dynamic_bucket_part\$snapshots
262+
"""
263+
assertTableEquals("t_overwrite_dynamic_bucket_part", "ORDER BY id")
264+
217265
// FT-015: Static partition overwrite replaces only the requested partition.
218266
sql """INSERT INTO t_overwrite_part VALUES
219267
(1, 'east_old', 'east'), (2, 'west_old', 'west')"""

0 commit comments

Comments
 (0)