Skip to content

Commit b2f346c

Browse files
committed
[fix](fe) Take the last-chance cancellation check on the auto-detect route too
An auto-detect overwrite (PARTITION(*)) publishes through the overwrite manager, so it bypassed the check the explicit-partition route got. With enable_strict_consistency_dml off, an empty-plan auto-detect overwrite has no exchange in its plan -- VEMPTYSET is the sink's direct child -- so it takes the insert's no-transaction path, and a cancellation landing while the swap waits for the table lock was answered with a success for a statement that did nothing. Both routes now publish through one place that holds the target table's write lock and reads the flag under it. Key changes: - publishTheOverwrite takes the publication as an action, so the explicit-partition swap and the auto-detect task group share the same last-chance decision, and the same failure for a target that was dropped. - The suite drives the auto-detect shape: with strict DML consistency off, a cancelled PARTITION(*) overwrite of an empty plan fails and the table keeps its rows. Unit Test: - regression-test/suites/insert_overwrite_p0/test_insert_overwrite_cancel.groovy - test_insert_overwrite_cancel, test_iot_auto_detect, test_iot_auto_detect_fail, test_iot_auto_detect_concurrent, test_iot_overwrite_and_create and mtmv_p0/ivm/test_ivm_basic_mtmv pass.
1 parent 9039a08 commit b2f346c

3 files changed

Lines changed: 58 additions & 16 deletions

File tree

‎fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertOverwriteTableCommand.java‎

Lines changed: 35 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -146,6 +146,17 @@ public class InsertOverwriteTableCommand extends Command
146146
public static final String DEBUG_POINT_CANCEL_WHILE_THE_SWAP_WAITS_FOR_THE_TABLE_LOCK =
147147
"InsertOverwriteTableCommand.cancelWhileTheSwapWaitsForTheTableLock";
148148

149+
/**
150+
* The swap that publishes an overwrite: replacing the temp partitions for an explicit-partition
151+
* overwrite, or making a task group's replacements visible for an auto-detect one. Both run through
152+
* {@link #publishTheOverwrite}, which holds the target table's write lock and takes the last look at the
153+
* cancellation flag before letting one run.
154+
*/
155+
@FunctionalInterface
156+
private interface OverwritePublication {
157+
void publish() throws UserException;
158+
}
159+
149160
private static final Logger LOG = LogManager.getLogger(InsertOverwriteTableCommand.class);
150161

151162
private LogicalPlan originLogicalQuery;
@@ -292,18 +303,23 @@ public void run(ConnectContext ctx, StmtExecutor executor) throws Exception {
292303
}
293304
if (isAutoDetectOverwrite(getLogicalQuery())) {
294305
// taskId here is a group id. it contains all replace tasks made and registered in rpc process.
295-
taskId = insertOverwriteManager.registerTaskGroup(targetTable);
306+
final long groupId = insertOverwriteManager.registerTaskGroup(targetTable);
307+
taskId = groupId;
296308
// When inserting, BE will call to replace partition by FrontendService. FE will register new temp
297309
// partitions and return. for transactional, the replacement will really occur when insert successed,
298310
// i.e. `insertInto` finished. then we call taskGroupSuccess to make replacement.
299-
InsertCommandContext insertCtx = insertIntoAutoDetect(ctx, executor, taskId);
311+
InsertCommandContext insertCtx = insertIntoAutoDetect(ctx, executor, groupId);
300312
if (isCancelled.get() && !insertCtx.hasCommitted()) {
301-
// The load committed no row, so the cancellation still has everything to take back: the
302-
// catch drops the group's temp partitions (there are none), and the statement fails
303-
// rather than publishing a replacement the client cancelled.
313+
// The load committed nothing -- an empty plan takes the path that begins no transaction --
314+
// so the cancellation still has everything to take back: the catch drops the group's temp
315+
// partitions (an empty plan registers none), and the statement fails rather than reporting
316+
// a replacement the client cancelled. A cancellation landing after this check, while the
317+
// swap waits for the table lock, is taken up again by publishTheOverwrite below.
304318
throw cancelledBeforeTheRowsWereCommitted("after a load that committed nothing", ctx);
305319
}
306-
insertOverwriteManager.taskGroupSuccess(taskId, (OlapTable) targetTable, isForceDropPartition());
320+
publishTheOverwrite(targetTable, insertCtx, ctx,
321+
() -> insertOverwriteManager.taskGroupSuccess(groupId, (OlapTable) targetTable,
322+
isForceDropPartition()));
307323
} else {
308324
// it's overwrite table(as all partitions) or specific partition(s)
309325
List<String> tempPartitionNames = InsertOverwriteUtil.generateTempPartitionNames(partitionNames);
@@ -348,7 +364,12 @@ public void run(ConnectContext ctx, StmtExecutor executor) throws Exception {
348364
+ " queryId: {}", ctx.getQueryIdentifier());
349365
}
350366
failBetweenTheTwoHalvesOfAnOverwrite(targetTable);
351-
publishTheOverwrite(targetTable, partitionNames, tempPartitionNames, insertCtx, ctx);
367+
// The publication below is a lambda, so the partitions it replaces need a name that is final:
368+
// partitionNames is assigned on more than one path above.
369+
final List<String> replacedPartitionNames = partitionNames;
370+
publishTheOverwrite(targetTable, insertCtx, ctx,
371+
() -> InsertOverwriteUtil.replacePartition(targetTable, replacedPartitionNames,
372+
tempPartitionNames, isForceDropPartition()));
352373
if (isCancelled.get()) {
353374
LOG.info("insert overwrite is cancelled before taskSuccess, do nothing, queryId: {}",
354375
ctx.getQueryIdentifier());
@@ -459,23 +480,22 @@ private void cancelTheOverwriteAt(String debugPointName, TableIf targetTable) {
459480
}
460481

461482
/**
462-
* Publishes this overwrite by swapping the temp partitions in, with a last look at the cancellation flag
463-
* taken under the lock the swap contends for.
483+
* Publishes this overwrite by running its swap, with a last look at the cancellation flag taken under the
484+
* lock the swap contends for.
464485
*
465486
* <p>{@link #run} reads the flag before the swap is issued, and the swap then waits for the table's write
466487
* lock, so a cancellation that arrives during that wait is the one place a check before the swap cannot
467488
* see. Reading it again here costs nothing and is where the wait happens: for a cancellation with nothing
468489
* committed there is nothing durable to publish, so refusing to swap costs the statement and leaves the
469-
* rows the client asked to keep -- while a swap that went ahead would replace them with empty partitions.
490+
* rows the client asked to keep -- while a swap that went ahead would replace them with an empty result.
470491
*
471492
* <p>Only a local table is wrapped: a remote table swaps on the frontend that owns it, where this lock
472493
* says nothing.
473494
*/
474-
private void publishTheOverwrite(TableIf targetTable, List<String> partitionNames,
475-
List<String> tempPartitionNames, InsertCommandContext insertCtx, ConnectContext ctx) throws UserException {
495+
private void publishTheOverwrite(TableIf targetTable, InsertCommandContext insertCtx, ConnectContext ctx,
496+
OverwritePublication publication) throws UserException {
476497
if (!(targetTable instanceof OlapTable) || targetTable instanceof RemoteOlapTable) {
477-
InsertOverwriteUtil.replacePartition(targetTable, partitionNames, tempPartitionNames,
478-
isForceDropPartition());
498+
publication.publish();
479499
return;
480500
}
481501
OlapTable olapTable = (OlapTable) targetTable;
@@ -493,8 +513,7 @@ private void publishTheOverwrite(TableIf targetTable, List<String> partitionName
493513
if (isCancelled.get() && !insertCtx.hasCommitted()) {
494514
throw cancelledBeforeTheRowsWereCommitted("while the swap waited for the table lock", ctx);
495515
}
496-
InsertOverwriteUtil.replacePartition(targetTable, partitionNames, tempPartitionNames,
497-
isForceDropPartition());
516+
publication.publish();
498517
} finally {
499518
olapTable.writeUnlock();
500519
}

‎regression-test/data/insert_overwrite_p0/test_insert_overwrite_cancel.out‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,10 @@
1313
-- !dst_after_the_cancelled_locked_swap --
1414
1 2026-01-10 100
1515

16+
-- !dst_after_the_cancelled_auto_detect_swap --
17+
3 2026-01-20 300
18+
4 2026-02-20 400
19+
1620
-- !dst_after_the_publish_timeout --
1721
3 2026-01-20 300
1822
4 2026-02-20 400

‎regression-test/suites/insert_overwrite_p0/test_insert_overwrite_cancel.groovy‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -186,6 +186,25 @@ suite("test_insert_overwrite_cancel", "nonConcurrent") {
186186
}
187187
order_qt_dst_after_the_cancelled_locked_swap """SELECT id, dt, amount FROM test_iot_cancel_flat_dst"""
188188

189+
// The auto-detect route takes the same decision at the same point, and it needs it: with
190+
// enable_strict_consistency_dml off, an empty-plan `PARTITION(*)` overwrite has no exchange in its plan
191+
// either, so it takes the insert's no-transaction path as well and its swap has nothing to publish. The
192+
// cancellation has to be honoured there too, and the table has to keep its rows.
193+
try {
194+
sql """SET enable_strict_consistency_dml = false"""
195+
GetDebugPoint().enableDebugPointForAllFEs(waitsPoint,
196+
[table_name: "test_iot_cancel_dst"])
197+
test {
198+
sql """INSERT OVERWRITE TABLE test_iot_cancel_dst PARTITION(*)
199+
SELECT id, dt, amount FROM test_iot_cancel_src WHERE 1 = 0"""
200+
exception "insert overwrite is cancelled while the swap waited for the table lock"
201+
}
202+
} finally {
203+
GetDebugPoint().disableDebugPointForAllFEs(waitsPoint)
204+
sql """SET enable_strict_consistency_dml = true"""
205+
}
206+
order_qt_dst_after_the_cancelled_auto_detect_swap """SELECT id, dt, amount FROM test_iot_cancel_dst"""
207+
189208
// An error response over committed rows: `insert_visible_timeout_return_mode=error` turns a publication
190209
// timeout that follows the commit into an error, but the rows the overwrite wrote are durable, so the
191210
// overwrite has to be published rather than dropped with the temporary partitions. The row the overwrite

0 commit comments

Comments
 (0)