Skip to content

Commit fa99d0d

Browse files
morningmanclaude
andcommitted
[improvement](fe) End connector planning when the planner releases its resources
### What problem does this PR solve? Issue Number: None Related PR: #66473, #66530 Problem Summary: NereidsPlanner.plan ended connector planning in a finally block of its own, next to StatementContext.releasePlannerResources, the cleanup that NereidsPlanner.planWithLock already runs in its finally once the plan is translated and its scans are finalized, and that StatementContext.close runs as a fallback. A planning path through planWithLock that came to keep scan memos would have had to remember the second call. releasePlannerResources now ends connector planning after it has released the planner resources and before it rethrows a failure to release one, so a table lock that fails to unlock does not keep the planning values either. plan() no longer ends connector planning itself, and StatementContext.endConnectorPlanning is private. Within plan() the memo is dropped at the same point as before. ### Release note None ### Check List (For Author) - Test: Unit Test - ConnectorStatementScopeTest.releasingPlannerResourcesEndsThePlanningOfTheScope (was statementContextEndsThePlanningOfItsScope) and releasingPlannerResourcesEndsPlanningWhenAResourceFailsToClose: a table whose read unlock throws still loses the memo, and the failure reaches the caller. Ending connector planning only after a successful release makes the second case fail; not ending it in releasePlannerResources makes both fail, and NereidsPlannerConnectorPlanningTest too. - NereidsPlannerConnectorPlanningTest keeps its assertions: plan() still ends connector planning whether it succeeds or fails. - The fe-core scope and planner tests (50) pass. - Behavior changed: No - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent 3ee1658 commit fa99d0d

5 files changed

Lines changed: 39 additions & 12 deletions

File tree

‎fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorStatementScopeImpl.java‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,8 @@
4444
* String-key SPI contract.</p>
4545
*
4646
* <p>A value registered through {@link #computeIfAbsentForPlanning} lives only until {@link #endPlanning}, which
47-
* {@link org.apache.doris.nereids.NereidsPlanner#plan} calls when it returns or throws.</p>
47+
* {@link org.apache.doris.nereids.StatementContext#releasePlannerResources} calls once planning is over, whether or
48+
* not the plan succeeded.</p>
4849
*/
4950
public class ConnectorStatementScopeImpl implements ConnectorStatementScope {
5051

‎fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java‎

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -170,9 +170,6 @@ public void plan(StatementBase queryStmt, org.apache.doris.thrift.TQueryOptions
170170
} finally {
171171
statementContext.getStopwatch().stop();
172172
NereidsPlanner.runningPlanNum.decrementAndGet();
173-
// Connectors plan the scans of the statement only inside plan(), so what they kept for planning, such as
174-
// the ranges identical scans share, is not needed once it returns or throws.
175-
statementContext.endConnectorPlanning();
176173
}
177174

178175
if (LOG.isDebugEnabled()) {

‎fe/fe-core/src/main/java/org/apache/doris/nereids/StatementContext.java‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -829,9 +829,9 @@ public synchronized ConnectorStatementScope getOrCreateConnectorStatementScope()
829829
/**
830830
* Drops the values of this statement's connector scope that only planning reads (see
831831
* {@link ConnectorStatementScope#endPlanning}), when a connector has opened the scope; unlike
832-
* {@link #getOrCreateConnectorStatementScope}, it never creates one.
832+
* {@link #getOrCreateConnectorStatementScope}, it never creates one. {@link #releasePlannerResources} calls it.
833833
*/
834-
public synchronized void endConnectorPlanning() {
834+
private synchronized void endConnectorPlanning() {
835835
if (this.connectorStatementScope != null) {
836836
this.connectorStatementScope.endPlanning();
837837
}
@@ -1145,6 +1145,10 @@ public synchronized void releasePlannerResources() {
11451145
}
11461146
}
11471147
}
1148+
// NereidsPlanner.planWithLock runs this once its plan is translated and the scans are finalized, so what
1149+
// connectors kept only for planning, such as the ranges identical scans share, can go, also when a resource
1150+
// failed to close.
1151+
endConnectorPlanning();
11481152
if (throwable != null) {
11491153
Throwables.throwIfInstanceOf(throwable, RuntimeException.class);
11501154
throw new IllegalStateException("Release resource failed", throwable);

‎fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorStatementScopeTest.java‎

Lines changed: 29 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@
1717

1818
package org.apache.doris.connector;
1919

20+
import org.apache.doris.catalog.TableIf;
2021
import org.apache.doris.connector.spi.ConnectorSession;
2122
import org.apache.doris.connector.spi.ConnectorStatementScope;
2223
import org.apache.doris.connector.spi.ConnectorWriteOps;
@@ -40,6 +41,7 @@
4041
import java.util.ArrayList;
4142
import java.util.Arrays;
4243
import java.util.List;
44+
import java.util.concurrent.TimeUnit;
4345
import java.util.concurrent.atomic.AtomicInteger;
4446
import java.util.function.Supplier;
4547

@@ -156,22 +158,44 @@ public void planningValueMustHoldNothingToClose() {
156158
}
157159

158160
@Test
159-
public void statementContextEndsThePlanningOfItsScope() {
160-
// NereidsPlanner.plan ends connector planning through the statement context; a statement no connector has
161-
// touched has no scope to end. MUTATION: endConnectorPlanning not reaching the scope -> red.
161+
public void releasingPlannerResourcesEndsThePlanningOfTheScope() {
162+
// NereidsPlanner.planWithLock releases the planner resources when planning is over, which ends connector
163+
// planning; a statement no connector has touched has no scope to end. MUTATION: releasePlannerResources not
164+
// ending connector planning -> red.
162165
StatementContext ctx = new StatementContext();
163-
ctx.endConnectorPlanning();
166+
ctx.releasePlannerResources();
164167
ConnectorStatementScope scope = ctx.getOrCreateConnectorStatementScope();
165168
Object memo = scope.computeIfAbsentForPlanning("memo", Object::new);
166169
Object table = scope.computeIfAbsent("table", Object::new);
167170

168-
ctx.endConnectorPlanning();
171+
ctx.releasePlannerResources();
169172

170173
Assertions.assertSame(scope, ctx.getOrCreateConnectorStatementScope(), "ending planning keeps the scope");
171174
Assertions.assertSame(table, scope.computeIfAbsent("table", Object::new));
172175
Assertions.assertNotSame(memo, scope.computeIfAbsentForPlanning("memo", Object::new));
173176
}
174177

178+
@Test
179+
public void releasingPlannerResourcesEndsPlanningWhenAResourceFailsToClose() {
180+
// A table lock that fails to unlock still reaches the caller, after the planning values are gone.
181+
// MUTATION: ending connector planning only after the release succeeded -> the memo survives -> red.
182+
StatementContext ctx = new StatementContext();
183+
ConnectorStatementScope scope = ctx.getOrCreateConnectorStatementScope();
184+
Object memo = scope.computeIfAbsentForPlanning("memo", Object::new);
185+
TableIf table = Mockito.mock(TableIf.class);
186+
Mockito.when(table.needReadLockWhenPlan()).thenReturn(true);
187+
Mockito.when(table.tryReadLock(Mockito.anyLong(), Mockito.any(TimeUnit.class))).thenReturn(true);
188+
Mockito.when(table.getNameWithFullQualifiers()).thenReturn("internal.db.t");
189+
IllegalStateException unlockFailure = new IllegalStateException("unlock failed");
190+
Mockito.doThrow(unlockFailure).when(table).readUnlock();
191+
ctx.getTables().put(Arrays.asList("internal", "db", "t"), table);
192+
ctx.lock();
193+
194+
Assertions.assertSame(unlockFailure,
195+
Assertions.assertThrows(IllegalStateException.class, ctx::releasePlannerResources));
196+
Assertions.assertNotSame(memo, scope.computeIfAbsentForPlanning("memo", Object::new));
197+
}
198+
175199
@Test
176200
public void statementContextScopeIsStableThenResetsForReusedPreparedContext() {
177201
// StatementContext lazily builds one scope and reuses it across the statement (getOrCreate...). A prepared

‎fe/fe-core/src/test/java/org/apache/doris/nereids/NereidsPlannerConnectorPlanningTest.java‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,8 @@ protected void runBeforeAll() throws Exception {
3737
public void planEndsConnectorPlanningWhetherItSucceedsOrFails() {
3838
// Identical scans share their planned ranges only within one plan, so what a connector keeps for planning
3939
// must not outlive the plan, whether the plan is complete or planning fails; the values the statement keeps
40-
// stay. MUTATION: not ending connector planning in plan()'s finally -> red for both statements.
40+
// stay. MUTATION: planWithLock's cleanup (releasePlannerResources) not ending connector planning -> red for
41+
// both statements.
4142
assertPlanEndsConnectorPlanning("select 1", true);
4243
assertPlanEndsConnectorPlanning("select * from no_such_table", false);
4344
}

0 commit comments

Comments
 (0)