Skip to content

Commit 1127003

Browse files
authored
[fix](arrow-flight) Keep coordinator alive across GetFlightInfo/DoGet for external table scan (#64799)
### What problem does this PR solve? Issue Number: close #62259 Related PR: #64797 Problem Summary: Arrow Flight SQL queries against Iceberg (and other external) tables in batch split mode crashed the BE / failed with `Split source X is released`. Arrow Flight executes a query in two phases: `GetFlightInfo` (plan + submit to BE) and `DoGet` (the client pulls results from the BE). For an external table scan in batch split mode, the BE keeps scanning during `DoGet` and lazily fetches file splits from the FE via the `fetchSplitBatch` RPC, using an async `SplitSource` that the FE coordinator holds (through its scan nodes). The FE closed the coordinator at the end of `GetFlightInfo` (`StmtExecutor.executeAndSendResult`'s `finally` → `Coordinator.close()` → `ScanNode.stop()` → `SplitSourceManager.removeSplitSource()`) and also unregistered it (`FlightSqlConnectProcessor.close()` → `StmtExecutor.finalizeQuery()`). So by the time the BE called `fetchSplitBatch` during `DoGet`, the `SplitSource` was already gone. The MySQL protocol is unaffected because plan + execute share one request, so the coordinator stays alive until all results are consumed. This PR keeps the coordinator (and its `SplitSource`) alive across the two phases and cleans it up reliably: - **StmtExecutor**: for an Arrow Flight query that produces results on the BE (`coordBase == coord`), mark it deferred, register the executor on the `ConnectContext`, and skip the eager `Coordinator.close()` in the `finally`. A failed query (whose `exec()` threw) is not deferred and is closed as before. - **ConnectContext**: hold the deferred executors and add `closeFlightSqlDeferredExecutors()`, which closes their coordinators (releasing the `SplitSource` and the query queue slot) and unregisters the queries. - **FlightSqlConnectProcessor.close()**: do not finalize deferred executors. - **DorisFlightSqlProducer**: finalize the previous query's deferred coordinator when the next query starts on the connection. - **FlightSqlConnectPoolMgr.unregisterConnection()**: finalize deferred coordinators when the connection is torn down. All teardown paths (idle/query timeout, bearer token expiry, explicit `CloseSession`) reach here, so an abandoned connection cannot leak the coordinator. Non-Arrow-Flight paths (MySQL, internal tables, point queries) are unchanged: `deferredForArrowFlight` can only become true for `ARROW_FLIGHT_SQL`. The BE-side error-path hardening (so any `fetchSplitBatch` failure fails gracefully instead of crashing the BE) is handled separately in #64797.
1 parent 513cd20 commit 1127003

10 files changed

Lines changed: 457 additions & 3 deletions

File tree

‎fe/fe-core/src/main/java/org/apache/doris/qe/ConnectContext.java‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -934,6 +934,40 @@ public void clear() {
934934
statementContext = null;
935935
}
936936

937+
// Arrow Flight SQL only.
938+
// Executors of already-planned queries whose results are produced on the BE and pulled later
939+
// during the DoGet phase. Their coordinators must stay alive until the BE finishes scanning:
940+
// an external-table scan in batch mode lazily fetches splits from the FE (a batch SplitSource
941+
// held by the coordinator's scan nodes), so closing the coordinator at the end of
942+
// GetFlightInfo would release the SplitSource too early and make the BE's fetchSplitBatch fail
943+
// with "Split source X is released". These executors are finalized when the next query starts
944+
// on this connection, or when the connection is torn down. See #62259.
945+
private final List<StmtExecutor> flightSqlDeferredExecutors = new ArrayList<>();
946+
947+
public void addFlightSqlDeferredExecutor(StmtExecutor executor) {
948+
synchronized (flightSqlDeferredExecutors) {
949+
flightSqlDeferredExecutors.add(executor);
950+
}
951+
}
952+
953+
public void closeFlightSqlDeferredExecutors() {
954+
List<StmtExecutor> toClose;
955+
synchronized (flightSqlDeferredExecutors) {
956+
if (flightSqlDeferredExecutors.isEmpty()) {
957+
return;
958+
}
959+
toClose = new ArrayList<>(flightSqlDeferredExecutors);
960+
flightSqlDeferredExecutors.clear();
961+
}
962+
for (StmtExecutor deferredExecutor : toClose) {
963+
try {
964+
deferredExecutor.finalizeArrowFlightQuery();
965+
} catch (Throwable t) {
966+
LOG.warn("failed to finalize deferred arrow flight executor", t);
967+
}
968+
}
969+
}
970+
937971
/**
938972
* This method is idempotent.
939973
*/

‎fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java‎

Lines changed: 43 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -199,6 +199,10 @@ public class StmtExecutor {
199199

200200
@Setter
201201
private volatile Coordinator coord = null;
202+
// Arrow Flight SQL: when true, this query's coordinator is kept alive past GetFlightInfo and
203+
// is finalized later by ConnectContext (see #62259), so the eager close in executeAndSendResult
204+
// is skipped.
205+
private volatile boolean deferredForArrowFlight = false;
202206
private MasterOpExecutor masterOpExecutor = null;
203207
private RedirectStatus redirectStatus = null;
204208
private Planner planner;
@@ -1035,6 +1039,23 @@ public void finalizeQuery() {
10351039
QeProcessorImpl.INSTANCE.unregisterQuery(context.queryId());
10361040
}
10371041

1042+
public boolean isDeferredForArrowFlight() {
1043+
return deferredForArrowFlight;
1044+
}
1045+
1046+
// Finalize an Arrow Flight query whose coordinator was kept alive across the
1047+
// GetFlightInfo -> DoGet phases: close the coordinator (releasing external-table batch
1048+
// SplitSources and the query queue slot) and then unregister the query. See #62259.
1049+
public void finalizeArrowFlightQuery() {
1050+
try {
1051+
if (coord != null) {
1052+
coord.close();
1053+
}
1054+
} finally {
1055+
finalizeQuery();
1056+
}
1057+
}
1058+
10381059
private void handleQueryWithRetry(TUniqueId queryId) throws Exception {
10391060
// queue query here
10401061
int retryTime = Config.max_query_retry_time;
@@ -1470,6 +1491,22 @@ public void executeAndSendResult(boolean isOutfileQuery, boolean isSendFields,
14701491
if (context.getConnectType().equals(ConnectType.ARROW_FLIGHT_SQL)) {
14711492
Preconditions.checkState(!context.isReturnResultFromLocal());
14721493
profile.getSummaryProfile().setTempStartTime();
1494+
// Defer closing the coordinator to ConnectContext (closed on the next query or
1495+
// connection teardown) instead of in the finally block below. This gate covers
1496+
// every Arrow Flight query whose results are produced on the BE (coordBase ==
1497+
// coord) -- internal-table and external, batch or not. It is REQUIRED only for an
1498+
// external-table scan in batch mode, where the BE lazily fetches splits from the FE
1499+
// during the later DoGet phase, so closing the coordinator here would release its
1500+
// batch SplitSource too early and break DoGet. Other remote-result queries do not
1501+
// need deferral (the BE buffers their result independently) but are captured by the
1502+
// same gate; the trade-off is their coordinator, query queue slot and query
1503+
// registration stay held until the next query / teardown instead of being released
1504+
// at the end of GetFlightInfo. Point queries use a different coordBase (not
1505+
// deferred). See #62259.
1506+
if (coordBase == coord) {
1507+
deferredForArrowFlight = true;
1508+
context.addFlightSqlDeferredExecutor(this);
1509+
}
14731510
return;
14741511
}
14751512

@@ -1579,7 +1616,12 @@ public void executeAndSendResult(boolean isOutfileQuery, boolean isSendFields,
15791616
this.coord = null;
15801617
throw e;
15811618
} finally {
1582-
coordBase.close();
1619+
// For deferred Arrow Flight queries the coordinator is closed later by ConnectContext
1620+
// (next query / connection teardown), so the BE can still fetch splits during DoGet.
1621+
// See #62259.
1622+
if (!deferredForArrowFlight) {
1623+
coordBase.close();
1624+
}
15831625
}
15841626
}
15851627

‎fe/fe-core/src/main/java/org/apache/doris/service/arrowflight/DorisFlightSqlProducer.java‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -189,6 +189,10 @@ private FlightInfo executeQueryStatement(String peerIdentity, ConnectContext con
189189
try {
190190
Preconditions.checkState(null != connectContext);
191191
Preconditions.checkState(!query.isEmpty());
192+
// Finalize the previous query's coordinator on this connection whose close was
193+
// deferred (Arrow Flight keeps it alive across GetFlightInfo -> DoGet so the BE can
194+
// fetch external-table splits during DoGet). By now the previous DoGet is done. #62259
195+
connectContext.closeFlightSqlDeferredExecutors();
192196
// After the previous query was executed, there was no getStreamStatement to take away the result.
193197
connectContext.getFlightSqlChannel().reset();
194198
connectContext.clearFlightSqlEndpointsLocations();
@@ -280,6 +284,14 @@ private FlightInfo executeQueryStatement(String peerIdentity, ConnectContext con
280284
}
281285
}
282286
} catch (Throwable e) {
287+
// GetFlightInfo failed (e.g. the BE Arrow schema fetch above timed out or returned an
288+
// error) after this query's coordinator may already have been deferred during planning.
289+
// No FlightInfo is returned, so no DoGet will ever pull this query's results; finalize
290+
// the deferred coordinator now (releasing its external-table batch SplitSource, query
291+
// queue slot and query registration) instead of leaking it until the next query starts
292+
// or the connection is torn down. The previous query's deferred coordinator was already
293+
// finalized at the top of this method, so this only closes this failed query. See #62259.
294+
connectContext.closeFlightSqlDeferredExecutors();
283295
String errMsg = "get flight info statement failed, " + e.getMessage() + ", " + Util.getRootCauseMessage(e)
284296
+ ", error code: " + connectContext.getState().getErrorCode() + ", error msg: "
285297
+ connectContext.getState().getErrorMessage();

‎fe/fe-core/src/main/java/org/apache/doris/service/arrowflight/FlightSqlConnectProcessor.java‎

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -196,11 +196,20 @@ public void fetchArrowFlightSchema(int timeoutMs) {
196196
@Override
197197
public void close() throws Exception {
198198
ctx.setCommand(MysqlCommand.COM_SLEEP);
199+
// Executors whose results are pulled from the BE keep their coordinator alive past
200+
// GetFlightInfo (registered as deferred executors on the ConnectContext) so the BE can
201+
// still fetch external-table splits during DoGet. Do NOT finalize those here; they are
202+
// finalized when the next query starts or the connection is torn down. Executors that are
203+
// not deferred (local results, or a query that already failed) are finalized now. See #62259.
199204
for (StmtExecutor asynExecutor : returnResultFromRemoteExecutor) {
200-
asynExecutor.finalizeQuery();
205+
if (!asynExecutor.isDeferredForArrowFlight()) {
206+
asynExecutor.finalizeQuery();
207+
}
201208
}
202209
returnResultFromRemoteExecutor.clear();
203-
executor.finalizeQuery();
210+
if (executor != null && !executor.isDeferredForArrowFlight()) {
211+
executor.finalizeQuery();
212+
}
204213
ctx.clear();
205214
ConnectContext.remove();
206215
}

‎fe/fe-core/src/main/java/org/apache/doris/service/arrowflight/sessions/FlightSqlConnectPoolMgr.java‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,11 @@ public int registerConnection(ConnectContext ctx) {
5555

5656
@Override
5757
public void unregisterConnection(ConnectContext ctx) {
58+
// Finalize any Arrow Flight query whose coordinator was kept alive across the
59+
// GetFlightInfo -> DoGet phases (see #62259), releasing its resources (e.g. external-table
60+
// batch SplitSources and the query queue slot) when the connection is torn down: idle/query
61+
// timeout, bearer token expiry, or explicit CloseSession all reach here.
62+
ctx.closeFlightSqlDeferredExecutors();
5863
ctx.closeTxn();
5964
if (connectionMap.remove(ctx.getConnectionId()) != null) {
6065
numberConnection.decrementAndGet();

‎fe/fe-core/src/test/java/org/apache/doris/qe/ConnectContextTest.java‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -753,4 +753,62 @@ public void testConnectAttributesSetNull() {
753753
Assert.assertNotNull(ctx.getConnectAttributes());
754754
Assert.assertTrue(ctx.getConnectAttributes().isEmpty());
755755
}
756+
757+
// Arrow Flight SQL keeps a query's coordinator alive across GetFlightInfo -> DoGet (see #62259).
758+
// closeFlightSqlDeferredExecutors() is the single place that releases those deferred coordinators
759+
// (and with them the external-table batch SplitSource and the query queue slot). The following
760+
// tests pin the leak-prevention contract of that method: every deferred executor is finalized,
761+
// the list is cleared so nothing is finalized twice or retained, and one failing executor does
762+
// not strand the others' resources.
763+
764+
@Test
765+
public void testCloseFlightSqlDeferredExecutorsFinalizesEachExecutor() {
766+
ConnectContext ctx = new ConnectContext();
767+
StmtExecutor deferred1 = Mockito.mock(StmtExecutor.class);
768+
StmtExecutor deferred2 = Mockito.mock(StmtExecutor.class);
769+
ctx.addFlightSqlDeferredExecutor(deferred1);
770+
ctx.addFlightSqlDeferredExecutor(deferred2);
771+
772+
ctx.closeFlightSqlDeferredExecutors();
773+
774+
// Both deferred coordinators must be finalized, otherwise their SplitSource and query queue
775+
// slot leak after the DoGet phase.
776+
Mockito.verify(deferred1).finalizeArrowFlightQuery();
777+
Mockito.verify(deferred2).finalizeArrowFlightQuery();
778+
}
779+
780+
@Test
781+
public void testCloseFlightSqlDeferredExecutorsClearsListSoSecondCallIsNoOp() {
782+
ConnectContext ctx = new ConnectContext();
783+
StmtExecutor deferred = Mockito.mock(StmtExecutor.class);
784+
ctx.addFlightSqlDeferredExecutor(deferred);
785+
786+
// More than one teardown path can fire for the same connection (e.g. the next query cleans
787+
// up, then the connection is later torn down). The list must be cleared after the first
788+
// call so the executor is finalized exactly once and is not retained (leaked) afterwards.
789+
ctx.closeFlightSqlDeferredExecutors();
790+
ctx.closeFlightSqlDeferredExecutors();
791+
792+
Mockito.verify(deferred, Mockito.times(1)).finalizeArrowFlightQuery();
793+
}
794+
795+
@Test
796+
public void testCloseFlightSqlDeferredExecutorsFinalizesRemainingWhenOneFails() {
797+
ConnectContext ctx = new ConnectContext();
798+
StmtExecutor failing = Mockito.mock(StmtExecutor.class);
799+
StmtExecutor healthy = Mockito.mock(StmtExecutor.class);
800+
Mockito.doThrow(new RuntimeException("finalize failed")).when(failing).finalizeArrowFlightQuery();
801+
ctx.addFlightSqlDeferredExecutor(failing);
802+
ctx.addFlightSqlDeferredExecutor(healthy);
803+
804+
// A single bad coordinator must not abort the cleanup: the call must not throw, and the
805+
// healthy executor must still be finalized so its resources are released.
806+
ctx.closeFlightSqlDeferredExecutors();
807+
Mockito.verify(healthy).finalizeArrowFlightQuery();
808+
809+
// The list is cleared up front, so neither executor is reprocessed on a later teardown.
810+
ctx.closeFlightSqlDeferredExecutors();
811+
Mockito.verify(failing, Mockito.times(1)).finalizeArrowFlightQuery();
812+
Mockito.verify(healthy, Mockito.times(1)).finalizeArrowFlightQuery();
813+
}
756814
}

‎fe/fe-core/src/test/java/org/apache/doris/qe/StmtExecutorTest.java‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,8 @@
2929
import org.apache.doris.planner.ResultFileSink;
3030
import org.apache.doris.qe.CommonResultSet.CommonResultSetMetaData;
3131
import org.apache.doris.qe.ConnectContext.ConnectType;
32+
import org.apache.doris.thrift.TQueryOptions;
33+
import org.apache.doris.thrift.TUniqueId;
3234
import org.apache.doris.utframe.TestWithFeService;
3335

3436
import com.google.common.collect.Lists;
@@ -71,6 +73,40 @@ public void testShowNull() throws Exception {
7173
Assert.assertEquals(QueryState.MysqlStateType.OK, connectContext.getState().getStateType());
7274
}
7375

76+
// Arrow Flight SQL keeps a query's coordinator alive across GetFlightInfo -> DoGet (see #62259);
77+
// it is released later by finalizeArrowFlightQuery(), which closes the coordinator and then
78+
// unregisters the query. The close and the unregister must be independent: if coord.close()
79+
// throws, the query registration must still be released (the try/finally), otherwise the query
80+
// leaks in QeProcessorImpl forever. The thrown error is expected to propagate to the caller
81+
// (ConnectContext.closeFlightSqlDeferredExecutors), which catches and logs it.
82+
@Test
83+
public void testFinalizeArrowFlightQueryUnregistersQueryEvenIfCoordCloseThrows() throws Exception {
84+
StmtExecutor stmtExecutor = new StmtExecutor(connectContext, "");
85+
TUniqueId queryId = new TUniqueId(0x6226259L, 0x62259L);
86+
connectContext.setQueryId(queryId);
87+
88+
Coordinator coord = Mockito.mock(Coordinator.class);
89+
Mockito.when(coord.getQueryOptions()).thenReturn(new TQueryOptions());
90+
Mockito.doThrow(new RuntimeException("coord close failed")).when(coord).close();
91+
stmtExecutor.setCoord(coord);
92+
93+
// Simulate the in-flight query whose results DoGet is still pulling.
94+
QeProcessorImpl.INSTANCE.registerQuery(queryId, new QeProcessorImpl.QueryInfo(coord));
95+
Assert.assertNotNull(QeProcessorImpl.INSTANCE.getCoordinator(queryId));
96+
97+
try {
98+
stmtExecutor.finalizeArrowFlightQuery();
99+
Assert.fail("expected coord.close() failure to propagate after the query is unregistered");
100+
} catch (RuntimeException e) {
101+
Assert.assertEquals("coord close failed", e.getMessage());
102+
}
103+
104+
// The coordinator close was attempted (releases SplitSource + query queue slot) ...
105+
Mockito.verify(coord).close();
106+
// ... and despite it failing, the query registration was still released (no leak).
107+
Assert.assertNull(QeProcessorImpl.INSTANCE.getCoordinator(queryId));
108+
}
109+
74110
@Test
75111
public void testKill() throws Exception {
76112
StmtExecutor stmtExecutor = new StmtExecutor(connectContext, "");

0 commit comments

Comments
 (0)