Skip to content

Commit 514da84

Browse files
morningmanyiguolei
authored andcommitted
[fix](arrow-flight) Keep coordinator alive across GetFlightInfo/DoGet for external table scan (apache#64799)
Issue Number: close apache#62259 Related PR: apache#64797 Problem Summary: Arrow Flight SQL queries against Iceberg (and other external) tables in batch split mode crashed the BE / failed with `Split source X is released`. Arrow Flight executes a query in two phases: `GetFlightInfo` (plan + submit to BE) and `DoGet` (the client pulls results from the BE). For an external table scan in batch split mode, the BE keeps scanning during `DoGet` and lazily fetches file splits from the FE via the `fetchSplitBatch` RPC, using an async `SplitSource` that the FE coordinator holds (through its scan nodes). The FE closed the coordinator at the end of `GetFlightInfo` (`StmtExecutor.executeAndSendResult`'s `finally` → `Coordinator.close()` → `ScanNode.stop()` → `SplitSourceManager.removeSplitSource()`) and also unregistered it (`FlightSqlConnectProcessor.close()` → `StmtExecutor.finalizeQuery()`). So by the time the BE called `fetchSplitBatch` during `DoGet`, the `SplitSource` was already gone. The MySQL protocol is unaffected because plan + execute share one request, so the coordinator stays alive until all results are consumed. This PR keeps the coordinator (and its `SplitSource`) alive across the two phases and cleans it up reliably: - **StmtExecutor**: for an Arrow Flight query that produces results on the BE (`coordBase == coord`), mark it deferred, register the executor on the `ConnectContext`, and skip the eager `Coordinator.close()` in the `finally`. A failed query (whose `exec()` threw) is not deferred and is closed as before. - **ConnectContext**: hold the deferred executors and add `closeFlightSqlDeferredExecutors()`, which closes their coordinators (releasing the `SplitSource` and the query queue slot) and unregisters the queries. - **FlightSqlConnectProcessor.close()**: do not finalize deferred executors. - **DorisFlightSqlProducer**: finalize the previous query's deferred coordinator when the next query starts on the connection. - **FlightSqlConnectPoolMgr.unregisterConnection()**: finalize deferred coordinators when the connection is torn down. All teardown paths (idle/query timeout, bearer token expiry, explicit `CloseSession`) reach here, so an abandoned connection cannot leak the coordinator. Non-Arrow-Flight paths (MySQL, internal tables, point queries) are unchanged: `deferredForArrowFlight` can only become true for `ARROW_FLIGHT_SQL`. The BE-side error-path hardening (so any `fetchSplitBatch` failure fails gracefully instead of crashing the BE) is handled separately in apache#64797.
1 parent 99ef194 commit 514da84

10 files changed

Lines changed: 457 additions & 3 deletions

File tree

‎fe/fe-core/src/main/java/org/apache/doris/qe/ConnectContext.java‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -941,6 +941,40 @@ public PlSqlOperation getPlSqlOperation() {
941941
return plSqlOperation;
942942
}
943943

944+
// Arrow Flight SQL only.
945+
// Executors of already-planned queries whose results are produced on the BE and pulled later
946+
// during the DoGet phase. Their coordinators must stay alive until the BE finishes scanning:
947+
// an external-table scan in batch mode lazily fetches splits from the FE (a batch SplitSource
948+
// held by the coordinator's scan nodes), so closing the coordinator at the end of
949+
// GetFlightInfo would release the SplitSource too early and make the BE's fetchSplitBatch fail
950+
// with "Split source X is released". These executors are finalized when the next query starts
951+
// on this connection, or when the connection is torn down. See #62259.
952+
private final List<StmtExecutor> flightSqlDeferredExecutors = new ArrayList<>();
953+
954+
public void addFlightSqlDeferredExecutor(StmtExecutor executor) {
955+
synchronized (flightSqlDeferredExecutors) {
956+
flightSqlDeferredExecutors.add(executor);
957+
}
958+
}
959+
960+
public void closeFlightSqlDeferredExecutors() {
961+
List<StmtExecutor> toClose;
962+
synchronized (flightSqlDeferredExecutors) {
963+
if (flightSqlDeferredExecutors.isEmpty()) {
964+
return;
965+
}
966+
toClose = new ArrayList<>(flightSqlDeferredExecutors);
967+
flightSqlDeferredExecutors.clear();
968+
}
969+
for (StmtExecutor deferredExecutor : toClose) {
970+
try {
971+
deferredExecutor.finalizeArrowFlightQuery();
972+
} catch (Throwable t) {
973+
LOG.warn("failed to finalize deferred arrow flight executor", t);
974+
}
975+
}
976+
}
977+
944978
/**
945979
* This method is idempotent.
946980
*/

‎fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java‎

Lines changed: 43 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -187,6 +187,10 @@ public class StmtExecutor {
187187

188188
@Setter
189189
private volatile Coordinator coord = null;
190+
// Arrow Flight SQL: when true, this query's coordinator is kept alive past GetFlightInfo and
191+
// is finalized later by ConnectContext (see #62259), so the eager close in executeAndSendResult
192+
// is skipped.
193+
private volatile boolean deferredForArrowFlight = false;
190194
private MasterOpExecutor masterOpExecutor = null;
191195
private RedirectStatus redirectStatus = null;
192196
private Planner planner;
@@ -944,6 +948,23 @@ public void finalizeQuery() {
944948
QeProcessorImpl.INSTANCE.unregisterQuery(context.queryId());
945949
}
946950

951+
public boolean isDeferredForArrowFlight() {
952+
return deferredForArrowFlight;
953+
}
954+
955+
// Finalize an Arrow Flight query whose coordinator was kept alive across the
956+
// GetFlightInfo -> DoGet phases: close the coordinator (releasing external-table batch
957+
// SplitSources and the query queue slot) and then unregister the query. See #62259.
958+
public void finalizeArrowFlightQuery() {
959+
try {
960+
if (coord != null) {
961+
coord.close();
962+
}
963+
} finally {
964+
finalizeQuery();
965+
}
966+
}
967+
947968
private void handleQueryWithRetry(TUniqueId queryId) throws Exception {
948969
// queue query here
949970
int retryTime = Config.max_query_retry_time;
@@ -1372,6 +1393,22 @@ public void executeAndSendResult(boolean isOutfileQuery, boolean isSendFields,
13721393
if (context.getConnectType().equals(ConnectType.ARROW_FLIGHT_SQL)) {
13731394
Preconditions.checkState(!context.isReturnResultFromLocal());
13741395
profile.getSummaryProfile().setTempStartTime();
1396+
// Defer closing the coordinator to ConnectContext (closed on the next query or
1397+
// connection teardown) instead of in the finally block below. This gate covers
1398+
// every Arrow Flight query whose results are produced on the BE (coordBase ==
1399+
// coord) -- internal-table and external, batch or not. It is REQUIRED only for an
1400+
// external-table scan in batch mode, where the BE lazily fetches splits from the FE
1401+
// during the later DoGet phase, so closing the coordinator here would release its
1402+
// batch SplitSource too early and break DoGet. Other remote-result queries do not
1403+
// need deferral (the BE buffers their result independently) but are captured by the
1404+
// same gate; the trade-off is their coordinator, query queue slot and query
1405+
// registration stay held until the next query / teardown instead of being released
1406+
// at the end of GetFlightInfo. Point queries use a different coordBase (not
1407+
// deferred). See #62259.
1408+
if (coordBase == coord) {
1409+
deferredForArrowFlight = true;
1410+
context.addFlightSqlDeferredExecutor(this);
1411+
}
13751412
return;
13761413
}
13771414

@@ -1487,7 +1524,12 @@ public void executeAndSendResult(boolean isOutfileQuery, boolean isSendFields,
14871524
this.coord = null;
14881525
throw e;
14891526
} finally {
1490-
coordBase.close();
1527+
// For deferred Arrow Flight queries the coordinator is closed later by ConnectContext
1528+
// (next query / connection teardown), so the BE can still fetch splits during DoGet.
1529+
// See #62259.
1530+
if (!deferredForArrowFlight) {
1531+
coordBase.close();
1532+
}
14911533
}
14921534
}
14931535

‎fe/fe-core/src/main/java/org/apache/doris/service/arrowflight/DorisFlightSqlProducer.java‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -189,6 +189,10 @@ private FlightInfo executeQueryStatement(String peerIdentity, ConnectContext con
189189
try {
190190
Preconditions.checkState(null != connectContext);
191191
Preconditions.checkState(!query.isEmpty());
192+
// Finalize the previous query's coordinator on this connection whose close was
193+
// deferred (Arrow Flight keeps it alive across GetFlightInfo -> DoGet so the BE can
194+
// fetch external-table splits during DoGet). By now the previous DoGet is done. #62259
195+
connectContext.closeFlightSqlDeferredExecutors();
192196
// After the previous query was executed, there was no getStreamStatement to take away the result.
193197
connectContext.getFlightSqlChannel().reset();
194198
connectContext.clearFlightSqlEndpointsLocations();
@@ -280,6 +284,14 @@ private FlightInfo executeQueryStatement(String peerIdentity, ConnectContext con
280284
}
281285
}
282286
} catch (Throwable e) {
287+
// GetFlightInfo failed (e.g. the BE Arrow schema fetch above timed out or returned an
288+
// error) after this query's coordinator may already have been deferred during planning.
289+
// No FlightInfo is returned, so no DoGet will ever pull this query's results; finalize
290+
// the deferred coordinator now (releasing its external-table batch SplitSource, query
291+
// queue slot and query registration) instead of leaking it until the next query starts
292+
// or the connection is torn down. The previous query's deferred coordinator was already
293+
// finalized at the top of this method, so this only closes this failed query. See #62259.
294+
connectContext.closeFlightSqlDeferredExecutors();
283295
String errMsg = "get flight info statement failed, " + e.getMessage() + ", " + Util.getRootCauseMessage(e)
284296
+ ", error code: " + connectContext.getState().getErrorCode() + ", error msg: "
285297
+ connectContext.getState().getErrorMessage();

‎fe/fe-core/src/main/java/org/apache/doris/service/arrowflight/FlightSqlConnectProcessor.java‎

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -195,11 +195,20 @@ public void fetchArrowFlightSchema(int timeoutMs) {
195195
@Override
196196
public void close() throws Exception {
197197
ctx.setCommand(MysqlCommand.COM_SLEEP);
198+
// Executors whose results are pulled from the BE keep their coordinator alive past
199+
// GetFlightInfo (registered as deferred executors on the ConnectContext) so the BE can
200+
// still fetch external-table splits during DoGet. Do NOT finalize those here; they are
201+
// finalized when the next query starts or the connection is torn down. Executors that are
202+
// not deferred (local results, or a query that already failed) are finalized now. See #62259.
198203
for (StmtExecutor asynExecutor : returnResultFromRemoteExecutor) {
199-
asynExecutor.finalizeQuery();
204+
if (!asynExecutor.isDeferredForArrowFlight()) {
205+
asynExecutor.finalizeQuery();
206+
}
200207
}
201208
returnResultFromRemoteExecutor.clear();
202-
executor.finalizeQuery();
209+
if (executor != null && !executor.isDeferredForArrowFlight()) {
210+
executor.finalizeQuery();
211+
}
203212
ctx.clear();
204213
ConnectContext.remove();
205214
}

‎fe/fe-core/src/main/java/org/apache/doris/service/arrowflight/sessions/FlightSqlConnectPoolMgr.java‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,11 @@ public int registerConnection(ConnectContext ctx) {
5555

5656
@Override
5757
public void unregisterConnection(ConnectContext ctx) {
58+
// Finalize any Arrow Flight query whose coordinator was kept alive across the
59+
// GetFlightInfo -> DoGet phases (see #62259), releasing its resources (e.g. external-table
60+
// batch SplitSources and the query queue slot) when the connection is torn down: idle/query
61+
// timeout, bearer token expiry, or explicit CloseSession all reach here.
62+
ctx.closeFlightSqlDeferredExecutors();
5863
ctx.closeTxn();
5964
if (connectionMap.remove(ctx.getConnectionId()) != null) {
6065
numberConnection.decrementAndGet();

‎fe/fe-core/src/test/java/org/apache/doris/qe/ConnectContextTest.java‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -871,4 +871,62 @@ public void testConnectAttributesSetNull() {
871871
Assert.assertNotNull(ctx.getConnectAttributes());
872872
Assert.assertTrue(ctx.getConnectAttributes().isEmpty());
873873
}
874+
875+
// Arrow Flight SQL keeps a query's coordinator alive across GetFlightInfo -> DoGet (see #62259).
876+
// closeFlightSqlDeferredExecutors() is the single place that releases those deferred coordinators
877+
// (and with them the external-table batch SplitSource and the query queue slot). The following
878+
// tests pin the leak-prevention contract of that method: every deferred executor is finalized,
879+
// the list is cleared so nothing is finalized twice or retained, and one failing executor does
880+
// not strand the others' resources.
881+
882+
@Test
883+
public void testCloseFlightSqlDeferredExecutorsFinalizesEachExecutor() {
884+
ConnectContext ctx = new ConnectContext();
885+
StmtExecutor deferred1 = Mockito.mock(StmtExecutor.class);
886+
StmtExecutor deferred2 = Mockito.mock(StmtExecutor.class);
887+
ctx.addFlightSqlDeferredExecutor(deferred1);
888+
ctx.addFlightSqlDeferredExecutor(deferred2);
889+
890+
ctx.closeFlightSqlDeferredExecutors();
891+
892+
// Both deferred coordinators must be finalized, otherwise their SplitSource and query queue
893+
// slot leak after the DoGet phase.
894+
Mockito.verify(deferred1).finalizeArrowFlightQuery();
895+
Mockito.verify(deferred2).finalizeArrowFlightQuery();
896+
}
897+
898+
@Test
899+
public void testCloseFlightSqlDeferredExecutorsClearsListSoSecondCallIsNoOp() {
900+
ConnectContext ctx = new ConnectContext();
901+
StmtExecutor deferred = Mockito.mock(StmtExecutor.class);
902+
ctx.addFlightSqlDeferredExecutor(deferred);
903+
904+
// More than one teardown path can fire for the same connection (e.g. the next query cleans
905+
// up, then the connection is later torn down). The list must be cleared after the first
906+
// call so the executor is finalized exactly once and is not retained (leaked) afterwards.
907+
ctx.closeFlightSqlDeferredExecutors();
908+
ctx.closeFlightSqlDeferredExecutors();
909+
910+
Mockito.verify(deferred, Mockito.times(1)).finalizeArrowFlightQuery();
911+
}
912+
913+
@Test
914+
public void testCloseFlightSqlDeferredExecutorsFinalizesRemainingWhenOneFails() {
915+
ConnectContext ctx = new ConnectContext();
916+
StmtExecutor failing = Mockito.mock(StmtExecutor.class);
917+
StmtExecutor healthy = Mockito.mock(StmtExecutor.class);
918+
Mockito.doThrow(new RuntimeException("finalize failed")).when(failing).finalizeArrowFlightQuery();
919+
ctx.addFlightSqlDeferredExecutor(failing);
920+
ctx.addFlightSqlDeferredExecutor(healthy);
921+
922+
// A single bad coordinator must not abort the cleanup: the call must not throw, and the
923+
// healthy executor must still be finalized so its resources are released.
924+
ctx.closeFlightSqlDeferredExecutors();
925+
Mockito.verify(healthy).finalizeArrowFlightQuery();
926+
927+
// The list is cleared up front, so neither executor is reprocessed on a later teardown.
928+
ctx.closeFlightSqlDeferredExecutors();
929+
Mockito.verify(failing, Mockito.times(1)).finalizeArrowFlightQuery();
930+
Mockito.verify(healthy, Mockito.times(1)).finalizeArrowFlightQuery();
931+
}
874932
}

‎fe/fe-core/src/test/java/org/apache/doris/qe/StmtExecutorTest.java‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,8 @@
2626
import org.apache.doris.mysql.MysqlSerializer;
2727
import org.apache.doris.qe.CommonResultSet.CommonResultSetMetaData;
2828
import org.apache.doris.qe.ConnectContext.ConnectType;
29+
import org.apache.doris.thrift.TQueryOptions;
30+
import org.apache.doris.thrift.TUniqueId;
2931
import org.apache.doris.utframe.TestWithFeService;
3032

3133
import com.google.common.collect.Lists;
@@ -66,6 +68,40 @@ public void testShowNull() throws Exception {
6668
Assert.assertEquals(QueryState.MysqlStateType.OK, connectContext.getState().getStateType());
6769
}
6870

71+
// Arrow Flight SQL keeps a query's coordinator alive across GetFlightInfo -> DoGet (see #62259);
72+
// it is released later by finalizeArrowFlightQuery(), which closes the coordinator and then
73+
// unregisters the query. The close and the unregister must be independent: if coord.close()
74+
// throws, the query registration must still be released (the try/finally), otherwise the query
75+
// leaks in QeProcessorImpl forever. The thrown error is expected to propagate to the caller
76+
// (ConnectContext.closeFlightSqlDeferredExecutors), which catches and logs it.
77+
@Test
78+
public void testFinalizeArrowFlightQueryUnregistersQueryEvenIfCoordCloseThrows() throws Exception {
79+
StmtExecutor stmtExecutor = new StmtExecutor(connectContext, "");
80+
TUniqueId queryId = new TUniqueId(0x6226259L, 0x62259L);
81+
connectContext.setQueryId(queryId);
82+
83+
Coordinator coord = Mockito.mock(Coordinator.class);
84+
Mockito.when(coord.getQueryOptions()).thenReturn(new TQueryOptions());
85+
Mockito.doThrow(new RuntimeException("coord close failed")).when(coord).close();
86+
stmtExecutor.setCoord(coord);
87+
88+
// Simulate the in-flight query whose results DoGet is still pulling.
89+
QeProcessorImpl.INSTANCE.registerQuery(queryId, new QeProcessorImpl.QueryInfo(coord));
90+
Assert.assertNotNull(QeProcessorImpl.INSTANCE.getCoordinator(queryId));
91+
92+
try {
93+
stmtExecutor.finalizeArrowFlightQuery();
94+
Assert.fail("expected coord.close() failure to propagate after the query is unregistered");
95+
} catch (RuntimeException e) {
96+
Assert.assertEquals("coord close failed", e.getMessage());
97+
}
98+
99+
// The coordinator close was attempted (releases SplitSource + query queue slot) ...
100+
Mockito.verify(coord).close();
101+
// ... and despite it failing, the query registration was still released (no leak).
102+
Assert.assertNull(QeProcessorImpl.INSTANCE.getCoordinator(queryId));
103+
}
104+
69105
@Test
70106
public void testKill() throws Exception {
71107
StmtExecutor stmtExecutor = new StmtExecutor(connectContext, "");

0 commit comments

Comments
 (0)