Skip to content

Commit 765185a

Browse files
branch-4.1: [fix](arrow-flight) Release a finished Flight query's coordinator instead of holding it until wait_timeout #67504 (#67558)
Cherry-picked from #67504 ### Backport notes Two conflicts, both resolved without changing behaviour: - `Config`: branch-4.1 still has the bilingual `autobucket_partition_size_per_bucket_gb` description that master rewrote, so that entry is kept as it is here and the new `arrow_flight_deferred_query_idle_timeout_second` is inserted above it, verbatim. - `StmtExecutor`: the comment above the deferral gate is taken from upstream verbatim, so it already mentions #67368. That fix is being backported separately (see below); the gate itself does not depend on it. One adaptation: `SplitAssignment` is `org.apache.doris.datasource.SplitAssignment` on this branch, not `org.apache.doris.datasource.split.SplitAssignment`, so `ArrowFlightDeferralGateTest` imports it from its package here. Everything else -- `ScanNode.hasBatchSplitSource`, `Coordinator.hasBatchSplitSource`, the `ConnectContext` idle reaper, `FlightSqlConnectProcessor`, `StmtExecutorTest`, `FlightSqlDeferredQueryIdleTimeoutTest` and both regression suites -- applied cleanly. ### Merge order Touches the same `StmtExecutor` comment as #67557 (the branch-4.1 backport of #67487). Merging that one first keeps this one conflict-free. ### Local verification `./build.sh --fe` on this branch: **BUILD SUCCESS** (2:26 min), no errors, checkstyle clean on every module -- `fe-common` (the new `Config` entry) and `fe-core` (the five main-source changes plus `ArrowFlightDeferralGateTest`, `FlightSqlDeferredQueryIdleTimeoutTest` and the new `StmtExecutorTest` case) included. The regression suites were not run locally. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01DzUYFcGHQH3bnLGCjpncVj Co-authored-by: Raghvendra Singh <[email protected]>
1 parent ee162ae commit 765185a

11 files changed

Lines changed: 544 additions & 23 deletions

File tree

‎fe/fe-common/src/main/java/org/apache/doris/common/Config.java‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3020,6 +3020,17 @@ public class Config extends ConfigBase {
30203020
"Maximal number of connections of Arrow Flight Server per FE."})
30213021
public static int arrow_flight_max_connections = 4096;
30223022

3023+
@ConfField(mutable = true, description = "Arrow Flight SQL only. A query that scans an external table in "
3024+
+ "batch mode keeps its FE coordinator alive after GetFlightInfo, so the BE can keep fetching splits "
3025+
+ "while the client pulls the results (DoGet); that coordinator is normally released when the "
3026+
+ "session runs its next query or is closed. Most Flight clients never close a session, so the "
3027+
+ "coordinator, and with it the query's workload group queue slot and its active_queries entry, "
3028+
+ "would otherwise stay held until wait_timeout. If the session stays idle for longer than this "
3029+
+ "many seconds after the query started, the coordinator is released anyway. The bound is never "
3030+
+ "shorter than the query's own execution timeout, and the session itself is not killed "
3031+
+ "(wait_timeout still governs that). 0 disables the bound.")
3032+
public static int arrow_flight_deferred_query_idle_timeout_second = 3600;
3033+
30233034
@ConfField(mutable = true, masterOnly = true, description = {
30243035
"Auto Buckets 中按照 partition size 去估算 bucket 数,存算一体 partition size 5G 估算一个 bucket,"
30253036
+ "但存算分离下 partition size 10G 估算一个 bucket。若配置小于 0,会在在代码中会自适应存算一体模式默认 5G,在存算分离默认 10G",

‎fe/fe-core/src/main/java/org/apache/doris/planner/ScanNode.java‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -140,6 +140,17 @@ public TupleDescriptor getTupleDesc() {
140140
return desc;
141141
}
142142

143+
/**
144+
* Whether this scan hands out its splits lazily through a batch {@link SplitSource} that the
145+
* BE fetches from the FE while it is scanning (external-table batch mode, see
146+
* {@link SplitGenerator#isBatchMode()}). Such a scan needs its coordinator alive until the BE
147+
* has finished scanning, even after the FE is done dispatching the query: closing the
148+
* coordinator releases the split source ({@link #stop()}) and the BE's next split fetch fails.
149+
*/
150+
public boolean hasBatchSplitSource() {
151+
return splitAssignment != null;
152+
}
153+
143154
protected abstract void createScanRangeLocations() throws UserException;
144155

145156
/**

‎fe/fe-core/src/main/java/org/apache/doris/qe/ConnectContext.java‎

Lines changed: 44 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1043,7 +1043,9 @@ public PlSqlOperation getPlSqlOperation() {
10431043
// held by the coordinator's scan nodes), so closing the coordinator at the end of
10441044
// GetFlightInfo would release the SplitSource too early and make the BE's fetchSplitBatch fail
10451045
// with "Split source X is released". These executors are finalized when the next query starts
1046-
// on this connection, or when the connection is torn down. See #62259.
1046+
// on this connection, when the connection is torn down, or by the idle reaper in checkTimeout
1047+
// once the connection has been sleeping for arrow_flight_deferred_query_idle_timeout_second.
1048+
// See #62259 and #67503.
10471049
private final List<StmtExecutor> flightSqlDeferredExecutors = new ArrayList<>();
10481050

10491051
public void addFlightSqlDeferredExecutor(StmtExecutor executor) {
@@ -1070,6 +1072,45 @@ public void closeFlightSqlDeferredExecutors() {
10701072
}
10711073
}
10721074

1075+
/**
1076+
* How long, in seconds, a sleeping connection may keep its deferred Arrow Flight executors
1077+
* before the timeout checker finalizes them without killing the connection
1078+
* (Config.arrow_flight_deferred_query_idle_timeout_second). A Flight client that opens a
1079+
* session per query and never closes it would otherwise pin each deferred query's query queue
1080+
* slot and query registration until wait_timeout (8h by default). The bound is never shorter
1081+
* than the execution timeout the deferred query was run with: the client may still be pulling
1082+
* that query's results from the BE, which still needs the batch split source the coordinator
1083+
* holds. Returns -1 when the bound is disabled or nothing is deferred.
1084+
*/
1085+
public long getFlightSqlDeferredExecutorsIdleTimeoutS() {
1086+
int configTimeoutS = Config.arrow_flight_deferred_query_idle_timeout_second;
1087+
if (configTimeoutS <= 0) {
1088+
return -1;
1089+
}
1090+
long execTimeoutS = -1;
1091+
synchronized (flightSqlDeferredExecutors) {
1092+
if (flightSqlDeferredExecutors.isEmpty()) {
1093+
return -1;
1094+
}
1095+
for (StmtExecutor deferredExecutor : flightSqlDeferredExecutors) {
1096+
execTimeoutS = Math.max(execTimeoutS, deferredExecutor.getDeferredExecTimeoutS());
1097+
}
1098+
}
1099+
return Math.max(configTimeoutS, execTimeoutS);
1100+
}
1101+
1102+
// Called by the timeout checker for a sleeping connection that is not past wait_timeout yet.
1103+
private void reapIdleFlightSqlDeferredExecutors(long idleMs) {
1104+
long timeoutS = getFlightSqlDeferredExecutorsIdleTimeoutS();
1105+
if (timeoutS < 0 || idleMs <= timeoutS * 1000L) {
1106+
return;
1107+
}
1108+
LOG.warn("release deferred arrow flight query of idle connection, connectionId: {}, remote: {}, "
1109+
+ "idle: {}ms, idle timeout: {}s",
1110+
connectionId, getRemoteHostPortString(), idleMs, timeoutS);
1111+
closeFlightSqlDeferredExecutors();
1112+
}
1113+
10731114
/**
10741115
* This method is idempotent.
10751116
*/
@@ -1342,6 +1383,8 @@ public void checkTimeout(long now) {
13421383
// Need kill this connection.
13431384
killFlag = true;
13441385
killConnection = true;
1386+
} else {
1387+
reapIdleFlightSqlDeferredExecutors(delta);
13451388
}
13461389
} else {
13471390
String timeoutTag = "query";

‎fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -816,6 +816,25 @@ public void exec() throws Exception {
816816
execInternal();
817817
}
818818

819+
/**
820+
* Whether the BE keeps calling back into this coordinator after {@link #exec()} returned: an
821+
* external-table scan in batch mode fetches its splits lazily from the split source that its
822+
* scan node holds, so the coordinator must not be closed until the BE has finished scanning.
823+
* Arrow Flight SQL uses this to decide whether a query's coordinator has to outlive
824+
* GetFlightInfo, the client pulling the results from the BE later in DoGet. See #62259.
825+
*/
826+
public boolean hasBatchSplitSource() {
827+
if (scanNodes == null) {
828+
return false;
829+
}
830+
for (ScanNode scanNode : scanNodes) {
831+
if (scanNode.hasBatchSplitSource()) {
832+
return true;
833+
}
834+
}
835+
return false;
836+
}
837+
819838
@Override
820839
public void close() {
821840
// NOTE: all close method should be no exception

‎fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java‎

Lines changed: 34 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -194,6 +194,10 @@ public class StmtExecutor {
194194
// is finalized later by ConnectContext (see #62259), so the eager close in executeAndSendResult
195195
// is skipped.
196196
private volatile boolean deferredForArrowFlight = false;
197+
// The execution timeout in effect when the coordinator was deferred. Captured at that moment
198+
// because per-statement SET_VAR values are reverted at the end of execute(), so reading
199+
// ConnectContext.getExecTimeoutS() later would report the session value instead.
200+
private volatile int deferredExecTimeoutS = -1;
197201
private MasterOpExecutor masterOpExecutor = null;
198202
private RedirectStatus redirectStatus = null;
199203
private Planner planner;
@@ -980,6 +984,21 @@ public boolean isDeferredForArrowFlight() {
980984
return deferredForArrowFlight;
981985
}
982986

987+
// Execution timeout (seconds) the deferred query was run with; -1 when the query is not deferred.
988+
public int getDeferredExecTimeoutS() {
989+
return deferredExecTimeoutS;
990+
}
991+
992+
// Keep this query's coordinator alive past GetFlightInfo (see the gate in executeAndSendResult)
993+
// and hand it to the ConnectContext, which finalizes it later. Records the execution timeout in
994+
// effect right now: it floors the idle reaper's bound and must be the value the query actually
995+
// ran with, not the session value left behind after SET_VAR hints are reverted.
996+
void deferForArrowFlight() {
997+
deferredForArrowFlight = true;
998+
deferredExecTimeoutS = context.getExecTimeoutS();
999+
context.addFlightSqlDeferredExecutor(this);
1000+
}
1001+
9831002
// Finalize an Arrow Flight query whose coordinator was kept alive across the
9841003
// GetFlightInfo -> DoGet phases: close the coordinator (releasing external-table batch
9851004
// SplitSources and the query queue slot) and then unregister the query. See #62259.
@@ -1435,23 +1454,21 @@ public void executeAndSendResult(boolean isOutfileQuery, boolean isSendFields,
14351454
if (context.getConnectType().equals(ConnectType.ARROW_FLIGHT_SQL)) {
14361455
Preconditions.checkState(!context.isReturnResultFromLocal());
14371456
profile.getSummaryProfile().setTempStartTime();
1438-
// Defer closing the coordinator to ConnectContext (closed on the next query or
1439-
// connection teardown) instead of in the finally block below. This gate covers
1440-
// every Arrow Flight query whose results are produced on the BE (coordBase ==
1441-
// coord) -- internal-table and external, batch or not. It is REQUIRED only for an
1442-
// external-table scan in batch mode, where the BE lazily fetches splits from the FE
1443-
// during the later DoGet phase, so closing the coordinator here would release its
1444-
// batch SplitSource too early and break DoGet. Other remote-result queries do not
1445-
// need deferral (the BE buffers their result independently) but are captured by the
1446-
// same gate; the trade-off is their coordinator, query queue slot and query
1447-
// registration stay held until the next query / teardown instead of being released
1448-
// at the end of GetFlightInfo. A short-circuit point query is the one case with a
1449-
// different coordBase, and it can no longer reach here: it has no Arrow result on
1450-
// either side, so LogicalResultSinkToShortCircuitPointQuery keeps Arrow Flight SQL
1451-
// on the normal execution path. See #62259 and #67368.
1452-
if (coordBase == coord) {
1453-
deferredForArrowFlight = true;
1454-
context.addFlightSqlDeferredExecutor(this);
1457+
// The client pulls the results from the BE later (DoGet). Only an external-table
1458+
// scan in batch mode still needs the coordinator after this point: the BE fetches
1459+
// its splits lazily from the split source the coordinator holds, so closing the
1460+
// coordinator here would release that source too early and break DoGet (#62259).
1461+
// Such a coordinator is closed later by ConnectContext: on the session's next
1462+
// query, on teardown, or by the idle reaper in checkTimeout. The trade-off is that
1463+
// its query queue slot and query registration stay held until then. Every other
1464+
// query closes its coordinator in the finally block below and releases both right
1465+
// away, the BE buffering its results independently of the coordinator (#67503).
1466+
// A short-circuit point query is the one case with a different coordBase, and it
1467+
// can no longer reach here: it has no Arrow result on either side, so
1468+
// LogicalResultSinkToShortCircuitPointQuery keeps Arrow Flight SQL on the normal
1469+
// execution path (#67368).
1470+
if (coordBase == coord && coord.hasBatchSplitSource()) {
1471+
deferForArrowFlight();
14551472
}
14561473
return;
14571474
}

‎fe/fe-core/src/main/java/org/apache/doris/service/arrowflight/FlightSqlConnectProcessor.java‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -196,11 +196,12 @@ public void fetchArrowFlightSchema(int timeoutMs) {
196196
@Override
197197
public void close() throws Exception {
198198
ctx.setCommand(MysqlCommand.COM_SLEEP);
199-
// Executors whose results are pulled from the BE keep their coordinator alive past
200-
// GetFlightInfo (registered as deferred executors on the ConnectContext) so the BE can
201-
// still fetch external-table splits during DoGet. Do NOT finalize those here; they are
202-
// finalized when the next query starts or the connection is torn down. Executors that are
203-
// not deferred (local results, or a query that already failed) are finalized now. See #62259.
199+
// An external-table scan in batch mode keeps its coordinator alive past GetFlightInfo
200+
// (registered as a deferred executor on the ConnectContext) so the BE can still fetch its
201+
// splits during DoGet. Do NOT finalize those here; they are finalized when the next query
202+
// starts, when the connection is torn down, or by the idle reaper in
203+
// ConnectContext.checkTimeout. Every other executor (local results, results the BE buffers
204+
// on its own, or a query that already failed) is finalized now. See #62259 and #67503.
204205
for (StmtExecutor asynExecutor : returnResultFromRemoteExecutor) {
205206
if (!asynExecutor.isDeferredForArrowFlight()) {
206207
asynExecutor.finalizeQuery();
Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
package org.apache.doris.qe;
19+
20+
import org.apache.doris.analysis.DescriptorTable;
21+
import org.apache.doris.datasource.SplitAssignment;
22+
import org.apache.doris.planner.PlanFragment;
23+
import org.apache.doris.planner.ScanNode;
24+
import org.apache.doris.thrift.TUniqueId;
25+
26+
import com.google.common.collect.Lists;
27+
import org.junit.jupiter.api.Assertions;
28+
import org.junit.jupiter.api.Test;
29+
import org.mockito.Mockito;
30+
31+
import java.lang.reflect.Field;
32+
import java.util.List;
33+
34+
/**
35+
* The predicate behind the Arrow Flight deferral gate in StmtExecutor.executeAndSendResult (#67503):
36+
* a coordinator has to outlive GetFlightInfo only when one of its scans still hands out splits to
37+
* the BE lazily, i.e. an external-table scan in batch mode holding a batch split source (#62259).
38+
*/
39+
public class ArrowFlightDeferralGateTest {
40+
41+
private static ScanNode scanNode(boolean batchSplitSource) throws Exception {
42+
ScanNode node = Mockito.mock(ScanNode.class, Mockito.CALLS_REAL_METHODS);
43+
if (batchSplitSource) {
44+
// FileQueryScanNode.createScanRangeLocations sets this only in batch mode.
45+
Field field = ScanNode.class.getDeclaredField("splitAssignment");
46+
field.setAccessible(true);
47+
field.set(node, Mockito.mock(SplitAssignment.class));
48+
}
49+
return node;
50+
}
51+
52+
private static Coordinator coordinator(List<ScanNode> scanNodes) {
53+
return new Coordinator(1L, new TUniqueId(1L, 2L), new DescriptorTable(), Lists.<PlanFragment>newArrayList(),
54+
scanNodes, "UTC", false, false);
55+
}
56+
57+
@Test
58+
public void testScanNodeHasBatchSplitSourceOnlyWhenSplitsAreHandedOutLazily() throws Exception {
59+
Assertions.assertFalse(scanNode(false).hasBatchSplitSource());
60+
Assertions.assertTrue(scanNode(true).hasBatchSplitSource());
61+
}
62+
63+
@Test
64+
public void testCoordinatorHasBatchSplitSourceIfAnyScanDoes() throws Exception {
65+
Assertions.assertFalse(coordinator(Lists.newArrayList()).hasBatchSplitSource());
66+
Assertions.assertFalse(coordinator(Lists.newArrayList(scanNode(false), scanNode(false))).hasBatchSplitSource());
67+
Assertions.assertTrue(coordinator(Lists.newArrayList(scanNode(false), scanNode(true))).hasBatchSplitSource());
68+
}
69+
}

‎fe/fe-core/src/test/java/org/apache/doris/qe/StmtExecutorTest.java‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,39 @@ public void testShowNull() throws Exception {
7373
Assert.assertEquals(QueryState.MysqlStateType.OK, connectContext.getState().getStateType());
7474
}
7575

76+
// The deferral gate (#67503): a coordinator is kept alive past GetFlightInfo only when the BE
77+
// still fetches splits from it (Coordinator.hasBatchSplitSource), and the execution timeout it
78+
// ran with is frozen at that moment. SET_VAR hint values are reverted when execute() ends, so
79+
// the idle reaper must not read the session value later.
80+
@Test
81+
public void testDeferForArrowFlightFreezesExecTimeoutInEffect() throws Exception {
82+
int savedQueryTimeout = connectContext.getSessionVariable().getQueryTimeoutS();
83+
int savedIdleTimeout = Config.arrow_flight_deferred_query_idle_timeout_second;
84+
connectContext.setQueryId(new TUniqueId(0x67503L, 0x1L));
85+
try {
86+
Config.arrow_flight_deferred_query_idle_timeout_second = 1;
87+
connectContext.getSessionVariable().setQueryTimeoutS(1234);
88+
StmtExecutor stmtExecutor = new StmtExecutor(connectContext, "");
89+
Assertions.assertFalse(stmtExecutor.isDeferredForArrowFlight());
90+
Assertions.assertEquals(-1, stmtExecutor.getDeferredExecTimeoutS());
91+
92+
stmtExecutor.deferForArrowFlight();
93+
94+
Assertions.assertTrue(stmtExecutor.isDeferredForArrowFlight());
95+
Assertions.assertEquals(1234, stmtExecutor.getDeferredExecTimeoutS());
96+
// the reaper's bound is floored at the frozen value ...
97+
Assertions.assertEquals(1234L, connectContext.getFlightSqlDeferredExecutorsIdleTimeoutS());
98+
// ... even after the session value moved on, as it does when a SET_VAR hint is reverted
99+
connectContext.getSessionVariable().setQueryTimeoutS(5);
100+
Assertions.assertEquals(1234, stmtExecutor.getDeferredExecTimeoutS());
101+
Assertions.assertEquals(1234L, connectContext.getFlightSqlDeferredExecutorsIdleTimeoutS());
102+
} finally {
103+
connectContext.closeFlightSqlDeferredExecutors();
104+
connectContext.getSessionVariable().setQueryTimeoutS(savedQueryTimeout);
105+
Config.arrow_flight_deferred_query_idle_timeout_second = savedIdleTimeout;
106+
}
107+
}
108+
76109
// Arrow Flight SQL keeps a query's coordinator alive across GetFlightInfo -> DoGet (see #62259);
77110
// it is released later by finalizeArrowFlightQuery(), which closes the coordinator and then
78111
// unregisters the query. The close and the unregister must be independent: if coord.close()

0 commit comments

Comments
 (0)