Skip to content

Commit 8608b8e

Browse files
morningmanclaude
andcommitted
[fix](remote-doris) Close the Flight SQL session a remote Doris scan opens on the remote frontend
A Remote Doris catalog with use_arrow_flight reads the other cluster over Arrow Flight SQL: for every scan RemoteDorisScanNode performs a handshake against a remote frontend, runs the query there and hands the endpoints to the BE, which reads the rows from the remote BEs with DoGet. The handshake opens a session on the remote frontend, and the scan closed the gRPC channel but never sent CloseSession, so every scan left one session behind until wait_timeout. Since apache#68101 a Flight session is a connection of the remote frontend's one pool, counted against the catalog user's max_user_connections; since apache#68266 nothing caps them any more. A hundred scans within eight hours and the catalog user, MySQL clients included, is refused there with "Reach limit of connections". The external regression hit exactly that on 2026-09-21: the remote_doris suites point the catalog at the frontend under test as root, and 48 leaked sessions took half of root's quota. RemoteDorisFlightSession is the session as an object: handshake, execute, and close() = CloseSession (bounded to five seconds, so a remote frontend that stopped answering cannot hang the local query's teardown) followed by the channel and the allocator. RemoteDorisScanNode keeps it from getSplits until stop(), which the coordinator calls when the local query closes or is cancelled - when the BE is done with the remote query's endpoints. It cannot be closed right after GetFlightInfo: the remote frontend cancels whatever a closed session was still running, and when the remote table is itself an external table in batch mode the remote query is deferred there while the BE reads. A query that fails closes its session at once, so the retry on the next node leaves nothing behind either. For the same reason the local coordinator has to outlive dispatch when the local query is an Arrow Flight SQL query, or apache#67503 would close it, and the session with it, right after exec() while the local BE may still be reading. The deferral gate's predicate is generalized from "has a batch split source" to "the BE still depends on this scan after dispatch": ScanNode.coordinatorMustOutliveDispatch() and Coordinator.mustOutliveDispatch(), with the open session as the second reason. Batch-mode external scans behave as before. RemoteDorisScanNodeTest drives the scan node against an in-process Flight SQL server that counts the sessions it is asked to close; the regression suite test_remote_doris_flight_session scans through a catalog logging in as a user of its own and asserts after each scan that the processlist holds no ArrowFlightSQL session of that user. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
1 parent 69b803b commit 8608b8e

9 files changed

Lines changed: 586 additions & 79 deletions

File tree

Lines changed: 153 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,153 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
package org.apache.doris.datasource.doris.source;
19+
20+
import org.apache.doris.common.Pair;
21+
import org.apache.doris.common.UserException;
22+
23+
import com.google.common.annotations.VisibleForTesting;
24+
import org.apache.arrow.flight.CallOptions;
25+
import org.apache.arrow.flight.CloseSessionRequest;
26+
import org.apache.arrow.flight.FlightClient;
27+
import org.apache.arrow.flight.FlightInfo;
28+
import org.apache.arrow.flight.Location;
29+
import org.apache.arrow.flight.grpc.CredentialCallOption;
30+
import org.apache.arrow.flight.sql.FlightSqlClient;
31+
import org.apache.arrow.memory.BufferAllocator;
32+
import org.apache.arrow.memory.RootAllocator;
33+
import org.apache.logging.log4j.LogManager;
34+
import org.apache.logging.log4j.Logger;
35+
36+
import java.io.Closeable;
37+
import java.net.URI;
38+
import java.util.Optional;
39+
import java.util.concurrent.TimeUnit;
40+
41+
/**
42+
* The Flight SQL session a {@link RemoteDorisScanNode} opens on a remote Doris frontend for one
43+
* scan, and ends with a CloseSession once the scan is over.
44+
*
45+
* <p>The handshake ({@code authenticateBasicToken}) opens a session on the remote frontend: a
46+
* connection in its pool, counted against {@code qe_max_connection}, the Arrow Flight SQL sub-quota
47+
* and the catalog user's {@code max_user_connections}, that only a CloseSession, a KILL or
48+
* {@code wait_timeout} ends. Closing the gRPC channel does not. A session opened per scan and never
49+
* closed therefore stays for hours and, one scan at a time, exhausts the catalog user's connection
50+
* quota on the remote frontend - refusing that user's MySQL connections there as well.
51+
*
52+
* <p>The session outlives GetFlightInfo on purpose: the query it ran serves the BE's DoGet of the
53+
* endpoints, and the remote frontend cancels whatever a closed session was still running - the
54+
* query itself, when the remote table is an external table scanned in batch mode and the query is
55+
* therefore deferred there. So {@link #close()} is called from {@link RemoteDorisScanNode#stop()},
56+
* when the coordinator of the local query closes, and that coordinator is kept alive until the BE
57+
* has finished scanning ({@link RemoteDorisScanNode#coordinatorMustOutliveDispatch()}).
58+
*/
59+
class RemoteDorisFlightSession implements Closeable {
60+
private static final Logger LOG = LogManager.getLogger(RemoteDorisFlightSession.class);
61+
62+
// A bound on the CloseSession round trip. close() runs on the local query's teardown path,
63+
// which must not hang on a remote frontend that has stopped answering; the session is then left
64+
// to the remote frontend's wait_timeout, as every session was before this class existed.
65+
@VisibleForTesting
66+
static final int CLOSE_SESSION_TIMEOUT_SECONDS = 5;
67+
68+
private final Pair<String, Integer> hostAndPort;
69+
private final BufferAllocator allocator;
70+
private final FlightSqlClient client;
71+
private final CredentialCallOption credential;
72+
private boolean closed = false;
73+
74+
private RemoteDorisFlightSession(Pair<String, Integer> hostAndPort, BufferAllocator allocator,
75+
FlightSqlClient client, CredentialCallOption credential) {
76+
this.hostAndPort = hostAndPort;
77+
this.allocator = allocator;
78+
this.client = client;
79+
this.credential = credential;
80+
}
81+
82+
/**
83+
* Opens a session on the remote frontend at {@code hostAndPort} with the catalog's credentials.
84+
* Nothing is left behind when this fails: a handshake that was refused opened no session, and
85+
* the channel and allocator are released before the exception propagates.
86+
*/
87+
static RemoteDorisFlightSession open(Pair<String, Integer> hostAndPort, String user, String password)
88+
throws Exception {
89+
BufferAllocator allocator = new RootAllocator();
90+
FlightClient flightClient = null;
91+
try {
92+
URI uri = new URI("grpc", null, hostAndPort.first, hostAndPort.second, null, null, null);
93+
flightClient = FlightClient.builder(allocator, new Location(uri)).build();
94+
Optional<CredentialCallOption> credential = flightClient.authenticateBasicToken(user, password);
95+
if (!credential.isPresent()) {
96+
throw new UserException("Authenticates with a username and password failure");
97+
}
98+
return new RemoteDorisFlightSession(hostAndPort, allocator, new FlightSqlClient(flightClient),
99+
credential.get());
100+
} catch (Throwable t) {
101+
closeQuietly(flightClient, allocator, hostAndPort);
102+
throw t;
103+
}
104+
}
105+
106+
/** Runs {@code sql} on the remote frontend; the endpoints of the result are where the BE reads it. */
107+
FlightInfo execute(String sql, int timeoutSec) {
108+
return client.execute(sql, credential, CallOptions.timeout(timeoutSec, TimeUnit.SECONDS));
109+
}
110+
111+
Pair<String, Integer> getHostAndPort() {
112+
return hostAndPort;
113+
}
114+
115+
/**
116+
* Ends the session on the remote frontend (CloseSession), then releases the channel and the
117+
* allocator. Never throws: this runs on the local query's teardown path, and a session the
118+
* remote frontend could not be told to close is only left to its wait_timeout. Idempotent.
119+
*/
120+
@Override
121+
public synchronized void close() {
122+
if (closed) {
123+
return;
124+
}
125+
closed = true;
126+
try {
127+
client.closeSession(new CloseSessionRequest(), credential,
128+
CallOptions.timeout(CLOSE_SESSION_TIMEOUT_SECONDS, TimeUnit.SECONDS));
129+
} catch (Throwable t) {
130+
LOG.warn("failed to close the Arrow Flight SQL session on remote Doris frontend {}:{}, it stays open there"
131+
+ " until its wait_timeout", hostAndPort.first, hostAndPort.second, t);
132+
}
133+
closeQuietly(client, allocator, hostAndPort);
134+
}
135+
136+
private static void closeQuietly(AutoCloseable client, BufferAllocator allocator,
137+
Pair<String, Integer> hostAndPort) {
138+
try {
139+
if (client != null) {
140+
client.close();
141+
}
142+
} catch (Throwable t) {
143+
LOG.warn("failed to close the Arrow Flight client to remote Doris frontend {}:{}",
144+
hostAndPort.first, hostAndPort.second, t);
145+
}
146+
try {
147+
allocator.close();
148+
} catch (Throwable t) {
149+
LOG.warn("failed to close the Arrow allocator of the Flight client to remote Doris frontend {}:{}",
150+
hostAndPort.first, hostAndPort.second, t);
151+
}
152+
}
153+
}

‎fe/fe-core/src/main/java/org/apache/doris/datasource/doris/source/RemoteDorisScanNode.java‎

Lines changed: 82 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -44,29 +44,21 @@
4444
import org.apache.doris.thrift.TRemoteDorisFileDesc;
4545
import org.apache.doris.thrift.TTableFormatFileDesc;
4646

47+
import com.google.common.annotations.VisibleForTesting;
4748
import com.google.common.base.Joiner;
4849
import com.google.common.collect.Lists;
49-
import org.apache.arrow.flight.CallOptions;
50-
import org.apache.arrow.flight.FlightClient;
5150
import org.apache.arrow.flight.FlightEndpoint;
5251
import org.apache.arrow.flight.FlightInfo;
5352
import org.apache.arrow.flight.Location;
54-
import org.apache.arrow.flight.grpc.CredentialCallOption;
55-
import org.apache.arrow.flight.sql.FlightSqlClient;
56-
import org.apache.arrow.memory.BufferAllocator;
57-
import org.apache.arrow.memory.RootAllocator;
5853
import org.apache.logging.log4j.LogManager;
5954
import org.apache.logging.log4j.Logger;
6055

61-
import java.net.URI;
6256
import java.nio.ByteBuffer;
6357
import java.util.ArrayList;
6458
import java.util.HashSet;
6559
import java.util.List;
6660
import java.util.Map;
67-
import java.util.Optional;
6861
import java.util.Set;
69-
import java.util.concurrent.TimeUnit;
7062
import java.util.stream.Collectors;
7163

7264
public class RemoteDorisScanNode extends FileQueryScanNode {
@@ -79,6 +71,13 @@ public class RemoteDorisScanNode extends FileQueryScanNode {
7971

8072
private RemoteDorisSource source;
8173

74+
// The Flight SQL session this scan opened on the remote frontend, from getSplits until stop()
75+
// closes it (see RemoteDorisFlightSession for why it must live that long and no longer). Both
76+
// guarded by this: stop() may run on another thread than the one that planned the query - a
77+
// KILL, the timeout checker - and more than once (cancel, then close).
78+
private RemoteDorisFlightSession flightSession;
79+
private boolean stopped;
80+
8281
public RemoteDorisScanNode(PlanNodeId id, TupleDescriptor desc, boolean needCheckColumnPriv,
8382
SessionVariable sv, ScanContext scanContext) {
8483
super(id, desc, "REMOTE_DORIS_SCAN_NODE", scanContext, needCheckColumnPriv, sv);
@@ -177,7 +176,8 @@ private List<Pair<String, ByteBuffer>> executeQuery() {
177176
source.nextHostAndArrowPort(),
178177
source.getCatalog().getUsername(),
179178
source.getCatalog().getPassword(),
180-
queryStr
179+
queryStr,
180+
source.getCatalog().getQueryTimeoutSec()
181181
);
182182
} catch (Exception e) {
183183
LOG.warn("arrow request node [{}] failures {}, try next nodes",
@@ -189,17 +189,78 @@ private List<Pair<String, ByteBuffer>> executeQuery() {
189189
throw new RuntimeException("Failed to execute query: " + queryStr, lastException);
190190
}
191191

192-
private List<Pair<String, ByteBuffer>> executeFlightSqlQuery(Pair<String, Integer> hostAndPort,
193-
String user, String psw, String sql) throws Exception {
194-
try (
195-
BufferAllocator allocatorFE = new RootAllocator();
196-
FlightClient clientFE = createFlightClient(allocatorFE, hostAndPort);
197-
FlightSqlClient sqlClientFE = new FlightSqlClient(clientFE)
198-
) {
199-
CredentialCallOption credentialCallOption = authenticate(clientFE, user, psw);
200-
FlightInfo info = executeSqlWithTimeout(sqlClientFE, sql, credentialCallOption);
201-
202-
return processFlightEndpoints(info.getEndpoints());
192+
// Opens a Flight SQL session on the remote frontend and runs the query in it. The session is
193+
// kept until stop(): its query serves the BE's DoGet of the endpoints returned here. A session
194+
// whose query failed is closed right away, so a retry on the next node leaves nothing behind.
195+
@VisibleForTesting
196+
List<Pair<String, ByteBuffer>> executeFlightSqlQuery(Pair<String, Integer> hostAndPort,
197+
String user, String psw, String sql, int timeoutSec) throws Exception {
198+
RemoteDorisFlightSession session = RemoteDorisFlightSession.open(hostAndPort, user, psw);
199+
FlightInfo info;
200+
try {
201+
info = session.execute(sql, timeoutSec);
202+
} catch (Throwable t) {
203+
session.close();
204+
throw t;
205+
}
206+
keepFlightSession(session);
207+
return processFlightEndpoints(info.getEndpoints());
208+
}
209+
210+
/**
211+
* Holds {@code session} until {@link #stop()}. A session handed over after stop() already ran,
212+
* or on top of one still held, is closed at once instead: this scan owns one session at most,
213+
* and none once stopped.
214+
*/
215+
@VisibleForTesting
216+
void keepFlightSession(RemoteDorisFlightSession session) {
217+
RemoteDorisFlightSession toClose;
218+
synchronized (this) {
219+
if (stopped) {
220+
toClose = session;
221+
} else {
222+
toClose = flightSession;
223+
flightSession = session;
224+
}
225+
}
226+
if (toClose != null) {
227+
toClose.close();
228+
}
229+
}
230+
231+
/**
232+
* True while this scan holds a Flight SQL session on the remote frontend: the local coordinator
233+
* has to stay alive until the BE has finished reading the remote query, since closing it is what
234+
* ends the session ({@link #stop()}) - and the remote frontend cancels what a closed session was
235+
* still running. Without this, an Arrow Flight SQL query on this frontend would close its
236+
* coordinator right after dispatch (#67503), while its BE may still be reading.
237+
*/
238+
@Override
239+
public boolean coordinatorMustOutliveDispatch() {
240+
if (super.coordinatorMustOutliveDispatch()) {
241+
return true;
242+
}
243+
synchronized (this) {
244+
return flightSession != null;
245+
}
246+
}
247+
248+
/**
249+
* Ends the Flight SQL session on the remote frontend, in addition to what {@code FileQueryScanNode}
250+
* releases. Called by the coordinator when the local query closes or is cancelled, i.e. when the
251+
* BE is done with (or gave up on) the remote query's endpoints.
252+
*/
253+
@Override
254+
public void stop() {
255+
super.stop();
256+
RemoteDorisFlightSession session;
257+
synchronized (this) {
258+
stopped = true;
259+
session = flightSession;
260+
flightSession = null;
261+
}
262+
if (session != null) {
263+
session.close();
203264
}
204265
}
205266

@@ -290,27 +351,6 @@ private boolean isExplainStatement() {
290351
.trim().toLowerCase().startsWith("explain");
291352
}
292353

293-
private FlightClient createFlightClient(BufferAllocator allocator,
294-
Pair<String, Integer> hostAndPort) throws Exception {
295-
URI uri = new URI("grpc", null, hostAndPort.first, hostAndPort.second, null, null, null);
296-
return FlightClient.builder(allocator, new Location(uri)).build();
297-
}
298-
299-
private CredentialCallOption authenticate(FlightClient client, String user, String psw) throws UserException {
300-
Optional<CredentialCallOption> credentialCallOption = client.authenticateBasicToken(user, psw);
301-
if (!credentialCallOption.isPresent()) {
302-
throw new UserException("Authenticates with a username and password failure");
303-
}
304-
return credentialCallOption.get();
305-
}
306-
307-
private FlightInfo executeSqlWithTimeout(FlightSqlClient sqlClient, String sql,
308-
CredentialCallOption credentialCallOption) {
309-
int timeoutSec = source.getCatalog().getQueryTimeoutSec();
310-
return sqlClient.execute(sql, credentialCallOption,
311-
CallOptions.timeout(timeoutSec, TimeUnit.SECONDS));
312-
}
313-
314354
private List<Pair<String, ByteBuffer>> processFlightEndpoints(List<FlightEndpoint> endpoints) {
315355
List<Pair<String, ByteBuffer>> uniquePairs = new ArrayList<>();
316356
Set<String> seenPairs = new HashSet<>();

‎fe/fe-core/src/main/java/org/apache/doris/planner/ScanNode.java‎

Lines changed: 9 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -135,13 +135,16 @@ public TupleDescriptor getTupleDesc() {
135135
}
136136

137137
/**
138-
* Whether this scan hands out its splits lazily through a batch {@link SplitSource} that the
139-
* BE fetches from the FE while it is scanning (external-table batch mode, see
140-
* {@link SplitGenerator#isBatchMode()}). Such a scan needs its coordinator alive until the BE
141-
* has finished scanning, even after the FE is done dispatching the query: closing the
142-
* coordinator releases the split source ({@link #stop()}) and the BE's next split fetch fails.
138+
* Whether the BE still depends on something this scan node holds on the FE while it is
139+
* scanning, so that the coordinator - closing it releases what the node holds, through
140+
* {@link #stop()} - has to stay alive until the BE has finished scanning, even after the FE is
141+
* done dispatching the query. Here: a batch {@link SplitSource} the BE fetches its splits from
142+
* lazily (external-table batch mode, see {@link SplitGenerator#isBatchMode()}); the BE's next
143+
* split fetch fails once the source is released. A subclass holding another such resource
144+
* adds its own reason, e.g. the Flight SQL session a remote Doris scan keeps open on the other
145+
* frontend for the query the BE reads (RemoteDorisScanNode).
143146
*/
144-
public boolean hasBatchSplitSource() {
147+
public boolean coordinatorMustOutliveDispatch() {
145148
return splitAssignment != null;
146149
}
147150

‎fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java‎

Lines changed: 10 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -797,18 +797,21 @@ public void exec() throws Exception {
797797
}
798798

799799
/**
800-
* Whether the BE keeps calling back into this coordinator after {@link #exec()} returned: an
801-
* external-table scan in batch mode fetches its splits lazily from the split source that its
802-
* scan node holds, so the coordinator must not be closed until the BE has finished scanning.
803-
* Arrow Flight SQL uses this to decide whether a query's coordinator has to outlive
804-
* GetFlightInfo, the client pulling the results from the BE later in DoGet. See #62259.
800+
* Whether the BE still depends on this coordinator after {@link #exec()} returned, so it must
801+
* not be closed until the BE has finished scanning: one of its scan nodes holds something on
802+
* the FE that the BE scans with and that {@link #close()} releases
803+
* ({@link ScanNode#coordinatorMustOutliveDispatch()}) - the split source an external-table
804+
* scan in batch mode fetches its splits from lazily, or the Flight SQL session a remote Doris
805+
* scan keeps open on the other frontend. Arrow Flight SQL uses this to decide whether a
806+
* query's coordinator has to outlive GetFlightInfo, the client pulling the results from the BE
807+
* later in DoGet. See #62259.
805808
*/
806-
public boolean hasBatchSplitSource() {
809+
public boolean mustOutliveDispatch() {
807810
if (scanNodes == null) {
808811
return false;
809812
}
810813
for (ScanNode scanNode : scanNodes) {
811-
if (scanNode.hasBatchSplitSource()) {
814+
if (scanNode.coordinatorMustOutliveDispatch()) {
812815
return true;
813816
}
814817
}

0 commit comments

Comments
 (0)