Skip to content

Commit 14c957e

Browse files
authored
[fix](remote-doris) End a remote Doris scan's Flight SQL session with the query; stop the regression framework leaking connections (#68338)
### What problem does this PR solve? Related PR: #68101 (the one connection pool, which made the leak visible), #68266 (the bearer token as the session's credential, which removed its last cap), #67503 / #62259 (the Arrow Flight deferral gate this generalizes) Problem Summary: **Context.** A Remote Doris catalog with `use_arrow_flight = true` reads another Doris cluster over Arrow Flight SQL: for every scan, `RemoteDorisScanNode` on the local FE performs a Flight SQL handshake against a remote FE (`authenticateBasicToken`), runs the query there (`GetFlightInfo`), and hands the endpoints - a ticket per remote BE - to the local BE, which reads the rows with `DoGet` straight from the remote BEs. The handshake is not free on the remote side: since #68101 a Flight SQL session is a connection in the remote FE's one connection pool, counted against `qe_max_connection`, the Arrow Flight SQL sub-quota and the catalog user's `max_user_connections`; since #68266 the bearer token is that session's name in the pool and nothing else, so the session ends only on `CloseSession`, `KILL CONNECTION`, `wait_timeout` (8h by default) or an FE restart. The regression framework has a leak of its own of the same shape: a suite's Doris connections are `ThreadLocal` to the thread that opened them, and only the suite thread and `Suite.thread()` close theirs; a `sql` on any other thread opens a connection nobody closes. **1. The problem, and what it cost** - `RemoteDorisScanNode.executeFlightSqlQuery` closed the gRPC channel and the allocator in a try-with-resources but never sent `CloseSession`. Every scan of a Remote Doris table therefore left one Flight SQL session behind on the remote FE, under the catalog user, until `wait_timeout`. Before #68101 this was invisible: Flight sessions had a pool of their own, were not counted per user, and a per-user LRU of `max_user_connections / 2` tokens evicted the oldest. After #68101 the leaked sessions eat the catalog user's quota on the remote FE; after #68266 nothing caps them at all. A hundred scans within 8h and the user - MySQL clients included - is refused there with `Reach limit of connections`. - This is what broke the external regression pipeline on 2026-09-21 (TeamCity 1053416 on #68308): the `remote_doris` suites point the catalog at the FE under test with user `root`; 49 scans left 48 Flight sessions (`Arrow Flight SQL: 512 (current: 48)` in the refusal), which took half of root's 100; the other half was taken by the framework leak below, and 14 MySQL connections were refused in a six-second window. - The framework leak: `Awaitility.await()...until { sql ... }` evaluates the condition on Awaitility's own thread, which dies with the `await()`. Each call leaked one root connection until the client JVM garbage-collected it (the FE logs those as `No more data to be read. Close connection`). 230 suites call `Awaitility.await()` directly; in the failing run one suite opened 22 such connections in 46 seconds, and 29 of them were collected in one GC at the moment the refusals stopped. Suites that call `sql` from threads of their own (`Thread.start { streamLoad }`, an `Executors` pool) leak the same way - a P0 run of the same day shows 76 Awaitility connections and 89 own-thread connections opened by root within one minute, all left to the garbage collector - and the two docker helpers dropped the connection their action opened without closing it. **2. What this PR does, and why it helps** FE: - `RemoteDorisFlightSession` (new): the session as an object - handshake, `execute`, and `close()` = `CloseSession` (bounded to 5s so a remote FE that stopped answering cannot hang the local query's teardown; the session is then left to its `wait_timeout` as before) followed by the channel and the allocator. `open()` leaves nothing behind when the handshake is refused; a query that fails closes its session at once, so the retry on the next node leaves nothing behind either. Idempotent. - `RemoteDorisScanNode` keeps the session from `getSplits` until `stop()`, which the coordinator calls when the local query closes or is cancelled - i.e. when the local BE is done with the remote query's endpoints. It cannot be closed right after `GetFlightInfo`: the remote FE cancels whatever a closed session was still running, and when the remote table is itself an external table scanned in batch mode the remote query is deferred there and still running while the local BE reads. - For the same reason the local coordinator has to outlive dispatch when the local query is itself an Arrow Flight SQL query (otherwise #67503 closes it right after `exec()`, while the local BE may still be reading). The deferral gate's predicate is generalized from "has a batch split source" to "the BE still depends on this scan after dispatch": `ScanNode.hasBatchSplitSource()` -> `coordinatorMustOutliveDispatch()`, `Coordinator.hasBatchSplitSource()` -> `mustOutliveDispatch()`; `RemoteDorisScanNode` adds its open session as the second reason. Batch-mode external scans behave exactly as before. - The statement is the fallback owner. The session is opened while the plan is translated, before any coordinator exists, and not every plan gets a coordinator or gets one that is closed: a statement that fails between planning and dispatch (a SQL block rule on the scan, an `INSERT` whose transaction cannot begin - a re-used `WITH LABEL`, the per-db txn limit), the plan `INSERT OVERWRITE` and every materialized-view refresh run only to locate the sink and discard, a load job created from the plan. `keepFlightSession` registers the node with the `StatementContext`, whose `close()` (the per-statement finally of `ConnectProcessor`, `TaskProcessor`, `MTMVTask`) stops what is still registered; the deferral gate hands the nodes over to the deferred coordinator before `deferForArrowFlight`, so a Flight query kept alive for DoGet is untouched. `INSERT OVERWRITE` releases its probe plan's scan nodes as soon as the plan has been read. - A same-plan retry must not reuse a plan whose scan node released what the BE scans with: `handleQueryWithRetry` re-dispatches the failed attempt's plan after its `cancel()` stopped the scan nodes, and the endpoints of a remote Doris scan belong to the query of a session that is then gone (the remote FE tears down a query it had deferred). `ScanNode.cannotBeRedispatched()` (true for a remote Doris scan once `stop()` ended its session) makes the retry rethrow the original error instead. Regression framework: - `Awaitility.pollInSameThread()` at framework start-up: every `until { }` now runs on the suite thread and reuses the suite's connection, as `Suite.awaitUntil` already did. The trade-off: an `atMost()` no longer bounds a condition that blocks (the poll runs to completion before the bound is checked). A condition that runs statements is bounded by their timeouts; a condition that waits on anything else bounds the wait itself - `SuiteCluster` now runs its `doris-compose` subprocess waits on a helper thread joined with the command's timeout and destroys the process on expiry (they used to rely on `atMost()`). - `SuiteContext` records every connection its thread-local accessors open, with the thread that opened it. On every statement it closes the connections of threads that have finished (a suite that starts a thread per step - `Thread.start { streamLoad }; join`, as the mow flexible suites do 72 times - now holds at most the connections of the threads still running, deterministically, where before the population depended on the JVM's next GC), and when the suite ends it closes whatever is left, with a warning naming the suite. - The two docker helpers (`docker`, `dockers`) close the connection their action opened before restoring the original one (and the multi-cluster one now types that original as the `ConnectionInfo` it is). Tests: - `RemoteDorisScanNodeTest`: an in-process Flight SQL server counts the sessions it is asked to close. The session lives from the query until `stop()`; `stop()` twice closes once; a failed query closes at once; a refused handshake opens nothing; a session handed over after `stop()` is closed at once; the coordinator of a query with such a scan `mustOutliveDispatch()`; a session no coordinator takes ends with the statement, one handed to a deferred coordinator does not; and a node whose `stop()` ended a session `cannotBeRedispatched()`. - `ArrowFlightDeferralGateTest` follows the rename. - Regression `external_table_p0/remote_doris/test_remote_doris_flight_session`: a catalog logging in as a user of its own scans a table five times, then `INSERT OVERWRITE`s from it, then runs an `INSERT ... WITH LABEL` twice (the second is refused after planning), and asserts after each statement that `information_schema.processlist` holds no `ArrowFlightSQL` session of that user - assertions that fail on master. What it buys: a Remote Doris scan costs the remote FE one session for exactly the duration of the local query, whatever the protocol of the local client; the catalog user's quota on the remote FE is no longer consumed by history; and the regression framework no longer manufactures the MySQL half of the pressure. **3. The classes, and how they call each other** - `RemoteDorisScanNode` (existing): `getSplits` -> `executeQuery` -> `executeFlightSqlQuery(host, user, password, sql, timeout)`: `RemoteDorisFlightSession.open` + `execute`, then `keepFlightSession` (which also registers the node with `StatementContext.stopScanNodeAtClose`). `stop()` (from `Coordinator.close()` / `cancel()`, or `StatementContext.close()` as the fallback) closes the session; `coordinatorMustOutliveDispatch()` is true while one is held; `cannotBeRedispatched()` once `stop()` ended one. - `RemoteDorisFlightSession` (new): `open` (FlightClient + `authenticateBasicToken`), `execute` (`FlightSqlClient.execute`), `close` (`closeSession` with a 5s deadline, then client and allocator). - `ScanNode.coordinatorMustOutliveDispatch()` (renamed from `hasBatchSplitSource`): `splitAssignment != null`, overridable. `ScanNode.cannotBeRedispatched()` (new): false by default. - `Coordinator.mustOutliveDispatch()` (renamed): any scan node's `coordinatorMustOutliveDispatch()`. - `StmtExecutor.executeAndSendResult`: the deferral gate now reads `coord.mustOutliveDispatch()`; the deferral gate calls `StatementContext.handOverScanNodesToDeferredCoordinator` before `deferForArrowFlight`; `handleQueryWithRetry` rethrows instead of retrying when `planCannotBeRedispatched()`. - `StatementContext`: `stopScanNodeAtClose`, `handOverScanNodesToDeferredCoordinator`, and `close()` stopping what is left (after the table locks, before the connector scope). - `InsertOverwriteTableCommand.run`: stops the scan nodes of the plan it probes and discards. - `SuiteCluster.waitForDorisCompose`: the bounded subprocess wait `runCmd` / `runCmdList` use instead of `Awaitility.await().atMost(...)`. - Remote FE, untouched: `DorisFlightSqlProducer.closeSession` -> `FlightSessionsInConnectPool.closeConnectContext` -> `ConnectContext.cleanup()` + `cancelQuery`. - `RegressionTest.initGroovyEnv`: `Awaitility.pollInSameThread()`. - `SuiteContext`: `openedDorisConnections` (connection -> opening thread), `trackDorisConnection`, `closeConnectionsOfFinishedThreads` (from `getConnection()`, i.e. every statement), `closeDorisConnection`, `closeLeftoverDorisConnections` (from `close()`); `Suite.dockerImpl` / `dockers` call `closeDorisConnection`. ``` local FE remote FE remote BE RemoteDorisScanNode.getSplits '- executeFlightSqlQuery |- RemoteDorisFlightSession.open ---- handshake --> openSession (pool: +1 for the catalog user) |- session.execute ----------------- GetFlightInfo --> runs the query ---------------> result buffered '- keepFlightSession (ticket per BE) Coordinator.exec -> local BE ------------------------------ DoGet(ticket) --------------------> rows (Arrow Flight local client: coordinator kept alive, mustOutliveDispatch() == true) Coordinator.close / cancel '- scanNode.stop '- session.close ------------------- CloseSession --> closeConnectContext (pool: -1) ```
1 parent 3750a81 commit 14c957e

15 files changed

Lines changed: 937 additions & 114 deletions

File tree

Lines changed: 153 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,153 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
package org.apache.doris.datasource.doris.source;
19+
20+
import org.apache.doris.common.Pair;
21+
import org.apache.doris.common.UserException;
22+
23+
import com.google.common.annotations.VisibleForTesting;
24+
import org.apache.arrow.flight.CallOptions;
25+
import org.apache.arrow.flight.CloseSessionRequest;
26+
import org.apache.arrow.flight.FlightClient;
27+
import org.apache.arrow.flight.FlightInfo;
28+
import org.apache.arrow.flight.Location;
29+
import org.apache.arrow.flight.grpc.CredentialCallOption;
30+
import org.apache.arrow.flight.sql.FlightSqlClient;
31+
import org.apache.arrow.memory.BufferAllocator;
32+
import org.apache.arrow.memory.RootAllocator;
33+
import org.apache.logging.log4j.LogManager;
34+
import org.apache.logging.log4j.Logger;
35+
36+
import java.io.Closeable;
37+
import java.net.URI;
38+
import java.util.Optional;
39+
import java.util.concurrent.TimeUnit;
40+
41+
/**
42+
* The Flight SQL session a {@link RemoteDorisScanNode} opens on a remote Doris frontend for one
43+
* scan, and ends with a CloseSession once the scan is over.
44+
*
45+
* <p>The handshake ({@code authenticateBasicToken}) opens a session on the remote frontend: a
46+
* connection in its pool, counted against {@code qe_max_connection}, the Arrow Flight SQL sub-quota
47+
* and the catalog user's {@code max_user_connections}, that only a CloseSession, a KILL or
48+
* {@code wait_timeout} ends. Closing the gRPC channel does not. A session opened per scan and never
49+
* closed therefore stays for hours and, one scan at a time, exhausts the catalog user's connection
50+
* quota on the remote frontend - refusing that user's MySQL connections there as well.
51+
*
52+
* <p>The session outlives GetFlightInfo on purpose: the query it ran serves the BE's DoGet of the
53+
* endpoints, and the remote frontend cancels whatever a closed session was still running - the
54+
* query itself, when the remote table is an external table scanned in batch mode and the query is
55+
* therefore deferred there. So {@link #close()} is called from {@link RemoteDorisScanNode#stop()},
56+
* when the coordinator of the local query closes, and that coordinator is kept alive until the BE
57+
* has finished scanning ({@link RemoteDorisScanNode#coordinatorMustOutliveDispatch()}).
58+
*/
59+
class RemoteDorisFlightSession implements Closeable {
60+
private static final Logger LOG = LogManager.getLogger(RemoteDorisFlightSession.class);
61+
62+
// A bound on the CloseSession round trip. close() runs on the local query's teardown path,
63+
// which must not hang on a remote frontend that has stopped answering; the session is then left
64+
// to the remote frontend's wait_timeout, as every session was before this class existed.
65+
@VisibleForTesting
66+
static final int CLOSE_SESSION_TIMEOUT_SECONDS = 5;
67+
68+
private final Pair<String, Integer> hostAndPort;
69+
private final BufferAllocator allocator;
70+
private final FlightSqlClient client;
71+
private final CredentialCallOption credential;
72+
private boolean closed = false;
73+
74+
private RemoteDorisFlightSession(Pair<String, Integer> hostAndPort, BufferAllocator allocator,
75+
FlightSqlClient client, CredentialCallOption credential) {
76+
this.hostAndPort = hostAndPort;
77+
this.allocator = allocator;
78+
this.client = client;
79+
this.credential = credential;
80+
}
81+
82+
/**
83+
* Opens a session on the remote frontend at {@code hostAndPort} with the catalog's credentials.
84+
* Nothing is left behind when this fails: a handshake that was refused opened no session, and
85+
* the channel and allocator are released before the exception propagates.
86+
*/
87+
static RemoteDorisFlightSession open(Pair<String, Integer> hostAndPort, String user, String password)
88+
throws Exception {
89+
BufferAllocator allocator = new RootAllocator();
90+
FlightClient flightClient = null;
91+
try {
92+
URI uri = new URI("grpc", null, hostAndPort.first, hostAndPort.second, null, null, null);
93+
flightClient = FlightClient.builder(allocator, new Location(uri)).build();
94+
Optional<CredentialCallOption> credential = flightClient.authenticateBasicToken(user, password);
95+
if (!credential.isPresent()) {
96+
throw new UserException("Authenticates with a username and password failure");
97+
}
98+
return new RemoteDorisFlightSession(hostAndPort, allocator, new FlightSqlClient(flightClient),
99+
credential.get());
100+
} catch (Throwable t) {
101+
closeQuietly(flightClient, allocator, hostAndPort);
102+
throw t;
103+
}
104+
}
105+
106+
/** Runs {@code sql} on the remote frontend; the endpoints of the result are where the BE reads it. */
107+
FlightInfo execute(String sql, int timeoutSec) {
108+
return client.execute(sql, credential, CallOptions.timeout(timeoutSec, TimeUnit.SECONDS));
109+
}
110+
111+
Pair<String, Integer> getHostAndPort() {
112+
return hostAndPort;
113+
}
114+
115+
/**
116+
* Ends the session on the remote frontend (CloseSession), then releases the channel and the
117+
* allocator. Never throws: this runs on the local query's teardown path, and a session the
118+
* remote frontend could not be told to close is only left to its wait_timeout. Idempotent.
119+
*/
120+
@Override
121+
public synchronized void close() {
122+
if (closed) {
123+
return;
124+
}
125+
closed = true;
126+
try {
127+
client.closeSession(new CloseSessionRequest(), credential,
128+
CallOptions.timeout(CLOSE_SESSION_TIMEOUT_SECONDS, TimeUnit.SECONDS));
129+
} catch (Throwable t) {
130+
LOG.warn("failed to close the Arrow Flight SQL session on remote Doris frontend {}:{}, it stays open there"
131+
+ " until its wait_timeout", hostAndPort.first, hostAndPort.second, t);
132+
}
133+
closeQuietly(client, allocator, hostAndPort);
134+
}
135+
136+
private static void closeQuietly(AutoCloseable client, BufferAllocator allocator,
137+
Pair<String, Integer> hostAndPort) {
138+
try {
139+
if (client != null) {
140+
client.close();
141+
}
142+
} catch (Throwable t) {
143+
LOG.warn("failed to close the Arrow Flight client to remote Doris frontend {}:{}",
144+
hostAndPort.first, hostAndPort.second, t);
145+
}
146+
try {
147+
allocator.close();
148+
} catch (Throwable t) {
149+
LOG.warn("failed to close the Arrow allocator of the Flight client to remote Doris frontend {}:{}",
150+
hostAndPort.first, hostAndPort.second, t);
151+
}
152+
}
153+
}

‎fe/fe-core/src/main/java/org/apache/doris/datasource/doris/source/RemoteDorisScanNode.java‎

Lines changed: 107 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -44,29 +44,21 @@
4444
import org.apache.doris.thrift.TRemoteDorisFileDesc;
4545
import org.apache.doris.thrift.TTableFormatFileDesc;
4646

47+
import com.google.common.annotations.VisibleForTesting;
4748
import com.google.common.base.Joiner;
4849
import com.google.common.collect.Lists;
49-
import org.apache.arrow.flight.CallOptions;
50-
import org.apache.arrow.flight.FlightClient;
5150
import org.apache.arrow.flight.FlightEndpoint;
5251
import org.apache.arrow.flight.FlightInfo;
5352
import org.apache.arrow.flight.Location;
54-
import org.apache.arrow.flight.grpc.CredentialCallOption;
55-
import org.apache.arrow.flight.sql.FlightSqlClient;
56-
import org.apache.arrow.memory.BufferAllocator;
57-
import org.apache.arrow.memory.RootAllocator;
5853
import org.apache.logging.log4j.LogManager;
5954
import org.apache.logging.log4j.Logger;
6055

61-
import java.net.URI;
6256
import java.nio.ByteBuffer;
6357
import java.util.ArrayList;
6458
import java.util.HashSet;
6559
import java.util.List;
6660
import java.util.Map;
67-
import java.util.Optional;
6861
import java.util.Set;
69-
import java.util.concurrent.TimeUnit;
7062
import java.util.stream.Collectors;
7163

7264
public class RemoteDorisScanNode extends FileQueryScanNode {
@@ -79,6 +71,18 @@ public class RemoteDorisScanNode extends FileQueryScanNode {
7971

8072
private RemoteDorisSource source;
8173

74+
// The Flight SQL session this scan opened on the remote frontend, from getSplits until stop()
75+
// closes it (see RemoteDorisFlightSession for why it must live that long and no longer). All
76+
// three guarded by this: stop() may run on another thread than the one that planned the query -
77+
// a KILL, the timeout checker - and more than once (cancel, then close).
78+
private RemoteDorisFlightSession flightSession;
79+
private boolean stopped;
80+
// Whether stop() ended a session this scan had opened: the endpoints handed to the backend
81+
// belong to that session's query, and a plan dispatched again with them (the same-plan retry
82+
// of StmtExecutor.handleQueryWithRetry) would read what the remote frontend may have torn
83+
// down with the session.
84+
private boolean sessionClosedByStop;
85+
8286
public RemoteDorisScanNode(PlanNodeId id, TupleDescriptor desc, boolean needCheckColumnPriv,
8387
SessionVariable sv, ScanContext scanContext) {
8488
super(id, desc, "REMOTE_DORIS_SCAN_NODE", scanContext, needCheckColumnPriv, sv);
@@ -177,7 +181,8 @@ private List<Pair<String, ByteBuffer>> executeQuery() {
177181
source.nextHostAndArrowPort(),
178182
source.getCatalog().getUsername(),
179183
source.getCatalog().getPassword(),
180-
queryStr
184+
queryStr,
185+
source.getCatalog().getQueryTimeoutSec()
181186
);
182187
} catch (Exception e) {
183188
LOG.warn("arrow request node [{}] failures {}, try next nodes",
@@ -189,17 +194,98 @@ private List<Pair<String, ByteBuffer>> executeQuery() {
189194
throw new RuntimeException("Failed to execute query: " + queryStr, lastException);
190195
}
191196

192-
private List<Pair<String, ByteBuffer>> executeFlightSqlQuery(Pair<String, Integer> hostAndPort,
193-
String user, String psw, String sql) throws Exception {
194-
try (
195-
BufferAllocator allocatorFE = new RootAllocator();
196-
FlightClient clientFE = createFlightClient(allocatorFE, hostAndPort);
197-
FlightSqlClient sqlClientFE = new FlightSqlClient(clientFE)
198-
) {
199-
CredentialCallOption credentialCallOption = authenticate(clientFE, user, psw);
200-
FlightInfo info = executeSqlWithTimeout(sqlClientFE, sql, credentialCallOption);
201-
202-
return processFlightEndpoints(info.getEndpoints());
197+
// Opens a Flight SQL session on the remote frontend and runs the query in it. The session is
198+
// kept until stop(): its query serves the BE's DoGet of the endpoints returned here. A session
199+
// whose query failed is closed right away, so a retry on the next node leaves nothing behind.
200+
@VisibleForTesting
201+
List<Pair<String, ByteBuffer>> executeFlightSqlQuery(Pair<String, Integer> hostAndPort,
202+
String user, String psw, String sql, int timeoutSec) throws Exception {
203+
RemoteDorisFlightSession session = RemoteDorisFlightSession.open(hostAndPort, user, psw);
204+
FlightInfo info;
205+
try {
206+
info = session.execute(sql, timeoutSec);
207+
} catch (Throwable t) {
208+
session.close();
209+
throw t;
210+
}
211+
keepFlightSession(session);
212+
return processFlightEndpoints(info.getEndpoints());
213+
}
214+
215+
/**
216+
* Holds {@code session} until {@link #stop()}. A session handed over after stop() already ran,
217+
* or on top of one still held, is closed at once instead: this scan owns one session at most,
218+
* and none once stopped. The statement registers the node as well: stop() is the coordinator's
219+
* to call, but a plan that never gets one, or whose coordinator nobody closes, is stopped when
220+
* the statement ends instead ({@link org.apache.doris.nereids.StatementContext#stopScanNodeAtClose}).
221+
*/
222+
@VisibleForTesting
223+
void keepFlightSession(RemoteDorisFlightSession session) {
224+
RemoteDorisFlightSession toClose;
225+
synchronized (this) {
226+
if (stopped) {
227+
toClose = session;
228+
} else {
229+
toClose = flightSession;
230+
flightSession = session;
231+
}
232+
}
233+
if (toClose != null) {
234+
toClose.close();
235+
}
236+
if (toClose != session) {
237+
ConnectContext.get().getStatementContext().stopScanNodeAtClose(this);
238+
}
239+
}
240+
241+
/**
242+
* True once {@link #stop()} ended the session this scan opened: the endpoints in its scan
243+
* ranges belong to that session's query on the remote frontend, so the same plan must not be
244+
* dispatched again (see {@link ScanNode#cannotBeRedispatched()}).
245+
*/
246+
@Override
247+
public boolean cannotBeRedispatched() {
248+
synchronized (this) {
249+
return sessionClosedByStop;
250+
}
251+
}
252+
253+
/**
254+
* True while this scan holds a Flight SQL session on the remote frontend: the local coordinator
255+
* has to stay alive until the BE has finished reading the remote query, since closing it is what
256+
* ends the session ({@link #stop()}) - and the remote frontend cancels what a closed session was
257+
* still running. Without this, an Arrow Flight SQL query on this frontend would close its
258+
* coordinator right after dispatch (#67503), while its BE may still be reading.
259+
*/
260+
@Override
261+
public boolean coordinatorMustOutliveDispatch() {
262+
if (super.coordinatorMustOutliveDispatch()) {
263+
return true;
264+
}
265+
synchronized (this) {
266+
return flightSession != null;
267+
}
268+
}
269+
270+
/**
271+
* Ends the Flight SQL session on the remote frontend, in addition to what {@code FileQueryScanNode}
272+
* releases. Called by the coordinator when the local query closes or is cancelled, i.e. when the
273+
* BE is done with (or gave up on) the remote query's endpoints.
274+
*/
275+
@Override
276+
public void stop() {
277+
super.stop();
278+
RemoteDorisFlightSession session;
279+
synchronized (this) {
280+
stopped = true;
281+
session = flightSession;
282+
flightSession = null;
283+
if (session != null) {
284+
sessionClosedByStop = true;
285+
}
286+
}
287+
if (session != null) {
288+
session.close();
203289
}
204290
}
205291

@@ -290,27 +376,6 @@ private boolean isExplainStatement() {
290376
.trim().toLowerCase().startsWith("explain");
291377
}
292378

293-
private FlightClient createFlightClient(BufferAllocator allocator,
294-
Pair<String, Integer> hostAndPort) throws Exception {
295-
URI uri = new URI("grpc", null, hostAndPort.first, hostAndPort.second, null, null, null);
296-
return FlightClient.builder(allocator, new Location(uri)).build();
297-
}
298-
299-
private CredentialCallOption authenticate(FlightClient client, String user, String psw) throws UserException {
300-
Optional<CredentialCallOption> credentialCallOption = client.authenticateBasicToken(user, psw);
301-
if (!credentialCallOption.isPresent()) {
302-
throw new UserException("Authenticates with a username and password failure");
303-
}
304-
return credentialCallOption.get();
305-
}
306-
307-
private FlightInfo executeSqlWithTimeout(FlightSqlClient sqlClient, String sql,
308-
CredentialCallOption credentialCallOption) {
309-
int timeoutSec = source.getCatalog().getQueryTimeoutSec();
310-
return sqlClient.execute(sql, credentialCallOption,
311-
CallOptions.timeout(timeoutSec, TimeUnit.SECONDS));
312-
}
313-
314379
private List<Pair<String, ByteBuffer>> processFlightEndpoints(List<FlightEndpoint> endpoints) {
315380
List<Pair<String, ByteBuffer>> uniquePairs = new ArrayList<>();
316381
Set<String> seenPairs = new HashSet<>();

0 commit comments

Comments
 (0)