Repository navigation
Commit 14c957e
authored
[fix](remote-doris) End a remote Doris scan's Flight SQL session with the query; stop the regression framework leaking connections (#68338)
### What problem does this PR solve?
Related PR: #68101 (the one connection pool, which made the leak
visible), #68266 (the bearer token as the session's credential, which
removed its last cap), #67503 / #62259 (the Arrow Flight deferral gate
this generalizes)
Problem Summary:
**Context.** A Remote Doris catalog with `use_arrow_flight = true` reads
another Doris cluster over Arrow Flight SQL: for every scan,
`RemoteDorisScanNode` on the local FE performs a Flight SQL handshake
against a remote FE (`authenticateBasicToken`), runs the query there
(`GetFlightInfo`), and hands the endpoints - a ticket per remote BE - to
the local BE, which reads the rows with `DoGet` straight from the remote
BEs.
The handshake is not free on the remote side: since #68101 a Flight SQL
session is a connection in the remote FE's one connection pool, counted
against `qe_max_connection`, the Arrow Flight SQL sub-quota and the
catalog user's `max_user_connections`; since #68266 the bearer token is
that session's name in the pool and nothing else, so the session ends
only on `CloseSession`, `KILL CONNECTION`, `wait_timeout` (8h by
default) or an FE restart.
The regression framework has a leak of its own of the same shape: a
suite's Doris connections are `ThreadLocal` to the thread that opened
them, and only the suite thread and `Suite.thread()` close theirs; a
`sql` on any other thread opens a connection nobody closes.
**1. The problem, and what it cost**
- `RemoteDorisScanNode.executeFlightSqlQuery` closed the gRPC channel
and the allocator in a try-with-resources but never sent `CloseSession`.
Every scan of a Remote Doris table therefore left one Flight SQL session
behind on the remote FE, under the catalog user, until `wait_timeout`.
Before #68101 this was invisible: Flight sessions had a pool of their
own, were not counted per user, and a per-user LRU of
`max_user_connections / 2` tokens evicted the oldest. After #68101 the
leaked sessions eat the catalog user's quota on the remote FE; after
#68266 nothing caps them at all. A hundred scans within 8h and the user
- MySQL clients included - is refused there with `Reach limit of
connections`.
- This is what broke the external regression pipeline on 2026-09-21
(TeamCity 1053416 on #68308): the `remote_doris` suites point the
catalog at the FE under test with user `root`; 49 scans left 48 Flight
sessions (`Arrow Flight SQL: 512 (current: 48)` in the refusal), which
took half of root's 100; the other half was taken by the framework leak
below, and 14 MySQL connections were refused in a six-second window.
- The framework leak: `Awaitility.await()...until { sql ... }` evaluates
the condition on Awaitility's own thread, which dies with the `await()`.
Each call leaked one root connection until the client JVM
garbage-collected it (the FE logs those as `No more data to be read.
Close connection`). 230 suites call `Awaitility.await()` directly; in
the failing run one suite opened 22 such connections in 46 seconds, and
29 of them were collected in one GC at the moment the refusals stopped.
Suites that call `sql` from threads of their own (`Thread.start {
streamLoad }`, an `Executors` pool) leak the same way - a P0 run of the
same day shows 76 Awaitility connections and 89 own-thread connections
opened by root within one minute, all left to the garbage collector -
and the two docker helpers dropped the connection their action opened
without closing it.
**2. What this PR does, and why it helps**
FE:
- `RemoteDorisFlightSession` (new): the session as an object -
handshake, `execute`, and `close()` = `CloseSession` (bounded to 5s so a
remote FE that stopped answering cannot hang the local query's teardown;
the session is then left to its `wait_timeout` as before) followed by
the channel and the allocator. `open()` leaves nothing behind when the
handshake is refused; a query that fails closes its session at once, so
the retry on the next node leaves nothing behind either. Idempotent.
- `RemoteDorisScanNode` keeps the session from `getSplits` until
`stop()`, which the coordinator calls when the local query closes or is
cancelled - i.e. when the local BE is done with the remote query's
endpoints. It cannot be closed right after `GetFlightInfo`: the remote
FE cancels whatever a closed session was still running, and when the
remote table is itself an external table scanned in batch mode the
remote query is deferred there and still running while the local BE
reads.
- For the same reason the local coordinator has to outlive dispatch when
the local query is itself an Arrow Flight SQL query (otherwise #67503
closes it right after `exec()`, while the local BE may still be
reading). The deferral gate's predicate is generalized from "has a batch
split source" to "the BE still depends on this scan after dispatch":
`ScanNode.hasBatchSplitSource()` -> `coordinatorMustOutliveDispatch()`,
`Coordinator.hasBatchSplitSource()` -> `mustOutliveDispatch()`;
`RemoteDorisScanNode` adds its open session as the second reason.
Batch-mode external scans behave exactly as before.
- The statement is the fallback owner. The session is opened while the
plan is translated, before any coordinator exists, and not every plan
gets a coordinator or gets one that is closed: a statement that fails
between planning and dispatch (a SQL block rule on the scan, an `INSERT`
whose transaction cannot begin - a re-used `WITH LABEL`, the per-db txn
limit), the plan `INSERT OVERWRITE` and every materialized-view refresh
run only to locate the sink and discard, a load job created from the
plan. `keepFlightSession` registers the node with the
`StatementContext`, whose `close()` (the per-statement finally of
`ConnectProcessor`, `TaskProcessor`, `MTMVTask`) stops what is still
registered; the deferral gate hands the nodes over to the deferred
coordinator before `deferForArrowFlight`, so a Flight query kept alive
for DoGet is untouched. `INSERT OVERWRITE` releases its probe plan's
scan nodes as soon as the plan has been read.
- A same-plan retry must not reuse a plan whose scan node released what
the BE scans with: `handleQueryWithRetry` re-dispatches the failed
attempt's plan after its `cancel()` stopped the scan nodes, and the
endpoints of a remote Doris scan belong to the query of a session that
is then gone (the remote FE tears down a query it had deferred).
`ScanNode.cannotBeRedispatched()` (true for a remote Doris scan once
`stop()` ended its session) makes the retry rethrow the original error
instead.
Regression framework:
- `Awaitility.pollInSameThread()` at framework start-up: every `until {
}` now runs on the suite thread and reuses the suite's connection, as
`Suite.awaitUntil` already did. The trade-off: an `atMost()` no longer
bounds a condition that blocks (the poll runs to completion before the
bound is checked). A condition that runs statements is bounded by their
timeouts; a condition that waits on anything else bounds the wait itself
- `SuiteCluster` now runs its `doris-compose` subprocess waits on a
helper thread joined with the command's timeout and destroys the process
on expiry (they used to rely on `atMost()`).
- `SuiteContext` records every connection its thread-local accessors
open, with the thread that opened it. On every statement it closes the
connections of threads that have finished (a suite that starts a thread
per step - `Thread.start { streamLoad }; join`, as the mow flexible
suites do 72 times - now holds at most the connections of the threads
still running, deterministically, where before the population depended
on the JVM's next GC), and when the suite ends it closes whatever is
left, with a warning naming the suite.
- The two docker helpers (`docker`, `dockers`) close the connection
their action opened before restoring the original one (and the
multi-cluster one now types that original as the `ConnectionInfo` it
is).
Tests:
- `RemoteDorisScanNodeTest`: an in-process Flight SQL server counts the
sessions it is asked to close. The session lives from the query until
`stop()`; `stop()` twice closes once; a failed query closes at once; a
refused handshake opens nothing; a session handed over after `stop()` is
closed at once; the coordinator of a query with such a scan
`mustOutliveDispatch()`; a session no coordinator takes ends with the
statement, one handed to a deferred coordinator does not; and a node
whose `stop()` ended a session `cannotBeRedispatched()`.
- `ArrowFlightDeferralGateTest` follows the rename.
- Regression
`external_table_p0/remote_doris/test_remote_doris_flight_session`: a
catalog logging in as a user of its own scans a table five times, then
`INSERT OVERWRITE`s from it, then runs an `INSERT ... WITH LABEL` twice
(the second is refused after planning), and asserts after each statement
that `information_schema.processlist` holds no `ArrowFlightSQL` session
of that user - assertions that fail on master.
What it buys: a Remote Doris scan costs the remote FE one session for
exactly the duration of the local query, whatever the protocol of the
local client; the catalog user's quota on the remote FE is no longer
consumed by history; and the regression framework no longer manufactures
the MySQL half of the pressure.
**3. The classes, and how they call each other**
- `RemoteDorisScanNode` (existing): `getSplits` -> `executeQuery` ->
`executeFlightSqlQuery(host, user, password, sql, timeout)`:
`RemoteDorisFlightSession.open` + `execute`, then `keepFlightSession`
(which also registers the node with
`StatementContext.stopScanNodeAtClose`). `stop()` (from
`Coordinator.close()` / `cancel()`, or `StatementContext.close()` as the
fallback) closes the session; `coordinatorMustOutliveDispatch()` is true
while one is held; `cannotBeRedispatched()` once `stop()` ended one.
- `RemoteDorisFlightSession` (new): `open` (FlightClient +
`authenticateBasicToken`), `execute` (`FlightSqlClient.execute`),
`close` (`closeSession` with a 5s deadline, then client and allocator).
- `ScanNode.coordinatorMustOutliveDispatch()` (renamed from
`hasBatchSplitSource`): `splitAssignment != null`, overridable.
`ScanNode.cannotBeRedispatched()` (new): false by default.
- `Coordinator.mustOutliveDispatch()` (renamed): any scan node's
`coordinatorMustOutliveDispatch()`.
- `StmtExecutor.executeAndSendResult`: the deferral gate now reads
`coord.mustOutliveDispatch()`; the deferral gate calls
`StatementContext.handOverScanNodesToDeferredCoordinator` before
`deferForArrowFlight`; `handleQueryWithRetry` rethrows instead of
retrying when `planCannotBeRedispatched()`.
- `StatementContext`: `stopScanNodeAtClose`,
`handOverScanNodesToDeferredCoordinator`, and `close()` stopping what is
left (after the table locks, before the connector scope).
- `InsertOverwriteTableCommand.run`: stops the scan nodes of the plan it
probes and discards.
- `SuiteCluster.waitForDorisCompose`: the bounded subprocess wait
`runCmd` / `runCmdList` use instead of `Awaitility.await().atMost(...)`.
- Remote FE, untouched: `DorisFlightSqlProducer.closeSession` ->
`FlightSessionsInConnectPool.closeConnectContext` ->
`ConnectContext.cleanup()` + `cancelQuery`.
- `RegressionTest.initGroovyEnv`: `Awaitility.pollInSameThread()`.
- `SuiteContext`: `openedDorisConnections` (connection -> opening
thread), `trackDorisConnection`, `closeConnectionsOfFinishedThreads`
(from `getConnection()`, i.e. every statement), `closeDorisConnection`,
`closeLeftoverDorisConnections` (from `close()`); `Suite.dockerImpl` /
`dockers` call `closeDorisConnection`.
```
local FE remote FE remote BE
RemoteDorisScanNode.getSplits
'- executeFlightSqlQuery
|- RemoteDorisFlightSession.open ---- handshake --> openSession (pool: +1 for the catalog user)
|- session.execute ----------------- GetFlightInfo --> runs the query ---------------> result buffered
'- keepFlightSession (ticket per BE)
Coordinator.exec -> local BE ------------------------------ DoGet(ticket) --------------------> rows
(Arrow Flight local client: coordinator kept alive, mustOutliveDispatch() == true)
Coordinator.close / cancel
'- scanNode.stop
'- session.close ------------------- CloseSession --> closeConnectContext (pool: -1)
```1 parent 3750a81 commit 14c957e
15 files changed
Lines changed: 937 additions & 114 deletions
File tree
- fe/fe-core/src
- main/java/org/apache/doris
- datasource/doris/source
- nereids
- trees/plans/commands/insert
- planner
- qe
- test/java/org/apache/doris
- datasource/doris/source
- qe
- regression-test
- framework/src/main/groovy/org/apache/doris/regression
- suite
- suites/external_table_p0/remote_doris
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
44 | 44 | | |
45 | 45 | | |
46 | 46 | | |
| 47 | + | |
47 | 48 | | |
48 | 49 | | |
49 | | - | |
50 | | - | |
51 | 50 | | |
52 | 51 | | |
53 | 52 | | |
54 | | - | |
55 | | - | |
56 | | - | |
57 | | - | |
58 | 53 | | |
59 | 54 | | |
60 | 55 | | |
61 | | - | |
62 | 56 | | |
63 | 57 | | |
64 | 58 | | |
65 | 59 | | |
66 | 60 | | |
67 | | - | |
68 | 61 | | |
69 | | - | |
70 | 62 | | |
71 | 63 | | |
72 | 64 | | |
| |||
79 | 71 | | |
80 | 72 | | |
81 | 73 | | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
82 | 86 | | |
83 | 87 | | |
84 | 88 | | |
| |||
177 | 181 | | |
178 | 182 | | |
179 | 183 | | |
180 | | - | |
| 184 | + | |
| 185 | + | |
181 | 186 | | |
182 | 187 | | |
183 | 188 | | |
| |||
189 | 194 | | |
190 | 195 | | |
191 | 196 | | |
192 | | - | |
193 | | - | |
194 | | - | |
195 | | - | |
196 | | - | |
197 | | - | |
198 | | - | |
199 | | - | |
200 | | - | |
201 | | - | |
202 | | - | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
203 | 289 | | |
204 | 290 | | |
205 | 291 | | |
| |||
290 | 376 | | |
291 | 377 | | |
292 | 378 | | |
293 | | - | |
294 | | - | |
295 | | - | |
296 | | - | |
297 | | - | |
298 | | - | |
299 | | - | |
300 | | - | |
301 | | - | |
302 | | - | |
303 | | - | |
304 | | - | |
305 | | - | |
306 | | - | |
307 | | - | |
308 | | - | |
309 | | - | |
310 | | - | |
311 | | - | |
312 | | - | |
313 | | - | |
314 | 379 | | |
315 | 380 | | |
316 | 381 | | |
| |||
0 commit comments