Skip to content

Commit 46e1c67

Browse files
authored
[refactor](arrow-flight) One connection pool for MySQL connections and Arrow Flight SQL sessions (#68101)
### What problem does this PR solve? Issue Number: #67577 Related PR: #67966 (the Flight session teardown contract this keeps, D-13) Problem Summary: Stage 3, item 1 of #67577 (connection governance): one connection pool for both protocols. Arrow Flight SQL sessions lived in a pool of their own, `FlightSqlConnectPoolMgr`, held to `arrow_flight_max_connections` alone: not to `qe_max_connection`, not to the user's `max_user_connections`, refused with a sentence of their own ("the number of arrow flight bearer tokens should be equal to arrow flight sql max connections"), and summed into every scheduler-wide answer — the processlist, `KILL`, the timeout checker, the connection metrics — by `ConnectScheduler` walking two pools. There is one pool now: - `ConnectPoolMgr` registers a connection of either protocol against `qe_max_connection` and the user's `max_user_connections`, and a Flight session additionally against its sub-quota. It indexes Flight sessions by their peer identity (the bearer token), since that is how Flight requests name their session. `FlightSqlConnectPoolMgr` is deleted; `ConnectScheduler` delegates to the one pool. - Admission is one atomic transition: `ConnectPoolMgr.registerConnection` checks and reserves the pool, Flight sub-quota and per-user quotas under a single lock (`admissionLock`), so a doomed attempt cannot hold one counter's reservation across another attempt's checks. Session creation is serialized too (`sessionCreationLock` in `FlightSessionsWithTokenManager`): exactly one `ConnectContext` is published per bearer token, and after registering it the token is re-validated under the same lock and the session unregistered if the token was invalidated meanwhile, so no session lingers in the pool without its token. - The bearer token cache size floors only the effective sub-quota at 1 (a legal sub-quota of 0 at `qe_max_connection = 1` still needs one token so the first request reaches the pool and is refused with `RESOURCE_EXHAUSTED`); an illegal `arrow_flight_token_cache_size` (<= 0) is left as-is, keeping the existing startup failure. - `unregisterConnection` is where every teardown path of a connection meets — a MySQL channel closing, a Flight token expiring or being evicted, `CloseSession`, `KILL`, the timeout checker — so it is where the protocol releases what it still holds for the session, through the new `ProtocolAdapter.releaseSession`: for Flight the channel-cached results and the deferred query coordinators (`FlightProtocolAdapter.tearDown`, unchanged from #67966), nothing for MySQL. The transitional `ProtocolAdapter.connectPool` goes. - A Flight session refused for a limit is refused in the words a MySQL client is refused in (`Reach limit of connections. Total: %d, User: %d, Current: %d`, naming the Flight sub-quota only when it is the tighter one), as the `RESOURCE_EXHAUSTED` status of the request that would have opened it, and its bearer token is invalidated with the refusal. - `arrow_flight_max_connections` becomes the sub-quota of Flight sessions within the pool: `-1`, the new default, is half of `qe_max_connection`; an explicit value never exceeds it. `connection_max` reports the pool's limit, `arrow_flight_connection_total` / `arrow_flight_connection_max` the Flight share of it. The token cache is sized from the effective sub-quota and is otherwise untouched: the token's lifecycle is the next PR's.
1 parent bd52103 commit 46e1c67

23 files changed

Lines changed: 1171 additions & 353 deletions

‎fe/fe-common/src/main/java/org/apache/doris/common/Config.java‎

Lines changed: 27 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -758,7 +758,9 @@ public class Config extends ConfigBase {
758758
+ "Set long enough to fit your tablet size.")
759759
public static long check_consistency_default_timeout_second = 600; // 10 min
760760

761-
@ConfField(description = "Maximum number of MySQL server connections per FE.")
761+
@ConfField(description = "Maximum number of connections per FE. MySQL connections and Arrow Flight SQL "
762+
+ "sessions share this one pool (see arrow_flight_max_connections for the share Flight sessions "
763+
+ "may take of it: half by default).")
762764
public static int qe_max_connection = 1024;
763765

764766
@ConfField(mutable = true, description = "Colocate join PlanFragment instance memory limit penalty factor. The "
@@ -2650,8 +2652,23 @@ public class Config extends ConfigBase {
26502652
+ "automatically. Set to 0 or negative value to disable " + "this limit for user-specified buckets.")
26512653
public static int max_bucket_num_per_partition = 768;
26522654

2653-
@ConfField(description = "Maximum number of connections for the Arrow Flight Server per FE.")
2654-
public static int arrow_flight_max_connections = 4096;
2655+
@ConfField(description = "Arrow Flight SQL sessions share the one connection pool with MySQL connections:"
2656+
+ " both count against qe_max_connection and the user's max_user_connections. This is the sub-quota of"
2657+
+ " Arrow Flight SQL sessions within that pool: -1 (the default) is half of qe_max_connection (512 with"
2658+
+ " the default pool of 1024), and an explicit value never exceeds qe_max_connection (a larger one is"
2659+
+ " capped, with a warning at startup). Mind how a Flight session ends: with CloseSession, a KILL"
2660+
+ " CONNECTION from another connection, wait_timeout, or the expiry or eviction of its bearer token"
2661+
+ " (arrow_flight_token_alive_time_second; the token cache, see below; and per user at most"
2662+
+ " max_user_connections / 2 tokens). Most Flight clients never send CloseSession, so a session whose"
2663+
+ " client has gone stays in the pool until wait_timeout (8 hours by default) or its token's expiry"
2664+
+ " (24 hours by default), whichever comes first; the default leaves the other half of the pool to"
2665+
+ " MySQL connections however many such sessions there are. Raise it with qe_max_connection, or set"
2666+
+ " it to qe_max_connection on an FE that serves Arrow Flight SQL only. The bearer token cache is"
2667+
+ " sized to this sub-quota (capped by arrow_flight_token_cache_size), so the Flight limit shows as"
2668+
+ " the eviction of the oldest token and its session rather than as a refusal. -1 is accepted from"
2669+
+ " this version on: an older FE that serves Arrow Flight SQL exits at startup with -1 in fe.conf;"
2670+
+ " remove the setting or set a positive value before a downgrade.")
2671+
public static int arrow_flight_max_connections = -1;
26552672

26562673
@ConfField(mutable = true, description = "Arrow Flight SQL only. A query that scans an external table in "
26572674
+ "batch mode keeps its FE coordinator alive after GetFlightInfo, so the BE can keep fetching splits "
@@ -2679,10 +2696,13 @@ public class Config extends ConfigBase {
26792696
+ "an abnormal case and triggers an alert.")
26802697
public static double autobucket_out_of_bounds_percent_threshold = 0.5;
26812698

2682-
@ConfField(description = "(Deprecated, replaced by arrow_flight_max_connection) The cache limit of all user "
2683-
+ "tokens in Arrow Flight Server, which will be eliminated by LRU rules after exceeding "
2684-
+ "the limit. Arrow Flight SQL is a stateless protocol; the connection is usually not "
2685-
+ "actively disconnected. A bearer token evicted from the cache will unregister its " + "ConnectContext.")
2699+
@ConfField(description = "The cap of the bearer token cache of the Arrow Flight SQL server. The cache holds"
2700+
+ " as many tokens as the Arrow Flight SQL sub-quota of the connection pool allows"
2701+
+ " (arrow_flight_max_connections, half of qe_max_connection by default) but never more than"
2702+
+ " this; beyond that the oldest token is evicted by LRU, and the session it names is closed with it."
2703+
+ " Arrow Flight SQL clients rarely close their session, so the effective cache size - the sub-quota"
2704+
+ " unless this is smaller - is what bounds their sessions in practice, and per user"
2705+
+ " max_user_connections / 2 tokens. The effective cache size is logged when the server starts.")
26862706
public static int arrow_flight_token_cache_size = 4096;
26872707

26882708
@ConfField(description = "The alive time of the user token in Arrow Flight Server (expire after write), in "

‎fe/fe-core/src/main/java/org/apache/doris/arrowflight/DorisFlightSqlService.java‎

Lines changed: 37 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -24,8 +24,10 @@
2424
import org.apache.doris.arrowflight.tokens.FlightTokenManager;
2525
import org.apache.doris.arrowflight.tokens.FlightTokenManagerImpl;
2626
import org.apache.doris.common.Config;
27+
import org.apache.doris.qe.ConnectPoolMgr;
2728
import org.apache.doris.service.FrontendOptions;
2829

30+
import com.google.common.annotations.VisibleForTesting;
2931
import io.grpc.ServerBuilder;
3032
import org.apache.arrow.flight.FlightServer;
3133
import org.apache.arrow.flight.Location;
@@ -48,12 +50,40 @@ public class DorisFlightSqlService {
4850
private final FlightSessionsManager flightSessionsManager;
4951
private volatile boolean running;
5052

53+
/**
54+
* The bearer token cache size: the effective Flight sub-quota (a session opens on a token, so the
55+
* sub-quota is what bounds live sessions), capped by {@code arrow_flight_token_cache_size}. The
56+
* sub-quota is floored at 1 -- a legal sub-quota of 0 ({@code qe_max_connection = 1}) still needs
57+
* one token so the first request reaches the pool and is refused with RESOURCE_EXHAUSTED, instead
58+
* of a {@code maximumSize(0)} cache evicting the freshly issued token and answering UNAUTHENTICATED.
59+
* The floor is NOT applied to {@code arrow_flight_token_cache_size}: an illegal value there
60+
* ({@literal <= 0}) stays the loud failure it is on the base (Guava rejects a negative maximumSize;
61+
* 0 evicts every token) rather than the FE silently running on a one-token cache.
62+
*/
63+
@VisibleForTesting
64+
static int effectiveTokenCacheSize(int flightMaxConnections, int tokenCacheConfig) {
65+
return Math.min(Math.max(1, flightMaxConnections), tokenCacheConfig);
66+
}
67+
5168
public DorisFlightSqlService(int port) {
5269
BufferAllocator allocator = new RootAllocator();
5370
// arrow flight sql is a stateless protocol, connection is usually not actively disconnected.
5471
// bearer token is evict from the cache will unregister ConnectContext.
55-
this.flightTokenManager = new FlightTokenManagerImpl(
56-
Math.min(Config.arrow_flight_max_connections, Config.arrow_flight_token_cache_size),
72+
int flightMaxConnections = ConnectPoolMgr.effectiveFlightMaxConnections(
73+
Config.qe_max_connection, Config.arrow_flight_max_connections);
74+
if (Config.arrow_flight_max_connections > Config.qe_max_connection) {
75+
// A fe.conf from before the pools were merged may still carry the old default, 4096: capped
76+
// to the whole pool, that is exactly the sharing the default of half is there to prevent.
77+
LOG.warn("arrow_flight_max_connections={} exceeds qe_max_connection={}: Arrow Flight SQL sessions are"
78+
+ " connections of the one pool, so the sub-quota is capped at {}, the whole pool."
79+
+ " Flight sessions, which their clients mostly never close, can then hold every"
80+
+ " connection until wait_timeout and refuse MySQL logins. On an FE that serves both"
81+
+ " protocols, remove the setting (the default is half of qe_max_connection; 4096 was"
82+
+ " the default before the pools were merged) or set it below qe_max_connection",
83+
Config.arrow_flight_max_connections, Config.qe_max_connection, flightMaxConnections);
84+
}
85+
int tokenCacheSize = effectiveTokenCacheSize(flightMaxConnections, Config.arrow_flight_token_cache_size);
86+
this.flightTokenManager = new FlightTokenManagerImpl(tokenCacheSize,
5787
Config.arrow_flight_token_alive_time_second);
5888
this.flightSessionsManager = new FlightSessionsWithTokenManager(flightTokenManager);
5989

@@ -63,9 +93,11 @@ public DorisFlightSqlService(int port) {
6393
.transportHint(GRPC_BUILDER_CONSUMER, (Consumer<ServerBuilder<?>>) builder ->
6494
builder.addStreamTracerFactory(new FlightRemoteIpServerStreamTracer.Factory()))
6595
.headerAuthenticator(new FlightBearerTokenAuthenticator(flightTokenManager)).build();
66-
LOG.info("Arrow Flight SQL service is created, port: {}, arrow_flight_max_connections: {},"
67-
+ "arrow_flight_token_alive_time_second: {}", port, Config.arrow_flight_max_connections,
68-
Config.arrow_flight_token_alive_time_second);
96+
LOG.info("Arrow Flight SQL service is created, port: {}, arrow_flight_max_connections: {} (effective: {},"
97+
+ " within qe_max_connection: {}), token cache size: {} (arrow_flight_token_cache_size: {}),"
98+
+ " arrow_flight_token_alive_time_second: {}", port,
99+
Config.arrow_flight_max_connections, flightMaxConnections, Config.qe_max_connection,
100+
tokenCacheSize, Config.arrow_flight_token_cache_size, Config.arrow_flight_token_alive_time_second);
69101
}
70102

71103
// start Arrow Flight SQL service, return true if success, otherwise false

‎fe/fe-core/src/main/java/org/apache/doris/arrowflight/protocol/FlightProtocolAdapter.java‎

Lines changed: 22 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -24,10 +24,9 @@
2424
import org.apache.doris.common.ErrorCode;
2525
import org.apache.doris.common.Status;
2626
import org.apache.doris.common.util.DebugUtil;
27+
import org.apache.doris.common.util.TokenMasker;
2728
import org.apache.doris.qe.ConnectContext;
2829
import org.apache.doris.qe.ConnectContext.ConnectType;
29-
import org.apache.doris.qe.ConnectPoolMgr;
30-
import org.apache.doris.qe.ConnectScheduler;
3130
import org.apache.doris.qe.QueryState;
3231
import org.apache.doris.qe.ShowResultSet;
3332
import org.apache.doris.qe.StmtExecutor;
@@ -266,9 +265,27 @@ public boolean returnsResultFromLocal(ConnectContext ctx) {
266265
public void fillForwardRequest(ConnectContext ctx, TMasterOpRequest request) {
267266
}
268267

268+
/**
269+
* Every Flight SQL session teardown path - the idle timeout (wait_timeout), bearer token expiry
270+
* or eviction, CloseSession, a KILL CONNECTION from another connection - reaches here through
271+
* the pool's unregisterConnection. The
272+
* channel-cached Arrow results go first, then the session is closed for good: teardown does not
273+
* wait for a command that may still be running, and what that command defers afterwards is
274+
* finalized on the spot ({@link #tearDown}).
275+
*/
269276
@Override
270-
public ConnectPoolMgr connectPool(ConnectScheduler scheduler) {
271-
return scheduler.getFlightSqlConnectPoolMgr();
277+
public void releaseSession(ConnectContext ctx) {
278+
try {
279+
channel.close();
280+
} catch (Throwable t) {
281+
// RootAllocator.close() marks the allocator closed before it reports outstanding
282+
// bytes. The error is actionable, but session teardown must still release the
283+
// coordinator, transaction and pool/token bookkeeping. The peer identity IS the bearer
284+
// token, so it is logged as a masked id, the same one FlightTokenManagerImpl uses.
285+
LOG.warn("failed to close Flight SQL channel while unregistering connection {}, peer identity {}",
286+
ctx.getConnectionId(), TokenMasker.tokenId(peerIdentity), t);
287+
}
288+
tearDown();
272289
}
273290

274291
/**
@@ -345,7 +362,7 @@ public void afterStatement(ConnectContext ctx) {
345362
@Override
346363
public void closeConnection(ConnectContext ctx) {
347364
// Releases the channel, the deferred executors and the transaction of the session.
348-
connectPool(ctx.getConnectScheduler()).unregisterConnection(ctx);
365+
ctx.getConnectScheduler().getConnectPoolMgr().unregisterConnection(ctx);
349366
}
350367

351368
public String getPeerIdentity() {

‎fe/fe-core/src/main/java/org/apache/doris/arrowflight/sessions/FlightSessionsWithTokenManager.java‎

Lines changed: 50 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -23,17 +23,21 @@
2323
import org.apache.doris.common.util.TokenMasker;
2424
import org.apache.doris.common.util.Util;
2525
import org.apache.doris.qe.ConnectContext;
26+
import org.apache.doris.qe.ConnectPoolMgr;
2627
import org.apache.doris.qe.ConnectScheduler;
2728
import org.apache.doris.service.ExecuteEnv;
2829

2930
import org.apache.arrow.flight.CallStatus;
31+
import org.apache.arrow.flight.FlightRuntimeException;
3032
import org.apache.logging.log4j.LogManager;
3133
import org.apache.logging.log4j.Logger;
3234

3335
public class FlightSessionsWithTokenManager implements FlightSessionsManager {
3436
private static final Logger LOG = LogManager.getLogger(FlightSessionsWithTokenManager.class);
3537

3638
private final FlightTokenManager flightTokenManager;
39+
// Serializes session creation so exactly one ConnectContext is published per bearer token.
40+
private final Object sessionCreationLock = new Object();
3741

3842
public FlightSessionsWithTokenManager(FlightTokenManager flightTokenManager) {
3943
this.flightTokenManager = flightTokenManager;
@@ -42,13 +46,27 @@ public FlightSessionsWithTokenManager(FlightTokenManager flightTokenManager) {
4246
@Override
4347
public ConnectContext getConnectContext(String peerIdentity) {
4448
try {
45-
ConnectContext connectContext = ExecuteEnv.getInstance().getScheduler().getFlightSqlConnectPoolMgr()
46-
.getContextWithFlightToken(peerIdentity);
47-
if (null == connectContext) {
48-
connectContext = createConnectContext(peerIdentity);
49+
ConnectContext connectContext = ExecuteEnv.getInstance().getScheduler()
50+
.getContextWithPeerIdentity(peerIdentity);
51+
if (null != connectContext) {
4952
return connectContext;
5053
}
51-
return connectContext;
54+
// Publish exactly one context per bearer token. Two concurrent first requests would both
55+
// find none above and, without this, both build and register one: two pool slots and quota
56+
// counters for one token, with the peer-identity index naming only the last, so evicting
57+
// that one leaves the other orphaned in the pool until wait_timeout. Serialize creation and
58+
// re-check the index inside, so the loser returns the winner's published context.
59+
synchronized (sessionCreationLock) {
60+
connectContext = ExecuteEnv.getInstance().getScheduler()
61+
.getContextWithPeerIdentity(peerIdentity);
62+
if (null != connectContext) {
63+
return connectContext;
64+
}
65+
return createConnectContext(peerIdentity);
66+
}
67+
} catch (FlightRuntimeException e) {
68+
// Already the status the client is meant to see (a connection refused for its limit).
69+
throw e;
5270
} catch (Exception e) {
5371
LOG.warn("get ConnectContext failed, " + e.getMessage(), e);
5472
throw CallStatus.INTERNAL.withDescription(Util.getRootCauseMessage(e)).withCause(e).toRuntimeException();
@@ -71,15 +89,34 @@ public ConnectContext createConnectContext(String peerIdentity) {
7189
flightTokenDetails.getUserIdentity(), flightTokenDetails.getRemoteIp());
7290
ConnectScheduler connectScheduler = ExecuteEnv.getInstance().getScheduler();
7391
connectScheduler.submit(connectContext);
74-
int res = connectScheduler.getFlightSqlConnectPoolMgr().registerConnection(connectContext);
92+
// The one pool every protocol registers in: qe_max_connection, the user's
93+
// max_user_connections and the Arrow Flight SQL sub-quota, refused in the words a MySQL
94+
// client is refused in. The token goes with the refusal, so that the client does not keep a
95+
// credential that can never open a session.
96+
ConnectPoolMgr pool = connectScheduler.getConnectPoolMgr();
97+
int res = pool.registerConnection(connectContext);
7598
if (res >= 0) {
76-
String errMsg = String.format(
77-
"Register arrow flight sql connection failed, Unknown Error, the number of arrow flight "
78-
+ "bearer tokens should be equal to arrow flight sql max connections, "
79-
+ "max connections: %d, used: %d.",
80-
connectScheduler.getFlightSqlConnectPoolMgr().getMaxConnections(), res);
81-
connectContext.getState().setError(ErrorCode.ERR_UNKNOWN_ERROR, errMsg);
82-
throw new IllegalArgumentException(errMsg);
99+
String errMsg = pool.limitReachedMessage(connectContext, res);
100+
connectContext.getState().setError(ErrorCode.ERR_TOO_MANY_USER_CONNECTIONS, errMsg);
101+
// The refused session never entered the pool, so nothing else releases what its
102+
// adapter allocated (the channel's allocator).
103+
connectContext.releaseProtocolSession();
104+
flightTokenManager.invalidateToken(peerIdentity);
105+
LOG.warn("refuse arrow flight sql session, bearer token id: {}, user: {}: {}",
106+
TokenMasker.tokenId(peerIdentity), connectContext.getQualifiedUser(), errMsg);
107+
throw CallStatus.RESOURCE_EXHAUSTED.withDescription(errMsg).toRuntimeException();
108+
}
109+
// Called under sessionCreationLock (see getConnectContext). The token can be invalidated between
110+
// the validateToken above and publishing the session here -- a concurrent CloseSession, or the
111+
// same-segment LRU evicting it -- which would leave a session in the pool whose token no longer
112+
// exists, holding its pool/user/Flight slots until wait_timeout. Re-check the token and, if it is
113+
// gone, unregister the session so no orphan lingers. (The token lifecycle moves under one owner
114+
// in the next PR, which closes the residual window after this re-check.)
115+
try {
116+
flightTokenManager.validateToken(peerIdentity);
117+
} catch (IllegalArgumentException e) {
118+
connectScheduler.getConnectPoolMgr().unregisterConnection(connectContext);
119+
throw e;
83120
}
84121
return connectContext;
85122
}

0 commit comments

Comments
 (0)