Skip to content

Commit 83ebddd

Browse files
authored
[refactor](session) Bind a ConnectContext to one ProtocolAdapter instead of a Flight subclass (#67835)
### What problem does this PR solve? Issue Number: #67577 -- the tracking issue for the protocol-agnostic session and execution layer. This is the second PR of its Stage 1 (after the golden baseline #67789) and does not close it. The shape after this PR: ``` +---------------------------+ +-----------------------------+ | MySQL client | | Arrow Flight SQL client | +-------------+-------------+ +--------------+--------------+ | | v v +---------------------------+ +-----------------------------+ | MysqlServer | | DorisFlightSqlProducer | | AcceptListener | | every call goes through | | ReadListener | | adapter.runCommand() | | (xnio: one command | | (lock: one command | | at a time) | | at a time) | +-------------+-------------+ +--------------+--------------+ | | v v +---------------------------+ +-----------------------------+ | MysqlConnectProcessor | | FlightSqlConnectProcessor | +-------------+-------------+ +--------------+--------------+ | | +----------------------------+-----------------------------+ | v +-------------------------------------------------------------------------------------------+ | ConnectContext -- the session, one per connection | | | | user, catalog / db, SessionVariable, transaction, prepared statements, | | queryId / stmtId, executor, audit, ... | | | | protocolAdapter : ProtocolAdapter | | bound once, by forMysql() | forMysqlProxy() | forFlight() | new ConnectContext() | | | | getMysqlChannel() getCapability() getFlightSqlChannel() isReturnResultFromLocal() .. | | same signatures as before, now delegate to protocolAdapter | +---------------------------------------------+---------------------------------------------+ | v +-----------------------------------------------+ | <<interface>> qe.protocol.ProtocolAdapter | | | | type() | | remoteHostPortString(ctx) | | resultSinkType() | | connectPool(scheduler) | | afterStatement(ctx) | | closeConnection(ctx) | +-----------------------+-----------------------+ | +--------------------+----------------------------+ | | +-----------------------+----------------------+ +-----------------------+----------------------+ | mysql.protocol.MysqlProtocolAdapter | | service.arrowflight.protocol | | | | .FlightProtocolAdapter | | MysqlChannel: | | | | a socket | | peerIdentity (the bearer token) | | ProxyMysqlChannel (statement forwarded | | FlightSqlChannel (FE-side result cache) | | to the master) | | endpoints of the last query | | DummyMysqlChannel (internal context) | | returnResultFromLocal | | server / negotiated MysqlCapability | | prepared queries | | handshake packet, MysqlSslContext | | deferred executors + idle bound | | COM_STMT_EXECUTE packet, cursor flag | | (#62259, #67503) | | clientConsumesCursorMetadataTerminator() | | per-session command lock: | | accept-query loop | | runCommand() / callCommand() | | (start / suspend / resume / stop) | | | +----------------------------------------------+ +----------------------------------------------+ Not touched by this PR: ConnectProcessor / StmtExecutor / Coordinator keep calling the ConnectContext getters. The result-encoding half (ResultSender) and the capability predicates that replace the remaining ConnectType branches come in the next PRs. ``` **In plain terms.** A client session in the frontend is a `ConnectContext`. Today that one class holds the state of both wire protocols at the same time: the MySQL socket, the capabilities negotiated with the MySQL client, the handshake and SSL state, the prepared-statement packet being executed -- and, next to them, the Arrow Flight SQL result cache, the backend endpoints of the last Flight query, the prepared queries and the deferred coordinators. Which half is real is decided by a subclass, `FlightSqlConnectContext`, that overrides six methods and leaves every other Flight member sitting on the base class, where a MySQL connection carries it as dead weight and a Flight session throws from the MySQL ones. This PR gives each protocol its own object, a `ProtocolAdapter`, and binds a session to exactly one of them when it is created. Nothing a client sees changes: the golden byte-for-byte baseline recorded in #67789 is identical before and after. Problem Summary: `ConnectContext` mixes three things: the session (user, catalog and database, session variables, transaction, prepared statements, the running statement), the MySQL protocol state, and the Arrow Flight SQL protocol state. The next steps of #67577 move the result path of both protocols onto one shared implementation, which needs a place for "what only this protocol knows" that is not the session itself. This PR creates that place and moves the state, without touching the execution layer yet: `StmtExecutor`, `ConnectProcessor` and the coordinators still call the same `ConnectContext` getters, which now delegate. ### What is changed? **`qe/protocol/ProtocolAdapter`** -- the wire-protocol half of a connection: `type()`, the client address for processlist and the audit log, the result sink type the backend must use, the pool the connection is registered in (there is still one per protocol), a per-statement cleanup hook and `closeConnection`. `qe/protocol` holds only the interface; each front end implements it in a `protocol` subpackage of its own package (`mysql/protocol`, `service/arrowflight/protocol`), which is also where the result senders of the next step go. **`mysql/protocol/MysqlProtocolAdapter`** -- owns the `MysqlChannel` (a socket, the `ProxyMysqlChannel` of a forwarded statement on the master, or the `DummyMysqlChannel` of an internal context), the server and negotiated capabilities, the handshake packet, the SSL context, the `COM_STMT_EXECUTE` packet and its cursor flag, and the xnio accept-query loop that `AcceptListener` / `ReadListener` drive. It also owns the decision `StmtExecutor` and `FEOpExecutor` used to compute from `ConnectContext` fields -- whether the Connector/J release on the other end consumes the metadata terminator of a cursor result (#67520) -- as `clientConsumesCursorMetadataTerminator`. **`service/arrowflight/protocol/FlightProtocolAdapter`** -- owns the peer identity (bearer token), the `FlightSqlChannel`, the prepared queries, the endpoints of the last query, `returnResultFromLocal` and the deferred executors of #62259 / #67503, together with their idle bound. `ConnectContext` keeps `checkTimeout` and the idle reaper unchanged; only the list moved. It also serializes the commands of a session. gRPC runs each call of a session on whatever thread it likes and nothing in the Flight transport orders them, while `ConnectContext` is not thread-safe (the existing `DorisFlightSqlProducerTest` spells that out). `runCommand` / `callCommand` take a per-session lock, make the session the thread's current `ConnectContext` for the duration, restore the previous one afterwards, and give up with `UNAVAILABLE` after the session's query timeout if another command is still running. `DorisFlightSqlProducer` runs statement execution, prepared statement creation and close, DoGet of a frontend-side result and the catalog / schema / table metadata requests through it. DoGet of a frontend-side result streams under the lock on purpose: the next statement of the session resets the channel, whose removal listener closes the `VectorSchemaRoot` being streamed. Session teardown (token expiry, `CloseSession`, `KILL`) does not take the lock; that path is reworked when the token becomes the session credential. **`ConnectContext`** -- gets `protocolAdapter` and three factories: `forMysql(StreamConnection)`, `forMysqlProxy(sessionId)` (replaces the `new ConnectContext(null, true, sessionId)` call in `FrontendServiceImpl`) and `forFlight(peerIdentity)` (replaces the subclass in `FlightSessionsManager`). The existing constructors stay as thin wrappers, so the ~115 test files that call `new ConnectContext()` are untouched. Every protocol-specific getter keeps its signature and delegates: `getMysqlChannel()`, `getCapability()`, `getFlightSqlChannel()`, `isReturnResultFromLocal()` and so on. A getter that only makes sense on the other protocol throws `IllegalStateException` naming the actual protocol (the subclass used to throw a `RuntimeException` for `getMysqlChannel()`; the base class used to return `null` / an empty list for the Flight ones, which no caller relied on). `FlightSqlConnectContext` is deleted: its `getClientIP` / `getRemoteHostPortString` / `closeChannel` / `setQueryId` overrides are the adapter's `remoteHostPortString` / `closeConnection` / `connectPool`, and its `kill` override only differed in log text. Removed as dead code while touching the class: `isSend` / `setIsSend` (nothing read them; the real flag lives on `MysqlChannel`), `cloneContext()` (no caller, and it would have to share a channel between two adapters), and the two lines of `resetConnection()` that cleared Flight-only fields (`COM_RESET_CONNECTION` is only sent by MySQL clients). Not in this PR, deliberately: the execution layer still branches on `ConnectType`, and an internal context is still a MySQL context over a `DummyMysqlChannel`, exactly as before. Both go away in the follow-up PRs that introduce the result sender and the capability bits. ### Verification - **Golden baseline of #67789**: `MysqlPacketGoldenTest` (27 cases, byte for byte) and `FlightResultGoldenTest` pass unchanged. Not a byte of the recorded traffic moved. - New unit tests: `FlightProtocolAdapterTest` (commands of one session run one at a time, a waiting command fails with `UNAVAILABLE` after the query timeout, the thread's current context is set and restored, a failing command releases the session, `KILL` unregisters the session from the Flight pool, the trace id lands in the Flight pool) and `MysqlProtocolAdapterTest` (internal and proxy contexts, the cursor-terminator decision and its per-statement reset, the accept-query loop and close going through the channel). - Existing tests adjusted to the factories: the ones that built a `FlightSqlConnectContext`, poked `mysqlChannel` / `connectType` through reflection, or used a plain `new ConnectContext()` as a Flight session (`ShortCircuitPointQueryTest`, `AuditLogWorkloadGroupTest`, `StmtExecutorTest`, `ConnectContextTest`, `MysqlProtoTest`, `ConnectionExceedTest`). 26 test classes around the session, the MySQL channel and the Flight producer: 166 tests, 0 failures. - Regression on a local cluster built from this branch: `arrow_flight_sql_p0` (8 suites, including the forward-to-master, query-release, point-query, SQL cache and `DatabaseMetaData.getColumns` paths) and `prepared_stmt_p0` (cursor fetch and server-side prepare over MySQL). - `checkstyle:check` on fe-core (main and test sources): 0 violations.
1 parent d1cefa9 commit 83ebddd

29 files changed

Lines changed: 1175 additions & 453 deletions

‎fe/fe-core/src/main/java/org/apache/doris/mysql/AcceptListener.java‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@
2222
import org.apache.doris.common.ErrorCode;
2323
import org.apache.doris.mysql.ProxyProtocolHandler.ProtocolType;
2424
import org.apache.doris.mysql.ProxyProtocolHandler.ProxyProtocolResult;
25+
import org.apache.doris.mysql.protocol.MysqlProtocolAdapter;
2526
import org.apache.doris.qe.ConnectContext;
2627
import org.apache.doris.qe.ConnectProcessor;
2728
import org.apache.doris.qe.ConnectScheduler;
@@ -63,7 +64,7 @@ public void handleEvent(AcceptingChannel<StreamConnection> channel) {
6364
}
6465
// connection has been established, so need to call context.cleanup()
6566
// if exception happens.
66-
ConnectContext context = new ConnectContext(connection);
67+
ConnectContext context = ConnectContext.forMysql(connection);
6768
if (context.getSessionVariable().getQueryTimeoutS() <= 0) {
6869
LOG.warn("Connection query timeout is invalid: {}", context.getSessionVariable().getQueryTimeoutS());
6970
}
@@ -132,7 +133,7 @@ public void handleConnection(ConnectContext context, StreamConnection connection
132133
context.setUserInsertTimeout(
133134
context.getEnv().getAuth().getInsertTimeout(context.getQualifiedUser()));
134135
ConnectProcessor processor = new MysqlConnectProcessor(context);
135-
context.startAcceptQuery(processor);
136+
MysqlProtocolAdapter.of(context).startAcceptQuery(context, processor);
136137
} catch (AfterConnectedException e) {
137138
// do not need to print log for this kind of exception.
138139
// just clean up the context;

‎fe/fe-core/src/main/java/org/apache/doris/mysql/ReadListener.java‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@
1717

1818
package org.apache.doris.mysql;
1919

20+
import org.apache.doris.mysql.protocol.MysqlProtocolAdapter;
2021
import org.apache.doris.qe.ConnectContext;
2122
import org.apache.doris.qe.ConnectProcessor;
2223

@@ -34,10 +35,12 @@
3435
public class ReadListener implements ChannelListener<ConduitStreamSourceChannel> {
3536
private static final Logger LOG = LogManager.getLogger(ReadListener.class);
3637
private ConnectContext ctx;
38+
private MysqlProtocolAdapter protocol;
3739
private ConnectProcessor connectProcessor;
3840

3941
public ReadListener(ConnectContext connectContext, ConnectProcessor connectProcessor) {
4042
this.ctx = connectContext;
43+
this.protocol = MysqlProtocolAdapter.of(connectContext);
4144
this.connectProcessor = connectProcessor;
4245
}
4346

@@ -46,17 +49,17 @@ public void handleEvent(ConduitStreamSourceChannel channel) {
4649
// suspend must be call sync in current thread (the IO-Thread notify the read event),
4750
// otherwise multi handler(task thread) would be waked up by once query.
4851
XnioIoThread.requireCurrentThread();
49-
ctx.suspendAcceptQuery();
52+
protocol.suspendAcceptQuery();
5053
// start async query handle in task thread.
5154
try {
5255
channel.getWorker().execute(() -> {
5356
ctx.setThreadLocalInfo();
5457
try {
5558
connectProcessor.processOnce();
5659
if (!ctx.isKilled()) {
57-
ctx.resumeAcceptQuery();
60+
protocol.resumeAcceptQuery();
5861
} else {
59-
ctx.stopAcceptQuery();
62+
protocol.stopAcceptQuery();
6063
ctx.cleanup();
6164
}
6265
} catch (Throwable e) {
Lines changed: 177 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,177 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
package org.apache.doris.mysql.protocol;
19+
20+
import org.apache.doris.mysql.MysqlCapability;
21+
import org.apache.doris.mysql.MysqlChannel;
22+
import org.apache.doris.mysql.MysqlCursorFetchCompatibility;
23+
import org.apache.doris.mysql.MysqlHandshakePacket;
24+
import org.apache.doris.mysql.MysqlSslContext;
25+
import org.apache.doris.qe.ConnectContext;
26+
import org.apache.doris.qe.ConnectContext.ConnectType;
27+
import org.apache.doris.qe.ConnectPoolMgr;
28+
import org.apache.doris.qe.ConnectProcessor;
29+
import org.apache.doris.qe.ConnectScheduler;
30+
import org.apache.doris.qe.protocol.ProtocolAdapter;
31+
import org.apache.doris.thrift.TResultSinkType;
32+
33+
import java.io.IOException;
34+
import java.nio.ByteBuffer;
35+
36+
/**
37+
* The MySQL protocol side of a connection: the channel to the client, the capabilities negotiated
38+
* with it, the handshake and SSL state, and the COM_STMT_EXECUTE packet being processed.
39+
*
40+
* <p>The channel decides what kind of MySQL connection this is: a {@link MysqlChannel} over a live
41+
* socket for a client, a {@code ProxyMysqlChannel} for the context the master builds to replay a
42+
* forwarded statement, or a {@code DummyMysqlChannel} for an internal context that never talks to
43+
* a client.
44+
*/
45+
public class MysqlProtocolAdapter implements ProtocolAdapter {
46+
private static final String SSL_PROTOCOL = "TLS";
47+
48+
private final MysqlChannel channel;
49+
// the protocol capability which server say it can support
50+
private final MysqlCapability serverCapability = MysqlCapability.DEFAULT_CAPABILITY;
51+
// the protocol capability after server and client negotiate
52+
private volatile MysqlCapability capability;
53+
// This context is used for SSL connection between server and mysql client.
54+
private final MysqlSslContext sslContext = new MysqlSslContext(SSL_PROTOCOL);
55+
private MysqlHandshakePacket handshakePacket;
56+
// The COM_STMT_EXECUTE packet being processed, forwarded as-is when the statement goes to the master.
57+
private ByteBuffer prepareExecuteBuffer;
58+
// Whether the current COM_STMT_EXECUTE requested a server-side read-only cursor.
59+
private boolean cursorFetchRequested;
60+
61+
public MysqlProtocolAdapter(MysqlChannel channel) {
62+
this.channel = channel;
63+
}
64+
65+
/** The MySQL side of {@code ctx}; throws if the connection speaks another protocol. */
66+
public static MysqlProtocolAdapter of(ConnectContext ctx) {
67+
ProtocolAdapter adapter = ctx.getProtocolAdapter();
68+
if (adapter instanceof MysqlProtocolAdapter) {
69+
return (MysqlProtocolAdapter) adapter;
70+
}
71+
throw new IllegalStateException("not a MySQL connection: " + adapter.type());
72+
}
73+
74+
@Override
75+
public ConnectType type() {
76+
return ConnectType.MYSQL;
77+
}
78+
79+
@Override
80+
public String remoteHostPortString(ConnectContext ctx) {
81+
return channel.getRemoteHostPortString();
82+
}
83+
84+
@Override
85+
public TResultSinkType resultSinkType() {
86+
return TResultSinkType.MYSQL_PROTOCOL;
87+
}
88+
89+
@Override
90+
public ConnectPoolMgr connectPool(ConnectScheduler scheduler) {
91+
return scheduler.getConnectPoolMgr();
92+
}
93+
94+
@Override
95+
public void afterStatement(ConnectContext ctx) {
96+
cursorFetchRequested = false;
97+
}
98+
99+
@Override
100+
public void closeConnection(ConnectContext ctx) {
101+
channel.close();
102+
}
103+
104+
public MysqlChannel getChannel() {
105+
return channel;
106+
}
107+
108+
public MysqlCapability getServerCapability() {
109+
return serverCapability;
110+
}
111+
112+
public MysqlCapability getCapability() {
113+
return capability;
114+
}
115+
116+
public void setCapability(MysqlCapability capability) {
117+
this.capability = capability;
118+
}
119+
120+
public MysqlSslContext getSslContext() {
121+
return sslContext;
122+
}
123+
124+
public void setHandshakePacket(MysqlHandshakePacket handshakePacket) {
125+
this.handshakePacket = handshakePacket;
126+
}
127+
128+
public byte[] getAuthPluginData() {
129+
return handshakePacket == null ? null : handshakePacket.getAuthPluginData();
130+
}
131+
132+
public ByteBuffer getPrepareExecuteBuffer() {
133+
return prepareExecuteBuffer;
134+
}
135+
136+
public void setPrepareExecuteBuffer(ByteBuffer prepareExecuteBuffer) {
137+
this.prepareExecuteBuffer = prepareExecuteBuffer;
138+
}
139+
140+
public boolean isCursorFetchRequested() {
141+
return cursorFetchRequested;
142+
}
143+
144+
public void setCursorFetchRequested(boolean cursorFetchRequested) {
145+
this.cursorFetchRequested = cursorFetchRequested;
146+
}
147+
148+
/**
149+
* Whether the client will consume the first OK packet after the column definitions of the
150+
* result it asked a cursor for. Connector/J before 9.5 does, while probing whether the cursor
151+
* was created; Doris never creates one, so the result must carry one more end marker or an
152+
* empty result would leave the client waiting.
153+
*/
154+
public boolean clientConsumesCursorMetadataTerminator(ConnectContext ctx) {
155+
return cursorFetchRequested
156+
&& MysqlCursorFetchCompatibility.resolve(ctx.getConnectAttributes())
157+
!= MysqlCursorFetchCompatibility.Behavior.STANDARD;
158+
}
159+
160+
// The xnio read loop. A command is read and processed while reads are suspended, which is what
161+
// serializes the commands of one connection.
162+
public void startAcceptQuery(ConnectContext ctx, ConnectProcessor connectProcessor) {
163+
channel.startAcceptQuery(ctx, connectProcessor);
164+
}
165+
166+
public void suspendAcceptQuery() {
167+
channel.suspendAcceptQuery();
168+
}
169+
170+
public void resumeAcceptQuery() {
171+
channel.resumeAcceptQuery();
172+
}
173+
174+
public void stopAcceptQuery() throws IOException {
175+
channel.stopAcceptQuery();
176+
}
177+
}

0 commit comments

Comments
 (0)