Skip to content

Commit efe7dec

Browse files
authored
[refactor](session) Move result encoding behind ResultSender (#67883)
### What problem does this PR solve? Issue Number: #67577 Related PR: #67835 (`ProtocolAdapter`), #67866 (package move), #67789 (protocol goldens) Problem Summary: Second step of the protocol-independent session work (#67577, stage 1). #67835 moved the *connection* half of the two wire protocols behind `ProtocolAdapter`; the *result* half was still spread over `StmtExecutor` and `ConnectProcessor` as `ConnectType` branches, a `MysqlSerializer` field, `MysqlChannel` parameters and three copies of "send a text result set". ``` +---------------------------+ +-----------------------------+ | MySQL client | | Arrow Flight SQL client | +-------------+-------------+ +--------------+--------------+ | | v v +---------------------------+ +-----------------------------+ | MysqlServer | | DorisFlightSqlProducer | | AcceptListener | | every call goes through | | ReadListener | | adapter.runCommand() | +-------------+-------------+ +--------------+--------------+ | | v v +---------------------------+ +-----------------------------+ | MysqlConnectProcessor | | FlightSqlConnectProcessor | | COM_FIELD_LIST, and | | | | finalizeCommand() = | | | | adapter.finishCommand() | | | +-------------+-------------+ +--------------+--------------+ | | +----------------------------+-----------------------------+ | v +-------------------------------------------------------------------------------------------+ | ConnectProcessor.executeQuery -- parse, one StmtExecutor per statement, audit | | for each statement: executor.execute() | | adapter.finishStatement(ctx, executor, i, n) <-- NEW | +---------------------------------------------+---------------------------------------------+ | v +-------------------------------------------------------------------------------------------+ | StmtExecutor -- plans and runs one statement, protocol-agnostic result path | | | | FE-side result (SHOW / EXPLAIN / REPLAY / dry run / FE-computable SELECT / forwarded): | | sendResultSet(rs) -> sender.sendResultSet(rs, fieldInfos, binaryRows) | | BE result stream: | | sender.sendFields(...) then sender.sendRow(row) per row | | internal executor streaming to the caller's client: | | executeInternalQueryAndSend(plan, callerCtx.getResultSender()) | | | | no MysqlSerializer field, no MysqlChannel parameter, no ConnectType branch on this path | +---------------------------------------------+---------------------------------------------+ | v +-------------------------------------------------------------------------------------------+ | ConnectContext -- the session, one per connection | | protocolAdapter : ProtocolAdapter getResultSender() = adapter.resultSender(this)| +---------------------------------------------+---------------------------------------------+ | v +-----------------------------------------------+ | <<interface>> qe.protocol.ProtocolAdapter | | type() remoteHostPortString(ctx) | | resultSinkType() connectPool(scheduler) | | afterStatement(ctx) closeConnection(ctx) | | resultSender(ctx) <-- NEW | | supportsSqlCacheReplay() <-- NEW | | finishStatement(ctx, executor, i, n) NEW | +-----------------------+-----------------------+ | +--------------------+----------------------------+ | | +-----------------------+----------------------+ +-----------------------+----------------------+ | mysql.protocol.MysqlProtocolAdapter | | arrowflight.protocol.FlightProtocolAdapter | | (unchanged state: channel, capability, | | (unchanged state: result cache, endpoints, | | handshake, SSL, execute packet, cursor) | | deferred executors, command lock) | | finishStatement: SERVER_MORE_RESULTS_EXISTS| | finishStatement: carry the outcome of a | | + flush the intermediate response when | | statement forwarded to the master; only | | CLIENT_MULTI_STATEMENTS | | the last statement may return a result | | finishCommand: OK/EOF/ERR or the master's | | | | packets; responsePacket(ctx) | | | +-----------------------+----------------------+ +-----------------------+----------------------+ | | v v +----------------------------------------------+ +----------------------------------------------+ | <<interface>> qe.protocol.ResultSender (NEW) | | | | sendResultSet(rs, fieldInfos, binaryRows) | | | | sendFields(names, fieldInfos, types) | | | | sendRow(wireRow) | | | | reset() | | | +----------------------------------------------+ +----------------------------------------------+ | mysql.protocol.MysqlResultSender | | arrowflight.protocol.FlightResultSender | | column count + column definitions + | | caches the ResultSet as Utf8 vectors under | | terminator (EOF / cursor OK) + text or | | the query id for the client's DoGet | | binary rows, through the channel's | | sendFields/sendRow: never called, the | | serializer; a raw row passes through | | client pulls BE results from the BE | | MySQL-only: sendStmtPrepareOK, | | reset: nothing pending | | sendFieldList | | | +----------------------------------------------+ +----------------------------------------------+ ``` **`ResultSender`** (`qe.protocol`, interface; implementations in `mysql.protocol` and `arrowflight.protocol`, mirroring the adapters): how a statement's result reaches the client. Four operations, all of them already used: `sendResultSet` for a result the frontend materialized, `sendFields` + `sendRow` for a backend result stream, `reset` for what `MysqlChannel.reset()` did at the start of a query. `MysqlResultSender` is the old `sendMetaData / sendFields / sendTextResultRow / sendBinaryResultRow / sendMetadataTerminatorIfNeeded / sendStmtPrepareOK` of `StmtExecutor` plus the `COM_FIELD_LIST` body of `ConnectProcessor`, moved without changes to the byte layout; it uses the channel's serializer, so the executor's `serializer` field is gone. `FlightResultSender` wraps `FlightSqlChannel.addResult` (every column still `Utf8`, typing them is stage 2). **`StmtExecutor`** no longer takes a `MysqlChannel`: `executeAndSendResult`, `sendCachedValues` and `executeInternalQueryAndSend` take a `ResultSender`. The three "text result" methods (`handleExplainStmt`, `handleReplayStmt`, `handleExplainPlanProcessStmt`) build a `ShowResultSet` and go through the one `sendResultSet`, which fixes `EXPLAIN PLAN PROCESS` returning nothing on an Arrow Flight SQL session (it had no Flight branch). The `MysqlChannel` overloads #67753 added for the IVM dry run become "hand the internal executor the caller's sender": `RefreshMTMVCommand` passes `ctx.getResultSender()`, and the internal executor's rows are encoded with the caller's negotiated capabilities instead of the internal context's defaults. **`ConnectProcessor`** loses its `connectType` field and every branch on it. The per-statement protocol work of a multi-statement request is one call, `adapter.finishStatement(ctx, executor, i, n)`: for MySQL it sets `SERVER_MORE_RESULTS_EXISTS` and flushes the intermediate response when the client negotiated `CLIENT_MULTI_STATEMENTS`; for Flight it carries a forwarded statement's outcome into the session (the former `carryForwardedOutcomeToFlightSession`) and enforces "only the last statement may return a result". `finalizeCommand` / `getResultPacket` move to `MysqlProtocolAdapter.finishCommand` / `responsePacket` and `COM_FIELD_LIST` to `MysqlConnectProcessor`, the only processor that dispatches it. The SQL-cache guard is `adapter.supportsSqlCacheReplay()` (true only for MySQL, whose wire rows the cache stores). Also folded in, from the local review of #67835: `FlightProtocolAdapter.acquireCommandLock` waits up to `getExecTimeoutS()` (a sync load may legitimately run past `query_timeout`) and logs when it gives up; `getFlightInfoStatement` / `streamMetadata` pass a `FlightRuntimeException` through instead of re-wrapping `UNAVAILABLE` as `INTERNAL`; `of(ctx)` names a null adapter instead of throwing NPE from the error path; `testFailedCommandReleasesTheSession` checks the lock from a second thread; `ConnectProcessorForwardProtocolTest` binds its recording channel through the adapter instead of overriding `getMysqlChannel()`. Not in this PR (next one, PR-1.3): the remaining `ConnectType` branches outside the result path (`returnResultFromLocal`, the Flight early return in `executeAndSendResult`, the retry condition, `supportHandleByFe`, the nine `getMysqlChannel().reset()` in insert / transaction commands, `FEOpExecutor`, the coordinators), which become capability bits on the adapter.
1 parent e797e99 commit efe7dec

28 files changed

Lines changed: 1479 additions & 966 deletions

‎fe/fe-core/src/main/java/org/apache/doris/arrowflight/DorisFlightSqlProducer.java‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,7 @@
4343
import org.apache.arrow.flight.FlightDescriptor;
4444
import org.apache.arrow.flight.FlightEndpoint;
4545
import org.apache.arrow.flight.FlightInfo;
46+
import org.apache.arrow.flight.FlightRuntimeException;
4647
import org.apache.arrow.flight.FlightStream;
4748
import org.apache.arrow.flight.Location;
4849
import org.apache.arrow.flight.PutResult;
@@ -317,6 +318,10 @@ public FlightInfo getFlightInfoStatement(final CommandStatementQuery request, fi
317318
return FlightProtocolAdapter.of(connectContext).callCommand(connectContext,
318319
() -> executeQueryStatement(context.peerIdentity(), connectContext, request.getQuery(),
319320
descriptor));
321+
} catch (FlightRuntimeException e) {
322+
// Already carries the status meant for the client, e.g. UNAVAILABLE from the session's
323+
// command lock; wrapping it as INTERNAL would hide that.
324+
throw e;
320325
} catch (Throwable e) {
321326
String errMsg = "get flight info statement failed, " + e.getMessage();
322327
LOG.error(errMsg, e);
@@ -660,6 +665,11 @@ private void streamMetadata(final CallContext context, final ServerStreamListene
660665
try {
661666
ConnectContext connectContext = flightSessionsManager.getConnectContext(context.peerIdentity());
662667
FlightProtocolAdapter.of(connectContext).runCommand(connectContext, () -> stream.send(connectContext));
668+
} catch (FlightRuntimeException e) {
669+
// Same as in getFlightInfoStatement: keep the status the session's command lock chose.
670+
LOG.error("stream metadata failed", e);
671+
listener.error(e);
672+
throw e;
663673
} catch (final Throwable e) {
664674
handleStreamException(e, "", listener);
665675
}

‎fe/fe-core/src/main/java/org/apache/doris/arrowflight/FlightSqlConnectProcessor.java‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,6 @@
2828
import org.apache.doris.proto.InternalService;
2929
import org.apache.doris.proto.Types;
3030
import org.apache.doris.qe.ConnectContext;
31-
import org.apache.doris.qe.ConnectContext.ConnectType;
3231
import org.apache.doris.qe.ConnectProcessor;
3332
import org.apache.doris.qe.StmtExecutor;
3433
import org.apache.doris.rpc.BackendServiceProxy;
@@ -64,7 +63,6 @@ public class FlightSqlConnectProcessor extends ConnectProcessor implements AutoC
6463

6564
public FlightSqlConnectProcessor(ConnectContext context) {
6665
super(context);
67-
connectType = ConnectType.ARROW_FLIGHT_SQL;
6866
context.setThreadLocalInfo();
6967
context.setReturnResultFromLocal(true);
7068
}

‎fe/fe-core/src/main/java/org/apache/doris/arrowflight/protocol/FlightProtocolAdapter.java‎

Lines changed: 82 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -21,20 +21,26 @@
2121
import org.apache.doris.arrowflight.results.FlightSqlChannel;
2222
import org.apache.doris.arrowflight.results.FlightSqlEndpointsLocation;
2323
import org.apache.doris.common.Config;
24+
import org.apache.doris.common.ErrorCode;
2425
import org.apache.doris.qe.ConnectContext;
2526
import org.apache.doris.qe.ConnectContext.ConnectType;
2627
import org.apache.doris.qe.ConnectPoolMgr;
2728
import org.apache.doris.qe.ConnectScheduler;
29+
import org.apache.doris.qe.QueryState;
30+
import org.apache.doris.qe.ShowResultSet;
2831
import org.apache.doris.qe.StmtExecutor;
2932
import org.apache.doris.qe.protocol.ProtocolAdapter;
3033
import org.apache.doris.thrift.TResultSinkType;
3134

35+
import com.google.common.annotations.VisibleForTesting;
36+
import com.google.common.base.Preconditions;
3237
import com.google.common.base.Strings;
3338
import com.google.common.collect.Lists;
3439
import org.apache.arrow.flight.CallStatus;
3540
import org.apache.logging.log4j.LogManager;
3641
import org.apache.logging.log4j.Logger;
3742

43+
import java.io.IOException;
3844
import java.util.ArrayList;
3945
import java.util.HashMap;
4046
import java.util.List;
@@ -84,7 +90,8 @@ public static FlightProtocolAdapter of(ConnectContext ctx) {
8490
if (adapter instanceof FlightProtocolAdapter) {
8591
return (FlightProtocolAdapter) adapter;
8692
}
87-
throw new IllegalStateException("not an Arrow Flight SQL connection: " + adapter.type());
93+
throw new IllegalStateException("not an Arrow Flight SQL connection: "
94+
+ (adapter == null ? "no protocol adapter" : adapter.type()));
8895
}
8996

9097
@Override
@@ -106,11 +113,77 @@ public TResultSinkType resultSinkType() {
106113
return TResultSinkType.ARROW_FLIGHT_PROTOCOL;
107114
}
108115

116+
@Override
117+
public FlightResultSender resultSender(ConnectContext ctx) {
118+
return new FlightResultSender(ctx, this);
119+
}
120+
121+
/**
122+
* The SQL cache keeps the result rows in MySQL wire format, which cannot be turned into the
123+
* Arrow batches a Flight client needs; the cached rows would be wrong for it anyway (object
124+
* types such as HLL / BITMAP / QUANTILE_STATE were serialized as NULL under
125+
* return_object_data_as_binary=false). A Flight session always re-executes the query.
126+
*/
127+
@Override
128+
public boolean supportsSqlCacheReplay() {
129+
return false;
130+
}
131+
109132
@Override
110133
public ConnectPoolMgr connectPool(ConnectScheduler scheduler) {
111134
return scheduler.getFlightSqlConnectPoolMgr();
112135
}
113136

137+
/**
138+
* A statement forwarded to the master has its outcome carried into this session here, the
139+
* way {@code MysqlProtocolAdapter.finishCommand} replays it to a MySQL client. And of the
140+
* statements of one request only the last may produce a result: the FlightInfo returned for
141+
* the request describes exactly one.
142+
*/
143+
@Override
144+
public boolean finishStatement(ConnectContext ctx, StmtExecutor executor, int stmtIndex, int stmtCount)
145+
throws IOException {
146+
if (executor.hasForwardedToMaster()) {
147+
carryForwardedOutcome(ctx, executor);
148+
}
149+
Preconditions.checkState(channel.resultNum() <= 1);
150+
if (channel.resultNum() == 1 && stmtIndex != stmtCount - 1) {
151+
String errMsg = "Only be one stmt that returns the result and it is at the end. "
152+
+ "stmts.size(): " + stmtCount;
153+
LOG.warn(errMsg);
154+
ctx.getState().setError(ErrorCode.ERR_ARROW_FLIGHT_SQL_MUST_ONLY_RESULT_STMT, errMsg);
155+
ctx.getState().setErrType(QueryState.ErrType.OTHER_ERR);
156+
return false;
157+
}
158+
return true;
159+
}
160+
161+
// The master answers a forwarded statement with its status and, for a SHOW, its rows. Without
162+
// this a forwarded statement leaves ctx.getState() at the OK that executeQuery() set with
163+
// reset() and leaves the FlightSqlChannel empty, so DorisFlightSqlProducer answers with
164+
// addOKResult()'s synthesized StatusResult=0 -- reporting success for a statement that failed
165+
// on the master, and an empty status row instead of the rows a forwarded SHOW produced.
166+
@VisibleForTesting
167+
void carryForwardedOutcome(ConnectContext ctx, StmtExecutor executor) throws IOException {
168+
if (executor.getProxyStatusCode() != 0) {
169+
// The master rejected the statement, e.g. CREATE TABLE on a table that already exists.
170+
// TMasterOpResult carries the master's error code as a plain int and ErrorCode has no
171+
// reverse lookup, so the master's code travels in the message instead.
172+
String errMsg = "forwarded statement failed on master FE, error code: "
173+
+ executor.getProxyStatusCode() + ", error message: " + executor.getProxyErrMsg();
174+
LOG.warn(errMsg);
175+
ctx.getState().setError(ErrorCode.ERR_UNKNOWN_ERROR, errMsg);
176+
return;
177+
}
178+
// Set exactly when the forwarded statement produced rows: proxyExecute() fills
179+
// TMasterOpResult.resultSet from getProxyShowResultSet(). A forwarded DDL produces none,
180+
// and the synthesized StatusResult=0 is the right answer for it.
181+
ShowResultSet resultSet = executor.getShowResultSet();
182+
if (resultSet != null) {
183+
executor.sendResultSet(resultSet);
184+
}
185+
}
186+
114187
@Override
115188
public void afterStatement(ConnectContext ctx) {
116189
// The channel keeps the statement's result until the client pulls it with DoGet, and the
@@ -238,7 +311,7 @@ public interface SessionCommand<T, E extends Exception> {
238311
* Runs one command of the session, and no other one at the same time: a statement, a prepared
239312
* statement action, a DoGet of a frontend-side result, a metadata request. The session's
240313
* {@link ConnectContext} is the thread's current context while the command runs. A command
241-
* that finds another one still running waits for it up to the session's query timeout and
314+
* that finds another one still running waits for it up to the session's execution timeout and
242315
* then fails with {@code UNAVAILABLE} instead of running concurrently on the same context.
243316
*
244317
* <p>Session teardown (bearer token expiry, CloseSession, KILL) does not go through here.
@@ -268,16 +341,22 @@ public <T, E extends Exception> T callCommand(ConnectContext ctx, SessionCommand
268341
}
269342

270343
private void acquireCommandLock(ConnectContext ctx) {
271-
long waitS = ctx.getQueryTimeoutS();
344+
// Wait as long as the running command is allowed to run: for a synchronous load statement
345+
// that is max(insert_timeout, query_timeout), the bound the timeout checker applies to it.
346+
long waitS = ctx.getExecTimeoutS();
272347
boolean locked;
273348
try {
274349
locked = commandLock.tryLock(waitS, TimeUnit.SECONDS);
275350
} catch (InterruptedException e) {
276351
Thread.currentThread().interrupt();
352+
LOG.warn("interrupted while waiting for the running command of Arrow Flight SQL connection {}",
353+
ctx.getConnectionId());
277354
throw CallStatus.CANCELLED.withDescription("interrupted while waiting for the previous command of "
278355
+ "this Arrow Flight SQL session to finish").withCause(e).toRuntimeException();
279356
}
280357
if (!locked) {
358+
LOG.warn("a command of Arrow Flight SQL connection {} gave up after waiting {}s for the running one",
359+
ctx.getConnectionId(), waitS);
281360
throw CallStatus.UNAVAILABLE.withDescription(String.format("another command of this Arrow Flight SQL "
282361
+ "session is still running after %d seconds, connection id: %d", waitS, ctx.getConnectionId()))
283362
.toRuntimeException();
Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
package org.apache.doris.arrowflight.protocol;
19+
20+
import org.apache.doris.catalog.Type;
21+
import org.apache.doris.common.util.DebugUtil;
22+
import org.apache.doris.mysql.FieldInfo;
23+
import org.apache.doris.qe.ConnectContext;
24+
import org.apache.doris.qe.ResultSet;
25+
import org.apache.doris.qe.protocol.ResultSender;
26+
27+
import java.nio.ByteBuffer;
28+
import java.util.List;
29+
30+
/**
31+
* Delivers the results of an Arrow Flight SQL session. A result the frontend materialized is
32+
* cached on the session's {@code FlightSqlChannel} under the statement's query id, and
33+
* {@code DorisFlightSqlProducer} hands it out when the client's DoGet arrives. A result a backend
34+
* produces never passes through the frontend: the client pulls it from the backend with the
35+
* endpoints GetFlightInfo returned, so this sender has no row stream.
36+
*
37+
* <p>Every column of a cached result is a {@code Utf8} vector for now; typing them by the result
38+
* set's column types is a later step.
39+
*/
40+
public class FlightResultSender implements ResultSender {
41+
private final ConnectContext ctx;
42+
private final FlightProtocolAdapter adapter;
43+
44+
FlightResultSender(ConnectContext ctx, FlightProtocolAdapter adapter) {
45+
this.ctx = ctx;
46+
this.adapter = adapter;
47+
}
48+
49+
@Override
50+
public void sendResultSet(ResultSet resultSet, List<FieldInfo> fieldInfos, boolean binaryRows) {
51+
adapter.getChannel().addResult(DebugUtil.printId(ctx.queryId()), adapter.getRunningQuery(), resultSet);
52+
// The statement's result is on this frontend, whatever the query path decided earlier: an
53+
// EXPLAIN goes through the query path, which marks the result as coming from the backend
54+
// before it knows the statement will not run there.
55+
adapter.setReturnResultFromLocal(true);
56+
}
57+
58+
@Override
59+
public void sendFields(List<String> colNames, List<FieldInfo> fieldInfos, List<Type> types) {
60+
throw new IllegalStateException(
61+
"an Arrow Flight SQL client pulls query results from the backend, not through the frontend");
62+
}
63+
64+
@Override
65+
public void sendRow(ByteBuffer row) {
66+
throw new IllegalStateException(
67+
"an Arrow Flight SQL client pulls query results from the backend, not through the frontend");
68+
}
69+
70+
@Override
71+
public void reset() {
72+
// Results are cached per query id and the cache is cleared when the next request of the
73+
// session starts (DorisFlightSqlProducer.executeQueryStatement); nothing is pending here.
74+
}
75+
}

‎fe/fe-core/src/main/java/org/apache/doris/arrowflight/results/FlightSqlChannel.java‎

Lines changed: 0 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -84,34 +84,6 @@ public void addResult(String queryId, String runningQuery, ResultSet resultSet)
8484
resultCache.put(queryId, flightSqlResultCacheEntry);
8585
}
8686

87-
public void addResult(String queryId, String runningQuery, ResultSetMetaData metaData, String result) {
88-
List<Field> schemaFields = new ArrayList<>();
89-
List<FieldVector> dataFields = new ArrayList<>();
90-
91-
// TODO: only support varchar type
92-
for (Column col : metaData.getColumns()) {
93-
schemaFields.add(new Field(col.getName(), FieldType.nullable(new Utf8()), null));
94-
VarCharVector varCharVector = new VarCharVector(col.getName(), allocator);
95-
varCharVector.allocateNew();
96-
varCharVector.setValueCount(result.split("\n").length);
97-
dataFields.add(varCharVector);
98-
}
99-
100-
int rowNum = 0;
101-
for (String item : result.split("\n")) {
102-
if (item == null || item.equals(FeConstants.null_string)) {
103-
dataFields.get(0).setNull(rowNum);
104-
} else {
105-
((VarCharVector) dataFields.get(0)).setSafe(rowNum, item.getBytes());
106-
}
107-
rowNum += 1;
108-
}
109-
VectorSchemaRoot vectorSchemaRoot = new VectorSchemaRoot(schemaFields, dataFields);
110-
final FlightSqlResultCacheEntry flightSqlResultCacheEntry = new FlightSqlResultCacheEntry(vectorSchemaRoot,
111-
runningQuery);
112-
resultCache.put(queryId, flightSqlResultCacheEntry);
113-
}
114-
11587
/**
11688
* Create a SchemaRoot with one row and one column.
11789
*/

0 commit comments

Comments
 (0)