Skip to content

Commit 674e491

Browse files
morningmanclaude
andcommitted
[feature](arrow-flight) Close the session to what is deferred after teardown, count a backend result as one, fence the empty schema
Three gaps left by the previous commits, each closed where its rule is stated. - Teardown (CloseSession, the bearer token's expiry, KILL, the timeout checker: all through FlightSqlConnectPoolMgr.unregisterConnection) does not wait for the command that may be running on the session. Such a command could still defer its query after teardown had taken everything the list held, and nothing would ever take that executor: the session runs no next request, and the timeout checker no longer sees the context. The coordinator, its batch split source, its queue slot and its query registration stayed until the frontend restarted. Teardown now tombstones the adapter (FlightProtocolAdapter.tearDown, set under the list's monitor): an executor deferred after it is finalized on the spot by the command that deferred it, and a command that gets its turn on the lock after teardown fails with UNAUTHENTICATED, as any later call of the session would, instead of running on a context nothing tracks. The exactly-once rule in the class comment holds across teardown now. - finishStatement enforced "only the last statement of a request may produce a result" for results cached on this frontend alone; a result left on the backends was not counted, so SELECT; SELECT delivered both queries' endpoints in one FlightInfo (read as the partitions of one result when the schemas agree) and SELECT; SET dropped the query's endpoints for the SET's status. A query that is not the last statement is now refused with the same error, before the next statement runs -- which also finalizes it while the session's query id is still its own. A statement that failed produced no result and keeps its own error. - The empty schema value, which puts the session back into no database without running a statement, went around the one gate every statement passes first: a transaction takes only its own statements, and USE, SWITCH and SET are refused while one is open. Leaving the database is refused the same way now (ERROR, as USE answers there), with nothing mutated. Tests: FlightProtocolAdapterTest (a torn-down session, through the real KILL path with the command still holding the lock: what it defers afterwards is finalized once by the command, and the next command is refused; a backend result before the last statement is refused, as the last it is the request's, a failed query keeps its error), FlightSessionOptionsTest.testNoOptionChangesTheSessionInsideATransaction (real BEGIN / ROLLBACK, the empty value and the empty string), FlightSqlConnectPoolMgrTest (teardown reaches tearDown), and the lifecycle regression suite (SELECT; SELECT and SELECT; SET refused, the SET not run). Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01Ja986K4PEm2LD8u44L9jd2
1 parent ad7c70b commit 674e491

8 files changed

Lines changed: 228 additions & 22 deletions

File tree

‎fe/fe-core/src/main/java/org/apache/doris/arrowflight/FlightSessionOptions.java‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -163,8 +163,16 @@ private static ErrorValue useDatabase(ConnectContext ctx, SessionOptionValue val
163163
// No database: the state the session started in, which GetSessionOptions reports as the
164164
// empty string and the empty value asks for back (Flight's way of unsetting an option; what
165165
// the ADBC driver sends to erase one). No statement leads there -- there is no USE of
166-
// nothing -- so it is not one the session runs, and there is nothing in it to check.
166+
// nothing -- so it is not one the session runs, and nothing of the value is left to check.
167+
// What every statement is checked for first still holds, though: a transaction takes only
168+
// its own statements (the executor refuses USE, SWITCH and SET while one is open), and the
169+
// session leaves its database no more than it enters another.
167170
if (database.isEmpty()) {
171+
if (ctx.isTxnModel()) {
172+
LOG.warn("session option {} of Arrow Flight SQL connection {} could not be set, the session is "
173+
+ "in a transaction", SCHEMA, ctx.getConnectionId());
174+
return ErrorValue.ERROR;
175+
}
168176
ctx.clearDatabase();
169177
return null;
170178
}

‎fe/fe-core/src/main/java/org/apache/doris/arrowflight/protocol/FlightProtocolAdapter.java‎

Lines changed: 63 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -76,10 +76,13 @@
7676
* that runs no command of the session (the timeout checker, a token expiry). It is finalized
7777
* exactly once, by whoever takes it out of the list under the list's lock -- the next request
7878
* ({@link #beginRequest}), the timeout checker ({@link #takeExpiredDeferredExecutors}, each
79-
* executor by its own deadline) or teardown ({@link #closeDeferredExecutors}) -- deciding and
80-
* taking in one critical section, finalizing outside it. The frontend has no signal for the moment
81-
* a query is done on the backends; the next request stands in for it, as it has since #62259,
82-
* and the deadline bounds the wait when no request comes.
79+
* executor by its own deadline) or teardown ({@link #tearDown}) -- deciding and taking in one
80+
* critical section, finalizing outside it. Teardown does not wait for a running command, so a
81+
* query may be deferred after teardown took everything the list held; it is then finalized on the
82+
* spot by the command that deferred it ({@link #addDeferredExecutor}), since nothing would take it
83+
* later. The frontend has no signal for the moment a query is done on the backends; the next
84+
* request stands in for it, as it has since #62259, and the deadline bounds the wait when no
85+
* request comes.
8386
*/
8487
public class FlightProtocolAdapter implements ProtocolAdapter {
8588
private static final Logger LOG = LogManager.getLogger(FlightProtocolAdapter.class);
@@ -110,6 +113,10 @@ public class FlightProtocolAdapter implements ProtocolAdapter {
110113
// the commands of the session add and take under the command lock, the timeout checker and
111114
// teardown take without it.
112115
private final List<StmtExecutor> deferredExecutors = new ArrayList<>();
116+
// Whether the session has been torn down (tearDown): set under the monitor of
117+
// deferredExecutors, so that a publication sees either the list or the tombstone; read at the
118+
// entry of every command as well.
119+
private volatile boolean closed = false;
113120
// Serializes the commands of this session, see runCommand.
114121
private final ReentrantLock commandLock = new ReentrantLock();
115122

@@ -265,7 +272,10 @@ public ConnectPoolMgr connectPool(ConnectScheduler scheduler) {
265272
* A statement forwarded to the master has its outcome carried into this session here, the
266273
* way {@code MysqlProtocolAdapter.finishCommand} replays it to a MySQL client. And of the
267274
* statements of one request only the last may produce a result: the FlightInfo returned for
268-
* the request describes exactly one.
275+
* the request describes exactly one, wherever it is. A result this frontend cached and one
276+
* left on the backends count the same -- the endpoints of two queries in one FlightInfo would
277+
* be read as the partitions of one result, and a query before a SET would have its endpoints
278+
* dropped for the SET's status.
269279
*/
270280
@Override
271281
public boolean finishStatement(ConnectContext ctx, StmtExecutor executor, int stmtIndex, int stmtCount)
@@ -274,7 +284,13 @@ public boolean finishStatement(ConnectContext ctx, StmtExecutor executor, int st
274284
carryForwardedOutcome(ctx, executor);
275285
}
276286
Preconditions.checkState(channel.resultNum() <= 1);
277-
if (channel.resultNum() == 1 && stmtIndex != stmtCount - 1) {
287+
// A statement that succeeded produced a result when it cached one on the channel or left
288+
// one on the backends for the client to pull (beforeQuery); one that failed produced none,
289+
// whatever it registered before failing, and the request stops with the statement's own
290+
// error.
291+
boolean producedResult = ctx.getState().getStateType() != QueryState.MysqlStateType.ERR
292+
&& (channel.resultNum() == 1 || !returnResultFromLocal);
293+
if (producedResult && stmtIndex != stmtCount - 1) {
278294
String errMsg = "Only be one stmt that returns the result and it is at the end. "
279295
+ "stmts.size(): " + stmtCount;
280296
LOG.warn(errMsg);
@@ -374,16 +390,27 @@ public void beginRequest() {
374390
returnResultFromLocal = true;
375391
}
376392

393+
/**
394+
* Keeps a deferred executor for the session to finalize later -- unless the session has been
395+
* torn down meanwhile ({@link #tearDown}). Teardown does not wait for a running command, so
396+
* its query may be deferred after teardown took everything the list held, and nothing would
397+
* take it then: the session runs no next request, and the timeout checker no longer sees it.
398+
* Such an executor is finalized here and now, by the command that deferred it.
399+
*/
377400
public void addDeferredExecutor(StmtExecutor executor) {
378401
synchronized (deferredExecutors) {
379-
deferredExecutors.add(executor);
402+
if (!closed) {
403+
deferredExecutors.add(executor);
404+
return;
405+
}
380406
}
407+
finalizeDeferredExecutors(Collections.singletonList(executor));
381408
}
382409

383410
/**
384411
* Takes every deferred executor out of the list and finalizes it: the next request of the
385412
* session ({@link #beginRequest}), which stands in for the end of the previous query's DoGet, and
386-
* the session's teardown.
413+
* a request that failed after deferring its query, which no DoGet will ever pull.
387414
*/
388415
public void closeDeferredExecutors() {
389416
List<StmtExecutor> taken;
@@ -397,6 +424,24 @@ public void closeDeferredExecutors() {
397424
finalizeDeferredExecutors(taken);
398425
}
399426

427+
/**
428+
* Tears the session down: takes every deferred executor out of the list and finalizes it, and
429+
* closes the list for good. Teardown -- CloseSession, the bearer token's expiry, KILL, the
430+
* timeout checker -- does not wait for the command that may be running, so that command may
431+
* still defer its query afterwards, which is then finalized on the spot
432+
* ({@link #addDeferredExecutor}); and a command that was waiting for the session does not run
433+
* on it ({@link #callCommand}). Reached through the pool's unregisterConnection.
434+
*/
435+
public void tearDown() {
436+
List<StmtExecutor> taken;
437+
synchronized (deferredExecutors) {
438+
closed = true;
439+
taken = new ArrayList<>(deferredExecutors);
440+
deferredExecutors.clear();
441+
}
442+
finalizeDeferredExecutors(taken);
443+
}
444+
400445
/**
401446
* Takes out of the list the deferred executors whose own bound has passed by {@code now} and
402447
* returns them for the caller to finalize: the timeout checker's half of the exactly-once rule
@@ -485,7 +530,9 @@ public interface SessionCommand<T, E extends Exception> {
485530
* finds another one still running waits for it up to the session's execution timeout and then
486531
* fails with {@code UNAVAILABLE} instead of running concurrently on the same context.
487532
*
488-
* <p>Session teardown (bearer token expiry, CloseSession, KILL) does not go through here.
533+
* <p>Session teardown (bearer token expiry, CloseSession, KILL) does not go through here and
534+
* does not wait for the running command ({@link #tearDown}); a command that gets its turn
535+
* after teardown fails with {@code UNAUTHENTICATED}, as any later call of the session would.
489536
*/
490537
public <E extends Exception> void runCommand(ConnectContext ctx, SessionAction<E> action) throws E {
491538
this.<Void, E>callCommand(ctx, () -> {
@@ -497,10 +544,15 @@ public <E extends Exception> void runCommand(ConnectContext ctx, SessionAction<E
497544
/** {@link #runCommand} for a command that returns a value. */
498545
public <T, E extends Exception> T callCommand(ConnectContext ctx, SessionCommand<T, E> command) throws E {
499546
acquireCommandLock(ctx);
500-
ctx.refreshStartTime();
501547
ConnectContext previous = ConnectContext.get();
502-
ctx.setThreadLocalInfo();
503548
try {
549+
if (closed) {
550+
throw CallStatus.UNAUTHENTICATED.withDescription(String.format("this Arrow Flight SQL session "
551+
+ "was closed, connection id: %d; reconnect to run further commands", ctx.getConnectionId()))
552+
.toRuntimeException();
553+
}
554+
ctx.refreshStartTime();
555+
ctx.setThreadLocalInfo();
504556
return command.call();
505557
} finally {
506558
if (previous == null) {

‎fe/fe-core/src/main/java/org/apache/doris/arrowflight/sessions/FlightSqlConnectPoolMgr.java‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -76,8 +76,10 @@ public void unregisterConnection(ConnectContext ctx) {
7676
}
7777
// Finalize any Arrow Flight query whose coordinator was kept alive across the
7878
// GetFlightInfo -> DoGet phases (see #62259), releasing its resources (e.g. external-table
79-
// batch SplitSources and the query queue slot).
80-
ctx.closeFlightSqlDeferredExecutors();
79+
// batch SplitSources and the query queue slot), and close the session for good: teardown
80+
// does not wait for a command that may still be running, and what that command defers
81+
// afterwards is finalized on the spot (FlightProtocolAdapter.tearDown).
82+
ctx.tearDownFlightSqlSession();
8183
ctx.closeTxn();
8284
if (connectionMap.remove(ctx.getConnectionId()) != null) {
8385
numberConnection.decrementAndGet();

‎fe/fe-core/src/main/java/org/apache/doris/qe/ConnectContext.java‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1049,6 +1049,15 @@ public void closeFlightSqlDeferredExecutors() {
10491049
}
10501050
}
10511051

1052+
// The session is over (see FlightProtocolAdapter.tearDown): its deferred executors are
1053+
// finalized, and it keeps none and runs no command from now on. Nothing to do for a
1054+
// connection of any other protocol.
1055+
public void tearDownFlightSqlSession() {
1056+
if (protocolAdapter instanceof FlightProtocolAdapter) {
1057+
((FlightProtocolAdapter) protocolAdapter).tearDown();
1058+
}
1059+
}
1060+
10521061
// A snapshot; empty for a connection of any other protocol.
10531062
@VisibleForTesting
10541063
public List<StmtExecutor> getFlightSqlDeferredExecutors() {

‎fe/fe-core/src/test/java/org/apache/doris/arrowflight/FlightSessionOptionsTest.java‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@
2525
import org.apache.doris.catalog.ScalarType;
2626
import org.apache.doris.common.util.DebugUtil;
2727
import org.apache.doris.qe.ConnectContext;
28+
import org.apache.doris.qe.QueryState.MysqlStateType;
2829
import org.apache.doris.qe.ShowResultSet;
2930
import org.apache.doris.qe.ShowResultSetMetaData;
3031
import org.apache.doris.qe.SqlModeHelper;
@@ -103,6 +104,18 @@ private Map<String, SetSessionOptionsResult.Error> setAll(ConnectContext ctx,
103104
}
104105
}
105106

107+
// Runs a statement of the session the way a request does, and leaves the test's own context on
108+
// the thread afterwards.
109+
private void run(ConnectContext ctx, String statement) throws Exception {
110+
try (FlightSqlConnectProcessor processor = new FlightSqlConnectProcessor(ctx)) {
111+
processor.handleQuery(statement);
112+
Assertions.assertNotEquals(MysqlStateType.ERR, ctx.getState().getStateType(),
113+
statement + " failed: " + ctx.getState().getErrorMessage());
114+
} finally {
115+
connectContext.setThreadLocalInfo();
116+
}
117+
}
118+
106119
private static String get(ConnectContext ctx, String name) {
107120
SessionOptionValue value = FlightSessionOptions.get(ctx).get(name);
108121
Assertions.assertNotNull(value, "option " + name + " is not reported");
@@ -179,6 +192,38 @@ public void testTheEmptyValueOrStringLeavesTheDatabase() {
179192
FlightSessionOptions.SCHEMA, SessionOptionValueFactory.makeEmptySessionOptionValue()));
180193
}
181194

195+
// A transaction takes only its own statements -- an insert, a commit, a rollback -- so while one
196+
// is open every option is refused as the statement it stands for is. Leaving the database is
197+
// refused with them, though no statement is run for it: the one change an option makes to the
198+
// session on its own goes through the same gate as the ones the statements make.
199+
@Test
200+
public void testNoOptionChangesTheSessionInsideATransaction() throws Exception {
201+
ConnectContext ctx = rootSession();
202+
Assertions.assertNull(set(ctx, FlightSessionOptions.SCHEMA, str(DB)));
203+
int queryTimeout = ctx.getSessionVariable().getQueryTimeoutS();
204+
run(ctx, "BEGIN");
205+
Assertions.assertTrue(ctx.isTxnModel());
206+
try {
207+
Assertions.assertEquals(ErrorValue.ERROR,
208+
set(ctx, FlightSessionOptions.SCHEMA, SessionOptionValueFactory.makeEmptySessionOptionValue()));
209+
Assertions.assertEquals(ErrorValue.ERROR, set(ctx, FlightSessionOptions.SCHEMA, str("")));
210+
Assertions.assertEquals(ErrorValue.ERROR, set(ctx, FlightSessionOptions.SCHEMA, str(OTHER_DB)));
211+
Assertions.assertEquals(ErrorValue.ERROR, set(ctx, FlightSessionOptions.CATALOG, str("internal")));
212+
// Every refusal of SET is reported as the value's (see setVariable), this one included.
213+
Assertions.assertEquals(ErrorValue.INVALID_VALUE,
214+
set(ctx, "query_timeout", SessionOptionValueFactory.makeSessionOptionValue(queryTimeout + 1L)));
215+
Assertions.assertEquals(DB, ctx.getDatabase());
216+
Assertions.assertEquals(DB, get(ctx, FlightSessionOptions.SCHEMA));
217+
Assertions.assertEquals(queryTimeout, ctx.getSessionVariable().getQueryTimeoutS());
218+
Assertions.assertTrue(ctx.isTxnModel());
219+
} finally {
220+
run(ctx, "ROLLBACK");
221+
}
222+
Assertions.assertFalse(ctx.isTxnModel());
223+
Assertions.assertNull(set(ctx, FlightSessionOptions.SCHEMA, str("")));
224+
Assertions.assertEquals("", ctx.getDatabase());
225+
}
226+
182227
@Test
183228
public void testWhatTheStatementRefusesForLackOfPrivilegeIsAnError() throws Exception {
184229
ConnectContext ctx = flightSession(UserIdentity.createAnalyzedUserIdentWithIp(USER, "%"));

0 commit comments

Comments
 (0)