Skip to content

Commit ad7c70b

Browse files
morningmanclaude
andcommitted
[feature](arrow-flight) Make a deferred query a closed object, finalized once and by its own deadline
A Flight query does not end where the frontend's command does: the client pulls the results from the backends by itself and the frontend hears nothing of it, so a query whose coordinator must outlive GetFlightInfo (an external-table scan in batch mode, #62259) is parked on the session as a deferred executor and finalized later -- by the next request, by teardown, or by the timeout checker (#67503). Until now that executor still read the session's live state when it was finalized, and the checker judged all deferred executors at once. Both held only as long as nothing happened on the session in between, which the session actions of this PR no longer guarantee (a SetSessionOptions request runs statements on the session without being a request of it), and which never held for the checker's thread. The contract, now in FlightProtocolAdapter's class comment: a deferred executor carries what finalizing it needs and reads nothing of the session afterwards; it is finalized exactly once, by whoever takes it out of the session's list under the list's lock, deciding and taking in one critical section. Concretely: - The timeout checker takes each deferred executor by its own deadline (its start plus max(config, the execution timeout it ran with)) in a single synchronized pass (takeExpiredDeferredExecutors) and finalizes the taken ones outside the lock. Before, it combined the earliest start with the longest timeout over three separate lock acquisitions and then closed the whole list: the statements of one multi-statement request, each with a start and a SET_VAR timeout of its own, were reaped by an aggregate that fits none of them, and a beginRequest() clearing the list between the reads turned the "-1 = nothing deferred" sentinel into "infinitely overdue", so a query deferred right after could be closed under a decision that was never about it. - Whether a statement's profile is reported is decided once, when its profile is first updated, and followed by the final update (StmtExecutor.reportsProfile). Re-reading enable_profile at the end left the RUNNING profile of a deferred query unfinished whenever a SetSessionOptions SET turned the variable off in between -- and, before this PR already, that of any Flight query run with a SET_VAR(enable_profile) hint, reverted before FlightSqlConnectProcessor.close() finalizes it. - The final summary of a deferred query adds only what ends it (end time, total time, task state), counted from its own start; what was recorded when it ran stays, since SummaryProfile merges by key. It no longer re-reads the context's catalog, database, state and variables, which the session's later statements own by then. Tests: FlightSqlDeferredQueryIdleTimeoutTest (each query by its own execution timeout and its own start; the checker interleaved with the next request finalizes each executor exactly once and leaves one deferred after its decision alone; one failing finalization does not stop the rest), StmtExecutorTest.testADeferredQueryIsFinalizedFromItsOwnRecordNotTheSessions (RUNNING published, then enable_profile off, USE, a new query id and start time on the session, finalization from a thread without the session's context: finished, and the record untouched). Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01Ja986K4PEm2LD8u44L9jd2
1 parent ec06096 commit ad7c70b

8 files changed

Lines changed: 387 additions & 117 deletions

File tree

‎fe/fe-common/src/main/java/org/apache/doris/common/Config.java‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2656,8 +2656,10 @@ public class Config extends ConfigBase {
26562656
+ "while the client pulls the results (DoGet); that coordinator is normally released when the "
26572657
+ "session runs its next query or is closed. Most Flight clients never close a session, so the "
26582658
+ "coordinator, and with it the query's workload group queue slot and its active_queries entry, "
2659-
+ "would otherwise stay held until wait_timeout. If the session stays idle for longer than this "
2660-
+ "many seconds after the query started, the coordinator is released anyway. The bound is never "
2659+
+ "would otherwise stay held until wait_timeout. Once this many seconds have passed since the query "
2660+
+ "started and the session is not running a statement, the coordinator is released anyway; each "
2661+
+ "such query is bounded on its own, and the session's other commands in the meantime (a session "
2662+
+ "option, a metadata request) neither release it earlier nor keep it longer. The bound is never "
26612663
+ "shorter than the query's own execution timeout, and the session itself is not killed "
26622664
+ "(wait_timeout still governs that). 0 disables the bound.")
26632665
public static int arrow_flight_deferred_query_idle_timeout_second = 3600;

‎fe/fe-core/src/main/java/org/apache/doris/arrowflight/protocol/FlightProtocolAdapter.java‎

Lines changed: 87 additions & 43 deletions
Original file line numberDiff line numberDiff line change
@@ -36,14 +36,17 @@
3636
import com.google.common.annotations.VisibleForTesting;
3737
import com.google.common.base.Preconditions;
3838
import com.google.common.base.Strings;
39+
import com.google.common.collect.ImmutableList;
3940
import com.google.common.collect.Lists;
4041
import org.apache.arrow.flight.CallStatus;
4142
import org.apache.logging.log4j.LogManager;
4243
import org.apache.logging.log4j.Logger;
4344

4445
import java.io.IOException;
4546
import java.util.ArrayList;
47+
import java.util.Collections;
4648
import java.util.HashMap;
49+
import java.util.Iterator;
4750
import java.util.List;
4851
import java.util.Map;
4952
import java.util.concurrent.TimeUnit;
@@ -58,6 +61,25 @@
5861
* arrive on its own thread, and nothing in the transport serializes them. {@link #runCommand}
5962
* does, so that a session's {@link ConnectContext}, which is not thread-safe, is only ever used
6063
* by one command at a time.
64+
*
65+
* <p>Nor does a query end where the frontend's command does. The client pulls the results from the
66+
* backends by itself (DoGet), and the frontend hears nothing of it -- not when it starts, not when
67+
* it is done -- so a query may still be running on the backends while the session runs its next
68+
* command here. Two things of a request therefore outlive the command that made them, and the
69+
* contract for both is that the command stream owns them and nothing else does: the result a
70+
* statement materialized on this frontend, cached on the channel until the client's DoGet takes it
71+
* or the next request drops it; and the executors of queries whose coordinator must stay alive
72+
* until the backends are done with it, the {@link #deferredExecutors}. A deferred executor is a
73+
* closed object from the moment it is deferred: it carries what finalizing it needs and reads
74+
* nothing of the session's live state afterwards, since the session moves on without it (a session
75+
* option's SET, a metadata request, the next request) and since it may be finalized from a thread
76+
* that runs no command of the session (the timeout checker, a token expiry). It is finalized
77+
* exactly once, by whoever takes it out of the list under the list's lock -- the next request
78+
* ({@link #beginRequest}), the timeout checker ({@link #takeExpiredDeferredExecutors}, each
79+
* executor by its own deadline) or teardown ({@link #closeDeferredExecutors}) -- deciding and
80+
* taking in one critical section, finalizing outside it. The frontend has no signal for the moment
81+
* a query is done on the backends; the next request stands in for it, as it has since #62259,
82+
* and the deadline bounds the wait when no request comes.
6183
*/
6284
public class FlightProtocolAdapter implements ProtocolAdapter {
6385
private static final Logger LOG = LogManager.getLogger(FlightProtocolAdapter.class);
@@ -83,8 +105,10 @@ public class FlightProtocolAdapter implements ProtocolAdapter {
83105
// GetFlightInfo would release the SplitSource too early and make the BE's fetchSplitBatch fail
84106
// with "Split source X is released". These executors are finalized when the next query starts
85107
// on this connection, when the connection is torn down, or by the idle reaper in
86-
// ConnectContext.checkTimeout once arrow_flight_deferred_query_idle_timeout_second has passed
87-
// since the query started. See #62259 and #67503.
108+
// ConnectContext.checkTimeout once each one's own bound has passed since its query started.
109+
// See #62259 and #67503, and the class comment for who owns them. Guarded by its own monitor:
110+
// the commands of the session add and take under the command lock, the timeout checker and
111+
// teardown take without it.
88112
private final List<StmtExecutor> deferredExecutors = new ArrayList<>();
89113
// Serializes the commands of this session, see runCommand.
90114
private final ReentrantLock commandLock = new ReentrantLock();
@@ -356,68 +380,88 @@ public void addDeferredExecutor(StmtExecutor executor) {
356380
}
357381
}
358382

383+
/**
384+
* Takes every deferred executor out of the list and finalizes it: the next request of the
385+
* session ({@link #beginRequest}), which stands in for the end of the previous query's DoGet, and
386+
* the session's teardown.
387+
*/
359388
public void closeDeferredExecutors() {
360-
List<StmtExecutor> toClose;
389+
List<StmtExecutor> taken;
361390
synchronized (deferredExecutors) {
362391
if (deferredExecutors.isEmpty()) {
363392
return;
364393
}
365-
toClose = new ArrayList<>(deferredExecutors);
394+
taken = new ArrayList<>(deferredExecutors);
366395
deferredExecutors.clear();
367396
}
368-
for (StmtExecutor deferredExecutor : toClose) {
369-
try {
370-
deferredExecutor.finalizeArrowFlightQuery();
371-
} catch (Throwable t) {
372-
LOG.warn("failed to finalize deferred arrow flight executor", t);
373-
}
374-
}
397+
finalizeDeferredExecutors(taken);
375398
}
376399

377400
/**
378-
* When the oldest deferred query started, in epoch milliseconds; -1 when nothing is deferred.
379-
* The bound of {@link #getDeferredExecutorsIdleTimeoutS} counts from here, not from the
380-
* session's last command: a session option or a metadata request that comes while the client
381-
* is still pulling the query's results neither says it is done nor may keep the coordinator
382-
* alive for another bound.
401+
* Takes out of the list the deferred executors whose own bound has passed by {@code now} and
402+
* returns them for the caller to finalize: the timeout checker's half of the exactly-once rule
403+
* in the class comment. Each executor is judged and removed in the same critical section, by
404+
* its own deadline -- when its query started plus {@link #deferredBoundMs} -- so that of the
405+
* executors of one multi-statement request, each with a start and an execution timeout of its
406+
* own, only the overdue ones go, and an executor added while the checker runs, which it never
407+
* judged, stays. Takes nothing when the bound is disabled
408+
* (Config.arrow_flight_deferred_query_idle_timeout_second is 0).
383409
*/
384-
public long getDeferredExecutorsStartTimeMs() {
385-
long startTimeMs = -1;
410+
public List<StmtExecutor> takeExpiredDeferredExecutors(long now) {
411+
int configTimeoutS = Config.arrow_flight_deferred_query_idle_timeout_second;
412+
if (configTimeoutS <= 0) {
413+
return Collections.emptyList();
414+
}
415+
List<StmtExecutor> expired = new ArrayList<>();
386416
synchronized (deferredExecutors) {
387-
for (StmtExecutor deferredExecutor : deferredExecutors) {
388-
long deferredStartTimeMs = deferredExecutor.getDeferredStartTimeMs();
389-
startTimeMs = startTimeMs < 0 ? deferredStartTimeMs : Math.min(startTimeMs, deferredStartTimeMs);
417+
Iterator<StmtExecutor> iterator = deferredExecutors.iterator();
418+
while (iterator.hasNext()) {
419+
StmtExecutor deferredExecutor = iterator.next();
420+
long deferredMs = now - deferredExecutor.getDeferredStartTimeMs();
421+
if (deferredMs > deferredBoundMs(deferredExecutor, configTimeoutS)) {
422+
expired.add(deferredExecutor);
423+
iterator.remove();
424+
}
390425
}
391426
}
392-
return startTimeMs;
427+
return expired;
393428
}
394429

395430
/**
396-
* How long, in seconds, a sleeping connection may keep its deferred executors before the
397-
* timeout checker finalizes them without killing the connection
398-
* (Config.arrow_flight_deferred_query_idle_timeout_second), counted from when the deferred
399-
* query started ({@link #getDeferredExecutorsStartTimeMs}). A Flight client that opens a
400-
* session per query and never closes it would otherwise pin each deferred query's query queue
401-
* slot and query registration until wait_timeout (8h by default). The bound is never shorter
402-
* than the execution timeout the deferred query was run with: the client may still be pulling
403-
* that query's results from the BE, which still needs the batch split source the coordinator
404-
* holds. Returns -1 when the bound is disabled or nothing is deferred.
431+
* How long, in milliseconds, a deferred executor may be kept after its query started before the
432+
* timeout checker finalizes it without killing the session: {@code configTimeoutS}
433+
* (Config.arrow_flight_deferred_query_idle_timeout_second) floored at the execution timeout the
434+
* query ran with -- the client may still be pulling its results from the BE, which still needs
435+
* the batch split source the coordinator holds. It counts from the query's own start, not from
436+
* the session's last command: a session option or a metadata request that came since neither
437+
* finished the query nor may keep its coordinator alive for another bound. A Flight client that
438+
* opens a session per query and never closes it would otherwise pin each deferred query's query
439+
* queue slot and query registration until wait_timeout (8h by default).
405440
*/
406-
public long getDeferredExecutorsIdleTimeoutS() {
407-
int configTimeoutS = Config.arrow_flight_deferred_query_idle_timeout_second;
408-
if (configTimeoutS <= 0) {
409-
return -1;
441+
public static long deferredBoundMs(StmtExecutor deferredExecutor, int configTimeoutS) {
442+
return Math.max(configTimeoutS, deferredExecutor.getDeferredExecTimeoutS()) * 1000L;
443+
}
444+
445+
/**
446+
* Finalizes executors taken out of the list (see {@link StmtExecutor#finalizeArrowFlightQuery}).
447+
* One failing does not keep the next from being finalized.
448+
*/
449+
public static void finalizeDeferredExecutors(List<StmtExecutor> takenExecutors) {
450+
for (StmtExecutor deferredExecutor : takenExecutors) {
451+
try {
452+
deferredExecutor.finalizeArrowFlightQuery();
453+
} catch (Throwable t) {
454+
LOG.warn("failed to finalize deferred arrow flight executor", t);
455+
}
410456
}
411-
long execTimeoutS = -1;
457+
}
458+
459+
/** A snapshot of the deferred executors, in the order they were deferred. */
460+
@VisibleForTesting
461+
public List<StmtExecutor> getDeferredExecutors() {
412462
synchronized (deferredExecutors) {
413-
if (deferredExecutors.isEmpty()) {
414-
return -1;
415-
}
416-
for (StmtExecutor deferredExecutor : deferredExecutors) {
417-
execTimeoutS = Math.max(execTimeoutS, deferredExecutor.getDeferredExecTimeoutS());
418-
}
463+
return ImmutableList.copyOf(deferredExecutors);
419464
}
420-
return Math.max(configTimeoutS, execTimeoutS);
421465
}
422466

423467
/** The body of a command run by {@link #runCommand}. */

‎fe/fe-core/src/main/java/org/apache/doris/qe/ConnectContext.java‎

Lines changed: 24 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -87,6 +87,7 @@
8787
import org.apache.doris.transaction.TransactionEntry;
8888
import org.apache.doris.transaction.TransactionStatus;
8989

90+
import com.google.common.annotations.VisibleForTesting;
9091
import com.google.common.base.Strings;
9192
import com.google.common.collect.Lists;
9293
import com.google.common.collect.Maps;
@@ -812,8 +813,7 @@ public void setStartTime() {
812813
* The client was active just now: wait_timeout counts from here. Unlike {@link #setStartTime},
813814
* which starts a statement, this leaves what the last statement recorded (its rows, its backend
814815
* selection) as it is, for a command that runs no statement of its own -- an Arrow Flight SQL
815-
* session option or metadata request -- while a deferred query of the session may still be
816-
* finished later from that record.
816+
* session option or metadata request.
817817
*/
818818
public void refreshStartTime() {
819819
startTimeInstant = Instant.now();
@@ -1049,35 +1049,34 @@ public void closeFlightSqlDeferredExecutors() {
10491049
}
10501050
}
10511051

1052-
// Returns -1 when the connection has nothing deferred or the bound is disabled.
1053-
public long getFlightSqlDeferredExecutorsIdleTimeoutS() {
1052+
// A snapshot; empty for a connection of any other protocol.
1053+
@VisibleForTesting
1054+
public List<StmtExecutor> getFlightSqlDeferredExecutors() {
10541055
return protocolAdapter instanceof FlightProtocolAdapter
1055-
? ((FlightProtocolAdapter) protocolAdapter).getDeferredExecutorsIdleTimeoutS() : -1;
1056+
? ((FlightProtocolAdapter) protocolAdapter).getDeferredExecutors() : Collections.emptyList();
10561057
}
10571058

1058-
// When the oldest deferred query started; -1 when the connection has nothing deferred.
1059-
public long getFlightSqlDeferredExecutorsStartTimeMs() {
1060-
return protocolAdapter instanceof FlightProtocolAdapter
1061-
? ((FlightProtocolAdapter) protocolAdapter).getDeferredExecutorsStartTimeMs() : -1;
1062-
}
1063-
1064-
// Called by the timeout checker for a sleeping connection that is not past wait_timeout yet.
1065-
// The bound counts from when the deferred query started, not from startTime: the session's
1066-
// later commands (a session option, a metadata request) move startTime on without finishing
1067-
// that query, and must not keep its coordinator alive either.
1059+
// Called by the timeout checker for a sleeping connection that is not past wait_timeout yet:
1060+
// finalizes each deferred query whose own bound has passed since it started (see
1061+
// FlightProtocolAdapter.takeExpiredDeferredExecutors), not the ones whose bound has not, and
1062+
// not the connection. The bound counts from the query's start, not from startTime: the
1063+
// session's later commands (a session option, a metadata request) move startTime on without
1064+
// finishing that query, and must not keep its coordinator alive either.
10681065
private void reapIdleFlightSqlDeferredExecutors(long now) {
1069-
long timeoutS = getFlightSqlDeferredExecutorsIdleTimeoutS();
1070-
if (timeoutS < 0) {
1071-
return;
1072-
}
1073-
long deferredMs = now - getFlightSqlDeferredExecutorsStartTimeMs();
1074-
if (deferredMs <= timeoutS * 1000L) {
1066+
if (!(protocolAdapter instanceof FlightProtocolAdapter)) {
10751067
return;
10761068
}
1077-
LOG.warn("release deferred arrow flight query of idle connection, connectionId: {}, remote: {}, "
1078-
+ "deferred for: {}ms, idle timeout: {}s",
1079-
connectionId, getRemoteHostPortString(), deferredMs, timeoutS);
1080-
closeFlightSqlDeferredExecutors();
1069+
List<StmtExecutor> expired = ((FlightProtocolAdapter) protocolAdapter).takeExpiredDeferredExecutors(now);
1070+
for (StmtExecutor deferredExecutor : expired) {
1071+
LOG.warn("release deferred arrow flight query {} of idle connection, connectionId: {}, remote: {}, "
1072+
+ "deferred for: {}ms, bound: {}ms",
1073+
deferredExecutor.getDeferredQueryId() == null
1074+
? "unknown" : DebugUtil.printId(deferredExecutor.getDeferredQueryId()),
1075+
connectionId, getRemoteHostPortString(), now - deferredExecutor.getDeferredStartTimeMs(),
1076+
FlightProtocolAdapter.deferredBoundMs(deferredExecutor,
1077+
Config.arrow_flight_deferred_query_idle_timeout_second));
1078+
}
1079+
FlightProtocolAdapter.finalizeDeferredExecutors(expired);
10811080
}
10821081

10831082
/**

0 commit comments

Comments
 (0)