Repository navigation
Commit ad7c70b
[feature](arrow-flight) Make a deferred query a closed object, finalized once and by its own deadline
A Flight query does not end where the frontend's command does: the
client pulls the results from the backends by itself and the frontend
hears nothing of it, so a query whose coordinator must outlive
GetFlightInfo (an external-table scan in batch mode, #62259) is parked on
the session as a deferred executor and finalized later -- by the next
request, by teardown, or by the timeout checker (#67503). Until now that
executor still read the session's live state when it was finalized, and
the checker judged all deferred executors at once. Both held only as long
as nothing happened on the session in between, which the session actions
of this PR no longer guarantee (a SetSessionOptions request runs
statements on the session without being a request of it), and which
never held for the checker's thread.
The contract, now in FlightProtocolAdapter's class comment: a deferred
executor carries what finalizing it needs and reads nothing of the session
afterwards; it is finalized exactly once, by whoever takes it out of the
session's list under the list's lock, deciding and taking in one critical
section. Concretely:
- The timeout checker takes each deferred executor by its own deadline
(its start plus max(config, the execution timeout it ran with)) in a
single synchronized pass (takeExpiredDeferredExecutors) and finalizes
the taken ones outside the lock. Before, it combined the earliest start
with the longest timeout over three separate lock acquisitions and then
closed the whole list: the statements of one multi-statement request,
each with a start and a SET_VAR timeout of its own, were reaped by an
aggregate that fits none of them, and a beginRequest() clearing the list
between the reads turned the "-1 = nothing deferred" sentinel into
"infinitely overdue", so a query deferred right after could be closed
under a decision that was never about it.
- Whether a statement's profile is reported is decided once, when its
profile is first updated, and followed by the final update
(StmtExecutor.reportsProfile). Re-reading enable_profile at the end left
the RUNNING profile of a deferred query unfinished whenever a
SetSessionOptions SET turned the variable off in between -- and, before
this PR already, that of any Flight query run with a SET_VAR(enable_profile)
hint, reverted before FlightSqlConnectProcessor.close() finalizes it.
- The final summary of a deferred query adds only what ends it (end time,
total time, task state), counted from its own start; what was recorded
when it ran stays, since SummaryProfile merges by key. It no longer
re-reads the context's catalog, database, state and variables, which the
session's later statements own by then.
Tests: FlightSqlDeferredQueryIdleTimeoutTest (each query by its own
execution timeout and its own start; the checker interleaved with the
next request finalizes each executor exactly once and leaves one deferred
after its decision alone; one failing finalization does not stop the
rest), StmtExecutorTest.testADeferredQueryIsFinalizedFromItsOwnRecordNotTheSessions
(RUNNING published, then enable_profile off, USE, a new query id and start
time on the session, finalization from a thread without the session's
context: finished, and the record untouched).
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01Ja986K4PEm2LD8u44L9jd21 parent ec06096 commit ad7c70b
8 files changed
Lines changed: 387 additions & 117 deletions
File tree
- fe
- fe-common/src/main/java/org/apache/doris/common
- fe-core/src
- main/java/org/apache/doris
- arrowflight/protocol
- qe
- test/java/org/apache/doris
- arrowflight
- protocol
- sessions
- mysql/protocol
- qe
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2656 | 2656 | | |
2657 | 2657 | | |
2658 | 2658 | | |
2659 | | - | |
2660 | | - | |
| 2659 | + | |
| 2660 | + | |
| 2661 | + | |
| 2662 | + | |
2661 | 2663 | | |
2662 | 2664 | | |
2663 | 2665 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
36 | 36 | | |
37 | 37 | | |
38 | 38 | | |
| 39 | + | |
39 | 40 | | |
40 | 41 | | |
41 | 42 | | |
42 | 43 | | |
43 | 44 | | |
44 | 45 | | |
45 | 46 | | |
| 47 | + | |
46 | 48 | | |
| 49 | + | |
47 | 50 | | |
48 | 51 | | |
49 | 52 | | |
| |||
58 | 61 | | |
59 | 62 | | |
60 | 63 | | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
61 | 83 | | |
62 | 84 | | |
63 | 85 | | |
| |||
83 | 105 | | |
84 | 106 | | |
85 | 107 | | |
86 | | - | |
87 | | - | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
88 | 112 | | |
89 | 113 | | |
90 | 114 | | |
| |||
356 | 380 | | |
357 | 381 | | |
358 | 382 | | |
| 383 | + | |
| 384 | + | |
| 385 | + | |
| 386 | + | |
| 387 | + | |
359 | 388 | | |
360 | | - | |
| 389 | + | |
361 | 390 | | |
362 | 391 | | |
363 | 392 | | |
364 | 393 | | |
365 | | - | |
| 394 | + | |
366 | 395 | | |
367 | 396 | | |
368 | | - | |
369 | | - | |
370 | | - | |
371 | | - | |
372 | | - | |
373 | | - | |
374 | | - | |
| 397 | + | |
375 | 398 | | |
376 | 399 | | |
377 | 400 | | |
378 | | - | |
379 | | - | |
380 | | - | |
381 | | - | |
382 | | - | |
| 401 | + | |
| 402 | + | |
| 403 | + | |
| 404 | + | |
| 405 | + | |
| 406 | + | |
| 407 | + | |
| 408 | + | |
383 | 409 | | |
384 | | - | |
385 | | - | |
| 410 | + | |
| 411 | + | |
| 412 | + | |
| 413 | + | |
| 414 | + | |
| 415 | + | |
386 | 416 | | |
387 | | - | |
388 | | - | |
389 | | - | |
| 417 | + | |
| 418 | + | |
| 419 | + | |
| 420 | + | |
| 421 | + | |
| 422 | + | |
| 423 | + | |
| 424 | + | |
390 | 425 | | |
391 | 426 | | |
392 | | - | |
| 427 | + | |
393 | 428 | | |
394 | 429 | | |
395 | 430 | | |
396 | | - | |
397 | | - | |
398 | | - | |
399 | | - | |
400 | | - | |
401 | | - | |
402 | | - | |
403 | | - | |
404 | | - | |
| 431 | + | |
| 432 | + | |
| 433 | + | |
| 434 | + | |
| 435 | + | |
| 436 | + | |
| 437 | + | |
| 438 | + | |
| 439 | + | |
405 | 440 | | |
406 | | - | |
407 | | - | |
408 | | - | |
409 | | - | |
| 441 | + | |
| 442 | + | |
| 443 | + | |
| 444 | + | |
| 445 | + | |
| 446 | + | |
| 447 | + | |
| 448 | + | |
| 449 | + | |
| 450 | + | |
| 451 | + | |
| 452 | + | |
| 453 | + | |
| 454 | + | |
| 455 | + | |
410 | 456 | | |
411 | | - | |
| 457 | + | |
| 458 | + | |
| 459 | + | |
| 460 | + | |
| 461 | + | |
412 | 462 | | |
413 | | - | |
414 | | - | |
415 | | - | |
416 | | - | |
417 | | - | |
418 | | - | |
| 463 | + | |
419 | 464 | | |
420 | | - | |
421 | 465 | | |
422 | 466 | | |
423 | 467 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
87 | 87 | | |
88 | 88 | | |
89 | 89 | | |
| 90 | + | |
90 | 91 | | |
91 | 92 | | |
92 | 93 | | |
| |||
812 | 813 | | |
813 | 814 | | |
814 | 815 | | |
815 | | - | |
816 | | - | |
| 816 | + | |
817 | 817 | | |
818 | 818 | | |
819 | 819 | | |
| |||
1049 | 1049 | | |
1050 | 1050 | | |
1051 | 1051 | | |
1052 | | - | |
1053 | | - | |
| 1052 | + | |
| 1053 | + | |
| 1054 | + | |
1054 | 1055 | | |
1055 | | - | |
| 1056 | + | |
1056 | 1057 | | |
1057 | 1058 | | |
1058 | | - | |
1059 | | - | |
1060 | | - | |
1061 | | - | |
1062 | | - | |
1063 | | - | |
1064 | | - | |
1065 | | - | |
1066 | | - | |
1067 | | - | |
| 1059 | + | |
| 1060 | + | |
| 1061 | + | |
| 1062 | + | |
| 1063 | + | |
| 1064 | + | |
1068 | 1065 | | |
1069 | | - | |
1070 | | - | |
1071 | | - | |
1072 | | - | |
1073 | | - | |
1074 | | - | |
| 1066 | + | |
1075 | 1067 | | |
1076 | 1068 | | |
1077 | | - | |
1078 | | - | |
1079 | | - | |
1080 | | - | |
| 1069 | + | |
| 1070 | + | |
| 1071 | + | |
| 1072 | + | |
| 1073 | + | |
| 1074 | + | |
| 1075 | + | |
| 1076 | + | |
| 1077 | + | |
| 1078 | + | |
| 1079 | + | |
1081 | 1080 | | |
1082 | 1081 | | |
1083 | 1082 | | |
| |||
0 commit comments