Repository navigation
Commit 46e1c67
authored
[refactor](arrow-flight) One connection pool for MySQL connections and Arrow Flight SQL sessions (#68101)
### What problem does this PR solve?
Issue Number: #67577
Related PR: #67966 (the Flight session teardown contract this keeps,
D-13)
Problem Summary:
Stage 3, item 1 of #67577 (connection governance): one connection pool
for both protocols.
Arrow Flight SQL sessions lived in a pool of their own,
`FlightSqlConnectPoolMgr`, held to `arrow_flight_max_connections` alone:
not to `qe_max_connection`, not to the user's `max_user_connections`,
refused with a sentence of their own ("the number of arrow flight bearer
tokens should be equal to arrow flight sql max connections"), and summed
into every scheduler-wide answer — the processlist, `KILL`, the timeout
checker, the connection metrics — by `ConnectScheduler` walking two
pools.
There is one pool now:
- `ConnectPoolMgr` registers a connection of either protocol against
`qe_max_connection` and the user's `max_user_connections`, and a Flight
session additionally against its sub-quota. It indexes Flight sessions
by their peer identity (the bearer token), since that is how Flight
requests name their session. `FlightSqlConnectPoolMgr` is deleted;
`ConnectScheduler` delegates to the one pool.
- Admission is one atomic transition:
`ConnectPoolMgr.registerConnection` checks and reserves the pool, Flight
sub-quota and per-user quotas under a single lock (`admissionLock`), so
a doomed attempt cannot hold one counter's reservation across another
attempt's checks. Session creation is serialized too
(`sessionCreationLock` in `FlightSessionsWithTokenManager`): exactly one
`ConnectContext` is published per bearer token, and after registering it
the token is re-validated under the same lock and the session
unregistered if the token was invalidated meanwhile, so no session
lingers in the pool without its token.
- The bearer token cache size floors only the effective sub-quota at 1
(a legal sub-quota of 0 at `qe_max_connection = 1` still needs one token
so the first request reaches the pool and is refused with
`RESOURCE_EXHAUSTED`); an illegal `arrow_flight_token_cache_size` (<= 0)
is left as-is, keeping the existing startup failure.
- `unregisterConnection` is where every teardown path of a connection
meets — a MySQL channel closing, a Flight token expiring or being
evicted, `CloseSession`, `KILL`, the timeout checker — so it is where
the protocol releases what it still holds for the session, through the
new `ProtocolAdapter.releaseSession`: for Flight the channel-cached
results and the deferred query coordinators
(`FlightProtocolAdapter.tearDown`, unchanged from #67966), nothing for
MySQL. The transitional `ProtocolAdapter.connectPool` goes.
- A Flight session refused for a limit is refused in the words a MySQL
client is refused in (`Reach limit of connections. Total: %d, User: %d,
Current: %d`, naming the Flight sub-quota only when it is the tighter
one), as the `RESOURCE_EXHAUSTED` status of the request that would have
opened it, and its bearer token is invalidated with the refusal.
- `arrow_flight_max_connections` becomes the sub-quota of Flight
sessions within the pool: `-1`, the new default, is half of
`qe_max_connection`; an explicit value never exceeds it.
`connection_max` reports the pool's limit,
`arrow_flight_connection_total` / `arrow_flight_connection_max` the
Flight share of it. The token cache is sized from the effective
sub-quota and is otherwise untouched: the token's lifecycle is the next
PR's.1 parent bd52103 commit 46e1c67
23 files changed
Lines changed: 1171 additions & 353 deletions
File tree
- fe
- fe-common/src/main/java/org/apache/doris/common
- fe-core/src
- main/java/org/apache/doris
- arrowflight
- protocol
- sessions
- tokens
- metric
- mysql
- protocol
- qe
- protocol
- test/java/org/apache/doris
- arrowflight
- protocol
- sessions
- metric
- mysql
- protocol
- qe
- regression-test/suites/arrow_flight_sql_p0
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
758 | 758 | | |
759 | 759 | | |
760 | 760 | | |
761 | | - | |
| 761 | + | |
| 762 | + | |
| 763 | + | |
762 | 764 | | |
763 | 765 | | |
764 | 766 | | |
| |||
2650 | 2652 | | |
2651 | 2653 | | |
2652 | 2654 | | |
2653 | | - | |
2654 | | - | |
| 2655 | + | |
| 2656 | + | |
| 2657 | + | |
| 2658 | + | |
| 2659 | + | |
| 2660 | + | |
| 2661 | + | |
| 2662 | + | |
| 2663 | + | |
| 2664 | + | |
| 2665 | + | |
| 2666 | + | |
| 2667 | + | |
| 2668 | + | |
| 2669 | + | |
| 2670 | + | |
| 2671 | + | |
2655 | 2672 | | |
2656 | 2673 | | |
2657 | 2674 | | |
| |||
2679 | 2696 | | |
2680 | 2697 | | |
2681 | 2698 | | |
2682 | | - | |
2683 | | - | |
2684 | | - | |
2685 | | - | |
| 2699 | + | |
| 2700 | + | |
| 2701 | + | |
| 2702 | + | |
| 2703 | + | |
| 2704 | + | |
| 2705 | + | |
2686 | 2706 | | |
2687 | 2707 | | |
2688 | 2708 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
| 27 | + | |
27 | 28 | | |
28 | 29 | | |
| 30 | + | |
29 | 31 | | |
30 | 32 | | |
31 | 33 | | |
| |||
48 | 50 | | |
49 | 51 | | |
50 | 52 | | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
51 | 68 | | |
52 | 69 | | |
53 | 70 | | |
54 | 71 | | |
55 | | - | |
56 | | - | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
57 | 87 | | |
58 | 88 | | |
59 | 89 | | |
| |||
63 | 93 | | |
64 | 94 | | |
65 | 95 | | |
66 | | - | |
67 | | - | |
68 | | - | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
69 | 101 | | |
70 | 102 | | |
71 | 103 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
| 27 | + | |
27 | 28 | | |
28 | 29 | | |
29 | | - | |
30 | | - | |
31 | 30 | | |
32 | 31 | | |
33 | 32 | | |
| |||
266 | 265 | | |
267 | 266 | | |
268 | 267 | | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
269 | 276 | | |
270 | | - | |
271 | | - | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
272 | 289 | | |
273 | 290 | | |
274 | 291 | | |
| |||
345 | 362 | | |
346 | 363 | | |
347 | 364 | | |
348 | | - | |
| 365 | + | |
349 | 366 | | |
350 | 367 | | |
351 | 368 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
| 26 | + | |
26 | 27 | | |
27 | 28 | | |
28 | 29 | | |
29 | 30 | | |
| 31 | + | |
30 | 32 | | |
31 | 33 | | |
32 | 34 | | |
33 | 35 | | |
34 | 36 | | |
35 | 37 | | |
36 | 38 | | |
| 39 | + | |
| 40 | + | |
37 | 41 | | |
38 | 42 | | |
39 | 43 | | |
| |||
42 | 46 | | |
43 | 47 | | |
44 | 48 | | |
45 | | - | |
46 | | - | |
47 | | - | |
48 | | - | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
49 | 52 | | |
50 | 53 | | |
51 | | - | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
52 | 70 | | |
53 | 71 | | |
54 | 72 | | |
| |||
71 | 89 | | |
72 | 90 | | |
73 | 91 | | |
74 | | - | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
75 | 98 | | |
76 | | - | |
77 | | - | |
78 | | - | |
79 | | - | |
80 | | - | |
81 | | - | |
82 | | - | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
83 | 120 | | |
84 | 121 | | |
85 | 122 | | |
| |||
0 commit comments