You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 23b5ee5
Browse filesBrowse the repository at this point in the historyBrowse files
authored
[fix](arrow-flight) Release a finished Flight query's coordinator instead of holding it until wait_timeout (#67504)
### What problem does this PR solve?
Issue Number: close#67503
Related PR: #64799 (kept the coordinator alive across GetFlightInfo →
DoGet so the BE can fetch external-table splits; this PR narrows that to
the queries that actually need it and bounds how long an abandoned
session can hold the rest)
Problem Summary:
Since #64799 every Arrow Flight SQL query whose results are produced on
the BE kept its coordinator alive after `GetFlightInfo`, until the
session ran its next query or was closed. The coordinator holds the
query's workload-group queue slot and its `active_queries` registration.
Most Flight clients open a session per query and never send
`CloseSession` (the ADBC/JDBC drivers don't do it on connection close,
as the comment in `DorisFlightSqlProducer.closeSession` notes), so each
abandoned session pinned one slot for up to `wait_timeout` (8h by
default).
Observed on a 2-FE cluster with a `max_concurrency=8` workload group:
eight finished Flight queries sat in `information_schema.active_queries`
as RUNNING for over an hour, and every later query in the group failed
with `query queue timeout, timeout: 60000 ms`. Full write-up in #67503.
This PR fixes it in two steps:
1. **Only defer the coordinator when the BE still needs it.** The
deferral added by #64799 is required only for an external-table scan in
batch mode, where the BE lazily fetches splits during `DoGet` from the
split source the coordinator holds. Every other query (internal tables,
external tables in non-batch mode) now closes its coordinator at the end
of `GetFlightInfo` again, releasing the queue slot and the
`active_queries` entry right away. That is safe: the BE buffers its
results independently of the coordinator, and finalizing the FE side
does not cancel BE execution (`QeProcessorImpl.reportExecStatus` accepts
reports for an unregistered query, and `Coordinator.close()` only
releases the queue token and the split sources). New predicates
`ScanNode.hasBatchSplitSource()` / `Coordinator.hasBatchSplitSource()`
drive the gate in `StmtExecutor.executeAndSendResult`.
2. **Bound the remaining deferred queries without killing the session.**
New mutable FE config `arrow_flight_deferred_query_idle_timeout_second`
(default 3600). Once a Flight session has been sleeping for longer than
this since its last query started, the connection timeout checker
finalizes the session's deferred executors (releasing the slot and the
registration) and leaves the session alive; `wait_timeout` still governs
the session itself. The bound is floored at the execution timeout the
deferred query actually ran with, captured when the executor is deferred
(`SET_VAR` hint values are reverted at the end of `execute()`, so the
session value cannot be read later). `0` disables the bound.
Why not kill the session (the first revision of this PR): a killed
Flight session's bearer token stays in the token cache marked as already
used, so the client's next call on it fails with `UserSession expire
after access` and has to re-handshake. With a 1h bound, every pooled or
BI-tool Flight connection that idles for an hour would fail once on its
next use. Reaping only the deferred query releases the leaked resources
with no client-visible change.
### Release note
Arrow Flight SQL: a query no longer holds its workload-group queue slot
and `active_queries` entry after `GetFlightInfo` unless it is an
external-table scan in batch mode, the only case where the BE still
fetches splits from the FE during `DoGet`. For that case a new FE config
`arrow_flight_deferred_query_idle_timeout_second` (default 1h) releases
the coordinator of an idle, never-closed session without killing the
session.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
(`arrow_flight_sql_p0/test_arrow_flight_query_release`: a finished
Flight query on an internal table no longer occupies a
`max_concurrency=1` group and is gone from `active_queries`;
`external_table_p0/iceberg/test_iceberg_arrow_flight_split_source`: the
batch-mode scan stays registered after `DoGet` and is released by the
idle reaper while the session survives)
- [x] Unit Test (`FlightSqlDeferredQueryIdleTimeoutTest`: the reaper
through `checkTimeout`, the exec-timeout floor, `0` disables, nothing
deferred, MySQL untouched; `ArrowFlightDeferralGateTest`: the
batch-split-source predicates;
`StmtExecutorTest.testDeferForArrowFlightFreezesExecTimeoutInEffect`)
- [x] Manual test (the first revision was verified on a test cluster
with an 8s bound, see the PR history; the current revision is covered by
the regression tests above)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [x] Yes. (1) A Flight query that is not an external-table batch-mode
scan releases its coordinator, queue slot and `active_queries` entry at
the end of `GetFlightInfo`, as it did before #64799. (2) The deferred
coordinator of a batch-mode scan on an idle session is released after 1h
by default instead of at `wait_timeout`; set
`arrow_flight_deferred_query_idle_timeout_second=0` for the previous
behavior. Sessions are never killed by this change.
- Does this need documentation?
- [ ] No.
- [x] Yes. New FE config
`arrow_flight_deferred_query_idle_timeout_second` — doris-website PR to
follow once this is reviewed.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- #64799 is in 4.0.8 / 4.1.4 -->
---------
Signed-off-by: Raghvendra Singh <[email protected]>
Co-authored-by: Raghav <[email protected]>
Co-authored-by: Claude Fable 5 <[email protected]>
Co-authored-by: morningman <[email protected]>
0 commit comments