You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 1127003
Browse filesBrowse the repository at this point in the historyBrowse files
[fix](arrow-flight) Keep coordinator alive across GetFlightInfo/DoGet for external table scan (#64799)
### What problem does this PR solve?
Issue Number: close#62259
Related PR: #64797
Problem Summary:
Arrow Flight SQL queries against Iceberg (and other external) tables in
batch split mode crashed the BE / failed with `Split source X is
released`.
Arrow Flight executes a query in two phases: `GetFlightInfo` (plan +
submit to BE) and `DoGet` (the client pulls results from the BE). For an
external table scan in batch split mode, the BE keeps scanning during
`DoGet` and lazily fetches file splits from the FE via the
`fetchSplitBatch` RPC, using an async `SplitSource` that the FE
coordinator holds (through its scan nodes).
The FE closed the coordinator at the end of `GetFlightInfo`
(`StmtExecutor.executeAndSendResult`'s `finally` → `Coordinator.close()`
→ `ScanNode.stop()` → `SplitSourceManager.removeSplitSource()`) and also
unregistered it (`FlightSqlConnectProcessor.close()` →
`StmtExecutor.finalizeQuery()`). So by the time the BE called
`fetchSplitBatch` during `DoGet`, the `SplitSource` was already gone.
The MySQL protocol is unaffected because plan + execute share one
request, so the coordinator stays alive until all results are consumed.
This PR keeps the coordinator (and its `SplitSource`) alive across the
two phases and cleans it up reliably:
- **StmtExecutor**: for an Arrow Flight query that produces results on
the BE (`coordBase == coord`), mark it deferred, register the executor
on the `ConnectContext`, and skip the eager `Coordinator.close()` in the
`finally`. A failed query (whose `exec()` threw) is not deferred and is
closed as before.
- **ConnectContext**: hold the deferred executors and add
`closeFlightSqlDeferredExecutors()`, which closes their coordinators
(releasing the `SplitSource` and the query queue slot) and unregisters
the queries.
- **FlightSqlConnectProcessor.close()**: do not finalize deferred
executors.
- **DorisFlightSqlProducer**: finalize the previous query's deferred
coordinator when the next query starts on the connection.
- **FlightSqlConnectPoolMgr.unregisterConnection()**: finalize deferred
coordinators when the connection is torn down. All teardown paths
(idle/query timeout, bearer token expiry, explicit `CloseSession`) reach
here, so an abandoned connection cannot leak the coordinator.
Non-Arrow-Flight paths (MySQL, internal tables, point queries) are
unchanged: `deferredForArrowFlight` can only become true for
`ARROW_FLIGHT_SQL`.
The BE-side error-path hardening (so any `fetchSplitBatch` failure fails
gracefully instead of crashing the BE) is handled separately in #64797.
0 commit comments