-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Description
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
I want to improve the FlightSQL client contributed by @avantgardnerio in #3207
Specifically FlightSQL is in terms of RecordBatches but at the moment to get record batches out out of a stream of FlightDatas is non trivial (especially as the schema and dictionaries are sent individually)
I think this is a common need as I found similar code in IOx for our own flight API as was in the FlightSQL client from #3207
@xudong963 also mentions there is similar code in DataBend: https://github.com/datafuselabs/databend/tree/main/src/query/service/src/api/rpc (uses the crate https://github.com/DataEngineeringLabs/arrow-format)
Describe the solution you'd like
I would like a mid-level flight client that handles these details - the FlightSQL client can then be built on top of this mid level client, and I think the mid level client is useful on its own (for writing clients for other Arrow Flight based APIs).
Describe alternatives you've considered
Additional context
We have a version of this code in https://github.com/influxdata/influxdb_iox/pull/6427