Skip to content

Mid-level ArrowFlight Client #3371

@alamb

Description

@alamb

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
I want to improve the FlightSQL client contributed by @avantgardnerio in #3207

Specifically FlightSQL is in terms of RecordBatches but at the moment to get record batches out out of a stream of FlightDatas is non trivial (especially as the schema and dictionaries are sent individually)

I think this is a common need as I found similar code in IOx for our own flight API as was in the FlightSQL client from #3207

@xudong963 also mentions there is similar code in DataBend: https://github.com/datafuselabs/databend/tree/main/src/query/service/src/api/rpc (uses the crate https://github.com/DataEngineeringLabs/arrow-format)

Describe the solution you'd like
I would like a mid-level flight client that handles these details - the FlightSQL client can then be built on top of this mid level client, and I think the mid level client is useful on its own (for writing clients for other Arrow Flight based APIs).

Describe alternatives you've considered

Additional context
We have a version of this code in https://github.com/influxdata/influxdb_iox/pull/6427

Metadata

Metadata

Assignees

Labels

arrow-flightChanges to the arrow-flight crateenhancementAny new improvement worthy of a entry in the changelog

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions