-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Description
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Arrow Flight SQL has just been officially announced https://arrow.apache.org/blog/2022/02/16/introducing-arrow-flight-sql/
It would be very cool if we had a Rust implementation of this protocol to make it easier to build systems in Rust that used this efficient new protocol and would allow better ecosystem integration.
Self servingly, we would likely use it in influxdb_iox and contribute to such an implementation, but it isn't high on our priority list quite yet.
Implementation steps:
Describe the solution you'd like
I propose creating an ArrowFlightSQLService type structure that handles the protocol details, and present an idiomatic rust trait for users.
Something like:
pub trait SqlService {
/// get list of database schemas as an array of Strings
fn schemas(&self, catalog_name: Option<&str>, db_schema_filter_pattern: Option<&str>) -> Result<ArrayRef>;
/// Execute the query and return an async stream of record batches
fn query(&self, sql: &str) -> Result<SendableRecordBatchStream>;
...
}
/// Implementation of Arrow Flight SQL for a `SqlService`
struct ArrowFlightSQLService<S: SqlService> {
inner: S
}
/// implement service from https://github.com/apache/arrow/blob/release-7.0.0/format/Flight.proto
/// That handles messages in https://github.com/apache/arrow/blob/release-7.0.0/format/FlightSql.proto
impl protobuf::generated::FlightService for ArrowFlightSQLService {
... code to translate to/from protobuf messages
}Describe alternatives you've considered
One alternative is to make this part of the official arrow-rs release (perhaps with a sql feature in the arrow-flight crate)?
Another alternative is to make this a separate project (e.g. in https://github.com/apache/arrow-datafusion or in https://github.com/datafusion-contrib )
Additional context
DataFusion