Skip to content

Consider support / implementation of Arrow Flight SQL #1323

@alamb

Description

@alamb

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Arrow Flight SQL has just been officially announced https://arrow.apache.org/blog/2022/02/16/introducing-arrow-flight-sql/

It would be very cool if we had a Rust implementation of this protocol to make it easier to build systems in Rust that used this efficient new protocol and would allow better ecosystem integration.

Self servingly, we would likely use it in influxdb_iox and contribute to such an implementation, but it isn't high on our priority list quite yet.

Implementation steps:

Describe the solution you'd like
I propose creating an ArrowFlightSQLService type structure that handles the protocol details, and present an idiomatic rust trait for users.

Something like:

pub trait SqlService {
  /// get  list of database schemas as an array of Strings
  fn schemas(&self, catalog_name: Option<&str>, db_schema_filter_pattern: Option<&str>) -> Result<ArrayRef>;

  /// Execute the query and return an async stream of record batches
  fn query(&self, sql: &str) -> Result<SendableRecordBatchStream>;
 ... 
}

/// Implementation of Arrow Flight SQL for a `SqlService`
struct ArrowFlightSQLService<S: SqlService>  {
  inner: S
}

/// implement service from https://github.com/apache/arrow/blob/release-7.0.0/format/Flight.proto
/// That handles messages in https://github.com/apache/arrow/blob/release-7.0.0/format/FlightSql.proto
impl protobuf::generated::FlightService for ArrowFlightSQLService {

... code to translate to/from protobuf messages

}

Describe alternatives you've considered
One alternative is to make this part of the official arrow-rs release (perhaps with a sql feature in the arrow-flight crate)?

Another alternative is to make this a separate project (e.g. in https://github.com/apache/arrow-datafusion or in https://github.com/datafusion-contrib )

Additional context
DataFusion

Metadata

Metadata

Assignees

No one assigned

    Labels

    arrow-flightChanges to the arrow-flight crateenhancementAny new improvement worthy of a entry in the changeloghelp wanted

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions