-
Notifications
You must be signed in to change notification settings - Fork 135
Description
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
While this FR is similar to apache/arrow-rs#1803, this is not just specific to parquet or any format.
There is a lack of reader-like API for object_store.
Describe the solution you'd like
If you look at put_multipart API, it returns a Box<dyn AsyncWrite + Unpin + Send>. I would suggest exposing a new API get_reader or get_aysnc_reader which returns something like Box<dyn AsyncRead + AsyncSeek + Send + Unpin>.
The signature(s) can look like,
async fn get_reader(
&self,
location: &Path,
) -> Result<Box<dyn AsyncRead + AsyncSeek + Send + Unpin>>;
async fn get_reader(
&self,
location: &ObjectMeta,
) -> Result<Box<dyn AsyncRead + AsyncSeek + Send + Unpin>>;
For end users, they can use it like,
let location: Path = "..."; // can be overloaded for ObjectMeta as well
let reader = store.get_reader(&location);
Describe alternatives you've considered
The alternative will require the users to implement such a reader by themselves.
Additional context
This can also help object_store work in a more versatile/ general manner. Example interop with the async parts of arrow2 like here