Skip to content

object-store: Expose an async reader API for object store #142

@chitralverma

Description

@chitralverma

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
While this FR is similar to apache/arrow-rs#1803, this is not just specific to parquet or any format.

There is a lack of reader-like API for object_store.

Describe the solution you'd like
If you look at put_multipart API, it returns a Box<dyn AsyncWrite + Unpin + Send>. I would suggest exposing a new API get_reader or get_aysnc_reader which returns something like Box<dyn AsyncRead + AsyncSeek + Send + Unpin>.

The signature(s) can look like,

async fn get_reader(
        &self,
        location: &Path,
    ) -> Result<Box<dyn AsyncRead + AsyncSeek + Send + Unpin>>;

async fn get_reader(
        &self,
        location: &ObjectMeta,
    ) -> Result<Box<dyn AsyncRead + AsyncSeek + Send + Unpin>>;

For end users, they can use it like,

let location: Path = "..."; // can be overloaded for ObjectMeta as well
let reader = store.get_reader(&location);

Describe alternatives you've considered
The alternative will require the users to implement such a reader by themselves.

Additional context
This can also help object_store work in a more versatile/ general manner. Example interop with the async parts of arrow2 like here

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions