Skip to content

[object_store] Run requests on a different tokio runtime #13

Closed

Description

@alamb

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
There are cases where one would like to use a different Tokio threadpool for making object store requests. For example, see this ticket in DataFusion

Note it is now possible to use object_store without reqwest (and therefore potentially without tokio), however I think tokio is still important enough to warrant this feature

Describe the solution you'd like
I would like some way, with an example, of making object store requests from an async function running on one tokio runtime (Handle) and execute on another

Describe alternatives you've considered
@tustvold has a proposal to do this in apache/arrow-rs#7253, so this is the corresponding ticket explaining the rationale

Additional context

Activity

  1. tustvold commented on Mar 18, 2025

    @tustvold
    Contributor

    and therefore potentially without tokio

    This isn't actually true - filed #1.

  2. ndemir commented on Mar 18, 2025

    @ndemir

    Just to be clear, this request will extend the current interface. Right?

    Example Usage:

    use object_store::{aws::AmazonS3Builder};
    use tokio::runtime::Handle;
    
    // Create a dedicated IO runtime 
    let io_runtime = tokio::runtime::Builder::new_multi_thread()
        .worker_threads(4)
        .thread_name("object-store-io")
        .build()
        .unwrap();
    
    // Create store with specific runtime handle
    // or, something like that
    let store = AmazonS3Builder::from_env()
        .with_runtime_handle(io_runtime.handle().clone())
        .build()?;
  3. alamb commented on Mar 19, 2025

    @alamb
    ContributorAuthor

    I am not 100% sure @ndemir -- one of the big things I want to do before merging apache/arrow-rs#7253 is write up some more documentation on how it would be used (and an example as you show is important)

  4. ndemir commented on Mar 19, 2025

    @ndemir

    @alamb

    I initially thought this was a new issue, but I realized that PR apache/arrow-rs#7253 already addresses and closes it.

    I had asked the previous question to clarify whether this issue was expecting a new change.

    Long story short, my comment can be ignored since I mistakenly assumed this was a new issue without an existing PR.

  5. alamb commented on Mar 20, 2025

    @alamb
    ContributorAuthor

    Migrating from arrow-rs issue #7304

  6. transferred this issue fromapache/arrow-rson Mar 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions