You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[object_store] Run requests on a different tokio runtime #13
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
There are cases where one would like to use a different Tokio threadpool for making object store requests. For example, see this ticket in DataFusion
Note it is now possible to use object_store without reqwest(and therefore potentially without tokio), however I think tokio is still important enough to warrant this feature
Describe the solution you'd like
I would like some way, with an example, of making object store requests from an async function running on one tokio runtime (Handle) and execute on another
Describe alternatives you've considered @tustvold has a proposal to do this in apache/arrow-rs#7253, so this is the corresponding ticket explaining the rationale
Just to be clear, this request will extend the current interface. Right?
Example Usage:
use object_store::{aws::AmazonS3Builder};use tokio::runtime::Handle;// Create a dedicated IO runtime let io_runtime = tokio::runtime::Builder::new_multi_thread().worker_threads(4).thread_name("object-store-io").build().unwrap();// Create store with specific runtime handle// or, something like thatlet store = AmazonS3Builder::from_env().with_runtime_handle(io_runtime.handle().clone()).build()?;
I am not 100% sure @ndemir -- one of the big things I want to do before merging apache/arrow-rs#7253 is write up some more documentation on how it would be used (and an example as you show is important)
Activity
tustvold commentedon Mar 18, 2025
This isn't actually true - filed #1.
ndemir commentedon Mar 18, 2025
Just to be clear, this request will extend the current interface. Right?
Example Usage:
alamb commentedon Mar 19, 2025
I am not 100% sure @ndemir -- one of the big things I want to do before merging apache/arrow-rs#7253 is write up some more documentation on how it would be used (and an example as you show is important)
ndemir commentedon Mar 19, 2025
@alamb
I initially thought this was a new issue, but I realized that PR apache/arrow-rs#7253 already addresses and closes it.
I had asked the previous question to clarify whether this issue was expecting a new change.
Long story short, my comment can be ignored since I mistakenly assumed this was a new issue without an existing PR.
alamb commentedon Mar 20, 2025
Migrating from arrow-rs issue #7304