-
-
Notifications
You must be signed in to change notification settings - Fork 1.8k
Description
As requested by @jakirkham I'm creating a github issue for this. I started this conversation on gitter and was about to make a SO question but have posted it here instead.
I have a custom array-like object that wraps a file-like object and I pass this custom object to Dask's store function to save a dask array. You can see an earlier implementation of that class here but it shouldn't matter for this question. So the issue I'm having is how to pass a file-like object as a target and have it explicitly closed. In normal execution the file seems to be closed after dask computes everything, in my unit tests with python's unittest the file seems to get closed before dask computes when I call it like this:
with MyFileObject(...) as my_file:
return da.store(my_data, my_file, compute=False)
...
result.compute() # compute the above delayed
I then decided to try using dask.delayed around a function that opens the file, runs store, then closes the file. But then I realized that delayed will compute my dask array before passing it to my function, defeating the purpose of using da.store for storing chunks.
So my question is, is there any way to use da.store with a file-like object, explicitly close that file-like object, and get the benefits of da.store storing individual chunks?