It is repeatedly mentioned to us that __setitem__ style da.store function doesn't work well for a majority of user cases as this means they need to open a file object and put in the graph. This then can fail for a variety of reasons like it is not safe to pickle, send across process boundaries, etc. Would be nice to have a form of store that takes a UDF that can handle all the file open/close logic as well as writing out the chunk.
cc @djhoese
It is repeatedly mentioned to us that
__setitem__styleda.storefunction doesn't work well for a majority of user cases as this means they need to open a file object and put in the graph. This then can fail for a variety of reasons like it is not safe to pickle, send across process boundaries, etc. Would be nice to have a form ofstorethat takes a UDF that can handle all the file open/close logic as well as writing out the chunk.cc @djhoese