After seeing @stuartarchibald's post on gitter, I was a bit curious how to_hdf5 actually worked and whether it would be viable to call from multiple processes.
Turns out to_hdf5 just calls store underneath the hood while holding the file open locally. The biggest issue of course is that the file is held open in write mode on the scheduler (meaning nothing else can write to the HDF5 file without corrupting it). Since the type of locking was not specified and store uses a threading.Lock by default, this is also incompatible with the multiprocessing use case (i.e. threading.Lock cannot be serialized/locked across processes). IOW the current implementation of to_hdf5 is not friendly even for locked parallel writes.
That said, it should be feasible to change to_hdf5's behavior to be more friendly for storage from multiple processes (though it will still need to be locked). In this case, it would still create the datasets initially (as it already does), but then would close the file. Instead of passing raw HDF5 Datasets as targets, a wrapper class would need to be used (to allow for pickling). The wrapper class would need to provide a __setitem__ method that would open the HDF5 file and write to the HDF5 Dataset at the selection specified in a process safe manner (probably with locket.lock_file). Ideally it would provide a __getitem__ method as well. Doing this should allow for HDF5 file to be written to in parallel. This assumes the filesystem is very robust and syncing changes between different nodes.
An alternative to this proposal that would avoid locking would be to have data serialized back to the scheduler, which then writes each piece to the HDF5 file as it arrives. This would avoid the overhead of the previous strategy (and any potential issues locking) by guaranteeing only one process ever opens the HDF5 file and keeps open until everything is written. This strategy would continue to work well for non-parallel use cases with arguably less overhead than is present now. The only thing to do before writing out all results would be to optimize the graphs since store would no longer be used.
After seeing @stuartarchibald's post on gitter, I was a bit curious how
to_hdf5actually worked and whether it would be viable to call from multiple processes.Turns out
to_hdf5just callsstoreunderneath the hood while holding the file open locally. The biggest issue of course is that the file is held open in write mode on the scheduler (meaning nothing else can write to the HDF5 file without corrupting it). Since the type of locking was not specified andstoreuses athreading.Lockby default, this is also incompatible with the multiprocessing use case (i.e.threading.Lockcannot be serialized/locked across processes). IOW the current implementation ofto_hdf5is not friendly even for locked parallel writes.That said, it should be feasible to change
to_hdf5's behavior to be more friendly for storage from multiple processes (though it will still need to be locked). In this case, it would still create the datasets initially (as it already does), but then would close the file. Instead of passing raw HDF5 Datasets as targets, a wrapper class would need to be used (to allow for pickling). The wrapper class would need to provide a__setitem__method that would open the HDF5 file and write to the HDF5 Dataset at the selection specified in a process safe manner (probably withlocket.lock_file). Ideally it would provide a__getitem__method as well. Doing this should allow for HDF5 file to be written to in parallel. This assumes the filesystem is very robust and syncing changes between different nodes.An alternative to this proposal that would avoid locking would be to have data serialized back to the scheduler, which then writes each piece to the HDF5 file as it arrives. This would avoid the overhead of the previous strategy (and any potential issues locking) by guaranteeing only one process ever opens the HDF5 file and keeps open until everything is written. This strategy would continue to work well for non-parallel use cases with arguably less overhead than is present now. The only thing to do before writing out all results would be to optimize the graphs since
storewould no longer be used.