-
-
Notifications
You must be signed in to change notification settings - Fork 750
Description
Currently Dask fails to communicate when we try to cross Python versions or have mismatched compression. The Python version mismatch is new-ish ever since we added Pickle protocol 5, the compression is a long-standing issue. In principle we ask that users provide consistent versions across all machines, however we may still be able to accomodate things here with a little bit of work on our end.
When establishing a connection between two endpoints we might have the two sides publish their capabilities. This might inform the other side which compression libraries they have available, whether or not they support Pickle5, or whether or not they have a GPU. This might make communication in heterogeneous situations a bit easier.
We already have some work here. There is a context= keyword that is passed down from Comm.write to to_frames, which in turn handles serialization. We could start populating that with more information and using it in serialization/compression.
This would allow the following situations:
- More relaxation around Python versions
- More relaxation around compression libraries
- Downgrading GPU results smoothly to CPU clients