Skip to content

Increase flexibility around Python versions and compression #4011

@mrocklin

Description

@mrocklin

Currently Dask fails to communicate when we try to cross Python versions or have mismatched compression. The Python version mismatch is new-ish ever since we added Pickle protocol 5, the compression is a long-standing issue. In principle we ask that users provide consistent versions across all machines, however we may still be able to accomodate things here with a little bit of work on our end.

When establishing a connection between two endpoints we might have the two sides publish their capabilities. This might inform the other side which compression libraries they have available, whether or not they support Pickle5, or whether or not they have a GPU. This might make communication in heterogeneous situations a bit easier.

We already have some work here. There is a context= keyword that is passed down from Comm.write to to_frames, which in turn handles serialization. We could start populating that with more information and using it in serialization/compression.

This would allow the following situations:

  1. More relaxation around Python versions
  2. More relaxation around compression libraries
  3. Downgrading GPU results smoothly to CPU clients

cc @quasiben @jakirkham

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions