Description

Use dask distributed for data computations.

Works with the development version iris.

Use with ESMValGroup/ESMValTool#3151 to pass the cluster address on to Python diagnostics so they make use of the Dask cluster to run their array computations.

Example usage

esmvaltool run examples/recipe_python.yml --max-parallel-tasks=0 --dask='{"cluster": {"type": "dask_jobqueue.SLURMCluster", "n_workers": 2}}'

this requires installing dask_jobqueue, or locally

esmvaltool run examples/recipe_python.yml --dask='{"client": {}}"

Configuration

The settings max_parallel_tasks can be used to control the how tasks are run:

The dask: cluster setting can be used to configure the Dask cluster, the argument type can be used to specify the Python cluster class (defaults to distributed.LocalCluster), any other arguments will be passed to the class.

The dask: client setting can be used to configure the distributed.Client, any arguments will be passed on.

Example config-user.yml settings for running locally using a LocalCluster:

dask:
  cluster:
    type: distributed.LocalCluster

Example settings for using an externally managed cluster (e.g. set it up from a Jupyter notebook)

dask:
  client:
    address: tcp://127.0.0.1:45695

Example settings for running on Levante:

dask:
  client: {}
  cluster:
    type: dask_jobqueue.SLURMCluster
    queue: interactive
    account: bk1088
    cores: 8
    memory: 16GiB
    local_directory: "/work/bd0854/b381141/dask-tmp"
    n_workers: 2

Example for recovering the previous behaviour: specify no dask item in config-user.yml.

TO DO

• Record info/debug log messages when creating tasks from the cluster
• Fix --resume-from with delayed save
• Investigate why there is a warning about sending a large graph and solve the issue
• Write documentation
• Make dask configuration more user friendly

May close #430

Link to documentation:

Before you get started

• ☝ Create an issue to discuss what you are going to do

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

• 🧪 The new functionality is relevant and scientifically sound
• 🛠 This pull request has a descriptive title and labels
• 🛠 Code is written according to the code quality guidelines
• 🧪 and 🛠 Documentation is available
• 🛠 Unit tests have been added
• 🛠 Changes are backward compatible
• 🛠 Any changed dependencies have been added or removed correctly
• 🛠 The list of authors is up to date
• 🛠 All checks below this pull request were successful

To help with the number pull requests:

• 🙏 We kindly ask you to review two other open pull requests in this repository