Skip to content

Lazy regrid? #3700

@bouweandela

Description

@bouweandela

@bjlittle Would it be possible to make regridding in iris lazy? It seems relatively straightforward, because it is an operation that acts only to the horizontal dimensions of a cube and the dask chunks are typically along the time dimension, at least for the files in CMIP, so it could be done one chunk at a time.

Here are some use cases that make this a really desirable feature for the ESMValTool community:

  • One of the most memory consuming operations we have in ESMValTool is computing multi-model statistics. iris supports computing various statistics in a lazy way, but we cannot benefit from this feature, because models typically come on different grids, so we need to regrid the cubes first. Because regridding is not lazy, we can only compute multi-model statistics for a small number of years.
  • When working with larger datasets, e.g. ERA5 data at an hourly resolution, regridding very quickly runs out of memory.

At the moment regridding is not lazy in iris, data appears to be realized e.g. here:

data = self._regrid(
src.data,

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions