Skip to content

Improve blockwise_meta exception handling #4952

@pentschev

Description

@pentschev

Current blockwise_meta/blockwise implementation defaults to dtype construction of Dask.Array when an non-trivial exception is raised when computing _meta. This is suboptimal for two reasons:

  1. It's difficult to trace that the defaulting happened within blockwise for somewhat complex user code;
  2. Exceptions that come from the array compute task (with NumPy, CuPy, etc.) are not raised to the user during graph construction.

Ideally, we would handle all possible exceptions, with all remaining ones assumed to be user's mistake, such as passing casting to cupy.einsum (which isn't supported), as discussed in #4914.

Given that _meta now exists, Dask has become a universal scheduler, which probably means we can't handle every single possible exception, but we should at least allow developers to know about such cases, as well as letting users know they possibly made a mistake. We should therefore let most exceptions pass back to the user in blockwise.

This is loosely related to #4951, where we wish to allow users/developers be alerted of conditions that Dask can't handle and should probably be better handled by the library which Dask is using as a backend through __array_function__.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions