Skip to content

Design: nested _meta #5329

@crusaderky

Description

@crusaderky

I'm opening this on the dask board as it is the first package that came up with the idea of the _meta property, but this is really of interest of all NEP18-compatible packages that wrap around other NEP18 packages.

I'm in the early stages of a project where I will likely need to use xarray, pint, dask, and sparse within the same object. In addition to this, I believe I saw on the sparse board some discussion about using CuPy (or similar) for GPU acceleration.

This could potentially translate to
an xarray.DataArray
that wraps around a pint.Quantity
that wraps around a dask.array.Array
that wraps around a sparse.COO
that wraps around a cupy.array

and in theory, it should just work as long as all the libraries in the stack implement the relevant numpy functions.

Ideally, as a final user I would like to see all of this information when invoking repr() on my top level object, something like

<xarray.DataArray (x: 100)>
<pint.Quantity::dask.array::sparse.COO::cupy.array, 
 unit="seconds", dtype=int64, nnz=2, fill_value=0, Device=0>

This thread is to understand:

  1. how to best represent this kind of arbitrarily nested meta data, both as a data structure and as a __repr__
  2. if it should be formalized through a NEP

@shoyer
@keewis
@hameerabbasi
@mrocklin

Metadata

Metadata

Assignees

No one assigned

    Labels

    discussionDiscussing a topic with no specific actions yet

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions