Dataset.eval does not support N-dimensional objects with N > 2

What is your issue?

Using

xarray 2025.12.0
pandas 2.3.3

Dataset.eval uses Pandas under the hood to handle the eval of arbitrary expressions

Lines 9538 to 9598 in f8bc4f4

    
               def eval( 
        
                   self, 
        
                   statement: str, 
        
                   *, 
        
                   parser: QueryParserOptions = "pandas", 
        
               ) -> Self | T_DataArray: 
        
                   """ 
        
                   Calculate an expression supplied as a string in the context of the dataset. 
        
                   This is currently experimental; the API may change particularly around 
        
                   assignments, which currently return a ``Dataset`` with the additional variable. 
        
                   Currently only the ``python`` engine is supported, which has the same 
        
                   performance as executing in python. 
        
                   Parameters 
        
                   ---------- 
        
                   statement : str 
        
                       String containing the Python-like expression to evaluate. 
        
                   Returns 
        
                   ------- 
        
                   result : Dataset or DataArray, depending on whether ``statement`` contains an 
        
                   assignment. 
        
                   Examples 
        
                   -------- 
        
                   >>> ds = xr.Dataset( 
        
                   ...     {"a": ("x", np.arange(0, 5, 1)), "b": ("x", np.linspace(0, 1, 5))} 
        
                   ... ) 
        
                   >>> ds 
        
                   <xarray.Dataset> Size: 80B 
        
                   Dimensions:  (x: 5) 
        
                   Dimensions without coordinates: x 
        
                   Data variables: 
        
                       a        (x) int64 40B 0 1 2 3 4 
        
                       b        (x) float64 40B 0.0 0.25 0.5 0.75 1.0 
        
                   >>> ds.eval("a + b") 
        
                   <xarray.DataArray (x: 5)> Size: 40B 
        
                   array([0.  , 1.25, 2.5 , 3.75, 5.  ]) 
        
                   Dimensions without coordinates: x 
        
                   >>> ds.eval("c = a + b") 
        
                   <xarray.Dataset> Size: 120B 
        
                   Dimensions:  (x: 5) 
        
                   Dimensions without coordinates: x 
        
                   Data variables: 
        
                       a        (x) int64 40B 0 1 2 3 4 
        
                       b        (x) float64 40B 0.0 0.25 0.5 0.75 1.0 
        
                       c        (x) float64 40B 0.0 1.25 2.5 3.75 5.0 
        
                   """ 
        
                   return pd.eval(  # type: ignore[return-value] 
        
                       statement, 
        
                       resolvers=[self], 
        
                       target=self, 
        
                       parser=parser, 
        
                       # Because numexpr returns a numpy array, using that engine results in 
        
                       # different behavior. We'd be very open to a contribution handling this. 
        
                       engine="python", 
        
                   )

However, Pandas explicitly does not support data with dimensionality > 2:

https://github.com/pandas-dev/pandas/blob/9b4461268867c548f575c75733f46fee7c5ad585/pandas/core/computation/ops.py#L118-L120

Since I was interested in simple additions and subtractions among variables, I tried to disable the check. The code runs on benchmark 3D data, but painstakingly slowly.

I came up with this solution

def _ds_eval(ds: xr.Dataset, expr: str) -> xr.DataArray:
    import ast

    ALLOWED_NODES = (
        ast.Expression, ast.BinOp, ast.Name, ast.Load, ast.Constant,
        ast.Add, ast.Sub, ast.Mult, ast.Div,
    )

    def validate(expr):
        tree = ast.parse(expr, mode="eval")
        for node in ast.walk(tree):
            if not isinstance(node, ALLOWED_NODES):
                raise ValueError(f"Disallowed expression: {node}")
        return tree

    tree = validate(expr)
    compiled = compile(tree, "<expr>", "eval")

    return eval(
        compiled,
        {"__builtins__": {}},  # globals
        ds.data_vars,  # locals
    )

# example usage, assuming ds has variables x and y
z = ds_eval(ds, "x + y")

Since ds.eval is read-only, it is not possible to monkeypatch the Dataset object.

I'm putting it here in case someone else needs it and to open a discussion on possible improvements on the current implementation, which I would be happy to help on.

Thanks for opening your first issue here at xarray! Be sure to follow the issue template!
If you have an idea for a solution, we would really welcome a Pull Request with proposed changes.
See the Contributing Guide for more.
It may take us a while to respond here, but we really value your contribution. Contributors like you help make xarray better.
Thank you!

~~do you have a repro please?~~ ok it repros

@max-sixty

@max-sixty, here is a MWE

import numpy as np
import xarray as xr

data_3d = np.arange(24).reshape(2, 3, 4)
data_2d = np.arange(12).reshape(3, 4)

ds = xr.Dataset(
    {
        "v0": (["z", "x", "y"], data_3d),
        "v1": (["z", "x", "y"], data_3d * 2),
        "v2": (["x", "y"], data_2d),
        "v3": (["x", "y"], data_2d * 3),
    },
    coords={"z": [0, 1], "x": [0, 1, 2], "y": [0, 1, 2, 3]},
)

this_works = ds.eval("v2 + v3")
this_doesnt = ds.eval("v0 + v1")

@jacopo-exact

yes, sorry about this @jacopo-exact . will fix...

@max-sixty

yes, sorry about this @jacopo-exact . will fix...

Nothing to be sorry about, @max-sixty. Let me know if I can be of any help. Thanks!

fixed in #11064

Navigation Menu

Repository navigation

Dataset.eval does not support N-dimensional objects with N > 2 #11062

What is your issue?

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	def eval(
	self,
	statement: str,
	*,
	parser: QueryParserOptions = "pandas",
	) -> Self \| T_DataArray:
	"""
	Calculate an expression supplied as a string in the context of the dataset.

	This is currently experimental; the API may change particularly around
	assignments, which currently return a ``Dataset`` with the additional variable.
	Currently only the ``python`` engine is supported, which has the same
	performance as executing in python.

	Parameters
	----------
	statement : str
	String containing the Python-like expression to evaluate.

	Returns
	-------
	result : Dataset or DataArray, depending on whether ``statement`` contains an
	assignment.

	Examples
	--------
	>>> ds = xr.Dataset(
	... {"a": ("x", np.arange(0, 5, 1)), "b": ("x", np.linspace(0, 1, 5))}
	... )
	>>> ds
	<xarray.Dataset> Size: 80B
	Dimensions: (x: 5)
	Dimensions without coordinates: x
	Data variables:
	a (x) int64 40B 0 1 2 3 4
	b (x) float64 40B 0.0 0.25 0.5 0.75 1.0

	>>> ds.eval("a + b")
	<xarray.DataArray (x: 5)> Size: 40B
	array([0. , 1.25, 2.5 , 3.75, 5. ])
	Dimensions without coordinates: x

	>>> ds.eval("c = a + b")
	<xarray.Dataset> Size: 120B
	Dimensions: (x: 5)
	Dimensions without coordinates: x
	Data variables:
	a (x) int64 40B 0 1 2 3 4
	b (x) float64 40B 0.0 0.25 0.5 0.75 1.0
	c (x) float64 40B 0.0 1.25 2.5 3.75 5.0
	"""

	return pd.eval( # type: ignore[return-value]
	statement,
	resolvers=[self],
	target=self,
	parser=parser,
	# Because numexpr returns a numpy array, using that engine results in
	# different behavior. We'd be very open to a contribution handling this.
	engine="python",
	)

Dataset.eval does not support N-dimensional objects with N > 2 #11062

Description

What is your issue?

Activity

welcome commented on Dec 31, 2025

max-sixty commented on Dec 31, 2025

jacopo-exact commented on Dec 31, 2025

max-sixty commented on Dec 31, 2025

jacopo-exact commented on Dec 31, 2025

max-sixty commented on Dec 31, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions