Skip to content

feat: add experimental geoxarray reader - #914

Merged
vincentsarago merged 9 commits into
mainfrom
feat/experimental-geoxarray
May 29, 2026
Merged

vincentsarago merged 9 commits into
mainfrom
feat/experimental-geoxarray

Conversation

@vincentsarago

@vincentsarago vincentsarago commented Apr 15, 2026 •

Copy link
Copy Markdown
Member

ref #909

import xarray
from obstore.store import HTTPStore
from zarr.storage import ObjectStore
from affine import Affine
from rasterio.crs import CRS
from matplotlib.pyplot import imshow

from rio_tiler.experimental.xarray import GeoArrayReader

# Create Obstore Store
store = HTTPStore("https://s3.explorer.eopf.copernicus.eu/esa-zarr-sentinel-explorer-fra/tests-output/sentinel-2-l2a/S2B_MSIL2A_20260216T142149_N0512_R096_T25WFV_20260216T165051.zarr")
zarr_store = ObjectStore(store=store, read_only=True)
dt = xarray.open_datatree(
    zarr_store,
    engine="zarr",
    create_default_indexes=False,
    consolidated=True,
)

ds = dt["/measurements/reflectance/r720m"]

attributes = ds.attrs

tr = attributes["spatial:transform"]
transform = Affine(*tr)
crs = CRS.from_user_input(attributes["proj:code"])

with GeoArrayReader(ds["b02"], transform=transform, crs=crs) as dst:
    img = dst.preview()

@emmanuelmathot

Copy link
Copy Markdown
Contributor

@vincentsarago
vincentsarago force-pushed the feat/experimental-geoxarray branch from 61bd124 to 32c44e0 Compare May 12, 2026 19:39
@emmanuelmathot

Copy link
Copy Markdown
Contributor

Bug: zarr v3 fill_value: "NaN" is not detected as nodata in GeoArrayReader, causing NaN pixels to render as 0

The nodata detection in GeoArrayReader._read() (and info()) checks:

self.options.get("nodata", self.input.attrs.get("fill_value"))

For zarr v3 arrays, the fill value lives in the array-level metadata (fill_value: "NaN" in zarr.json), not in user attributes. When xarray opens such an array it stores this in da.encoding["_FillValue"] — not da.attrs — so the lookup returns None.

With nodata = None the masking block is skipped entirely. The raw float32 data already contains NaN values (the cast_value codec decodes stored zeros back to NaN), but they are never masked, so they are cast to 0 on render.

For reference, the async zarr.Reader handles this correctly with:

nodata = getattr(self.input.metadata, "fill_value", None)

which returns the actual float("nan") directly from the zarr-python metadata object.

Two possible fixes (combined or separately):

  1. Extend the detection chain to also probe self.input.encoding.get("_FillValue") — where xarray stores the zarr fill value.

  2. Auto-mask NaN for float arrays as a fallback (mirrors what da.to_masked_array() does in XarrayReader):

elif numpy.issubdtype(data.dtype, numpy.floating):
    masked_data.mask = numpy.isnan(masked_data.data)

This is robust regardless of fill value storage, since zarr v3 always decodes to actual NaN in the float array when fill_value: "NaN".

@vincentsarago

Copy link
Copy Markdown
Member Author

@emmanuelmathot regarding your comment, I'm not quite sure to follow everything.

I know there is an issue with Nodata because I don't think there is a clear convention about where to store this nodata sentinel value. The zarr V3 fill_value metadata describe how the missing chuck should be interpreted but this is not the same as nodata within an array.

Xarray and Zarr python seems to have different implementation on this (see slack thread).

I will want to avoid to do like rioxarray https://github.com/corteva/rioxarray/blob/67e7b63e77c5ad16f6c97ff122faaa0850470fd0/rioxarray/raster_array.py#L162-L196

@emmanuelmathot

Copy link
Copy Markdown
Contributor

@vincentsarago You're right that zarr v3 fill_value and CF-style nodata are semantically distinct concepts. But in practice, for this use case they converge:

The zarr v3 spec defines fill_value as the value used to represent uninitialized / missing data. In Sentinel-2 GeoZarr products, fill_value: "NaN" is the intended sentinel for "no valid observation here". This is not ambiguous, OpenLayers recently implemented exactly this in openlayers/openlayers#17355: read fill_value from zarr metadata, use it as nodata, discard those pixels.

What's wrong with the rioxarray approach (rioxarray/raster_array.py#L162–196): it walks a long priority chain over many different attribute keys (_FillValue, missing_value, nodata, nodatavals, rasterio manager…), trying to reconcile CF, HDF5 and rasterio conventions. It is complex, brittle, and still doesn't robustly handle zarr v3 because it primarily inspects da.attrs rather than zarr metadata.

What xarray actually does today: when opening a zarr v3 array, xarray reads fill_value from the zarr array metadata and surfaces it in da.encoding["_FillValue"]. There is an open xarray issue (pydata/xarray#11332) where @maxrjones is working on making _FillValue in zarr attributes accept JSON-native scalars properly. Until that lands, da.encoding.get("_FillValue") is the right place to read it which is what the async zarr.Reader already does via self.input.metadata.fill_value.

Proposed minimal fix for _read() — no complex convention chain, no rioxarray pattern:

# Read zarr fill_value from xarray encoding (where xarray stores the zarr metadata fill_value)
if nodata is None:
    nodata = self.options.get(
        "nodata",
        self.input.encoding.get("_FillValue")
    )

And as a safety net for float arrays (matches da.to_masked_array() behavior):

masked_data = numpy.ma.masked_invalid(data)  # masks NaN and Inf for float dtypes
if nodata is not None and not numpy.isnan(nodata):
    masked_data = numpy.ma.masked_equal(masked_data, nodata)

The same fix applies to info().

@vincentsarago
vincentsarago force-pushed the feat/experimental-geoxarray branch from 70c0521 to 3e7ba9a Compare May 19, 2026 20:48
"missing_value",
self.input.attrs.get("fill_value", self.input.attrs.get("nodata")),
),
)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CF style nodata

Comment thread tests/test_geoxarray.py
fill_value=-9999.0,
dimension_names=("y", "x"),
attributes={
"_FillValue": FillValueCoder.encode(-9999.0, numpy.dtype("float32")),

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in this case, xarray will remove _FillValue from the attribute and replace those values by nan in the array.

We adapted the code to first use DataArray.to_masked() which will mask those nan value but also allow nodata overwriting

@vincentsarago
vincentsarago marked this pull request as ready for review May 19, 2026 21:02
@vincentsarago
vincentsarago force-pushed the feat/experimental-geoxarray branch 2 times, most recently from 21c0070 to 151d6c5 Compare May 20, 2026 14:15
@vincentsarago
vincentsarago force-pushed the feat/experimental-geoxarray branch from 45051a0 to dc5ef2a Compare May 21, 2026 17:23
@vincentsarago

Copy link
Copy Markdown
Member Author

tests are still failing for python 3.14

here is what Claude is telling me

Root cause analysis:

  • _apply_mask (xarray/coding/variables.py:132) calls np.where(condition, decoded_fill_value, data)
  • data comes from ZarrArrayWrapper.getitem → zarr v3's sync wrapper → zarr's I/O pipeline
  • If zarr v3 returns an array that shares memory with an internal chunk cache buffer (common in chunked I/O), and zarr's background threads evict/reuse that buffer while np.where holds a reference to it, you get a use-after-free crash at the C level
  • CPython 3.14 may have changed internal memory management in ways that make this easier to trigger
    The xarray fix:
    Change _apply_mask line 128 from np.asarray (may return a view) to np.array (always copies), ensuring the data is independent of zarr's buffer before numpy operations:
# xarray/coding/variables.py  
def _apply_mask(data, encoded_fill_values, decoded_fill_value, dtype):
-   data = np.asarray(data, dtype=dtype)
+   data = np.array(data, dtype=dtype)   # always copies; zarr chunks may share memory
    condition = False
    for fv in encoded_fill_values:
        condition |= data == fv
    return np.where(condition, decoded_fill_value, data)

@vincentsarago vincentsarago mentioned this pull request May 26, 2026
3 of 6 tasks
@vincentsarago

Copy link
Copy Markdown
Member Author

xarray doesn't support python3.14 officially yet so we can ignore the segfault for now

@vincentsarago
vincentsarago force-pushed the feat/experimental-geoxarray branch from 9434da1 to f30f72a Compare May 26, 2026 20:33
@vincentsarago
vincentsarago force-pushed the feat/experimental-geoxarray branch from f30f72a to b531710 Compare May 26, 2026 20:45
@vincentsarago
vincentsarago merged commit d7d1863 into main May 29, 2026
4 checks passed
@vincentsarago
vincentsarago deleted the feat/experimental-geoxarray branch May 29, 2026 13:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants