Repository navigation
feat: add experimental geoxarray reader - #914
Conversation
61bd124 to
32c44e0
Compare
|
Bug: zarr v3 The nodata detection in self.options.get("nodata", self.input.attrs.get("fill_value"))For zarr v3 arrays, the fill value lives in the array-level metadata ( With For reference, the async nodata = getattr(self.input.metadata, "fill_value", None)which returns the actual Two possible fixes (combined or separately):
elif numpy.issubdtype(data.dtype, numpy.floating):
masked_data.mask = numpy.isnan(masked_data.data)This is robust regardless of fill value storage, since zarr v3 always decodes to actual NaN in the float array when |
|
@emmanuelmathot regarding your comment, I'm not quite sure to follow everything. I know there is an issue with Nodata because I don't think there is a clear convention about where to store this Xarray and Zarr python seems to have different implementation on this (see slack thread). I will want to avoid to do like rioxarray https://github.com/corteva/rioxarray/blob/67e7b63e77c5ad16f6c97ff122faaa0850470fd0/rioxarray/raster_array.py#L162-L196 |
|
@vincentsarago You're right that zarr v3 The zarr v3 spec defines What's wrong with the rioxarray approach (rioxarray/raster_array.py#L162–196): it walks a long priority chain over many different attribute keys ( What xarray actually does today: when opening a zarr v3 array, xarray reads Proposed minimal fix for # Read zarr fill_value from xarray encoding (where xarray stores the zarr metadata fill_value)
if nodata is None:
nodata = self.options.get(
"nodata",
self.input.encoding.get("_FillValue")
)And as a safety net for float arrays (matches masked_data = numpy.ma.masked_invalid(data) # masks NaN and Inf for float dtypes
if nodata is not None and not numpy.isnan(nodata):
masked_data = numpy.ma.masked_equal(masked_data, nodata)The same fix applies to |
70c0521 to
3e7ba9a
Compare
| "missing_value", | ||
| self.input.attrs.get("fill_value", self.input.attrs.get("nodata")), | ||
| ), | ||
| ) |
| fill_value=-9999.0, | ||
| dimension_names=("y", "x"), | ||
| attributes={ | ||
| "_FillValue": FillValueCoder.encode(-9999.0, numpy.dtype("float32")), |
There was a problem hiding this comment.
in this case, xarray will remove _FillValue from the attribute and replace those values by nan in the array.
We adapted the code to first use DataArray.to_masked() which will mask those nan value but also allow nodata overwriting
21c0070 to
151d6c5
Compare
45051a0 to
dc5ef2a
Compare
|
tests are still failing for python 3.14 here is what Claude is telling me Root cause analysis:
# xarray/coding/variables.py
def _apply_mask(data, encoded_fill_values, decoded_fill_value, dtype):
- data = np.asarray(data, dtype=dtype)
+ data = np.array(data, dtype=dtype) # always copies; zarr chunks may share memory
condition = False
for fv in encoded_fill_values:
condition |= data == fv
return np.where(condition, decoded_fill_value, data) |
|
xarray doesn't support python3.14 officially yet so we can ignore the segfault for now |
9434da1 to
f30f72a
Compare
f30f72a to
b531710
Compare
ref #909