Saving zarr to remote location lower cases all data_vars #5028

Activity

1. fwrite commented on Mar 15, 2021

   

   fwrite

   on Mar 15, 2021

   More actions

   Interestingly, the same applies to locally created and remotely opened data sets. The variables are recognized correctly from the consolidated metadata, but on opening are made lower case.

   A quick example:

   import xarray as xr, numpy as np
   x, y, t = np.arange(500), np.arange(500), np.arange(0,500,6)
   
   xr.Dataset(
       data_vars={'Group/Variable': (('t', 'y', 'x'), np.ones((t.size, x.size, y.size)))},
       coords={'x': x, 'y': y, 'Time': t}
   ).to_zarr('/tmp/zarr2058', consolidated=True)

   Locally, everything is fine (every value should be 1).

   xr.open_zarr('/tmp/zarr2058', group='Group').Variable[0,0,0].compute() # = 1

   However, if we access the data via http, the values are not read correctly.

   To test, start a local webserver:

   python3 -m http.server -d /tmp/zarr2058 8080

   Open our array, the group and variable are properly cased (and derived from .zmetadata):

   xr.open_zarr('http://localhost:8080/', group='Group', consolidated=True)

   However, access to one of the variables will yield a lower case request to the server, and output nan for any array position:

   xr.open_zarr('http://localhost:8080/', group='Group', consolidated=True).Variable[0,0,0].compute() # = nan

   The terminal will read the requested url, all in lower case:

   "GET /group/variable/0.0.0 HTTP/1.1" 404
   

   However, the problem appears to be in Zarr, not (only) in xarray:

   zarr.open_group('http://localhost:8080/')['Group']

   Behaves exactly the same.

   Environment:

   Output of xr.show_versions()

   INSTALLED VERSIONS

   commit: None
   python: 3.7.7 (default, Mar 23 2020, 22:36:06)
   [GCC 7.3.0]
   python-bits: 64
   OS: Linux
   OS-release: 5.8.0-44-generic
   machine: x86_64
   processor: x86_64
   byteorder: little
   LC_ALL: None
   LANG: en_US.UTF-8
   LOCALE: en_US.UTF-8
   libhdf5: 1.10.4
   libnetcdf: 4.6.1

   xarray: 0.15.0
   pandas: 1.1.3
   numpy: 1.18.1
   scipy: 1.4.1
   netCDF4: 1.4.2
   pydap: None
   h5netcdf: 0.10.0
   h5py: 2.10.0
   Nio: None
   zarr: 2.6.1
   cftime: 1.4.1
   nc_time_axis: None
   PseudoNetCDF: None
   rasterio: 1.1.0
   cfgrib: None
   iris: None
   bottleneck: None
   dask: 2.13.0
   distributed: 2.30.1
   matplotlib: 3.3.2
   cartopy: None
   seaborn: None
   numbagg: None
   setuptools: 52.0.0.post20210125
   pip: 21.0.1
   conda: None
   pytest: None
   IPython: 7.13.0
   sphinx: None
2. fwrite commented on Mar 15, 2021

   

   fwrite

   on Mar 15, 2021

   Last edited by fwrite

   More actions

   The issue appears to be in zarr.storage.FSStore, and is also discussed in zarr#649.

   Recently, a storage_options argument was added to open_zarr(). This argument might be used to provide the extra option storage_options={'normalize_keys': False} to Zarr.

   Coming back to my last example:

   zarr.open_consolidated('http://localhost:8080/', storage_options={'normalize_keys': False})['Group'].Variable[0,0,0]

   Works.

   Work-around, instantiate the store first:

   xr.open_zarr(store=zarr.storage.FSStore('http://localhost:8080/', normalize_keys=False),
                group='Group',
                consolidated=True).Variable[0,0,0].compute() # = 1
3. dcherian

   added

   topic-zarrRelated to zarr storage libraryRelated to zarr storage library

   on Apr 24, 2021
4. joshmoore commented on Jun 14, 2021

   

   joshmoore

   on Jun 14, 2021

   More actions

   A heads up that zarr-developers/zarr-python#755 reverses the default behavior of FSStore and so as of Zarr 2.8.3 this might "just work"?

   👍React with 👍2Reacted by Brendan McAndrew and fwrite
5. raybellwaves commented on Jun 15, 2021

   

   raybellwaves

   on Jun 15, 2021

   ContributorAuthor

   More actions

   Works. Thanks @joshmoore !

   

   👍React with 👍2Reacted by Josh Moore and Ryan Abernathey
6. raybellwaves

   closed this as completedon Jun 15, 2021
7. rabernat commented on Jun 17, 2021

   

   rabernat

   on Jun 17, 2021

   Last edited by rabernat

   Contributor

   More actions

   So glad this got fixed upstream! That's how it is supposed to work! 🏆 Thanks to everyone for making this happen.