Missing rectangles in zarr dataset loaded from GCS

I have had some issue with loading slices from some zarr dataset stored on GCS. I don't know where the problem could be coming from so am posting it here in case someone has experienced something similar.

The figure below shows what is going on. Upon slicing and loading some zarr dataset stored on GCS using xarray, I'm getting missing squares (often in the northern hemisphere). The squares change their positions each time I reload the same slice, sometimes they appear for one single variable, sometimes for more than one (at different locations), sometimes they don't happen at all. The rectangle sizes match the chunking which for this dataset corresponds to 25 degrees in lat/lon so it looks like entire chunks are randomly failing to be transferred for some reason.

Has someone seen something similar or would have an idea about this please?

Thanks

@rafa-guedes

Hi @rafa-guedes - could you share more information about your dataset? Like the xarray repr and the zarr .info property of one of the arrays?

@rabernat

Hi @rabernat please find some more info below.

In [1]: import xarray as xr

In [2]: import zarr

In [3]: from fsspec import get_mapper

In [4]: fsmap = get_mapper('gs://oceanum-era5/wind_10m.zarr')

In [5]: dset = xr.open_zarr(fsmap, consolidated=True)

In [6]: dset
Out[6]: 
<xarray.Dataset>
Dimensions:    (latitude: 721, longitude: 1440, time: 354264)
Coordinates:
  * latitude   (latitude) float32 90.0 89.75 89.5 89.25 ... -89.5 -89.75 -90.0
  * longitude  (longitude) float32 0.0 0.25 0.5 0.75 ... 359.25 359.5 359.75
  * time       (time) datetime64[ns] 1979-01-01 ... 2019-05-31T23:00:00
Data variables:
    u10        (time, latitude, longitude) float32 dask.array<shape=(354264, 721, 1440), chunksize=(744, 100, 100)>                                                                                         
    v10        (time, latitude, longitude) float32 dask.array<shape=(354264, 721, 1440), chunksize=(744, 100, 100)>                                                                                         
Attributes:
    Conventions:  CF-1.6
    history:      2019-08-01 10:44:37 GMT by grib_to_netcdf-2.10.0: /opt/ecmw...

In [7]: g = zarr.open_group(fsmap)

In [8]: g.info
Out[8]: 
Name        : /
Type        : zarr.hierarchy.Group
Read-only   : False
Store type  : fsspec.mapping.FSMap
No. members : 5
No. arrays  : 5
No. groups  : 0
Arrays      : latitude, longitude, time, u10, v10

In [9]: g.u10.info
Out[9]: 
Name               : /u10
Type               : zarr.core.Array
Data type          : int16
Shape              : (354264, 721, 1440)
Chunk shape        : (744, 100, 100)
Order              : C
Read-only          : False
Compressor         : Blosc(cname='zstd', clevel=3, shuffle=BITSHUFFLE,
                   : blocksize=0)
Store type         : fsspec.mapping.FSMap
No. bytes          : 735622110720 (685.1G)
Chunks initialized : 57240/57240

In [10]: g.v10.info
Out[10]: 
Name               : /v10
Type               : zarr.core.Array
Data type          : int16
Shape              : (354264, 721, 1440)
Chunk shape        : (744, 100, 100)
Order              : C
Read-only          : False
Compressor         : Blosc(cname='zstd', clevel=3, shuffle=BITSHUFFLE,
                   : blocksize=0)
Store type         : fsspec.mapping.FSMap
No. bytes          : 735622110720 (685.1G)
Chunks initialized : 57240/57240

In [11]: dset.u10.encoding
Out[11]: 
{'chunks': (744, 100, 100),
 'compressor': Blosc(cname='zstd', clevel=3, shuffle=BITSHUFFLE, blocksize=0),
 'filters': None,
 '_FillValue': -32767,
 'scale_factor': 0.01,
 'dtype': dtype('int16')}

@martindurant

Perhaps GCS is not returning the data reliably? So @martindurant might have some suggestions on how to get some debugging out of gcsfs. I seem to recall a trick to turn on http logging.

You can set the level gcsfs.core.logger.set_level("DEBUG"), so long as you have some logging initialised (like logging.basicConfig()).

@rabernat

Sorry for taking long to reply, I am only now getting back on this. Thanks for the suggestions @rabernat, @martindurant. There is nothing obvious from those logs though, and the problem still persists, intermittently. I may try and recreate this zarr dataset and check again. I'd be interested to hear if someone has found some similar behaviour in the past, this looks like a bit of a serious issue and I don't know where it would be coming from..

I'm seeing similar intermittent errors with data in the gs://pangeo-parcels bucket.

Here's a gist showing that calculations fail with group not found at path ValueErrors and, if computation is done with retries=10, the statistics change intermittently:

https://nbviewer.jupyter.org/gist/willirath/b7af4bc9a93a79b81772910f8ee5c630

Is there a way of forcing Zarr to retry if a chunk cannot be read rather than setting the chunk to all NaNs?

I would open an issue in gcsfs for this. https://github.com/dask/gcsfs/issues/

…

On Tue, Oct 1, 2019 at 7:41 AM Willi Rath ***@***.***> wrote: Is there a way of forcing Zarr to retry if a chunk cannot be read rather than setting the chunk to all NaNs? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#691?email_source=notifications&email_token=AAJEKJRFUUMYX3KTDKVQZSDQMMZPHA5CNFSM4IK5ONJ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAA64LI#issuecomment-536997421>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAJEKJXNQVQJSBXY3QWJ64DQMMZPHANCNFSM4IK5ONJQ> .

Does it happen if you set the number of threads per worker to one?

Does it happen if you set the number of threads per worker to one?

Yes, configuring KubeCluster to use --nthreads 2 does not seem to change anything.

and --nthreads 1 ?

Errr, yes, I meant and --nthreads 1.

I honestly cannot think of why this is happening - help appreciated in debugging when/how this happens, the exact error, etc. If it happens in one thread, apparently not a race condition or effect of session sharing. If we are lucky, it is merely one particular exception from the remote backend which we don't consider as retriable, but should (e.g., API rate limit).

Is there an easy way of preventing GCSFS from retrying at all?

Not super-easy: the GCSFileSystem class has an attribute .retries, so this can be set to 1 in every worker with a client.run call

def set_gcsfs_retry():
    import gcsfs
    gcsfs.GCSFileSystem.retries = 1

client.run(set_gcsfs_retry)

I would also run with GCSFS_DEBUG=true

gcsfs.utils.is_retriable is the place where errors are sorted.

Finished without a NaN:

ds_combined.distance.data.sum().compute()
2445454000000.0

(I do this via the dask array so that I don't have to worry about interpretation of NaN - even one would make the result be NaN)

Could we close this?

@willirath

Waiting for the OK from @willirath ; who may want to run with Client() (i.e., a LocalCluster of dask processes, launched from the same environment) to test

I've just created a setup with KubeCluster on the pangeo binder. Stay tuned...

I still get NaN's.

Gist (still running, so there will be an update) is here: https://gist.github.com/willirath/b40b0b6f281cb1a46fcf848150ca0367

My impression is that the errors occur a lot less frequent. But this might be due to adapted rate limits on the GCS side?

results = []
for n in range(20):
    
    vlist = [
        ds[vname].isnull().data.sum()
        for vname in ds.data_vars.keys()
    ]
    
    results.append(sum(vlist).compute(retries=10))
    
    print(n, results[-1])
    
    sleep(20) # allow for scaling down

0 96200000
1 0
2 0
3 0
4 0
5 0
6 0
7 0
8 0
9 0
10 0
11 0
12 0
13 181433200
14 0
15 0

I repeated my tests 1000 times now on a 4-worker local cluster, no nans at all for me

@willirath

OK, so closing this, and maybe @willirath is facing a different issue (another rate limit?). More logging may tell.

I've raised zarr-developers/zarr-python#486 hoping to find out how to more easily trigger an Exception upon errors when loading chunks.

Zarr deciding that where a chunk does not exist, it should be nan is the right behaviour. The question is, what error is actually coming out of gcsfs (FileNotFound instead of some permission error in the file-system layer, and/or IndexError in the mapper) and what errors would be right for zarr. Is gcsfs retrying the underlying error? This info should be in the logs, or else the logging should be improved!

Just to say we're hitting something like this currently, looking at the fsspec code I think there may still be a possibility that transient errors from storage are getting propagated as KeyErrors to zarr, which then fills in the chunk with missing data. Have raised an issue here: fsspec/filesystem_spec#255.

We're currently using this workaround which can be used where you know all chunks should be there for an array, i.e., you never expect a KeyError to get raised from the store for any reason.

For information, changes merged upstream in fsspec that should protect against this happening: fsspec/filesystem_spec#259

@alimanfoo

Thanks for tracking this down @alimanfoo!

Over in pydata/xarray#3831, we have been discussing how to set up ecosystem-wide integration testing for all of the interconnected packages one needs to use xarray + zarr on the cloud. I'd love to get your thoughts on how best to do this.

Navigation Menu

Repository navigation

Missing rectangles in zarr dataset loaded from GCS #691

26 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Missing rectangles in zarr dataset loaded from GCS #691

Description

Activity

rabernat commented on Aug 12, 2019

rafa-guedes commented on Aug 12, 2019

rabernat commented on Aug 14, 2019

martindurant commented on Aug 14, 2019

rafa-guedes commented on Sep 13, 2019

willirath commented on Oct 1, 2019

willirath commented on Oct 1, 2019

rabernat commented on Oct 1, 2019

martindurant commented on Oct 1, 2019

willirath commented on Oct 1, 2019

martindurant commented on Oct 1, 2019

willirath commented on Oct 1, 2019

martindurant commented on Oct 1, 2019

willirath commented on Oct 1, 2019

martindurant commented on Oct 1, 2019

26 remaining items

martindurant commented on Oct 11, 2019

rafa-guedes commented on Oct 12, 2019

martindurant commented on Oct 12, 2019

willirath commented on Oct 12, 2019

willirath commented on Oct 12, 2019

willirath commented on Oct 12, 2019

rafa-guedes commented on Oct 14, 2019

martindurant commented on Oct 14, 2019

willirath commented on Oct 19, 2019

martindurant commented on Oct 19, 2019

alimanfoo commented on Mar 24, 2020

alimanfoo commented on Mar 24, 2020

alimanfoo commented on Mar 27, 2020

rabernat commented on Mar 27, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions