xesmf included in pangeo.pydata.org ?

Hi all,
I have put together a use-case for reproducing the canonical ICCP-AR5 Figure12.5 using the CMIP5 multi-model surface temperature data. I have converted the needed netcdf files to zarr and uploaded to pangeo.pydata.org. I just noticed that the xesmf packages has not been included, so I can't do the fast and easy spatial regridding provided by xesmf to combine all models on the same grid. See:
use case on pangeo.pydata.org.
Any chance we can add xesmf to the standard packages?

Here is the figure this notebook can reproduce:

@naomi-henderson

This is a great point @naomi-henderson. (FYI, we can't follow the link http://pangeo.pydata.org/user/naomi-henderson/lab/tree/CMIP5-MakeFigure12.5.ipynb. If you would like to share a notebook from pangeo, your best bet right now is to download it and upload it as a gist.)

We have discussed xesmf use cases in #197, but your example is much more concrete. It would be a great example to highlight the capabilities of pangeo to the climate community.

The main roadblock right now is pangeo-data/helm-chart#29, which is holding us up from adding new packages to pangeo.pydata.org. Hopefully that will be resolved very soon, and we can move ahead with this.

@rabernat

Ah, thanks @rabernat - I will wait for this to be sorted out. I will also need xESMF to finish the atmospheric moisture budget use case. The ability to do regridding is essential in dealing with the CMIP5 (and soon to come CMIP6) data.

@JiaweiZhuang

xesmf is now on the stock software environment on Pangeo. I believe that this is now resolved. I'm curious to see it used.

cc @JiaweiZhuang

@mrocklin

@mrocklin fantastic! So far so good, xesmf is happily calculating the weight files and reusing them when appropriate. Thanks much

I wonder how xesmf will work with distributed. Will it scatter the weight matrix to the workers?

…

Sent from my iPhone

On Jun 13, 2018, at 4:57 PM, Naomi Henderson ***@***.***> wrote: @mrocklin fantastic! So far so good, xesmf is happily calculating the weight files and reusing them when appropriate. Thanks much — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

@naomi-henderson

Glad to see this! @naomi-henderson could you share your notebook (I still cannot access it)? Let me try to tweak it to support distributed.

@JiaweiZhuang

That would be great, @JiaweiZhuang ! The notebook runs fine (under 3 minutes) on my linux box, but unfortunately gets bogged down at the beginning when reading the zarr data on pangeo.pydata.org. The xesmf regridding is so fast in comparison that I haven't had a chance to check out how the workers are handling this part. (It is monthly data chunked (12,ny,nx) in (time,lat,lon) - is that the problem?) If you scroll down to the bottom of each gist notebook you will see that it takes 4x as long using 10workers/20cores on pangeo.pydata.org as on my linux box (where the zarr-ed data is on an internal drive) with 2 threads.

If anyone has words of wisdom on the data reading/ startup delay issues, I am very open to suggestions and would very much like to get this working properly.

Here is my notebook (with output) configured/run on linux box

and here is the same notebook (with output) configured/run on pangeo.pydata.org

@martindurant

cc @martindurant , putting this on your backlog if I may. I suspect that you'll enjoy diving into performance issues here.

I am just heading out for a weekend away, but if no one beats me to it, I can look early next week.
My immediate thoughts are, that GCS file-listing and reading metadata must still be done from the client process, and there may be many of these - that's the most likely slow-down in the initial stage.

# ~time to list all files
%time data_fs.du(base_path, True, True)
CPU times: user 2.33 s, sys: 171 ms, total: 2.51 s
Wall time: 25.5 s
12336380536

# size in GB
_ / 2**30
11.489149682223797

# number of files
len(data_fs.walk(base_path))
23832

Can you estimate the per-node data throughput, perhaps by doing a trivial operation such as sum(), versus your local disc? Network data rates >100MB/s are typical, but your disc may be orders of magnitude faster. There may be an opportunity to eat the cost just once if the data can be persisted.

Thanks for looking at this. I will be gone for the next week, but will do some simple benchmarking of GCS vs my local disc when I return. Thanks for showing how to get the time to list all files and get the number of files - very useful!

I'm curious about the practice of listing files from GCS. Are there ways that we might be able to get around this? I would expect this to become an increasingly large issue as our datasets increase in size. I've heard from others that you should avoid listing buckets generally.

…

On Fri, Jun 22, 2018 at 1:42 PM, Naomi Henderson ***@***.***> wrote: Thanks for looking at this. I will be gone for the next week, but will do some simple benchmarking of GCS vs my local disc when I return. Thanks for showing how to get the time to list all files and get the number of files - very useful! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#309 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszH-ZBIV7LzVHjtF2axC6pN2v-WByks5t_SyXgaJpZM4UlQjI> .

Quite some time ago, now, we realised that it simply took far too long to list all the files in a bucket for data-heavy buckets, and we moved to per-directory listing. This means, though, a new call for every directory, and many calls for a deeply nested set of directories, such as zarr CDF structures seem to have. The following is a set of exists calls seen for accessing one of the standard pangeo datastets. There are repeats in the list, and many calls here are for free, since listings are cached; however, apparent non-existent directories are also tried, the full tree is interrogated up-front, and every single .zarray and .zgroup are opened and read, all small files (often < 1kB) but requiring a new HTTP call every time, and all these happen in serial

exists(args=('pangeo-data/newman-met-ensemble/.zarray',), kwargs={})
exists(args=('pangeo-data/newman-met-ensemble/.zgroup',), kwargs={})
exists(args=('pangeo-data/newman-met-ensemble/.zarray',), kwargs={})
exists(args=('pangeo-data/newman-met-ensemble/.zattrs/.zarray',), kwargs={})
exists(args=('pangeo-data/newman-met-ensemble/.zgroup/.zarray',), kwargs={})
exists(args=('pangeo-data/newman-met-ensemble/elevation/.zarray',), kwargs={})
exists(args=('pangeo-data/newman-met-ensemble/elevation/.zarray',), kwargs={})
exists(args=('pangeo-data/newman-met-ensemble/lat/.zarray',), kwargs={})
exists(args=('pangeo-data/newman-met-ensemble/lat/.zarray',), kwargs={})
exists(args=('pangeo-data/newman-met-ensemble/lon/.zarray',), kwargs={})
exists(args=('pangeo-data/newman-met-ensemble/lon/.zarray',), kwargs={})
exists(args=('pangeo-data/newman-met-ensemble/mask/.zarray',), kwargs={})
exists(args=('pangeo-data/newman-met-ensemble/mask/.zarray',), kwargs={})
exists(args=('pangeo-data/newman-met-ensemble/pcp/.zarray',), kwargs={})
exists(args=('pangeo-data/newman-met-ensemble/pcp/.zarray',), kwargs={})
exists(args=('pangeo-data/newman-met-ensemble/t_max/.zarray',), kwargs={})
exists(args=('pangeo-data/newman-met-ensemble/t_max/.zarray',), kwargs={})
exists(args=('pangeo-data/newman-met-ensemble/t_mean/.zarray',), kwargs={})
exists(args=('pangeo-data/newman-met-ensemble/t_mean/.zarray',), kwargs={})
exists(args=('pangeo-data/newman-met-ensemble/t_min/.zarray',), kwargs={})
exists(args=('pangeo-data/newman-met-ensemble/t_min/.zarray',), kwargs={})
exists(args=('pangeo-data/newman-met-ensemble/t_range/.zarray',), kwargs={})
exists(args=('pangeo-data/newman-met-ensemble/t_range/.zarray',), kwargs={})
exists(args=('pangeo-data/newman-met-ensemble/time/.zarray',), kwargs={})
exists(args=('pangeo-data/newman-met-ensemble/time/.zarray',), kwargs={})
exists(args=('pangeo-data/newman-met-ensemble/time/.zarray',), kwargs={})
exists(args=('pangeo-data/newman-met-ensemble/time/.zarray',), kwargs={})
exists(args=('pangeo-data/newman-met-ensemble/lat/.zarray',), kwargs={})
exists(args=('pangeo-data/newman-met-ensemble/lon/.zarray',), kwargs={})
exists(args=('pangeo-data/newman-met-ensemble/time/.zarray',), kwargs={})
exists(args=('pangeo-data/newman-met-ensemble/.zarray',), kwargs={})
exists(args=('pangeo-data/newman-met-ensemble/.zgroup',), kwargs={})
exists(args=('pangeo-data/newman-met-ensemble/.zarray',), kwargs={})
exists(args=('pangeo-data/newman-met-ensemble/_ipython_canary_method_should_not_exist_/.zarray',), kwargs={})
exists(args=('pangeo-data/newman-met-ensemble/_ipython_canary_method_should_not_exist_/.zgroup',), kwargs={})
exists(args=('pangeo-data/newman-met-ensemble/_repr_mimebundle_/.zarray',), kwargs={})
exists(args=('pangeo-data/newman-met-ensemble/_repr_mimebundle_/.zgroup',), kwargs={})

(note that I have no idea what the _ipython_canary.. or _repr_mimebundle_ are, they do not appear in any file listing as far as I can tell)

All of this aside, the general rule is still to have data chunks which are large enough that the download outweighs overheads from filesystem operations: for newmann there is 120GB in 1930 files, for CMIP5-ts 11GB in 23800 files

@alimanfoo

@alimanfoo, the zarr storage of cdf-like nested group data allows access at any level in the hierarchy, which is very convenient, but I wonder is there any possibility of consolidating the group and attr files such that so many list and open operations would not be required at metadata inference time?

@naomi-henderson

@naomi-henderson if you pip install again all should be well

@martindurant

zarr-2.2.1.dev6 it is! thanks @martindurant and @mrocklin

@martindurant

@martindurant , perhaps I don't quite understand the process. If I already have the zarr data, then I should be able to run consolidate_metadata and it should add a file called .zmetadata
Here I can open the dataset, but then get an error - what am I missing?

group = 'zarr/CMIP5-ts/MRI-CGCM3/rcp85'
#xr.open_zarr(group)
zarr.convenience.consolidate_metadata(group)

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-27-90161f14a048> in <module>()
      1 group = 'zarr/CMIP5-ts/MRI-CGCM3/rcp85'
      2 xr.open_zarr(group)
----> 3 zarr.convenience.consolidate_metadata(group)

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/zarr/convenience.py in consolidate_metadata(mapping, out_key)
   1090 
   1091     out = {key: mapping[key].decode() for key in mapping if is_zarr_key(key)}
-> 1092     mapping[out_key] = json.dumps(out).encode()

TypeError: 'str' object does not support item assignment

I should have been clearer - the function works on a mapping, so for local paths you would do

from zarr.storage import DirectoryStore
d = DirectoryStore('zarr/CMIP5-ts/MRI-CGCM3/rcp85')
consolidate_metadata(d)

data = xr.open_zarr('zarr/CMIP5-ts/MRI-CGCM3/rcp85')  # or reuse d here

for working directly on GCS

gcs = gcsfs.GCSFileSystem()
mapping = gcsfs.GCSMap(gcs=gcs, root='bucket/path.zarr')
consolidate_metadata(mapping)

data = xr.open_zarr(mapping)

@martindurant

thanks @martindurant , it is working like a charm.

With consolidated metadata:

%%time

gcs = gcsfs.GCSFileSystem(token='anon')
mapper = gcsfs.GCSMap(gcs=gcs, root='pangeo-data/CMIP5-ts/meta/CanESM2/rcp45/')
d = xr.open_zarr(mapper)

CPU times: user 31.2 ms, sys: 3.16 ms, total: 34.4 ms
Wall time: 739 ms

Original data

%%time

gcs = gcsfs.GCSFileSystem(token='anon')
mapper = gcsfs.GCSMap(gcs=gcs, root='pangeo-data/CMIP5-ts/20mb/CanESM2/rcp45/')
d = xr.open_zarr(mapper)

CPU times: user 85.7 ms, sys: 7.01 ms, total: 92.8 ms
Wall time: 2.4 s

@martindurant

Wow, @martindurant , I can't wait to use this on pangeo.pydata.org ! I can't start a server this afternoon for some reason, though.

Working directly on GCS from my Columbia linux server I see a factor of 4 improvement on opening the 49 historical model simulations: ~4 min (original zarr) and ~1 min (consolidated metadata). The first reduce step (annual means), which touches all of the data, takes about the same time for original and consolidated.

For now I can use your zarr to fix the data, thanks!

@naomi-henderson

Will it scatter the weight matrix to the workers?

No, currently it will cast distributed dask arrays to in-memory numpy arrays. In @naomi-henderson's example, the original data were reduced significantly (taking time average) before passing to xESMF. At that stage there's no need of distributed computing at all.

If the workflow is to reduce a large mount of data and finally make some 1D&2D plots, I'd argue that distributed regridding is rarely needed. It is much better to take average before regridding. Distributed regridding would only be useful for read-regrid-write workflow, to pre-process large volumes of data for later analysis. See more at #334.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

We should follow up on this discussion. Has their been any progress on either the xesmf side or the zarr side that brings us closer to making this use case work well?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

This issue has been automatically closed because it had not seen recent activity. The issue can always be reopened at a later date.

Navigation Menu

Repository navigation

xesmf included in pangeo.pydata.org ? #309

38 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

xesmf included in pangeo.pydata.org ? #309

Description

Activity

rabernat commented on Jun 13, 2018

naomi-henderson commented on Jun 13, 2018

mrocklin commented on Jun 13, 2018

naomi-henderson commented on Jun 13, 2018

rabernat commented on Jun 13, 2018

JiaweiZhuang commented on Jun 14, 2018

naomi-henderson commented on Jun 14, 2018

mrocklin commented on Jun 22, 2018

martindurant commented on Jun 22, 2018

naomi-henderson commented on Jun 22, 2018

mrocklin commented on Jun 22, 2018

martindurant commented on Jun 25, 2018

martindurant commented on Jun 25, 2018

38 remaining items

mrocklin commented on Jul 2, 2018

naomi-henderson commented on Jul 2, 2018

naomi-henderson commented on Jul 3, 2018

martindurant commented on Jul 3, 2018

naomi-henderson commented on Jul 3, 2018

naomi-henderson commented on Jul 3, 2018

JiaweiZhuang commented on Jul 10, 2018

stale commented on Sep 8, 2018

rabernat commented on Sep 8, 2018

stale commented on Nov 7, 2018

stale commented on Nov 14, 2018

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions