Implement a new tropical cyclone rain model by tovogt · Pull Request #85 · CLIMADA-project/climada_petals

tovogt · 2023-07-05T17:23:46Z

Changes proposed in this PR:

This implements a new TC rainfall model (called TCR) as part of the climada_petals.hazard.tc_rainfield module.
The existing implementation (of R-CLIPER) is refactored a bit, with improved documentation, and it is also covered by the tutorial notebook.
Apart from the tutorial notebook (~1 MB), this adds three further binary files to the repository:
1. climada_petals/data/tc_rainfield/tcrain_examples.nc (80 KB): Two TC tracks with additional variables extracted from ERA5. The data is used in the tutorial notebook as well as in the unit tests.
2. climada_petals/data/tc_rainfield/c_drag_500.tif (~1 MB): A global gridded data set of the surface drag coefficient for gradient winds (from ERA5). The resolution is sufficient for almost all applications of the model, and it's still very small.
3. climada_petals/data/tc_rainfield/topography_land_360as.tif (~3 MB): A global gridded data set of surface elevation at 0.1 degree resolution (derived from SRTM). The resolution is fine for applications of TCR, but is too low for many other applications, such as surge or flood simulations. It's public domain and comparably small, so I think we can distribute it with CLIMADA.

As of July 6, this PR is ready to be reviewed. I'm looking forward to your feedback!

PR Author Checklist

PR Reviewer Checklist

climada_petals/hazard/tc_rainfield.py

tovogt · 2023-07-06T13:15:41Z

Hey everyone, the new rain module is ready for review!

Please note that running the notebook and the tests will require data sets that are included in the ./climada_petals/data/tc_rainfield/ directory of the repository. If you struggle with error messages about missing data files, you might need to copy those files to the CLIMADA data and config directory on your computer.

Furthermore, the implementation is based on a proprietary MATLAB implementation. If you would like to have a look at the original code, please drop me a message, and I can send you the MATLAB source code via mail.

climada_petals/hazard/tc_rainfield.py

climada_petals/hazard/test/test_tc_rainfield.py

# Conflicts: # climada_petals/hazard/tc_rainfield.py

aleeciu · 2023-09-29T13:03:54Z

climada_petals/hazard/tc_rainfield.py

+        The implementation of the R-CLIPER model currently does not allow modifications, so that
+        `model_kwargs` is ignored with `model="R-CLIPER"`. While the TCR model can be configured in
+        several ways, it is usually safe to go with the default settings. Here is the complete list
+        of `model_kwargs` and their meaning with `model="TCR"` (in alphabetical order):


I'd expect the explanation of what model_kwargs is about in the function's parameters description just below, under model_kwargs.

Good point, I moved the description accordingly.

aleeciu · 2023-09-29T13:05:18Z

climada_petals/hazard/tc_rainfield.py

+            Path to a GeoTIFF file containing digital elevation model data (in m). If not
+            specified, a topography at 0.1 degree resolution provided with CLIMADA is used.
+            Default: None
+        matlab_ref_mode : bool, optional


Why do we need this. If you are sure it's a bug, it's better if it's gone, no? I'd rather add in the docstring that this is the original implementation, with a bug fix, and explain what the bug fix is.

I agree that bugs should just be fixed if we are sure that they are bugs. I updated the description of the parameter.

Background: When I originally implemented this, I found quite a few strange "issues" in the MATLAB code. And it took some time to write them down and get feedback from the original authors about these "issues". I didn't want to remove the issues unless the original authors definitely confirm that they are unintended ("bugs"). When I wrote this docstring I still didn't have feedback from the authors. Today, the situation is different: I know that some of the issues really were bugs. Those bugs are now fixed in my implementation and the original authors also decided to fix them in their implementation. But some of the "issues" were actually deliberate decisions by the original authors that I, personally, don't like (for example, using a parameter value that is not documented anywhere in the literature, not even in their own papers), and, at the same time, only make a comparably small difference. For testing purposes, I still think that it's useful to be able to replicate the behavior of the reference implementation. That's why I keep this parameter, but make clear what it actually does.

aleeciu · 2023-09-29T13:22:29Z

Thanks @tovogt , I screened through the code and ran the tutorial. I cannot check it thoroughly but I wanted to provide a user type of feedback.

Code: It looks great, I just had a couple of documentation-related comments.

Tutorial: I could run it but I had to specifiy the paths of the required additional files (c_drag_500, topography_land_360 and tcrain_examples) via the model_kwargs arguments as my machine was looking for data under the 'Users/user_name/climada/...' folder instead of the CLIMADA project's folder.

I am not completely sure, but my understanding is that 'Users/user_name/climada/...' is where these data are supposed to be by default? Anyway, beside that, it ran flawlessly and the text is also very clear with a good balance of explanation and links for further knowledge.

A couple of more points :

What is the origin of topo data? would be great to know (haven't seen that, but perhaps is there)
It seems this new model's results are waaay different than R-CLIPER, should we just kill R-CLIPER for good or we still want to keep both?
My understanding is that this code is not applicable to the Knutson-scaled future TCs, right? But what about scaling the generated rainfields with knutson's rain rate scaling factors? would that be sensible ?

tovogt · 2023-10-10T11:04:41Z

Thanks for your feedback!

Well spotted, I forgot to add the data files to the REPO_DATA in ./climada_petals/__init__.py. This should be fixed now and the files will be automatically copied over to the right locations during import.
The topography data is the one from the MATLAB reference implementation. which is just SRTM upscaled to 0.1 degree resolution. The advantage of this data set is that it is extremely small (~3 MB) so that it can be included in the CLIMADA repository. Since SRTM is public domain, there shouldn't be a licensing issue.
I would keep R-CLIPER since it's really very simple code (not hard to maintain) and nice for future reference.
I haven't worked with the Knutson scaling that is currently implemented in CLIMADA, so I cannot really comment on that.

EDIT: I was finally able to reproduce the topography file from SRTM using gdalwarp and the "average" interpolation option. This is now also mentioned in several docstrings:

climada_petals/climada_petals/hazard/tc_rainfield.py

Lines 111 to 115 in d649a5a

    
           DEF_ELEVATION_TIF = u_const.SYSTEM_DIR / "topography_land_360as.tif" 
        
           """Topography (land surface elevation, 0 over oceans) raster data at 0.1 degree resolution 
        
           SRTM data upscaled to 0.1 degree resolution using the "average" method of gdalwarp. 
        
           """

aleeciu · 2023-10-10T14:59:52Z

Thanks for your feedback!

Well spotted, I forgot to add the data files to the REPO_DATA in ./climada_petals/__init__.py. This should be fixed now and the files will be automatically copied over to the right locations during import.

The topography data is the one from the MATLAB reference implementation. which is just SRTM upscaled to 0.1 degree resolution. The advantage of this data set is that it is extremely small (~3 MB) so that it can be included in the CLIMADA repository. Since SRTM is public domain, there shouldn't be a licensing issue.

I would keep R-CLIPER since it's really very simple code (not hard to maintain) and nice for future reference.

I haven't worked with the Knutson scaling that is currently implemented in CLIMADA, so I cannot really comment on that.

EDIT: I was finally able to reproduce the topography file from SRTM using gdalwarp and the "average" interpolation option. This is now also mentioned in several docstrings:

climada_petals/climada_petals/hazard/tc_rainfield.py

Lines 111 to 115 in d649a5a

DEF_ELEVATION_TIF = u_const.SYSTEM_DIR / "topography_land_360as.tif"

"""Topography (land surface elevation, 0 over oceans) raster data at 0.1 degree resolution

SRTM data upscaled to 0.1 degree resolution using the "average" method of gdalwarp.

"""

Super ! thanks for addressing these points and especially for figuring out where the elevation data come from. All good for me regarding this PR.

emanuel-schmid · 2023-11-08T12:17:46Z

I'm a wee bit uneasy about the data files being part of the repository. Alternatively we could get them from the data-api, maybe?

tovogt · 2023-11-08T13:35:07Z

Yes, sounds great! Please go ahead and copy the files to the data-api and tell me how I can access them from within the code, or just push your proposed changes.

…equivalents downloaded from the data-api

Thomas Vogt added 8 commits July 3, 2023 18:06

TCRain: refactor and use util functions

48ddba0

TCRain: implement TCR model

181906e

tc_rain: typos

e0e0208

tc_rain: cleanup

508a984

tc_rain: cleanup

d70eda1

tc_rain: cleanup

afe27c8

tc_rain: fix store_rainrates

2f2d8fc

tc_rainfield: adapt names of imports from trop_cyclone

76889e8

tovogt requested review from aleeciu and peanutfun July 5, 2023 17:24

tovogt mentioned this pull request Jul 5, 2023

Refactor TropCyclone for external use in other modules CLIMADA-project/climada_python#749

Merged

13 tasks

bguillod reviewed Jul 6, 2023

View reviewed changes

climada_petals/hazard/tc_rainfield.py Show resolved Hide resolved

Thomas Vogt added 7 commits July 6, 2023 09:51

tc_rainfield: improve climada_petals/hazard/tc_rainfield.py

1e654d8

tc_rainfield: add tutorial notebook

001e4f7

tc_rainfield: smooth gradient of raster data

98bb4d8

tc_rainfield: provide default topography and cdrag files with CLIMADA

d547d4e

tc_rainfield: example track data for tutorial notebook

5354490

tc_rainfield: docstrings

3f55c5c

tc_rainfield: more unit tests

6314c2c

tovogt marked this pull request as ready for review July 6, 2023 13:12

Thomas Vogt added 9 commits July 6, 2023 17:11

tc_rainfield: fix _qs_from_t_diff_level

91e1b25

tc_rainfield: _r_from_t_same_level

3feb0c8

TCRain: simplify chunking

74a9f04

TCRain: list used si_track data variables in docstrings

a80e0dd

tc_rainfield: comments

96215dc

tc_rainfield: use different gradient-to-surface wind factor

c77b57c

tc_rainfield: fix when using pool

610ec3c

tc_rainfield: adjust chunking

7190e81

tc_rainfield: fix chunking

fc2b4bb

Thomas Vogt added 3 commits July 17, 2023 09:16

tc_rainfield: update TCR according to MATLAB reference

a5975be

tc_rain: update tutorial

98b18bc

Update CHANGELOG

9ecd497

tovogt commented Jul 17, 2023

View reviewed changes

climada_petals/hazard/tc_rainfield.py Show resolved Hide resolved

tovogt commented Jul 17, 2023

View reviewed changes

climada_petals/hazard/tc_rainfield.py Show resolved Hide resolved

tovogt commented Jul 17, 2023

View reviewed changes

climada_petals/hazard/test/test_tc_rainfield.py Show resolved Hide resolved

tovogt commented Jul 17, 2023

View reviewed changes

climada_petals/hazard/test/test_tc_rainfield.py Show resolved Hide resolved

Thomas Vogt and others added 6 commits July 17, 2023 16:28

Mention TCRain tutorial in overview

a409054

Merge branch 'develop' into feature/tc_rain

183daf0

tc_rain: unused arguments

2d55d79

Merge branch 'develop' into feature/tc_rain

b34675b

# Conflicts: # climada_petals/hazard/tc_rainfield.py

tc_rainfield: get rid of Tag

b9ad11a

Merge branch 'develop' into feature/tc_rain

d98b9bb

aleeciu reviewed Sep 29, 2023

View reviewed changes

Thomas Vogt added 2 commits October 10, 2023 11:17

Merge branch 'develop' into feature/tc_rain

99a4546

tc_rainfield: improve docstrings

20362eb

Thomas Vogt added 3 commits October 10, 2023 13:05

tc_rainfield: include files in REPO_DATA

94459da

tc_rainfield: mention source of topo data

d649a5a

tc_rainfield: source of topo data in all docstrings

fe7c65f

Merge branch 'develop' into feature/tc_rain

283f1e1

emanuel-schmid and others added 2 commits November 17, 2023 16:26

suggestion for replacing climada_petals/data/tc_fainfield files with …

3450047

…equivalents downloaded from the data-api

Merge branch 'develop' into feature/tc_rain

d0c1e77

tovogt merged commit 209cc23 into develop Dec 12, 2023

emanuel-schmid deleted the feature/tc_rain branch December 13, 2023 13:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement a new tropical cyclone rain model#85

Implement a new tropical cyclone rain model#85
tovogt merged 45 commits intodevelopfrom
feature/tc_rain

tovogt commented Jul 5, 2023 •

edited

Loading

Uh oh!

Uh oh!

tovogt commented Jul 6, 2023 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aleeciu Sep 29, 2023

Uh oh!

tovogt Oct 10, 2023

Uh oh!

aleeciu Sep 29, 2023

Uh oh!

tovogt Oct 10, 2023

Uh oh!

aleeciu commented Sep 29, 2023 •

edited

Loading

Uh oh!

tovogt commented Oct 10, 2023 •

edited

Loading

Uh oh!

aleeciu commented Oct 10, 2023

Uh oh!

emanuel-schmid commented Nov 8, 2023

Uh oh!

tovogt commented Nov 8, 2023 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

tovogt commented Jul 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Author Checklist

PR Reviewer Checklist

Uh oh!

Uh oh!

tovogt commented Jul 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aleeciu Sep 29, 2023

Choose a reason for hiding this comment

Uh oh!

tovogt Oct 10, 2023

Choose a reason for hiding this comment

Uh oh!

aleeciu Sep 29, 2023

Choose a reason for hiding this comment

Uh oh!

tovogt Oct 10, 2023

Choose a reason for hiding this comment

Uh oh!

aleeciu commented Sep 29, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tovogt commented Oct 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aleeciu commented Oct 10, 2023

Uh oh!

emanuel-schmid commented Nov 8, 2023

Uh oh!

tovogt commented Nov 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

tovogt commented Jul 5, 2023 •

edited

Loading

tovogt commented Jul 6, 2023 •

edited

Loading

aleeciu commented Sep 29, 2023 •

edited

Loading

tovogt commented Oct 10, 2023 •

edited

Loading

tovogt commented Nov 8, 2023 •

edited

Loading