New plot_type 1d_profile in monitor #3178

FranziskaWinterstein · 2023-05-16T11:18:45Z

Description

This implements a new one dimensional profile plot option '1d_profile' in the diagnostic monitor/multi_datasets.py. It allows one dimensional preprocessed data with only dimension height or pressure and plots all datasets in one figure. It is possible to plot the absolute values or differences. The latter as absolute differences or relative differences.

To prevent misunderstanding the former plot_type 'profile' is renamed to 'zonal_mean_profile'. Backward compatibility is given until the option 'profile' can be removed in future versions.

Example:

Before you get started

☝ Create an issue to discuss what you are going to do

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

🛠 This pull request has a descriptive title
🛠 Code is written according to the code quality guidelines
🛠 Documentation is available
🛠 Tests run successfully
🛠 The list of authors is up to date
🛠 All checks below this pull request were successful

New or updated recipe/diagnostic

🧪 Recipe runs successfully
🧪 Figure(s) and data look as expected from literature
🛠 Provenance information has been added

…al_mean_profile

schlunma · 2023-05-16T16:19:22Z

Hi @FranziskaWinterstein, thanks for this, this looks great!

I will try to review this soon. In the meantime, could I ask you to add an example usage of this to https://github.com/ESMValGroup/ESMValTool/blob/main/esmvaltool/recipes/monitor/recipe_monitor_with_refs.yml and an example plot to the bottom https://github.com/ESMValGroup/ESMValTool/blob/main/doc/sphinx/source/recipes/recipe_monitor.rst? That would be awesome, thanks!!

schlunma

I just had a brief look at the code, looks very good already!! 👍

My two main comments:

At the very beginning of the diagnostic (in the __init__ method), you can add a check for plot_type == 'profile' and replace this if necessary with
```
self.plots['zonal_mean_profile'] = self.plots.pop('profile')
```
Then you can basically remove all further appearances of profile in the remaining code, which makes it much simpler.
I would recommend to not add a separate plot_bias option here but rather use the bias preprocessor in the recipe instead and then plot the output of this simply with 1d_profile. That will make your code here much simpler.

Thanks!! Let me know if you have any questions/comments!

esmvaltool/diag_scripts/monitor/multi_datasets.py

Co-authored-by: Manuel Schlund <[email protected]>

valeriupredoi · 2023-05-25T11:10:29Z

cheers both you guys! I can have a final look and merge when review's over and passed, remember to fix the flake issues and also please fix the PR description boxes (remove or strikethrough the ones that are not relevant, check mark the ones that are), yeah sorry, I sound like an accountant 😁

Co-authored-by: Manuel Schlund <[email protected]>

Co-authored-by: valeriupredoi <[email protected]>

FranziskaWinterstein · 2023-05-25T14:55:10Z

So, finally I think I addressed all comments. @schlunma would you be so kind as to review the changes again? There are still some codacy issues - can these be ignored?

schlunma

Thanks, this is almost ready 👍 Just got a couple of minor comments below.

Could I also ask you to add an example usage of this to https://github.com/ESMValGroup/ESMValTool/blob/main/esmvaltool/recipes/monitor/recipe_monitor_with_refs.yml and an example plot to the bottom https://github.com/ESMValGroup/ESMValTool/blob/main/doc/sphinx/source/recipes/recipe_monitor.rst? It would also be good to modify profile -> zonal_mean_profile there.

Alternatively, could you post your recipe here? Then I can try to include a similar diagnostic into our recipe. Thanks!!

esmvaltool/diag_scripts/monitor/multi_datasets.py

Co-authored-by: Manuel Schlund <[email protected]>

…e bot

schlunma · 2023-06-07T14:34:09Z

@esmvalbot please run monitor/recipe_monitor_with_refs.yml

esmvalbot · 2023-06-07T14:34:13Z

Since @schlunma asked, ESMValBot will run recipe monitor/recipe_monitor_with_refs.yml as soon as possible, output will be generated here

valeriupredoi · 2023-06-07T14:48:17Z

Since @schlunma asked, ESMValBot will run recipe monitor/recipe_monitor_with_refs.yml as soon as possible, output will be generated here

yeah bot's on the ropes today, possibly due to Lustre FS issues at DKRZ, RIP poor bot 🤖

schlunma

I added the new plot type to our example recipe and update the doc, this should be ready now!

Thanks @FranziskaWinterstein again for this great addition!

@valeriupredoi would you do us the honor and perform a final check and merge please? I ran the recipe locally to produce the new plots for the doc, everything went well 🚀

valeriupredoi

looking great, beautiful plots too, well done you two! @schlunma would you want to add Franziska as an author to esmvaltool/recipes/monitor/recipe_monitor_with_refs.yml? If not, I can merge right now 👍

schlunma · 2023-06-07T15:25:36Z

looking great, beautiful plots too, well done you two! @schlunma would you want to add Franziska as an author to esmvaltool/recipes/monitor/recipe_monitor_with_refs.yml? If not, I can merge right now +1

Good point, just did that 👍

valeriupredoi · 2023-06-07T15:37:30Z

great work here, guys! 🍺 x2

esmvalbot · 2023-06-08T06:34:38Z

ESMValBot is sorry to report it failed to run recipe monitor/recipe_monitor_with_refs.yml: exit is 1, output has been generated here

FranziskaWinterstein · 2023-06-09T06:30:08Z

Thank you, too for reviewing and merging! =))

…old_and_clone_task_rtw * recipe_test_workflow_prototype: (199 commits) #3169: Upgrade the RTW to work with ESMValTool v2.8.0 [Condalock] Update Linux condalock file (#3237) Modified links to the tutorial (#3236) Add ESMValCore release `v2.8.1` into the documentation (#3235) Generate climatology on the fly for AutoAssess soil moisture (#3197) New recipe and diagnostic for Arctic-midlatitude research (#3021) Fixed pandas diagnostics for pandas>=2.0.0 (#3209) Update obs4MIPs dataset to the current naming scheme in recipe_smpi.yml (#2991) Add variable long names to provenance record in monitoring diagnostics (#3222) Extension of NASA MERRA2 CMORizer (cl, cli, clivi, clw, clwvi) (#3167) Remove "fx_variable" from recipe_wenzel14jgr.yml (#3212) [Condalock] Update Linux condalock file (#3217) Add Seaborn diagnostic (#3155) Remove fx_variables from ipccwg1ar5ch9 recipes (#3215) Remove "fx_variable" from recipe_tebaldi21esd.yml (#3211) Update recipe_impact.yml to work with newer versions of `pandas` (#3220) Use ESMValCore v2.9.0 release candidates (#3219) [Github Actions ] Check if python minor version changed after Julia install in development installation test (#3213) New plot_type 1d_profile in monitor (#3178) Add support for using a dask distributed scheduler (#3151) ...

Co-authored-by: Manuel Schlund <[email protected]> Co-authored-by: Valeriu Predoi <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: valeriupredoi <[email protected]> Co-authored-by: Manuel Schlund <[email protected]>

FranziskaWinterstein added 7 commits April 24, 2023 22:34

included 1d_profiles and renamed previous profile consequently to zon…

46b0506

…al_mean_profile

aspect ratio and gridlines included

7f00b60

aspect ratio can be set by namelist

ca8d2d4

aspect ratio can be set by namelist

f14c365

code cleanup

a7dde6c

bugfix

fde976c

backward compatibility

04b7002

FranziskaWinterstein added enhancement diagnostic labels May 16, 2023

FranziskaWinterstein requested a review from schlunma May 16, 2023 11:18

Merge branch 'main' into monitor_1d_profile

6e5bde7

schlunma added this to the v2.9.0 milestone May 16, 2023

schlunma reviewed May 22, 2023

View reviewed changes

FranziskaWinterstein and others added 7 commits May 25, 2023 12:57

Update esmvaltool/diag_scripts/monitor/multi_datasets.py

3196518

Co-authored-by: Manuel Schlund <[email protected]>

Update esmvaltool/diag_scripts/monitor/multi_datasets.py

ce2ea14

Co-authored-by: Manuel Schlund <[email protected]>

Update esmvaltool/diag_scripts/monitor/multi_datasets.py

4240e14

Co-authored-by: Manuel Schlund <[email protected]>

Update esmvaltool/diag_scripts/monitor/multi_datasets.py

0ef829f

Co-authored-by: Manuel Schlund <[email protected]>

Update esmvaltool/diag_scripts/monitor/multi_datasets.py

fc28846

Co-authored-by: Manuel Schlund <[email protected]>

Update esmvaltool/diag_scripts/monitor/multi_datasets.py

310e260

Co-authored-by: Manuel Schlund <[email protected]>

Update esmvaltool/diag_scripts/monitor/multi_datasets.py

7d8446f

Co-authored-by: Manuel Schlund <[email protected]>

FranziskaWinterstein and others added 8 commits May 25, 2023 13:14

Update esmvaltool/diag_scripts/monitor/multi_datasets.py

0d42d97

Co-authored-by: Manuel Schlund <[email protected]>

Update esmvaltool/diag_scripts/monitor/multi_datasets.py

8bc674e

Co-authored-by: Manuel Schlund <[email protected]>

formatted minor ticks on log x axis

d01b381

Pin numpy !=1.24.3 due to severe masking bug (#3182)

a2898a2

[Condalock] Update Linux condalock file (#3186)

42e47b9

Co-authored-by: valeriupredoi <[email protected]>

included handling of plot_type and corresponding warning

7eda394

Merge branch 'main' into monitor_1d_profile

2925208

changed call of _get_gridline_kwargs and removed all specific parts

99ec16e

code cleanup

911c724

schlunma reviewed May 25, 2023

View reviewed changes

FranziskaWinterstein and others added 11 commits May 25, 2023 17:51

Update esmvaltool/diag_scripts/monitor/multi_datasets.py

c400264

Co-authored-by: Manuel Schlund <[email protected]>

Update esmvaltool/diag_scripts/monitor/multi_datasets.py

b57cfa1

Co-authored-by: Manuel Schlund <[email protected]>

options are in alphabetic order and complete

ae56c27

Merge branch 'main' into monitor_1d_profile

2c23492

Removed unused option

128974a

Explicitly set default options for each plot type separately

f400997

Allow using gridline_kwargs for other XY plots

c2de356

Replaced ERA5 dataset with CMIP6 to be able to run the recipe with th…

031fa46

…e bot

Replaced profile -> zonal_mean_profile in the recipe

0596fb6

Add 1D profile to recipe

7fe1e8b

Updated doc with new plots

f1f281b

schlunma self-requested a review June 7, 2023 15:00

schlunma approved these changes Jun 7, 2023

View reviewed changes

valeriupredoi approved these changes Jun 7, 2023

View reviewed changes

Add Franzsika as recipe author

8e6c1ea

valeriupredoi merged commit 54ace22 into main Jun 7, 2023

valeriupredoi deleted the monitor_1d_profile branch June 7, 2023 15:37

bouweandela mentioned this pull request Jun 30, 2023

Recipe testing and output comparison for release 2.9.0 #3239

Closed

New plot_type 1d_profile in monitor #3178

New plot_type 1d_profile in monitor #3178

Uh oh!

Conversation

FranziskaWinterstein commented May 16, 2023 • edited by schlunma Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Before you get started

Checklist

New or updated recipe/diagnostic

Uh oh!

schlunma commented May 16, 2023

Uh oh!

schlunma left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

valeriupredoi commented May 25, 2023

Uh oh!

FranziskaWinterstein commented May 25, 2023

Uh oh!

schlunma left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

schlunma commented Jun 7, 2023

Uh oh!

esmvalbot bot commented Jun 7, 2023

Uh oh!

valeriupredoi commented Jun 7, 2023

Uh oh!

schlunma left a comment

Choose a reason for hiding this comment

Uh oh!

valeriupredoi left a comment

Choose a reason for hiding this comment

Uh oh!

schlunma commented Jun 7, 2023

Uh oh!

valeriupredoi commented Jun 7, 2023

Uh oh!

esmvalbot bot commented Jun 8, 2023

Uh oh!

FranziskaWinterstein commented Jun 9, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

FranziskaWinterstein commented May 16, 2023 •

edited by schlunma

Loading