Skip to content

Add scorecard section to scripts - #530

Merged
AUdaltsova merged 10 commits into
mainfrom
scorecard
Jun 12, 2026
Merged

Add scorecard section to scripts#530
AUdaltsova merged 10 commits into
mainfrom
scorecard

Conversation

@AUdaltsova

@AUdaltsova AUdaltsova commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Pull Request

Description

This PR adds a new folder to scripts section which houses a script to generate the scorecard and an HTML template for it.

This is the "v0" of the scorecard. It will only calculate the metrics for the "national" forecasts, so it's not suitable for the sites pipeline! I have included the following metrics/plots:

  1. t0 availability by hour and date
  2. NMAE: by forecast horizon, averaged over forecast horizons, by month, comparative % improvement between models
  3. Quantile calibration plot

I have left in some paths as an example in the script. For this configuration (2 backtest files, each for a year of data), the script take 2-3 minutes to run.

Potentially controversial choices:

  1. Table contents get printed out. This is in case the template fails to compile, as this data will be lost otherwise. It's not necessarily a good/clean way of having a back-up, but I've ran into the final rendering failing many times, especially when editing the template, and it can be quite frustrating to wait all this time and not see the result. (.png of all plots also get saved but I don't think it's controversial, also lets you actually reuse plots instead of taking screenshots of the pdf)
  2. There is a progress bar which is largely fiction, as different steps take up wildly different amounts of time (e g NMAE and quantile calibration section together probably take up ~60% of the runtime). I like having at least some idea on how the script is progressing, which is why I kept it in, but I'm happy to be overruled if people think it's too misleading!

Also, obviously, this is a long way from perfect, so any other criticism is also welcome; again though, this is a v0, and the idea is to let it develop as people use it and hopefully add more metrics/optimise it in a way that suits the workflow!

Fixes #408

Checklist:

  • My code follows OCF's coding style guidelines
  • I have performed a self-review of my own code
  • I have made corresponding changes to the documentation
  • I have added tests that prove my fix is effective or that my feature works
  • I have checked my code and corrected any misspellings

Example of outputs

This is what the printout of the script looks like:

alex@donatello % python scorecard/generate_scorecard.py          
 33%|████████████████████████████████████████████████████                                                                                                        | 2/6 [00:07<00:13,  3.26s/it]
NMAE leaderboard
                          norm_abs_error
model                                   
nl_regional_48h_pv_ecmwf        0.016849
nl_regional_pv_ecmwf_sat        0.017033

NMAE cross-comparison
                          nl_regional_48h_pv_ecmwf  nl_regional_pv_ecmwf_sat
nl_regional_48h_pv_ecmwf                  0.000000                  0.010799
nl_regional_pv_ecmwf_sat                 -0.010916                  0.000000
 50%|██████████████████████████████████████████████████████████████████████████████                                                                              | 3/6 [00:41<00:51, 17.14s/it]
Quantile calibration
model     nl_regional_48h_pv_ecmwf  nl_regional_pv_ecmwf_sat
quantile                                                    
0.02                          0.03                      0.04
0.10                          0.13                      0.14
0.25                          0.27                      0.29
0.50                          0.52                      0.52
0.75                          0.75                      0.74
0.90                          0.89                      0.87
0.98                          0.97                      0.95
 67%|████████████████████████████████████████████████████████████████████████████████████████████████████████                                                    | 4/6 [01:50<01:15, 37.94s/it]
NMAE by month
model     nl_regional_48h_pv_ecmwf  nl_regional_pv_ecmwf_sat
month                                                       
Jan 2024                  0.008401                  0.008520
Feb 2024                  0.013348                  0.013402
Mar 2024                  0.017494                  0.0
[scorecard.pdf](https://github.com/user-attachments/files/28758457/scorecard.pdf)
17875
Apr 2024                  0.024492                  0.025181
May 2024                  0.024388                  0.024652
Jun 2024                  0.024564                  0.024459
Jul 2024                  0.023339                  0.022897
Aug 2024                  0.020309                  0.019889
Sep 2024                  0.016499                  0.016539
Oct 2024                  0.012144                  0.012575
Nov 2024                  0.008548                  0.009258
Dec 2024                  0.004653                  0.004976
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [02:32<00:00, 25.39s/it]

Report saved as backtest_report_demo_val_nl_sat_ablation_2/scorecard_nl_regional_48h_pv_ecmwf_nl_regional_pv_ecmwf_sat_20240101T0115_20241230T1145.pdf

I didn't want to paste the whole pdf in here so a semi-helpful screenshot:
image

And these are the plots that went into it and were also saved as .png:

image image image image

peterdudfield
peterdudfield reviewed Jun 9, 2026
View reviewed changes
Comment thread scripts/scorecard/generate_scorecard.py Resolved
@felix-e-h-p
felix-e-h-p self-requested a review June 9, 2026 16:15
felix-e-h-p
felix-e-h-p reviewed Jun 9, 2026
View reviewed changes
Comment thread scripts/scorecard/generate_scorecard.py Outdated

backtest_dict = {
"nl_regional_48h_pv_ecmwf":
"/mnt/storage_u2_30tb_a/scratch/nl_backtests/nl_val_backtests/"

@felix-e-h-p felix-e-h-p Jun 9, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this! Be all OK to move to a config file?

@AUdaltsova AUdaltsova Jun 9, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will do!

@AUdaltsova
AUdaltsova requested a review from felix-e-h-p June 10, 2026 13:12
@felix-e-h-p

felix-e-h-p commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Looks all good to me! Just to check, perhaps team wide question - defo yaml as opposed to say argparse?

@peterdudfield

peterdudfield commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Looks all good to me! Just to check, perhaps team wide question - defo yaml as opposed to say argparse?

For me yaml is better, slightly more overhead but much easier to track things

@AUdaltsova

AUdaltsova commented Jun 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

@peterdudfield @felix-e-h-p re: yaml, the reasoning here was that:

  1. user variables should be removed out of the script to avoid the annoyance of, "oh damn I want to commit something but I've changed all these paths and they're in the way"/"I want to do something that requires a clean tree and git wouldn't let me because I changed all these paths";
  2. yaml is a bit of overhead but it's a lot easier to track, as with argparse you'll have to pass all the reeeeeally long filenames in the cli and it's hard to see at a glance what you've added and what you haven't.
  3. yaml has a disadvantage that you can't easily wrap it in a bash script to e.g. run a loop with different arguments (like that backtest script for example, where you might want to backtest 5 models on the same data), but with the scorecard you can compare as many backtests as you want in one report, so that wouldn't be an issue.

Re: having a clean tree, I'll add the config to .gitignore now, slipped my mind!

@AUdaltsova
AUdaltsova merged commit 0411f4d into main Jun 12, 2026
6 checks passed
@AUdaltsova
AUdaltsova deleted the scorecard branch June 12, 2026 10:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@peterdudfield peterdudfield peterdudfield left review comments
@felix-e-h-p felix-e-h-p felix-e-h-p approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add a "score card" script/notebook (v0 release)

3 participants

@AUdaltsova @felix-e-h-p @peterdudfield