Skip to content

Generalize the benchmarking framework at country level#543

Merged
tgilon merged 66 commits intomasterfrom
feat/540-bench-by-ct
Mar 27, 2026
Merged

Generalize the benchmarking framework at country level#543
tgilon merged 66 commits intomasterfrom
feat/540-bench-by-ct

Conversation

@tgilon
Copy link
Copy Markdown
Member

@tgilon tgilon commented Mar 13, 2026

Closes #540.

Changes proposed in this Pull Request

This PR extends the benchmarking framework with a new bus-level and country-level overview. Where data is available, the benchmarks are produced both at EU27-level and at a spatially resolved level for each technology in a table.

The carrier-level tables now report on missing buses/countries (i.e. where a carrier is not defined for a given bus/country). The table-level tables report on missing carriers (i.e. where Open-TYNDP and reference carriers do not match) and the maximum number of missing buses/countries from the underlying carrier data.

Tasks

Workflow

  • (new configuration) benchmarking.spatial.by_bus and benchmarking.spatial.by_country, which control whether benchmarking at this level is performed
  • clean_tyndp_output_benchmark produces bus-level data where possible
  • clean_tyndp_report_benchmark has been refactored for clarity
  • make_benchmark and plot_benchmark always generate by_bus and by_country outputs. These outputs are empty if the configuration is disabled.
  • General improvements to the mappings and to the code structure

Open issues

Power capacities

  • Belgium: Open-TYNDP reports an extra 3GW of wind offshore compared to the Market Model (both 2030 and 2040). However, the capacity exists in the model in Denmark. This is a known issue with the naming conventions for the offshore topology ([SPIKE] investigate differences in offshore hub modelling #513).
  • Denmark: Except from the wind offshore issue described in the Belgium section, Open-TYNDP reports an extra of 1.4 GW of wind onshore compared to the Market Model in 2040 (Fix wrong onwind capacity in DK for NT 2040 #558).
  • Spain: The PEMMDB for ES00 only has the 20.5 GW of Gas CCGT present 1. However, the MM reports a value of 24.49 GW of CCGT present 1. If we add the Other Non-RES value for ES00 to the PEMMDB value, we get the exact correct value for Gas CCGT present 1: 20,517.56 + 3,980.011 = 24,497.571 GW. It could be that there is a double count of Other Non-RES Gas power plant.
  • Malta: Open-TYNDP reports an extra of 0.1 GW of solar compared to the MM in 2040 (Fix wrong solar capacity in MT for NT 2040 #559).
  • Netherlands: Open-TYNDP reports a lack of 1 GW of wind offshore compared to the MM (both 2030 and 2040). It's a rounding issue. Open-TYNDP reports 50 300 MW (rounded at 50 GW) and MM 50 542.5 MW (rounded at 51 GW).

Hydrogen demand

  • Hydrogen demand for power generation is still too low, especially in 2040.

Notes

Using the latest produced results (20260311), we get the following figures and tables. A set of figures has been made available here to make the review easier.

Overall overview by countries

kpis_s_all___all_years_by_country
  sMPE sMAPE sMdAPE RMSLE Growth Error Missing carriers Missing countries reference version
biomass_supply 0.21 0.63 0.32 1.24 0 0   TYNDP 2024 Scenarios Report v0.5.1+g0cfc7814b
elec_demand 0 0 0 0 0 0   TYNDP 2024 Scenarios Report v0.5.1+g0cfc7814b
energy_imports 0.58 0.58 0.4 1.11 0.02 1 0 TYNDP 2024 Scenarios Report v0.5.1+g0cfc7814b
final_energy_demand -0.1 0.27 0.09 0.56 0.01 0   TYNDP 2024 Scenarios Report v0.5.1+g0cfc7814b
generation_profiles           NA NA   v0.5.1+g0cfc7814b
hydrogen_demand -0.27 0.35 0 2.99 0 0   TYNDP 2024 Market Model Outputs v0.5.1+g0cfc7814b
hydrogen_supply -0.24 0.64 0.4 1.64 -0.41 3 0 TYNDP 2024 Market Model Outputs v0.5.1+g0cfc7814b
methane_demand 0.04 0.11 0.09 0.14 0 0   TYNDP 2024 Scenarios Report v0.5.1+g0cfc7814b
methane_supply 0.15 0.15 0.12 0.16 0.01 4 0 TYNDP 2024 Scenarios Report v0.5.1+g0cfc7814b
power_capacity 0.06 0.16 0 3.61 0 3 0 TYNDP 2024 Market Model Outputs v0.5.1+g0cfc7814b
power_generation -0.07 0.29 0.01 3.98 0.02 3 0 TYNDP 2024 Market Model Outputs v0.5.1+g0cfc7814b
Total (excl. time series) 0.04 0.38 0.05 1.56 0 11     v0.5.1+g0cfc7814b

Overall overview by bus

kpis_s_all___all_years_by_bus
  sMPE sMAPE sMdAPE RMSLE Growth Error Missing carriers Missing buses reference version
biomass_supply 0.21 0.63 0.32 1.24 0 0 0 TYNDP 2024 Scenarios Report v0.5.1+g0cfc7814b
elec_demand 0 0 0 0 0 0 0 TYNDP 2024 Scenarios Report v0.5.1+g0cfc7814b
energy_imports 0.58 0.58 0.4 1.11 0.02 1 0 TYNDP 2024 Scenarios Report v0.5.1+g0cfc7814b
final_energy_demand -0.1 0.27 0.09 0.56 0.01 0 0 TYNDP 2024 Scenarios Report v0.5.1+g0cfc7814b
generation_profiles           NA NA   v0.5.1+g0cfc7814b
hydrogen_demand -0.27 0.35 0 2.99 0 0 0 TYNDP 2024 Market Model Outputs v0.5.1+g0cfc7814b
hydrogen_supply -0.24 0.64 0.4 1.64 -0.41 3 0 TYNDP 2024 Market Model Outputs v0.5.1+g0cfc7814b
methane_demand 0.04 0.11 0.09 0.14 0 0 0 TYNDP 2024 Scenarios Report v0.5.1+g0cfc7814b
methane_supply 0.15 0.15 0.12 0.16 0.01 4 0 TYNDP 2024 Scenarios Report v0.5.1+g0cfc7814b
power_capacity 0.06 0.19 0 3.91 0 3 32 TYNDP 2024 Market Model Outputs v0.5.1+g0cfc7814b
power_generation -0.08 0.27 0 4.1 0.02 3 32 TYNDP 2024 Market Model Outputs v0.5.1+g0cfc7814b
Total (excl. time series) -0.1 0.21 0 1.93 0 16     v0.5.1+g0cfc7814b

Power capacities at country-level

benchmark_power_capacity_EU27_cy2009_2040 benchmark_power_capacity_BE_cy2009_2040
  sMPE sMAPE sMdAPE RMSLE Growth Error Missing buses Missing countries reference version
battery 0.47 1.25 1.41 10.34 -1.75 0   TYNDP 2024 Market Model Outputs v0.5.1+g0cfc7814b
chp and small thermal 0 0 0 0 0 0   TYNDP 2024 Market Model Outputs v0.5.1+g0cfc7814b
coal + other fossil (incl. biofuels) -0.02 0.02 0 0.06 -0.02 0   TYNDP 2024 Market Model Outputs v0.5.1+g0cfc7814b
hydro and pumped storage 0 0 0 0 0 0   TYNDP 2024 Market Model Outputs v0.5.1+g0cfc7814b
hydrogen 0 0 0 0 0 0   TYNDP 2024 Market Model Outputs v0.5.1+g0cfc7814b
methane 0 0.01 0 0.04 0.01 0   TYNDP 2024 Market Model Outputs v0.5.1+g0cfc7814b
nuclear 0 0 0 0 0 0   TYNDP 2024 Market Model Outputs v0.5.1+g0cfc7814b
oil (incl. biofuels) 0 0 0 0 0 0   TYNDP 2024 Market Model Outputs v0.5.1+g0cfc7814b
small scale res 0 0 0 0 0 0   TYNDP 2024 Market Model Outputs v0.5.1+g0cfc7814b
solar 0.01 0.01 0 0.06 0 0   TYNDP 2024 Market Model Outputs v0.5.1+g0cfc7814b
wind offshore 0 0.03 0 0.11 -0.04 0   TYNDP 2024 Market Model Outputs v0.5.1+g0cfc7814b
wind onshore 0 0 0 0.02 0 0   TYNDP 2024 Market Model Outputs v0.5.1+g0cfc7814b
solar thermal             0 TYNDP 2024 Market Model Outputs v0.5.1+g0cfc7814b
demand shedding             0 TYNDP 2024 Market Model Outputs v0.5.1+g0cfc7814b
OCGT             0 TYNDP 2024 Market Model Outputs v0.5.1+g0cfc7814b

Power capacities at bus-level

benchmark_power_capacity_EU27_cy2009_2040 benchmark_power_capacity_ITCA_cy2009_2040
  sMPE sMAPE sMdAPE RMSLE Growth Error Missing buses reference version
battery 0.52 1.21 1.33 10.53 -1.75 2 TYNDP 2024 Market Model Outputs v0.5.1+g0cfc7814b
chp and small thermal 0 0 0 0.01 0 0 TYNDP 2024 Market Model Outputs v0.5.1+g0cfc7814b
coal + other fossil (incl. biofuels) -0.02 0.02 0 0.06 -0.02 0 TYNDP 2024 Market Model Outputs v0.5.1+g0cfc7814b
hydro and pumped storage 0 0 0 0 0 0 TYNDP 2024 Market Model Outputs v0.5.1+g0cfc7814b
hydrogen 0 0 0 0 0 0 TYNDP 2024 Market Model Outputs v0.5.1+g0cfc7814b
methane 0 0.01 0 0.04 0.01 0 TYNDP 2024 Market Model Outputs v0.5.1+g0cfc7814b
nuclear 0 0 0 0 0 0 TYNDP 2024 Market Model Outputs v0.5.1+g0cfc7814b
oil (incl. biofuels) 0 0 0 0 0 0 TYNDP 2024 Market Model Outputs v0.5.1+g0cfc7814b
small scale res 0 0 0 0 0 0 TYNDP 2024 Market Model Outputs v0.5.1+g0cfc7814b
solar 0 0 0 0.05 0 0 TYNDP 2024 Market Model Outputs v0.5.1+g0cfc7814b
wind offshore -0.17 0.22 0 0.5 0.03 32 TYNDP 2024 Market Model Outputs v0.5.1+g0cfc7814b
wind onshore 0 0 0 0.03 0 0 TYNDP 2024 Market Model Outputs v0.5.1+g0cfc7814b
solar thermal           0 TYNDP 2024 Market Model Outputs v0.5.1+g0cfc7814b
demand shedding           0 TYNDP 2024 Market Model Outputs v0.5.1+g0cfc7814b
OCGT           0 TYNDP 2024 Market Model Outputs v0.5.1+g0cfc7814b

Checklist

  • I tested my contribution locally and it works as intended.
  • Code and workflow changes are sufficiently documented.
  • Changed dependencies are added to pixi.toml (using pixi add <dependency-name>).
  • Changes in configuration options are added in config/config.default.yaml.
  • Changes in configuration options are documented in doc/configtables/*.csv.
  • Changes in configuration options are added in config/test/*.yaml.
  • Open-TYNDP SPDX license header added to all touched files.
  • For new data sources or versions, these instructions have been followed.
  • New rules are documented in the appropriate doc/*.rst files.
  • A release note doc/release_notes.rst is added.
  • Major features are documented with up-to-date information in README and doc/index.rst.
  • Module docstrings added to new Python scripts.

tgilon and others added 30 commits March 4, 2026 11:14
@tgilon tgilon marked this pull request as ready for review March 20, 2026 10:57
@tgilon tgilon requested review from daniel-rdt and lisazeyen March 20, 2026 10:57
Copy link
Copy Markdown
Collaborator

@lisazeyen lisazeyen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot @tgilon ! This is a massive improvement for the benchmarking and helps us a lot to analyse the results.

Comment thread config/benchmarking.default.yaml
Comment thread doc/benchmarking.rst Outdated
Comment thread doc/release_notes.rst Outdated
Comment thread scripts/sb/build_statistics.py
Comment thread scripts/sb/build_statistics.py Outdated
Comment thread scripts/sb/build_statistics.py Outdated
Comment thread scripts/sb/build_statistics.py Outdated
Comment thread scripts/sb/clean_tyndp_output_benchmark.py
Comment thread scripts/sb/plot_benchmark.py Outdated
ax.bar_label(c, fmt="%.0f", padding=3, fontsize=8)
ax.bar_label(
c,
fmt=lambda x: f"{x:.1f}" if 0 < abs(x) < 10 else f"{x:.0f}",
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it is not good practice to have different rounding in the same plot because it reduces comparability, better to have all numbers in the same format

Suggested change
fmt=lambda x: f"{x:.1f}" if 0 < abs(x) < 10 else f"{x:.0f}",
fmt=lambda x: f"{x:.1f}",

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I get your point. Some of the numbers are already overlapping. Adding more numbers will make it even worse.

E.g.:

Image

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with both sentiments. Maybe you could increase the width of the plot so that the bars are wider giving more room to the figure labels?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could also reduce the font or tilt the text

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good ideas, we can make the figure wider, rotate the text, reduce font size, have different label positions

for i, (bar, val) in enumerate(zip(bars, values)):
    offset = 0.05 if i % 2 == 0 else 0.15
    ax.text(bar.get_x() + bar.get_width()/2, bar.get_height() + offset,
            f'{val:.1f}', ha='center', va='bottom')

I would favour that instead of having different roundings.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is my suggestion: when we have large numbers and / or many sources, the bar labels are rotated.

benchmark_power_generation_PL_cy2009_2040 benchmark_power_generation_IT_cy2009_2030 benchmark_power_capacity_SI_cy2009_2030 benchmark_power_capacity_NL_cy2009_2040 benchmark_power_capacity_EU27_cy2009_2040

Any further thoughts? @lisazeyen @daniel-rdt

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To me, this looks much better! Thank you @tgilon! We could even have it always rotated by 90° so it is consistent. But no strong opinion on that.

@tgilon tgilon requested a review from lisazeyen March 24, 2026 13:51
Copy link
Copy Markdown
Collaborator

@lisazeyen lisazeyen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tgilon thanks for the changes! I think it is nearly ready to be merged, only the merge conflicts need to be resolved and it would be good if we can change the layout of the figure so that we can have one more decimal in the bar plots.

tgilon added 2 commits March 25, 2026 22:37
 Conflicts:
	README.md
	config/benchmarking.default.yaml
        doc/release_notes.rst
@lisazeyen lisazeyen removed this from the Release v0.6 milestone Mar 27, 2026
@tgilon tgilon merged commit 625e84a into master Mar 27, 2026
6 checks passed
@tgilon tgilon deleted the feat/540-bench-by-ct branch March 27, 2026 16:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

SB Scenario Building

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[SUB] Generalize the benchmarking framework at country level

3 participants