Add risk-averse optimization #1345

Irieo · 2025-09-01T13:34:10Z

Add CVaR-based risk-averse optimization

we build here on top of #1154

This PR integrates a linearized Conditional Value-at-Risk (CVaR) formulation as a new feature for PyPSA’s stochastic optimization. It augments the objective with a convex risk term and adds auxiliary variables and constraints to represent CVaR over scenario-dependent operating costs.

New risk preference API on the Network: set_risk_preference(alpha: float, omega: float).
Works seamlessly for stochastic networks; falls back to the original objective for deterministic networks or when risk preference is not set.

Mathematical formulation

This formulation was introduced by Rockafellar & Uryasev (2002), and later adopted by folks from the energy economics & modelling fields, eg Munoz et al. (2017) is a great example!

Let $s \in S$ index scenarios with probability $p_s$, and let $\mathrm{OPEX}(s)$ denote the operating cost in scenario $s$. We introduce auxiliary decision variables:

a_s ≥ 0 (scenario excess costs)
θ (VaR level)
CVaR (the CVaR value)

The risk-augmented objective is

$$ \min\ OBJ := \mathrm{CAPEX} + (1-\omega)*\mathbb{E}[\mathrm{OPEX}] + \omega*\mathrm{CVaR}_{\alpha}, $$

subject to the standard PyPSA constraints and the following auxiliary constraints for CVaR linearization:

$$ \theta + \frac{1}{1-\alpha} \sum_{s\in S} p_s* a_s \le \mathrm{CVaR}. $$

$$ a_s \ge \mathrm{OPEX}(s) - \theta. $$

Here,

$\mathbb{E}[\mathrm{OPEX}] = \sum_{s\in S} p_s\ * \mathrm{OPEX}(s)$
$\alpha$ is the CVaR confidence level, so $1-\alpha$ is the tail mass
$\omega$ is the risk preference weight

Variables and constraints added

Variables
- CVaR-a[scenario] (nonnegative): excess loss $a_s$ for each scenario $s$.
- CVaR-theta (free): VaR level $\theta$.
- CVaR (free): CVaR value.
Constraints
- CVaR-excess-<scenario>: $a_s - \mathrm{OPEX}(s) + \theta \ge 0$.
- CVaR-def: $\theta + \tfrac{1}{1-\alpha} \sum_s p_s a_s \le \mathrm{CVaR}$.

Notes

The auxiliary variables are not written back to component tables; they are internal to the optimization model.
Scenario probabilities come from Network.scenario_weightings and are used both in E[OPEX] and the CVaR-def constraint.
Alpha (CVaR confidence level) has bounds of $0 < \alpha < 1$. At $\alpha = 1$ the denominator is zero (undefined). At $\alpha = 0$, the confidence level covers the entire distribution and the formulation degenerates (equivalent to expectation).
Omega (risk weight) has bounds of $0 \le \omega \le 1$. The objective uses a convex combination $(1-\omega)*\mathbb{E}[\mathrm{OPEX}] + \omega*\mathrm{CVaR}$ Allowing the closed interval includes meaningful endpoints: $\omega=0$ yields the risk‑neutral expected‑cost objective; $\omega=1$ yields pure CVaR minimization.
Our CVaR constraint uses tail $1/(1 − alpha)$, meaning it averages losses over the worst (1 − alpha) mass.
For a case with quadratic marginal costs, CVaR constraint becomes a convex quadratic inequality, which is not supported by linopy. So marginal_cost_quadratic is the only option we're not supporting with CVaR. I add a guard for that.

API for users

Network.set_risk_preference(alpha: float, omega: float)
- Sets the CVaR level α and risk weight ω on the network. When present and the network has scenarios, the optimization model automatically includes CVaR variables and constraints and augments the objective.
- If not set, or if the network has no scenarios, the model behaves as before (risk‑neutral expected cost for stochastic; plain deterministic for non-stochastic).

Backward compatibility

Deterministic networks: unchanged objective and constraints.
Stochastic networks without a configured risk preference: unchanged (risk‑neutral expected cost).

How to use

import pypsa
n = pypsa.examples.model_energy()
n.set_scenarios({"volcano": 0.1, "no_volcano": 0.9})
n.generators_t.p_max_pu.loc[:, ("volcano", "solar")] *= 0.3

# Risk-neutral baseline (expected cost)
n.optimize()

# CVaR with moderate risk aversion
n.set_risk_preference(alpha=0.9, omega=0.5)
n.optimize()

# Edge case: CVaR with omega=0 equals risk-neutral
n.set_risk_preference(alpha=0.9, omega=0.0)
n.optimize()

# Edge case: CVaR capturing the results of the worst-case scenario (omega=1, alpha so only worst scenario is in tail)
p_worst = float(n.scenario_weightings.loc["volcano", "weight"])  # 0.1
n.set_risk_preference(alpha=1 - p_worst, omega=1.0)
n.optimize()

Example of results to expect

Capacity diff (omega=0.5 - neutral) [GW]:
component    carrier         
Generator    load shedding          0.000000
             solar                 10.882582
             wind                   4.636405
Link         electrolysis           0.594093
             turbine                0.364935
StorageUnit  battery storage       -1.313774
Store        hydrogen storage    1579.986342

CVaR penalizes expensive tail operations (worst-scenario OPEX like load shedding and costly peakers). It trades more CAPEX now to avoid high OPEX in the tail. In this system, the cheapest hedge is to overbuild cheap VRE (solar) even if it is hit in “volcano” scenario. Here “more solar + long-duration storage” is the least-cost way to suppress worst-case costs; however, what hedge risk-averse optimization yields depends on cost assumptions and degrees of freedom available. In short: risk aversion hedges against the risky scenario by shifting costs from tail OPEX to upfront CAPEX.

A dedicated PR with a notebooks and user guide will go v1-docs branch #1250

References

Rockafellar & Uryasev (2002). Conditional Value‑at‑Risk for General Loss Distributions. Journal of Banking & Finance, 26(7), 1443–1471. DOI: https://doi.org/10.1016/S0378-4266(02)00271-6
Munoz et al. (2017). Does risk aversion affect transmission and generation planning? A Western North America case study. Energy Economics, Volume 64. DOI: https://doi.org/10.1016/j.eneco.2017.03.025

TODOs

Other
-> [into v1-docs] add user guide and example notebook to docs (and patch where docs saying risk aversion will be in future releases)
-> is there a better way for a workaround in constraints.py to avoid circular import with optimize.py?

fneum · 2025-09-03T16:31:50Z

I copy your user guide suggestions @fneum, thanks. I hide it here to keep this PR discussion shorter, and open a new PR to v1-docs branch with docs & examples.

doc/references/release-notes.rst

pypsa/network/index.py

pypsa/optimization/constraints.py

pypsa/optimization/optimize.py

fneum · 2025-09-03T17:46:47Z

Functionality-wise all excellent! Played a bit with the following code to calculate insurance premium, tail risk reduction, risk-hedging cost ratio and risk-cost frontiers. These would make excellent functions like planned for expected value of information.

import matplotlib.pyplot as plt
import pandas as pd

import pypsa

n = pypsa.examples.model_energy()
n.add(
    "Generator",
    "gas",
    carrier="gas",
    bus="electricity",
    p_nom=5000,
    efficiency=0.5,
    marginal_cost=100,
)

n.set_scenarios({"volcano": 0.1, "no_volcano": 0.9})
n.generators_t.p_max_pu.loc[:, ("volcano", "solar")] *= 0.3

n.set_risk_preference(alpha=0.9, omega=0)
n.optimize(log_to_console=False, solver_name="gurobi")

cvar_risk_neutral = float(n.model["CVaR"].solution) / 1e9

etc_risk_neutral = (
    n.statistics.capex().div(1e9).sum()
    + n.statistics.opex().mul(n.scenario_weightings.squeeze()).div(1e9).sum()
)
etc_risk_neutral

n.set_risk_preference(alpha=0.9, omega=0.5)
n.optimize(log_to_console=False, solver_name="gurobi")

cvar_risk_averse = float(n.model["CVaR"].solution) / 1e9

etc_risk_averse = (
    n.statistics.capex().div(1e9).sum()
    + n.statistics.opex().mul(n.scenario_weightings.squeeze()).div(1e9).sum()
)
etc_risk_averse

insurance_premium = etc_risk_averse - etc_risk_neutral
insurance_premium

tail_risk_reduction = cvar_risk_neutral - cvar_risk_averse
tail_risk_reduction

risk_hedging_cost = insurance_premium / tail_risk_reduction
risk_hedging_cost

cvar = {}
etc = {}
for omega in [0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]:
    n.set_risk_preference(0.9, omega)
    n.optimize(log_to_console=False, solver_name="gurobi")
    cvar[omega] = float(n.model["CVaR"].solution) / 1e9
    etc[omega] = (
        n.statistics.capex().div(1e9).sum()
        + n.statistics.opex().mul(n.scenario_weightings.squeeze()).div(1e9).sum()
    )

pd.Series(etc).plot(label="expected total cost")
pd.Series(cvar).plot(label="conditional value at risk", ylim=(0, 12))
plt.legend()
plt.grid()

plt.scatter(etc.values(), cvar.values(), c=cvar.keys())
plt.grid()
plt.ylabel("conditional value at risk")
plt.xlabel("expected total cost")
plt.ylim(0, 3)
plt.xlim(9, 11)

CVaR linearization requires per-scenario OPEX. Since OPEX is assembled inside define_objective, constraints must reconstruct the same OPEX terms. The prototype version in 558afb7 has much less code, but goes against architecture principles. I could make shared OPEX builder to avoid duplication but I don't like this idea too much.

- ensure strict monotonicity of CVaR objective vs omega (neutral < 0.25 < 0.5 < 0.75) - ensure CVaR omega=0 equals SP risk-neutral baseline (capacities and objective) - ensure CVaR omega=1 & alpha=1-p_worst equals deterministic run for worst-case scenario"

…optimization modes and component settings

CVaR aux constraints a(s) ≥ OPEX(s) − θ are linear only if OPEX is affine. When marginal_cost_quadratic is there, we get a QCP problem. Here we add a showstopper in define_cvar_constraints that raises an error and tells to remove or approximate quadratic costs.

…omponents

Co-authored-by: Fabian Neumann <[email protected]>

lkstrp

A couple of minor stuff and:

If I understand correctly, no risk preference is the same as having alpha=0 and omega=0. I am not sure if I then prefer to just have those as defaults in the risk_preference dict. Then we don't need to mix types there.

What do you think?

Also, if we will never have risk_preference without stochastic networks, just disallow setting it on non stochastic networks.

pypsa/network/index.py

pypsa/optimization/optimize.py

pypsa/optimization/variables.py

fneum · 2025-09-05T15:18:39Z

Also, if we will never have risk_preference without stochastic networks, just disallow setting it on non stochastic networks.

Yes, it's correct that risk_preference should only be set for stochastic networks.

If I understand correctly, no risk preference is the same as having alpha=0 and omega=0. I am not sure if I then prefer to just have those as defaults in the risk_preference dict. Then we don't need to mix types there.

I think that works. Actually omega=0 would be enough regardless of alpha (not that it matters for the default).

Co-authored-by: Lukas Trippe <[email protected]>

Irieo · 2025-09-12T14:03:53Z

Also, if we will never have risk_preference without stochastic networks, just disallow setting it on non stochastic networks.

Yes, it's correct that risk_preference should only be set for stochastic networks.

If I understand correctly, no risk preference is the same as having alpha=0 and omega=0. I am not sure if I then prefer to just have those as defaults in the risk_preference dict. Then we don't need to mix types there.

I think that works. Actually omega=0 would be enough regardless of alpha (not that it matters for the default).

regarding the point whether risk_preference should only be set for stochastic networks, or not (ie for any but ignore if not stochastic), it might be this point contradicts this #1345 (comment)

@lkstrp what do you think?

upd: we decided for risk_preference only for stoch network way.

Co-authored-by: Lukas Trippe <[email protected]>

- set_risk_preference requires set_scenarios() first - remove redundant warning in optimize.py - remove warning test, add test_set_risk_preference_requires_scenarios

Irieo · 2025-09-15T13:22:03Z

@lkstrp all your points are addressed now, I think.

All suggestions are addressed

* feat: add network methods for risk preferences * feat: prototype first working version * feat: move CVaR constraints to constraints.py CVaR linearization requires per-scenario OPEX. Since OPEX is assembled inside define_objective, constraints must reconstruct the same OPEX terms. The prototype version in 558afb7 has much less code, but goes against architecture principles. I could make shared OPEX builder to avoid duplication but I don't like this idea too much. * tests: add tests for new network methods and properties * tests: add optimization tests - ensure strict monotonicity of CVaR objective vs omega (neutral < 0.25 < 0.5 < 0.75) - ensure CVaR omega=0 equals SP risk-neutral baseline (capacities and objective) - ensure CVaR omega=1 & alpha=1-p_worst equals deterministic run for worst-case scenario" * docs: add release notes * types: make mypy happier * tests: make codecov/patch happy with tests covering CVaR for various optimization modes and component settings * feat: add a guard for quadratic marginal costs CVaR aux constraints a(s) ≥ OPEX(s) − θ are linear only if OPEX is affine. When marginal_cost_quadratic is there, we get a QCP problem. Here we add a showstopper in define_cvar_constraints that raises an error and tells to remove or approximate quadratic costs. * docs: better docstrings for new network methods * codecov: add tests for guards, incorrect value checks and more OPEX components * revision: trim release notes * revision: update pypsa/network/index.py Co-authored-by: Fabian Neumann <[email protected]> * revision: relax condition for setting risk preference * revision: address minor comments * revision: revert 48a1895 * revision: make mypy happy again * Update nomorerisks with latest master * commit lkstrp suggestion to pypsa/network/index.py Co-authored-by: Lukas Trippe <[email protected]> * revision: minor touch docstrings * revision: minor touch optimize.py * revision: update pypsa/optimization/optimize.py Co-authored-by: Lukas Trippe <[email protected]> * revision: minor touch pypsa/optimization/variables.py Co-authored-by: Lukas Trippe <[email protected]> * revision: docstrings in variables.py * revision: capture QCP problem in test * revision: remove redundant docstring lines * restore: require scenarios before setting risk preference - set_risk_preference requires set_scenarios() first - remove redundant warning in optimize.py - remove warning test, add test_set_risk_preference_requires_scenarios * revision: drop guards from optimize.py & drop related tests * fix: log custom variables without dash as intended in #1353 * fix: doctest problem * fix: types * revision: no more RuntimeError --------- Co-authored-by: Fabian Neumann <[email protected]> Co-authored-by: Lukas Trippe <[email protected]>

* initial draft to add prices in statistics. needs further modularization. * refined prices statistics function with tests * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Deprecate define_nominal_constraints_per_bus_carrier (#1294) * Deprecate * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update pypsa/optimization/global_constraints.py Co-authored-by: Lukas Trippe <[email protected]> --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Lukas Trippe <[email protected]> * fix: activate doctests (#1327) * Add additional MGA functionality and example (#1269) * Add additional MGA functionality Introduces new functions in optimization/abstract.py to facilitate the analysis of trade-offs between multiple objectives/dimensions using near-optimal methods. Allows for solving a network in a direction given in the coordinate space of user-specified dimensions, and also introduces a parallelized function to solve in multiple directions at once. Three different functions generating (random) directions are also provided. * Fix typing * Add test coverage for new mga functionality * Use `is_solved` attribute * Fix docstrings; return coordinates a pd.Series * refactor: to `mga.py` * refactor parallel mga to avoid pickling; robustness Also improve docstrings and add guardrails to make parallel code more robust * Fix mga docstring example * fix: tests --------- Co-authored-by: Koen van Greevenbroek <[email protected]> Co-authored-by: Lukas Trippe <[email protected]> * feat: exclude inactive components from model build (#1310) * squashed commit xarray-view * deactivate deprecation checks * Xarray view 2 (#1164) * feat: introduce component class * refactor: rename `core.py` -> `network.py` * refactor: move data to data dir * rename: `types.py` -> `typing.py` * move standard_types to `ComponentTypeInfo` * add `list_as_string()` * feat: add component specific class * fix: tests * feat: add component data view * Merge branch 'master' into xarray-view * refactor: add explicit component definitions, add operational_attr map * refac: move components descriptors to components module * reset constraints.py from master * fix: doc test * reset parts of optimization/variables.py * start converting variables * feat: make as_xarray robust for scenario dim * fix: ignore type for nominal attr in Loads * revert test on snapshot weightings index name * fix indentation * fix: iterate over components values in set_investment_persiods * review --------- Co-authored-by: lkstrp <[email protected]> * adjust components methods * make methods internal * refactor: components methods import * Refactor optimization code (#1166) * make get_bounds_pu support DataArrays * refactor: operational constraints for non-ext * refactor: operational constraints for ext * refactor: nominal constraints for ext * patch: FH feedback * refactor: make get_bounds_pu individual for component classes * refactor: nodal balance constraint * fix type checker issues (mypy) * refactor: operational constraints for committables * tests: cover the case with ramp_rates & committable * refactor: ramp limit constraints 1/2 * fix: correct status_prev in ramp limit constraints * refactor: ramp limit con 2/2, handle rolling horizon properly * fix: minor correction of types in func signatures * refactor: fixed operation cons & add related tests * revert: fixed operation cons (to fix p_set issue) * tests: add test covering case with p_set set to zero * fix: handle p_set default values for lopf case * fix: update p_set default values to 'n/a' and make exception for power flow calculations * fix: update shadow price attributes to default to 'n/a' in component attribute files * refactor: fixed operation cons (incl. storage level) * add test for fixed operation value of extendable component, fix cons name * refactor: modular constraints * further components refactor * apply components convention * refactor: storage_unit_constraints * refactor: store_constraints, improve docstrings * refactor: define_loss_constraints, add docstring * update docstring based on numpy style guide * docstrings: add good docs everywhere * refactor: total_supply_constraints * remove `-ext`, `-com`, `-fix` * remove previous helpers * adjust deprecations * rename component dim name (`c.name` to `"component"`) * fix: fix SCLOPF broken with 22a9c3e * fix tests * refactor: objective, add docstring --------- Co-authored-by: lkstrp <[email protected]> * initial commit * remove scenarios * Revert "remove scenarios" This reverts commit 20b0264. * fix * fix types * fix: capex broadcasting problem introduced in #1166 and not covered by tests * add `set_scenarios` interface * add tests * add scenarios info to __repr__ * tests: simple fixture and test placeholders * make as_xarray work with scenarios * tests: add benchmark solved problem, add fixture, simplify properties test * Revert "make as_xarray work with scenarios" This reverts commit 343d989. * put scenario back on columns axis for `n.dynamic` * make as_xarray work again based on different axis * fix renamed index in `define_nodal_balance_constraints` * fix types * some refinement * fix: test_bugs run through * fix: tests -- handle networks with no loads * tests: mute unfinished tests for now * add stochastic network fixture * add network cycles function * refactor kvl constraint to use cycles function * fixes: many fixes to make scenario network work * wip: handle case when no components and scenarios present * feat: support kvl constraints for stochastic network * add network cycles function * refactor kvl constraint to use cycles function * fix: remove breakpoint * fix: skip kvl constraints if no cycles there * wip: handle assign solution * wip: prep tests * fix spatial clustering * stoch: correct objective weights with scen probabilities * more post_processing fixes * remove tables pin * Revert "stoch: correct objective weights with scen probabilities" This reverts commit 44bfb7c. * stoch: correct objective weights with scen probabilities (now with KVL) * fix: write back p_nom_opt to network * add full test for stoch problem obj, dispatch, status * tests: mute mutiinvest tests until KVL is fixed * final tests fixed * more final tests and fix types * fix: make proper snapshot selection in kvl constraint * fix: statistics for stoch networks * refactor: Remove unused 'as_xarray' parameter in get_bounds_pu * feat: add xarray accessor for attribute style lazy access * refac: make adajacency matrix backwards compat * fix: tests * fix: Handle NaN p_set in LPF and fix allocation function call. Fixes test_lpf_ac_dc.py::test_lpf failure * test: Add stochastic variable/constraint dimension test * test: Add stochastic multiperiod optimization test * test: minor cleanup * fix: adds stochastic network support to growth limits * tests: add single-scenario stochastic network test * feat: add support for nested members of collection * fix: types * fix: missed new index name * fix: tests * fix: ensure correct column names in get_switchable_as_dense for stoch networks * fix: set index names in lpf_contingency * refactor: operational attrs and bounds pu redefinment * docs: add component/attr scope into constraint functions * feat: make adjacency matrix fully backwards compatible * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: udpate doc example * fix: pre-commit * fix: types * fix: docs build * fix: treat special case with aux dim in dual variable assignment (fix for failing tests for security-constrained optimization) * fix: extended pre-commits * fix: support multi-dimensional duals in security-constrained lopf * fix: resolve problems dual assignment - we treat here 2D cases with aux dimensions ('cycle' or sclopf-related stuff) - patch from_xarray to treat all cases - refactor assign_duals() so it can be read by humans * fix: TypeError in stochastic optimization constant handling * fix: doctest failures in networks.py (skip examples requiring optimized network) * fix: types * refactor: rename new index names `component` -> `name` * refactor: use `c.da` instead of `c.as_xarray` * fix: correct calculation of objective constant term * fix: make 82f51d8 work * fix: reserve power example * docs: remove old stochastic example * fix: correct dimension name in loss constraints * feat: add consistency check and unit test for scenario probabilities * feat: add scenario invariant attributes check and related unit tests * wip: add io for `.nc` * add example network * fix: types * fix: stochastic network handling bus 'control' attr & rework consistency checks - fixes test_optimization_simple() test fail because bus 'control' attribute varied across scenarios, violating consistency check - context: Bus 'control' is derived from generators during topology determination, not user-set - Remove 'control' from invariant_attrs in check_scenario_invariant_attributes() since it's a user-set attr - feat power_flow.py -> find_slack_bus() to select the same slack generators in stochastic networks - Add dedicated consistency check to ensure same slack bus across scenarios - Add unit tests for all these things * refactor: align and adjust how index data is stored * fix: types * fix: excel io * fix: missing `obj` col in for sub_networks * fix: stochastic network support for post-processing calculations - Skip power flow calculations for stochastic networks in optimize.py post_processing() - Fix determine_network_topology() to handle MultiIndex structure of stoch network by workign with first scenario only (possible since consistency_check() ensures all scenarios have the same topology) - Fix find_cycles() to access scenarios via sub_network.n.scenarios instead of sub_network.scenarios * fix: stoch optimization with Store component bug -> handle MultiIndex dime in elapsed hours. Add two unit tests. * Fix scenario ordering bug in stochastic optimization - Ensure scenario coordinates preserve correct order in DataArrays - Add tests for this * make mypy happy * fix: compatible multiperiod stochastic optimization 🐉 • Fix broadcasting error in storage unit constraint when multiperiod invest is enabled • Add many tests for multiperiod stochastic optimization • Test storage unit & multiinvest & stochastic network issue specifically * add test for determine_network_topology * refac: make determine_network_topology compatible with multiindexed components * fix io: keep column order in determine_network_topology * refac: write it a tiny bit more compact * Revert "refactor: use `c.da` instead of `c.as_xarray`" This reverts commit 401f641. * Revert "feat: add xarray accessor for attribute style lazy access" This reverts commit 28bd950. * fix: tests * Revert "Revert "feat: add xarray accessor for attribute style lazy access"" This reverts commit e48f5f1. * Revert "Revert "refactor: use `c.da` instead of `c.as_xarray`"" This reverts commit 2e68217. * Revert "fix: tests" This reverts commit be7c8fd. * perf: fix memory issues in `_as_dynamic` * perf: wrap `_as_dynamic` in `get_switchable_as_dense` * perf: optimize subset validation * perf: lazy load sklearn `HAC` * perf: remove redundant reindex * refac: remove inspect-based power flow detection in array module * test: skip SCLOPF test on Python 3.10 due to unstable numerics * revert: skip SCLOPF test on Python 3.10 due to unstable numerics * test: tighten solver tolerances as honest attempt to fix numerical instability of SCLOPF * fix: exclude standard types from scenario broadcasting * Revert "fix: exclude standard types from scenario broadcasting" This reverts commit aaa7018. * wip: fixing unaligned index names * feat: treat MultiIndex case of line_types in stochastic networks - Fix apply_line_types() to handle MultiIndex case - Add check_line_types_consistency() to ensure line_types are identical across scenarios - Add tests for line_types consistency check * fix: handle meshed bus index in optimize.py for stoch networks * fix: coordinate alignment in case of multiperiod + stoch + storage unit - fix in constraints.py - remove TODO from stoch hackaton period - add unit test * fix: correct dims of total_supply_constraints for stochastic networks * feat: make primary/operational limit (global constraints) compatible with scenarios * feat: add stoch opt support to growth limit constraints - we find and apply strictest (minimum) max_growth and max_relative_growth values across scenarios - add unit tests to ensure that growth limits are applied and applied correctly - reuse our helper _extract_names_for_multiindex to cut lines of code - revert bebff25 which I admit was dumb * add test on primary energy limit * apply primary energy limit to individual scenarios * raise error for non supported global constraint (stoch case) * feat: add test for operational limit (global constraint) * refactor: remove `drop_scenarios` argument from `_as_xarray` * refactor: clean up `_as_xarray` * feat: add runtime verification option and simplfy `_as_xarray` * fix: doctests * fix: remove redundant func in global constraint & refine unit tests for growth limits * fix: add missed module * fix types and add tests * fix: attribute error in `define_operational_limit` * feat: add not implemented decorator for clustering * test: add test for `c.ds` * refactor: remove unused helper methods and small fixes * refactor: more minor changes * fix: buggy storage constraints RHS sorting. * test: add unit test for storage unit RHS problem * test: improve tests and `from_xarray` write back * fix: unneccessary except * fix: flaky doctest * test: ensure that duals are correctly assigned in stoch networks. * fix: relax tolerance in scenario sum probability check. Avoids rounding error if scenarios are passed via list. * Reverted line_types_to_use to a2213ee, as code refactoring introduced bug. * refactor: let uc status always return single dim name index * feat: exclude inactive components from model build * Update pypsa/networks.py Co-authored-by: Fabian Neumann <[email protected]> * refactor primary energy limit to use scenario dim * fix: correct rhs in multi-invest & storage & global_cons case caused by 171dc0f. Add missing test for --all-- + stochastic network * fix: write back duals for static/dynamic global constraints * fix: avoid integer-location index in multi-invest objective * tests: make stoch solution test better, remove redundant * tests: more patches to stoch unit tests * tests: check that ramp_limit_start_up attr works as intended * refactor: align operational_limit constraint logic with 171dc0f - We build one Linopy expression per scenario grouped by cons name - Merge them into a single constraint with a scenario dimension - Add this as single constraint into n.model - The same logic as in constraints.py * tests: update assertions to reflect changes in 6626f7f * feat(global-constraints): make transmission_volume_expansion_limit stochastic-aware Implement scenario dim in define_transmission_volume_expansion_limit. Build per-scenario expression over extendable Lines/Links filtered by carrier. Merge into a single constraint per name (GlobalConstraint-<name>). Remove NotImplementedError. Add test: test_transmission_volume_expansion_limit_constraint_stochastic. * test: add network data guard for optimize and conistency * fix: small changes from review * fix: fix and improve assign_duals behavior * fix: update usage of `unique` to `drop_duplicates` to keep order * fix: linear power_flow * test: add more (inactive) guards * fix: default investment_periods not assigned * fix: pre-commit * fix: removed SubNetwork `obj` during io * remove comment line with redundant TODO * Update pypsa/networks.py Co-authored-by: Fabian Neumann <[email protected]> * ci: log installed versions * fix: typo in merge * fix: types * rename cycles to cycle_matrix, adjust docstring * add unit test placeholder to catch dual assignemnt problem * fix: assign duals for GlobalConstraints * tests: add assertions to placeholder (temporary v. until assign_duals debugged) * fix: types * better solution write back, test and remove also from masks * Add additional activity mask tests * activity mask fixes * additional fix to store constraints building * Fix global constraints in case of activity mask; add test --------- Co-authored-by: Irieo <[email protected]> Co-authored-by: Fabian Hofmann <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Bobby Xiong <[email protected]> Co-authored-by: Fabian Neumann <[email protected]> Co-authored-by: Koen van Greevenbroek <[email protected]> * fix: rename strongly meshed buses to Bus (#1304) * fix: rename strongly meshed buses to Bus Follow-up to #1274 and fix for PyPSA/linopy#470. * Update release-notes.rst --------- Co-authored-by: Lukas Trippe <[email protected]> * feat: full support for current and new components API (#1329) * rename option * rename static data * rename dynamic data * add pytest flag `--new-components-api` * ci: add ci job for new api * fix: ci * ci: run for all versions * fix * Update pypsa/plot/maps/interactive.py Co-authored-by: Fabian Neumann <[email protected]> * ignore FutureWarning --------- Co-authored-by: Fabian Neumann <[email protected]> * fix: small fix for new api (#1331) * prepare release `v0.35.2` * prepare release `v1.0.0rc1` * [github-actions.ci] prepare release v1.0.0rc1 * fix: temporary ignore raised 1.0 deprecations (#1334) * build(deps): bump the github-actions group with 2 updates (#1344) Bumps the github-actions group with 2 updates: [actions/checkout](https://github.com/actions/checkout) and [actions/download-artifact](https://github.com/actions/download-artifact). Updates `actions/checkout` from 4 to 5 - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v4...v5) Updates `actions/download-artifact` from 4 to 5 - [Release notes](https://github.com/actions/download-artifact/releases) - [Commits](actions/download-artifact@v4...v5) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '5' dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions - dependency-name: actions/download-artifact dependency-version: '5' dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * add additional line types from pandapower (#1342) * Bugfix: Handle alphanumeric versions (like "1.0.0rc1") (#1338) * Bugfix version.py for alphanumeric versions like 1.0.0rc1 * Extended unit tests by alphanumeric versions and parse_version_tuple_function * Using packaging.version to do version comparison * Fixed conversion back to str. * refactor: refactor version attributes and remove tuple versions * fix: simplify regex version match --------- Co-authored-by: lkstrp <[email protected]> * fix: update url for carbon management file (#1346) * feat: improved optional dependency handling (#1348) * feat: make return of added component names optional (#1347) * feat: make return of added component names optional * feat: add default option * Fix custom variable bug (#1353) * add test test_assign_custom_variable Tests if the optimization raises an error after adding a custom variable to the linopy model. * fix: handle custom variable names (#1351) Solutions of custom variables in the linopy model that can not be mapped to a network component are skipped. Log message for user, to allow user to understand why the solution of the custom variable was not mapped. * test: add test for warnings --------- Co-authored-by: lkstrp <[email protected]> * Remove 3.13 dependency pins (#1355) * refactor: resolve deprecations and make components iter compatible (#1349) * fix: address deprecations in network mixins * Fix more deprecations * Address more deprecations (claude supported) * Add warnings filters for network collection get_active_assets * Add pytest warns fences for testing old components api * Revert "fix: temporary ignore raised 1.0 deprecations (#1334)" This reverts commit 1871897. * refactor: resolve deprecations and make components iter compatible * fix: types * fix: don't raise new API deprecation warnings * test: now check for warnings instead of errors * mute assertion in test_sclopf_scigrid, see #1356 * fix: consistent ordering during iter * fix: for inconsistencys --------- Co-authored-by: Jonas Hoersch <[email protected]> Co-authored-by: Irieo <[email protected]> * Add risk-averse optimization (#1345) * feat: add network methods for risk preferences * feat: prototype first working version * feat: move CVaR constraints to constraints.py CVaR linearization requires per-scenario OPEX. Since OPEX is assembled inside define_objective, constraints must reconstruct the same OPEX terms. The prototype version in 558afb7 has much less code, but goes against architecture principles. I could make shared OPEX builder to avoid duplication but I don't like this idea too much. * tests: add tests for new network methods and properties * tests: add optimization tests - ensure strict monotonicity of CVaR objective vs omega (neutral < 0.25 < 0.5 < 0.75) - ensure CVaR omega=0 equals SP risk-neutral baseline (capacities and objective) - ensure CVaR omega=1 & alpha=1-p_worst equals deterministic run for worst-case scenario" * docs: add release notes * types: make mypy happier * tests: make codecov/patch happy with tests covering CVaR for various optimization modes and component settings * feat: add a guard for quadratic marginal costs CVaR aux constraints a(s) ≥ OPEX(s) − θ are linear only if OPEX is affine. When marginal_cost_quadratic is there, we get a QCP problem. Here we add a showstopper in define_cvar_constraints that raises an error and tells to remove or approximate quadratic costs. * docs: better docstrings for new network methods * codecov: add tests for guards, incorrect value checks and more OPEX components * revision: trim release notes * revision: update pypsa/network/index.py Co-authored-by: Fabian Neumann <[email protected]> * revision: relax condition for setting risk preference * revision: address minor comments * revision: revert 48a1895 * revision: make mypy happy again * Update nomorerisks with latest master * commit lkstrp suggestion to pypsa/network/index.py Co-authored-by: Lukas Trippe <[email protected]> * revision: minor touch docstrings * revision: minor touch optimize.py * revision: update pypsa/optimization/optimize.py Co-authored-by: Lukas Trippe <[email protected]> * revision: minor touch pypsa/optimization/variables.py Co-authored-by: Lukas Trippe <[email protected]> * revision: docstrings in variables.py * revision: capture QCP problem in test * revision: remove redundant docstring lines * restore: require scenarios before setting risk preference - set_risk_preference requires set_scenarios() first - remove redundant warning in optimize.py - remove warning test, add test_set_risk_preference_requires_scenarios * revision: drop guards from optimize.py & drop related tests * fix: log custom variables without dash as intended in #1353 * fix: doctest problem * fix: types * revision: no more RuntimeError --------- Co-authored-by: Fabian Neumann <[email protected]> Co-authored-by: Lukas Trippe <[email protected]> * feat: more default params options for optimization module (#1359) * feat: more default params options for optimization module * fix: tests * Add undocumented bus attribute of GlobalConstraint (#1293) * Add undocumented bus attribute of GlobalConstraint * fix: make optional --------- Co-authored-by: Lukas Trippe <[email protected]> * Optimize _sort_attrs for already ordered axes (#1362) * Optimize _sort_attrs for already ordered axes * Add release note * Further refactor _sort_attrs to just take column list, not dataframe * refactor: refine statistic arguments (#1358) * refactor: refine statistic arguments * docs: add release note * adjust prices and statistics * wip * improved schema for statistics plots * rename baseline files * test: adjust mpl tests and upload comparison * fix: path * fix * fix windows tests * fix: rename missed baseline files --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Fabian Neumann <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Lukas Trippe <[email protected]> Co-authored-by: Koen van Greevenbroek <[email protected]> Co-authored-by: Koen van Greevenbroek <[email protected]> Co-authored-by: Irieo <[email protected]> Co-authored-by: Fabian Hofmann <[email protected]> Co-authored-by: Bobby Xiong <[email protected]> Co-authored-by: Koen van Greevenbroek <[email protected]> Co-authored-by: Jonas Hörsch <[email protected]> Co-authored-by: lkstrp <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Bobby Xiong <[email protected]> Co-authored-by: Tom Welfonder <[email protected]>

Irieo marked this pull request as ready for review September 3, 2025 09:17

fneum previously requested changes Sep 3, 2025

View reviewed changes

fneum added this to the v1.0 milestone Sep 3, 2025

Irieo and others added 17 commits September 5, 2025 15:49

feat: add network methods for risk preferences

2d94c5b

feat: prototype first working version

762bfd5

tests: add tests for new network methods and properties

533d513

tests: add optimization tests

1a43998

- ensure strict monotonicity of CVaR objective vs omega (neutral < 0.25 < 0.5 < 0.75) - ensure CVaR omega=0 equals SP risk-neutral baseline (capacities and objective) - ensure CVaR omega=1 & alpha=1-p_worst equals deterministic run for worst-case scenario"

docs: add release notes

7091160

types: make mypy happier

b5a2d85

tests: make codecov/patch happy with tests covering CVaR for various …

06da90b

…optimization modes and component settings

docs: better docstrings for new network methods

87e331c

codecov: add tests for guards, incorrect value checks and more OPEX c…

d74772a

…omponents

revision: trim release notes

b9f5324

revision: update pypsa/network/index.py

25cf5b4

Co-authored-by: Fabian Neumann <[email protected]>

revision: relax condition for setting risk preference

2b35eb7

revision: address minor comments

5668984

revision: revert 48a1895

1a7ede3

revision: make mypy happy again

145c813

Irieo force-pushed the nomorerisks branch from 847fc95 to 145c813 Compare September 5, 2025 13:53

Update nomorerisks with latest master

0c08e52

lkstrp requested changes Sep 5, 2025

View reviewed changes

Irieo and others added 4 commits September 12, 2025 15:12

commit lkstrp suggestion to pypsa/network/index.py

c8b3288

Co-authored-by: Lukas Trippe <[email protected]>

revision: minor touch docstrings

597f65a

revision: minor touch optimize.py

c166b7f

revision: update pypsa/optimization/optimize.py

c51d097

Co-authored-by: Lukas Trippe <[email protected]>

Irieo and others added 8 commits September 12, 2025 16:04

revision: minor touch pypsa/optimization/variables.py

6fbe2b8

Co-authored-by: Lukas Trippe <[email protected]>

revision: docstrings in variables.py

279fb22

revision: capture QCP problem in test

68bcb35

revision: remove redundant docstring lines

3b2fd67

Merge branch 'master' into nomorerisks

c8bce9a

Merge branch 'master' into nomorerisks

d49ffa1

restore: require scenarios before setting risk preference

0a79662

- set_risk_preference requires set_scenarios() first - remove redundant warning in optimize.py - remove warning test, add test_set_risk_preference_requires_scenarios

revision: drop guards from optimize.py & drop related tests

bc98fa7

Irieo and others added 3 commits September 15, 2025 15:52

fix: log custom variables without dash as intended in #1353

627d4de

fix: doctest problem

e817b89

fix: types

69283e4

lkstrp approved these changes Sep 15, 2025

View reviewed changes

revision: no more RuntimeError

d381129

Irieo merged commit 7f9dd44 into master Sep 16, 2025
25 of 26 checks passed

Irieo deleted the nomorerisks branch September 16, 2025 11:13

lkstrp mentioned this pull request Sep 16, 2025

v1.0.0 Documentation #1250

Merged

51 tasks

Irieo mentioned this pull request Oct 9, 2025

docs: add user-guide for risk-averse optimization #1387

Merged

1 task

Irieo mentioned this pull request Oct 16, 2025

Add value-of-information and solution quality metrics for stochastic optimization #1408

Draft

16 tasks

FabianHofmann mentioned this pull request Oct 22, 2025

harmonize features & update optimization framework enricogiglio/PyPSA#2

Merged

Irieo mentioned this pull request Nov 14, 2025

Make model instance conveniently inspectable (separate problem prep and solving) PyPSA/pypsa-eur#1902

Merged

6 tasks

Add risk-averse optimization #1345

Add risk-averse optimization #1345

Uh oh!

Conversation

Irieo commented Sep 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Add CVaR-based risk-averse optimization

Mathematical formulation

Variables and constraints added

API for users

Backward compatibility

How to use

References

TODOs

Uh oh!

fneum commented Sep 3, 2025 • edited by Irieo Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fneum commented Sep 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lkstrp left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fneum commented Sep 5, 2025

Uh oh!

Irieo commented Sep 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Irieo commented Sep 15, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Irieo commented Sep 1, 2025 •

edited

Loading

fneum commented Sep 3, 2025 •

edited by Irieo

Loading

fneum commented Sep 3, 2025 •

edited

Loading

Irieo commented Sep 12, 2025 •

edited

Loading