* Add additional MGA functionality

Introduces new functions in optimization/abstract.py to facilitate the
analysis of trade-offs between multiple objectives/dimensions using
near-optimal methods. Allows for solving a network in a direction
given in the coordinate space of user-specified dimensions, and also
introduces a parallelized function to solve in multiple directions at
once. Three different functions generating (random) directions are
also provided.

* Fix typing

* Add test coverage for new mga functionality

* Use `is_solved` attribute

* Fix docstrings; return coordinates a pd.Series

* refactor: to `mga.py`

* refactor parallel mga to avoid pickling; robustness

Also improve docstrings and add guardrails to make parallel code more robust

* Fix mga docstring example

* fix: tests

---------

Co-authored-by: Koen van Greevenbroek <[email protected]>
Co-authored-by: Lukas Trippe <[email protected]>

* initial draft to add prices in statistics. needs further modularization.

* refined prices statistics function with tests

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Deprecate define_nominal_constraints_per_bus_carrier (#1294)

* Deprecate

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update pypsa/optimization/global_constraints.py

Co-authored-by: Lukas Trippe <[email protected]>

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lukas Trippe <[email protected]>

* fix: activate doctests (#1327)

* Add additional MGA functionality and example (#1269)

* Add additional MGA functionality

Introduces new functions in optimization/abstract.py to facilitate the
analysis of trade-offs between multiple objectives/dimensions using
near-optimal methods. Allows for solving a network in a direction
given in the coordinate space of user-specified dimensions, and also
introduces a parallelized function to solve in multiple directions at
once. Three different functions generating (random) directions are
also provided.

* Fix typing

* Add test coverage for new mga functionality

* Use `is_solved` attribute

* Fix docstrings; return coordinates a pd.Series

* refactor: to `mga.py`

* refactor parallel mga to avoid pickling; robustness

Also improve docstrings and add guardrails to make parallel code more robust

* Fix mga docstring example

* fix: tests

---------

Co-authored-by: Koen van Greevenbroek <[email protected]>
Co-authored-by: Lukas Trippe <[email protected]>

* feat: exclude inactive components from model build (#1310)

* squashed commit xarray-view

* deactivate deprecation checks

* Xarray view 2 (#1164)

* feat: introduce component class

* refactor: rename `core.py` -> `network.py`

* refactor: move data to data dir

* rename: `types.py` -> `typing.py`

* move standard_types to `ComponentTypeInfo`

* add `list_as_string()`

* feat: add component specific class

* fix: tests

* feat: add component data view

* Merge branch 'master' into xarray-view

* refactor: add explicit component definitions, add operational_attr map

* refac: move components descriptors to components module

* reset constraints.py from master

* fix: doc test

* reset parts of optimization/variables.py

* start converting variables

* feat: make as_xarray robust for scenario dim

* fix: ignore type for nominal attr in Loads

* revert test on snapshot weightings index name

* fix indentation

* fix: iterate over components values in set_investment_persiods

* review

---------

Co-authored-by: lkstrp <[email protected]>

* adjust components methods

* make methods internal

* refactor: components methods import

* Refactor optimization code (#1166)

* make get_bounds_pu support DataArrays

* refactor: operational constraints for non-ext

* refactor: operational constraints for ext

* refactor: nominal constraints for ext

* patch: FH feedback

* refactor: make get_bounds_pu individual for component classes

* refactor: nodal balance constraint

* fix type checker issues (mypy)

* refactor: operational constraints for committables

* tests: cover the case with ramp_rates & committable

* refactor: ramp limit constraints 1/2

* fix: correct status_prev in ramp limit constraints

* refactor: ramp limit con 2/2, handle rolling horizon properly

* fix: minor correction of types in func signatures

* refactor: fixed operation cons & add related tests

* revert: fixed operation cons (to fix p_set issue)

* tests: add test covering case with p_set set to zero

* fix: handle p_set default values for lopf case

* fix: update p_set default values to 'n/a' and make exception for power flow calculations

* fix: update shadow price attributes to default to 'n/a' in component attribute files

* refactor: fixed operation cons (incl. storage level)

* add test for fixed operation value of extendable component, fix cons name

* refactor: modular constraints

* further components refactor

* apply components convention

* refactor: storage_unit_constraints

* refactor: store_constraints, improve docstrings

* refactor: define_loss_constraints, add docstring

* update docstring based on numpy style guide

* docstrings: add good docs everywhere

* refactor: total_supply_constraints

* remove `-ext`, `-com`, `-fix`

* remove previous helpers

* adjust deprecations

* rename component dim name (`c.name` to `"component"`)

* fix: fix SCLOPF broken with 22a9c3e

* fix tests

* refactor: objective, add docstring

---------

Co-authored-by: lkstrp <[email protected]>

* initial commit

* remove scenarios

* Revert "remove scenarios"

This reverts commit 20b0264.

* fix

* fix types

* fix: capex broadcasting problem introduced in #1166 and not covered by tests

* add `set_scenarios` interface

* add tests

* add scenarios info to __repr__

* tests: simple fixture and test placeholders

* make as_xarray work with scenarios

* tests: add benchmark solved problem, add fixture, simplify properties test

* Revert "make as_xarray work with scenarios"

This reverts commit 343d989.

* put scenario back on columns axis for `n.dynamic`

* make as_xarray work again based on different axis

* fix renamed index in `define_nodal_balance_constraints`

* fix types

* some refinement

* fix: test_bugs run through

* fix: tests -- handle networks with no loads

* tests: mute unfinished tests for now

* add stochastic network fixture

* add network cycles function

* refactor kvl constraint to use cycles function

* fixes: many fixes to make scenario network work

* wip: handle case when no components and scenarios present

* feat: support kvl constraints for stochastic network

* add network cycles function

* refactor kvl constraint to use cycles function

* fix: remove breakpoint

* fix: skip kvl constraints if no cycles there

* wip: handle assign solution

* wip: prep tests

* fix spatial clustering

* stoch: correct objective weights with scen probabilities

* more post_processing fixes

* remove tables pin

* Revert "stoch: correct objective weights with scen probabilities"

This reverts commit 44bfb7c.

* stoch: correct objective weights with scen probabilities (now with KVL)

* fix: write back p_nom_opt to network

* add full test for stoch problem obj, dispatch, status

* tests: mute mutiinvest tests until KVL is fixed

* final tests fixed

* more final tests and fix types

* fix: make proper snapshot selection in kvl constraint

* fix: statistics for stoch networks

* refactor: Remove unused 'as_xarray' parameter in get_bounds_pu

* feat: add xarray accessor for attribute style lazy access

* refac: make adajacency matrix backwards compat

* fix: tests

* fix: Handle NaN p_set in LPF and fix allocation function call. Fixes test_lpf_ac_dc.py::test_lpf failure

* test: Add stochastic variable/constraint dimension test

* test: Add stochastic multiperiod optimization test

* test: minor cleanup

* fix: adds stochastic network support to growth limits

* tests: add single-scenario stochastic network test

* feat: add support for nested members of collection

* fix: types

* fix: missed new index name

* fix: tests

* fix: ensure correct column names in get_switchable_as_dense for stoch networks

* fix: set index names in lpf_contingency

* refactor: operational attrs and bounds pu redefinment

* docs: add component/attr scope into constraint functions

* feat: make adjacency matrix fully backwards compatible

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix: udpate doc example

* fix: pre-commit

* fix: types

* fix: docs build

* fix: treat special case with aux dim in dual variable assignment (fix for failing tests for security-constrained optimization)

* fix: extended pre-commits

* fix: support multi-dimensional duals in security-constrained lopf

* fix: resolve problems dual assignment
- we treat here 2D cases with aux dimensions ('cycle' or sclopf-related stuff)
- patch from_xarray to treat all cases
- refactor assign_duals() so it can be read by humans

* fix: TypeError in stochastic optimization constant handling

* fix: doctest failures in networks.py (skip examples requiring optimized network)

* fix: types

* refactor: rename new index names `component` -> `name`

* refactor: use `c.da` instead of `c.as_xarray`

* fix: correct calculation of objective constant term

* fix: make 82f51d8 work

* fix: reserve power example

* docs: remove old stochastic example

* fix: correct dimension name in loss constraints

* feat: add consistency check and unit test for scenario probabilities

* feat: add scenario invariant attributes check and related unit tests

* wip: add io for `.nc`

* add example network

* fix: types

* fix: stochastic network handling bus 'control' attr & rework consistency checks
- fixes test_optimization_simple() test fail because bus 'control' attribute
  varied across scenarios, violating consistency check
- context: Bus 'control' is derived from generators during topology determination,
  not user-set
- Remove 'control' from invariant_attrs in check_scenario_invariant_attributes()
  since it's a user-set attr
- feat power_flow.py -> find_slack_bus() to select the same slack generators in
  stochastic networks
- Add dedicated consistency check to ensure same slack bus across scenarios
- Add unit tests for all these things

* refactor: align and adjust how index data is stored

* fix: types

* fix: excel io

* fix: missing `obj` col in for sub_networks

* fix: stochastic network support for post-processing calculations
- Skip power flow calculations for stochastic networks in optimize.py post_processing()
- Fix determine_network_topology() to handle MultiIndex structure of stoch network by workign with first scenario only
(possible since consistency_check() ensures all scenarios have the same topology)
- Fix find_cycles() to access scenarios via sub_network.n.scenarios instead of sub_network.scenarios

* fix: stoch optimization with Store component bug -> handle MultiIndex dime in elapsed hours. Add two unit tests.

* Fix scenario ordering bug in stochastic optimization
- Ensure scenario coordinates preserve correct order in DataArrays
- Add tests for this

* make mypy happy

* fix: compatible multiperiod stochastic optimization 🐉
• Fix broadcasting error in storage unit constraint when multiperiod invest is enabled
• Add many tests for multiperiod stochastic optimization
• Test storage unit & multiinvest & stochastic network issue specifically

* add test for determine_network_topology

* refac: make determine_network_topology compatible with multiindexed components

* fix io: keep column order in determine_network_topology

* refac: write it a tiny bit more compact

* Revert "refactor: use `c.da` instead of `c.as_xarray`"

This reverts commit 401f641.

* Revert "feat: add xarray accessor for attribute style lazy access"

This reverts commit 28bd950.

* fix: tests

* Revert "Revert "feat: add xarray accessor for attribute style lazy access""

This reverts commit e48f5f1.

* Revert "Revert "refactor: use `c.da` instead of `c.as_xarray`""

This reverts commit 2e68217.

* Revert "fix: tests"

This reverts commit be7c8fd.

* perf: fix memory issues in `_as_dynamic`

* perf: wrap `_as_dynamic` in `get_switchable_as_dense`

* perf: optimize subset validation

* perf: lazy load sklearn `HAC`

* perf: remove redundant reindex

* refac: remove inspect-based power flow detection in array module

* test: skip SCLOPF test on Python 3.10 due to unstable numerics

* revert: skip SCLOPF test on Python 3.10 due to unstable numerics

* test: tighten solver tolerances as honest attempt to fix numerical instability of SCLOPF

* fix: exclude standard types from scenario broadcasting

* Revert "fix: exclude standard types from scenario broadcasting"

This reverts commit aaa7018.

* wip: fixing unaligned index names

* feat: treat MultiIndex case of line_types in stochastic networks
- Fix apply_line_types() to handle MultiIndex case
- Add check_line_types_consistency() to ensure line_types are identical across scenarios
- Add tests for line_types consistency check

* fix: handle meshed bus index in optimize.py for stoch networks

* fix: coordinate alignment in case of multiperiod + stoch + storage unit
- fix in constraints.py
- remove TODO from stoch hackaton period
- add unit test

* fix: correct dims of total_supply_constraints for stochastic networks

* feat: make primary/operational limit (global constraints) compatible with scenarios

* feat: add stoch opt support to growth limit constraints
- we find and apply strictest (minimum) max_growth and max_relative_growth values across scenarios
- add unit tests to ensure that growth limits are applied and applied correctly
- reuse our helper _extract_names_for_multiindex to cut lines of code
- revert bebff25 which I admit was dumb

* add test on primary energy limit

* apply primary energy limit to individual scenarios

* raise error for non supported global constraint (stoch case)

* feat: add test for operational limit (global constraint)

* refactor: remove `drop_scenarios` argument from `_as_xarray`

* refactor: clean up `_as_xarray`

* feat: add runtime verification option and simplfy `_as_xarray`

* fix: doctests

* fix: remove redundant func in global constraint & refine unit tests for growth limits

* fix: add missed module

* fix types and add tests

* fix: attribute error in `define_operational_limit`

* feat: add not implemented decorator for clustering

* test: add test for `c.ds`

* refactor: remove unused helper methods and small fixes

* refactor: more minor changes

* fix: buggy storage constraints RHS sorting.

* test: add unit test for storage unit RHS problem

* test: improve tests and `from_xarray` write back

* fix: unneccessary except

* fix: flaky doctest

* test: ensure that duals are correctly assigned in stoch networks.

* fix: relax tolerance in scenario sum probability check. Avoids rounding error if scenarios are passed via list.

* Reverted line_types_to_use to a2213ee, as code refactoring introduced bug.

* refactor: let uc status always return single dim name index

* feat: exclude inactive components from model build

* Update pypsa/networks.py

Co-authored-by: Fabian Neumann <[email protected]>

* refactor primary energy limit to use scenario dim

* fix: correct rhs in multi-invest & storage & global_cons case caused by 171dc0f. Add missing test for --all-- + stochastic network

* fix: write back duals for static/dynamic  global constraints

* fix: avoid integer-location index in multi-invest objective

* tests: make stoch solution test better, remove redundant

* tests: more patches to stoch unit tests

* tests: check that ramp_limit_start_up attr works as intended

* refactor: align operational_limit constraint logic with 171dc0f
- We build one Linopy expression per scenario grouped by cons name
- Merge them into a single constraint with a scenario dimension
- Add this as single constraint into n.model
- The same logic as in constraints.py

* tests: update assertions to reflect changes in 6626f7f

* feat(global-constraints): make transmission_volume_expansion_limit stochastic-aware

Implement scenario dim in define_transmission_volume_expansion_limit. Build per-scenario expression over extendable Lines/Links filtered by carrier. Merge into a single constraint per name (GlobalConstraint-<name>). Remove NotImplementedError. Add test: test_transmission_volume_expansion_limit_constraint_stochastic.

* test: add network data guard for optimize and conistency

* fix: small changes from review

* fix: fix and improve assign_duals behavior

* fix: update usage of `unique` to `drop_duplicates` to keep order

* fix: linear power_flow

* test: add more (inactive) guards

* fix: default investment_periods not assigned

* fix: pre-commit

* fix: removed SubNetwork `obj` during io

* remove comment line with redundant TODO

* Update pypsa/networks.py

Co-authored-by: Fabian Neumann <[email protected]>

* ci: log installed versions

* fix: typo in merge

* fix: types

* rename cycles to cycle_matrix, adjust docstring

* add unit test placeholder to catch dual assignemnt problem

* fix: assign duals for GlobalConstraints

* tests: add assertions to placeholder (temporary v. until assign_duals debugged)

* fix: types

* better solution write back, test and remove also from masks

* Add additional activity mask tests

* activity mask fixes

* additional fix to store constraints building

* Fix global constraints in case of activity mask; add test

---------

Co-authored-by: Irieo <[email protected]>
Co-authored-by: Fabian Hofmann <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Bobby Xiong <[email protected]>
Co-authored-by: Fabian Neumann <[email protected]>
Co-authored-by: Koen van Greevenbroek <[email protected]>

* fix: rename strongly meshed buses to Bus (#1304)

* fix: rename strongly meshed buses to Bus

Follow-up to #1274 and fix for PyPSA/linopy#470.

* Update release-notes.rst

---------

Co-authored-by: Lukas Trippe <[email protected]>

* feat: full support for current and new components API (#1329)

* rename option

* rename static data

* rename dynamic data

* add pytest flag `--new-components-api`

* ci: add ci job for new api

* fix: ci

* ci: run for all versions

* fix

* Update pypsa/plot/maps/interactive.py

Co-authored-by: Fabian Neumann <[email protected]>

* ignore FutureWarning

---------

Co-authored-by: Fabian Neumann <[email protected]>

* fix: small fix for new api (#1331)

* prepare release `v0.35.2`

* prepare release `v1.0.0rc1`

* [github-actions.ci] prepare release v1.0.0rc1

* fix: temporary ignore raised 1.0 deprecations (#1334)

* build(deps): bump the github-actions group with 2 updates (#1344)

Bumps the github-actions group with 2 updates: [actions/checkout](https://github.com/actions/checkout) and [actions/download-artifact](https://github.com/actions/download-artifact).


Updates `actions/checkout` from 4 to 5
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](actions/checkout@v4...v5)

Updates `actions/download-artifact` from 4 to 5
- [Release notes](https://github.com/actions/download-artifact/releases)
- [Commits](actions/download-artifact@v4...v5)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '5'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions
- dependency-name: actions/download-artifact
  dependency-version: '5'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* add additional line types from pandapower (#1342)

* Bugfix: Handle alphanumeric versions (like "1.0.0rc1") (#1338)

* Bugfix version.py for alphanumeric versions like 1.0.0rc1

* Extended unit tests by alphanumeric versions and parse_version_tuple_function

* Using packaging.version to do version comparison

* Fixed conversion back to str.

* refactor: refactor version attributes and remove tuple versions

* fix: simplify regex version match

---------

Co-authored-by: lkstrp <[email protected]>

* fix: update url for carbon management file (#1346)

* feat: improved optional dependency handling (#1348)

* feat: make return of added component names optional (#1347)

* feat: make return of added component names optional

* feat: add default option

* Fix custom variable bug (#1353)

* add test test_assign_custom_variable

Tests if the optimization raises an error after adding a custom variable to the linopy model.

* fix: handle custom variable names (#1351)

Solutions of custom variables in the linopy model that can not be mapped to a network component are skipped.

Log message for user, to allow user to understand why the solution of the custom variable was not mapped.

* test: add test for warnings

---------

Co-authored-by: lkstrp <[email protected]>

* Remove 3.13 dependency pins (#1355)

* refactor: resolve deprecations and make components iter compatible (#1349)

* fix: address deprecations in network mixins

* Fix more deprecations

* Address more deprecations (claude supported)

* Add warnings filters for network collection get_active_assets

* Add pytest warns fences for testing old components api

* Revert "fix: temporary ignore raised 1.0 deprecations (#1334)"

This reverts commit 1871897.

* refactor: resolve deprecations and make components iter compatible

* fix: types

* fix: don't raise new API deprecation warnings

* test: now check for warnings instead of errors

* mute assertion in test_sclopf_scigrid, see #1356

* fix: consistent ordering during iter

* fix: for inconsistencys

---------

Co-authored-by: Jonas Hoersch <[email protected]>
Co-authored-by: Irieo <[email protected]>

* Add risk-averse optimization (#1345)

* feat: add network methods for risk preferences

* feat: prototype first working version

* feat: move CVaR constraints to constraints.py

CVaR linearization requires per-scenario OPEX. Since OPEX is assembled inside define_objective, constraints must reconstruct the same OPEX terms. The prototype version in 558afb7 has much less code, but goes against architecture principles. I could make shared OPEX builder to avoid duplication but I don't like this idea too much.

* tests: add tests for new network methods and properties

* tests: add optimization tests
- ensure strict monotonicity of CVaR objective vs omega (neutral < 0.25 < 0.5 < 0.75)
- ensure CVaR omega=0 equals SP risk-neutral baseline (capacities and objective)
- ensure CVaR omega=1 & alpha=1-p_worst equals deterministic run for worst-case scenario"

* docs: add release notes

* types: make mypy happier

* tests: make codecov/patch happy with tests covering CVaR for various optimization modes and component settings

* feat: add a guard for quadratic marginal costs
CVaR aux constraints a(s) ≥ OPEX(s) − θ are linear only if OPEX is affine. When marginal_cost_quadratic is there, we get a QCP problem. Here we add a showstopper in define_cvar_constraints that raises an error and tells to remove or approximate quadratic costs.

* docs: better docstrings for new network methods

* codecov: add tests for guards, incorrect value checks and more OPEX components

* revision: trim release notes

* revision: update pypsa/network/index.py

Co-authored-by: Fabian Neumann <[email protected]>

* revision: relax condition for setting risk preference

* revision: address minor comments

* revision: revert 48a1895

* revision: make mypy happy again

* Update nomorerisks with latest master

* commit lkstrp suggestion to pypsa/network/index.py

Co-authored-by: Lukas Trippe <[email protected]>

* revision: minor touch docstrings

* revision: minor touch optimize.py

* revision: update pypsa/optimization/optimize.py

Co-authored-by: Lukas Trippe <[email protected]>

* revision: minor touch pypsa/optimization/variables.py

Co-authored-by: Lukas Trippe <[email protected]>

* revision: docstrings in variables.py

* revision: capture QCP problem in test

* revision: remove redundant docstring lines

* restore: require scenarios before setting risk preference
- set_risk_preference requires set_scenarios() first
- remove redundant warning in optimize.py
- remove warning test, add test_set_risk_preference_requires_scenarios

* revision: drop guards from optimize.py & drop related tests

* fix: log custom variables without dash as intended in #1353

* fix: doctest problem

* fix: types

* revision: no more RuntimeError

---------

Co-authored-by: Fabian Neumann <[email protected]>
Co-authored-by: Lukas Trippe <[email protected]>

* feat: more default params options for optimization module (#1359)

* feat: more default params options for optimization module

* fix: tests

* Add undocumented bus attribute of GlobalConstraint (#1293)

* Add undocumented bus attribute of GlobalConstraint

* fix: make optional

---------

Co-authored-by: Lukas Trippe <[email protected]>

* Optimize _sort_attrs for already ordered axes (#1362)

* Optimize _sort_attrs for already ordered axes

* Add release note

* Further refactor _sort_attrs to just take column list, not dataframe

* refactor: refine statistic arguments (#1358)

* refactor: refine statistic arguments

* docs: add release note

* adjust prices and statistics

* wip

* improved schema for statistics plots

* rename baseline files

* test: adjust mpl tests and upload comparison

* fix: path

* fix

* fix windows tests

* fix: rename missed baseline files

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Fabian Neumann <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lukas Trippe <[email protected]>
Co-authored-by: Koen van Greevenbroek <[email protected]>
Co-authored-by: Koen van Greevenbroek <[email protected]>
Co-authored-by: Irieo <[email protected]>
Co-authored-by: Fabian Hofmann <[email protected]>
Co-authored-by: Bobby Xiong <[email protected]>
Co-authored-by: Koen van Greevenbroek <[email protected]>
Co-authored-by: Jonas Hörsch <[email protected]>
Co-authored-by: lkstrp <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Bobby Xiong <[email protected]>
Co-authored-by: Tom Welfonder <[email protected]>