* Revert "Transition from stores to storage units for LTES, introducing energy-…" (PyPSA#1520)

This reverts commit 0e6a8f8.

* use pypi `pycipopt` (PyPSA#1524)

* Update pinned environment files for all platforms (PyPSA#1523)

Co-authored-by: lkstrp <[email protected]>

* fix: ensure correct dtype of "reverse" column (PyPSA#1525)

* fix: ensure correct dtype of "reverse" column

* apply boolean dtype to reverse column in perfect-foresight

* reinsert objective check

* fix: improve selection of bidirectionals

* fix: relax condition on sanitizing "reversed" column

* revert some unneeded changes

* debug: print dataframes in summary (to be reverted)

* filter empty dfs in make summary

* use highs-simplex instead of highs-default for ci

---------

Co-authored-by: Lukas Trippe <[email protected]>

* fix(add_electricity): Fix nuclear p_max_pu time-series for greenfield nuclear (PyPSA#1519)

If nuclear is only part of extendable carriers, but not of convential carriers,
then the typical "conventional: nuclear: p_max_pu" setting writes the file name
all over the p_max_pu time-series, breaking the add_electricity rule.

Co-authored-by: Fabian Hofmann <[email protected]>

* solve_network: add customisable logging frequency (PyPSA#1521)

Co-authored-by: Fabian Hofmann <[email protected]>
Co-authored-by: Fabian Neumann <[email protected]>

* Configurable cutout directory (PyPSA#1515)

* make cutout dir configurable

* feat: add global cutout_directory to config

* add release notes

* feat: add description to configtables

---------

Co-authored-by: Fabian Neumann <[email protected]>

* remove `highspy` pin (PyPSA#1528)

* remove `highspy` pin

* Update environment.yaml

* Update environment.yaml

---------

Co-authored-by: Fabian Hofmann <[email protected]>

* Update pinned environment files for all platforms (PyPSA#1533)

Co-authored-by: lkstrp <[email protected]>

* Fixing snakemake shadow directory bug that occures with Windows/SMB drives (PyPSA#1535)

* Fixing snakemake shadow directory bug that occures with Windows/SMB/network drives

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* [pre-commit.ci] pre-commit autoupdate (PyPSA#1526)

updates:
- [github.com/astral-sh/ruff-pre-commit: v0.9.3 → v0.9.4](astral-sh/ruff-pre-commit@v0.9.3...v0.9.4)
- [github.com/codespell-project/codespell: v2.4.0 → v2.4.1](codespell-project/codespell@v2.4.0...v2.4.1)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Fabian Hofmann <[email protected]>

* Update validate.yaml (PyPSA#1536)

* Update validate.yaml

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update validate.yaml

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Consistent function scope (PyPSA#1537)

* start removing snakemake from functions

* move snakemake objects out of prepare_sector_functions

* continue with add_existing_baseyear

* continue with prepare_perfect_foresight

* apply it to add_electricity

* continue with solve network

* add pylint check to reusable functions

* iron out follow up issues

* adjust pylint check

* Update pinned environment files for all platforms (PyPSA#1538)

Co-authored-by: FabianHofmann <[email protected]>

* Increase default mem_mb for config.default scope (PyPSA#1541)

* Add a note to avoid updating pins without testing (PyPSA#1545)

* [pre-commit.ci] pre-commit autoupdate (PyPSA#1539)

updates:
- [github.com/astral-sh/ruff-pre-commit: v0.9.4 → v0.9.6](astral-sh/ruff-pre-commit@v0.9.4...v0.9.6)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Fabian Neumann <[email protected]>

* Update pinned environment files for all platforms (PyPSA#1544)

Co-authored-by: lkstrp <[email protected]>

* Fix geothermal heat potentials (PyPSA#1516)

* feat: refactor build_heat_source_potentials into build_geothermal_heat_potential and and use ISI Excel output as data source

* feat: remove unnecessary snapshot index

* chore: refactor accordingly

* feat: fix scaling factor

* feat: remove obsolete retrieve

* docs: update docs

* docs: update release notes

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* feat: use retrieve separate retrieve rules

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* feat: remove conda env from new retrieve rules

* fix file names

* fix config passing

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* style: remove snakemake access in add_heat

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* [pre-commit.ci] pre-commit autoupdate (PyPSA#1552)

* Change heat_vent config.default setting (PyPSA#1556)

* change central_heat_vent in config.default to true, since water tank charger and discharger cant be used as heat vent anymore with branch ltes_stores

* change release notes

* corrected release notes

* fix(benchmark): ignore keyboard interrupt in memory logging subprocess (PyPSA#1555)

ensures a C-c is not consumed by the logging child process, but instead
properly terminates the solver or main python process.

* feat(ci): Remove defaults channel from conda setup and use strict channel prio (PyPSA#1554)

* feat(ci): Remove defaults channel from conda setup and use strict channel prio

With the default settings the generated pinned environments may still contain
references to the defaults channel in certain cases (currently to zstandard),
and the test workflow does not fail because it also falls back to the
defaults channel.

* Use miniforge version

Avoids dependencies on proprietary anaconda pkgs channels

* Update pinned environment files for all platforms (PyPSA#1553)

Co-authored-by: coroa <[email protected]>

* Updated eia hydro generation and capacity csv file to include data up to 2023 (PyPSA#1543)

Updated the eia hydro generation and capacity up to 2023

Co-authored-by: Lukas Trippe <[email protected]>
Co-authored-by: Fabian Neumann <[email protected]>

* extend pre-built cutout years available (PyPSA#1559)

* fix(solve): Fix planning_horizons in solve_network (PyPSA#1560)

* fix: the planning_horizons params in CCL need to be string

* docs: Update the docstring for planning_horizons.

* fix: update planning_horizons parameter to use config_provider in perfect foresight

* fix(solve): Consolidate use of planning horizons

In perfect foresight, the planning_horizons param remains the list
of all planning horizons (to trigger re-runs).

The current planning horizons is always taken from the planning_horizons
wildcard and passed into extra_functionality as a partial kwarg.

* Add a specific not implemented AssertionError for CCL in perfect foresight

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: yechenyan <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* [pre-commit.ci] pre-commit autoupdate (PyPSA#1564)

updates:
- [github.com/astral-sh/ruff-pre-commit: v0.9.7 → v0.9.9](astral-sh/ruff-pre-commit@v0.9.7...v0.9.9)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* refactor: Rename "fixed" to "capital_cost" for annualised investment … (PyPSA#1566)

* refactor: Rename "fixed" to "capital_cost" for annualised investment costs in sector-coupled networks

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Follow-up to PyPSA#1532 (PyPSA#1565)

* overwrite cost table attributes specified in config in prepare_costs and load_costs (#23)

* overwrite cost table attributes specified in config in prepare_costs and load_costs

* add custom fuel prices from EEE project

---------

Co-authored-by: Thomas Gilon <[email protected]>
(cherry picked from commit 926f763)

* Add logging to existing code

* Document the changes

* Fix issues with configuration

* Add example in the default configuration file

* Add release note

* Remove fuel cost suggestion

* feat(costs): Add support for overwriting investment, lifetime, FOM, VOM, efficiency, and fuel attributes in configuration

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* refactor(config): simplify overwrites structure in default configuration

* refactor(costs): streamline overwriting of marginal and capital costs in configuration

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix(prepare_sector_network): iterate over overwrites dictionary items for clarity

* fix(prepare_costs): ensure overwrites are applied only to existing costs

---------

Co-authored-by: Daniel Rüdt <[email protected]>
Co-authored-by: Thomas Gilon <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Allow heat-venting in all heating systems and impose small penalty for water tank chargers (PyPSA#1563)

* feat: add marginal cost for water tank chargers

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* docs: update add_retrofit_gas_boiler_constraint docs

* docs: update return type annotation for get_heat_system_type method

* feat: add heat vents for decentral and rural heating

* feat: remove unnecessary central_heat_vent config

* feat: pass heat vent marginal cost to prepare_sector_network

* refactor: rename marginal_cost_storage to marginal_cost_home_battery_storage

* docs: update configtables

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* docs: update release notes

* feat: use :config:cost:marginal_cost where possible

* Apply suggestions from code review

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Fabian Neumann <[email protected]>

* [pre-commit.ci] pre-commit autoupdate (PyPSA#1569)

updates:
- [github.com/astral-sh/ruff-pre-commit: v0.9.9 → v0.9.10](astral-sh/ruff-pre-commit@v0.9.9...v0.9.10)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* enh: enable running mock_snakemake from a separate workdir (PyPSA#1571)

* enh: enable running mock_snakemake from a separate workdir

Working directories are a simple mechanism of snakemake to have
multiple not competing runs of the same workflow.

Data, config and logfiles are expected relative to the directory
specified with `--directory`, while all the source files (mainly script
snakemake and environment definitions are resolved relative from the
current snakefile).

A new working directory can be quickly set up:
```shell
mkdir workdir
cp -lr -t workdir --parents cutouts $(git ls-files data)
cp -r config workdir
```

Running it then works in the repo root like
```shell
snakemake --directory workdir ...
```

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: Jonas Hoersch <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Cluster network by shapes (NUTS0, NUTS1, NUTS2, NUTS3, and ADM1) (PyPSA#1502)

* Updated shapes to high-res 1M NUTS2024 shapes and geoboundaries for non-NUTS countries (UA, MD, BA).

* Updated nuts to 2021 and non-nuts to gadm.

* Updated non NUTS country adm 1 boundaries to OSM via Overpass (MD, BA, UA, XK

* Minor order update in rule.

* Updated eurostat population data

* Updated dataset to JRC ARDECO 2021, including UK, RS, CH

* Finished new implementation of build_shapes.

* Small fix.

* Running sector-coupling workflow

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Updated build_shapes

* Added release nots and updated data-bundle docs.

* Added sandbox databundle for temporary CI testing.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix in build_shapes: Build only for selected countries in config.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Removed jrc-data from repo, use REST API instead. removed params from build_osm_boundaries.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Re-added shape/polygon simplification.

* Made build_osm_boundaries more robust, if wrong country components are retrieved. Reduced nuts shapes to 03m resolution.

* Updated build_shapes with simplifying europe function to remove remote islands.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Updated back to 01m resolution. takes longer.

* Removed test_build_shapes from testing, as outdated and not compatible with shapes PR

* Readded test_build_shapes.py, however reduced testing, as 'countries()' does not exist anymore, replaced by create_regions().

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix for major Danish islands and Kopenhagen.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Initial working commit.

* Initial working commit.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Finalised PR.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Reversed default mock_snakemake params.

* Reversed prepare_network.py mock_snakemake

* Updated workflow: Voronoi cell generation within admin shapes.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Code clean in cluster_network.py

* Reverted mock snakemake to default.

* Fixed deprecated typing.

* Implemented robust merging algorith for clustering substation-less regions.

* Finalised: Updating x,y coordinates to internal point; Always aggregating London area.

* Bug fixes.

* Reversed mock_snakemake.

* Added in simplify_network: Stub-removal only within admin regions.

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Update common.smk (PyPSA#1577)

Wrong variable description in has_internet_access (mentioned Google and should be Zenodo)

* Windows snakefile (PyPSA#1575)

* Bugfix wdpa downloads on windows

Bugfix:
- add special handling to the rules download_wdpa and download_wdpa_marine in retrieve.smk, since Linux shell commands don't work on Windows.

* Remove unused lines of code

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Switch wdpa downloads to shutil

Minor changes:
- change the download wdpa rules in rules/retrieve.smk to shutil to properly work on Windows

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* clustering with NUTS3 in Spain breaks (PyPSA#1580)

* Update base_network.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Minor bugfix in contains columns: Only unique IDs

* Dropped redundant else: ...

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: bobbyxng <[email protected]>
Co-authored-by: Bobby Xiong <[email protected]>

* match CH and any number of digits (PyPSA#1585)

* Add PTES and introduce PTES/TES energy-to-power ratios (PyPSA#1546)

* introduce etpr for TES using stores

* add constraints

* copleted implementation of constraints for TES stores

* updated release notes

* adjusted mock snakemake default

* library change

* update release notes

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Delete .DS_Store

* change retrieval of etpr values in etpr_constraint

* changed TES constraints check based on the configs

* constraint checks

* enhance error handling for the tes constraints

* small changes

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* changed mock snakemake back to default settings

* feat: add marginal cost for water tank chargers

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* docs: update add_retrofit_gas_boiler_constraint docs

* docs: update return type annotation for get_heat_system_type method

* feat: add heat vents for decentral and rural heating

* feat: remove unnecessary central_heat_vent config

* feat: pass heat vent marginal cost to prepare_sector_network

* refactor: rename marginal_cost_storage to marginal_cost_home_battery_storage

* docs: update configtables

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* docs: update release notes

* add suggestions to ensure that charger and discharger align for charger ratio constraints and energy power constraints

* add lifetime to PTES and TTES charger/discharger, so that they are built with correct baseyear

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* feat: add small penalty on ptes charging

* feat: update fixed cost to capital cost str

* feat: remove unnecessary passing of obsolete config

* feat: remove unneeded config passing

* feat: remove unneeded config passing

* feat: remove unneeded config passing

* Update scripts/solve_network.py

* Update solve_network.py

* feat: re-add TES necessary parameters

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Refactor string contains checks

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Amos Schledorn <[email protected]>
Co-authored-by: Amos Schledorn <[email protected]>

* Update pinned environment files for all platforms (PyPSA#1588)

Co-authored-by: lkstrp <[email protected]>

* [pre-commit.ci] pre-commit autoupdate (PyPSA#1592)

updates:
- [github.com/astral-sh/ruff-pre-commit: v0.9.10 → v0.11.0](astral-sh/ruff-pre-commit@v0.9.10...v0.11.0)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Failing on 'warning' solving status (PyPSA#1591)

* Reduce pipeline entrypoints to country coverage / exclude RU+BY (PyPSA#1582)

* handle pipeline entrypoints for reduced country coverage

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

---------

Co-authored-by: Lukas Trippe <[email protected]>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: lkstrp <[email protected]>
Co-authored-by: Fabian Hofmann <[email protected]>
Co-authored-by: Jonas Hörsch <[email protected]>
Co-authored-by: Philipp Glaum <[email protected]>
Co-authored-by: Fabian Neumann <[email protected]>
Co-authored-by: Bobby Xiong <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: FabianHofmann <[email protected]>
Co-authored-by: Amos Schledorn <[email protected]>
Co-authored-by: TomKae00 <[email protected]>
Co-authored-by: Jonas Hörsch <[email protected]>
Co-authored-by: coroa <[email protected]>
Co-authored-by: Lukas <[email protected]>
Co-authored-by: yechenyan <[email protected]>
Co-authored-by: Daniel Rüdt <[email protected]>
Co-authored-by: Jonas Hoersch <[email protected]>
Co-authored-by: Markus Groissböck <[email protected]>
Co-authored-by: Eric Nitschke <[email protected]>
Co-authored-by: Cristóbal José Gallego Castillo <[email protected]>
Co-authored-by: bobbyxng <[email protected]>
Co-authored-by: pworschischek-aggmag <[email protected]>
Co-authored-by: Amos Schledorn <[email protected]>