Fix cutout directory on Windows by Eric-Nitschke · Pull Request #1602 · PyPSA/pypsa-eur

Eric-Nitschke · 2025-03-20T16:26:59Z

Bugfixes:

change CDIR in Snakefile to always contain forward slashes instead of depending on the OS.

Closes #1598

Changes proposed in this Pull Request

change the definition of CDIR in Snakefile to not include os.path.join anymore. More details on this can be found in Snakemake can't find retrieve_cutouts #1598.

Checklist

I tested my contribution locally and it works as intended.
Code and workflow changes are sufficiently documented.
Changed dependencies are added to envs/environment.yaml.
Changes in configuration options are added in config/config.default.yaml.
Changes in configuration options are documented in doc/configtables/*.csv.
Sources of newly added data are documented in doc/data_sources.rst.
A release note doc/release_notes.rst is added.

Bugfixes: - change CDIR in Snakefile to always contain forward slashes instead of depending on the OS.

lkstrp

Did you also try pathlib here? It is generally better for cross-platform compatibility. And we wanna leave those string paths behind.

I was under the impression that Snakemake works with "Linux-Style" paths regardless of OS

Not sure how this interferes with Snakemake. But I don't like to rely on it

fneum · 2025-03-23T09:50:00Z

I agree with @lkstrp — try pathlib. Also, the CDIR changes proposed behave differently regarding "/" suffixes. Make sure this is the same as before.

Eric-Nitschke · 2025-03-27T16:58:46Z

@lkstrp, @fneum I've tried it today with pathlib. Changing it to

CDIR = PurePosixPath(cutout_dir).joinpath("" if run["shared_cutouts"] else RDIR)

and changing every other occurance to

str(CDIR.joinpath("{cutout}.nc"))

(or similar), seemed to work in a dry-run.
Note that changing it back to a str is required by Snakemake, since all inputs, outputs, etc. have to be strings.
Additionally it is important to change all str concatenations to PurePosixPath.joinpath(), since the + operator is not supported between a PurePosixPath and str or another PurePosixPath.

Eric-Nitschke · 2025-03-28T08:07:18Z

Changing it to
CDIR = PurePosixPath(cutout_dir).joinpath("" if run["shared_cutouts"] else RDIR)
and changing every other occurance to
str(CDIR.joinpath("{cutout}.nc"))
(or similar), seemed to work in a dry-run.

It might be slighty cleaner to go with

CDIR = Path(cutout_dir, "" if run["shared_cutouts"] else RDIR)

and

CDIR.joinpath("{cutout}.nc").as_posix()

I'll try a full run with it today.

@lkstrp I've found the Snakemake pull request, that changes all Windows paths to posix paths.

Bug fix: - change CDIR in the Snakemake files to pathlib to properly function on Windows

for more information, see https://pre-commit.ci

fneum

Here's a suggestion to simplify. Should hopefully also work on Windows.

fneum · 2025-04-01T14:13:03Z

rules/build_electricity.smk

            regions_offshore=resources("regions_offshore.geojson"),
        output:
-            protected(CDIR + "{cutout}.nc"),
+            protected(CDIR.joinpath("{cutout}.nc").as_posix()),


Suggested change

protected(CDIR.joinpath("{cutout}.nc").as_posix()),

protected(CDIR / "{cutout}.nc"),

Instead of joinpath, you should be able to use the/ operator (simplifies code)

.as_posix() should also not be necessary as

In [8]: from pathlib import Path In [9]: Path("test") / "mypath" Out[9]: PosixPath('test/mypath') In [10]: Path("test").joinpath("mypath") Out[10]: PosixPath('test/mypath') In [11]: Path("") / Path("") Out[11]: PosixPath('.') In [12]: Path("") Out[12]: PosixPath('.') In [13]: "" / Path("test") Out[13]: PosixPath('test')

This minimises code changes in .smk files to changing "+" to "/".

Hey @fneum,

I agree that "/" is nicer than .joinpath() (and it works just fine on Windows).

as_posix() however has to stay for two reasons:

a PosixPath is required to resolve issue Snakemake can't find retrieve_cutouts #1598. On Windows all your examples output a WindowsPath, which leads to the issue. Another option to ensure this, would be defining CDIR as a PurePosixPath. Pure is required because using just PosixPath does not function on Windows.

input, output and log files have to be strings. At least when using a PurePosixPath this always leads to errors, if the Path is not converted to a string first (either with str() or with .as_posix()). Below you will find a few error messages I got when testing it witout converting the Paths to strings or just partially converting them to strings.

RuleException in file C:[...]\pypsa-eur\rules/retrieve.smk, line 136: Input and output files have to be specified as strings or lists of strings. RuleException in file C:[...]\pypsa-eur\rules/retrieve.smk, line 136: Log files have to be specified as strings. InputFunctionException in rule build_renewable_profiles in file C:\[...]\pypsa-eur\rules/build_electricity.smk, line 306: Error: Function did not return str or iterable of str. Encountered: [PurePosixPath('cutouts/europe-2013-sarah3-era5.nc')] (<class 'list'>) ``

coroa · 2025-04-30T20:16:01Z

Sorry for dropping in here late, but i was also fighting this path nonsense with snakemake a while ago. Snakemake is in principle fine with / as a path separator or the corresponding backslash \, if they are used uniformly.

ie a string may not mix between them.

The way CDIR as a string is used and compounded with other paths you are introducing in so many places "/" into the string after the CDIR that this is a loosing battle and you should just use a literal "/" as a separator.

That works reliably because snakemake runs on all paths os.path.normpath and that converts / to \ on windows.

At least that was my take-away.

Note that you can play around with an ipython on windows using wine:
(https://github.com/PietJankbal/powershell-wrapper-for-wine, and uv)

wget https://github.com/PietJankbal/powershell-wrapper-for-wine/raw/master/install_pwshwrapper.exe
wine install_pwshwrapper.exe
wine powershell

and then in the new powershell window:

powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

and after reopening it:

uvx ipython

Have fun.

fneum

This version is acceptable to me! Thanks for your patience dealing with Windows paths! Iaf we find a better alternative later, we can still update. We just need Windows to run as this is 90% of first-time users.

Fix cutout directory on Windows

0dd1e4d

Bugfixes: - change CDIR in Snakefile to always contain forward slashes instead of depending on the OS.

lkstrp reviewed Mar 21, 2025

View reviewed changes

Eric-Nitschke and others added 2 commits April 1, 2025 14:11

Change CDIR to pathlib

617f617

Bug fix: - change CDIR in the Snakemake files to pathlib to properly function on Windows

[pre-commit.ci] auto fixes from pre-commit.com hooks

eacba86

for more information, see https://pre-commit.ci

fneum requested changes Apr 1, 2025

View reviewed changes

fneum approved these changes May 25, 2025

View reviewed changes

fneum merged commit 063d8ae into PyPSA:master May 25, 2025
2 checks passed

fneum mentioned this pull request Nov 3, 2025

data: Intermediate layer for versioning of datasets #1675

Merged

25 tasks

Eric-Nitschke mentioned this pull request Nov 4, 2025

Adjust file paths to work on Windows open-energy-transition/pypsa-eur#70

Merged

8 tasks

euronion mentioned this pull request Nov 5, 2025

Cross-platform path handling: str(Path) breaks shell commands on Windows snakemake/snakemake#3637

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix cutout directory on Windows#1602

Fix cutout directory on Windows#1602
fneum merged 3 commits intoPyPSA:masterfrom
Eric-Nitschke:windows_cutouts

Eric-Nitschke commented Mar 20, 2025

Uh oh!

lkstrp left a comment •

edited

Loading

Uh oh!

fneum commented Mar 23, 2025

Uh oh!

Eric-Nitschke commented Mar 27, 2025 •

edited

Loading

Uh oh!

Eric-Nitschke commented Mar 28, 2025 •

edited

Loading

Uh oh!

fneum left a comment •

edited

Loading

Uh oh!

fneum Apr 1, 2025

Uh oh!

Eric-Nitschke Apr 1, 2025

Uh oh!

coroa commented Apr 30, 2025 •

edited

Loading

Uh oh!

fneum left a comment •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	protected(CDIR.joinpath("{cutout}.nc").as_posix()),
	protected(CDIR / "{cutout}.nc"),

Conversation

Eric-Nitschke commented Mar 20, 2025

Changes proposed in this Pull Request

Checklist

Uh oh!

lkstrp left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fneum commented Mar 23, 2025

Uh oh!

Eric-Nitschke commented Mar 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Eric-Nitschke commented Mar 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fneum left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fneum Apr 1, 2025

Choose a reason for hiding this comment

Uh oh!

Eric-Nitschke Apr 1, 2025

Choose a reason for hiding this comment

Uh oh!

coroa commented Apr 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fneum left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

lkstrp left a comment •

edited

Loading

Eric-Nitschke commented Mar 27, 2025 •

edited

Loading

Eric-Nitschke commented Mar 28, 2025 •

edited

Loading

fneum left a comment •

edited

Loading

coroa commented Apr 30, 2025 •

edited

Loading

fneum left a comment •

edited

Loading