Snakemake version
snakemake-minimal 9.13.4 pyhdfd78af_1 bioconda
Describe the bug
The issue occurs when running snakemake rules that fulfill the following requirements (such as in PyPSA-Eur #1675 and the corresponding discussion here: open-energy-transition/pypsa-eur#70) on Windows:
- a rule requires an input which is determined by a callable, such as a lambda function or unpack(). Called rule1 from now on.
- the output of the rule that generates this input is not dependend on a callable. Instead the output is specified in the rule file via a pathlib.Path. Called rule2 from now on.
In io._init.py class _IOFile(str, AnnotatedStringInterface): inputs and outputs that are already a pathlib.Path are converted to a posix style string:
def __new__(
cls,
file: Union[str, Path, "AnnotatedString", Callable],
rule: Optional["snakemake.rules.Rule"],
):
is_annotated = isinstance(file, AnnotatedString)
is_callable = (
isfunction(file) or ismethod(file) or (is_annotated and bool(file.callable))
)
if isinstance(file, Path):
file = str(file.as_posix())Callables are flagged, but not yet dealt with. Thus rule2 now "knows" that it has a posix style output, regardless of the OS.
The callables are evaluated in rules.py by expand_input() and _apply_wildcards(), which handle the outputs using concretize_iofile(). There pathlib.Path outputs from the callable are directly converted to strings, without converting them to posix style paths:
def concretize_iofile(f, wildcards, from_callable, incomplete):
if from_callable is not None:
if isinstance(f, Path):
f = str(f)
iofile = IOFile(f, rule=self).apply_wildcards(wildcards)As a result, rule1 is now looking for a Windows style input while rule2 only provides a posix style output.
This might be related to #3637.
Fix
This can be fixed through a minor adjustment to concretize_iofile():
def concretize_iofile(f, wildcards, from_callable, incomplete):
if from_callable is not None:
if isinstance(f, Path):
f = str(f.as_posix())
iofile = IOFile(f, rule=self).apply_wildcards(wildcards)I can create a PR for this.
Logs
Missing input files for rule add_electricity:
output: resources/Versioning/Test01/networks/base_s_5_elec.nc
wildcards: clusters=5
affected files:
data\costs\primary\v0.13.3\costs_2050.csv
Minimal example
Snakefile:
from pathlib import Path
# rule1
rule add_electricity:
input:
tech_costs=lambda w: Path("test/test") / "costs_2030.csv",
# rule2
rule retrieve_cost_data:
output:
costs=Path("test/test") / "costs_{year}.csv",Command that will fail on Windows:
snakemake -c1 add_electricity -n
Snakefile that will also fail:
from pathlib import Path
def input_add_electricity(w):
return {"tech_costs": Path("test/test") / "costs_2030.csv"}
# rule1
rule add_electricity:
input:
unpack(input_add_electricity),
# rule2
rule retrieve_cost_data:
output:
costs=Path("test/test") / "costs_{year}.csv",
Snakemake version
snakemake-minimal 9.13.4 pyhdfd78af_1 bioconda
Describe the bug
The issue occurs when running snakemake rules that fulfill the following requirements (such as in PyPSA-Eur #1675 and the corresponding discussion here: open-energy-transition/pypsa-eur#70) on Windows:
In
io._init.pyclass _IOFile(str, AnnotatedStringInterface):inputs and outputs that are already a pathlib.Path are converted to a posix style string:Callables are flagged, but not yet dealt with. Thus rule2 now "knows" that it has a posix style output, regardless of the OS.
The callables are evaluated in
rules.pybyexpand_input()and_apply_wildcards(), which handle the outputs usingconcretize_iofile(). There pathlib.Path outputs from the callable are directly converted to strings, without converting them to posix style paths:As a result, rule1 is now looking for a Windows style input while rule2 only provides a posix style output.
This might be related to #3637.
Fix
This can be fixed through a minor adjustment to
concretize_iofile():I can create a PR for this.
Logs
Minimal example
Snakefile:
Command that will fail on Windows:
Snakefile that will also fail: