fix: correct Windows callable path handling by Eric-Nitschke · Pull Request #3832 · snakemake/snakemake

Eric-Nitschke · 2025-11-07T10:59:34Z

Bugfix:

fix path handling on Windows when using callables for input/output files in rules by converting Path objects to POSIX-style strings

QC

The PR contains a test case for the changes or the changes are already covered by an existing test case.
The documentation (docs/) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).

Summary by CodeRabbit

Bug Fixes
- Improved cross-platform path handling so input file paths are consistently formatted when workflows use callables.
Tests
- Added an automated workflow test and accompanying test input to validate the callable-based input handling.

Bugfix: - fix path handling on Windows when using callables for input/output files in rules by converting Path objects to POSIX-style strings

coderabbitai · 2025-11-07T10:59:47Z

📝 Walkthrough

Walkthrough

Convert pathlib.Path values returned from callables into POSIX-style strings (via as_posix()) when concretizing inputs in expand_input, so IOFile construction receives normalized paths across platforms.

Changes

Cohort / File(s)	Summary
Path normalization in callable handling `src/snakemake/rules.py`	Change `concretize_iofile()` so that when `from_callable` is set and `f` is a `Path`, `f` is converted with `f.as_posix()` before creating the `IOFile`.
New test workflow and input `tests/test_callable/Snakefile`, `tests/test_callable/inputs/test.in`	Add a small Snakemake workflow (uses `pathlib.Path` and `shutil.copy2`) and a test input file containing "test" to validate callable/path handling on different OSes.

Sequence Diagram(s)

sequenceDiagram
    autonumber
    participant Rules as rules.expand_input
    participant Concretize as concretize_iofile
    participant IO as IOFile

    Note over Rules,Concretize: Input value originates from a callable
    Rules->>Concretize: pass f (Path), wildcards, from_callable=True
    alt f is pathlib.Path
        Concretize-->>Concretize: convert f -> f.as_posix()  %%{style:dashed,stroke:`#1f77b4`}%%
    else f is str or other
        Concretize-->>Concretize: leave f unchanged
    end
    Concretize->>IO: construct IOFile(f) and apply_wildcards
    IO-->>Rules: concrete input string (POSIX-normalized if Path)

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Changes are localized but include a test addition.
Review attention:
- src/snakemake/rules.py — verify correct use of as_posix() and no unintended side effects.
- tests/test_callable/* — ensure test path expectations and file contents match normalization behavior.

Possibly related PRs

fix: use proper callable when accessing rule proxy with associated path modifier #3763 — Related work on callable handling and unwrapping IOFile/AnnotatedString in src/snakemake/rules.py, which overlaps with this PR's path normalization fix.

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	You can run `@coderabbitai generate docstrings` to improve docstring coverage.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately describes the main change: fixing Windows callable path handling by converting Path objects to POSIX-style strings in the expand_input function.
Linked Issues check	✅ Passed	The PR correctly implements the fix specified in issue #3830 by converting Path objects to POSIX-style strings using as_posix() in the concretize_iofile function, and includes a test case demonstrating the fix.
Out of Scope Changes check	✅ Passed	All changes are directly related to fixing Windows callable path handling: the code fix in rules.py, the test Snakefile, and test input file are all necessary and in-scope for addressing issue #3830.
Description check	✅ Passed	The PR description follows the required template with a clear bugfix statement, issue reference, and both QC checklist items marked as complete.

✨ Finishing touches

📝 Generate docstrings

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

Minor changes: - add a new test for callable inputs in Snakemake

coderabbitai

Actionable comments posted: 0

🧹 Nitpick comments (2)

tests/test_callable/Snakefile (2)
7-11: Consider adding a comment explaining the Windows path issue.

While the test is functionally correct, adding a brief comment would help future maintainers understand why this specific test exists.

For example:
 rule rule1:
+	# Test callable returning Path - previously failed on Windows due to
+	# Path being converted to Windows string instead of POSIX (issue #3830)
 	input: lambda w: Path("inputs") / "test.inter"
1-17: Optionally consider testing the unpack case.

Issue #3830 mentions both "lambda/unpack returning pathlib.Path". While the current lambda test is sufficient for the bugfix, you could enhance coverage by adding a test with unpack() that returns Path objects.

Example additional rule:
rule rule3:
	input: unpack(lambda w: {"file": Path("inputs") / "test.in"})
	output: "test_unpack.out"
	run:
		copy2(input.file, output[0])
This is entirely optional and can be deferred if desired.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 8f2c55d and 2232371.

⛔ Files ignored due to path filters (1)

tests/test_callable/expected-results/test.out is excluded by !**/*.out

📒 Files selected for processing (2)

tests/test_callable/Snakefile (1 hunks)
tests/test_callable/inputs/test.in (1 hunks)

✅ Files skipped from review due to trivial changes (1)

tests/test_callable/inputs/test.in

🧰 Additional context used

🧠 Learnings (4)

📚 Learning: 2024-10-06T14:09:54.370Z

Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3117
File: tests/test_wrapper/Snakefile:11-11
Timestamp: 2024-10-06T14:09:54.370Z
Learning: Changes made within test cases, such as in `tests/test_wrapper/Snakefile`, are for testing purposes and do not require updates to the project documentation.

Applied to files:

tests/test_callable/Snakefile

📚 Learning: 2024-12-21T15:10:31.992Z

Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 2925
File: tests/test_nonstr_params/Snakefile:14-15
Timestamp: 2024-12-21T15:10:31.992Z
Learning: The file "test.jsonl" in tests/test_nonstr_params is automatically created by Snakemake, rather than manually generated in the Snakefile.

Applied to files:

tests/test_callable/Snakefile

📚 Learning: 2025-01-17T12:00:09.368Z

Learnt from: leoschwarz
Repo: snakemake/snakemake PR: 3176
File: tests/test_output_index.py:99-157
Timestamp: 2025-01-17T12:00:09.368Z
Learning: New test dependencies for Snakemake should be introduced in separate PRs rather than being added as part of feature or refactoring PRs.

Applied to files:

tests/test_callable/Snakefile

📚 Learning: 2024-11-07T00:32:44.137Z

Learnt from: mbhall88
Repo: snakemake/snakemake PR: 3188
File: tests/test_script/scripts/test.sh:4-4
Timestamp: 2024-11-07T00:32:44.137Z
Learning: In test scripts within the Snakemake project, concise code is preferred over verbose error handling when simplicity suffices.

Applied to files:

tests/test_callable/Snakefile

🔇 Additional comments (1)

tests/test_callable/Snakefile (1)

1-17: LGTM! Test demonstrates the Windows path normalization issue.

This test effectively reproduces issue #3830 by combining:

A callable returning a Path object (line 8, which would previously fail on Windows)

A direct Path output (line 15, which already normalized correctly)

A dependency between them that would fail without proper POSIX normalization

The workflow will fail on Windows without the fix and succeed with it, making this a solid regression test.

🤖 I have created a release *beep* *boop* --- ## [9.14.0](v9.13.7...v9.14.0) (2025-11-27) ### Features * Support Hy in script directive ([#3824](#3824)) ([2329c9e](2329c9e)) ### Bug Fixes * Add support for pathlib in `notebook` field ([#3811](#3811)) ([7b2180a](7b2180a)) * Addressed race condition in workdir_handler.py ([#3844](#3844)) ([8dbfcfb](8dbfcfb)) * cleanup update-marked output files of failed jobs if there was no backup to restore them ([#3843](#3843)) ([41f1ce8](41f1ce8)) * correct Windows callable path handling ([#3832](#3832)) ([5caad70](5caad70)) * expand env vars on resources ([#3823](#3823)) ([fcfa1bc](fcfa1bc)) * fix backup for output marked by `update` ([#3839](#3839)) ([09c64b7](09c64b7)) * Minor fixes/additions to logging module. ([#3802](#3802)) ([3b3986d](3b3986d)) * mount local storage prefix into containers ([#3840](#3840)) ([f1e8b62](f1e8b62)) * properly format input/output files in case of missing rule to produce them ([#3849](#3849)) ([69d5d24](69d5d24)) * Unpack AnnotatedString in _apply_wildcards ([#3798](#3798)) ([7886508](7886508)) ### Performance Improvements * retrieve storage inputs immediately before scheduling jobs instead of before running the entire workflow ([#3850](#3850)) ([4ac6cda](4ac6cda)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

Fixes snakemake#3830 Bugfix: - fix path handling on Windows when using callables for input/output files in rules by converting Path objects to POSIX-style strings ### QC  * [x] The PR contains a test case for the changes or the changes are already covered by an existing test case. * [x] The documentation (`docs/`) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).  ## Summary by CodeRabbit * **Bug Fixes** * Improved cross-platform path handling so input file paths are consistently formatted when workflows use callables. * **Tests** * Added an automated workflow test and accompanying test input to validate the callable-based input handling.

🤖 I have created a release *beep* *boop* --- ## [9.14.0](snakemake/snakemake@v9.13.7...v9.14.0) (2025-11-27) ### Features * Support Hy in script directive ([snakemake#3824](snakemake#3824)) ([2329c9e](snakemake@2329c9e)) ### Bug Fixes * Add support for pathlib in `notebook` field ([snakemake#3811](snakemake#3811)) ([7b2180a](snakemake@7b2180a)) * Addressed race condition in workdir_handler.py ([snakemake#3844](snakemake#3844)) ([8dbfcfb](snakemake@8dbfcfb)) * cleanup update-marked output files of failed jobs if there was no backup to restore them ([snakemake#3843](snakemake#3843)) ([41f1ce8](snakemake@41f1ce8)) * correct Windows callable path handling ([snakemake#3832](snakemake#3832)) ([5caad70](snakemake@5caad70)) * expand env vars on resources ([snakemake#3823](snakemake#3823)) ([fcfa1bc](snakemake@fcfa1bc)) * fix backup for output marked by `update` ([snakemake#3839](snakemake#3839)) ([09c64b7](snakemake@09c64b7)) * Minor fixes/additions to logging module. ([snakemake#3802](snakemake#3802)) ([3b3986d](snakemake@3b3986d)) * mount local storage prefix into containers ([snakemake#3840](snakemake#3840)) ([f1e8b62](snakemake@f1e8b62)) * properly format input/output files in case of missing rule to produce them ([snakemake#3849](snakemake#3849)) ([69d5d24](snakemake@69d5d24)) * Unpack AnnotatedString in _apply_wildcards ([snakemake#3798](snakemake#3798)) ([7886508](snakemake@7886508)) ### Performance Improvements * retrieve storage inputs immediately before scheduling jobs instead of before running the entire workflow ([snakemake#3850](snakemake#3850)) ([4ac6cda](snakemake@4ac6cda)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

Fix Windows callable path handling

8f2c55d

Bugfix: - fix path handling on Windows when using callables for input/output files in rules by converting Path objects to POSIX-style strings

Eric-Nitschke requested a review from johanneskoester as a code owner November 7, 2025 10:59

Eric-Nitschke changed the title ~~Fix Windows callable path handling~~ fix: correct Windows callable path handling Nov 10, 2025

Add a test for callable inputs

2232371

Minor changes: - add a new test for callable inputs in Snakemake

coderabbitai Bot reviewed Nov 17, 2025

View reviewed changes

johanneskoester approved these changes Nov 27, 2025

View reviewed changes

johanneskoester enabled auto-merge (squash) November 27, 2025 13:23

johanneskoester merged commit 5caad70 into snakemake:main Nov 27, 2025
62 checks passed

snakemake-bot mentioned this pull request Nov 27, 2025

chore(main): release 9.14.0 #3835

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: correct Windows callable path handling#3832

fix: correct Windows callable path handling#3832
johanneskoester merged 2 commits intosnakemake:mainfrom
Eric-Nitschke:windows-callable-paths

Eric-Nitschke commented Nov 7, 2025 •

edited

Loading

Uh oh!

coderabbitai Bot commented Nov 7, 2025 •

edited

Loading

Walkthrough

Changes

Sequence Diagram(s)

Estimated code review effort

Possibly related PRs

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Eric-Nitschke commented Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

QC

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

Estimated code review effort

Possibly related PRs

Pre-merge checks and finishing touches

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Eric-Nitschke commented Nov 7, 2025 •

edited

Loading

coderabbitai Bot commented Nov 7, 2025 •

edited

Loading