fix: Unpack AnnotatedString in _apply_wildcards#3798
fix: Unpack AnnotatedString in _apply_wildcards#3798johanneskoester merged 1 commit intosnakemake:mainfrom
Conversation
📝 WalkthroughWalkthroughUse the underlying callable for unpacked AnnotatedString items when checking derivation in Changes
Sequence Diagram(s)sequenceDiagram
participant Caller as _apply_wildcards
participant Item as item (maybe AnnotatedString, is_unpack)
participant Probe as _is_deriving_function
participant Apply as apply_input_function
Caller->>Item: receive item
alt item.is_unpack and item is AnnotatedString
Caller->>Item: use callable_item = item.callable
else
Caller->>Item: use callable_item = item
end
Caller->>Probe: _is_deriving_function(callable_item)
Probe-->>Caller: is_derived
Caller->>Apply: apply_input_function(item, is_derived)
Apply-->>Caller: result
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes
Pre-merge checks and finishing touches✅ Passed checks (4 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
📜 Recent review detailsConfiguration used: CodeRabbit UI Review profile: CHILL Plan: Pro ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (3)
🚧 Files skipped from review as they are similar to previous changes (1)
🧰 Additional context used📓 Path-based instructions (1)**/*.py⚙️ CodeRabbit configuration file
Files:
🧠 Learnings (8)📚 Learning: 2025-08-29T11:51:44.625ZApplied to files:
📚 Learning: 2024-10-06T14:09:54.370ZApplied to files:
📚 Learning: 2024-12-21T15:10:31.992ZApplied to files:
📚 Learning: 2024-10-08T17:41:54.542ZApplied to files:
📚 Learning: 2025-01-17T12:00:09.368ZApplied to files:
📚 Learning: 2024-08-13T16:22:09.641ZApplied to files:
📚 Learning: 2024-10-08T17:41:54.542ZApplied to files:
📚 Learning: 2024-12-14T13:10:48.450ZApplied to files:
🧬 Code graph analysis (1)src/snakemake/rules.py (1)
🔇 Additional comments (2)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
src/snakemake/common/__init__.py(1 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
**/*.py
⚙️ CodeRabbit configuration file
**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of theselfargument of methods.
Do not suggest type annotation of theclsargument of classmethods.
Do not suggest return type annotation if a function or method does not contain areturnstatement.
Files:
src/snakemake/common/__init__.py
🧬 Code graph analysis (1)
src/snakemake/common/__init__.py (1)
src/snakemake/io/__init__.py (1)
is_flagged(974-977)
🪛 Ruff (0.14.0)
src/snakemake/common/__init__.py
296-296: Undefined name is_flagged
(F821)
🔇 Additional comments (1)
src/snakemake/common/__init__.py (1)
296-297:unpackapplies exclusively toAnnotatedString, which always initializes a.callableattribute, so the attribute access is safe.
dee9e88 to
e011235
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
src/snakemake/common/__init__.py(2 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
**/*.py
⚙️ CodeRabbit configuration file
**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of theselfargument of methods.
Do not suggest type annotation of theclsargument of classmethods.
Do not suggest return type annotation if a function or method does not contain areturnstatement.
Files:
src/snakemake/common/__init__.py
🧬 Code graph analysis (1)
src/snakemake/common/__init__.py (1)
src/snakemake/io/__init__.py (1)
is_flagged(974-977)
🔇 Additional comments (1)
src/snakemake/common/__init__.py (1)
26-26: LGTM! Critical import issue resolved.The import of
is_flaggedfromsnakemake.iois correctly placed and addresses the criticalNameErrorthat was flagged in the previous review.
e011235 to
d56f4b1
Compare
d56f4b1 to
54a4001
Compare
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
src/snakemake/rules.py (1)
786-787: LGTM! The fix correctly handles unpacked callables.The change properly extracts the underlying callable from an AnnotatedString before passing it to
_is_deriving_function, which resolves the TypeError described in issue #3797. The logic is sound: whenis_unpackis True, the item is an AnnotatedString wrapping a callable, and_is_deriving_functionneeds the raw callable to inspect its parameters.For consistency with the established pattern in this codebase (lines 686-688, 1091-1092) and defensive programming, consider adding an explicit
isinstancecheck:if non_derived_items is not None: - callable_item = item.callable if is_unpack else item + if is_unpack and isinstance(item, AnnotatedString): + callable_item = item.callable + else: + callable_item = item is_derived = self._is_deriving_function(callable_item)This makes the code more robust and aligns with the pattern used elsewhere in the file. However, the current implementation is likely safe since
is_unpackshould only be True for AnnotatedString objects.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
src/snakemake/rules.py(1 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
**/*.py
⚙️ CodeRabbit configuration file
**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of theselfargument of methods.
Do not suggest type annotation of theclsargument of classmethods.
Do not suggest return type annotation if a function or method does not contain areturnstatement.
Files:
src/snakemake/rules.py
54a4001 to
9016c69
Compare
johanneskoester
left a comment
There was a problem hiding this comment.
To avoid a future regression, would you mind adding a small test case?
9016c69 to
932e57f
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
tests/tests.py (1)
2647-2652: Consider asserting the specific error message.The test correctly expects a
WorkflowError, but it catches anyWorkflowErrorwithout verifying it's the intended "unpack() is not allowed with params." message. This could mask other unexpected errors.Consider asserting the error message:
def test_wildcard_annotatedstrings(): - try: - run(dpath("test_wildcard_annotatedstrings"), targets=["test.out"]) - except WorkflowError: - # WorkflowError is expected - return + with pytest.raises(WorkflowError, match="unpack.*is not allowed with params"): + run(dpath("test_wildcard_annotatedstrings"), targets=["test.out"])
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
tests/test_wildcard_annotatedstrings/expected-results/test.outis excluded by!**/*.out
📒 Files selected for processing (3)
src/snakemake/rules.py(1 hunks)tests/test_wildcard_annotatedstrings/Snakefile(1 hunks)tests/tests.py(2 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
**/*.py
⚙️ CodeRabbit configuration file
**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of theselfargument of methods.
Do not suggest type annotation of theclsargument of classmethods.
Do not suggest return type annotation if a function or method does not contain areturnstatement.
Files:
src/snakemake/rules.pytests/tests.py
🧬 Code graph analysis (2)
src/snakemake/rules.py (1)
src/snakemake/io/__init__.py (1)
AnnotatedString(948-968)
tests/tests.py (1)
tests/common.py (2)
run(152-503)dpath(33-36)
🔇 Additional comments (2)
tests/tests.py (1)
22-22: LGTM!The import of
WorkflowErroris correctly added and follows the existing pattern.src/snakemake/rules.py (1)
786-790: Perfect! I found the key information in the search results. Let me verify one more aspect directly in the codebase:#!/bin/bash # Find the exact code context around lines 786-790 sed -n '780,800p' src/snakemake/rules.py # Also check is_callable definition to confirm it's from snakemake.io rg -nA8 'def is_callable' src/snakemake/io.pyBased on my analysis from the web search results, I can now verify the edge cases:
From the Snakemake documentation (index 22-6), I found the
AnnotatedStringclass definition:
AnnotatedStringhas acallableproperty that is set tovalue if is_callable(value) else NoneAnd the
is_callablefunction checks if a value is callable by usingcallable(value)or checking if it's an_IOFileorAnnotatedStringwith a non-Nonecallableproperty.From the earlier script output,
_is_deriving_functionat line 943 checkscallable()returns True if the object is callable, otherwise False, and sincecallable(None)returns False, passingNoneto_is_deriving_functionis handled gracefully at line 951 which returnsFalse.The fix is correct and handles edge cases appropriately:
Null callable edge case: If
item.callableisNone,_is_deriving_function(None)will returnFalsebecause thecallable()function returns False for non-callable objects like None.
_IOFilehandling: No evidence found in the codebase that_IOFileis used with theis_unpackflag in a way requiring unwrapping before the derivation check. The fix correctly handlesAnnotatedStringunpacking, and_IOFilehandling already occurs inapply_input_functionat the beginning of that method.
Of course, just added one. Since this is fixing an incorrect If you'd prefer a different approach, I'm happy to modify the test. |
Head branch was pushed to by a user without write access
822b5a0 to
9094907
Compare
Unpacked functions (AnnotatedStrings) can be passed to _is_deriving_function. Ensure these unpacked functions are converted to callable functions first.
|
Thanks a lot! |
🤖 I have created a release *beep* *boop* --- ## [9.14.0](v9.13.7...v9.14.0) (2025-11-27) ### Features * Support Hy in script directive ([#3824](#3824)) ([2329c9e](2329c9e)) ### Bug Fixes * Add support for pathlib in `notebook` field ([#3811](#3811)) ([7b2180a](7b2180a)) * Addressed race condition in workdir_handler.py ([#3844](#3844)) ([8dbfcfb](8dbfcfb)) * cleanup update-marked output files of failed jobs if there was no backup to restore them ([#3843](#3843)) ([41f1ce8](41f1ce8)) * correct Windows callable path handling ([#3832](#3832)) ([5caad70](5caad70)) * expand env vars on resources ([#3823](#3823)) ([fcfa1bc](fcfa1bc)) * fix backup for output marked by `update` ([#3839](#3839)) ([09c64b7](09c64b7)) * Minor fixes/additions to logging module. ([#3802](#3802)) ([3b3986d](3b3986d)) * mount local storage prefix into containers ([#3840](#3840)) ([f1e8b62](f1e8b62)) * properly format input/output files in case of missing rule to produce them ([#3849](#3849)) ([69d5d24](69d5d24)) * Unpack AnnotatedString in _apply_wildcards ([#3798](#3798)) ([7886508](7886508)) ### Performance Improvements * retrieve storage inputs immediately before scheduling jobs instead of before running the entire workflow ([#3850](#3850)) ([4ac6cda](4ac6cda)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
Unpacked functions (AnnotatedStrings) can be passed to `_is_deriving_function`. Ensure these unpacked functions are converted to callable functions first. Fixes snakemake#3797 ### QC * [X] The PR contains a test case for the changes or the changes are already covered by an existing test case. * [X] The documentation (`docs/`) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake). <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Fixed wildcard handling so unpacked annotated callables are evaluated correctly when determining derived vs non‑derived inputs, preventing incorrect derivation detection. * **Tests** * Added a test covering wildcard‑annotated parameters that verifies error behavior when unpacked annotated params are used. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
🤖 I have created a release *beep* *boop* --- ## [9.14.0](snakemake/snakemake@v9.13.7...v9.14.0) (2025-11-27) ### Features * Support Hy in script directive ([snakemake#3824](snakemake#3824)) ([2329c9e](snakemake@2329c9e)) ### Bug Fixes * Add support for pathlib in `notebook` field ([snakemake#3811](snakemake#3811)) ([7b2180a](snakemake@7b2180a)) * Addressed race condition in workdir_handler.py ([snakemake#3844](snakemake#3844)) ([8dbfcfb](snakemake@8dbfcfb)) * cleanup update-marked output files of failed jobs if there was no backup to restore them ([snakemake#3843](snakemake#3843)) ([41f1ce8](snakemake@41f1ce8)) * correct Windows callable path handling ([snakemake#3832](snakemake#3832)) ([5caad70](snakemake@5caad70)) * expand env vars on resources ([snakemake#3823](snakemake#3823)) ([fcfa1bc](snakemake@fcfa1bc)) * fix backup for output marked by `update` ([snakemake#3839](snakemake#3839)) ([09c64b7](snakemake@09c64b7)) * Minor fixes/additions to logging module. ([snakemake#3802](snakemake#3802)) ([3b3986d](snakemake@3b3986d)) * mount local storage prefix into containers ([snakemake#3840](snakemake#3840)) ([f1e8b62](snakemake@f1e8b62)) * properly format input/output files in case of missing rule to produce them ([snakemake#3849](snakemake#3849)) ([69d5d24](snakemake@69d5d24)) * Unpack AnnotatedString in _apply_wildcards ([snakemake#3798](snakemake#3798)) ([7886508](snakemake@7886508)) ### Performance Improvements * retrieve storage inputs immediately before scheduling jobs instead of before running the entire workflow ([snakemake#3850](snakemake#3850)) ([4ac6cda](snakemake@4ac6cda)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
Unpacked functions (AnnotatedStrings) can be passed to
_is_deriving_function. Ensure these unpacked functions are converted to callable functions first.Fixes #3797
QC
docs/) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).Summary by CodeRabbit