Comparing changes

Add support for [Hy](https://hylang.org/) in the script directive. This will allow the use of a Lisp-like syntax for writing scripts, e.g.: ```hy (require hyrule [-> ->>]) (defn is-odd? [n] (!= (% n 2) 0)) (setv result (->> (get snakemake.input 0) open .readlines (map int) (filter is-odd?) sum)) (print result :file (-> (get snakemake.output 0) (open "w"))) ``` ### QC  * [X] The PR contains a test case for the changes or the changes are already covered by an existing test case. * [X] The documentation (`docs/`) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).  ## Summary by CodeRabbit * **New Features** * Added runtime support for Hy scripts. * **Documentation** * Added Hy and Xonsh language sections with examples and cross-references. * Fixed Bash example formatting and removed a duplicated Xonsh example. * **Tests** * Added end-to-end test exercising Hy script execution with a Snakefile, Hy script, and sample input/output.  --------- Co-authored-by: Johannes Köster <[email protected]> Co-authored-by: Johannes Köster <[email protected]>

- Set the `initialized` attribute in `LoggerManager.setup()`. - Remove incorrect `if` statement in `DefaultFilter.filter()` that never matched. - Remove unused `DefaultFormatter` methods `format_info()` and `format_d3dag()`. - Replace the `get_event_level()` function with `get_event()` as the other return value wasn't being used. - Correct `LoggerManager.settings` annotation from `OutputSettingsLoggerInterface` to `OutputSettings` (uses attributes not defined in the former). - Add parameter annotations to some `DefaultFormatter` methods. - Document attributes for for `OutputSettings`, `LoggerManager`, and `DefaultFilter` classes. ### QC  * [x] The PR contains a test case for the changes or the changes are already covered by an existing test case. * [x] The documentation (`docs/`) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).  ## Summary by CodeRabbit * **Refactor** * Improved internal logging/event extraction and formatting flow, with stricter type annotations for more predictable behavior and maintainability. * **Chores** * Expanded documentation for output settings and simplified a default setting representation for clarity.  --------- Co-authored-by: Cade Mirchandani <[email protected]>

When an output file is marked to be updated snakemake fails if that file does not exist. That might happen when a file needs to be created by the rule before being updated in later workflow executions. This is fixed by validating if the output exists before creating a backup. In addition the behavior of restoring a backup after an error has been thrown needs to be adjusted. Previously an error was raised when no backup file exists for a file marked with `update`. As the newly introduced scenario might not have created a backup a waring is thrown instead of an error. ### QC  * [ ] The PR contains a test case for the changes or the changes are already covered by an existing test case. * [ ] The documentation (`docs/`) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).  ## Summary by CodeRabbit ## Release Notes * **Bug Fixes** * Enhanced backup restoration to issue warnings instead of errors when backups are unavailable * Improved backup efficiency by only backing up files that exist * Refined directory backup handling to better preserve backup data after restoration

Race condition existed in workdir_handler.py when for some reason (such as testing suites that might run in parallel) snakemake is run in parallel, and change_to is called, an exception is called, though control-flow can continue as normal without any issues. Here is a more thorough description of how the race condition can occur For reference, take a look at mkdir from pathlib ```python def mkdir(self, mode=0o777, parents=False, exist_ok=False): """ Create a new directory at this given path. """ try: os.mkdir(self, mode) except FileNotFoundError: if not parents or self.parent == self: raise self.parent.mkdir(parents=True, exist_ok=True) self.mkdir(mode, parents=False, exist_ok=exist_ok) except OSError: # Cannot rely on checking for EEXIST, since the operating system # could give priority to other errors like EACCES or EROFS if not exist_ok or not self.is_dir(): raise ``` Take two threads that are run on snakemake, and both try to cd into "/fake/address" Thread 1 and 2: checks if "/fake/address" (returns False) calls self.workdir.mkdir(parents=True) (with exists_ok defaulting to false) inside the try block: tries to mkdir "/fake/address" which fails with FileNotFoundError because "/fake" does not exist catches exception Thread 1: calls `self.parent.mkdir(parents=True, exist_ok=True)` successfully creates "/fake" Thread 2: calls `self.parent.mkdir(parents=True, exist_ok=True)` "/fake" already exists, but that is not a problem calls `self.mkdir(mode, parents=False, exist_ok=exist_ok)` to create "/fake/address" succeeds Thread 1: calls `self.mkdir(mode, parents=False, exist_ok=exist_ok)` to create "/fake/address" fails because "/fake/address" exists Whereby there is no problem here that "/fake/address" exists. Not sure if I should write a test for this, I personally wouldn't even know how to begin testing this issue, but it seems to have already been solved by pathlib, so just adding this small param should fix this niche issue. This PR does not change intended behavior. ### QC  * [x] The PR contains a test case for the changes or the changes are already covered by an existing test case. * [x] The documentation (`docs/`) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).  ## Summary by CodeRabbit # Release Notes * **Bug Fixes** * Enhanced working directory handling to gracefully manage scenarios where directories already exist, improving reliability during concurrent operations and preventing unnecessary errors. <sub>✏️ Tip: You can customize this high-level summary in your review settings.</sub>

…oduce them (#3849)  ### QC  * [x] The PR contains a test case for the changes or the changes are already covered by an existing test case. * [x] The documentation (`docs/`) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).  ## Summary by CodeRabbit * **Bug Fixes** * Improved formatting of file paths in missing-rule error messages so file locations are displayed more clearly and consistently. * This only affects the error message presentation; no functional behavior changes to rule handling. <sub>✏️ Tip: You can customize this high-level summary in your review settings.</sub>

…o backup to restore them (#3843)  ### QC  * [x] The PR contains a test case for the changes or the changes are already covered by an existing test case. * [x] The documentation (`docs/`) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).  ## Summary by CodeRabbit * **Bug Fixes** * Cleanup now preserves outputs that were restored or explicitly marked to be kept after errors. * Restore-after-update now reports success/failure so restored outputs are correctly retained during cleanup. * Group-level cleanup simplified to rely on per-job cleanup to avoid accidental removal of recovered files. * **Tests** * Added a test validating cleanup behavior after a failed update to ensure updated outputs are removed or preserved as intended. <sub>✏️ Tip: You can customize this high-level summary in your review settings.</sub>  --------- Co-authored-by: Felix Mölder <[email protected]>

Envvars on resources (e.g. `tmpdir`) are not expanded correctly (e.g. runtime). This PR fixes that, as well as expansion of envvars on profiles. For that reason, changes from PR #3597 were reverted. Tests were also added for both cases. ### QC  * [x] The PR contains a test case for the changes or the changes are already covered by an existing test case. * [x] The documentation (`docs/`) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).  ## Summary by CodeRabbit * **New Features** * Configuration values and resource expressions now expand environment variables (e.g., $VAR) and user home (~), allowing dynamic paths and settings. * **Tests** * Test workflows updated to validate environment-variable and home-directory expansion in configs and resource handling.

Fixes #3830 Bugfix: - fix path handling on Windows when using callables for input/output files in rules by converting Path objects to POSIX-style strings ### QC  * [x] The PR contains a test case for the changes or the changes are already covered by an existing test case. * [x] The documentation (`docs/`) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).  ## Summary by CodeRabbit * **Bug Fixes** * Improved cross-platform path handling so input file paths are consistently formatted when workflows use callables. * **Tests** * Added an automated workflow test and accompanying test input to validate the callable-based input handling.

### Description  ### QC  * [x] The PR contains a test case for the changes or the changes are already covered by an existing test case. * [x] The documentation (`docs/`) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).  ## Summary by CodeRabbit * **New Features** * Workflows expose runtime paths that are propagated to scripts, notebooks, wrappers, CWL jobs, and executors. * A configurable local storage prefix is accepted and forwarded across relevant components. * Container executions now accept and apply runtime/local storage binds so runtime paths are available inside containers. <sub>✏️ Tip: You can customize this high-level summary in your review settings.</sub>

@staadecker

PR #3636 added support for Pathlike `notebook` specifications but @staadecker (the original author of that PR) did not provide a matching test. This PR superseeds @staadecker's original PR #3636 by merging in current `main` and adding a commit that adds such a test based on @staadecker's example. I have verified that this test * fails (with the expected `TypeError`: ``` TypeError in file "</path/to/testdir>/Snakefile", line 7: expected str, got PosixPath ``` ) at commit 8bd1df9 (v9.6.0; parent of @staadecker's original PR #3636), * passes after applying the changes of commit 4445bb8 (@staadecker's original PR #3636), * still fails (with the same expected `TypeError`, see above) at commit 395a5e6 (current `main`), and * still passes at commit 756656d merging the current `main` (see above) on top of @staadecker's original PR #3636). --- closes #3636  ## Summary by CodeRabbit * **Bug Fixes** * Ensure pathlike notebook resources are converted to strings before processing to avoid inconsistent handling. * **Tests** * Added test coverage verifying Jupyter notebook pathlike object support with Conda deployment.  --------- Co-authored-by: Martin <[email protected]>

Unpacked functions (AnnotatedStrings) can be passed to `_is_deriving_function`. Ensure these unpacked functions are converted to callable functions first. Fixes #3797 ### QC * [X] The PR contains a test case for the changes or the changes are already covered by an existing test case. * [X] The documentation (`docs/`) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).  ## Summary by CodeRabbit * **Bug Fixes** * Fixed wildcard handling so unpacked annotated callables are evaluated correctly when determining derived vs non‑derived inputs, preventing incorrect derivation detection. * **Tests** * Added a test covering wildcard‑annotated parameters that verifies error behavior when unpacked annotated params are used.

…ead of before running the entire workflow (#3850) This saves local disk space. After each job is finished, Snakemake already ensures that local storage copies are deleted once not needed anymore. Hence, in combination with this PR, the lifetime of local copies on disk will be minimized. ### QC  * [x] The PR contains a test case for the changes or the changes are already covered by an existing test case. * [x] The documentation (`docs/`) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).  ## Summary by CodeRabbit * **Refactor** * Simplified and unified storage-input retrieval flow, changing when preflight retrieval runs and clarifying external-input determination. * **New Features** * Asynchronous prefetching of storage inputs for remote jobs when local copies are configured. * **Style** * Consistent file I/O formatting and logging output across job reporting. <sub>✏️ Tip: You can customize this high-level summary in your review settings.</sub>

🤖 I have created a release *beep* *boop* --- ## [9.14.0](v9.13.7...v9.14.0) (2025-11-27) ### Features * Support Hy in script directive ([#3824](#3824)) ([2329c9e](2329c9e)) ### Bug Fixes * Add support for pathlib in `notebook` field ([#3811](#3811)) ([7b2180a](7b2180a)) * Addressed race condition in workdir_handler.py ([#3844](#3844)) ([8dbfcfb](8dbfcfb)) * cleanup update-marked output files of failed jobs if there was no backup to restore them ([#3843](#3843)) ([41f1ce8](41f1ce8)) * correct Windows callable path handling ([#3832](#3832)) ([5caad70](5caad70)) * expand env vars on resources ([#3823](#3823)) ([fcfa1bc](fcfa1bc)) * fix backup for output marked by `update` ([#3839](#3839)) ([09c64b7](09c64b7)) * Minor fixes/additions to logging module. ([#3802](#3802)) ([3b3986d](3b3986d)) * mount local storage prefix into containers ([#3840](#3840)) ([f1e8b62](f1e8b62)) * properly format input/output files in case of missing rule to produce them ([#3849](#3849)) ([69d5d24](69d5d24)) * Unpack AnnotatedString in _apply_wildcards ([#3798](#3798)) ([7886508](7886508)) ### Performance Improvements * retrieve storage inputs immediately before scheduling jobs instead of before running the entire workflow ([#3850](#3850)) ([4ac6cda](4ac6cda)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comparing changes

Open a pull request

Commits on Nov 8, 2025

Commits on Nov 18, 2025

Commits on Nov 19, 2025

Commits on Nov 27, 2025

This comparison is taking too long to generate.

Uh oh!