fix: expand env vars on resources by fgvieira · Pull Request #3823 · snakemake/snakemake

fgvieira · 2025-11-02T11:00:18Z

Envvars on resources (e.g. tmpdir) are not expanded correctly (e.g. runtime). This PR fixes that, as well as expansion of envvars on profiles. For that reason, changes from PR #3597 were reverted.

Tests were also added for both cases.

QC

The PR contains a test case for the changes or the changes are already covered by an existing test case.
The documentation (docs/) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).

Summary by CodeRabbit

New Features
- Configuration values and resource expressions now expand environment variables (e.g., $VAR) and user home (~), allowing dynamic paths and settings.
Tests
- Test workflows updated to validate environment-variable and home-directory expansion in configs and resource handling.

coderabbitai · 2025-11-02T11:00:28Z

📝 Walkthrough

Walkthrough

Profile parsing and resource evaluation now apply both user-home and environment-variable expansion (os.path.expanduser(os.path.expandvars(...))) at parse/eval points. Helper formatters in profiles stopped doing expansion. Tests were updated to use environment variables and to assert expanded values.

Changes

Cohort / File(s)	Summary
Profile parsing helpers & main parsing `src/snakemake/profiles.py`	Removed `os.path.expandvars` from `format_one_level_dict` and `format_two_level_dict`; moved value expansion to the main parsing branch and now apply `os.path.expanduser(os.path.expandvars(str(value)))` when parsing profile values.
Resource expression evaluation `src/snakemake/resources.py`	In `eval_resource_expression` → `generic_callable`, import `os` locally and expand resource expression values via `os.path.expanduser(os.path.expandvars(val))` before `eval`.
Test configuration `tests/test_profile/config.yaml`	Replaced `configfile` static path with `$WORKFLOW_CONFIG`; added `a.tmpdir: "/tmp/$TMPDIR_SUBFOLDER"`; added `b=4` to `set-threads`.
Test workflows `tests/test_profile/Snakefile`, `tests/test_profile/Snakefile.internal`	`Snakefile`: added `import os` and set `WORKFLOW_CONFIG` and `TMPDIR_SUBFOLDER` env vars for the test. `Snakefile.internal`: added `Path` import, derived `cfg_file` from `workflow.configfiles[0].name`, and extended shell output to include `tmpdir` and `configfile`.

Sequence Diagram(s)

mermaid
sequenceDiagram
participant Parser as Profile parser
participant Helpers as format_*_dict
participant Resources as Resource evaluator
participant OS as os.path.expanduser/expandvars
Note over Helpers: Previously applied expandvars
Parser->>Helpers: parse profile structure (no expansion)
Parser->>OS: expanduser + expandvars on parsed values
Parser->>Resources: pass expanded values
Resources->>OS: expanduser + expandvars on resource expressions before eval
Resources->>Resources: eval(resource expression)

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Inspect profiles.py parsing branch to ensure all value types (strings, nested structures) are correctly converted to str before expansion.
Verify local os import and expansion in resources.py does not change evaluation context or introduce security concerns with eval.
Check tests for environment isolation so CI runs are unaffected by host env variables.

Possibly related PRs

fix: expandvars for special profile keys #3597: Related changes around moving/adding environment-variable expansion in profiles.py (similar area of profile parsing and expansion behavior).

Suggested reviewers

johanneskoester

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	You can run `@coderabbitai generate docstrings` to improve docstring coverage.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	The PR description follows the repository template and includes all required sections. The author provided a clear explanation of the issue (environment variables not expanding correctly in resources like tmpdir), described the fix (expanding env vars in both profiles and resources with reference to reverting PR #3597), and noted that tests were added for both cases. Both QC checkboxes are properly marked as completed, indicating that test cases are included and documentation updates were considered. The description is substantially complete and meets the template requirements.
Title Check	✅ Passed	The pull request title "fix: expand env vars on resources" directly relates to the main changes in the changeset. The PR's primary objective is to enable environment variable expansion (along with user home directory expansion via `os.path.expanduser`) in resource expressions within `src/snakemake/resources.py`, as evidenced by the change adding `os.path.expanduser(os.path.expandvars(val))` before evaluation. The title is concise, specific, and clearly indicates what is being fixed without unnecessary noise or vagueness. While the changes also affect profile parsing in `src/snakemake/profiles.py`, the "resources" focus correctly identifies the primary intent of the fix.

✨ Finishing touches

📝 Generate docstrings

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

🤖 I have created a release *beep* *boop* --- ## [9.14.0](v9.13.7...v9.14.0) (2025-11-27) ### Features * Support Hy in script directive ([#3824](#3824)) ([2329c9e](2329c9e)) ### Bug Fixes * Add support for pathlib in `notebook` field ([#3811](#3811)) ([7b2180a](7b2180a)) * Addressed race condition in workdir_handler.py ([#3844](#3844)) ([8dbfcfb](8dbfcfb)) * cleanup update-marked output files of failed jobs if there was no backup to restore them ([#3843](#3843)) ([41f1ce8](41f1ce8)) * correct Windows callable path handling ([#3832](#3832)) ([5caad70](5caad70)) * expand env vars on resources ([#3823](#3823)) ([fcfa1bc](fcfa1bc)) * fix backup for output marked by `update` ([#3839](#3839)) ([09c64b7](09c64b7)) * Minor fixes/additions to logging module. ([#3802](#3802)) ([3b3986d](3b3986d)) * mount local storage prefix into containers ([#3840](#3840)) ([f1e8b62](f1e8b62)) * properly format input/output files in case of missing rule to produce them ([#3849](#3849)) ([69d5d24](69d5d24)) * Unpack AnnotatedString in _apply_wildcards ([#3798](#3798)) ([7886508](7886508)) ### Performance Improvements * retrieve storage inputs immediately before scheduling jobs instead of before running the entire workflow ([#3850](#3850)) ([4ac6cda](4ac6cda)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

Envvars on resources (e.g. `tmpdir`) are not expanded correctly (e.g. runtime). This PR fixes that, as well as expansion of envvars on profiles. For that reason, changes from PR snakemake#3597 were reverted. Tests were also added for both cases. ### QC  * [x] The PR contains a test case for the changes or the changes are already covered by an existing test case. * [x] The documentation (`docs/`) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).  ## Summary by CodeRabbit * **New Features** * Configuration values and resource expressions now expand environment variables (e.g., $VAR) and user home (~), allowing dynamic paths and settings. * **Tests** * Test workflows updated to validate environment-variable and home-directory expansion in configs and resource handling.

🤖 I have created a release *beep* *boop* --- ## [9.14.0](snakemake/snakemake@v9.13.7...v9.14.0) (2025-11-27) ### Features * Support Hy in script directive ([snakemake#3824](snakemake#3824)) ([2329c9e](snakemake@2329c9e)) ### Bug Fixes * Add support for pathlib in `notebook` field ([snakemake#3811](snakemake#3811)) ([7b2180a](snakemake@7b2180a)) * Addressed race condition in workdir_handler.py ([snakemake#3844](snakemake#3844)) ([8dbfcfb](snakemake@8dbfcfb)) * cleanup update-marked output files of failed jobs if there was no backup to restore them ([snakemake#3843](snakemake#3843)) ([41f1ce8](snakemake@41f1ce8)) * correct Windows callable path handling ([snakemake#3832](snakemake#3832)) ([5caad70](snakemake@5caad70)) * expand env vars on resources ([snakemake#3823](snakemake#3823)) ([fcfa1bc](snakemake@fcfa1bc)) * fix backup for output marked by `update` ([snakemake#3839](snakemake#3839)) ([09c64b7](snakemake@09c64b7)) * Minor fixes/additions to logging module. ([snakemake#3802](snakemake#3802)) ([3b3986d](snakemake@3b3986d)) * mount local storage prefix into containers ([snakemake#3840](snakemake#3840)) ([f1e8b62](snakemake@f1e8b62)) * properly format input/output files in case of missing rule to produce them ([snakemake#3849](snakemake#3849)) ([69d5d24](snakemake@69d5d24)) * Unpack AnnotatedString in _apply_wildcards ([snakemake#3798](snakemake#3798)) ([7886508](snakemake@7886508)) ### Performance Improvements * retrieve storage inputs immediately before scheduling jobs instead of before running the entire workflow ([snakemake#3850](snakemake#3850)) ([4ac6cda](snakemake@4ac6cda)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

Expand env vars

42c9e62

fgvieira requested a review from johanneskoester as a code owner November 2, 2025 11:00

Code format

72968ce

fgvieira changed the title ~~fix: expand env vars~~ fix: expand env vars on resources Nov 3, 2025

Merge branch 'main' into env_vars

3d5bc0a

johanneskoester approved these changes Nov 27, 2025

View reviewed changes

johanneskoester merged commit fcfa1bc into snakemake:main Nov 27, 2025
59 checks passed

fgvieira deleted the env_vars branch November 27, 2025 13:28

snakemake-bot mentioned this pull request Nov 27, 2025

chore(main): release 9.14.0 #3835

Merged

fgvieira mentioned this pull request Dec 2, 2025

Use of env variabled in resources not working properly snakemake/snakemake-executor-plugin-slurm#370

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: expand env vars on resources#3823

fix: expand env vars on resources#3823
johanneskoester merged 3 commits intosnakemake:mainfrom
fgvieira:env_vars

fgvieira commented Nov 2, 2025 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Nov 2, 2025 •

edited

Loading

Walkthrough

Changes

Sequence Diagram(s)

Estimated code review effort

Possibly related PRs

Suggested reviewers

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

fgvieira commented Nov 2, 2025 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

QC

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Nov 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

Estimated code review effort

Possibly related PRs

Suggested reviewers

Pre-merge checks and finishing touches

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

fgvieira commented Nov 2, 2025 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Nov 2, 2025 •

edited

Loading