Skip to content

Not all values in profile config support environment variables #3592

@bentyeh

Description

@bentyeh

Snakemake's documentation is explicit about supporting the environment variables like $USER in profiles:

Values in profiles can make use of globally available environment variables, e.g. the $USER variable. For example, the following would set the default prefix for storing local copies of remote storage files to a user specific directory

local-storage-prefix: /local/work/$USER/snakemake-scratch

Any such environment variables are automatically expanded when evaluating the profile.

Unfortunately, it does not appear that environment variables are expanded for all profile values. Specifically, $USER is not expanded when part of the tmpdir resource value.

Not sure whether this should be considered a bug or a documentation issue.


Example

Tested with Snakemake version 9.3.3.

workflow/Snakefile

rule all:
    input:
        "out.txt",

rule rule1:
    output:
        "out.txt",
    shell:
        """
        echo "USER: $USER" >> "{output}"
        echo "TMPDIR: $TMPDIR" >> "{output}"
        """

workflow/profiles/default/config.yaml

executor: local # or slurm
jobs: 2
latency-wait: 10
printshellcmds: True
verbose: True
default-resources:
    runtime: 5 # minutes
    mem_mb: 1000
    tmpdir: /central/scratch/$USER

Command: snakemake --snakefile workflow/Snakefile --workflow-profile workflow/profiles/default

Output (out.txt)

USER: bentyeh
TMPDIR: /central/scratch/$USER

The failure for $USER to be expanded results in the literal string '/central/scratch/$USER' to be used as the value of the $TMPDIR environment variable within a rule. This is the result for both local and slurm executors.


Is there a recommended way for users to configure the temporary directory used by a workflow? I thought that the profile would be the most appropriate place, but since that does not appear to work (for the moment), I could expose a variable in config/config.yaml for the user to set. But then each rule in the Snakefile would need to set the tmpdir resource, since it does not appear that the workflow profile value for default-resources: {tmpdir: <value>} has access to variable values in the Snakefile.

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions