Snakemake's documentation is explicit about supporting the environment variables like $USER in profiles:
Values in profiles can make use of globally available environment variables, e.g. the $USER variable. For example, the following would set the default prefix for storing local copies of remote storage files to a user specific directory
local-storage-prefix: /local/work/$USER/snakemake-scratch
Any such environment variables are automatically expanded when evaluating the profile.
Unfortunately, it does not appear that environment variables are expanded for all profile values. Specifically, $USER is not expanded when part of the tmpdir resource value.
Not sure whether this should be considered a bug or a documentation issue.
Example
Tested with Snakemake version 9.3.3.
workflow/Snakefile
rule all:
input:
"out.txt",
rule rule1:
output:
"out.txt",
shell:
"""
echo "USER: $USER" >> "{output}"
echo "TMPDIR: $TMPDIR" >> "{output}"
"""
workflow/profiles/default/config.yaml
executor: local # or slurm
jobs: 2
latency-wait: 10
printshellcmds: True
verbose: True
default-resources:
runtime: 5 # minutes
mem_mb: 1000
tmpdir: /central/scratch/$USER
Command: snakemake --snakefile workflow/Snakefile --workflow-profile workflow/profiles/default
Output (out.txt)
USER: bentyeh
TMPDIR: /central/scratch/$USER
The failure for $USER to be expanded results in the literal string '/central/scratch/$USER' to be used as the value of the $TMPDIR environment variable within a rule. This is the result for both local and slurm executors.
Is there a recommended way for users to configure the temporary directory used by a workflow? I thought that the profile would be the most appropriate place, but since that does not appear to work (for the moment), I could expose a variable in config/config.yaml for the user to set. But then each rule in the Snakefile would need to set the tmpdir resource, since it does not appear that the workflow profile value for default-resources: {tmpdir: <value>} has access to variable values in the Snakefile.
Snakemake's documentation is explicit about supporting the environment variables like
$USERin profiles:Unfortunately, it does not appear that environment variables are expanded for all profile values. Specifically,
$USERis not expanded when part of thetmpdirresource value.Not sure whether this should be considered a bug or a documentation issue.
Example
Tested with Snakemake version 9.3.3.
workflow/Snakefile
workflow/profiles/default/config.yaml
Command:
snakemake --snakefile workflow/Snakefile --workflow-profile workflow/profiles/defaultOutput (out.txt)
The failure for $USER to be expanded results in the literal string
'/central/scratch/$USER'to be used as the value of the$TMPDIRenvironment variable within a rule. This is the result for bothlocalandslurmexecutors.Is there a recommended way for users to configure the temporary directory used by a workflow? I thought that the profile would be the most appropriate place, but since that does not appear to work (for the moment), I could expose a variable in config/config.yaml for the user to set. But then each rule in the Snakefile would need to set the
tmpdirresource, since it does not appear that the workflow profilevaluefordefault-resources: {tmpdir: <value>}has access to variable values in the Snakefile.