Skip to content

Commit e406351

Browse files
fgvieirafgvieira
authored
Merge branch 'main' into input_helper_func
2 parents 8fd3e33 + 477e5f8 commit e406351

File tree

19 files changed

+195
-71
lines changed

19 files changed

+195
-71
lines changed

‎docs/snakefiles/deployment.rst‎

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -285,8 +285,12 @@ At the moment, only a single, random conda environment is shown.
285285

286286
.. note::
287287

288-
Note that conda environments are only used with ``shell``, ``script``, ``notebook`` and the ``wrapper`` directive, not the ``run`` directive.
289-
The reason is that the ``run`` directive has access to the rest of the Snakefile (e.g. globally defined variables) and therefore must be executed in the same process as Snakemake itself. If used with ``notebook`` directive, the associated conda environment should have package ``jupyter`` installed (this package contains dependencies required to execute the notebook).
288+
Note that conda environments can be used with the ``shell``, ``script``, ``notebook``, ``wrapper`` and ``run`` directives.
289+
290+
However, the ``run`` directive is a special case, as it has access to the rest of the Snakefile (e.g. globally defined variables) and therefore must be executed in the same process as Snakemake itself.
291+
The ``conda`` directive for rules with a ``run`` directive therefore only affects ``shell`` function calls that are executed from the within ``run`` script.
292+
293+
If used with ``notebook`` directive, the associated conda environment should have package ``jupyter`` installed (this package contains dependencies required to execute the notebook).
290294

291295
Further, note that search path modifying environment variables like ``R_LIBS`` and ``PYTHONPATH`` can interfere with your conda environments.
292296
Therefore, Snakemake automatically deactivates them for a job when a conda environment definition is used.
@@ -457,8 +461,10 @@ Defining global container images
457461

458462
.. note::
459463

460-
Note that apptainer integration is only used with ``shell``, ``script`` and the ``wrapper`` directive, not the ``run`` directive.
461-
The reason is that the ``run`` directive has access to the rest of the Snakefile (e.g. globally defined variables) and therefore must be executed in the same process as Snakemake itself.
464+
Note that apptainer integration can be used with the ``shell``, ``script``, ``wrapper`` and ``run`` directives.
465+
466+
However, the ``run`` directive is a special case, as it has access to the rest of the Snakefile (e.g. globally defined variables) and therefore must be executed in the same process as Snakemake itself.
467+
The ``container`` directive for rules with a ``run`` directive therefore only affects ``shell`` function calls that are executed from within the ``run`` script.
462468

463469
A global definition of a container image can be given:
464470

‎docs/snakefiles/reporting.rst‎

Lines changed: 68 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -4,9 +4,17 @@
44
Reports
55
=======
66

7-
From Snakemake 5.1 on, it is possible to automatically generate detailed self-contained HTML reports that encompass runtime statistics, provenance information, workflow topology and results.
7+
From Snakemake 5.1 on, it is possible to automatically generate detailed self-contained HTML reports that you can easily move and share.
8+
They encompass runtime statistics, provenance information and workflow topology by default, and workflow developers can also specify and annotate results for inclusion.
9+
For smaller default reports, these can be a single :ref:`snakefiles-self_contained_html_file`.
10+
For more complex reports, one can generate a :ref:`snakefiles-self_contained_zip_archive`, with a contained ``report.html`` as the main entry point.
811
**As an example, the report of the Snakemake rolling paper can be found** `here <https://snakemake.github.io/resources/report.html>`__.
912

13+
.. _snakefiles-including_results_in_a_report:
14+
15+
Including results in a report
16+
-----------------------------
17+
1018
For including results into the report, the Snakefile has to be annotated with additional information.
1119
Each output file that shall be part of the report has to be marked with the ``report`` flag, which optionally points to a caption in `restructured text format <https://docutils.sourceforge.io/docs/user/rst/quickstart.html>`_ and allows to define a ``category`` for grouping purposes.
1220
Moreover, a global workflow description can be defined via the ``report`` directive.
@@ -94,7 +102,7 @@ This works as follows:
94102
"""
95103
96104
Defining file labels
97-
--------------------
105+
^^^^^^^^^^^^^^^^^^^^
98106

99107
In addition to category, and subcategory, it is possible (and highly recommended!) to define a dictionary of labels for each report item.
100108
By that, the actual filename will be hidden in the report and instead a table with the label keys as columns and the values in the respective row for the file will be displayed.
@@ -131,17 +139,17 @@ This behavior can be used to, for example, switch between different versions of
131139

132140

133141
Determining category, subcategory, and labels dynamically via functions
134-
-----------------------------------------------------------------------
142+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
135143

136144
Similar to e.g. with input file and parameter definition (see :ref:`snakefiles-input_functions`), ``category`` and a ``subcategory`` and ``labels`` can be specified by pointing to a function that takes ``wildcards`` as the first argument (and optionally in addition ``input``, ``output``, ``params`` in any order).
137145
The function is expected to return a string or number (int, float, numpy types), or, in case of labels, a dict with strings as keys and strings or numbers as values.
138146

139147

140148
Linking between items
141-
---------------------
149+
^^^^^^^^^^^^^^^^^^^^^
142150

143151
From captions
144-
^^^^^^^^^^^^^
152+
"""""""""""""
145153

146154
In every ``.rst`` document (i.e. in the captions), you can link to
147155

@@ -153,7 +161,7 @@ In every ``.rst`` document (i.e. in the captions), you can link to
153161
For details about the hyperlink mechanism of restructured text see `here <https://docutils.sourceforge.io/docs/user/rst/quickref.html#hyperlink-targets>`__.
154162

155163
From results
156-
^^^^^^^^^^^^
164+
""""""""""""
157165

158166
From within results that are included into the report, you can link to other report items.
159167
This works by using the ``snakemake.report_href()`` method that is available from within :ref:`python scripts <snakefiles-external_scripts>`.
@@ -246,36 +254,78 @@ Further, using ``url_args()`` you can add URL arguments and using ``anchor()`` y
246254
file=f,
247255
)
248256
257+
.. _snakefiles-rendering_reports:
258+
249259
Rendering reports
250260
-----------------
251261

252-
To create the report simply run
262+
All the metadata contained in the report (e.g. runtime statistics) are automatically collected during the rendering of the report.
263+
These statistics are obtained from the metadata that is stored in the ``.snakemake`` directory inside your working directory.
264+
265+
.. _snakefiles-self_contained_html_file:
266+
267+
Self-contained HTML file
268+
^^^^^^^^^^^^^^^^^^^^^^^^
269+
270+
To create a default report simply run
253271

254272
.. code-block:: bash
255273
256-
snakemake --report report.html
274+
snakemake --report
257275
258-
after your workflow has finished.
259-
All other information contained in the report (e.g. runtime statistics) is automatically collected during creation.
260-
These statistics are obtained from the metadata that is stored in the ``.snakemake`` directory inside your working directory.
276+
after your workflow has finished successfully.
277+
This will create a self-contained HTML report with the default filename ``report.html``.
278+
If you want to give your report a custom name, you can do so by specifying this on the command-line:
279+
280+
.. code-block:: bash
281+
282+
snakemake --report my_report.html
283+
284+
As specifying an HTML file for the report will always embed any user-defined report outputs into the same HTML file, this report output type is only suitable for smaller reports.
261285

286+
.. _snakefiles-self_contained_zip_archive:
262287

263-
You can define an institute specific stylesheet with:
288+
Self-contained ZIP archive
289+
^^^^^^^^^^^^^^^^^^^^^^^^^^
290+
291+
For anything more complex, it is recommended to generate a ``ZIP`` archive report.
292+
For example, if you have lots of samples you report on, or if you specified lots of different workflow outputs for report inclusion.
293+
To get such a self-contained ``ZIP`` archive, simply specify a ``.zip`` file name instead:
264294

265295
.. code-block:: bash
266296
267-
snakemake --report report.html --report-stylesheet custom-stylesheet.css
297+
snakemake --report some_report.zip
268298
269-
In particular, this allows you to e.g. set a logo at the top (by using CSS to inject a background for the placeholder ``<div id="brand">``, or overwrite colors.
270-
For an example custom stylesheet defining the logo, see :download:`here <../../tests/test_report/custom-stylesheet.css>`.
271-
The report for above example can be found :download:`here <../../tests/test_report/expected-results/report.html>` (with a custom branding for the University of Duisburg-Essen).
272-
The full example source code can be found `here <https://github.com/snakemake/snakemake/tree/main/tests/test_report/>`__.
299+
You can move this file wherever you want to view it and then unpack it right there.
300+
This includes sending this file to collaboration partners or customers for whom you might have done an analysis.
301+
The main entry point is always the ``report.html`` file in the main folder that this creates (for the above example, this folder will be named ``some_report``).
302+
Just open that file in a web browser and explore your results.
273303

274-
Note that the report can be restricted to particular jobs and results by specifying targets at the command line, analog to normal Snakemake execution.
304+
Partial reports
305+
^^^^^^^^^^^^^^^
306+
307+
The report can be restricted to particular jobs and results by specifying targets at the command line, analog to normal Snakemake execution.
275308
For example, with
276309

277310
.. code-block:: bash
278311
279312
snakemake fig1.svg --report report-short.html
280313
281314
the report contains only ``fig1.svg``.
315+
This can be useful when a larger workflow has not yet run to completion, but you already want to explore some intermediate outputs in the report.
316+
Or when you have multiple alternative target rules within the same workflow.
317+
318+
Custom layout
319+
^^^^^^^^^^^^^
320+
321+
You can define an institute specific layout by providing a custom stylesheet:
322+
323+
.. code-block:: bash
324+
325+
snakemake --report report.html --report-stylesheet custom-stylesheet.css
326+
327+
For example, this allows you to set a logo at the top (by using CSS to inject a background for the placeholder ``<div id="brand">``), or overwrite colors.
328+
329+
For an example with a custom stylesheet defining a logo, see :download:`the report here <../../tests/test_report/expected-results/report.html>` (with a custom branding for the University of Duisburg-Essen).
330+
For the complete mechanics, you can also have a look at the `full example source code <https://github.com/snakemake/snakemake/tree/main/tests/test_report/>`__ and :download:`the custom stylesheet with the logo definition <../../tests/test_report/custom-stylesheet.css>`.
331+

‎docs/snakefiles/rules.rst‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1631,6 +1631,27 @@ as ``"${snakemake_input[0]}"`` and ``"${snakemake_input[1]}"``.
16311631

16321632
For technical reasons, scripts are executed in ``.snakemake/scripts``. The original script directory is available as ``scriptdir`` in the ``snakemake`` object.
16331633

1634+
1635+
Xonsh_
1636+
~~~~~~
1637+
1638+
.. _Xonsh: https://xon.sh
1639+
1640+
.. code-block:: python
1641+
1642+
rule NAME:
1643+
input:
1644+
"path/to/inputfile",
1645+
"path/to/other/inputfile"
1646+
output:
1647+
"path/to/outputfile",
1648+
"path/to/another/outputfile"
1649+
script:
1650+
"path/to/script.xsh"
1651+
1652+
Because Xonsh is a superset of Python, you can use a Xonsh script as you would a Python script (see above), but with all the additional shell primitives that Xonsh provides.
1653+
1654+
16341655
.. _snakefiles_notebook-integration:
16351656

16361657
Jupyter notebook integration

‎pyproject.toml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,8 @@
11
[build-system]
22
build-backend = "setuptools.build_meta"
33
requires = [
4-
"setuptools>=42",
4+
# TODO relax upper bound, as of March 9 2025 there is a hash mismatch in setuptools on pypi
5+
"setuptools>=42,<76",
56
'tomli; python_version < "3.11"',
67
]
78

‎snakemake/cli.py‎

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -954,10 +954,15 @@ def get_argument_parser(profiles=None):
954954
const="report.html",
955955
metavar="FILE",
956956
type=Path,
957-
help="Create an HTML report with results and statistics. "
958-
"This can be either a .html file or a .zip file. "
959-
"In the former case, all results are embedded into the .html (this only works for small data). "
960-
"In the latter case, results are stored along with a file report.html in the zip archive. "
957+
help="Create a self-contained HTML report with default statistics, "
958+
"provenance information and user-specified results. "
959+
"For smaller datasets with a limited report complexity, you can specify "
960+
"an '.html' file and all results will be embedded directly into this file. "
961+
"For customized reports on larger sample sizes, it makes more sense to "
962+
"specify a '.zip' file. The resulting archive will spread the contents "
963+
"across a folder structure, for a quicker loading of individual results. "
964+
"You can unpack this archive anywhere and open the 'report.html` file in "
965+
"its main folder to view the report in any web browser. "
961966
"If no filename is given, an embedded report.html is the default.",
962967
)
963968
group_report.add_argument(

‎snakemake/script/__init__.py‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1597,6 +1597,11 @@ def execute_script(self, fname, edit=False):
15971597
self._execute_cmd("bash {fname:q}", fname=fname)
15981598

15991599

1600+
class XonshScript(PythonScript):
1601+
def execute_script(self, fname, edit=False):
1602+
self._execute_cmd("xonsh {fname:q}", fname=fname)
1603+
1604+
16001605
def strip_re(regex: Pattern, s: str) -> Tuple[str, str]:
16011606
"""Strip a substring matching a regex from a string and return the stripped part
16021607
and the remainder of the original string.
@@ -1659,6 +1664,8 @@ def get_language(source_file, source):
16591664
language = "rust"
16601665
elif filename.endswith(".sh"):
16611666
language = "bash"
1667+
elif filename.endswith(".xsh"):
1668+
language = "xonsh"
16621669

16631670
# detect kernel language for Jupyter Notebooks
16641671
if language == "jupyter":
@@ -1722,10 +1729,11 @@ def script(
17221729
"julia": JuliaScript,
17231730
"rust": RustScript,
17241731
"bash": BashScript,
1732+
"xonsh": XonshScript,
17251733
}.get(language, None)
17261734
if exec_class is None:
17271735
raise ValueError(
1728-
"Unsupported script: Expecting either Python (.py), R (.R), RMarkdown (.Rmd) or Julia (.jl) script."
1736+
"Unsupported script: Expecting either Python (.py), R (.R), RMarkdown (.Rmd), Julia (.jl), Rust (.rs), Bash (.sh), or Xonsh (.xsh) script."
17291737
)
17301738

17311739
executor = exec_class(

‎snakemake/workflow.py‎

Lines changed: 14 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -1776,66 +1776,41 @@ def get_resource_value(value):
17761776
)
17771777
# TODO retrieve suitable singularity image
17781778

1779+
def check_may_use_software_deployment(method):
1780+
if ruleinfo.template_engine:
1781+
raise RuleException(
1782+
f"{method} directive is only allowed with "
1783+
"run, shell, script, notebook, or wrapper "
1784+
"directives (not with template_engine)",
1785+
rule=rule,
1786+
)
1787+
17791788
if ruleinfo.env_modules:
17801789
# If using environment modules and they are defined for the rule,
17811790
# ignore conda and singularity directive below.
17821791
# The reason is that this is likely intended in order to use
17831792
# a software stack specifically compiled for a particular
17841793
# HPC cluster.
1785-
invalid_rule = not (
1786-
ruleinfo.script
1787-
or ruleinfo.wrapper
1788-
or ruleinfo.shellcmd
1789-
or ruleinfo.notebook
1790-
)
1791-
if invalid_rule:
1792-
raise RuleException(
1793-
"envmodules directive is only allowed with "
1794-
"shell, script, notebook, or wrapper directives (not with run or the template_engine)",
1795-
rule=rule,
1796-
)
1794+
check_may_use_software_deployment("envmodules")
17971795
from snakemake.deployment.env_modules import EnvModules
17981796

17991797
rule.env_modules = EnvModules(*ruleinfo.env_modules)
18001798

18011799
if ruleinfo.conda_env:
1802-
if not (
1803-
ruleinfo.script
1804-
or ruleinfo.wrapper
1805-
or ruleinfo.shellcmd
1806-
or ruleinfo.notebook
1807-
):
1808-
raise RuleException(
1809-
"Conda environments are only allowed "
1810-
"with shell, script, notebook, or wrapper directives "
1811-
"(not with run or template_engine).",
1812-
rule=rule,
1813-
)
1800+
check_may_use_software_deployment("conda")
18141801

18151802
if isinstance(ruleinfo.conda_env, Path):
18161803
ruleinfo.conda_env = str(ruleinfo.conda_env)
18171804

18181805
rule.conda_env = ruleinfo.conda_env
18191806

1820-
invalid_rule = not (
1821-
ruleinfo.script
1822-
or ruleinfo.wrapper
1823-
or ruleinfo.shellcmd
1824-
or ruleinfo.notebook
1825-
)
18261807
if ruleinfo.container_img:
1827-
if invalid_rule:
1828-
raise RuleException(
1829-
"Singularity directive is only allowed "
1830-
"with shell, script, notebook or wrapper directives "
1831-
"(not with run or template_engine).",
1832-
rule=rule,
1833-
)
1808+
check_may_use_software_deployment("container/singularity")
18341809
rule.container_img = ruleinfo.container_img
18351810
rule.is_containerized = ruleinfo.is_containerized
18361811
elif self.global_container_img:
1837-
if not invalid_rule and ruleinfo.container_img != False:
1838-
# skip rules with run directive or empty image
1812+
if not ruleinfo.template_engine and ruleinfo.container_img != False:
1813+
# skip rules with template_engine directive or empty image
18391814
rule.container_img = self.global_container_img
18401815
rule.is_containerized = self.global_is_containerized
18411816

‎test-environment.yml‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ channels:
44
- nodefaults
55
dependencies:
66
- python >=3.11
7+
- setuptools
78
- yte
89
- packaging
910
- boto3
@@ -57,10 +58,10 @@ dependencies:
5758
- nodejs # for cwltool
5859
- immutables
5960
- conda
61+
- cwltool
62+
- cwl-utils
6063
- pip
6164
- pip:
62-
- cwltool
63-
- cwl-utils
6465
- snakemake-interface-executor-plugins >=9.3.1
6566
- snakemake-executor-plugin-cluster-generic >=1.0.9
6667
- snakemake-storage-plugin-http

‎tests/test_conda_python_3_7_script/Snakefile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,4 +4,4 @@ rule random_python_conda_script:
44
conda:
55
"test_python_env.yaml"
66
script:
7-
"test_script.py"
7+
"test_script_python_3_7.py"

tests/test_conda_python_3_7_script/test_script.py renamed to tests/test_conda_python_3_7_script/test_script_python_3_7.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
import platform
22
import PIL
33

4-
with open('version.txt', 'w') as f:
4+
with open("version.txt", "w") as f:
55
f.write(platform.python_version())

0 commit comments

Comments
 (0)