Merge branch 'main' into xonsh-support

johanneskoester · web-flow · commit c2e9c52ff12c · 2025-03-10T18:16:26.000+01:00
diff --git a/docs/snakefiles/deployment.rst b/docs/snakefiles/deployment.rst
@@ -285,8 +285,12 @@ At the moment, only a single, random conda environment is shown.
 
 .. note::
 
-   Note that conda environments are only used with ``shell``, ``script``, ``notebook`` and the ``wrapper`` directive, not the ``run`` directive.
-   The reason is that the ``run`` directive has access to the rest of the Snakefile (e.g. globally defined variables) and therefore must be executed in the same process as Snakemake itself. If used with ``notebook`` directive, the associated conda environment should have package ``jupyter`` installed (this package contains dependencies required to execute the notebook).
+   Note that conda environments can be used with the ``shell``, ``script``, ``notebook``, ``wrapper`` and ``run`` directives.
+   
+   However, the ``run`` directive is a special case, as it has access to the rest of the Snakefile (e.g. globally defined variables) and therefore must be executed in the same process as Snakemake itself. 
+   The ``conda`` directive for rules with a ``run`` directive therefore only affects ``shell`` function calls that are executed from the within ``run`` script.
+   
+   If used with ``notebook`` directive, the associated conda environment should have package ``jupyter`` installed (this package contains dependencies required to execute the notebook).
 
    Further, note that search path modifying environment variables like ``R_LIBS`` and ``PYTHONPATH`` can interfere with your conda environments.
    Therefore, Snakemake automatically deactivates them for a job when a conda environment definition is used.
@@ -457,8 +461,10 @@ Defining global container images
 
 .. note::
 
-   Note that apptainer integration is only used with ``shell``, ``script`` and the ``wrapper`` directive, not the ``run`` directive.
-   The reason is that the ``run`` directive has access to the rest of the Snakefile (e.g. globally defined variables) and therefore must be executed in the same process as Snakemake itself.
+   Note that apptainer integration can be used with the ``shell``, ``script``, ``wrapper`` and ``run`` directives.
+   
+   However, the ``run`` directive is a special case, as it has access to the rest of the Snakefile (e.g. globally defined variables) and therefore must be executed in the same process as Snakemake itself. 
+   The ``container`` directive for rules with a ``run`` directive therefore only affects ``shell`` function calls that are executed from within the ``run`` script.
 
 A global definition of a container image can be given:
 
diff --git a/docs/snakefiles/reporting.rst b/docs/snakefiles/reporting.rst
@@ -4,9 +4,17 @@
 Reports
 =======
 
-From Snakemake 5.1 on, it is possible to automatically generate detailed self-contained HTML reports that encompass runtime statistics, provenance information, workflow topology and results.
+From Snakemake 5.1 on, it is possible to automatically generate detailed self-contained HTML reports that you can easily move and share. 
+They encompass runtime statistics, provenance information and workflow topology by default, and workflow developers can also specify and annotate results for inclusion.
+For smaller default reports, these can be a single :ref:`snakefiles-self_contained_html_file`.
+For more complex reports, one can generate a :ref:`snakefiles-self_contained_zip_archive`, with a contained ``report.html`` as the main entry point.
 **As an example, the report of the Snakemake rolling paper can be found** `here <https://snakemake.github.io/resources/report.html>`__.
 
+.. _snakefiles-including_results_in_a_report:
+
+Including results in a report
+-----------------------------
+
 For including results into the report, the Snakefile has to be annotated with additional information.
 Each output file that shall be part of the report has to be marked with the ``report`` flag, which optionally points to a caption in `restructured text format <https://docutils.sourceforge.io/docs/user/rst/quickstart.html>`_ and allows to define a ``category`` for grouping purposes.
 Moreover, a global workflow description can be defined via the ``report`` directive.
@@ -94,7 +102,7 @@ This works as follows:
             """
 
 Defining file labels
---------------------
+^^^^^^^^^^^^^^^^^^^^
 
 In addition to category, and subcategory, it is possible (and highly recommended!) to define a dictionary of labels for each report item.
 By that, the actual filename will be hidden in the report and instead a table with the label keys as columns and the values in the respective row for the file will be displayed.
@@ -131,17 +139,17 @@ This behavior can be used to, for example, switch between different versions of
 
 
 Determining category, subcategory, and labels dynamically via functions
------------------------------------------------------------------------
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Similar to e.g. with input file and parameter definition (see :ref:`snakefiles-input_functions`), ``category`` and a ``subcategory`` and ``labels`` can be specified by pointing to a function that takes ``wildcards`` as the first argument (and optionally in addition ``input``, ``output``, ``params`` in any order).
 The function is expected to return a string or number (int, float, numpy types), or, in case of labels, a dict with strings as keys and strings or numbers as values.
 
 
 Linking between items
----------------------
+^^^^^^^^^^^^^^^^^^^^^
 
 From captions
-^^^^^^^^^^^^^
+"""""""""""""
 
 In every ``.rst`` document (i.e. in the captions), you can link to
 
@@ -153,7 +161,7 @@ In every ``.rst`` document (i.e. in the captions), you can link to
 For details about the hyperlink mechanism of restructured text see `here <https://docutils.sourceforge.io/docs/user/rst/quickref.html#hyperlink-targets>`__.
 
 From results
-^^^^^^^^^^^^
+""""""""""""
 
 From within results that are included into the report, you can link to other report items.
 This works by using the ``snakemake.report_href()`` method that is available from within :ref:`python scripts <snakefiles-external_scripts>`.
@@ -246,36 +254,78 @@ Further, using ``url_args()`` you can add URL arguments and using ``anchor()`` y
             file=f,
         )
 
+.. _snakefiles-rendering_reports:
+
 Rendering reports
 -----------------
 
-To create the report simply run
+All the metadata contained in the report (e.g. runtime statistics) are automatically collected during the rendering of the report.
+These statistics are obtained from the metadata that is stored in the ``.snakemake`` directory inside your working directory.
+
+.. _snakefiles-self_contained_html_file:
+
+Self-contained HTML file
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+To create a default report simply run
 
 .. code-block:: bash
 
-    snakemake --report report.html
+    snakemake --report
 
-after your workflow has finished.
-All other information contained in the report (e.g. runtime statistics) is automatically collected during creation.
-These statistics are obtained from the metadata that is stored in the ``.snakemake`` directory inside your working directory.
+after your workflow has finished successfully.
+This will create a self-contained HTML report with the default filename ``report.html``.
+If you want to give your report a custom name, you can do so by specifying this on the command-line:
+
+.. code-block:: bash
+
+    snakemake --report my_report.html
+
+As specifying an HTML file for the report will always embed any user-defined report outputs into the same HTML file, this report output type is only suitable for smaller reports.
 
+.. _snakefiles-self_contained_zip_archive:
 
-You can define an institute specific stylesheet with:
+Self-contained ZIP archive
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+For anything more complex, it is recommended to generate a ``ZIP`` archive report.
+For example, if you have lots of samples you report on, or if you specified lots of different workflow outputs for report inclusion.
+To get such a self-contained ``ZIP`` archive, simply specify a ``.zip`` file name instead:
 
 .. code-block:: bash
 
-    snakemake --report report.html --report-stylesheet custom-stylesheet.css
+    snakemake --report some_report.zip
 
-In particular, this allows you to e.g. set a logo at the top (by using CSS to inject a background for the placeholder ``<div id="brand">``, or overwrite colors.
-For an example custom stylesheet defining the logo, see :download:`here <../../tests/test_report/custom-stylesheet.css>`.
-The report for above example can be found :download:`here <../../tests/test_report/expected-results/report.html>` (with a custom branding for the University of Duisburg-Essen).
-The full example source code can be found `here <https://github.com/snakemake/snakemake/tree/main/tests/test_report/>`__.
+You can move this file wherever you want to view it and then unpack it right there.
+This includes sending this file to collaboration partners or customers for whom you might have done an analysis.
+The main entry point is always the ``report.html`` file in the main folder that this creates (for the above example, this folder will be named ``some_report``).
+Just open that file in a web browser and explore your results.
 
-Note that the report can be restricted to particular jobs and results by specifying targets at the command line, analog to normal Snakemake execution.
+Partial reports
+^^^^^^^^^^^^^^^
+
+The report can be restricted to particular jobs and results by specifying targets at the command line, analog to normal Snakemake execution.
 For example, with
 
 .. code-block:: bash
 
     snakemake fig1.svg --report report-short.html
 
 the report contains only ``fig1.svg``.
+This can be useful when a larger workflow has not yet run to completion, but you already want to explore some intermediate outputs in the report.
+Or when you have multiple alternative target rules within the same workflow.
+
+Custom layout
+^^^^^^^^^^^^^
+
+You can define an institute specific layout by providing a custom stylesheet:
+
+.. code-block:: bash
+
+    snakemake --report report.html --report-stylesheet custom-stylesheet.css
+
+For example, this allows you to set a logo at the top (by using CSS to inject a background for the placeholder ``<div id="brand">``), or overwrite colors.
+
+For an example with a custom stylesheet defining a logo, see :download:`the report here <../../tests/test_report/expected-results/report.html>` (with a custom branding for the University of Duisburg-Essen).
+For the complete mechanics, you can also have a look at the `full example source code  <https://github.com/snakemake/snakemake/tree/main/tests/test_report/>`__ and :download:`the custom stylesheet with the logo definition <../../tests/test_report/custom-stylesheet.css>`.
+
diff --git a/snakemake/cli.py b/snakemake/cli.py
@@ -954,10 +954,15 @@ def get_argument_parser(profiles=None):
         const="report.html",
         metavar="FILE",
         type=Path,
-        help="Create an HTML report with results and statistics. "
-        "This can be either a .html file or a .zip file. "
-        "In the former case, all results are embedded into the .html (this only works for small data). "
-        "In the latter case, results are stored along with a file report.html in the zip archive. "
+        help="Create a self-contained HTML report with default statistics, "
+        "provenance information and user-specified results. "
+        "For smaller datasets with a limited report complexity, you can specify "
+        "an '.html' file and all results will be embedded directly into this file. "
+        "For customized reports on larger sample sizes, it makes more sense to "
+        "specify a '.zip' file. The resulting archive will spread the contents "
+        "across a folder structure, for a quicker loading of individual results. "
+        "You can unpack this archive anywhere and open the 'report.html` file in "
+        "its main folder to view the report in any web browser. "
         "If no filename is given, an embedded report.html is the default.",
     )
     group_report.add_argument(
diff --git a/snakemake/workflow.py b/snakemake/workflow.py
@@ -1776,66 +1776,41 @@ def get_resource_value(value):
                 )
                 # TODO retrieve suitable singularity image
 
+            def check_may_use_software_deployment(method):
+                if ruleinfo.template_engine:
+                    raise RuleException(
+                        f"{method} directive is only allowed with "
+                        "run, shell, script, notebook, or wrapper "
+                        "directives (not with template_engine)",
+                        rule=rule,
+                    )
+
             if ruleinfo.env_modules:
                 # If using environment modules and they are defined for the rule,
                 # ignore conda and singularity directive below.
                 # The reason is that this is likely intended in order to use
                 # a software stack specifically compiled for a particular
                 # HPC cluster.
-                invalid_rule = not (
-                    ruleinfo.script
-                    or ruleinfo.wrapper
-                    or ruleinfo.shellcmd
-                    or ruleinfo.notebook
-                )
-                if invalid_rule:
-                    raise RuleException(
-                        "envmodules directive is only allowed with "
-                        "shell, script, notebook, or wrapper directives (not with run or the template_engine)",
-                        rule=rule,
-                    )
+                check_may_use_software_deployment("envmodules")
                 from snakemake.deployment.env_modules import EnvModules
 
                 rule.env_modules = EnvModules(*ruleinfo.env_modules)
 
             if ruleinfo.conda_env:
-                if not (
-                    ruleinfo.script
-                    or ruleinfo.wrapper
-                    or ruleinfo.shellcmd
-                    or ruleinfo.notebook
-                ):
-                    raise RuleException(
-                        "Conda environments are only allowed "
-                        "with shell, script, notebook, or wrapper directives "
-                        "(not with run or template_engine).",
-                        rule=rule,
-                    )
+                check_may_use_software_deployment("conda")
 
                 if isinstance(ruleinfo.conda_env, Path):
                     ruleinfo.conda_env = str(ruleinfo.conda_env)
 
                 rule.conda_env = ruleinfo.conda_env
 
-            invalid_rule = not (
-                ruleinfo.script
-                or ruleinfo.wrapper
-                or ruleinfo.shellcmd
-                or ruleinfo.notebook
-            )
             if ruleinfo.container_img:
-                if invalid_rule:
-                    raise RuleException(
-                        "Singularity directive is only allowed "
-                        "with shell, script, notebook or wrapper directives "
-                        "(not with run or template_engine).",
-                        rule=rule,
-                    )
+                check_may_use_software_deployment("container/singularity")
                 rule.container_img = ruleinfo.container_img
                 rule.is_containerized = ruleinfo.is_containerized
             elif self.global_container_img:
-                if not invalid_rule and ruleinfo.container_img != False:
-                    # skip rules with run directive or empty image
+                if not ruleinfo.template_engine and ruleinfo.container_img != False:
+                    # skip rules with template_engine directive or empty image
                     rule.container_img = self.global_container_img
                     rule.is_containerized = self.global_is_containerized
 
diff --git a/tests/test_conda_python_3_7_script/Snakefile b/tests/test_conda_python_3_7_script/Snakefile
@@ -4,4 +4,4 @@ rule random_python_conda_script:
 	conda:
 		"test_python_env.yaml"
 	script:
-		"test_script.py"
+		"test_script_python_3_7.py"
diff --git a/tests/test_conda_python_3_7_script/test_script_python_3_7.py b/tests/test_conda_python_3_7_script/test_script_python_3_7.py
@@ -1,5 +1,5 @@
 import platform
 import PIL
 
-with open('version.txt', 'w') as f:
+with open("version.txt", "w") as f:
     f.write(platform.python_version())
diff --git a/tests/test_conda_run/Snakefile b/tests/test_conda_run/Snakefile
@@ -0,0 +1,13 @@
+rule random_python_conda_script:
+    output:
+        "test.txt"
+    conda:
+        "test_python_env.yaml"
+    params: 
+        script=workflow.source_path('test_script_run.py')
+    run:
+        # Calling a script like this is usually unnecessary with Snakemake
+        # we use this pattern just for testing purposes here.
+        # In a real workflow, use the script directive instead which
+        # is cleaner and gives you many more benefits.
+        shell("python {params.script}")
diff --git a/tests/test_conda_run/expected-results/test.txt b/tests/test_conda_run/expected-results/test.txt
@@ -0,0 +1 @@
+3.0
diff --git a/tests/test_conda_run/test_python_env.yaml b/tests/test_conda_run/test_python_env.yaml
@@ -0,0 +1,6 @@
+channels:
+  - conda-forge
+  - defaults
+dependencies:
+  - numpy
+  - python
diff --git a/tests/test_conda_run/test_script_run.py b/tests/test_conda_run/test_script_run.py
@@ -0,0 +1,4 @@
+import numpy
+
+with open("test.txt", "w") as f:
+    f.write(str(numpy.log2(8)))
diff --git a/tests/tests_using_conda.py b/tests/tests_using_conda.py
@@ -315,6 +315,11 @@ def test_script_xsh():
     )
 
 
+@conda
+def test_conda_run():
+    run(dpath("test_conda_run"), deployment_method={DeploymentMethod.CONDA})
+
+
 # These tests have no explicit dependency on Conda and do not build new conda envs,
 # but will fail if 'conda info --json' does not work as expected, because the wrapper
 # code uses this to examine the installed software environment.
