would instruct Snakemake to cache and reuse the results of the rules ``download_data`` and ``create_index``.

The environment variable definition that happens in the first line (defining the location of the cache) should of course be done only once and system wide in reality.

In any case, the provided location should be shared between all workflows of your group, institute or computing environment, in order to benefit from the reuse of previously obtained intermediate results.

Alternatively, rules can be marked as eligible for caching via the ``cache`` directive:

Both mechanisms can be particularly handy when used in combination with :ref:`non-local execution <non-local-exec>`.

.. _non-local-exec:

Non-local execution

In addition, we have moved the former remote provider functionality into so called :ref:`storage plugins <storage-support>`.

Most of the old remote providers have been migrated into the new storage plugins

Two former remote providers have been migrated into Snakemake wrappers instead, namely

the NCBI and EGA remote providers, which are now replaced by the

`entrez/efetch <https://snakemake-wrappers.readthedocs.io/en/stable/wrappers/entrez/efetch.html>`_ and

the `ega <https://snakemake-wrappers.readthedocs.io/en/stable/wrappers/ega/fetch.html>`_ wrappers.

Command line interface

Moreover, some options have been renamed:

* The ``--use-conda`` and ``--use-singularity`` options are deprecated. Instead you should now use ``--software-deployment-method conda`` or ``--software-deployment-method apptainer`` or ``--software-deployment-method conda apptainer`` if you need both.

* There is a new executor plugin for `Google Cloud Batch <https://cloud.google.com/batch/docs/get-started>`_.

This is meant as a replacement for the old Google Life Sciences executor.

The new executor is called ``googlebatch`` and can be used with ``--executor googlebatch``.

Note that in principle it is fine to re-add google-lifesciences support as a plugin as well.

Any help with getting this tested and released despite the fact that google lifesciences will be shut down this year would still be valued.

.. list-table:: Interface comparison

The Snakemake API has been completely rewritten into a modern `dataclass <https://docs.python.org/3/library/dataclasses.html>`_ based approach.

The traditional central ``snakemake()`` function is gone.

If you introduce a new feature you should add a new test to the tests directory. See the folder for examples.

Documentation Guidelines

My shell command fails with exit code != 0 from within a pipe, what's wrong?

Instructions for doing this are located `here

Note that you can also format Snakefiles in Vim using :ref:`snakefmt

<How should Snakefiles be formatted?>`, with instructions located `here

I want to import some helper functions from another python file. Is that possible?

I want to pass Python variables between rules. Is that possible?

Snakemake expects shell commands to behave properly, meaning that failures should cause an exit status other than zero. If a command does not exit with a status other than zero, Snakemake assumes everything worked fine, even if output files are empty. This is because empty output files are also a reasonable tool to indicate progress where no real output was produced. However, sometimes you will have to deal with tools that do not properly report their failure with an exit status. Here, you can use the :ref:`ensure function <snakefiles_ensure>` to mark output files that should not be empty, e.g.:

.. code-block:: python

You can then run ``gitmodtimes`` to update the modification times of all tracked files on the current branch to their last commit time in git; BE CAREFUL--this does not account for local changes that have not been committed.

How do I exit a running Snakemake workflow?

To activate this autocompletion permanently, put this line in ``~/.zshrc``.

How can I avoid system /tmp to be used when combining apptainer and conda?

It is **highly recommended** to run the linter before publishing any workflow, asking questions on Stack Overflow or filing issues on Github.

* There is an automatic formatter for Snakemake workflows, called `Snakefmt <https://github.com/snakemake/snakefmt>`_, which should be applied to any Snakemake workflow before publishing it.

* When publishing your workflow in a `Github <https://github.com>`_ repository, it is a good idea to add some minimal test data and configure `Github Actions <https://github.com/features/actions>`_ for continuously testing the workflow on each new commit.

* For publishing and distributing a Snakemake workflow, it is a good idea to stick to a :ref:`standardized structure <distribution_and_reproducibility>` that is expected by frequent users of Snakemake.

The `Snakemake workflow catalog <https://snakemake.github.io/snakemake-workflow-catalog>`_ automatically lists Snakemake workflows hosted on `Github <https://github.com>`_ if they follow certain `rules <https://snakemake.github.io/snakemake-workflow-catalog/?rules=true>`_.

By complying to these `rules <https://snakemake.github.io/snakemake-workflow-catalog/?rules=true>`_ you can make your workflow more discoverable and even automate its usage documentation (see `"Standardized usage" <https://snakemake.github.io/snakemake-workflow-catalog/?rules=true>`_).

Use such configuration for metadata and experiment information, **not for runtime specific configuration** like threads, resources and output folders.

For those, just rely on Snakemake's CLI arguments like ``--set-threads``, ``--set-resources``, ``--set-default-resources``, and ``--directory``.

This makes workflows more readable, scalable, and portable.

* Try to keep filenames short (thus easier on the eye), but informative. Avoid mixing of too many special characters (e.g. decide whether to use ``_`` or ``-`` as a separator and do that consistently throughout the workflow).

* Try to keep Python code like helper functions separate from rules (e.g. in a ``workflow/rules/common.smk`` file). This way, you help non-experts to read the workflow without needing to parse internals that are irrelevant for them. The helper function names should be chosen in a way that makes them sufficiently informative without looking at their content. Also avoid ``lambda`` expressions inside of rules.

If the `configfile` statement is not used, the config variable provides an empty array.

.. code-block:: console

Tabular configuration

Here, the workflow is executed as usual until this rule is reached.

Then, Snakemake passes all resources to the nextflow workflow management system, which generates certain files.

Instead, the invoked other workflow management system is responsible for that.

E.g., in case of `Nextflow <https://nextflow.io>`_, submission behavior can be configured via a ``nextflow.conf`` file or environment variables.

Common-Workflow-Language (CWL) support

A CWL tool definition can be used as follows.

.. code-block:: python

When using in combination with :ref:`--software-deployment-method apptainer <apptainer>` (``--sdm`` for short), Snakemake will instruct `cwltool` to execute the command via Singularity in user space.

Otherwise, `cwltool` will in most cases use a Docker container, which requires Docker to be set up properly.

In contrast to a :ref:`Snakemake wrapper <snakefiles-wrappers>`, CWL tool definitions are in general not suited to alter the behavior of a tool, e.g., by normalizing output names or special input handling.

As you can see in comparison to the analog :ref:`wrapper declaration <snakefiles-wrappers>` above, the rule becomes slightly more verbose, because input, output, and params have to be dispatched to the specific expectations of the CWL tool definition.