Skip to content

Commit a978c4e

Browse files
johanneskoesterjohanneskoester
authored
Merge branch 'main' into fix/issue3009
2 parents 38aed18 + 67fa392 commit a978c4e

File tree

107 files changed

+109074
-348
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

107 files changed

+109074
-348
lines changed

‎.github/workflows/main.yml‎

Lines changed: 20 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -49,10 +49,8 @@ jobs:
4949
matrix:
5050
test_group: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
5151
# see pyprojec.toml: [tool.pixi.feature.test] for available test types
52-
os: [ubuntu-latest, windows-latest] # macos-latest, macos-13 not supported yet
52+
os: [ubuntu-latest, windows-latest, macos-latest] # , macos-13 not supported yet
5353
env: ["py311", "py312"]
54-
shell:
55-
- bash
5654

5755
runs-on: ${{ matrix.os }}
5856

@@ -61,7 +59,6 @@ jobs:
6159
GCP_AVAILABLE: "${{ secrets.GCP_SA_KEY }}"
6260
ZENODO_SANDBOX_PAT: "${{ secrets.ZENODO_SANDBOX_PAT }}"
6361
CI: true
64-
6562
steps:
6663
- uses: actions/checkout@v4
6764
with:
@@ -105,27 +102,35 @@ jobs:
105102
pixi run --environment ${{matrix.env}} test-all \
106103
--splits 10 \
107104
--group ${{ matrix.test_group }} \
108-
--splitting-algorithm=least_duration
109-
110-
- name: Run Tests for Windows (os='Windows')
111-
if: runner.os == 'Windows'
112-
run: |
113-
pixi run --environment ${{matrix.env}} test-simple --splits 10 --group ${{ matrix.test_group }} --splitting-algorithm=least_duration
105+
--splitting-algorithm=least_duration \
106+
--showlocals \
107+
--show-capture=all
114108
115-
- name: Run Report Generation Tests (os='Linux')
116-
if: runner.os == 'Linux'
117-
shell: bash -el {0}
118-
run: |
119109
cd tests/test_report
120110
pixi run -e ${{ matrix.env }} snakemake \
121111
--use-conda \
122112
--cores 1 \
123113
--report report.zip
124114
115+
- name: Run Tests for MacOS (os='macOS')
116+
if: runner.os == 'macOS'
117+
run: |
118+
pixi run --environment ${{matrix.env}} test-simple \
119+
--splits 10 \
120+
--group ${{ matrix.test_group }} \
121+
--splitting-algorithm=least_duration \
122+
--showlocals \
123+
--show-capture=all
124+
125+
- name: Run Tests for Windows (os='Windows')
126+
if: runner.os == 'Windows'
127+
run: |
128+
pixi run --environment ${{matrix.env}} test-simple --splits 10 --group ${{ matrix.test_group }} --splitting-algorithm=least_duration --showlocals --show-capture=all
129+
125130
build-container-image:
126131
if: github.event.pull_request.merged != true || github.ref != 'refs/heads/main'
127132
runs-on: ubuntu-latest
128-
needs: tests
133+
needs: tests
129134
steps:
130135
- uses: actions/checkout@v4
131136

‎.readthedocs.yml‎

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -4,10 +4,13 @@ build:
44
os: ubuntu-22.04
55
tools:
66
python: "3.11"
7-
7+
# this ensures a viable `mamba` is on `$PATH``
8+
python: mambaforge-latest
9+
commands:
10+
- mamba install -c conda-forge -c nodefaults pixi
11+
- pixi install --environment docs
12+
- pixi run --environment docs build-docs
13+
- mkdir -p $READTHEDOCS_OUTPUT/html
14+
- rm -rf $READTHEDOCS_OUTPUT/html && mv docs/_build/html $READTHEDOCS_OUTPUT/html
815
sphinx:
916
configuration: docs/conf.py
10-
11-
python:
12-
install:
13-
- requirements: docs/requirements.txt

‎.test_durations‎

Lines changed: 283 additions & 235 deletions
Large diffs are not rendered by default.

‎apidocs/conf.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@
2222
# If extensions (or modules to document with autodoc) are in another directory,
2323
# add these directories to sys.path here. If the directory is relative to the
2424
# documentation root, use os.path.abspath to make it absolute, like shown here.
25-
sys.path.insert(0, os.path.abspath("../"))
25+
sys.path.insert(0, os.path.abspath("../src/"))
2626

2727
# -- General configuration ------------------------------------------------
2828

‎docs/conf.py‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,6 @@
1313
# All configuration values have a default; values that are commented out
1414
# serve to show the default.
1515

16-
import sys
1716
import os
1817
from datetime import datetime
1918
from sphinxawesome_theme.postprocess import Icons
@@ -22,7 +21,7 @@
2221
# If extensions (or modules to document with autodoc) are in another directory,
2322
# add these directories to sys.path here. If the directory is relative to the
2423
# documentation root, use os.path.abspath to make it absolute, like shown here.
25-
sys.path.insert(0, os.path.abspath("../"))
24+
# sys.path.insert(0, os.path.abspath("../src/"))
2625

2726
# -- General configuration ------------------------------------------------
2827

‎docs/project_info/contributing.rst‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -208,6 +208,24 @@ directly with the test file and the test name.
208208
--group 1 \
209209
--splitting-algorithm=least_duration
210210
211+
Marked tests
212+
------------
213+
Some tests have been marked using `pytest markers <https://docs.pytest.org/en/stable/mark.html>`_.
214+
These allow for running specific tests or *excluding* specific tests.
215+
For example, the `pixi run test-simple` currently excludes the `needs_envmodules` tests.
216+
There is also another marker for ``needs_s3`` which will skip tests that require an S3 connection.
217+
If you are not looking to test the S3 functionality, you can modify the
218+
test command to exclude these tests.
219+
220+
.. code-block:: console
221+
222+
$ pixi run test-simple -m "not needs_envmodules and not needs_s3"
223+
224+
For a full list of available markers, you can run:
225+
226+
.. code-block:: console
227+
228+
$ pixi run pytest --markers
211229
212230
Warnings and oddities
213231
---------------------

‎docs/snakefiles/deployment.rst‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -456,6 +456,10 @@ Note that the isolation of jobs running in containers depends on the container e
456456
For example, Docker does not pass any host environment variables to the container, whereas Apptainer/Singularity passes everything.
457457
To override the default behaviour, consider using ``--apptainer-args`` or ``--singularity-args``, e.g. to pass ``--cleanenv``.
458458

459+
Files that are mounted using `params` using `workflow.source_path` are also automatically available in the container. This is realized by mounting the snakemake cache in the container (/home/<user>/.cache/snakemake/snakemake/source-cache) where the sourced files will be cached.
460+
461+
In general, it should be noted that only trusted containers should be used!
462+
459463
Defining global container images
460464
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
461465

‎docs/snakefiles/modularization.rst‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -201,6 +201,47 @@ Otherwise, you will have two versions of the same rule, which might be unintende
201201

202202
Of course, it is possible to combine the use of rules from multiple modules (see :ref:`use_with_modules`), and via modifying statements they can be rewired and reconfigured in an arbitrary way.
203203

204+
.. _snakefiles-dynamic-modules:
205+
206+
---------------
207+
Dynamic Modules
208+
---------------
209+
210+
With Snakemake 9.0 and later, it is possible to load modules dynamically by providing the ``name`` keyword inside the module definition.
211+
For example, by reading the module name from a config file or by iterating over several modules in a loop.
212+
For this, the module name is not specified directly after the ``module`` keyword, but by specifying the ``name`` parameter.
213+
214+
215+
.. code-block:: python
216+
217+
for module_name in ['module1', 'module2']:
218+
module:
219+
name: module_name
220+
snakefile: f"{module_name}/Snakefile"
221+
config: config[module_name]
222+
223+
use rule * from module_name as module_name*
224+
225+
.. note::
226+
It is not allowed to specify the module name both after the ``module`` keyword and inside the module definition after the ``name`` parameter.
227+
228+
In the ``use rule`` statement, it is first checked if the module name (here, ``'module_name'``) corresponds to a loaded module. If yes, the rules are imported from the loaded module and an arbitrary alias can be provided after the ``as`` keyword.
229+
230+
If ``module_name`` was not registered as a module (as in the example above), the module name is resolved dynamically by searching the name in the current python variable scope. In the example, it resolves to ``'module1'`` and ``'module2'``.
231+
Note that this means that if ``use rule`` is used with the optional ``as`` keyword inside the loop, the alias after ``as`` must be specified using a variable to ensure a one-to-one mapping between module names and their aliases. This can either be the same name variable (as in the above example) or a second variable (as in the example below).
232+
233+
In particular, it is not possible to modify the alias name in the ``use rule`` statement (e.g., writing directly ``use rule * from module as module_*`` is not allowed for dynamic modules).
234+
235+
.. code-block:: python
236+
237+
for module_name, alias in zip(['module1', 'module2'], ['module1_', 'module2_']):
238+
module:
239+
name: module_name
240+
snakefile: f"{module_name}/Snakefile"
241+
config: config[module_name]
242+
243+
use rule * from module_name as alias*
244+
204245
.. _snakefiles-meta-wrappers:
205246

206247
~~~~~~~~~~~~~

‎docs/snakefiles/rules.rst‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -698,6 +698,18 @@ The ``subpath`` function can be very handy in combination with :ref:`Snakemake's
698698
shell:
699699
"somecommand {input} --name {params.basename} --outdir {params.outdir}"
700700
701+
702+
.. _snakefiles-flatten:
703+
704+
flatten
705+
"""""""
706+
When selecting input files, sometimes you might end up with an irregular list of lists. To flatten in, you can use:
707+
708+
.. code-block:: python
709+
710+
flatten([1, "a", [2,"b"], ["c","d",["e", 3]]]) # returns ["1", "a", "2", "b", "c", "d", "e", "3"]
711+
712+
701713
.. _snakefiles-targets:
702714

703715
Target rules
@@ -1818,6 +1830,30 @@ Further, an output file marked as ``temp`` is deleted after all rules that use i
18181830
18191831
.. _snakefiles-directory_output:
18201832

1833+
Auto-grouping via temp files upon remote execution
1834+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1835+
1836+
For performance reasons, it is sometimes useful to write intermediate files on a faster storage, e.g., attached locally on the cluster compute node rather than shared over the network (and thus neither visible to the main snakemake process that submits jobs to the cluster, nor to other nodes of the cluster).
1837+
Snakemake (since version 9.0) allows files marked as ``temp`` to use the option ``group_jobs`` to indicate that rules creating and consuming them should be automatically :ref:`grouped <job_grouping>` together so Snakemake will schedule them to run on the same physical node:
1838+
1839+
.. code-block:: python
1840+
1841+
rule NAME1:
1842+
input:
1843+
"path/to/inputfile"
1844+
output:
1845+
temp("path/to/intermediatefile", group_jobs=True)
1846+
shell:
1847+
"somecommand {input} {output}"
1848+
1849+
rule NAME2:
1850+
input:
1851+
"path/to/intermediatefile"
1852+
output:
1853+
"path/to/outputfile"
1854+
shell:
1855+
"someothercommand {input} {output}"
1856+
18211857
Directories as outputs
18221858
----------------------
18231859

‎docs/snakefiles/storage.rst‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -138,6 +138,68 @@ each time providing a different tag::
138138
shell:
139139
"..."
140140

141+
Retrieving and keeping remote files locally
142+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
143+
144+
When the input file is a remote file, the default behaviour is to download the remote
145+
file to the local area and then remove it after the workflow no longer needs it.
146+
147+
This behaviour can be configured from the command line arguments, using::
148+
149+
snakemake --keep-storage-local-copies --not-retrieve-storage
150+
151+
where ``--keep-storage-local-copies`` directs snakemake to keep the local copies of
152+
remote files that it makes and ``--not-retrieve-storage`` directs snakemake to not download
153+
copies of remote files.
154+
155+
Additionally, this behaviour can be set at the level of the storage directive e.g.::
156+
157+
storage:
158+
provider="http",
159+
retrieve=False,
160+
161+
storage http_local:
162+
provider="http",
163+
keep_local=True,
164+
165+
rule example_remote:
166+
input:
167+
storage.http("http://example.com/example.txt")
168+
output:
169+
"example_remote.txt"
170+
shell:
171+
"..."
172+
173+
rule example_local:
174+
input:
175+
storage.http_local("http://example.com/example.txt")
176+
output:
177+
"example_local.txt"
178+
shell:
179+
"..."
180+
181+
Finally, ``retrieve`` and ``keep_local`` can also be set inside the call to the storage
182+
plugin within a rule::
183+
184+
storage:
185+
provider="http",
186+
retrieve=False,
187+
188+
rule example_remote:
189+
input:
190+
storage.http("http://example.com/example.txt", retrieve=False)
191+
output:
192+
"example_remote.txt"
193+
shell:
194+
"..."
195+
196+
rule example_local:
197+
input:
198+
storage.http("http://example.com/example.txt", keep_local=True)
199+
output:
200+
"example_local.txt"
201+
shell:
202+
"..."
141203

142204
Automatic inference
143205
^^^^^^^^^^^^^^^^^^^

0 commit comments

Comments
 (0)