Skip to content

Commit 6e2dcec

Browse files
SrPeter128SrPeter128
authored
Merge branch 'main' into fix/issue-3338
2 parents 44bb8c4 + 4ba62fe commit 6e2dcec

File tree

10 files changed

+736
-75
lines changed

10 files changed

+736
-75
lines changed

‎docs/project_info/codebase.rst‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -103,7 +103,7 @@ Besides these central classes, the following modules add additional functionalit
103103

104104
ioutils
105105
"""""""
106-
The ``ioutils`` module (`snakemake/ioutils <https://github.com/snakemake/snakemake/blob/main/snakemake/ioutils>`__) implements semantic helper functions functions for handling input and output files as well as non-file parameters in the workflow.
106+
The ``ioutils`` module (`snakemake/ioutils <https://github.com/snakemake/snakemake/blob/main/snakemake/ioutils>`__) implements semantic helper functions for handling input and output files as well as non-file parameters in the workflow.
107107

108108
linting
109109
"""""""

‎docs/project_info/contributing.rst‎

Lines changed: 17 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -185,46 +185,32 @@ For the documentation, please adhere to the following guidelines:
185185
- Put each sentence on its own line, this makes tracking changes through Git SCM easier.
186186
- Provide hyperlink targets, at least for the first two section levels.
187187
For this, use the format ``<document_part>-<section_name>``, e.g., ``project_info-doc_guidelines``.
188-
- Use the section structure from below.
188+
- Use the `section structure recommended by Sphinx <https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#sections>`_, which references the `recommendations in the Python Developer's Guide <https://devguide.python.org/documentation/markup/#sections>`_.
189+
Namely, the levels are:
189190

190191
::
191192

192-
.. document_part-heading_1:
193+
.. document_part-section_heading:
193194

194-
=========
195-
Heading 1
196-
=========
195+
===============
196+
Section heading
197+
===============
197198

198199

199-
.. document_part-heading_2:
200+
.. document_part-subsection_heading:
200201

201-
---------
202-
Heading 2
203-
---------
202+
Subsection heading
203+
------------------
204204

205+
.. document_part-subsubsection_heading:
205206

206-
.. document_part-heading_3:
207+
Subsubsection heading
208+
^^^^^^^^^^^^^^^^^^^^^
207209

208-
Heading 3
209-
=========
210+
.. document_part-paragraph_heading:
210211

211-
212-
.. document_part-heading_4:
213-
214-
Heading 4
215-
---------
216-
217-
218-
.. document_part-heading_5:
219-
220-
Heading 5
221-
~~~~~~~~~
222-
223-
224-
.. document_part-heading_6:
225-
226-
Heading 6
227-
:::::::::
212+
Paragraph heading
213+
"""""""""""""""""
228214

229215
.. _doc_setup:
230216

@@ -239,7 +225,8 @@ Snakemake development environment via
239225
240226
$ git clone [email protected]:snakemake/snakemake.git
241227
$ cd snakemake
242-
$ conda env create -f doc-environment.yml -n snakemake
228+
$ conda env create -f doc-environment.yml -n snakemake_docs
229+
$ conda activate snakemake_docs
243230
244231
You will also need to install your development version of Snakemake for the docs to be built correctly
245232

@@ -251,7 +238,6 @@ Then, the docs can be built with
251238

252239
.. code-block:: console
253240
254-
$ conda activate snakemake
255241
$ cd docs
256242
$ make html
257243
$ make clean && make html # force rebuild

‎docs/snakefiles/configuration.rst‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -112,8 +112,8 @@ Instead, for data provenance and reproducibility reasons, you are required to pa
112112
Validation
113113
----------
114114

115-
With Snakemake 5.1, it is possible to validate both types of configuration via `JSON schemas <https://json-schema.org>`_.
116-
The function ``snakemake.utils.validate`` takes a loaded configuration (a config dictionary or a Pandas data frame) and validates it with a given JSON schema.
115+
With Snakemake 5.1, it is possible to validate both types of configuration (standard and tabular) via `JSON schemas <https://json-schema.org>`_.
116+
The function ``snakemake.utils.validate`` takes a loaded configuration (a config dictionary, a Pandas DataFrame, Polars DataFrame or Polars LazyFrame) and validates it with a given JSON schema.
117117
Thereby, the schema can be provided in JSON or YAML format. Also, by using the defaults property it is possible to populate entries with default values. See `jsonschema FAQ on setting default values <https://python-jsonschema.readthedocs.io/en/latest/faq/>`_ for details.
118118
In case of the data frame, the schema should model the record that is expected in each row of the data frame.
119119
In the following example,

‎snakemake/assets/__init__.py‎

Lines changed: 400 additions & 0 deletions
Large diffs are not rendered by default.

‎snakemake/dag.py‎

Lines changed: 19 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -409,25 +409,28 @@ async def store_storage_outputs(self):
409409
if self.workflow.remote_exec:
410410
logger.info("Storing output in storage.")
411411
try:
412-
async with asyncio.TaskGroup() as tg:
413-
for job in self.needrun_jobs(exclude_finished=False):
414-
benchmark = [job.benchmark] if job.benchmark else []
415-
416-
async def tostore(f):
417-
return (
418-
f.is_storage
419-
and f
420-
not in self.workflow.storage_settings.unneeded_temp_files
421-
and await f.exists_local()
422-
)
412+
for level in self.toposorted(
413+
set(self.needrun_jobs(exclude_finished=False))
414+
):
415+
async with asyncio.TaskGroup() as tg:
416+
for job in level:
417+
benchmark = [job.benchmark] if job.benchmark else []
418+
419+
async def tostore(f):
420+
return (
421+
f.is_storage
422+
and f
423+
not in self.workflow.storage_settings.unneeded_temp_files
424+
and await f.exists_local()
425+
)
423426

424-
if self.finished(job):
425-
for f in chain(job.output, benchmark):
427+
if self.finished(job):
428+
for f in chain(job.output, benchmark):
429+
if await tostore(f):
430+
tg.create_task(f.store_in_storage())
431+
for f in job.log:
426432
if await tostore(f):
427433
tg.create_task(f.store_in_storage())
428-
for f in job.log:
429-
if await tostore(f):
430-
tg.create_task(f.store_in_storage())
431434
except ExceptionGroup as e:
432435
raise WorkflowError("Failed to store output in storage.", e)
433436

‎snakemake/report/html_reporter/data/packages.py‎

Lines changed: 137 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,14 +38,151 @@ def get_packages():
3838
source_path="vega/vega.js",
3939
license_path="vega/LICENSE",
4040
),
41+
# Begin dependencies for vega, included in vega/vega.js
42+
"d3-array": Package(
43+
license_path="d3-array/LICENSE",
44+
),
45+
"d3-format": Package(
46+
license_path="d3-format/LICENSE",
47+
),
48+
"d3-time-format": Package(
49+
license_path="d3-time-format/LICENSE",
50+
),
51+
"d3-time": Package(
52+
license_path="d3-time/LICENSE",
53+
),
54+
"d3-interpolate": Package(
55+
license_path="d3-interpolate/LICENSE",
56+
),
57+
"d3-color": Package(
58+
license_path="d3-color/LICENSE",
59+
),
60+
"d3-scale": Package(
61+
license_path="d3-scale/LICENSE",
62+
),
63+
"@types-estree": Package(
64+
license_path="@types-estree/LICENSE",
65+
),
66+
"d3-force": Package(
67+
license_path="d3-force/LICENSE",
68+
),
69+
"d3-dispatch": Package(
70+
license_path="d3-dispatch/LICENSE",
71+
),
72+
"d3-quadtree": Package(
73+
license_path="d3-quadtree/LICENSE",
74+
),
75+
"d3-timer": Package(
76+
license_path="d3-timer/LICENSE",
77+
),
78+
"d3-geo": Package(
79+
license_path="d3-geo/LICENSE",
80+
),
81+
"d3-hierarchy": Package(
82+
license_path="d3-hierarchy/LICENSE",
83+
),
84+
"d3-dsv": Package(
85+
license_path="d3-dsv/LICENSE",
86+
),
87+
"topojson-client": Package(
88+
license_path="topojson-client/LICENSE",
89+
),
90+
"d3-geo-projection": Package(
91+
license_path="d3-geo-projection/LICENSE",
92+
),
93+
"d3-path": Package(
94+
license_path="d3-path/LICENSE",
95+
),
96+
"d3-shape": Package(
97+
license_path="d3-shape/LICENSE",
98+
),
99+
"d3-delaunay": Package(
100+
license_path="d3-delaunay/LICENSE",
101+
),
102+
"delaunator": Package(
103+
license_path="delaunator/LICENSE",
104+
),
105+
# End dependencies for vega, included in vega/vega.js
106+
# Begin copied/derived/adapted code in vega, included in vega/vega.js
107+
"hashlru": Package(
108+
license_path="hashlru/LICENSE",
109+
),
110+
"d3-regression": Package(
111+
license_path="d3-regression/LICENSE",
112+
),
113+
"regression": Package(
114+
license_path="regression/LICENSE",
115+
),
116+
"science": Package(
117+
license_path="science/LICENSE",
118+
),
119+
"quickselect": Package(
120+
license_path="quickselect/LICENSE",
121+
),
122+
"commons-math": Package(
123+
license_path="commons-math/LICENSE.txt",
124+
# This must be included with LICENSE.txt.
125+
notice="commons-math/NOTICE.txt",
126+
),
127+
"esprima": Package(
128+
license_path="esprima/LICENSE.BSD",
129+
),
130+
"fabric": Package(
131+
license_path="fabric/LICENSE",
132+
),
133+
"d3-contour": Package(
134+
license_path="d3-contour/LICENSE",
135+
),
136+
# End copied/derived/adapted code in vega, included in vega/vega.js
41137
"vega-lite": Package(
42138
source_path="vega-lite/vega-lite.js",
43139
license_path="vega-lite/LICENSE",
44140
),
141+
# Begin dependencies for vega-lite, included in vega-lite/vega-lite.js
142+
# (excluding those shared with vega and therefore already documented)
143+
"@types-clone": Package(
144+
license_path="@types-estree/LICENSE",
145+
),
146+
"array-flat-polyfill": Package(
147+
license_path="array-flat-polyfill/LICENSE",
148+
),
149+
"clone": Package(
150+
license_path="clone/LICENSE",
151+
),
152+
"fast-deep-equal": Package(
153+
license_path="fast-deep-equal/LICENSE",
154+
),
155+
"fast-json-stable-stringify": Package(
156+
license_path="fast-json-stable-stringify/LICENSE",
157+
),
158+
"json-stringify-pretty-compact": Package(
159+
license_path="json-stringify-pretty-compact/LICENSE",
160+
),
161+
# End dependencies for vega-lite, included in vega-lite/vega-lite.js
162+
# Any copied/derived/adapted code in vega-lite is shared with vega.
45163
"vega-embed": Package(
46164
source_path="vega-embed/vega-embed.js",
47165
license_path="vega-embed/LICENSE",
48166
),
167+
# Begin dependencies for vega-embed, included in vega-embed/vega-embed.js
168+
# (excluding those shared with vega/vega-lite and therefore already documented)
169+
"fast-json-patch": Package(
170+
license_path="fast-json-patch/LICENSE",
171+
),
172+
"semver": Package(
173+
license_path="semver/LICENSE",
174+
),
175+
"vega-schema-url-parser": Package(
176+
license_path="vega-schema-url-parser/LICENSE",
177+
),
178+
"vega-themes": Package(
179+
license_path="vega-themes/LICENSE",
180+
),
181+
"vega-tooltip": Package(
182+
license_path="vega-tooltip/LICENSE",
183+
),
184+
# End dependencies for vega-embed, included in vega-embed/vega-embed.js
185+
# Any copied/derived/adapted code in vega-embed is shared with vega and/or vega-lite.
49186
"heroicons": Package(
50187
license_path="heroicons/LICENSE",
51188
),

0 commit comments

Comments
 (0)