… written (#3197)

This PR makes Snakemake remove the incomplete marker for a job thats
finished only after the metadata is written. Otherwise the jobs
starttime for the report was missing in the metadata and caused problems
in the plot for the runtimes of individual jobs.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **Bug Fixes**
- Adjusted the order of operations in the job completion process to
ensure that incomplete markers are only removed after successful
metadata recording, enhancing reliability.
  
- **Refactor**
- Improved the structure of the `finished` method for better clarity and
maintainability while retaining existing functionality.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Just a typo fix to please codespell.

This change would just replace "array" with "dictionary" in this bit of
the documentation on configuration:

> If the configfile statement is not used, the config variable provides
an empty array.

(What's an array?)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **Documentation**
- Updated documentation to clarify the behavior of the `config` variable
when no configuration file is specified, highlighting that it provides
an empty dictionary instead of an empty array.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Filipe G. Vieira <[email protected]>

<!--Add a description of your PR here-->

The apptainer command used in snakemake now does not have `--cleanenv`
(or `-e`) option. The envinronment variables in the host (e.g.
PYTHONPATH or CONDA_EXE) will be passed to the container. This may
affect reproducibility and even cause some errors. This PR proposes
adding the `--cleanenv` option to the apptainer commmand to avoid
passing host environment variables.

### QC
<!-- Make sure that you can tick the boxes below. -->

* [ ] The PR contains a test case for the changes or the changes are
already covered by an existing test case.
* [ ] The documentation (`docs/`) is updated to reflect the changes or
this is not necessary (e.g. if the change does neither modify the
language nor the behavior or functionalities of Snakemake).


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **New Features**
- Enhanced Singularity command execution by adding the `--cleanenv`
option, ensuring a clean environment for better handling of environment
variables during execution.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: Anfeng Li <[email protected]>
Co-authored-by: Johannes Köster <[email protected]>

<!--Add a description of your PR here-->
closes #3183 

rust-script's conda recipe was very out of date. As such, the test suite
didn't pick up a breaking CLI change (which isn't even in their
changelog annoyingly). `--features` now needs to be passed after `--` as
it is passed on to `cargo`.

I also moved the rust script tests into their own dedicated test suite.

### QC
<!-- Make sure that you can tick the boxes below. -->

* [x] The PR contains a test case for the changes or the changes are
already covered by an existing test case.
* [x] The documentation (`docs/`) is updated to reflect the changes or
this is not necessary (e.g. if the change does neither modify the
language nor the behavior or functionalities of Snakemake).


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **New Features**
- Enhanced Snakemake rules documentation with updated Rust script
requirements and expanded template rendering details.
- Introduced new features for continuous input and mechanisms for
updating existing output files.
- Added multiple new rules for R and Julia scripts in the Snakefile,
shifting focus from Rust processing.

- **Bug Fixes**
- Updated import statements and variable declarations in Rust scripts
for improved deserialization handling.

- **Documentation**
- Significant updates to the rules documentation for clarity and
usability.

- **Tests**
	- Added a new test function for the `test_script_rs` functionality.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

…n Snakemake report (#3203)

Support versioned URLs in Asset class and fix missing versions in
Snakemake report


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **New Features**
- Introduced versioning for assets, allowing dynamic URL generation
based on specified versions.
	- Added a method to retrieve the version of specified assets.

- **Bug Fixes**
- Enhanced logic for determining package versions with fallback options
for version retrieval.

- **Documentation**
- Updated documentation to reflect new attributes and methods related to
asset versioning.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

<!--Add a description of your PR here-->

### QC
<!-- Make sure that you can tick the boxes below. -->

* [x] The PR contains a test case for the changes or the changes are
already covered by an existing test case.
* [x] The documentation (`docs/`) is updated to reflect the changes or
this is not necessary (e.g. if the change does neither modify the
language nor the behavior or functionalities of Snakemake).


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **New Features**
	- Updated social media links in the documentation.
	- Added Bluesky and Mastodon badges for following Johannes Köster.
	- Removed the Twitter badge.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Fix: This PR adds back the removal of the incomplete marker file also
when keep_metadata is False (i.e. --drop-metadata is active).

The removal calls were merged and put at the start of the function in
#3162, but this caused loss of starttime info in the metadata record. In
#3197 this was fixed, but by moving the removal call to the end of the
function, the incomplete tracker file was not removed anymore when
--drop-metadata was active. This PR adds the call to the removal
function back also for that case, and adds a note to inform why the code
is structured this way.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **Bug Fixes**
- Improved handling of job metadata by ensuring the incomplete marker is
removed only after successful metadata record creation, preserving job
start time.
  
- **Documentation**
- Added clarifying comments to explain changes regarding the incomplete
marker removal.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Johannes Köster <[email protected]>

…us mtime errors when checking for consistency with output files (#3220)

<!--Add a description of your PR here-->

### QC
<!-- Make sure that you can tick the boxes below. -->

* [x] The PR contains a test case for the changes or the changes are
already covered by an existing test case.
* [x] The documentation (`docs/`) is updated to reflect the changes or
this is not necessary (e.g. if the change does neither modify the
language nor the behavior or functionalities of Snakemake).


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Enhanced output verification process with new checks for output file
modification times.
- Improved control over output handling during job postprocessing with
additional parameters.

- **Bug Fixes**
- Added detailed error handling to raise errors for outdated output
files.

- **Refactor**
- Updated method signatures in the `DAG` and `Job` classes to include
new parameters for better functionality.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

As discussed on Discord


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **Documentation**
- Updated rules documentation with new features, clarifications, and
examples to enhance usability.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

…3216)

This is a more correct terminology, IMO.

### QC
<!-- Make sure that you can tick the boxes below. -->

* [ ] The PR contains a test case for the changes or the changes are
already covered by an existing test case.
* [x] The documentation (`docs/`) is updated to reflect the changes or
this is not necessary (e.g. if the change does neither modify the
language nor the behavior or functionalities of Snakemake).


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **Documentation**
- Revised caching documentation to clarify the mechanism for recognizing
previously computed results.
	- Replaced "blockchain" with "Merkle tree" in the context of hashing.
- Specified that caching must be explicitly activated per rule, with
examples for command line usage and Python definitions.
- Reiterated eligibility criteria for caching rules and permissions for
cached data.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Johannes Köster <[email protected]>
Co-authored-by: Johannes Köster <[email protected]>

🤖 I have created a release *beep* *boop*
---


##
[8.25.4](v8.25.3...v8.25.4)
(2024-11-27)


### Bug Fixes

* clean env vars in apptainer
([#3199](#3199))
([76d5329](76d5329))
* ensure that intermediate files in job groups do not cause spurious
mtime errors when checking for consistency with output files
([#3220](#3220))
([4ba2bdf](4ba2bdf))
* Remove incomplete marker also when drop-metadata is active
([#3215](#3215))
([a4f2e5c](a4f2e5c))
* Remove incomplete marker for job finished only after metadata is
written ([#3197](#3197))
([6567e5f](6567e5f))
* Support versioned URLs in Asset class and fix missing versions in
Snakemake report
([#3203](#3203))
([f086f6c](f086f6c))
* update rust-script usage to recent version (v0.35.0)
[#3183](#3183)
([#3208](#3208))
([43885d7](43885d7))


### Documentation

* clarify continuously updated input section
([#3219](#3219))
([72a6994](72a6994))
* Fix typo in CHANGELOG.md
([#3198](#3198))
([0e445ed](0e445ed))
* refer to Merkle trees instead of "blockchain" in caching.rst
([#3216](#3216))
([282e5d9](282e5d9))
* remove twitter in favor of bluesky and mastodon
([#3217](#3217))
([231c6df](231c6df))
* use "dictionary" not "array" wording in config docs
([#3156](#3156))
([17aed41](17aed41))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>