feat: Support Hy in script directive#3824
feat: Support Hy in script directive#3824johanneskoester merged 17 commits intosnakemake:mainfrom jsquaredosquared:support-hy-script
Conversation
📝 WalkthroughWalkthroughAdds Hy (.hy) script support: new HyScript subclass, language detection and dispatch registration for "hy", documentation updates adding Xonsh and Hy examples, and a test bundle (Snakefile, Hy script, input) exercising a Hy script that sums odd numbers. Changes
Sequence Diagram(s)sequenceDiagram
participant SM as Snakemake
participant HS as HyScript
participant HY as hy runtime
participant FS as Filesystem
rect `#E8F5E9`
Note over SM,HS: detect `.hy` -> select HyScript
end
SM->>HS: script(...): instantiate HyScript
HS->>HS: write_script(preamble, fd)\n(emit Hy preamble + source)
HS->>HY: execute_script(fname) (spawn hy runtime)
HY->>FS: read inputs / write outputs
HY-->>HS: exit status / stdout
HS-->>SM: return completion / propagate errors
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes
Possibly related PRs
Pre-merge checks and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✨ Finishing touches🧪 Generate unit tests (beta)
📜 Recent review detailsConfiguration used: CodeRabbit UI Review profile: CHILL Plan: Pro 📒 Files selected for processing (1)
🧰 Additional context used🧠 Learnings (4)📚 Learning: 2024-10-06T14:09:54.370ZApplied to files:
📚 Learning: 2024-12-14T13:10:48.450ZApplied to files:
📚 Learning: 2024-11-07T00:32:44.137ZApplied to files:
📚 Learning: 2024-10-29T17:14:13.585ZApplied to files:
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (10)
🔇 Additional comments (1)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
tests/test_script_hy/Snakefile (1)
6-14: Misleading rule name.The rule is named
test_xonshbut it's actually testing Hy script support, not Xonsh. For clarity and maintainability, consider renaming it totest_hy.Apply this diff:
-rule test_xonsh: +rule test_hy: input: "test.in" output:Note: This also aligns better with the directory name
test_script_hyand the script being tested.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
tests/test_script_hy/expected-results/test.outis excluded by!**/*.out
📒 Files selected for processing (6)
docs/snakefiles/rules.rst(1 hunks)src/snakemake/script/__init__.py(3 hunks)tests/test_script_hy/Snakefile(1 hunks)tests/test_script_hy/envs/xonsh.yaml(1 hunks)tests/test_script_hy/scripts/test.hy(1 hunks)tests/test_script_hy/test.in(1 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
**/*.py
⚙️ CodeRabbit configuration file
**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of theselfargument of methods.
Do not suggest type annotation of theclsargument of classmethods.
Do not suggest return type annotation if a function or method does not contain areturnstatement.
Files:
src/snakemake/script/__init__.py
🧠 Learnings (5)
📚 Learning: 2024-10-06T14:09:54.370Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 3117
File: tests/test_wrapper/Snakefile:11-11
Timestamp: 2024-10-06T14:09:54.370Z
Learning: Changes made within test cases, such as in `tests/test_wrapper/Snakefile`, are for testing purposes and do not require updates to the project documentation.
Applied to files:
docs/snakefiles/rules.rsttests/test_script_hy/Snakefile
📚 Learning: 2024-12-14T13:10:48.450Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 2927
File: tests/test_subpath/Snakefile:11-11
Timestamp: 2024-12-14T13:10:48.450Z
Learning: In Snakemake, when a rule defines a single output file (even if the output is specified as a dictionary), `{output}` in the shell command can be used directly to refer to that file.
Applied to files:
docs/snakefiles/rules.rst
📚 Learning: 2024-10-29T17:14:13.585Z
Learnt from: zmbc
Repo: snakemake/snakemake PR: 2857
File: snakemake/notebook.py:84-90
Timestamp: 2024-10-29T17:14:13.585Z
Learning: In `snakemake/notebook.py`, when suggesting enhancements to the `execute_script` method in the `JupyterNotebook` class, ensure that changes align with both papermill and nbconvert use cases.
Applied to files:
docs/snakefiles/rules.rst
📚 Learning: 2024-12-21T15:10:31.992Z
Learnt from: johanneskoester
Repo: snakemake/snakemake PR: 2925
File: tests/test_nonstr_params/Snakefile:14-15
Timestamp: 2024-12-21T15:10:31.992Z
Learning: The file "test.jsonl" in tests/test_nonstr_params is automatically created by Snakemake, rather than manually generated in the Snakefile.
Applied to files:
tests/test_script_hy/Snakefile
📚 Learning: 2025-01-17T12:00:09.368Z
Learnt from: leoschwarz
Repo: snakemake/snakemake PR: 3176
File: tests/test_output_index.py:99-157
Timestamp: 2025-01-17T12:00:09.368Z
Learning: New test dependencies for Snakemake should be introduced in separate PRs rather than being added as part of feature or refactoring PRs.
Applied to files:
tests/test_script_hy/Snakefile
🧬 Code graph analysis (1)
src/snakemake/script/__init__.py (1)
src/snakemake/common/__init__.py (1)
write(421-422)
🪛 Ruff (0.14.2)
src/snakemake/script/__init__.py
1618-1618: Unused method argument: edit
(ARG002)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (46)
- GitHub Check: apidocs
- GitHub Check: tests (10, ubuntu-latest, py313)
- GitHub Check: tests (10, macos-latest, py313)
- GitHub Check: tests (10, windows-2022, py313)
- GitHub Check: tests (8, windows-2022, py313)
- GitHub Check: tests (8, ubuntu-latest, py312)
- GitHub Check: tests (10, ubuntu-latest, py311)
- GitHub Check: tests (9, ubuntu-latest, py312)
- GitHub Check: tests (10, ubuntu-latest, py312)
- GitHub Check: tests (9, ubuntu-latest, py313)
- GitHub Check: tests (9, windows-2022, py313)
- GitHub Check: tests (8, ubuntu-latest, py311)
- GitHub Check: tests (8, ubuntu-latest, py313)
- GitHub Check: tests (7, windows-2022, py313)
- GitHub Check: tests (9, ubuntu-latest, py311)
- GitHub Check: tests (6, ubuntu-latest, py313)
- GitHub Check: tests (7, ubuntu-latest, py313)
- GitHub Check: tests (6, ubuntu-latest, py312)
- GitHub Check: tests (7, ubuntu-latest, py311)
- GitHub Check: tests (7, ubuntu-latest, py312)
- GitHub Check: tests (6, windows-2022, py313)
- GitHub Check: tests (5, ubuntu-latest, py312)
- GitHub Check: tests (6, ubuntu-latest, py311)
- GitHub Check: tests (5, macos-latest, py313)
- GitHub Check: tests (5, ubuntu-latest, py311)
- GitHub Check: tests (5, windows-2022, py313)
- GitHub Check: tests (5, ubuntu-latest, py313)
- GitHub Check: tests (4, ubuntu-latest, py311)
- GitHub Check: tests (4, windows-2022, py313)
- GitHub Check: tests (4, macos-latest, py313)
- GitHub Check: tests (3, windows-2022, py313)
- GitHub Check: tests (4, ubuntu-latest, py312)
- GitHub Check: tests (2, macos-latest, py313)
- GitHub Check: tests (3, ubuntu-latest, py311)
- GitHub Check: tests (3, macos-latest, py313)
- GitHub Check: tests (2, ubuntu-latest, py313)
- GitHub Check: tests (2, ubuntu-latest, py312)
- GitHub Check: tests (3, ubuntu-latest, py313)
- GitHub Check: tests (2, windows-2022, py313)
- GitHub Check: tests (4, ubuntu-latest, py313)
- GitHub Check: tests (3, ubuntu-latest, py312)
- GitHub Check: tests (1, windows-2022, py313)
- GitHub Check: tests (2, ubuntu-latest, py311)
- GitHub Check: tests (1, ubuntu-latest, py312)
- GitHub Check: tests (1, ubuntu-latest, py311)
- GitHub Check: tests (1, ubuntu-latest, py313)
🔇 Additional comments (8)
docs/snakefiles/rules.rst (2)
1763-1763: LGTM! Documentation simplification.The change to use "..." as a placeholder for input/output makes the example more concise while maintaining clarity.
1770-1783: LGTM! Well-structured documentation addition.The Hy language section follows the established documentation pattern and provides clear guidance for users. The description accurately conveys Hy's purpose as a Lisp-like syntax for Python.
src/snakemake/script/__init__.py (4)
1613-1620: Implementation looks correct.The
HyScriptclass appropriately inherits fromPythonScriptsince Hy is a Lisp dialect that runs on Python. The preamble is wrapped in(pys #[[{preamble}]])which is Hy's syntax for embedding Python code, allowing the Python-based snakemake object to be accessible in Hy scripts.Note: The unused
editparameter flagged by static analysis is consistent with other script classes (e.g.,XonshScript) and matches the parent class signature.
1686-1687: LGTM! Language detection correctly implemented.The
.hyfile extension detection follows the established pattern and is correctly placed in the language detection logic.
1752-1752: LGTM! Script dispatch correctly configured.The HyScript class is properly registered in the dispatch table, enabling Snakemake to route
.hyscripts to the correct executor.
1756-1756: LGTM! Error message updated correctly.The error message now includes
.hyin the list of supported script file types, ensuring users receive accurate guidance.tests/test_script_hy/test.in (1)
1-20: LGTM! Appropriate test input.The test input file contains integers 1-20, which provides good coverage for testing the Hy script's functionality (reading input, converting to integers, filtering odd numbers, and computing the sum).
tests/test_script_hy/scripts/test.hy (1)
1-17: LGTM! Well-crafted Hy test script.The script effectively demonstrates Hy's capabilities:
- Uses threading macros from hyrule for clean data pipeline syntax
- Correctly accesses Snakemake's input/output via the injected
snakemakeobject- Implements proper file handling with context managers
- Shows idiomatic Hy code with functional programming patterns
The logic correctly:
- Reads integers from the input file
- Filters odd numbers
- Computes their sum (which should be 100 for inputs 1-20)
- Writes the result to the output file
johanneskoester
left a comment
There was a problem hiding this comment.
Thanks a lot, very nice! One request below.
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
src/snakemake/script/__init__.py (1)
1620-1627: LGTM! Clean implementation following established patterns.The
HyScriptclass correctly:
- Inherits from
PythonScriptto reuse preamble generation- Wraps the Python preamble in Hy's
pysmacro with bracket string syntax#[[...]]for inline Python embedding- Executes scripts via the
hycommand, consistent with other script typesThe unused
editparameter (flagged by static analysis) is part of the abstract interface and consistent with other script classes.Note: The bracket string format assumes the preamble doesn't contain
]]which could theoretically break the delimiter. Based on the preamble content this is not a practical concern, but if edge cases arise, consider using a different delimiter or escaping mechanism.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
src/snakemake/script/__init__.py(4 hunks)tests/test_script_hy/scripts/test.hy(1 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
- tests/test_script_hy/scripts/test.hy
🧰 Additional context used
📓 Path-based instructions (1)
**/*.py
⚙️ CodeRabbit configuration file
**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of theselfargument of methods.
Do not suggest type annotation of theclsargument of classmethods.
Do not suggest return type annotation if a function or method does not contain areturnstatement.
Files:
src/snakemake/script/__init__.py
🧬 Code graph analysis (1)
src/snakemake/script/__init__.py (1)
src/snakemake/common/__init__.py (2)
get_report_id(58-62)get_snakemake_searchpaths(51-55)
🪛 Ruff (0.14.3)
src/snakemake/script/__init__.py
1625-1625: Unused method argument: edit
(ARG002)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (46)
- GitHub Check: tests (9, ubuntu-latest, py311)
- GitHub Check: tests (10, windows-2022, py313)
- GitHub Check: tests (9, ubuntu-latest, py313)
- GitHub Check: tests (8, ubuntu-latest, py311)
- GitHub Check: tests (10, ubuntu-latest, py312)
- GitHub Check: tests (10, ubuntu-latest, py313)
- GitHub Check: tests (9, ubuntu-latest, py312)
- GitHub Check: tests (10, ubuntu-latest, py311)
- GitHub Check: tests (5, ubuntu-latest, py311)
- GitHub Check: tests (7, ubuntu-latest, py313)
- GitHub Check: tests (7, ubuntu-latest, py312)
- GitHub Check: tests (9, windows-2022, py313)
- GitHub Check: tests (7, windows-2022, py313)
- GitHub Check: tests (8, ubuntu-latest, py313)
- GitHub Check: tests (4, ubuntu-latest, py313)
- GitHub Check: tests (7, ubuntu-latest, py311)
- GitHub Check: tests (5, macos-latest, py313)
- GitHub Check: tests (8, windows-2022, py313)
- GitHub Check: tests (8, ubuntu-latest, py312)
- GitHub Check: tests (6, ubuntu-latest, py313)
- GitHub Check: tests (5, ubuntu-latest, py313)
- GitHub Check: tests (6, windows-2022, py313)
- GitHub Check: tests (4, macos-latest, py313)
- GitHub Check: tests (6, ubuntu-latest, py312)
- GitHub Check: tests (6, ubuntu-latest, py311)
- GitHub Check: tests (5, ubuntu-latest, py312)
- GitHub Check: tests (4, ubuntu-latest, py312)
- GitHub Check: tests (5, windows-2022, py313)
- GitHub Check: tests (4, ubuntu-latest, py311)
- GitHub Check: tests (4, windows-2022, py313)
- GitHub Check: tests (1, ubuntu-latest, py313)
- GitHub Check: tests (3, ubuntu-latest, py312)
- GitHub Check: tests (3, ubuntu-latest, py313)
- GitHub Check: tests (3, ubuntu-latest, py311)
- GitHub Check: tests (1, macos-latest, py313)
- GitHub Check: tests (3, windows-2022, py313)
- GitHub Check: tests (2, windows-2022, py313)
- GitHub Check: tests (2, macos-latest, py313)
- GitHub Check: tests (3, macos-latest, py313)
- GitHub Check: tests (2, ubuntu-latest, py311)
- GitHub Check: tests (1, windows-2022, py313)
- GitHub Check: tests (2, ubuntu-latest, py312)
- GitHub Check: tests (1, ubuntu-latest, py312)
- GitHub Check: tests (2, ubuntu-latest, py313)
- GitHub Check: tests (1, ubuntu-latest, py311)
- GitHub Check: apidocs
🔇 Additional comments (3)
src/snakemake/script/__init__.py (3)
1693-1694: LGTM!The language detection for
.hyfiles follows the same pattern as other supported languages.
1759-1759: LGTM!Correctly registers
HyScriptin the dispatch table for Hy language support.
1763-1763: LGTM!Error message correctly updated to include
.hyin the list of supported script file types.
|
Thanks a lot! |
🤖 I have created a release *beep* *boop* --- ## [9.14.0](v9.13.7...v9.14.0) (2025-11-27) ### Features * Support Hy in script directive ([#3824](#3824)) ([2329c9e](2329c9e)) ### Bug Fixes * Add support for pathlib in `notebook` field ([#3811](#3811)) ([7b2180a](7b2180a)) * Addressed race condition in workdir_handler.py ([#3844](#3844)) ([8dbfcfb](8dbfcfb)) * cleanup update-marked output files of failed jobs if there was no backup to restore them ([#3843](#3843)) ([41f1ce8](41f1ce8)) * correct Windows callable path handling ([#3832](#3832)) ([5caad70](5caad70)) * expand env vars on resources ([#3823](#3823)) ([fcfa1bc](fcfa1bc)) * fix backup for output marked by `update` ([#3839](#3839)) ([09c64b7](09c64b7)) * Minor fixes/additions to logging module. ([#3802](#3802)) ([3b3986d](3b3986d)) * mount local storage prefix into containers ([#3840](#3840)) ([f1e8b62](f1e8b62)) * properly format input/output files in case of missing rule to produce them ([#3849](#3849)) ([69d5d24](69d5d24)) * Unpack AnnotatedString in _apply_wildcards ([#3798](#3798)) ([7886508](7886508)) ### Performance Improvements * retrieve storage inputs immediately before scheduling jobs instead of before running the entire workflow ([#3850](#3850)) ([4ac6cda](4ac6cda)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
<!--Add a description of your PR here--> Add support for [Hy](https://hylang.org/) in the script directive. This will allow the use of a Lisp-like syntax for writing scripts, e.g.: ```hy (require hyrule [-> ->>]) (defn is-odd? [n] (!= (% n 2) 0)) (setv result (->> (get snakemake.input 0) open .readlines (map int) (filter is-odd?) sum)) (print result :file (-> (get snakemake.output 0) (open "w"))) ``` ### QC <!-- Make sure that you can tick the boxes below. --> * [X] The PR contains a test case for the changes or the changes are already covered by an existing test case. * [X] The documentation (`docs/`) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake). <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added runtime support for Hy scripts. * **Documentation** * Added Hy and Xonsh language sections with examples and cross-references. * Fixed Bash example formatting and removed a duplicated Xonsh example. * **Tests** * Added end-to-end test exercising Hy script execution with a Snakefile, Hy script, and sample input/output. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Johannes Köster <[email protected]> Co-authored-by: Johannes Köster <[email protected]>
🤖 I have created a release *beep* *boop* --- ## [9.14.0](snakemake/snakemake@v9.13.7...v9.14.0) (2025-11-27) ### Features * Support Hy in script directive ([snakemake#3824](snakemake#3824)) ([2329c9e](snakemake@2329c9e)) ### Bug Fixes * Add support for pathlib in `notebook` field ([snakemake#3811](snakemake#3811)) ([7b2180a](snakemake@7b2180a)) * Addressed race condition in workdir_handler.py ([snakemake#3844](snakemake#3844)) ([8dbfcfb](snakemake@8dbfcfb)) * cleanup update-marked output files of failed jobs if there was no backup to restore them ([snakemake#3843](snakemake#3843)) ([41f1ce8](snakemake@41f1ce8)) * correct Windows callable path handling ([snakemake#3832](snakemake#3832)) ([5caad70](snakemake@5caad70)) * expand env vars on resources ([snakemake#3823](snakemake#3823)) ([fcfa1bc](snakemake@fcfa1bc)) * fix backup for output marked by `update` ([snakemake#3839](snakemake#3839)) ([09c64b7](snakemake@09c64b7)) * Minor fixes/additions to logging module. ([snakemake#3802](snakemake#3802)) ([3b3986d](snakemake@3b3986d)) * mount local storage prefix into containers ([snakemake#3840](snakemake#3840)) ([f1e8b62](snakemake@f1e8b62)) * properly format input/output files in case of missing rule to produce them ([snakemake#3849](snakemake#3849)) ([69d5d24](snakemake@69d5d24)) * Unpack AnnotatedString in _apply_wildcards ([snakemake#3798](snakemake#3798)) ([7886508](snakemake@7886508)) ### Performance Improvements * retrieve storage inputs immediately before scheduling jobs instead of before running the entire workflow ([snakemake#3850](snakemake#3850)) ([4ac6cda](snakemake@4ac6cda)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
Add support for Hy in the script directive. This will allow the use of a Lisp-like syntax for writing scripts, e.g.:
QC
docs/) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).Summary by CodeRabbit
New Features
Documentation
Tests