Skip to content

feat: Add snakemake-storage-plugin-cached-http to address zenodo failures#1913

Merged
lkstrp merged 8 commits intomasterfrom
feat/add-snakemake-storage-plugin-cached-http
Nov 20, 2025
Merged

feat: Add snakemake-storage-plugin-cached-http to address zenodo failures#1913
lkstrp merged 8 commits intomasterfrom
feat/add-snakemake-storage-plugin-cached-http

Conversation

@coroa
Copy link
Copy Markdown
Member

@coroa coroa commented Nov 19, 2025 •
edited
Loading

Changes proposed in this Pull Request

Add snakemake-storage-plugin-cached-http as planned in #1675 to fix recurring Zenodo retrieval failures.

Fix a re-introduced stream decompression error in progress_retrieve (follow-up to #1840).

Checklist

  • I tested my contribution locally and it works as intended.
  • Code and workflow changes are sufficiently documented.
  • Changed dependencies are added to pixi.toml (using pixi add <dependency-name>).
  • A release note doc/release_notes.rst is added.

@coroa coroa requested a review from lkstrp November 19, 2025 15:43
@coroa coroa force-pushed the feat/add-snakemake-storage-plugin-cached-http branch from 9854eac to e8c362b Compare November 19, 2025 15:59
@coroa
Copy link
Copy Markdown
Member Author

coroa commented Nov 19, 2025 •
edited
Loading

Can't reproduce the CI failure locally. Is it possible that we have a broken costs.csv in the data cache? How can i clear that cache?

@coroa
Copy link
Copy Markdown
Member Author

coroa commented Nov 19, 2025 •
edited
Loading

At https://github.com/PyPSA/pypsa-eur/actions/runs/19509226553/job/55844401905#step:7:169:

INFO:__main__:Head of the file:
������n#K�.���" OuK�d2�j�9GE���]�@RU���4w���J29�I�T�����<����0������y��z���O�"��I&oR����RQɌo�XkŊ�u�i�?����6rCw@c���]�Lk��ע`�vi�f��}������7��`X�Ð����Gꆿ9��\��RB}ڍ�������nxKI�}����

ie. we are receiving transparently compressed data and storing the compressed stream instead.

i guess we re-introduced this in #1840 (and i was advocating for it 🤦 )

@coroa
Copy link
Copy Markdown
Member Author

coroa commented Nov 20, 2025

Ready to merge from my side.

@coroa coroa requested a review from fneum November 20, 2025 09:01
lkstrp
lkstrp approved these changes Nov 20, 2025
View reviewed changes
Copy link
Copy Markdown
Member

@lkstrp lkstrp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will be a blessing

@lkstrp lkstrp enabled auto-merge (squash) November 20, 2025 09:17
fneum
fneum approved these changes Nov 20, 2025
View reviewed changes
Copy link
Copy Markdown
Member

@fneum fneum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. If I understand the snakemake docs correctly (https://snakemake.readthedocs.io/en/stable/snakefiles/storage.html), the cached-http plugin is not yet used in this PR as it would need:

storage:
    provider="cached-http",

if it is to be used by default or calls to storage.cached_http(). Is that right?

@coroa
Copy link
Copy Markdown
Member Author

coroa commented Nov 20, 2025 •
edited
Loading

LGTM. If I understand the snakemake docs correctly (https://snakemake.readthedocs.io/en/stable/snakefiles/storage.html), the cached-http plugin is not yet used in this PR as it would need:

storage:
    provider="cached-http",

if it is to be used by default or calls to storage.cached_http(). Is that right?

No, there are two facilities how stuff is being used.

  1. By using the "tag"s how they call them explicitly, ie. storage.cached_http.
  2. If you use only storage(...) then all registered storage plugins are asked, by calling StorageProvider.is_valid_query with the url. If multiple plugins then report, yes please, snakemake raises an ambiguity error. Therefore, the plugin monkey-patches the response of the default http plugin to return "no, thank you" for zenodo urls. See also section in the README: https://github.com/PyPSA/snakemake-storage-plugin-cached-http#plugin-priority

ie. the plugin is active for zenodo urls.

@fneum
Copy link
Copy Markdown
Member

fneum commented Nov 20, 2025

Gotcha. Thanks for explaining!

@lkstrp lkstrp merged commit cbe69b6 into master Nov 20, 2025
6 checks passed
@lkstrp lkstrp deleted the feat/add-snakemake-storage-plugin-cached-http branch November 20, 2025 10:06
@tgilon tgilon mentioned this pull request Nov 20, 2025
Merged
12 tasks
toniseibold pushed a commit that referenced this pull request Dec 1, 2025
@coroa @lkstrp @toniseibold
feat: Add snakemake-storage-plugin-cached-http to address zenodo fail…
fdf521b 
…ures (#1913)

* feat: Add snakemake-storage-plugin-cached-http to alleviate zenodo failures

* Add release note

* fix: don't use progress bar for cost retrieval

* Desperate debugging

* More desperation

* fix: fix progress_retrieve

use iter_content to avoid storing transparently compressed data.

* chore: clean up desperation

* Apply suggestion from @lkstrp

---------

Co-authored-by: Lukas Trippe <[email protected]>
@lkstrp lkstrp mentioned this pull request Jan 2, 2026
Merged
8 tasks
@willu47 willu47 mentioned this pull request Mar 3, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lkstrp lkstrp lkstrp approved these changes

@fneum fneum fneum approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@coroa @fneum @lkstrp