Repository navigation
FEAT/BUG: caching interface is limited to the point of unusability #324
Description
xref: astropy/astropy#18835
I'm experimenting with caching abilities for the tox.yml workflow, but I think I'm hitting a wall where I find that:
- setting
cache-pathisn't sufficient in itself to enable caching (cache-keyalso needs to be set) - it's impossible to set different cache keys across a (generated) test matrix, because
cache-keycan only be set before jobs are created
As a result, caching is basically unusable: different jobs end up competing for a single cache-key, even though, in all generality, they need to cache different artifacts.
Am I missing anything ?
I suspect it could be possible to solve this with another input (say cache-key-expr) which would represent (and replace) a cache key expression, but whose expression would only be evaluated in the context of the generated job. Thoughts ?
Activity
Cadair commentedon Nov 4, 2025
SunPy has only ever used the cache to split our doc build into two parts, which only needs one cache per invocation of the workflow so it's always worked fine for us.
Perhaps the simplest solution would be an option to append the job name to the cache key?
neutrinoceros commentedon Nov 4, 2025
Is the job name already guaranteed to be unique ? Would it break the existing application in sunpy if this behavior was always enforced, rather than through a new option ?
Cadair commentedon Nov 5, 2025
ha, going to answer this question I went and looked at the sunpy code, and I realised you can already do what you want (all be it not that nicely). You can specify cache-key per env:
https://github.com/sunpy/sunpy/blob/179029a504826c36b9e836c86aca0b71f7b2e9f5/.github/workflows/ci.yml#L90-L96
We would need to change our workflow, so I'd rather not make the braking change, but we could.
neutrinoceros commentedon Nov 5, 2025
ah, this is a game changer. Thanks a lot ! nothing to fix after all !
Cadair commentedon Nov 5, 2025
I still think an option to make it less verbose would be good. Something like
is a lot neater than having to specify the key for each job?
neutrinoceros commentedon Nov 5, 2025
Yeah but then you also loose the ability to share keys between selected jobs, and you make the existing interface somewhat harder to discover. I'm generally very skeptical about the extensibility (or lack thereof) of boolean flags. I'd rather we avoid taking a suboptimal path now if it makes better solutions harder to implement in the future.
Cadair commentedon Nov 5, 2025
yeah fair enough. I'll let you experiment on astropy and we can see where it goes.