Update environment pinning workflow to use conda-lock#1660
Update environment pinning workflow to use conda-lock#1660lkstrp merged 7 commits intoPyPSA:masterfrom
Conversation
| @@ -1,5 +1,4 @@ | |||
| name: Update pinned envs | |||
|
|
|||
| name: Update pinned envs with conda-lock | |||
There was a problem hiding this comment.
| name: Update pinned envs with conda-lock | |
| name: Update pinned envs |
Can we keep that shorter?
lkstrp
left a comment
There was a problem hiding this comment.
Thanks for the contribution @stefan-oet !
I did not know about conda-lock. But adding build numbers without extra files for arm-based architecture will not work. Any silicon based macs can not resolve those, which is quite common. Linux ARM is probably less of an issue, but still.
We can either drop the build numbers entirely, or add environments for each architecture. I probably prefer the latter and then we can also break things and use new filenames (the defaults).
So could we do:
project/
├── envs/
│ ├── environment.yaml
│ ├── linux-64.lock.yaml
│ ├── osx-64.lock.yaml
│ ├── win-64.lock.yaml
│ ├── osx-arm64.lock.yaml
│ ├── linux-aarch64.lock.yaml
Any future rule specific environments can go into subdirs then.
And before we merge this, and when we change the filenames again, I'd like to see if we can add a uv lock as well. Not all dependencies can be installed via uv, but if the rest is already installed on the system, it would be such a convenience to have uv support.
|
Thanks for your quick response @lkstrp ! I agree that we should definitely include arm-based architecture support even though this will break the current default filenames. Regarding One way to have an implicit |
|
What about just adding a What would be the advantage of pixi over this process? It looks like could be used as well as another alternative on top of conda. |
|
and we don't need glpk anymore as we have highs :) |
|
I moved the |
|
#1663 looks good to me and it works as long as The proposed file structure makes sense as well. Perhaps we can call the directory containing the lockfiles Regarding the pixi discussion, it does not really make sense here since we have (or had?) only 1 system dependency. It's just a modern alternative to conda |
Not sure, Another thing is that the created file no longer has a name. So you always have to pass |
|
I do agree that dropping the subdir makes more sense now.
Fair point! I'll update the PR tomorrow |
|
I had to do some digging this morning as it turned out that adding support for It turns out that conda / conda-lock will always resolve the conda dependencies first before moving to those defined under pip. And this can have implications sometimes. Like in our case we have Note that this also explains why Any objections against moving |
|
No objections that I know of. Just feels a bit wrong to further mix pip in there. We can also ask Highs if there is a reason for not providing linux arm on conda, and move PyPSA back when they do. |
|
@lkstrp I've updated the PR. The HiGHS team has added support for ARM on linux 🙌 |
lkstrp
left a comment
There was a problem hiding this comment.
The HiGHS team has added support for ARM on linux 🙌
Lovely! I'll test the process once, and then it's ready to be merged.
One last thing: could you also add a release note for the changed file name?
I've added the missing lock files and have mentioned the breaking changes in the release notes. |
Changes proposed in this Pull Request
This PR improves our environment pinning workflow by replacing the previous multi-OS matrix approach with conda-lock.
Changes:
Checklist
envs/environment.yaml.config/config.default.yaml.doc/configtables/*.csv.doc/data_sources.rst.doc/release_notes.rstis added.