You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I propose that we transition to using pixi in CI to manage the majority of environments (and tasks), hopefully improving readability and clarity in the workflow files. Perhaps we keep one job each for plain pip and conda-forge, or something like that.
I think this seems useful and shouldn't be controversial, but perhaps good to ping resident CI wizards @andyfaff@larsoner in case there are any concerns?
As a side-note, this would be a good way to test the waters with pixi, and see whether we would like to start recommending it as a setup method for contributors in the future.
I think this seems useful and shouldn't be controversial,
"let's use a new tool" is almost always at least mildly controversial:)
I propose that we transition to using pixi in CI to manage the majority of environments (and tasks), hopefully improving readability and clarity in the workflow files. Perhaps we keep one job each for plain pip and conda-forge, or something like that.
There will be a number of jobs where we can't use it because the jobs are special-purpose and not covered by conda-forge. E.g. from linux.yml, the jobs with pre-release versions of dependencies, the 32-bit Linux job, the "distro multiple Pythons" job, the free-threaded job, and the "venv inside source tree" job all don't quite fit.
There will be significant advantages for some other jobs, and they will indeed get more readable and easier to maintain. Two very nice advantages are:
The whole "some Sphinx or pytest thingy did a release, and all of a sudden our CI job is mysteriously broken" set of annoyances will go away.
CI environments and tasks are very easily reproducible locally.
I'd prefer to take it quite slow here, since I don't yet have a good feeling for things like:
how updates to the lock file should work,
whether, as long as we only use it for CI, we want the pixi.toml to live at the top-level of the repo or not
I'd like to keep a mix of jobs, or at least to have clear instructions in the dev-docs as to how to use it. I find the pixi docs a little confusing. I'm kind of a rusted on conda-env/venv/pip user.
I'd like to have at least one job per ci file that's still vanilla.
I'll add to the list below:
what is Cargo? This is referred to on main GH landing page, but it's assumed that people know what it is. What does Cargo-like mean?
what is a lock file? It sounds like an environment file, but explanation of basic term would be good.
what is a project? Is that like a venv or conda environment? It's not explained on landing page.
Is a workspace similar to a project? The basic usage talks about the former, but doesn't actually say what it is. Is a workspace similar to a venv/environment?
What is a task? The basic usage shows things but there's not a basic explanation of the terms.
How do I tell it to use wheels from pypi as default? I prefer to use wheels throughout my workflow, not conda packages.
What is a manifest?
It's unclear as to why pixi is going to give me. I already have conda+pip, faster resolution of package dependencies doesn't help me. It's not the rate limiting step for me.
Is conda-forge the default conda channel?
the pixi command overlaps with the pixi PyPI package.
if I install pixi in a conda env, how do I use the python installed into a pixi project rather than the python provided by the conda env? Everything has to be prefixed with pixi run, e.g. pixi run python.
I'm coming from a situation where I've got to help users coming from a non-programming background. pip/venv/conda are just about simple enough for them, but pixi seems to be a steeper learning curve.
I'd like to have at least one job per ci file that's still vanilla.
Agreed, at least to the extent that we need a healthy mix and ensuring we still test dev versions, PyPI packages for dependencies, etc. (I care less about the per-file aspect).
It's unclear as to why pixi is going me. I already have conda+pip, faster resolution of package dependencies doesn't help me. It's not the rate limiting step for me.
The front page of the Pixi docs (https://pixi.sh) summarizes it well, but to make it more specific for SciPy maintenance/development: I think the key advantages are (a) much better reproducibility for environments, and (b) the ability to build up many environments and tasks, so it becomes way easier to do many dev/maint tasks in a single local clone of the SciPy repo. To illustrate (b): say you have a branch where you've been doing regular build/test's (which would be pixi r build, pixi r test, r is shorthand for run), and now you want to test that same branch with free-threading and with CuPy for array types support:
with Pixi that's as easy as running pixi r test-cupy, pixi r test-nogil - all the separate environments are automatically created, and the builds use different build directories (with the config in pixi-dev-scipystack).
with conda or pip, that's a lot more involved:
deactivate current environment
activate new environment. questions: remember the env name? or need to create that still? do the two environments contain the same versions of compilers, numpy et al. (pixi can guarantee this, conda/pip cannot)?
run spin build then find out that it will choke or error halfway through depending on the task, because it tries to reuse the build dir which may not work in that second env (Pixi will know to use spin build --build-dir=build-nogil).
run spin test .... wrong again, you find out too late. You want spin test --build-dir=build-nogil (also less ergonomic).
In addition pixi is faster, uses less disk space for environments, etc. But the main thing for daily SciPy maintainer needs is declarative envs and tasks that really speed up development. Hope that sells it well enough:)
I find the pixi docs a little confusing
I suspect this will go away after some experimentation. I read both the pixi and uv docs extensively, and both are way way better than conda/pip's docs, especially on the conceptual level (both more important and harder to write than the command reference docs).
It's unclear as to why pixi is going me. I already have conda+pip, faster resolution of package dependencies doesn't help me. It's not the rate limiting step for me.
And of course it isn't just for experienced maintainers. The fact that it makes the same workflow that experienced maintainers use accessible to new contributors in just a few commands is very valuable in itself.
Anecdote: when I first started my internship working on SciPy, it took multiple days for me to set up a Linux machine for the first time, install compilers/CUDA drivers, and set up mamba for contributing to SciPy. With Pixi I think I could now do the same in less than half an hour!
Yes. Where do you think would be a good place to mention that?
How do I tell it to use wheels from pypi as default? I prefer to use wheels throughout my workflow, not conda packages.
I don't think we have a config option for this, if you feel strongly please open an issue on the Pixi repo. It is a non-trivial request though, because the same package on PyPI and conda-forge (or other channels) may have a different name. It's quite easy to get things from PyPI-only though, you just need to add --pypi to each of your pixi add invocations, or add to [pypi-dependencies] instead of [dependencies] if you are editing the manifest (pixi.toml) manually. So I'll check this box off as covered here.
what is Cargo? This is referred to on main GH landing page, but it's assumed that people know what it is. What does Cargo-like mean?
I will add hyperlinks for cargo and yarn to the GH README.
what is a lock file? It sounds like an environment file, but explanation of basic term would be good.
what is a project? Is that like a venv or conda environment? It's not explained on landing page.
Is a workspace similar to a project? The basic usage talks about the former, but doesn't actually say what it is. Is a workspace similar to a venv/environment?
I had a look at the first_workspace page. It doesn't actually describe what a workspace is though. As I understand it it's a location where you do all work for a given topic, e.g. scipy development or a scientific project?
Yes. Where do you think would be a good place to mention that? (c.f. conda-forge)
I don't know. The reason it's important is that downstream users (at bigger organisations) got a jump scare about the licensing conditions of using the defaults channel, and don't want to be exposed to any licensing risks. Hopefully defaults should be solely opt-in and not an automatic fallback option. Hopefully it's possible to set a pixi option to never use defaults, even if e.g. a git cloned repository asks to grab packages from there.
How do I tell it to use wheels from pypi as default? I prefer to use wheels throughout my workflow
It's fine to tick that checkbox (the only reason I put the checkbox there was to act as a guide to the list). The reason I like to solely work with wheels is because we spend so much time and effort as projects creating them as 1st order release artifacts (with nightlies, etc). A large proportion of the wider community rely on wheels because other less used packages often don't have feedstocks. Conda-forge packages are arguably a 2nd order artifact, in that feedstock development takes place in separate downstream repositories, and changes are fed back to upstream. This isn't meant to denigrate conda packages, just reflect what the current development practice is from my pov.
what is a lock file?
That page is probably sufficient.
What is a manifest?
I don't think the preamble for that section is described in sufficiently simple terms, e.g. : "A manifest specifies what packages and their respective versions, are installed into a workspace. It also describes where those packages are sourced from."
with the merge of gh-23844, we will be able to begin work on gh-23637.
This issue can stay open to track increasing the coverage of CI jobs further, e.g. the CircleCI jobs. We are deliberately not aiming for 100% coverage, as some variety may help smoke out additional issues, but some more coverage might still be nice.
Activity
nickodell commentedon Aug 31, 2024
lucascolley commentedon Aug 31, 2024
rgommers commentedon Sep 2, 2024
"let's use a new tool" is almost always at least mildly controversial:)
There will be a number of jobs where we can't use it because the jobs are special-purpose and not covered by conda-forge. E.g. from
linux.yml, the jobs with pre-release versions of dependencies, the 32-bit Linux job, the "distro multiple Pythons" job, the free-threaded job, and the "venv inside source tree" job all don't quite fit.There will be significant advantages for some other jobs, and they will indeed get more readable and easier to maintain. Two very nice advantages are:
I'd prefer to take it quite slow here, since I don't yet have a good feeling for things like:
pixi.tomlto live at the top-level of the repo or notI'll also note that https://github.com/rgommers/pixi-dev-scipystack is still in flux. The array API environments as well as the BLAS switching is good to go, but I at least need to revisit the Windows support (the compiler setup isn't working robustly).
lucascolley commentedon Sep 19, 2024
Any thoughts on these points @glemaitre ?
glemaitre commentedon Sep 19, 2024
This one is an easy one, e.g. https://github.com/skrub-data/skrub/blob/main/.github/workflows/update_pixi_lock_files.yml
You can have a bot to try to update the lock file once a week.
I think this is something to look at. In
skrubit seems that we are having this issue:(but don't tell to GitHub that apparently you can overpass the limit).
Maybe there is a need for the workflow updating the lock file to also clean the old cache.
dhirschfeld commentedon Oct 11, 2024
I just used @rgommers'
pixi-dev-scipystackto buildscipy. So easy - amazing! 🚀lucascolley commentedon Jun 3, 2025
https://github.com/data-apis/array-api-extra/commits/main/ shows renovate working very well for a smaller project. To be seen whether it would be appropriate at scale.
andyfaff commentedon Sep 11, 2025
I'd like to keep a mix of jobs, or at least to have clear instructions in the dev-docs as to how to use it. I find the pixi docs a little confusing. I'm kind of a rusted on conda-env/venv/pip user.
lucascolley commentedon Sep 11, 2025
Any suggestions for what you would like to see?
Of course, this should be done.
I have bandwidth for addressing this — please let me know what is confusing!
lucascolley commentedon Sep 11, 2025
also happy to chat on Slack or elsewhere if it isn't so easy to put into words
andyfaff commentedon Sep 11, 2025
I'd like to have at least one job per ci file that's still vanilla.
I'll add to the list below:
if I install pixi in a conda env, how do I use the python installed into a pixi project rather than the python provided by the conda env?Everything has to be prefixed withpixi run, e.g.pixi run python.I'm coming from a situation where I've got to help users coming from a non-programming background. pip/venv/conda are just about simple enough for them, but pixi seems to be a steeper learning curve.
lucascolley commentedon Sep 11, 2025
great feedback, thanks Andrew. Please continue to add to the list as and when you find things.
rgommers commentedon Sep 11, 2025
Agreed, at least to the extent that we need a healthy mix and ensuring we still test dev versions, PyPI packages for dependencies, etc. (I care less about the per-file aspect).
The front page of the Pixi docs (https://pixi.sh) summarizes it well, but to make it more specific for SciPy maintenance/development: I think the key advantages are (a) much better reproducibility for environments, and (b) the ability to build up many environments and tasks, so it becomes way easier to do many dev/maint tasks in a single local clone of the SciPy repo. To illustrate (b): say you have a branch where you've been doing regular build/test's (which would be
pixi r build,pixi r test,ris shorthand forrun), and now you want to test that same branch with free-threading and with CuPy for array types support:pixi r test-cupy,pixi r test-nogil- all the separate environments are automatically created, and the builds use different build directories (with the config in pixi-dev-scipystack).condaorpip, that's a lot more involved:spin buildthen find out that it will choke or error halfway through depending on the task, because it tries to reuse thebuilddir which may not work in that second env (Pixi will know to usespin build --build-dir=build-nogil).spin test.... wrong again, you find out too late. You wantspin test --build-dir=build-nogil(also less ergonomic).In addition
pixiis faster, uses less disk space for environments, etc. But the main thing for daily SciPy maintainer needs is declarative envs and tasks that really speed up development. Hope that sells it well enough:)I suspect this will go away after some experimentation. I read both the
pixianduvdocs extensively, and both are way way better thanconda/pip's docs, especially on the conceptual level (both more important and harder to write than the command reference docs).lucascolley commentedon Sep 11, 2025
And of course it isn't just for experienced maintainers. The fact that it makes the same workflow that experienced maintainers use accessible to new contributors in just a few commands is very valuable in itself.
Anecdote: when I first started my internship working on SciPy, it took multiple days for me to set up a Linux machine for the first time, install compilers/CUDA drivers, and set up
mambafor contributing to SciPy. With Pixi I think I could now do the same in less than half an hour!lucascolley commentedon Sep 12, 2025
Or, you can
pixi shelllucascolley commentedon Sep 15, 2025
how can we check this box @andyfaff ? I think you can just rename the binary after downloading it if there is an unavoidable clash.
andyfaff commentedon Sep 15, 2025
I don't think it's possible. I got confused because I tried to pip install pixi instead of via conda.
lucascolley commentedon Sep 15, 2025
Yeah, we are also annoyed about the name clash, but there isn't much that can be done. Would a warning on https://pixi.sh/latest/installation/ against
pip install pixihave helped you? Note that the recommended installation method is the script at the top of that page.Yes. Where do you think would be a good place to mention that?
I don't think we have a config option for this, if you feel strongly please open an issue on the Pixi repo. It is a non-trivial request though, because the same package on PyPI and conda-forge (or other channels) may have a different name. It's quite easy to get things from PyPI-only though, you just need to add
--pypito each of yourpixi addinvocations, or add to[pypi-dependencies]instead of[dependencies]if you are editing the manifest (pixi.toml) manually. So I'll check this box off as covered here.I will add hyperlinks for
cargoandyarnto the GH README.where did you first encounter this? Would a hyperlink to https://pixi.sh/latest/workspace/lockfile/ at that location be sufficient?
where did you first encounter this? Would a hyperlink to https://pixi.sh/latest/workspace/advanced_tasks/ at that location be sufficient?
where did you first encounter this? Would a hyperlink to https://pixi.sh/latest/reference/pixi_manifest/ at that location be sufficient?
lucascolley commentedon Sep 15, 2025
After prefix-dev/pixi#4589, the landing page now points to https://pixi.sh/dev/first_workspace/, and mentions of
projectare largely gone. Does that satisfy these two points?andyfaff commentedon Sep 15, 2025
I had a look at the first_workspace page. It doesn't actually describe what a workspace is though. As I understand it it's a location where you do all work for a given topic, e.g. scipy development or a scientific project?
I don't know. The reason it's important is that downstream users (at bigger organisations) got a jump scare about the licensing conditions of using the
defaultschannel, and don't want to be exposed to any licensing risks. Hopefully defaults should be solely opt-in and not an automatic fallback option. Hopefully it's possible to set a pixi option to never usedefaults, even if e.g. a git cloned repository asks to grab packages from there.It's fine to tick that checkbox (the only reason I put the checkbox there was to act as a guide to the list). The reason I like to solely work with wheels is because we spend so much time and effort as projects creating them as 1st order release artifacts (with nightlies, etc). A large proportion of the wider community rely on wheels because other less used packages often don't have feedstocks. Conda-forge packages are arguably a 2nd order artifact, in that feedstock development takes place in separate downstream repositories, and changes are fed back to upstream. This isn't meant to denigrate conda packages, just reflect what the current development practice is from my pov.
That page is probably sufficient.
I don't think the preamble for that section is described in sufficiently simple terms, e.g. : "A manifest specifies what packages and their respective versions, are installed into a workspace. It also describes where those packages are sourced from."
lucascolley commentedon Sep 16, 2025
I see that the conda-forge default is mentioned on the homepage: https://pixi.sh/latest/#available-software.
I think you can do this by setting a mirror from any channel you want to block: https://pixi.sh/latest/reference/pixi_configuration/#mirrors.
andyfaff commentedon Sep 16, 2025
Conda forge by default is good, but one needs to know that there's no fallback to
defaults. In conda you do that by editing the condarc file.rgommers commentedon Sep 16, 2025
That's already one step beyond what Andrew asked I think, as part of blocking it at a site/org-wide level. The simpler answer is that yes,
conda-forgeonly is the default (see https://pixi.sh/latest/reference/pixi_configuration/#default-channels) and thedefaultschannel will never be used unless the user explicitly addsdefaultsto a config file.lucascolley commentedon Sep 16, 2025
@andyfaff I think we can check off the remaining boxes once
are merged. Unless you had any other suggestions for 'tasks'?
lucascolley commentedon Oct 23, 2025
with the merge of gh-23844, we will be able to begin work on gh-23637.
This issue can stay open to track increasing the coverage of CI jobs further, e.g. the CircleCI jobs. We are deliberately not aiming for 100% coverage, as some variety may help smoke out additional issues, but some more coverage might still be nice.
lucascolley commentedon Nov 13, 2025
we should also do another pass of the CI job names, some of them are quite unhelpful or misleading at the minute.