Skip to content

CI: use pixi to manage environments #21489

Open

Description

@lucascolley

Skrub is a nice example of using pixi to manage environments for CI jobs: https://github.com/skrub-data/skrub/blob/main/.github/workflows/testing.yml. It gives natural separation between the environment setup details and the actual content of the job. For some SciPy jobs right now, these parts are mixed in together and can be tricky to follow.

I propose that we transition to using pixi in CI to manage the majority of environments (and tasks), hopefully improving readability and clarity in the workflow files. Perhaps we keep one job each for plain pip and conda-forge, or something like that.

@rgommers has already done some experimental work with pixi for SciPy at https://github.com/rgommers/pixi-dev-scipystack, which works nicely for me locally.

I think this seems useful and shouldn't be controversial, but perhaps good to ping resident CI wizards @andyfaff @larsoner in case there are any concerns?


As a side-note, this would be a good way to test the waters with pixi, and see whether we would like to start recommending it as a setup method for contributors in the future.

Activity

  1. added
    CIItems related to the CI tools such as CircleCI, GitHub Actions or Azure
    on Aug 31, 2024
  2. nickodell commented on Aug 31, 2024

    @nickodell
  3. lucascolley commented on Aug 31, 2024

    @lucascolley
    Author
  4. rgommers commented on Sep 2, 2024

    @rgommers
    Member

    I think this seems useful and shouldn't be controversial,

    "let's use a new tool" is almost always at least mildly controversial:)

    I propose that we transition to using pixi in CI to manage the majority of environments (and tasks), hopefully improving readability and clarity in the workflow files. Perhaps we keep one job each for plain pip and conda-forge, or something like that.

    There will be a number of jobs where we can't use it because the jobs are special-purpose and not covered by conda-forge. E.g. from linux.yml, the jobs with pre-release versions of dependencies, the 32-bit Linux job, the "distro multiple Pythons" job, the free-threaded job, and the "venv inside source tree" job all don't quite fit.

    There will be significant advantages for some other jobs, and they will indeed get more readable and easier to maintain. Two very nice advantages are:

    1. The whole "some Sphinx or pytest thingy did a release, and all of a sudden our CI job is mysteriously broken" set of annoyances will go away.
    2. CI environments and tasks are very easily reproducible locally.

    I'd prefer to take it quite slow here, since I don't yet have a good feeling for things like:

    • how updates to the lock file should work,
    • whether, as long as we only use it for CI, we want the pixi.toml to live at the top-level of the repo or not
    • how caching of environments is going to scale

    I'll also note that https://github.com/rgommers/pixi-dev-scipystack is still in flux. The array API environments as well as the BLAS switching is good to go, but I at least need to revisit the Windows support (the compiler setup isn't working robustly).

  5. lucascolley commented on Sep 19, 2024

    @lucascolley
    MemberAuthor

    I don't yet have a good feeling for things like:

    • how updates to the lock file should work,
      ...
    • how caching of environments is going to scale

    Any thoughts on these points @glemaitre ?

  6. glemaitre commented on Sep 19, 2024

    @glemaitre
    Contributor

    how updates to the lock file should work,

    This one is an easy one, e.g. https://github.com/skrub-data/skrub/blob/main/.github/workflows/update_pixi_lock_files.yml

    You can have a bot to try to update the lock file once a week.

    how caching of environments is going to scale

    I think this is something to look at. In skrub it seems that we are having this issue:

    image

    (but don't tell to GitHub that apparently you can overpass the limit).

    Maybe there is a need for the workflow updating the lock file to also clean the old cache.

  7. dhirschfeld commented on Oct 11, 2024

    @dhirschfeld

    I just used @rgommers' pixi-dev-scipystack to build scipy. So easy - amazing! 🚀

  8. lucascolley commented on Jun 3, 2025

    @lucascolley
    MemberAuthor

    how updates to the lock file should work

    https://github.com/data-apis/array-api-extra/commits/main/ shows renovate working very well for a smaller project. To be seen whether it would be appropriate at scale.

  9. self-assigned this
    on Jun 16, 2025
  10. andyfaff commented on Sep 11, 2025

    @andyfaff
    Contributor

    I'd like to keep a mix of jobs, or at least to have clear instructions in the dev-docs as to how to use it. I find the pixi docs a little confusing. I'm kind of a rusted on conda-env/venv/pip user.

  11. lucascolley commented on Sep 11, 2025

    @lucascolley
    MemberAuthor

    I'd like to keep a mix of jobs

    Any suggestions for what you would like to see?

    at least to have clear instructions in the dev-docs as to how to use it.

    Of course, this should be done.

    I find the pixi docs a little confusing.

    I have bandwidth for addressing this — please let me know what is confusing!

  12. lucascolley commented on Sep 11, 2025

    @lucascolley
    MemberAuthor

    I find the pixi docs a little confusing.

    I have bandwidth for addressing this — please let me know what is confusing!

    also happy to chat on Slack or elsewhere if it isn't so easy to put into words

  13. andyfaff commented on Sep 11, 2025

    @andyfaff
    Contributor

    I'd like to have at least one job per ci file that's still vanilla.

    I'll add to the list below:

    • what is Cargo? This is referred to on main GH landing page, but it's assumed that people know what it is. What does Cargo-like mean?
    • what is a lock file? It sounds like an environment file, but explanation of basic term would be good.
    • what is a project? Is that like a venv or conda environment? It's not explained on landing page.
    • Is a workspace similar to a project? The basic usage talks about the former, but doesn't actually say what it is. Is a workspace similar to a venv/environment?
    • What is a task? The basic usage shows things but there's not a basic explanation of the terms.
    • How do I tell it to use wheels from pypi as default? I prefer to use wheels throughout my workflow, not conda packages.
    • What is a manifest?
    • It's unclear as to why pixi is going to give me. I already have conda+pip, faster resolution of package dependencies doesn't help me. It's not the rate limiting step for me.
    • Is conda-forge the default conda channel?
    • the pixi command overlaps with the pixi PyPI package.
    • if I install pixi in a conda env, how do I use the python installed into a pixi project rather than the python provided by the conda env? Everything has to be prefixed with pixi run, e.g. pixi run python.

    I'm coming from a situation where I've got to help users coming from a non-programming background. pip/venv/conda are just about simple enough for them, but pixi seems to be a steeper learning curve.

  14. lucascolley commented on Sep 11, 2025

    @lucascolley
    MemberAuthor

    great feedback, thanks Andrew. Please continue to add to the list as and when you find things.

  15. rgommers commented on Sep 11, 2025

    @rgommers
    Member

    I'd like to have at least one job per ci file that's still vanilla.

    Agreed, at least to the extent that we need a healthy mix and ensuring we still test dev versions, PyPI packages for dependencies, etc. (I care less about the per-file aspect).

    It's unclear as to why pixi is going me. I already have conda+pip, faster resolution of package dependencies doesn't help me. It's not the rate limiting step for me.

    The front page of the Pixi docs (https://pixi.sh) summarizes it well, but to make it more specific for SciPy maintenance/development: I think the key advantages are (a) much better reproducibility for environments, and (b) the ability to build up many environments and tasks, so it becomes way easier to do many dev/maint tasks in a single local clone of the SciPy repo. To illustrate (b): say you have a branch where you've been doing regular build/test's (which would be pixi r build, pixi r test, r is shorthand for run), and now you want to test that same branch with free-threading and with CuPy for array types support:

    • with Pixi that's as easy as running pixi r test-cupy, pixi r test-nogil - all the separate environments are automatically created, and the builds use different build directories (with the config in pixi-dev-scipystack).
    • with conda or pip, that's a lot more involved:
      • deactivate current environment
      • activate new environment. questions: remember the env name? or need to create that still? do the two environments contain the same versions of compilers, numpy et al. (pixi can guarantee this, conda/pip cannot)?
      • run spin build then find out that it will choke or error halfway through depending on the task, because it tries to reuse the build dir which may not work in that second env (Pixi will know to use spin build --build-dir=build-nogil).
      • run spin test .... wrong again, you find out too late. You want spin test --build-dir=build-nogil (also less ergonomic).

    In addition pixi is faster, uses less disk space for environments, etc. But the main thing for daily SciPy maintainer needs is declarative envs and tasks that really speed up development. Hope that sells it well enough:)

    I find the pixi docs a little confusing

    I suspect this will go away after some experimentation. I read both the pixi and uv docs extensively, and both are way way better than conda/pip's docs, especially on the conceptual level (both more important and harder to write than the command reference docs).

  16. lucascolley commented on Sep 11, 2025

    @lucascolley
    MemberAuthor

    It's unclear as to why pixi is going me. I already have conda+pip, faster resolution of package dependencies doesn't help me. It's not the rate limiting step for me.

    And of course it isn't just for experienced maintainers. The fact that it makes the same workflow that experienced maintainers use accessible to new contributors in just a few commands is very valuable in itself.

    Anecdote: when I first started my internship working on SciPy, it took multiple days for me to set up a Linux machine for the first time, install compilers/CUDA drivers, and set up mamba for contributing to SciPy. With Pixi I think I could now do the same in less than half an hour!

  17. lucascolley commented on Sep 12, 2025

    @lucascolley
    MemberAuthor

    Everything has to be prefixed with pixi run

    Or, you can pixi shell

  18. lucascolley commented on Sep 15, 2025

    @lucascolley
    MemberAuthor
    • the pixi command overlaps with the pixi PyPI package.

    how can we check this box @andyfaff ? I think you can just rename the binary after downloading it if there is an unavoidable clash.

  19. andyfaff commented on Sep 15, 2025

    @andyfaff
    Contributor

    I don't think it's possible. I got confused because I tried to pip install pixi instead of via conda.

  20. lucascolley commented on Sep 15, 2025

    @lucascolley
    MemberAuthor

    I got confused because I tried to pip install pixi instead of via conda.

    Yeah, we are also annoyed about the name clash, but there isn't much that can be done. Would a warning on https://pixi.sh/latest/installation/ against pip install pixi have helped you? Note that the recommended installation method is the script at the top of that page.

    Is conda-forge the default conda channel?

    Yes. Where do you think would be a good place to mention that?

    How do I tell it to use wheels from pypi as default? I prefer to use wheels throughout my workflow, not conda packages.

    I don't think we have a config option for this, if you feel strongly please open an issue on the Pixi repo. It is a non-trivial request though, because the same package on PyPI and conda-forge (or other channels) may have a different name. It's quite easy to get things from PyPI-only though, you just need to add --pypi to each of your pixi add invocations, or add to [pypi-dependencies] instead of [dependencies] if you are editing the manifest (pixi.toml) manually. So I'll check this box off as covered here.

    what is Cargo? This is referred to on main GH landing page, but it's assumed that people know what it is. What does Cargo-like mean?

    I will add hyperlinks for cargo and yarn to the GH README.

    what is a lock file? It sounds like an environment file, but explanation of basic term would be good.

    where did you first encounter this? Would a hyperlink to https://pixi.sh/latest/workspace/lockfile/ at that location be sufficient?

    What is a task? The basic usage shows things but there's not a basic explanation of the terms.

    where did you first encounter this? Would a hyperlink to https://pixi.sh/latest/workspace/advanced_tasks/ at that location be sufficient?

    What is a manifest?

    where did you first encounter this? Would a hyperlink to https://pixi.sh/latest/reference/pixi_manifest/ at that location be sufficient?

  21. lucascolley commented on Sep 15, 2025

    @lucascolley
    MemberAuthor
    • what is a project? Is that like a venv or conda environment? It's not explained on landing page.
    • Is a workspace similar to a project? The basic usage talks about the former, but doesn't actually say what it is. Is a workspace similar to a venv/environment?

    After prefix-dev/pixi#4589, the landing page now points to https://pixi.sh/dev/first_workspace/, and mentions of project are largely gone. Does that satisfy these two points?

  22. andyfaff commented on Sep 15, 2025

    @andyfaff
    Contributor

    I had a look at the first_workspace page. It doesn't actually describe what a workspace is though. As I understand it it's a location where you do all work for a given topic, e.g. scipy development or a scientific project?

    Yes. Where do you think would be a good place to mention that? (c.f. conda-forge)

    I don't know. The reason it's important is that downstream users (at bigger organisations) got a jump scare about the licensing conditions of using the defaults channel, and don't want to be exposed to any licensing risks. Hopefully defaults should be solely opt-in and not an automatic fallback option. Hopefully it's possible to set a pixi option to never use defaults, even if e.g. a git cloned repository asks to grab packages from there.

    How do I tell it to use wheels from pypi as default? I prefer to use wheels throughout my workflow

    It's fine to tick that checkbox (the only reason I put the checkbox there was to act as a guide to the list). The reason I like to solely work with wheels is because we spend so much time and effort as projects creating them as 1st order release artifacts (with nightlies, etc). A large proportion of the wider community rely on wheels because other less used packages often don't have feedstocks. Conda-forge packages are arguably a 2nd order artifact, in that feedstock development takes place in separate downstream repositories, and changes are fed back to upstream. This isn't meant to denigrate conda packages, just reflect what the current development practice is from my pov.

    what is a lock file?

    That page is probably sufficient.

    What is a manifest?

    I don't think the preamble for that section is described in sufficiently simple terms, e.g. : "A manifest specifies what packages and their respective versions, are installed into a workspace. It also describes where those packages are sourced from."

  23. lucascolley commented on Sep 16, 2025

    @lucascolley
    MemberAuthor

    I see that the conda-forge default is mentioned on the homepage: https://pixi.sh/latest/#available-software.

    Hopefully it's possible to set a pixi option to never use defaults, even if e.g. a git cloned repository asks to grab packages from there.

    I think you can do this by setting a mirror from any channel you want to block: https://pixi.sh/latest/reference/pixi_configuration/#mirrors.

  24. andyfaff commented on Sep 16, 2025

    @andyfaff
    Contributor

    Conda forge by default is good, but one needs to know that there's no fallback to defaults. In conda you do that by editing the condarc file.

  25. rgommers commented on Sep 16, 2025

    @rgommers
    Member

    I think you can do this by setting a mirror from any channel you want to bloc

    That's already one step beyond what Andrew asked I think, as part of blocking it at a site/org-wide level. The simpler answer is that yes, conda-forge only is the default (see https://pixi.sh/latest/reference/pixi_configuration/#default-channels) and the defaults channel will never be used unless the user explicitly adds defaults to a config file.

  26. lucascolley commented on Sep 16, 2025

    @lucascolley
    MemberAuthor
  27. lucascolley commented on Oct 23, 2025

    @lucascolley
    MemberAuthor

    with the merge of gh-23844, we will be able to begin work on gh-23637.

    This issue can stay open to track increasing the coverage of CI jobs further, e.g. the CircleCI jobs. We are deliberately not aiming for 100% coverage, as some variety may help smoke out additional issues, but some more coverage might still be nice.

  28. lucascolley commented on Nov 13, 2025

    @lucascolley
    MemberAuthor

    we should also do another pass of the CI job names, some of them are quite unhelpful or misleading at the minute.

  29. removed their assignment
    on Nov 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    CIItems related to the CI tools such as CircleCI, GitHub Actions or Azure

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions