Repository navigation
TST: decoupling stable CI and incompatibilities discovery (let's use lock files) #17596
Description
Disclaimer
As the length of this issue may suggest, I've put a lot of thoughts into this proposal, which comes as a culmination of one year serving as our first line of defense against environment incompatibilities (also known as dependency hell). Maybe this should be an APE ?
TL;DR
The one-liner version of this proposal is: let's start using lock files.
Now for a slightly longer summary:
We already have a distinction between regular, required CI and jobs that are allowed to fail (allowed-failure). I propose to consolidate this distinction into a stable/unstable paradigm, using a lock file in stable jobs to get exact reproducibility of the testing environment and improve our trust that failures are always related to the patch under test.
Since lock files are currently foreign to our shared maintenance culture, and I was myself completely new to them a couple months back, I'll take a long detour to explain the motivation as well as the solution: if you don't know what a lock file is, hang in there ! I'll get to it.
Here are some external resources describing the core concepts and stakes for cargo, the rust package manager, and uv, the Python package manager inspired by cargo:
- Cargo.toml vs Cargo.lock
- Why have Cargo.lock in version control?
- Verifying Latest Dependencies (cargo)
- Locking and syncing (uv)
Problem: required CI isn't stable
Our regular CI comprises jobs that are required for merging, and some that are explicitly allowed to fail. The former are meant to reveal failures caused by the patch under test, while the latter are aimed to continuously reveal incoming incompatibilities with a selection of important dependencies (first and foremost, numpy).
Incoming incompatibilities indeed shouldn't block most patches, as they are usually caused by the outside world, so I think this distinction between required and allowed-failures is a great thing to have. However, I see a problem with how it is currently implemented:
Even in required jobs, the Python environment that is installed isn't stable or easily reproducible; whatever is the latest version of one of our dependencies, is what gets installed and tested against. This regularly leads to situations where required CI is failing on every single PR at once because of some upstream change we didn't happen to test early against (again, we only run such early tests against a selection of our direct dependencies, leaving lots of room for failure). Such a situation, because it is so inconvenient to the team as a whole, is often treated as an emergency, and rightfully so. Regardless how much dopamine I get by serving as our first line (and I am grateful to be of help as often as I can), I need to point that solutions found while in crisis mode may not always be the best or most robust ones.
It's also not always obvious what (if anything) changed in the environment between two runs, and a defensive reflex when ones encounter a cryptic, seemingly unrelated error, is to diff the outputs of pip freeze found in CI logs.
This is bad for two reasons:
- it shows that we do not really trust required CI to reveal only issues with the patch under test
- it is a somewhat tedious and error-prone task
Proposed solution (spoilers: it's lock files)
What are lock files
Simply put, a lock file is a record of a resolved environment with metadata including exact versions of all dependencies (direct, indirect, mandatory, and optional ones).
A requirements.txt file obtained via pip freeze is conceptually similar (albeit generally less portable1, or secure2), in that it allows to recreate an environment exactly.
Practically, lock files allow pinning all dependencies in a way that's not visible to end users.
Expected gains
Locking CI means that we never get surprise updates on weekends or in indirect dependencies we don't pay attention to. Updates happen on our schedule rather than being imposed by the outside world. This proposed strategy is actually not that foreign to us: we already use exact pins (paired with automated updates) for GitHub Action workflows and for pre-commit hooks.
Having a version-controlled lock files also means that we can revisit previous states of the repo and run tests within the exact same environment
The current state of standards
I want to point out that lock files are currently not standardized in the Python packaging ecosystem, which means all workflow tools that support a version of this implement their own flavor (poetry.lock, pdm.lock, uv.lock ...) but there is a PEP under discussion.
Namely PEP 751 is being discussed on discourse., note that stake holders (poetry, pdm, uv ...) are all included in the discussion, which is a good sign that this one might succeed3.
Having a standard file format for lock files would greatly improve interoperability between tools that consume them, and would allow us to migrate from one tool to another as we see fit without converting the lock file.
Proposed plan
I propose we start using a uv lock file (uv.lock) in stable CI as well as documentation (#17052) and wheel builds. I choose uv over other options because:
- it integrates nicely with our existing toolbox
- tox (via
tox-uv) - ReadTheDocs
- cibuildwheel
- dependabot
- tox (via
- astral-sh/uv as a very good record on supporting newly accepted PEPs (e.g. PEP 723 or PEP 735, supported within 2 weeks after acceptance), so I'm confident that they would support converting to a standardized lock file format if there is one in the future.
- I think I have a good understanding of how this tool work: I've been using
uvmore and more over the past year that it's been available. - I don't have experience with the other options (
pdm,poetryandpixiare the ones I know about), but I hear some of them have non-ideal records in terms of standard adoption:pdmsupport at least one PEP that was ultimately rejected- poetry has been very slow/reluctant to support standards, effectively locking its users in)
- maybe it's worth mentioning
pixiis already bound to uv in a sense (it uses uv internally for dependency resolution)
Deployment
There are 3 easy steps needed to adopt to this testing strategy, which I believe can be performed in a single PR:
- generate and commit the file itself (trivial)
- use uv in CI (this can easily be achieved as a follow up to #16963)
- add uv-lock to
pre-commit(ensureuv.lockstays in sync when dependencies are added, updated or removed inpyproject.toml)
For the record, I have been doing exactly this on smaller packages that I control.
Long term maintenance
The core idea is to decouple environment updates, and the ensuing risk of surprise incompatibility, from regular testing. However, running regular CI in an up-to-date environment remains very much needed, so keeping the lock file up to date is crucial to this strategy.
There are two questions to discuss here:
- how frequently should it be updated ? Given how large astropy's testing stack is, I would recommend at least weekly updates, maybe even daily (5 days a week) ?
- what service (if any) can we use ?
dependabot doesn't support uv.lock (yet ?)- dependabot supports
uv.lock - renovate does too
- we could also choose to only perform/commit manual upgrades using
uv lock --upgradelocally
My personal recommendation for now would be to use renovate exclusively for this task (as opposed to replacing dependabot completely) until dependabot gains support for uv.lock or an equivalent, standardized format (which might take a while).
Expected drawbacks, side effects
In short: none that I'm aware of.
How does this affect end users and downstream packages ?
It doesn't. Lock files are not meant to be distributed and using them does not change how we specify our dependencies in pyproject.toml .
How does this affect individual maintainers ?
This does not force anyone to use uv locally4 If you manage your own local dev environment(s) with pip, conda, pyenv (you name it) and you don't care for uv, nothing changes for you.
Maintainers who do want to use uv will benefit from having a centralized and version-controlled lock file.
In general, lock file upgrades should not invalidate CI jobs that already ran on pending PRs, with the possible exception of someone working specifically around dependency issues.
How does this affect release managers ?
This removes the (admittedly small) risk of discovering incompatibilities with our dependency stack between the moment a release is tagged and when it's actually published (wheels are tested in a guaranteed-stable environment).
Lock file updates should probably be backported as much as possible.
Does this lock us into current major versions of our dependencies ?
It's been pointed out by @eerovaher that cargo, the inspiration for uv, assumes that every package in the rust ecosystem strictly follows semantic versioning, so cargo update (the command to upgrade Cargo.lock) will not perform major upgrades by default (one needs to specify --breaking to get major upgrades). uv lock --upgrade doesn't do this and treats all upgrades (major, minor, patch) the same, to the best of my knowledge, and as of uv 0.5.14.
How does this affect oldestdeps and devdeps jobs ?
It doesn't. uv simply ignore existing lock files if a different resolution strategy is requested, which is exactly what's done for oldestdeps (via --resolution=lowest-direct, see #16963), and devdeps (via --prerelease=allow, equivalent to pip's --pre flag).
Cultural impact
This proposal is a cultural shift in the testing strategy. While it is by no means required that packages downstream of astropy follow the same path, I believe that it would still be helpful to encourage it: the problem I want to resolve isn't specific to astropy. I would be happy to help coordinated and affiliated packages migrate to this new paradigm if it is adopted by astropy itself.
Footnotes
-
Actual lock files are more portable: they can lock on different versions of a package depending, e.g., on the target platform (Linux VS Windows), or the version of Python, depending on what's available. See https://github.com/astral-sh/uv/pull/9827, for instance. ↩
-
lock files typically contain a lot more metadata than package names and versions, and will also record sources (usually
pypi.org) andsha256hashes of artifacts (wheels and source distributions) to be downloaded. ↩ -
except in the context of running tox, but one doesn't need to know about uv to use uv-in-tox, just like one doesn't need to know about
pipto run pip-in-tox (current status quo). ↩
Activity
neutrinoceros commentedon Jan 3, 2025
We discussed this issue at length with @eerovaher over yesterday's telecon. I've incorporated most his suggestions in the original post.
One other important idea he had was that instead of trying to guess an optimal updating frequency, we could make an informed decision by manually maintaining a lock file (outside of VCS) for a little while and get a sense of how many upgrades we typically go through in a month or so.
The
uv lockcommand actually has a--exclude-neweroption that allows me to simulate what would have happened if I had this idea a couple month back, so I'll do this instead and hopefully get some data today !neutrinoceros commentedon Jan 3, 2025
I ran three small simulations, varying the update strategy:
weekdays)mondays-and-thursdays)mondays)My simulations are spanned over the last two months (from 2024-11-01 to 2024-12-27 to be exact). The span is completely arbitrary; I just took a recent period that spans an integer number of weeks. In all my simulations, I froze astropy to commit 7d08082 (2024-11-01) for simplicity, created a lock file at this date and ran
uv lock --upgradefor each selected following update days.raw data (logging the output of `uv lock --upgrade`)
weekdaysmondays-and-thursdaysmondaysA simple viz (update size is simply computed as the number of packages that get updated)
Finally, some reduced metrics
There is some tension between minimizing the number of PRs that are generated and minimizing the size of each update. I resisted the temptation to weight updates by assumed importance (assuming version numbers are semantic) because I'm not sure it would yield anything meaningful and I also don't think the scoring system can easily be made free of my biases.
mhvk commentedon Jan 3, 2025
I like this idea overall, as long as the effort associated with updating the lock files becomes truly minimal. Obviously, it should at the very least be less than what we currently spend dealing with flames every half year or so, but I think the updates probably should just be done without intervention, i.e., something like a weekly test run with unlocked dependencies, and if those work, just update the lock file without a PR (or one that auto-merges; i.e., effectively mimic what happens currently, except that the updated packages have been tested).
Of course, the lack of oversight is also slightly worrying, since we then rely on that update mechanism to not inject something malicious (but a human would, after doing a check every week for a year, not notice anything either...). So, I think we'd perhaps need some separate check that the update only changes the lockfile and only changes version numbers in it.
On the developer side, an advantage of using
toxis that it gives an environment as similar as possible as what is run in CI. Would this still be the case? And could one turn the lockfile off? E.g., would it be as easy as, say, invokingtox -e test-alldeps-lockedortox -e test-alldeps-nolock?On the frequency of updates: I think weekly should be fine. Maybe this even helps with CI caching of packages? (I think that is done but am actually not sure...)
neutrinoceros commentedon Jan 3, 2025
Well, we need to define "truly minimal" first because there are two metrics we would like to minimize that can't possibly be minimized at the same time: frequency and size of updates.
good news: renovate (the only service that currently supports managing
uv.lock) can be configured for auto-mergers.That's a technicality, but upgrades are do not just change version numbers because lock files also record exact source urls and sha256 hashes.
That said, I don't know how to automate merges but still somehow validate patches.
yes.
hum, I guess the simplest way to "ignore" the lock file would be to run
uv lock --upgraderight before tests are run. I haven't studied question this too closely but I think it should be doable. In the meantime, as I mentioned, the lock will already be ignored fordevdepsandoldestdeps.astral-sh/setup-uvsupports caching (and enables it by default since v5.0)pllim commentedon Jan 3, 2025
Expected drawbacks, side effects
This idea is great if we are maintaining a pipeline, but this is a fundamental library. Since as you pointed out, most users don't go to great lengths to use
uv(some barely uses Python because we are a diverse community), the CI will stop testing a realistic env.Which as you also pointed out, moves the risk of incompatibility from PR time to release time or lock file update time. Fires are unpleasant but they do not happen very often. But if we lock everything down, then they pile up and add to the stress for core maintainers. One fire at a time, we can pinpoint more easily what happened and when, but if you pile them up, then more time need to be spent to do code archeology. Which could be minimized with higher frequency update but the risk will never be zero. Also moves the work load from a subpackage maintainer that might know exactly what is going on during unlucky PR review, to only a few (2-3) of us who actually checks infrastructure stuff.
We also inherit any potential bugs from whatever lockfile system we choose to go with. If their bot stops working for a while, small fire turns into a very big fire.
And what about non-traditional env like pyinstaller?
Lock file does not fix the problem, it delays the problem and move it to another plate. So I am not super enthusiastic about it, but I am open to what other core maintainers think.
mhvk commentedon Jan 3, 2025
Hmm, I do share @pllim's worry about "yet more magic that too few people understand". The more painful issues we have had are in fact with updates to our infrastructure packages (like
tox...), mostly because we do not really have expertise in those.That said, if all the cron jobs run without the lockfiles, we do gain the separation of concerns that drives this idea, that PRs are always tested in a known-good environment.
@neutrinoceros - I think a more minimal verifier for any auto-merge feature would simply check that the PR only actually touches the lockfiles.
bsipocz commentedon Jan 3, 2025
I also do not support this. The lock file just delays and enlarges pain, similar to when projects used to or still do exact versions in their CI files and keep receiving user complaints from actual real-life scenarios. So this adds an extra layer of complexity to @pllim infrastructure chart without much added benefit.
And besides, this would add a lot of noise of PRs for the automated lockfile updates. When a project already has a lot of non-content related PRs (it has been complained about here a lot already, then this just adds more into that same pile.
mhvk commentedon Jan 3, 2025
@bsipocz - the idea is that the lockfiles get updated very regularly, and that the PRs would be merged automatically. But the extra complexity definitely is a worry; it has to be weighed against regularly (if infrequently) getting to a state where every PR is failing because of a change in a dependency.
bsipocz commentedon Jan 3, 2025
@mhvk - 1) That would still generate notification noise. 2) I do maintain projects where we use a lock file, they have their absolute usefulness for projects, website build, etc, but not for a library as if there is indeed a real problem with the dependencies the lock file update can be painful. (and if all can be done automatically, then there is no need for the lockfile to begin with as there is nothing breaking and changing). I also maintain a couple of libraries where, for historical reason, there are pinned versions in the CI config. And that is an actual hindrance and pain and we are slowly but surely removing those pins as they hide actual issues the users do run into.
And likelihood that a PR has to be rebased and CI restarted if an update is done to the pinned numpy, or some dependency that actually matters, or would you trust a passing status and then rather fix up broken
main? That scenario is even more a no on a lock file in my book.mhvk commentedon Jan 3, 2025
I'm definitely feeling less convinced now that this is a good idea... I'd love to reduce the complexity of our infrastructure...
neutrinoceros commentedon Jan 3, 2025
I feel that maybe I didn't stress this enough but problems from realistic (i.e. not locked) environments would not be made invisible in between updates: they would continue to pop up in jobs that are allowed to fail (which I pay a lot of attention to).
neutrinoceros commentedon Jan 3, 2025
Me too. But somehow I feel that proposing, e.g., dropping tox would be a lot more controversial still.
mhvk commentedon Jan 3, 2025
The advantage of
toxis that it unifies things -- and we've probably all gotten used to it now... But it is a pain that numpy has its ownspinon things.pllim commentedon Jan 3, 2025
I still don't use
toxlocally... 😝20 remaining items
neutrinoceros commentedon Apr 28, 2025
tooling update: pip 25.1 ships a new (and experimental)
pip lockcommandnamurphy commentedon Apr 29, 2025
That's good news about
pip locksupportingpylock.tomlso quickly! 🎉🎂🎆🪩🌌🥦One thing to note is that the
pip lockdocumentation says:I think that's a limitation of
pylock.toml. In contrast,uv.lockdefines environments for multiple Python versions, multiple platforms, and all dependency groups in a single file. 🌈For PlasmaPy, I've had a great experience using
uv lockto pin requirements for CI and doc builds, since doing so greatly reduces the likelihood of other PRs having spontaneous unrelated failures. Having weekly automated PRs to updateuv.locklets us run tests with new versions of dependencies (and fix any new failures) before we start using them in CI.Note
A downside of pinning requirements with
uv.lockis that the diffs end up including a lot of hashes, so it was hard to tell what was changed solely from the diffs. For PlasmaPy, I made it so that the weekly automated PRs to updateuv.lockinclude a table of the changes, like in PlasmaPy/PlasmaPy#2996. (This was before dependabot started supportinguv, so I probably should have waited a month or two before setting up my own thing. 😅)Important
"Pylock" would be a great name for a cartoonish supervillain from the 1980s.
eli-schwartz commentedon Apr 29, 2025
That's why I said that I elsewhere pin requirements via weekly rolling docker containers, no need to review hashes then. :)
In fact though, why do you care what changed? If the only thing that changes is a lockfile, and the purpose of the lockfile is to define the CI environment, all you really need to do is assume that it upgrades "stuff" and gets "stuff" from PyPI. You can treat it as an opaque binary file. The lockfile is a means to an end, and the end is not about security (which is what hashes do), just about recreating the same additional installed software on top of
runs-on: ubuntu-latest, in a way that avoids "some dependency released a new version and we automatically started using it in CI, but weren't ready yet". Even committing a lockfile to git is just the simplest place to store that environment seed.namurphy commentedon Apr 29, 2025
That is an important question!
New releases of dependencies occasionally cause CI to break due to backward incompatible changes. Tracking down the cause of the problem is a lot easier when we know exactly which dependencies were updated.
We had a problem recently where one of PlasmaPy's Sphinx extensions got updated, and our documentation build broke. We found this out in one of the automated PRs to update the lock file, so we were able to quarantine the new version of the extension before we started using it in other PRs. Since I was able to quickly find out which packages were updated from that PR, I was able to go to the changelog for the extension and then raise an issue detailing the problem, which then got fixed upstream. In this case, knowing which packages got updated probably saved me ∼20 minutes.
eli-schwartz commentedon Apr 29, 2025
This doesn't require a hashing lockfile, though, only a list of versions. And even that only matters when the build breaks, it doesn't matter if the update is a success.
(It also does not strictly require the list of versions to be stored in the same repository as the project being tested, though as I said above it's certainly the simplest approach to git commit the list of versions.)
namurphy commentedon Apr 30, 2025
Agreed; for my use case the hashes didn't provide me with any useful information. This turned out to be what motivated me to add tables like this to the automated PRs that update
uv.lock:argcomplete3.5.33.6.0We used to pin requirements for CI with
requirements.txt-style files in the past. While the diffs were actually helpful, we had to create multiple requirements files for different versions of Python and different dependency sets. That was a mess to maintain...and then we discovered that a requirements file created for macOS does not necessarily work on Windows. 😭 😭 😭 😭 😭It was a big relief when we went from ∼18
requirements.txtfiles to a singleuv.lockfile that uniquely defines environments for multiple versions of Python, multiple dependency sets, and multiple operating systems with a single command (uv lock --upgrade) that typically takes ∼1–3 seconds to run.neutrinoceros commentedon Jun 23, 2025
anocdoctally, we presently have a situation with CI that I believe illustrates why all dependencies are important, and potentially blocking, with our current (non locked) CI strategy:
scipy(major, recommended-but-optional dep) andjplephem(very niche, completely optional and barely tested dep) both released versions that are breaking a required test job at the same timejplephemtest case for newer versions #18331This means we cannot merge anything until both problem are resolved, but they are also mutually blocking, so we need to either force-merge one fix, or resolve both (unrelated) issues in the same PR. Either way, it's pretty inconvenient, especially because I only have a solution to one problem at the moment.
We were lucky enough that breaking updates happened in direct dependencies, which made them much easier to guess, but it could have been much worse with indirect dependencies.
neutrinoceros commentedon Jun 24, 2025
Another data point: I just hit an error on readthedocs that reads
Now imagine having to check 1701 dependencies for "recent"2 changes.
Footnotes
yes, that's the actual number of dependencies in astropy (including dev-only and indirect ones) ↩
defining this word is left as an exercise to the reader ↩
mhvk commentedon Jun 24, 2025
The current astropy meeting seems like a perfect opportunity to make a decision here -- my 2¢ is to give it a shot. I like the separation of concerns of testing PRs with a known-good environment and using cron to identify problems with new packages (and update the lock file if there are none). The real worry is added complication -- so a PR introducing it should have good descriptions (maybe simply in the workflow file -- or at least a reference there).
neutrinoceros commentedon Jun 24, 2025
I'll propose this as a breakout session
neutrinoceros commentedon Sep 29, 2025
Follow up: we had discussion about this at the CM, but it was inconclusive as far as I recall.
A new data point: #18646
neutrinoceros commentedon Dec 12, 2025
another data point: scipy has been using
pixi.lockin CI for at least a couple weeks as far as I can tell.xref: scipy/scipy#21489
xref: scipy/scipy#23637
neutrinoceros commentedon Jun 9, 2026
For visibility: security issues that'd be solved by locking CI envs:
#19879