[Backport release-25.05] workflows/eval: compare eval consistency and performance between Lix / Nix versions#433033
Merged
wolfgangwalther merged 5 commits intorelease-25.05from Aug 12, 2025
Conversation
The Dependabot update change the hashes to the latest main branch commit instead of the v5.0.0 tag - also it didn't adjust the tags in the comments accordingly. Last but not least, one of the references used a `@v5` reference instead of the commit hash. The latter is probably what Dependabot tripped on. (cherry picked from commit 51e6b0e)
Move "Packages" up, because it's much shorter and easier to scroll past. This way both Packages and Performance are visible immediately. (cherry picked from commit f05895f)
With this change, we start running Eval on all available Lix and Nix versions. Because this requires a lot of resources, this complete test is only run when `ci/pinned.json` is updated. The resulting outpaths are checked for consistency with the target branch. A difference will cause the `report` job to fail, thus blocking the merge, ensuring Eval consistency for Nixpkgs across different versions. This implements a kind of "ratchet style" check: Since we originally confirmed that the versions currently in Nixpkgs at the time of this commit match Eval behavior of Nix 2.3, we can ensure consistency with Nix 2.3 down the road, even without testing for it explicitly. There had been one regression in Eval consistency for Nix between 2.18 and 2.24 - two tests in `tests.devShellTools` produce different results between Lix 2.91+ (which was forked from Nix 2.18) and Nix 2.24+. I assume it's unlikely that such a change would be "fixed" by now, thus I added an exception for these. As a bonus, we also present the total time in seconds it takes for Eval to complete for every tested version in a summary table. This allows us to easily see performance improvements for Eval due to version updates. At this stage, this time only includes the "outpaths" step of Eval, but not the generation of attrpaths beforehand. (cherry picked from commit b523f25)
This gives us a fixed `lixPackageSets.git`. From the nixpkgs-unstable channel: https://hydra.nixos.org/build/304569381#tabs-buildinputs Changes for treefmt-nix: numtide/treefmt-nix@58bd4da...7d81f6f (cherry picked from commit 14a6d9d)
Contributor
|
Investigating #427724 (comment) first. |
The conclusion of the `versions` job propagates from through `eval` to `compare`, which meant the `compare` job was skipped. No rebuild labels, no reviewer requests. Also, we don't want to run eval when `versions` runs, but fails. (cherry picked from commit 267d5cd)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bot-based backport to
release-25.05, triggered by a label in #427724.