Skip to content

[Backport release-25.05] workflows/eval: compare eval consistency and performance between Lix / Nix versions#433033

Merged
wolfgangwalther merged 5 commits intorelease-25.05from
backport-427724-to-release-25.05
Aug 12, 2025
Merged

[Backport release-25.05] workflows/eval: compare eval consistency and performance between Lix / Nix versions#433033
wolfgangwalther merged 5 commits intorelease-25.05from
backport-427724-to-release-25.05

Conversation

@nixpkgs-ci
Copy link
Contributor

@nixpkgs-ci nixpkgs-ci bot commented Aug 12, 2025

Bot-based backport to release-25.05, triggered by a label in #427724.

  • Before merging, ensure that this backport is acceptable for the release.
    • Even as a non-committer, if you find that it is not acceptable, leave a comment.

The Dependabot update change the hashes to the latest main branch commit
instead of the v5.0.0 tag - also it didn't adjust the tags in the
comments accordingly. Last but not least, one of the references used a
`@v5` reference instead of the commit hash. The latter is probably what
Dependabot tripped on.

(cherry picked from commit 51e6b0e)
Move "Packages" up, because it's much shorter and easier to scroll past.
This way both Packages and Performance are visible immediately.

(cherry picked from commit f05895f)
With this change, we start running Eval on all available Lix and Nix
versions. Because this requires a lot of resources, this complete test
is only run when `ci/pinned.json` is updated.

The resulting outpaths are checked for consistency with the target
branch. A difference will cause the `report` job to fail, thus blocking
the merge, ensuring Eval consistency for Nixpkgs across different
versions.

This implements a kind of "ratchet style" check: Since we originally
confirmed that the versions currently in Nixpkgs at the time of this
commit match Eval behavior of Nix 2.3, we can ensure consistency with
Nix 2.3 down the road, even without testing for it explicitly.

There had been one regression in Eval consistency for Nix between 2.18
and 2.24 - two tests in `tests.devShellTools` produce different results
between Lix 2.91+ (which was forked from Nix 2.18) and Nix 2.24+. I
assume it's unlikely that such a change would be "fixed" by now, thus I
added an exception for these.

As a bonus, we also present the total time in seconds it takes for Eval
to complete for every tested version in a summary table. This allows us
to easily see performance improvements for Eval due to version updates.
At this stage, this time only includes the "outpaths" step of Eval, but
not the generation of attrpaths beforehand.

(cherry picked from commit b523f25)
This gives us a fixed `lixPackageSets.git`.

From the nixpkgs-unstable channel:
https://hydra.nixos.org/build/304569381#tabs-buildinputs

Changes for treefmt-nix:
numtide/treefmt-nix@58bd4da...7d81f6f

(cherry picked from commit 14a6d9d)
@nixpkgs-ci nixpkgs-ci bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin. 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux. 6.topic: continuous integration Affects continuous integration (CI) in Nixpkgs, including Ofborg and GitHub Actions 4.workflow: backport This targets a stable branch 6.topic: policy discussion Discuss policies to work in and around Nixpkgs labels Aug 12, 2025
@wolfgangwalther wolfgangwalther marked this pull request as draft August 12, 2025 08:47
@wolfgangwalther
Copy link
Contributor

Investigating #427724 (comment) first.

The conclusion of the `versions` job propagates from through `eval` to
`compare`, which meant the `compare` job was skipped. No rebuild labels,
no reviewer requests.

Also, we don't want to run eval when `versions` runs, but fails.

(cherry picked from commit 267d5cd)
@wolfgangwalther wolfgangwalther marked this pull request as ready for review August 12, 2025 09:31
@wolfgangwalther wolfgangwalther merged commit a88329d into release-25.05 Aug 12, 2025
80 of 83 checks passed
@wolfgangwalther wolfgangwalther deleted the backport-427724-to-release-25.05 branch August 12, 2025 09:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

4.workflow: backport This targets a stable branch 6.topic: continuous integration Affects continuous integration (CI) in Nixpkgs, including Ofborg and GitHub Actions 6.topic: policy discussion Discuss policies to work in and around Nixpkgs 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin. 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant