Skip to content

tests.eval: init#477760

Open
MattSturgeon wants to merge 2 commits intoNixOS:masterfrom
MattSturgeon:pkgs-eval-test
Open

tests.eval: init#477760
MattSturgeon wants to merge 2 commits intoNixOS:masterfrom
MattSturgeon:pkgs-eval-test

Conversation

@MattSturgeon
Copy link
Contributor

@MattSturgeon MattSturgeon commented Jan 7, 2026

Introduces pkgs/test/eval, which allows making assertions against a nix evaluation that is run within a test derivation's build script.

This is heavily inspired by lib/tests/test-with-nix.nix and Ci's eval comparison, which use the same technique of running nix within a derivation build script.

While this may be useful for any sufficiently complex pkgs tests, adding it is initially motivated by python's deprecated stdenv overriding. This needs test coverage but emits warnings, so cannot be directly included in the pkgs.tests evaluation. By using the eval within a build technique, we can not only retain test coverage, we can also make assertion about the emitted warning itself!

This is an alternative to the handleMsgStdenvArg workaround proposed in #477208 - cc @ShamrockLee

Things done


Add a 👍 reaction to pull requests you find important.

@MattSturgeon MattSturgeon changed the title tests/eval: init tests.eval: init Jan 7, 2026
@nixpkgs-ci nixpkgs-ci bot added 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. 10.rebuild-darwin: 1-10 This PR causes between 1 and 10 packages to rebuild on Darwin. 10.rebuild-darwin: 1 This PR causes 1 package to rebuild on Darwin. 10.rebuild-linux: 1 This PR causes 1 package to rebuild on Linux. labels Jan 7, 2026
@nixpkgs-ci nixpkgs-ci bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin. 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux. and removed 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. 10.rebuild-darwin: 1-10 This PR causes between 1 and 10 packages to rebuild on Darwin. 10.rebuild-darwin: 1 This PR causes 1 package to rebuild on Darwin. 10.rebuild-linux: 1 This PR causes 1 package to rebuild on Linux. labels Jan 9, 2026
Adds a test framework for evaluating Nixpkgs within a test derivation.
This allows testing things that should warn or throw, for example.
Restores the python deprecated stdenv override tests removed in
9667e93 and adapts them to also
validate the emitted warning.
Comment on lines 69 to 73
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With #477208 merged, we can also expect buildPythonPackage-fp-args-stdenv-deprecated to warn.

Tested with:

From 629af6c9da8cdcb8253a2a9d345ebf1281f745e4 Mon Sep 17 00:00:00 2001
From: Matt Sturgeon <[email protected]>
Date: Thu, 15 Jan 2026 11:27:10 +0000
Subject: [PATCH 1/2] tests.eval: check buildPythonPackage fixpoint args
 warning

---
 pkgs/test/eval/default.nix | 1 +
 1 file changed, 1 insertion(+)

diff --git a/pkgs/test/eval/default.nix b/pkgs/test/eval/default.nix
index 5eb709bab139..08fc6587aec8 100644
--- a/pkgs/test/eval/default.nix
+++ b/pkgs/test/eval/default.nix
@@ -70,6 +70,7 @@ runCommand "nixpkgs-pkgs-tests-eval"
       overridePythonAttrs-stdenv-deprecated
       overridePythonAttrs-override-clangStdenv-deprecated-nested
       buildPythonPackage-stdenv-deprecated
+      buildPythonPackage-fp-args-stdenv-deprecated
     )
     expected_warning='evaluation warning: python-package-stub: Passing `stdenv` directly to `buildPythonPackage` or `buildPythonApplication` is deprecated.'
 
-- 
2.52.0
From ff90863b81c129fb07d342056dd1dd5148f7976b Mon Sep 17 00:00:00 2001
From: Matt Sturgeon <[email protected]>
Date: Thu, 15 Jan 2026 11:36:29 +0000
Subject: [PATCH 2/2] buildPython*: include package name in fixpoint-args
 stdenv warning

In the simple case we attempt to get a name from the input args, as this
avoids evaluating the initial `result`.

In the fixpoint-args scenario, this isn't possible. Therefore, we should
extract the name from `result`.
---
 .../python/python-packages-base.nix           | 19 +++++++++++--------
 1 file changed, 11 insertions(+), 8 deletions(-)

diff --git a/pkgs/development/interpreters/python/python-packages-base.nix b/pkgs/development/interpreters/python/python-packages-base.nix
index 9537844d10a8..b62ec897de87 100644
--- a/pkgs/development/interpreters/python/python-packages-base.nix
+++ b/pkgs/development/interpreters/python/python-packages-base.nix
@@ -55,24 +55,27 @@ let
         args:
         let
           result = f args;
-          applyMsgStdenvArg = lib.warnIf (lib.oldestSupportedReleaseIsAtLeast 2511) ''
-            ${
-              args.name or args.pname or "<unnamed>"
-            }: Passing `stdenv` directly to `buildPythonPackage` or `buildPythonApplication` is deprecated. You should use their `.override` function instead, e.g:
-              buildPythonPackage.override { stdenv = customStdenv; } { }
-          '';
+          getName = x: x.pname or (lib.getName (x.name or "<unnamed>"));
+          applyMsgStdenvArg =
+            name:
+            lib.warnIf (lib.oldestSupportedReleaseIsAtLeast 2511) ''
+              ${name}: Passing `stdenv` directly to `buildPythonPackage` or `buildPythonApplication` is deprecated. You should use their `.override` function instead, e.g:
+                buildPythonPackage.override { stdenv = customStdenv; } { }
+            '';
         in
         if lib.isFunction args then
           if result ? __stdenvPythonCompat then
             # Less reliable, as constructing with the wrong `stdenv` might lead to evaluation errors in the package definition.
-            f'.override { stdenv = applyMsgStdenvArg result.__stdenvPythonCompat; } (
+            f'.override { stdenv = applyMsgStdenvArg (getName result) result.__stdenvPythonCompat; } (
               finalAttrs: removeAttrs (args finalAttrs) [ "stdenv" ]
             )
           else
             result
         else if args ? stdenv then
           # More reliable, but only works when args is not `(finalAttrs: { })`
-          f'.override { stdenv = applyMsgStdenvArg args.stdenv; } (removeAttrs args [ "stdenv" ])
+          f'.override { stdenv = applyMsgStdenvArg (getName args) args.stdenv; } (
+            removeAttrs args [ "stdenv" ]
+          )
         else
           result
       )
-- 
2.52.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin. 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant