splice.nix: make `pkgs` `splicedPackages` by Artturin · Pull Request #349316 · NixOS/nixpkgs

Artturin · 2024-10-17T16:13:48Z

This will make pkgs used in callPackage, and pkgsCross.X.pkgs have packages with __spliced.

https://www.github.com/NixOS/nixpkgs/blob/3029741718f4c765fbc5ebf76bea3d6c8ff15fe5/pkgs/development/interpreters/python/passthrufun.nix#L37

https://www.github.com/NixOS/nixpkgs/blob/d2bd9a39dec88eddd5c192abee69939e67f43d12/pkgs/top-level/python-packages.nix#L10720

nix-repl> pkgsCross.aarch64-multiplatform.python3Packages.protobuf4.protobuf.__spliced
error:
       … while evaluating the attribute 'aarch64-multiplatform.python3Packages.protobuf4.protobuf.__spliced'
         at /home/artturin/nixgits/my-nixpkgs/.worktree/1/pkgs/development/python-modules/protobuf/4.nix:119:13:
          118|   passthru = {
          119|     inherit protobuf;
             |             ^
          120|   };

       error: attribute '__spliced' missing
       at «string»:1:1:
            1| pkgsCross.aarch64-multiplatform.python3Packages.protobuf4.protobuf.__spliced
             | ^

to

nix-repl> pkgsCross.aarch64-multiplatform.python3Packages.protobuf4.protobuf.__spliced
{
  buildBuild = «derivation /nix/store/s7da5mfvx4h1n86j78knaj9cprglxqz6-protobuf-25.4.drv»;
  buildHost = «derivation /nix/store/s7da5mfvx4h1n86j78knaj9cprglxqz6-protobuf-25.4.drv»;
  buildTarget = «repeated»;
  hostHost = «derivation /nix/store/mszvybzs4zxh43awyrjnybsfcb265n9r-protobuf-aarch64-unknown-linux-gnu-25.4.drv»;
  hostTarget = «repeated»;
}

Will allow getting rid of many of these https://github.com/search?q=lang%3Anix+NOT+is%3Afork+%2Fpkgs.%2B__splicedPackages%2F&type=code
#251005 is required to get rid of all of them

Things done

Add a 👍 reaction to pull requests you find important.

Atemu · 2024-10-21T11:36:15Z

This sounds like an obvious win to me, is there anything left to test or verify?

kjeremy · 2024-10-23T15:58:41Z

Would #263082 have caught this? Maybe it would be worth reviving that.

Aleksanaa · 2024-10-23T16:05:31Z

I got the feeling of this may be doable a couple of weeks ago. But now there are not many people who really dare to change the cross logic, come on 🫣

Artturin · 2024-10-31T18:05:38Z

pkgs/top-level/splice.nix

@Ericson2314
I wonder why you didn't do this instead of adding __splicedPackages? f27f491

Artturin · 2024-11-01T15:24:07Z

Interesting, the fetchpatch changes in 8614381 cause infinite recursion https://gist.github.com/GrahamcOfBorg/784e668fe0b8a16a5bf11b5b622654cc

Because pkgs isn't __splicedPackages in all-packages.nix

diff --git a/pkgs/top-level/all-packages.nix b/pkgs/top-level/all-packages.nix
index a14ffa88fd95..a0e8dead9df9 100644
--- a/pkgs/top-level/all-packages.nix
+++ b/pkgs/top-level/all-packages.nix
@@ -11,6 +11,7 @@ res: pkgs: super:
 with pkgs;
 
 {
+  pkgs1 = pkgs;
   # A module system style type tag
   #
   # Allows the nixpkgs fixpoint, usually known as `pkgs` to be distinguished

nix-repl> pkgsStatic.pkgs1.bash ? __spliced
false

Artturin · 2024-11-01T16:58:33Z

pkgs/top-level/all-packages.nix

Oh yeah #58327
If the pkgs used in all-packages.nix was __splicedPackages it would break with infinite recursion

nix-repl> pkgsStatic.pkgs1.bash ? __spliced false nix-repl> pkgsStatic.pkgs1.pkgs.bash ? __spliced true

The pkgs in patchutils = pkgs.patchutils_0_3_3; in all-packages.nix is coming from this pkgs: https://github.com/NixOS/nixpkgs/blob/b6e486730fc875ec79a3dea0f1f46eaf9f6ffa9d/pkgs/top-level/all-packages.nix#L9
instead of the pkgs in with pkgs;

I've posted this patch a few times before, but it does still seem (when updated slightly (gccCrossStageStatic has a new attr name gccWithoutTargetLibc since that patch was written) and applied on top of this PR) to fix the infrec from inheriting from __splicedPackages: lilyinstarlight@9639e73

I didn't do rigorous testing, but it does result in the same .drvs for already-cross-safe attrs I tested

nixos-discourse · 2024-11-10T21:54:01Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/is-it-possible-for-us-to-remove-builtins-functionargs/51960/6

This will make `pkgs` used in `callPackage`, and `pkgsCross.X.pkgs` have packages with `__spliced`. https://www.github.com/NixOS/nixpkgs/blob/3029741718f4c765fbc5ebf76bea3d6c8ff15fe5/pkgs/development/interpreters/python/passthrufun.nix#L37 https://www.github.com/NixOS/nixpkgs/blob/d2bd9a39dec88eddd5c192abee69939e67f43d12/pkgs/top-level/python-packages.nix#L10720 ``` nix-repl> pkgsCross.aarch64-multiplatform.python3Packages.protobuf4.protobuf.__spliced error: … while evaluating the attribute 'aarch64-multiplatform.python3Packages.protobuf4.protobuf.__spliced' at /home/artturin/nixgits/my-nixpkgs/.worktree/1/pkgs/development/python-modules/protobuf/4.nix:119:13: 118| passthru = { 119| inherit protobuf; | ^ 120| }; error: attribute '__spliced' missing at «string»:1:1: 1| pkgsCross.aarch64-multiplatform.python3Packages.protobuf4.protobuf.__spliced | ^ ``` to ``` nix-repl> pkgsCross.aarch64-multiplatform.python3Packages.protobuf4.protobuf.__spliced { buildBuild = «derivation /nix/store/s7da5mfvx4h1n86j78knaj9cprglxqz6-protobuf-25.4.drv»; buildHost = «derivation /nix/store/s7da5mfvx4h1n86j78knaj9cprglxqz6-protobuf-25.4.drv»; buildTarget = «repeated»; hostHost = «derivation /nix/store/mszvybzs4zxh43awyrjnybsfcb265n9r-protobuf-aarch64-unknown-linux-gnu-25.4.drv»; hostTarget = «repeated»; } ```

Now that `pkgs` is `__splicedPackages` on cross we can remove these variables I set. Assignments like `patchutils = pkgs.patchutils_0_3_3;` in `all-packages.nix` cannot be removed because the `pkgs` in `patchutils = pkgs.patchutils_0_3_3;` in `all-packages.nix` is coming from this `pkgs:` https://www.github.com/NixOS/nixpkgs/blob/b6e486730fc875ec79a3dea0f1f46eaf9f6ffa9d/pkgs/top-level/all-packages.nix#L9 instead of the `pkgs` in `with pkgs;`

roberth · 2024-11-15T11:25:49Z

This seems like a massive improvement, at no apparent cost.

Would it be possible to do the same to the overlay fixpoint, final?
I believe that solution would then also have the same effect on the pkgs attribute.

Currently in this PR:

nix-repl> p.hello?__spliced
false

nix-repl> p.pkgs.hello?__spliced
true

philiptaron · 2024-12-06T16:17:08Z

This has the label 2.status: wait for branch‐off -- and the branch cutoff has happened. Let's get this merged!

Atemu · 2024-12-06T16:45:12Z

Is there anything left to test or think more closely about? Otherwise I'd be in favour of merging and seeing whether anything breaks.

Artturin · 2024-12-06T16:47:03Z

I want to get #341067 evalling first and then apply #341067 and 6761b8e (#341067) here to see the actual perf diff

Artturin · 2024-12-06T16:47:37Z

I was thinking of a different PR, there's no perf diff here :P I have 3 PRs touching splice.nix so got confused.

philiptaron

The changes look great -- and if they eval, let's merge.

Artturin · 2024-12-06T16:58:52Z

Weird seems like the github actions eval got inf rec https://github.com/NixOS/nixpkgs/actions/runs/12202727756/job/34044092263 and this PR didn't yet use github actions eval #362499 (comment)

Artturin · 2024-12-06T17:06:38Z

Take 2 #362499

wolfgangwalther · 2025-10-27T08:50:57Z

I'm working on this again in #456138.

ofborg bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin. 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux. labels Oct 17, 2024

SomeoneSerge mentioned this pull request Oct 17, 2024

python312Packages.protobuf: propagate pkgs.protobuf #349270

Merged

13 tasks

Artturin force-pushed the pkgsisspliced branch from 3e6ad22 to 0793f85 Compare October 31, 2024 16:46

Artturin marked this pull request as ready for review October 31, 2024 16:46

nix-owners bot requested a review from Ericson2314 October 31, 2024 16:47

github-actions bot added the 6.topic: erlang General-purpose, concurrent, functional high-level programming language label Oct 31, 2024

nix-owners bot requested review from marcusramberg, stigtsp and zakame October 31, 2024 17:21

Artturin force-pushed the pkgsisspliced branch 2 times, most recently from 4bb1955 to 47c47cc Compare October 31, 2024 17:28

Artturin commented Oct 31, 2024

View reviewed changes

ofborg bot added 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. and removed 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux. labels Oct 31, 2024

Artturin force-pushed the pkgsisspliced branch from 47c47cc to 65ca39b Compare November 1, 2024 01:28

github-actions bot added the 6.topic: python Python is a high-level, general-purpose programming language. label Nov 1, 2024

nix-owners bot requested a review from natsukium November 1, 2024 01:30

Artturin force-pushed the pkgsisspliced branch 2 times, most recently from 01b7508 to 8614381 Compare November 1, 2024 01:43

Artturin commented Nov 1, 2024

View reviewed changes

Artturin force-pushed the pkgsisspliced branch from 8614381 to 58d7711 Compare November 1, 2024 18:24

github-actions bot removed the 6.topic: python Python is a high-level, general-purpose programming language. label Nov 1, 2024

Artturin force-pushed the pkgsisspliced branch 2 times, most recently from 259c3f0 to 96fb948 Compare November 5, 2024 15:45

ofborg bot added the 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux. label Nov 5, 2024

ofborg bot removed the 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. label Nov 5, 2024

wegank added the 2.status: merge conflict This PR has merge conflicts with the target branch label Nov 10, 2024

Artturin added 2 commits November 11, 2024 00:08

Artturin force-pushed the pkgsisspliced branch from 96fb948 to a27e48e Compare November 10, 2024 22:25

Aleksanaa added the 2.status: wait for branch‐off Waiting for the next Nixpkgs branch‐off label Nov 11, 2024

ofborg bot removed the 2.status: merge conflict This PR has merge conflicts with the target branch label Nov 11, 2024

Artturin marked this pull request as draft December 6, 2024 16:47

Artturin marked this pull request as ready for review December 6, 2024 16:47

philiptaron approved these changes Dec 6, 2024

View reviewed changes

Artturin merged commit debf997 into NixOS:master Dec 6, 2024

Artturin deleted the pkgsisspliced branch December 6, 2024 16:49

This was referenced Dec 6, 2024

Revert "splice.nix: make pkgs splicedPackages" #362496

Merged

splice.nix: make pkgs splicedPackages when required #362499

Closed

kachick mentioned this pull request Dec 6, 2024

dprint: enable doCheck with some skips #362498

Merged

13 tasks

Aleksanaa mentioned this pull request Jun 9, 2025

nixos/man-db: fix cross‐compilation #415044

Merged

13 tasks

emilazy mentioned this pull request Jun 13, 2025

nixos/nixpkgs/read-only: fix cross‐compilation #416276

Merged

13 tasks

wolfgangwalther mentioned this pull request Oct 27, 2025

Reapply "splice.nix: make pkgs splicedPackages" #456138

Merged

3 tasks

wolfgangwalther mentioned this pull request Nov 3, 2025

all-packages: use pkgs.__splicedPackages scope #458111

Closed

3 tasks

Uh oh!

Conversation

Artturin commented Oct 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Things done

Uh oh!

Atemu commented Oct 21, 2024

Uh oh!

kjeremy commented Oct 23, 2024

Uh oh!

Aleksanaa commented Oct 23, 2024

Uh oh!

Artturin Oct 31, 2024

Choose a reason for hiding this comment

Uh oh!

Artturin commented Nov 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Artturin Nov 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lilyinstarlight Nov 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nixos-discourse commented Nov 10, 2024

Uh oh!

roberth commented Nov 15, 2024

Uh oh!

philiptaron commented Dec 6, 2024

Uh oh!

Atemu commented Dec 6, 2024

Uh oh!

Artturin commented Dec 6, 2024

Uh oh!

Artturin commented Dec 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

philiptaron left a comment

Choose a reason for hiding this comment

Uh oh!

Artturin commented Dec 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Artturin commented Dec 6, 2024

Uh oh!

wolfgangwalther commented Oct 27, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

Artturin commented Oct 17, 2024 •

edited

Loading

Artturin commented Nov 1, 2024 •

edited

Loading

Artturin Nov 1, 2024 •

edited

Loading

lilyinstarlight Nov 1, 2024 •

edited

Loading

Artturin commented Dec 6, 2024 •

edited

Loading

Artturin commented Dec 6, 2024 •

edited

Loading