Reapply "splice.nix: make pkgs splicedPackages" by wolfgangwalther · Pull Request #456138 · NixOS/nixpkgs

wolfgangwalther · 2025-10-27T08:49:45Z

This reapplies #349316, which was tried in #362499, but then abandoned.

Does #349316 (comment), too, but not #349316 (comment). The latter still causes me some infrec headache for darwin cross.

Dropping stdenv/native along the way, which was broken right now anyway - and causes infinite recursions through these nativeTools & co. arguments.

Things done

Basic tests work.
Do more sophisticated testing.
Fits CONTRIBUTING.md, pkgs/README.md, maintainers/README.md and other READMEs.

Add a 👍 reaction to pull requests you find important.

szlend · 2025-10-27T10:54:14Z

lib/customisation.nix

@@ -726,7 +726,7 @@ rec {
        pkgsTargetTarget = otherSplices.selfTargetTarget;
      };
      spliced = extra spliced0 // spliced0 // keep self;
-      self = f self // {
+      self = f spliced0 // {


If we change self to be already spliced, do we also need to adjust this, a few lines above?

pkgsHostTarget = self; # Not `otherSplices.selfHostTarget`;

My understanding is pkgs<host><target> package sets are supposed to be un-spliced, otherwise they could get unintentionally shifted when placed into inputs.

This doesn’t add the splices to the defined self, it just passes in the spliced packages to the scope’s function. The spliced packages were already accessible through self.callPackage, but not when using self directly – analogous to how packages in all-packages.nix aren’t from __splicedPackages.

(Admittedly though this means that f = self: { …; foo = self.bar; } would insert splices into the resulting self and therefore into its selfHostTarget – maybe that’s why it was not done?)

Sorry, what I meant was that we'd be generating foo.__spliced.hostTarget from a spliced package set, instead of an unspliced one. Though thinking about it some more, I'm no longer sure if this would actually even be an issue.

Right, what I mean is that if you do this:

makeScopeWithSplicing' { … f = self: { foo = self.callPackage … { }; bar = self.callPackage ({ foo }: …) { }; }; }

then bar’s foo dependency already has .__spliced today, because callPackage is defined in terms of spliced. Most packages are defined with callPackage already. (The return values of callPackage don’t have it, though; that’s added in later.)

What would change is that self.bar would gain .__spliced where it doesn’t currently have it – to me, this is less confusing than the status quo. OTOH, it does have the effect that:

makeScopeWithSplicing' { … f = self: { foo = self.callPackage … { }; bar = self.foo; }; }

…would put bar.__spliced in the unspliced versions of the scope used to splice together the scope. That could be a problem; I’d assume it’s what’s happening with #456138 (comment).

(Maybe that is what you meant, but just making sure.)

It's not exactly what I had in mind, but that indeed looks like a problem yeah.

It doesn't feel to me like this can be avoided with the way splicing works though.

What if we kept self = f self // ... and instead made makeScopeWithSplicing' return spliced0? There's quite a bit more that would need to change in that function, but it would sort of mimic what was done to pkgs where pkgs to the outside was redefined to splicedPackages.

(keeping this thread open, although not relevant anymore, because the commit was taken out again - will reference this thread in the follow up PR)

philiptaron · 2025-10-27T14:06:00Z

I'm smashing the subscribe button.

RossSmyth · 2025-10-27T15:04:38Z

Great idea

emilazy · 2025-10-27T19:47:53Z

Dropping stdenv/native along the way, which was broken right now anyway - and causes infinite recursions through these nativeTools & co. arguments.

The changes here wouldn’t cause that without the __splicedPackages in all-packages.nix, right? Honestly, I somewhat prefer keeping this around for now as “canaries in the coalmine” to show us what issues that would cause – although I do think it ought to be dropped in general.

wolfgangwalther · 2025-10-27T19:52:20Z

Dropping stdenv/native along the way, which was broken right now anyway - and causes infinite recursions through these nativeTools & co. arguments.

The changes here wouldn’t cause that without the __splicedPackages in all-packages.nix, right?

That's what I thought at first, but it doesn't seem to be the case, no, at least when I tried this morning. OTOH, I had received confusingly different results from time to time, so I'm not really trusting myself to always hold this correctly. I will test and confirm it again, but my current idea is that this is indeed already required for the actual reapply.

wolfgangwalther · 2025-10-27T20:46:03Z

I tested again: The reapply itself works in all my basic tests. It's the lib.customisation: make self spliced in scopes commit that causes stdenv.cc.nativeTools infrec for pkgsLLVM now, which makes the commit to drop stdenv/native required here.

wolfgangwalther · 2025-10-28T13:49:21Z

I tested again: The reapply itself works in all my basic tests. It's the lib.customisation: make self spliced in scopes commit that causes stdenv.cc.nativeTools infrec for pkgsLLVM now, which makes the commit to drop stdenv/native required here.

I also looked into the eval failure by CI in nixComponents_... - this is caused by the same commit. To reduce scope, I removed the commit about scopes for now.

I'll bring the drop for stdenv/native back with the scope commit in a separate PR.

wolfgangwalther · 2025-10-28T15:24:46Z

I now did the following verification:

ran a nix repl session with NIXPKGS_ALLOW_UNFREE=1
took the output of lib.mapAttrs (_: ps: ps.hello) pkgsCross (aka the hello package for every cross example we have)
diffed the output from before this PR to after this PR
on all 4 major platforms

I found a diff for the windows platforms, which I fixed in the latest push. Afterwards, there was no diff anymore.

I also [ pkgsStatic.hello pkgsLLVM.hello pkgsMusl.hello pkgs.i686Linux.hello pkgsExtraHardening.hello pkgsArocc.hello pkgsZig.hello ] on both x86_64-linux and aarch64-darwin. Same hashes (or same eval error as before).

I'll have to see whether any rebuilds remain after this push and look into them in more detail. In general, it'd not be unexpected to have some hashes changing - for cases where the packages should have been spliced before but were not.

wolfgangwalther · 2025-10-28T15:36:31Z

The changed packages now are:

extra-container
mmseqs2
nixos-container
nixos-install-tools
nixpkgs-manual
tests.nixos-functions.nixos-test

Everything except mmseqs2 is uncritical, because they just pull in the whole nixpkgs source or lib/ or so. But mmseqs2... is cursed:

• The set of input derivation names do not match:
    - zstd-static-x86_64-unknown-linux-musl-1.5.7
    + zstd-1.5.7

nixpkgs/pkgs/by-name/mm/mmseqs2/package.nix

Lines 22 to 25 in dd42bb1

    
           let 
        
             # require static library, libzstd.a 
        
             inherit (pkgsStatic) zstd; 
        
           in

Now, that's worrying in two aspects:

Why does anyone pull in a single dependency from pkgsStatic? How does that even work - mixing glibc and musl in the same build?
Why does this change with this PR? That's not good, I think.

wolfgangwalther · 2025-10-28T16:29:51Z

Why does this change with this PR? That's not good, I think.

Actually it is fine, from this PR's perspective. The mmseqs2 is even more cursed than expected: pkgsStatic.zstd is added to nativeBuildInputs, which means that it will use pkgsStatic.zstd.__spliced.buildHost, now that this is available. It was not spliced before, it is now because of 1093f4aa71ef5125d0c6bb479070e4c22db2b6bd that makes pkgsStatic spliced.

relibc itself has been removed in 2024 already.

This was removed recently when cleaning up aliases, but it breaks `pkgsCross.ben_nanonote`.

This will make `pkgs` used in `callPackage`, and `pkgsCross.X.pkgs` have packages with `__spliced`. https://www.github.com/NixOS/nixpkgs/blob/3029741718f4c765fbc5ebf76bea3d6c8ff15fe5/pkgs/development/interpreters/python/passthrufun.nix#L37 https://www.github.com/NixOS/nixpkgs/blob/d2bd9a39dec88eddd5c192abee69939e67f43d12/pkgs/top-level/python-packages.nix#L10720 ``` nix-repl> pkgsCross.aarch64-multiplatform.python3Packages.protobuf4.protobuf.__spliced error: … while evaluating the attribute 'aarch64-multiplatform.python3Packages.protobuf4.protobuf.__spliced' at /home/artturin/nixgits/my-nixpkgs/.worktree/1/pkgs/development/python-modules/protobuf/4.nix:119:13: 118| passthru = { 119| inherit protobuf; | ^ 120| }; error: attribute '__spliced' missing at «string»:1:1: 1| pkgsCross.aarch64-multiplatform.python3Packages.protobuf4.protobuf.__spliced | ^ ``` to ``` nix-repl> pkgsCross.aarch64-multiplatform.python3Packages.protobuf4.protobuf.__spliced { buildBuild = «derivation /nix/store/s7da5mfvx4h1n86j78knaj9cprglxqz6-protobuf-25.4.drv»; buildHost = «derivation /nix/store/s7da5mfvx4h1n86j78knaj9cprglxqz6-protobuf-25.4.drv»; buildTarget = «repeated»; hostHost = «derivation /nix/store/mszvybzs4zxh43awyrjnybsfcb265n9r-protobuf-aarch64-unknown-linux-gnu-25.4.drv»; hostTarget = «repeated»; } ``` Co-authored-by: Wolfgang Walther <[email protected]>

Now that `pkgs` is `__splicedPackages` on cross we can remove these variables I set. Assignments like `patchutils = pkgs.patchutils_0_3_3;` in `all-packages.nix` cannot be removed because the `pkgs` in `patchutils = pkgs.patchutils_0_3_3;` in `all-packages.nix` is coming from this `pkgs:` https://www.github.com/NixOS/nixpkgs/blob/b6e486730fc875ec79a3dea0f1f46eaf9f6ffa9d/pkgs/top-level/all-packages.nix#L9 instead of the `pkgs` in `with pkgs;`

wolfgangwalther · 2025-10-28T20:56:37Z

mmseqs2 was fixed in #456530. This rebuild should disappear now and I consider the current state ready for review / ready to go, as a first step.

Once this is merged, I will open two follow up PRs with the changes I backed out - but these will be draft for now, because they still hit infinite recursion in some places, and I will probably need help with these.

pkgs/top-level/splice.nix

philiptaron

I read through this commit-by-commit. Some questions were straightforward to answer, but a couple places could use bigger comments explaining what and why.

pkgs/top-level/aliases.nix

pkgs/top-level/all-packages.nix

philiptaron · 2025-10-29T15:25:38Z

pkgs/stdenv/booter.nix

 in
-dfold folder postStage (_: { }) withAllowCustomOverrides
+pkgs.__splicedPackages


This could use a wall of text (or just a couple sentences) that explains

Why we're returning __splicedPackages from this function.

What the various values are, vs. pkgs directly.

emilazy · 2025-11-01T18:31:55Z

Maybe this could use a release note?

I guess it’s a breaking change for people overlaying things into the bootstrap, but OTOH I don’t know if we provide compatibility guarantees for that.

philiptaron · 2025-11-01T19:38:53Z

I would very much shy away from providing anything other than best wishes for that scenario.

emilazy · 2025-11-01T19:44:45Z

Well... actually this is a breaking change for anyone doing nativeBuildInputs = [ pkgs.foo ]; which, while most likely a mistake, does concern me a bit.

wolfgangwalther · 2025-11-03T14:07:09Z

In brought back the additional changes in the following two draft PRs:

These don't work, yet, any help appreciated.

link2xt · 2025-11-16T20:35:00Z

This change broke naersk example of cross-compilation from Linux to Windows according to git bisect:
nix-community/naersk#371

wolfgangwalther mentioned this pull request Oct 27, 2025

splice.nix: make pkgs splicedPackages #349316

Merged

13 tasks

jopejoe1 mentioned this pull request Oct 27, 2025

windows: fix eval of some packages on linux #451244

Merged

13 tasks

szlend reviewed Oct 27, 2025

View reviewed changes

emilazy requested a review from Artturin October 27, 2025 19:48

wolfgangwalther force-pushed the pkgs-spliced branch from 34c2a3e to cb51825 Compare October 28, 2025 13:49

github-project-automation bot added this to Stdenv Oct 28, 2025

wolfgangwalther force-pushed the pkgs-spliced branch from cb51825 to 1093f4a Compare October 28, 2025 15:19

nixpkgs-ci bot added 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. and removed 10.rebuild-linux: 11-100 This PR causes between 11 and 100 packages to rebuild on Linux. 10.rebuild-darwin: 1 This PR causes 1 package to rebuild on Darwin. labels Oct 28, 2025

nixpkgs-ci bot added 6.topic: windows Running, or buiding, packages on Windows 6.topic: lib The Nixpkgs function library labels Oct 28, 2025

wolfgangwalther and others added 4 commits October 28, 2025 21:54

all-packages.nix: remove unused headers

568495b

systems/examples: remove x86_64-unknown-redox

4949737

relibc itself has been removed in 2024 already.

uclibc: bring back alias

1d4ca02

This was removed recently when cleaning up aliases, but it breaks `pkgsCross.ben_nanonote`.

wolfgangwalther force-pushed the pkgs-spliced branch from 1093f4a to 53391c3 Compare October 28, 2025 20:54

wolfgangwalther marked this pull request as ready for review October 28, 2025 20:56

nixpkgs-ci bot added the 10.rebuild-darwin: 1 This PR causes 1 package to rebuild on Darwin. label Oct 28, 2025

philiptaron reviewed Oct 28, 2025

View reviewed changes

pkgs/top-level/splice.nix Show resolved Hide resolved

philiptaron reviewed Oct 29, 2025

View reviewed changes

top-level: return __splicedPackages

f2640ee

wolfgangwalther force-pushed the pkgs-spliced branch from 53391c3 to f2640ee Compare October 31, 2025 19:34

nixpkgs-ci bot added 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 8.has: module (update) This PR changes an existing module in `nixos/` 6.topic: testing Tooling for automated testing of packages and modules labels Oct 31, 2025

philiptaron approved these changes Nov 1, 2025

View reviewed changes

philiptaron added this pull request to the merge queue Nov 1, 2025

Merged via the queue into NixOS:master with commit 21623b7 Nov 1, 2025
31 of 33 checks passed

github-project-automation bot moved this to Done in Stdenv Nov 1, 2025

wolfgangwalther deleted the pkgs-spliced branch November 1, 2025 18:46

This was referenced Nov 3, 2025

lib/customisation: make self spliced in scopes #458108

Closed

all-packages: use pkgs.__splicedPackages scope #458111

Closed

link2xt mentioned this pull request Nov 16, 2025

Windows cross-compilation example does not work with the latest nixpkgs nix-community/naersk#371

Open

wolfgangwalther mentioned this pull request Dec 10, 2025

postgresql: Fix cross #469584

Merged

13 tasks

Uh oh!

Conversation

wolfgangwalther commented Oct 27, 2025

Things done

Uh oh!

szlend Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

emilazy Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

szlend Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

emilazy Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

szlend Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

szlend Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wolfgangwalther Oct 28, 2025

Choose a reason for hiding this comment

Uh oh!

philiptaron commented Oct 27, 2025

Uh oh!

RossSmyth commented Oct 27, 2025

Uh oh!

emilazy commented Oct 27, 2025

Uh oh!

wolfgangwalther commented Oct 27, 2025

Uh oh!

wolfgangwalther commented Oct 27, 2025

Uh oh!

wolfgangwalther commented Oct 28, 2025

Uh oh!

wolfgangwalther commented Oct 28, 2025

Uh oh!

wolfgangwalther commented Oct 28, 2025

Uh oh!

wolfgangwalther commented Oct 28, 2025

Uh oh!

wolfgangwalther commented Oct 28, 2025

Uh oh!

Uh oh!

philiptaron left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

philiptaron Oct 29, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

emilazy commented Nov 1, 2025

Uh oh!

philiptaron commented Nov 1, 2025

Uh oh!

emilazy commented Nov 1, 2025

Uh oh!

wolfgangwalther commented Nov 3, 2025

Uh oh!

link2xt commented Nov 16, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

szlend Oct 27, 2025 •

edited

Loading

emilazy Oct 27, 2025 •

edited

Loading

szlend Oct 27, 2025 •

edited

Loading