Skip to content

lib: Discourage use of extend#376033

Merged
roberth merged 2 commits intoNixOS:masterfrom
roberth:stop-hurting-yourself-please
Feb 9, 2025
Merged

lib: Discourage use of extend#376033
roberth merged 2 commits intoNixOS:masterfrom
roberth:stop-hurting-yourself-please

Conversation

@roberth
Copy link
Member

@roberth roberth commented Jan 23, 2025

It creates interoperability issues that can not be reconciled by lib or maintainers of projects that use the Nixpkgs library. Occasionally, end users may be able to solve the problems they run into, but most are not prepared to deal with this set of problems, nor should they be.

Typical conflict:

  • User wants to propagate their own lib, because it has some function they like to use throughout their projects
  • Project maintainer requires the project's lib to be used

No sane language uses a single namespace for combining all the things. (Arguably not even C with its extensive use of prefixing)

Meanwhile, in Nix, all symbols are first class variables. We don't even have the concept of a global top-level namespace to pour everything into. With lib you can try to approximate that, I get the appeal of its apparent simplicity, but since lib can't be global, we just don't even get that apparent simplicity.

I apologize for not offering concrete solutions to this in the text. The manuals are limited to reference documentation. Alternatives - of which we have multiple - are best provided in task-oriented documentation, e.g. nix.dev.

Things done

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
    • sandbox = relaxed
    • sandbox = true
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 25.05 Release Notes (or backporting 24.11 and 25.05 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

Add a 👍 reaction to pull requests you find important.

It creates interoperability issues that can not be reconciled by
`lib` or maintainers of projects that use the Nixpkgs library.
Occasionally, end users may be able to solve the problems they run
into, but most are not prepared to deal with this set of problems,
nor should they be.

Typical conflict:
- User wants to propagate their own lib, because it has some function
  they like to use throughout their projects
- Project maintainer requires the project's lib to be used

No sane language uses a single namespace for combining all the things.
(Arguably not even C with its extensive use of prefixing)

Meanwhile, in Nix, all symbols are first class variables. We don't even
have the concept of a global top-level namespace to pour everything into.
With `lib` you can try to approximate that, I get the appeal of its
apparent simplicity, but since `lib` can't be global, we just don't even
get that apparent simplicity.

I apologize for not offering concrete solutions to this in the text.
The manuals are limited to reference documentation.
Alternatives - of which we have multiple - are best provided in
task-oriented documentation, e.g. nix.dev.
@github-actions github-actions bot added 8.has: documentation This PR adds or changes documentation 6.topic: flakes The experimental Nix feature 6.topic: lib The Nixpkgs function library labels Jan 23, 2025
@roberth roberth requested a review from philiptaron January 23, 2025 07:40
@nix-owners nix-owners bot requested a review from infinisil January 23, 2025 07:42
Copy link
Contributor

@hsjobeki hsjobeki left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm.
I dont know how to display the doc-comment in the repl or on noogle for this yet.

@roberth
Copy link
Member Author

roberth commented Jan 29, 2025

how to display the doc-comment in the repl

Seems to work in the repl, Nix 2.25

nix-repl> :doc lib.extend
Function extend
    … defined at /home/user/h/nixpkgs/lib/default.nix:38:20

    Patch the Nixpkgs library
[...]

I guess for nixdoc we'd have to construct a fake sublibrary to represent the top level or something to make it work? We can't just do lib because that has a copy of a majority of functions.

Co-authored-by: Johannes Kirschbauer <[email protected]>
@roberth roberth force-pushed the stop-hurting-yourself-please branch from 304df2b to 55a11de Compare January 29, 2025 21:13
@github-actions github-actions bot added 10.rebuild-darwin: 1 This PR causes 1 package to rebuild on Darwin. 10.rebuild-darwin: 1-10 This PR causes between 1 and 10 packages to rebuild on Darwin. 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. labels Jan 29, 2025
@roberth roberth merged commit a7c2010 into NixOS:master Feb 9, 2025
29 checks passed
@arianvp
Copy link
Member

arianvp commented Apr 11, 2025

Is there a way to make flake.nix not use a supposed anti-pattern?

@roberth
Copy link
Member Author

roberth commented Apr 12, 2025

lib should have been defined as

lib = {
  lib = import ./lib;
  nixos = ...;
  nixosSystem = ...;
};

Has that ship sailed? Possibly.

Alternatively, we could make nixpkgsFlake.lib.extend "drop" the non-Nixpkgs-library bits by using // instead of extend.
That will still upset some users, but far fewer.

lib = lib // {
  nixos = ...;
  nixosSystem = ...;
};

Maybe add a deprecation warning:

lib = lib.extend (this: old: { nixos = ...; nixosSystem = ...; }) // {
  nixos = ...;
  nixosSystem = ...;
};

Doing the lib.lib thing properly would look like this, and I would guess that over half of all flakes need to be updated for this.
inherit (nixpkgs) lib; -> inherit (nixpkgs.lib) lib;

lib =
  let
    nixpkgsLib = import ./lib;
    warnOnce =
      lib.warnIf
        ( # auto-enables after 24.11 EOL
          lib.oldestSupportedReleaseIsAtLeast 2505)
        # Needs better writing, maybe on nix.dev.
        "The Nixpkgs library is moving into `nixpkgs.lib.lib`. `nixpkgs.lib` is a standard flake output to expose functions and libraries. Nixpkgs exposes its standard library, as well as NixOS functions, such as the `nixos` library and `nixosSystem` function. These remain in `nixpkgs.lib`." null;
  in
    mapAttrs (k: v: builtins.seq warnOnce v) nixpkgsLib // {
      lib = nixpkgsLib;
      nixos = ...;
      nixosSystem = ...;
    }

What do you think?

Before doing any of this, we should have guides for handling dependencies on nix.dev. Things like how do you write a module file that uses bog standard lexical scope stuff instead of fancy tricks like lib.extend or specialArgs.
(hint: importApply or make an anonymous module in flake.nix inject the right option definitions, like { pkgs, ... }: { import = [ ./module.nix ]; services.foo.package = self.packages.${pkgs.stdenv.hostPlatform.system}.foo; })

@arianvp
Copy link
Member

arianvp commented Apr 13, 2025

I think in general the situation currently is quite broken and inconsistent. e.g. a lot of people do not use legacyPackages but import nixpkgs {} to be able to use overlays and they then do not get the extended version of lib anyway.

So I think a breaking change that introduces a lib.lib is worth it. My two cents.

The injecting into the global lib namespace is just ripe with confusion

@arianvp
Copy link
Member

arianvp commented Apr 13, 2025

NixOS/amis@ac17880 is a good example of why I'd love to get rid of all of this. The amount of hoops I need to jump to build a tool for helping release management in nixpkgs due to the oddness of our flake interface is not nice.

roberth added a commit to roberth/nixpkgs that referenced this pull request Apr 13, 2025
A simple preparation for plans discussed in
- NixOS#376033

Crucially, without these changes, the identifier `lib` has too many
meanings and therefore doesn't mean anything, making it unmaintainable.
@roberth roberth mentioned this pull request Apr 13, 2025
13 tasks
@roberth
Copy link
Member Author

roberth commented Apr 13, 2025

NixOS/amis@ac17880 is a good example of why I'd love to get rid of all of this. The amount of hoops I need to jump to build a tool for helping release management in nixpkgs due to the oddness of our flake interface is not nice.

This will then also need changes to lib/flake-version-info.nix. I hope its behavior can be made to match the released channel behavior.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

6.topic: flakes The experimental Nix feature 6.topic: lib The Nixpkgs function library 8.has: documentation This PR adds or changes documentation 10.rebuild-darwin: 1-10 This PR causes between 1 and 10 packages to rebuild on Darwin. 10.rebuild-darwin: 1 This PR causes 1 package to rebuild on Darwin. 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants