This is a meta tracking issue to aggregate & track issues & PR related to Python build time & runtime separation.

Related issues

• Python: build wheels in separate derivation
• Python: run tests in separate derivation

Motivations

Stop propagation of Python dependencies from applications

When depending on applications that are written in Python in a nix-shell Python dependencies are propagated into $PYTHONPATH.
This is very problematic when developing Python applications, and a number of packages (notably Poetry & PDM) have hacks that remove $out/nix-support/propagated-build-inputs.
Those hacks could be entirely removed.

Reducing the number of rebuilds

By not using build time propagation we don't even need to add runtime dependencies to the build.
This will drastically cut down on the amount of rebuilds we have to do.

Reducing evaluation overhead

Both through smaller build time closures, but also through possibly getting rid of the packageOverrides idiom (at least internally in nixpkgs).
This idiom is problematic because it means we rebootstrap the Python package set every time it's used.

By delaying composition we can even allow for things which today are not possible because they would lead to file collisions:

python3Packages.buildPythonApplication {
  dependencies = [
    (python3Packages.requests.overridePythonAttrs(old: {
      # This can actually work when we don't enforce propagation at build time!
      # `requiredPythonModules` just needs to take the first dependency for a given name from the list.
      # By putting something earlier in the `dependencies` list they get higher precedence.
    }))
  ];
}

As per the table in my comment a nixpkgs evaluation allocates an attrset with 26755 members 2491 times.
I hunted this down and this is variants of pythonPackages being instantiated.
That's only the tip of the iceberg.

Circular dependencies

In Nix, build time propagation requires the build graph to be a DAG.
This is not true for Python dependencies which sometimes has circular dependencies.

By only depending on the minimal build time dependencies we can reduce the risk of build-time cycles happening to almost zero.
Ensuring no infinite recursion happens would probably be the job of requiredPythonModules.

Expected breakage

I've tried to keep breakage to a minimum, but something has to give for such a drastic semantic change.
The good thing is that most of these issues can be fixed ahead of merging the breaking PR.

python3Packages.foo

Depending on a Python package like this will break:

stdenv.mkDerivation {
  propagatedBuildInputs = [ python3Packages.requests ];
}

You will instead need to:

stdenv.mkDerivation {
  propagatedBuildInputs = python3Packages.requiredPythonModules [ python3Packages.requests ];
}

This is because dependencies are no longer build-time propagated, so the full dependency graph calculation needs to happen in the Nix evaluator.

Note that you don't have to call requiredPythonModules manually when using buildPythonPackage/buildPythonApplication.
This quirk only applies when depending on pythonPackages in non-python contexts.
There are a handful of those cases in nixpkgs but I suspect the most common case is in external development shells:

pkgs.mkShell {
  packages = [ 
    python3
    python3Packages.requests
  ];
}

In those cases you'll need to either call requiredPythonModules or use withPackages/buildEnv.

python3Packages.buildPythonPackage vs python3Packages.buildPythonApplication

In #272179 the distinction between these two functions becomes much more firm:

• buildPythonPackage is for building a Python package without dependency propagation
• buildPythonApplication is for building application bundles. This internally calls buildPythonPackage, and also creates a wrapper derivation that takes care of calling requiredPythonModules.

All Python applications which has runnable binaries now needs to either be:

• Created using buildPythonApplication
• Converted from a Python module using toPythonApplication

Pull requests

I'm trying my best to allow us to work towards this goal without breaking things unnecessarily.
The first two pull requests in this list doesn't break any current behaviour.
Those allow us to work towards the goal of splitting dependencies incrementally.

The third PR introduces breaking behaviour, but manages to stay mostly compatible, with the breaking changes documented above being the notable exceptions.

• buildPythonPackage: Separate runtime & build time dependencies
  Introduces argument separation between build, runtime & optional dependencies to buildPythonPackage.
  This is a requisite change to start

• buildPythonPackage: Add support for running tests in a separate derivation
  Introduces a new argument to buildPythonPackage called separateChecks.
  A passthru.tests.python attribute is added where the tests are run separately from building the package.

• WIP: Python no runtime deps at build time
  Removes propagation and introduces the previously mentioned distinction between buildPythonPackage & buildPythonApplication.

Add a 👍 reaction to issues you find important.

cc @FRidh @mweinelt
cc @DavHau @phaer