I'm opening this issue to discuss about the problem of dependency conflicts errors in the python build system of nixpkgs.

The problem is specifically triggered when using lang2nix approaches where versions and sources of packages are generated from a lock file.

The issue

If the lock file introduces a custom version of any python package that is also used as a build tool by the nixpkgs own hooks, like, for example setuptools or packaging, this will likely trigger the following build error:

       > Found duplicated packages in closure for dependency 'packaging':
       > 	packaging 23.2 (/nix/store/l4l6p6x4hxirclz9n0nvgydb2xyl610j-python3.10-packaging-23.2/lib/python3.10/site-packages/packaging-23.2.dist-info)
       > 	packaging 23.2 (/nix/store/lrp3rjxmcb3m4kkkcirlsxqbbd6l963g-python3.10-packaging-23.2/lib/python3.10/site-packages/packaging-23.2.dist-info)

Why no override?

Now you probably wonder, why I don't just override the nixpkgs python fixpoint to ensure there is only one version of packaging. Well, some reasons are:

• there is no guarantee, that the nixpkgs hooks will still work with the overridden dependency
• it is easy to trigger hard to debug problems like infinite recursions when overriding the fixpoint in an automated fashion, due to aliasing issues etc.

When using lang2nix I just want to use the nixpkgs build system as a black box to build arbitrary externally defined packages. I am not interested in any of the nixpkgs python packages itself. Therefore it seems counter intuitive and smells like bad UX that I have to deal with the nixpkgs fix point in order to fix issues.

Optimally the build system itself would never introduce conflicts in the first place.

Are the conflicts real?

The weird thing is that in all cases where I ran into this issue, there haven't actually been any dependency conflicts in the build output.
The pythonCatchConflictsHook doesn't even scan $out for conflicts. Instead it scans$PYTHONPATH for conflicts. The issues here is that $PYTHNOPATH not only contains the installed dependencies but also build time only python dependencies and that's where the conflicts appear.

TL;DR; pythonCatchConflictsHookraises false positive errors in all my cases. There aren't actually any installed conflicts at runtime.

How to fix the issues?

So far I have been trying to fix these issues for individual hooks, like in #254547, but the issue gets re-introduced over time by new hooks added etc. I'd like to implement a more generic fix for this issue.

Ideas:

1. remove pythonCatchConflictsHook in general: Seems like a bad idea, because manually introduced conflicts by the user wouldn't be caught anymore
2. lift conflict detection up to eval time: Each python package could carry meta-data about it's build and runtime dependency versions, which is then compared with other packages at eval time. Sounds clean, but probably much more complex to implement
3. make the hook smarter: Improve the hook to do something that's smarter than scanning PYTHONPATH. Not sure exactly what that would look like. Maybe it can scan the build inputs somehow. Not sure how complex this would be due to input propagation.

I would like to fix this and am looking for some input.
cc @FRidh @mweinelt

reproducer

let
  pkgs = import (builtins.fetchTarball "https://github.com/nixos/nixpkgs/tarball/nixos-unstable") {};
  py = pkgs.python3.pkgs;
  # a custom version of 'packaging'
  packaging = py.buildPythonPackage rec {
    pname = "packaging";
    version = "23.2";
    src = pkgs.fetchPypi {
      inherit pname version;
      hash = "sha256-BI+w6UBQNlGOqvSKVZU8dQwR4aG2jg3RqdYu0MCSz8U=";
    };
    pyproject = true;
    buildInputs = [
      py.flit-core
    ];
  };
  # another package that depends on the custom 'packaging'
  pyprojectToml = builtins.toFile "pyproject.toml" ''
    [project]
    name = "my-project"
    version = "1.0.0"
  '';
  # the source of the example project
  projectSource = pkgs.runCommand "my-project-source" {} ''
    mkdir -p $out/src
    cp ${pyprojectToml} $out/pyproject.toml
    touch $out/src/__init__.py
  '';
in
  # a python package that depends on a custom version of packaging
  py.buildPythonPackage {
    pname = "my-project";
    version = "1.0.0";
    src = projectSource;
    pyproject = true;
    buildInputs = [
      py.setuptools
    ];
    propagatedBuildInputs = [
      packaging
    ];
  }

Add a 👍 reaction to issues you find important.