@zeuner unroll-src fails with:

nixpkgs on  update/python3Packages.torch took 3s
❮ ./result 2.8.0 > pkgs/development/python-modules/torch/source/src.nix
Loading required package: processx
Loading required package: jsonlite
Loading required package: stringr
Loading required package: plyr
Loading required package: dplyr

Attaching package: ‘dplyr’

The following objects are masked from ‘package:plyr’:

    arrange, count, desc, failwith, id, mutate, rename, summarise,
    summarize

The following objects are masked from ‘package:stats’:

    filter, lag

The following objects are masked from ‘package:base’:

    intersect, setdiff, setequal, union

Error in `run("bash", c("-c", paste(c(paste0("cd ", fetched), paste0("git checkout ", …`:
! System command 'bash' failed
---
Exit status: 128
Stderr:
fatal: unable to read tree (3ed8d2ec4ba35ef5d9d8353826209b6f868f63d3)
---
Backtrace:
1. global clone_missing()
2. processx::run("bash", c("-c", paste(c(paste0("cd ", fetched), paste0("git checkout "…
3. processx:::throw(new_process_error(res, call = sys.call(), echo = echo, …
Execution halted

Any idea?

Any idea?

It fails fetching the submodule external/cutlass of https://github.com/pytorch/fbgemm. The branch FBGEMM of https://github.com/jwfromm/cutlass (referenced from .gitmodules) does not provide the 3ed8d2ec4ba35ef5d9d8353826209b6f868f63d3 commit (as expected by git submodule status).

Looking further for a good way to mitigate this.

Any idea?

It fails fetching the submodule external/cutlass of https://github.com/pytorch/fbgemm. The branch FBGEMM of https://github.com/jwfromm/cutlass (referenced from .gitmodules) does not provide the 3ed8d2ec4ba35ef5d9d8353826209b6f868f63d3 commit (as expected by git submodule status).

Looking further for a good way to mitigate this.

Thanks a lot!

Hello, I don't know if this will help much, but I believe I have a working setup for pytorch 2.8.0 with ROCm.

https://github.com/benjaminhottell/nix-rocm-pytorch

I don't think all of the things I did were strictly necessary. But maybe some of the patches will help you. Feel free to take as little or as much from that repo as you wish :)

Hello, I don't know if this will help much, but I believe I have a working setup for pytorch 2.8.0 with ROCm.

https://github.com/benjaminhottell/nix-rocm-pytorch

I don't think all of the things I did were strictly necessary. But maybe some of the patches will help you. Feel free to take as little or as much from that repo as you wish :)

Thanks for proposing.
I'm currently block on a setuptools version issue:

ERROR Missing dependencies:
        setuptools<80.0,>=62.3.0

Hi, I got around this in my derivation by intentionally downloading setuptools 79.0.1 as you can see here: https://github.com/benjaminhottell/nix-rocm-pytorch/blob/main/derivations/setuptools-79_0_1.nix

I then set up a special python for pytorch's build process with that specific setuptools included:

{ pkgs ? import <nixpkgs> {} }:
let
  # Version specified in requirements as: <80.0
  setuptools = pkgs.callPackage ./setuptools-79_0_1.nix { inherit pkgs; };
 };

  # The python instance with pytorch's dependencies installed
  python-build = (pkgs.python312.withPackages(pypkgs: [
    
    # ...

    # Discovered by running and letting it crash
    setuptools

    # ...
  ]));

in
  python-build

This is because pytorch only supports (to my knowledge) versions under version 80.0.0

I believe the version of setuptools that you get from python3Packages.setuptools is greater than 80.0.0

nix-shell -p python3Packages.setuptools
python3 -c 'import setuptools; print(setuptools.__version__)'

Skimming through your PR, I believe you are currently patching out the requirements to make a version greater than 80.0.0 work anyway.

    substituteInPlace pyproject.toml \
      --replace-fail "setuptools>=62.3.0,<80.0" "setuptools"

I have not tried this, and I am unsure if that will work. In my experience the setup.py script is rather smart and you may need to patch more than the pyproject.toml entry.

Now failing during compilation (buildPhase) with:

^M[2151/7093] Building CXX object c10/CMakeFiles/c10.dir/core/Device.cpp.o^M[2152/7093] Building CXX object c10/CMakeFiles/c10.dir/core/ConstantSymNodeImpl.cpp.o^M[2152/7093] Building CXX object c10/CMakeFiles/c1>
FAILED: [code=1] /build/pytorch/torch/utils/data/datapipes/datapipe.pyi
cd /build/pytorch && /nix/store/0vnarm4qjnj16dr3zj9kwq6bn79c0icn-cmake-3.31.7/bin/cmake -E env PYTHONPATH="/build/pytorch/torch/.." /nix/store/iyff8129iampdw13nlfqalzhxy8y1hi9-python3-3.13.6/bin/python3.13 /build>
Traceback (most recent call last):
  File "/build/pytorch/torch/utils/data/datapipes/gen_pyi.py", line 6, in <module>
    from typing_extensions import deprecated
ModuleNotFoundError: No module named 'typing_extensions'
^M[2297/7093] Building C object sleef/src/libm/CMakeFiles/sleefdetsse2.dir/sleefsimddp.c.o^M[2298/7093] Building C object sleef/src/libm/CMakeFiles/sleefsse2.dir/sleefsimddp.c.o^M[2299/7093] Regenerating version >
ninja: build stopped: subcommand failed.

It's weird because typing-extensions is in dependencies...

Probably introduced in pytorch/pytorch@9b2a45a.

PYTHONPATH= override in the cmake doing it?

This should work:

    # path override breaks finding site packages
    substituteInPlace torch/CMakeLists.txt \
      --replace-fail 'env PYTHONPATH=' 'env PYTHONPATH_unused='

ROCm build still needs some work. Some substitutes no longer apply. LoadHIP changed and is crashing finding rocm-core. Can probably comment out find_package(rocm-core CONFIG) to workaround, may become unneeded after ROCm 6.4 bump.

Now failing with:

CMake Error: The source directory "/nix/store/iyff8129iampdw13nlfqalzhxy8y1hi9-python3-3.13.6/bin/python3.13" is a file, not a directory.

Review run looks much better now. torchWithRocm fails for 3.12 importing setuptools but is fine for 3.13.

       > Running phase: buildPhase
       > Executing pypaBuildPhase
       > Traceback (most recent call last):
       >   File "/build/pytorch/setup.py", line 263, in <module>
       >     import setuptools.command.build_ext
       > ModuleNotFoundError: No module named 'setuptools'

Does that make any sense?

Seems like rocmPackages.clr in buildInputs messes up the python 3.12 env. :/

Should we just mark broken if python version == 3.12 + rocmSupport enabled? That's probably an issue in rocmPackages.clr and I can try to address it separately.

Should we just mark broken if python version == 3.12 + rocmSupport enabled? That's probably an issue in rocmPackages.clr and I can try to address it separately.

Sure, fine by me.
@prusnak any further requests before merging?

Seems like rocmPackages.clr in buildInputs messes up the python 3.12 env. :/

Why exactly?

Why exactly?

@NickCao I think what's happening is that clr propagates rocminfo which propagates python3 which is ending up as python3.13 because rocmPackages aren't part of the python package set.

rocminfo probably needs this dropped: https://github.com/NixOS/nixpkgs/blame/2a54616c742ed35e28b1e599b528c4ead690af7f/pkgs/development/rocm-modules/6/rocminfo/default.nix#L39 - but might need more patching on top of that and will trigger rebuilds of the entire ROCm package set which includes CK and takes 5ever to review run.

Could try to fix it in this PR but the root cause been messed up for 3 years and is only breaking 3.12 torchWithRocm so I think addressing it in a followup PR is fine.

only breaking 3.12 torchWithRocm so I think addressing it in a followup PR is fine.

Breaking all non default python versions? Mark all of them as broken then.

Do you have a suggestion for the specific broken = expression?

rocminfo python leakage fix PR: #438399

only breaking 3.12 torchWithRocm so I think addressing it in a followup PR is fine.

Breaking all non default python versions? Mark all of them as broken then.

I managed to mark it as broken on non-default python versions.

nixpkgs-review result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 431973
Commit: ab2d7e40168939df18c44a68f3d59098a24ccb56

x86_64-linux

WARNING: Some logs were not included in this report: there were too many.

nixpkgs-review result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 431973 --package python3Packages.torchWithCuda --package python312Packages.torchWithCuda
Commit: ab2d7e40168939df18c44a68f3d59098a24ccb56

x86_64-linux

Thanks everyone for your help!