sci-libs/pytorch: add PyTorch 1.10.1#1123
Closed
Miezhiko wants to merge 10 commits intogentoo:masterfrom
Closed
Conversation
Contributor
|
Thank you @Miezhiko ! @littlewu2508 would you like to give a review? |
Contributor
|
@Miezhiko Can we use the system pybind headers and libs for compilation rather than using the third_party (stay the same as pytorch-1.8,1.9)? |
Contributor
Author
|
@littlewu2508 don't know, in this I'm just removing third party pylib in |
Contributor
Author
|
but you are right, I will try this: |
Contributor
Author
|
@littlewu2508 updated |
Contributor
|
In This error also occurs in the 1.9.0 version. Have you encountered a similar issue? |
Contributor
Author
|
mhm, seems like building with |
Contributor
Leave this to me. I've got a solution. I'll also update some patches for the ROCm. |
Contributor
Author
|
Okay, cool with me! |
| rm -rfv "${ED}/usr/${LIB}/cmake" | ||
|
|
||
| if use python; then | ||
| scanelf -r --fix "${BUILD_DIR}/caffe2/python" |
| python_foreach_impl python_optimize | ||
| fi | ||
|
|
||
| find "${ED}/usr/${LIB}" -name "*.a" -exec rm -fv {} \; |
|
|
||
| find "${ED}/usr/${LIB}" -name "*.a" -exec rm -fv {} \; | ||
|
|
||
| use test && rm -rfv "${ED}/usr/test" "${ED}"/usr/bin/test_{api,jit} |
Member
There was a problem hiding this comment.
Suggested change
| use test && rm -rfv "${ED}/usr/test" "${ED}"/usr/bin/test_{api,jit} | |
| if use test; then | |
| rm -r "${ED}/usr/test" "${ED}"/usr/bin/test_{api,jit} || die | |
| fi |
| cp -rv "${WORKDIR}/${P}/third_party/pybind11/include/pybind11" "${ED}/usr/include/" | ||
|
|
||
| rm -fv "${ED}/usr/${LIB}/libtbb.so" | ||
| rm -rfv "${ED}/usr/${LIB}/cmake" |
Member
There was a problem hiding this comment.
Suggested change
| rm -rfv "${ED}/usr/${LIB}/cmake" | |
| rm -r "${ED}/usr/${LIB}/cmake" || die |
|
|
||
| cp -rv "${WORKDIR}/${P}/third_party/pybind11/include/pybind11" "${ED}/usr/include/" | ||
|
|
||
| rm -fv "${ED}/usr/${LIB}/libtbb.so" |
Member
There was a problem hiding this comment.
Suggested change
| rm -fv "${ED}/usr/${LIB}/libtbb.so" | |
| rm -v "${ED}/usr/${LIB}/libtbb.so" || die |
Member
There was a problem hiding this comment.
And in the other lines above too.
If external commands are called we need to call die if they fail to stop the ebuild there. Also drop the -f argument, as this causes rm to exit successfully even if the file does not exist and therefore interferes with || die
| KEYWORDS="~x86 ~amd64" | ||
|
|
||
| IUSE="asan blas cuda +fbgemm ffmpeg gflags glog +gloo leveldb lmdb mkldnn mpi namedtensor +nnpack numa +observers opencl opencv +openmp +python +qnnpack redis rocm static test tools zeromq" | ||
| RESTRICT="!test? ( test )" |
Member
There was a problem hiding this comment.
Suggested change
| RESTRICT="!test? ( test )" |
| SLOT="0" | ||
| KEYWORDS="~x86 ~amd64" | ||
|
|
||
| IUSE="asan blas cuda +fbgemm ffmpeg gflags glog +gloo leveldb lmdb mkldnn mpi namedtensor +nnpack numa +observers opencl opencv +openmp +python +qnnpack redis rocm static test tools zeromq" |
Member
There was a problem hiding this comment.
Suggested change
| IUSE="asan blas cuda +fbgemm ffmpeg gflags glog +gloo leveldb lmdb mkldnn mpi namedtensor +nnpack numa +observers opencl opencv +openmp +python +qnnpack redis rocm static test tools zeromq" | |
| IUSE="asan blas cuda +fbgemm ffmpeg gflags glog +gloo leveldb lmdb mkldnn mpi namedtensor +nnpack numa +observers opencl opencv +openmp +python +qnnpack redis rocm static tools zeromq" |
|
|
||
| LICENSE="BSD" | ||
| SLOT="0" | ||
| KEYWORDS="~x86 ~amd64" |
Member
There was a problem hiding this comment.
Please sort the keywords, it helps with some tools if they are in the expected order. And also, repoman will complain if they are not alphabetically.
Suggested change
| KEYWORDS="~x86 ~amd64" | |
| KEYWORDS="~amd64 ~x86" |
| # Copyright 1999-2021 Gentoo Authors | ||
| # Distributed under the terms of the GNU General Public License v2 | ||
|
|
||
| EAPI=7 |
Member
There was a problem hiding this comment.
Can we bump this to EAPI=8 while we are touching this package anyway?
|
|
||
| EAPI=7 | ||
|
|
||
| PYTHON_COMPAT=( python3_{7..10} ) |
Member
There was a problem hiding this comment.
Suggested change
| PYTHON_COMPAT=( python3_{7..10} ) | |
| PYTHON_COMPAT=( python3_{8..10} ) |
Contributor
Author
|
@AndrewAmmerlaan thank you for review but first I'd wait for python bindings fix from @littlewu2508 |
Contributor
Hi, do you have a PR somewhere for this? |
Contributor
I have some local changes that fixes python bindings and rocm related issues but not ready for PR yet. Recently all the rocm dependencies are finally merged into portage tree, so I suggest we can land pytorch into ::gentoo.
@Miezhiko You can refine your ebuild according to the review comments anyway, then merge it into master. I'll open another PR to sci to fix python bindings and rocm-4.3 support. After that let's consider landing it to ::gentoo |
Contributor
Meanwhile, can open a PR for pytorch-1.9.1 to replace pytorch-1.9.0, which is also broken in python and rocm. |
Contributor
|
@littlewu2508 If you are ready with the Python binding fix, just open a new PR based on this one. Just donot forget to acknownledge @Miezhiko. |
Member
|
I'm getting a compile failure (happens both with default flags, and with all disabled): |
Signed-off-by: Miezhiko <[email protected]>
Signed-off-by: Miezhiko <[email protected]>
Signed-off-by: Miezhiko <[email protected]>
Miezhiko
changed the title
improve ebuild quality + switch to EAPI 8 Signed-off-by: Miezhiko <[email protected]>
Contributor
Author
|
@AndrewAmmerlaan updated ebuild, but it still has same problems with python and maybe some rocm patches? what @littlewu2508 said |
Signed-off-by: Miezhiko <[email protected]>
Contributor
Author
|
(fixed breakpad problem too) |
Signed-off-by: Miezhiko <[email protected]>
Contributor
Thanks for fixing breakpad. I've compiling with rocm support, and if everything goes well I'll update the changes. |
Signed-off-by: Miezhiko <[email protected]>
Contributor
Author
|
@littlewu2508 please pull small fix for |
Contributor
Author
|
Also trying to update breakpad on upstream |
Other changes: 1. Pytorch seems to be work fine with different version of protobuf, so no need to specify 0/30; however when protobuf version changed torch should be rebuilt. 2. BUILD_NAMEDTENSOR is not an option now 3. Verbose mode for rm in src_install Package-Manager: Portage-3.0.22, Repoman-3.0.3 Signed-off-by: Yiyang Wu <[email protected]>
Contributor
|
Masha#1 I create a PR to your branch for pushing python binding and rocm support |
sci-libs/pytorch: fix python and update rocm support
Contributor
Author
|
@littlewu2508 thank you, merged. |
Member
|
With USE="python" I get this error: And with USE="opencl" I get this error: |
Signed-off-by: Miezhiko <[email protected]>
Member
|
Thanks @Miezhiko and @littlewu2508 👍 USE="opencl" or USE="python" is not quite working yet, but I merged it anyway since it is such a huge improvement over the in-tree ebuild already (since those don't build at all). We can fix the python/opencl stuff in a separate PR |
Contributor
Author
|
@AndrewAmmerlaan thank you, but I haven't finished yet my own testing after PR merge, still building commit |
Contributor
I can't reproduce this. Maybe you can provide more information? It seems like pytorch tries to search a third party dir.
I think dev-util/opencl-headers is missing. Should add some dependencies |
Member
I have that installed, so that is not it.
It is probably related to either the python version or the setuptools version, I'm compiling with python3.10 and setuptools-59.8.0. |
Member
According to the portage file list there are several packages providing
It seems that the That specific line fails if I just |
Contributor
I*'m using python3.8 and setuptools-57.4.0-r2. Also, I noticed the that Let me try upgrading distutils and see if pytorch breaks. |
Contributor
So it's true only with |
littlewu2508
added a commit
to littlewu2508/sci
that referenced
this pull request
Dec 25, 2021
Backport an upstream PR: pytorch/pytorch#69904 Bug: gentoo#1123 (comment) Package-Manager: Portage-3.0.22, Repoman-3.0.3 Signed-off-by: Yiyang Wu <[email protected]>
gentoo-bot
pushed a commit
that referenced
this pull request
Dec 26, 2021
Reference: pytorch/pytorch@0776756 Reference: #1123 (comment) Package-Manager: Portage-3.0.22, Repoman-3.0.3 Closes: #1130 Signed-off-by: Yiyang Wu <[email protected]> Signed-off-by: Benda Xu <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://pytorch.org/blog/pytorch-1.10-released/