Set proper `LD_LIBRARY_PATH` on Linux in nightly venv in nightly pull tool #143262

XuehaiPan · 2024-12-14T21:09:28Z

Stack from ghstack (oldest at bottom):

-> Set proper LD_LIBRARY_PATH on Linux in nightly venv in nightly pull tool #143262

Before this change:

$ make setup-env-cuda PYTHON="${HOMEBREW_PREFIX}/bin/python3.12"
$ source venv/bin/activate
$ python3 -c 'import torch'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/PanXuehai/Projects/pytorch/torch/__init__.py", line 379, in <module>
    from torch._C import *  # noqa: F403
    ^^^^^^^^^^^^^^^^^^^^^^
ImportError: libcudnn.so.9: cannot open shared object file: No such file or directory

This PR adds site-packages/nvidia/**/lib to LD_LIBRARY_PATH in venv/bin/activate script to let NVIDIA PyPI packages can be loaded correctly.

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/143262

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

⏳ No Failures, 50 Pending

As of commit de9d701 with merge base a10b765 ():
💚 Looks good so far! There are no failures yet. 💚

UNSTABLE - The following jobs are marked as unstable, possibly due to flakiness on trunk:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

… tool Before this change: ```console $ make setup-env-cuda PYTHON="${HOMEBREW_PREFIX}/bin/python3.12" $ source venv/bin/activate $ python3 -c 'import torch' Traceback (most recent call last): File "<string>", line 1, in <module> File "/home/PanXuehai/Projects/pytorch/torch/__init__.py", line 379, in <module> from torch._C import * # noqa: F403 ^^^^^^^^^^^^^^^^^^^^^^ ImportError: libcudnn.so.9: cannot open shared object file: No such file or directory ``` This PR add `site-packages/nvidia/**/lib` to `LD_LIBRARY_PATH` to let NVIDIA PyPI packages to be loaded correctly. See also: - pytorch#141837 ghstack-source-id: b4d4b78 Pull Request resolved: pytorch#143262

[ghstack-poisoned]

… tool Before this change: ```console $ make setup-env-cuda PYTHON="${HOMEBREW_PREFIX}/bin/python3.12" $ source venv/bin/activate $ python3 -c 'import torch' Traceback (most recent call last): File "<string>", line 1, in <module> File "/home/PanXuehai/Projects/pytorch/torch/__init__.py", line 379, in <module> from torch._C import * # noqa: F403 ^^^^^^^^^^^^^^^^^^^^^^ ImportError: libcudnn.so.9: cannot open shared object file: No such file or directory ``` This PR add `site-packages/nvidia/**/lib` to `LD_LIBRARY_PATH` to let NVIDIA PyPI packages to be loaded correctly. See also: - pytorch#141837 ghstack-source-id: ba7bdd3 Pull Request resolved: pytorch#143262

[ghstack-poisoned]

… tool Before this change: ```console $ make setup-env-cuda PYTHON="${HOMEBREW_PREFIX}/bin/python3.12" $ source venv/bin/activate $ python3 -c 'import torch' Traceback (most recent call last): File "<string>", line 1, in <module> File "/home/PanXuehai/Projects/pytorch/torch/__init__.py", line 379, in <module> from torch._C import * # noqa: F403 ^^^^^^^^^^^^^^^^^^^^^^ ImportError: libcudnn.so.9: cannot open shared object file: No such file or directory ``` This PR add `site-packages/nvidia/**/lib` to `LD_LIBRARY_PATH` to let NVIDIA PyPI packages to be loaded correctly. See also: - #141837 ghstack-source-id: 4a1b1ab Pull Request resolved: #143262

… tool Before this change: ```console $ make setup-env-cuda PYTHON="${HOMEBREW_PREFIX}/bin/python3.12" $ source venv/bin/activate $ python3 -c 'import torch' Traceback (most recent call last): File "<string>", line 1, in <module> File "/home/PanXuehai/Projects/pytorch/torch/__init__.py", line 379, in <module> from torch._C import * # noqa: F403 ^^^^^^^^^^^^^^^^^^^^^^ ImportError: libcudnn.so.9: cannot open shared object file: No such file or directory ``` This PR add `site-packages/nvidia/**/lib` to `LD_LIBRARY_PATH` to let NVIDIA PyPI packages to be loaded correctly. See also: - pytorch#141837 ghstack-source-id: 4a1b1ab Pull Request resolved: pytorch#143262

albanD · 2024-12-25T21:40:52Z

This shouldn't be needed? My understanding is that we buildin appropriate rpath into our shared libraries so they can find the nvidia packages without any LD_LIBRARY path (which the user will never have locally).

XuehaiPan · 2024-12-26T08:39:26Z

This shouldn't be needed? My understanding is that we buildin appropriate rpath into our shared libraries so they can find the nvidia packages without any LD_LIBRARY path (which the user will never have locally).

For normal installation from PyPI, this is not needed. Because the torch package and nvidia-* dependencies are installed into site-packages. See also:

Invalid RPATH for _C.so for in tree installation: assuption fail "$ORIGIN/../.." -> site-packages #141837

This PR fixes the ld search path for in-tree installation for the nightly development tool. This does not affect most users.

[ghstack-poisoned]

… tool Before this change: ```console $ make setup-env-cuda PYTHON="${HOMEBREW_PREFIX}/bin/python3.12" $ source venv/bin/activate $ python3 -c 'import torch' Traceback (most recent call last): File "<string>", line 1, in <module> File "/home/PanXuehai/Projects/pytorch/torch/__init__.py", line 379, in <module> from torch._C import * # noqa: F403 ^^^^^^^^^^^^^^^^^^^^^^ ImportError: libcudnn.so.9: cannot open shared object file: No such file or directory ``` This PR add `site-packages/nvidia/**/lib` to `LD_LIBRARY_PATH` to let NVIDIA PyPI packages to be loaded correctly. See also: - #141837 ghstack-source-id: a1c9d21 Pull Request resolved: #143262

malfet

Please move typo fixes/refactors to the separate RP.
And changes to the environment variable proposed in this PR feels wrong to me: If I'm not mistaken, PyPI package sometimes missing header files, and this change is not updating system includes search path, is it?
I.e. you are building with one set of header, but than dynamically load other libraries

tools/nightly.py

[ghstack-poisoned]

XuehaiPan · 2024-12-26T21:00:59Z

If I'm not mistaken, PyPI package sometimes missing header files, and this change is not updating system includes search path, is it?

The goal of the nightly tool is to allow contributors to run and test Python code without building from source, if they want to contribute Python code only. Since the nightly installation is an in-tree git repository, all torch headers are in the local directory. The NVIDIA headers are under venv/lib/pythonX.Y/site-packages/nvidia/cu*/include.

github-actions · 2025-03-26T22:36:14Z

Looks like this PR hasn't been updated in a while so we're going to go ahead and mark this as Stale.
Feel free to remove the Stale label if you feel this was a mistake.
If you are unable to remove the Stale label please contact a maintainer in order to do so.
If you want the bot to never mark this PR stale again, add the no-stale label.
Stale pull requests will automatically be closed after 30 days of inactivity.

[ghstack-poisoned]

… tool Before this change: ```console $ make setup-env-cuda PYTHON="${HOMEBREW_PREFIX}/bin/python3.12" $ source venv/bin/activate $ python3 -c 'import torch' Traceback (most recent call last): File "<string>", line 1, in <module> File "/home/PanXuehai/Projects/pytorch/torch/__init__.py", line 379, in <module> from torch._C import * # noqa: F403 ^^^^^^^^^^^^^^^^^^^^^^ ImportError: libcudnn.so.9: cannot open shared object file: No such file or directory ``` This PR add `site-packages/nvidia/**/lib` to `LD_LIBRARY_PATH` to let NVIDIA PyPI packages to be loaded correctly. See also: - #141837 ghstack-source-id: 79e38fd Pull Request resolved: #143262

[ghstack-poisoned]

… tool Before this change: ```console $ make setup-env-cuda PYTHON="${HOMEBREW_PREFIX}/bin/python3.12" $ source venv/bin/activate $ python3 -c 'import torch' Traceback (most recent call last): File "<string>", line 1, in <module> File "/home/PanXuehai/Projects/pytorch/torch/__init__.py", line 379, in <module> from torch._C import * # noqa: F403 ^^^^^^^^^^^^^^^^^^^^^^ ImportError: libcudnn.so.9: cannot open shared object file: No such file or directory ``` This PR add `site-packages/nvidia/**/lib` to `LD_LIBRARY_PATH` to let NVIDIA PyPI packages to be loaded correctly. See also: - #141837 ghstack-source-id: 424d923 Pull Request resolved: #143262

[ghstack-poisoned]

… tool Before this change: ```console $ make setup-env-cuda PYTHON="${HOMEBREW_PREFIX}/bin/python3.12" $ source venv/bin/activate $ python3 -c 'import torch' Traceback (most recent call last): File "<string>", line 1, in <module> File "/home/PanXuehai/Projects/pytorch/torch/__init__.py", line 379, in <module> from torch._C import * # noqa: F403 ^^^^^^^^^^^^^^^^^^^^^^ ImportError: libcudnn.so.9: cannot open shared object file: No such file or directory ``` This PR add `site-packages/nvidia/**/lib` to `LD_LIBRARY_PATH` to let NVIDIA PyPI packages to be loaded correctly. See also: - #141837 ghstack-source-id: e25e05f Pull Request resolved: #143262

malfet

Sure, if lint is green...

malfet · 2025-04-01T16:49:09Z

@pytorchbot merge -f "Lint + MPS are green"

pytorchmergebot · 2025-04-01T16:50:54Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

… tool (pytorch#143262) Before this change: ```console $ make setup-env-cuda PYTHON="${HOMEBREW_PREFIX}/bin/python3.12" $ source venv/bin/activate $ python3 -c 'import torch' Traceback (most recent call last): File "<string>", line 1, in <module> File "/home/PanXuehai/Projects/pytorch/torch/__init__.py", line 379, in <module> from torch._C import * # noqa: F403 ^^^^^^^^^^^^^^^^^^^^^^ ImportError: libcudnn.so.9: cannot open shared object file: No such file or directory ``` This PR adds `site-packages/nvidia/**/lib` to `LD_LIBRARY_PATH` in `venv/bin/activate` script to let NVIDIA PyPI packages can be loaded correctly. See also: - pytorch#141837 Pull Request resolved: pytorch#143262 Approved by: https://github.com/malfet

Update

85f8bd5

[ghstack-poisoned]

pytorch-bot bot added the topic: not user facing topic category label Dec 14, 2024

XuehaiPan mentioned this pull request Dec 14, 2024

[Easy] Bump CUDA nightly version to 11.8 / 12.4 / 12.6 in nightly pull tool #143263

Closed

pytorchbot added the open source label Dec 14, 2024

XuehaiPan requested review from atalman and seemethere December 14, 2024 21:50

Update

ce82ae7

[ghstack-poisoned]

XuehaiPan mentioned this pull request Dec 15, 2024

[Easy] Add ROCm support to nightly pull tool #141282

Closed

Update

acc49f0

[ghstack-poisoned]

XuehaiPan requested review from albanD, ezyang, huydhn and malfet December 24, 2024 11:26

XuehaiPan added the module: devx Related to PyTorch contribution experience (HUD, pytorchbot) label Dec 24, 2024

Update

3d3f682

[ghstack-poisoned]

Update

4fc5bc8

[ghstack-poisoned]

malfet requested changes Dec 26, 2024

View reviewed changes

tools/nightly.py Show resolved Hide resolved

tools/nightly.py Outdated Show resolved Hide resolved

tools/nightly.py Outdated Show resolved Hide resolved

Update

2b8f746

[ghstack-poisoned]

github-actions bot added the Stale label Mar 26, 2025

XuehaiPan added the no-stale label Mar 27, 2025

Update

469cd5a

[ghstack-poisoned]

Update

a1e5010

[ghstack-poisoned]

Update

de9d701

[ghstack-poisoned]

malfet approved these changes Apr 1, 2025

View reviewed changes

pytorchmergebot added the merging label Apr 1, 2025

pytorchmergebot closed this in ae74ef9 Apr 1, 2025

pytorchmergebot added Merged and removed merging labels Apr 1, 2025

github-actions bot deleted the gh/XuehaiPan/211/head branch May 12, 2025 02:20

Tony-Y mentioned this pull request May 26, 2025

ImportError: libcudnn.so.9: cannot open shared object file: No such file or directory #154299

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Set proper `LD_LIBRARY_PATH` on Linux in nightly venv in nightly pull tool #143262

Set proper `LD_LIBRARY_PATH` on Linux in nightly venv in nightly pull tool #143262

Uh oh!

XuehaiPan commented Dec 14, 2024 •

edited

Loading

Uh oh!

pytorch-bot bot commented Dec 14, 2024 •

edited

Loading

Uh oh!

albanD commented Dec 25, 2024 •

edited

Loading

Uh oh!

XuehaiPan commented Dec 26, 2024

Uh oh!

malfet left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

XuehaiPan commented Dec 26, 2024

Uh oh!

github-actions bot commented Mar 26, 2025

Uh oh!

malfet left a comment

Uh oh!

malfet commented Apr 1, 2025

Uh oh!

pytorchmergebot commented Apr 1, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Set proper LD_LIBRARY_PATH on Linux in nightly venv in nightly pull tool #143262

Set proper LD_LIBRARY_PATH on Linux in nightly venv in nightly pull tool #143262

Uh oh!

Conversation

XuehaiPan commented Dec 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Dec 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/143262

⏳ No Failures, 50 Pending

Uh oh!

albanD commented Dec 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

XuehaiPan commented Dec 26, 2024

Uh oh!

malfet left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

XuehaiPan commented Dec 26, 2024

Uh oh!

github-actions bot commented Mar 26, 2025

Uh oh!

malfet left a comment

Choose a reason for hiding this comment

Uh oh!

malfet commented Apr 1, 2025

Uh oh!

pytorchmergebot commented Apr 1, 2025

Merge started

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Set proper `LD_LIBRARY_PATH` on Linux in nightly venv in nightly pull tool #143262

Set proper `LD_LIBRARY_PATH` on Linux in nightly venv in nightly pull tool #143262

XuehaiPan commented Dec 14, 2024 •

edited

Loading

pytorch-bot bot commented Dec 14, 2024 •

edited

Loading

albanD commented Dec 25, 2024 •

edited

Loading

malfet left a comment •

edited

Loading