Skip to content

Error due to PyTorch when trying to install in development mode #2044

@jonahpearl

Description

@jonahpearl

Is there an existing issue for this?

  • I have searched the existing issues

Bug description

I wanted to be able to do some hacking in the DLC code (related to #2036; specifically, adding an option to pass a manually-chosen list of frames for refinement via cli), so I tried to make a developer install. I removed the deeplabcut requirement from the yaml, made a conda env with everything except deeplabcut, then ran pip install -e ".[tf]" from within the cloned directory to install dlc. (I'm on a CLI so no gui.)This almost worked perfectly, except I got an odd unexpected error. Running import deeplabcut from a python interpreter gave an error about circular imports, but then it succeeds the second time:

2022-11-10 14:02:49.250879: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F AVX512_VNNI FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-11-10 14:02:49.370427: I tensorflow/core/util/util.cc:169] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2022-11-10 14:02:49.404578: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2022-11-10 14:02:50.503183: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /n/app/cuda/11.2/lib64:/n/app/gcc/6.2.0/lib64:/n/app/gcc/6.2.0/lib
2022-11-10 14:02:50.503364: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /n/app/cuda/11.2/lib64:/n/app/gcc/6.2.0/lib64:/n/app/gcc/6.2.0/lib
2022-11-10 14:02:50.503419: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
Loading DLC 2.3rc2...
DLC loaded in light mode; you cannot use any GUI (labeling, relabeling and standalone GUI)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jop9552/datta-lab/DeepLabCut/deeplabcut/__init__.py", line 35, in <module>
    from deeplabcut.create_project import (
  File "/home/jop9552/datta-lab/DeepLabCut/deeplabcut/create_project/__init__.py", line 2, in <module>
    from deeplabcut.create_project.demo_data import load_demo_data
  File "/home/jop9552/datta-lab/DeepLabCut/deeplabcut/create_project/demo_data.py", line 15, in <module>
    from deeplabcut.utils import auxiliaryfunctions
  File "/home/jop9552/datta-lab/DeepLabCut/deeplabcut/utils/__init__.py", line 1, in <module>
    from deeplabcut.utils.auxfun_multianimal import *
  File "/home/jop9552/datta-lab/DeepLabCut/deeplabcut/utils/auxfun_multianimal.py", line 24, in <module>
    from deeplabcut.utils import auxiliaryfunctions, conversioncode
  File "/home/jop9552/datta-lab/DeepLabCut/deeplabcut/utils/auxiliaryfunctions.py", line 21, in <module>
    from deeplabcut.pose_estimation_tensorflow.lib.trackingutils import TRACK_METHODS
  File "/home/jop9552/datta-lab/DeepLabCut/deeplabcut/pose_estimation_tensorflow/__init__.py", line 20, in <module>
    from deeplabcut.pose_estimation_tensorflow.predict_videos import *
  File "/home/jop9552/datta-lab/DeepLabCut/deeplabcut/pose_estimation_tensorflow/predict_videos.py", line 36, in <module>
    from deeplabcut.refine_training_dataset.stitch import stitch_tracklets
  File "/home/jop9552/datta-lab/DeepLabCut/deeplabcut/refine_training_dataset/__init__.py", line 12, in <module>
    from deeplabcut.refine_training_dataset.tracklets import *
  File "/home/jop9552/datta-lab/DeepLabCut/deeplabcut/refine_training_dataset/tracklets.py", line 5, in <module>
    from deeplabcut.post_processing import columnwise_spline_interp
  File "/home/jop9552/datta-lab/DeepLabCut/deeplabcut/post_processing/__init__.py", line 12, in <module>
    from deeplabcut.post_processing.filtering import *
  File "/home/jop9552/datta-lab/DeepLabCut/deeplabcut/post_processing/filtering.py", line 18, in <module>
    from deeplabcut.refine_training_dataset.outlier_frames import FitSARIMAXModel
  File "/home/jop9552/datta-lab/DeepLabCut/deeplabcut/refine_training_dataset/outlier_frames.py", line 24, in <module>
    from deeplabcut.utils import (
  File "/home/jop9552/datta-lab/DeepLabCut/deeplabcut/utils/visualization.py", line 20, in <module>
    from deeplabcut.utils.auxiliaryfunctions import attempttomakefolder
ImportError: cannot import name 'attempttomakefolder' from partially initialized module 'deeplabcut.utils.auxiliaryfunctions' (most likely due to a circular import) (/home/jop9552/datta-lab/DeepLabCut/deeplabcut/utils/auxiliaryfunctions.py)
>>> import deeplabcut
Loading DLC 2.3rc2...
DLC loaded in light mode; you cannot use any GUI (labeling, relabeling and standalone GUI)

After this happened, I made a clean conda env + install of normal DLC (keeping it in the conda yaml) and it worked fine. I noticed the version of DLC differs between the normal and the dev versions: the normal one I just cloned says Loading DLC 2.2.3... vs Loading DLC 2.3rc2... in the development version. But I made both from the same clone which I grabbed just an hour ago! So I'm confused as to how they can have different versions.

Please let me know if there's a better way to run DLC in dev mode so that I can run tests and contribute :)

Operating System

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"

3.10.0-1160.76.1.el7.x86_64

DeepLabCut version

2.2.3 / 2.3rc2

DeepLabCut mode

single animal

Device type

gpu

Steps To Reproduce

YAML file (can't put yamls on gh so here it is as a txt): DEEPLABCUT_dev.txt

"""
conda env create -f DEEPLABCUT_dev.yaml
cd [your dlc clone]
pip install -e ".[tf]"
cd [anywhere other than the dlc directory]
conda activate DEEPLABCUT_dev
python

import deeplabcut
"""

Relevant log output

No response

Anything else?

No response

Code of Conduct

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions