CUDA-GDB to debug Python TensorFlow scripts

Developer Tools CUDA Developer Tools CUDA-GDB
1

Hello! I have a debug build of TensorFlow built with symbols for Cuda as well as Tensorflow. My system has 2 Nvidia Quadro M2000s.
I am trying to debug a Python script I made, using Cuda GDB, however I keep getting the error :
fatal: All CUDA devices are used for display and cannot be used while debugging. (error code = CUDBG_ERROR_ALL_DEVICES_WATCHDOGGED(0x18)

or: Attaching to process running on watchdogged GPU is not possible. Please repeat the attempt in console mode or restart the process with CUDA_VISIBLE_DEVICES environment variable set.

For the first error I am simply running my program like:
cuda-gdb --args python3.6 script.py
r

The second error I get when I use PyCharm, set a breakpoint at a line, print the process ID then attach cuda-gdb to that process.

I tried following some of the advice online such as setting software preemption mode etc, but to no avail. New to cuda/cuda-gdb so sorry if I missed something obvious.

Thanks!

2

Hi @davido,

Please try setting the interactive option for false for your GPUs: CUDA Visual Profiler 'Interactive' X config option? - Stack Overflow

Also, can you provide more details about how you enabled the SW preemption?

And the output of the following commands:

nvidia-smi
env | grep CUDA_ | sort
3

Thank you this helped, also changing environment variables in the PyCharm runtime config was necessary.