-
Notifications
You must be signed in to change notification settings - Fork 75.2k
Closed
Description
Problem description
This issue happens randomly. For same machine, same CPU,works randomly. I run it for 10 times, 6 times succeed, 4 times fail. Same code is running for every time. The code is
import tensorflow as tf
# with tf.device('/cpu:0'):
a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')
b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2], name='b')
c = tf.matmul(a, b)
with tf.Session(config=tf.ConfigProto(log_device_placement=True)) as sess:
print sess.run(c)
System information
Information of cluster:
Basically our cluster is made of IBM Power System S822LC for commercial computing
OS: Red Hat Enterprise Linux Server release 7.2 (Maipo)
TF version: 0.10.0rc0
protobuf: 3.0.0b2
Python: 2.7.5
Error
>>> import os
>>> cwd = os.getcwd()
>>> cwd
'/gpfs/gpfs0/groups/duraisamy/shawnpan/test/script'
>>> import tensorflow
import tensorflow
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcublas.so.8.0 locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcudnn.so.5.1.5 locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcufft.so.8.0 locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcurand.so.8.0 locally
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/gpfs/gpfs0/software/rhel72/packages/tensorflow_gpu/tensorflow/__init__.py", line 23, in <module>
from tensorflow.python import *
File "/gpfs/gpfs0/software/rhel72/packages/tensorflow_gpu/tensorflow/python/__init__.py", line 58, in <module>
raise ImportError(msg)
ImportError: Traceback (most recent call last):
File "/gpfs/gpfs0/software/rhel72/packages/tensorflow_gpu/tensorflow/python/__init__.py", line 52, in <module>
from tensorflow.core.framework.graph_pb2 import *
File "/gpfs/gpfs0/software/rhel72/packages/tensorflow_gpu/tensorflow/core/framework/graph_pb2.py", line 6, in <module>
from google.protobuf import descriptor as _descriptor
ImportError: No module named google.protobuf
Error importing tensorflow. Unless you are using bazel,
you should not try to import tensorflow from its source directory;
please exit the tensorflow source tree, and relaunch your python interpreter
from there.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels