Step1_EvaluateModelon...py: Error when snapshotindex = all #8
Description
To evaluate all models (snapshots of training stages) I set on myconfig.py the variable snapshotindex = all.
There is a variable mistake in the Step1_EvaluateModelonDataset.py file, since the conditioned variable snapindex was not used, but the imported variable snapshotindex was.
Correction:
##################################################
# Compute predictions over images
##################################################
if snapshotindex == -1:
snapindices = [-1]
elif snapshotindex == all:
snapindices = range(len(Snapshots))
else:
print("Invalid choice, only -1 or all!")
for snapindex in snapindices:
cfg['init_weights'] = modelfolder + \
'/train/' + Snapshots[snapindex] # xxx[snapshotindex]
trainingsiterations = (
cfg['init_weights'].split('/')[-1]).split('-')[-1]
scorer = 'DeepCut' + "_resnet" + str(cfg["net_type"]) + "_" + str(
int(trainFraction *
100)) + 'shuffle' + str(shuffle) + '_' + str(
trainingsiterations) + "forTask:" + Task
print("Running ", scorer, trainingsiterations)
try:
Data = pd.read_hdf('Data_h5/' + scorer + '.h5',
'df_with_missing')
print("This net has already been evaluated!")
except:
# Specifying state of model (snapshot / training state)
cfg['init_weights'] = modelfolder + \
'/train/' + Snapshots[snapindex] # xxx[snapshotindex]
sess, inputs, outputs = predict.setup_pose_prediction(cfg)
However, there is still a tf error when running this script in this mode:
Traceback (most recent call last):
File "Step1_EvaluateModelonDataset.py", line 132, in <module>
sess, inputs, outputs = **predict**.setup_pose_prediction(cfg)
...
**_ValueError:** Variable resnet_v1_50/conv1/weights already exists, disallowed. Did you mean to set reuse=True in VarScope? Originally defined at:
File "/tensorflow-3.0/local/lib/python2.7/site-packages/tensorflow/contrib/framework/python/ops/variables.py", line 216, in variable
use_resource=use_resource)
...
To solve this what I did (it worked but not sure if it is the best solution) is to reset the graph for each new iteration of snapshots. By adding tf.reset_default_graph() to the method setup_pose_prediction(cfg) of the file DeepLabCut/pose-tensorflow/nnet/predict.py
def setup_pose_prediction(cfg):
tf.reset_default_graph() # NEW LINE
inputs = tf.placeholder(tf.float32, shape=[cfg.batch_size , None, None, 3])
net_heads = pose_net(cfg).test(inputs)
outputs = [net_heads['part_prob']]
if cfg.location_refinement:
outputs.append(net_heads['locref'])
restorer = tf.train.Saver()
sess = tf.Session()
sess.run(tf.global_variables_initializer())
sess.run(tf.local_variables_initializer())
# Restore variables from disk.
restorer.restore(sess, cfg.init_weights)
return sess, inputs, outputs