Basic Models in TensorFlow
CS 20SI:
TensorFlow for Deep Learning Research
Lecture 3
1/20/2017
1
2
Agenda
Review
Linear regression in TensorFlow
Optimizers
Logistic regression on MNIST
Loss functions
3
Review
4
Computation graph
TensorFlow separates definition of computations from their execution
Phase 1: assemble a graph
Phase 2: use a session to execute operations in the graph.
5
TensorBoard
a = 2
b = 3 useless z
a
x = tf.add(a, b)
y = tf.mul(a, b)
useless = tf.mul(a, x)
a x a y
z = tf.pow(y, x)
b b
with tf.Session() as sess:
Create a FileWriter object to write your
z = sess.run(z)
graph to event files
6
tf.constant and tf.Variable
Constant values are stored in the graph definition
Sessions allocate memory to store variable values
7
tf.placeholder and feed_dict
Feed values into placeholders by dictionary (feed_dict)
You can feed values in variables too
8
Avoid lazy loading
1. Separate the assembling of graph and executing ops
2. Use Python attribute to ensure a function is only loaded the first time it’s
called
9
Go to GitHub
From examples
03_linear_regression_starter.py
03_logistic_regression_mnist_starter.py
From data
Get the file fire_theft.xls
10
Linear Regression
11
Model relationship between a scalar
dependent variable y and independent
variables X
12
The City of Chicago
X: number of incidents of fire
Y: number of incidents of theft
13
Want
X: number of incidents of fire
Y: number of incidents of theft
Predict Y from X
14
Model
w * X + b
(Y - Y_predicted)2
15
Phase 1: Assemble our graph
16
Step 1: Read in data
I already did that for you
17
Step 2: Create placeholders for
inputs and labels
tf.placeholder(dtype, shape=None, name=None)
18
Step 3: Create weight and bias
tf.Variable(initial_value=None, trainable=True, collections=None,
name=None, dtype=None, ...)
19
Step 4: Build model to predict Y
Y_predicted = X * w + b
20
Step 5: Specify loss function
tf.square(Y - Y_predicted, name="loss")
21
Step 6: Create optimizer
tf.train.GradientDescentOptimizer(learning_rate=0.001).minimize(loss)
22
Phase 2: Train our model
Initialize variables
Run optimizer op
(with data fed into placeholders for inputs and labels)
23
See your model in TensorBoard
Step 1: writer = tf.summary.FileWriter('./my_graph/03/linear_reg',
sess.graph)
Step 2: $ tensorboard --logdir='./my_graph'
24
25
Plot the results with matplotlib
Step 1: Uncomment the plotting code at the end of your program
Step 2: Run it again
If run into problem of matplotlib in virtual environment, go to
GitHub/setups and see the file possible setup problems
26
27
ValueError?
28
ValueError?
w, b = sess.run([w, b])
29
How does TensorFlow know what variables
to update?
30
Optimizer
optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.001).minimize(loss)
_, l = sess.run([optimizer, loss], feed_dict={X: x, Y:y})
31
Optimizer
optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.001).minimize(loss)
_, l = sess.run([optimizer, loss], feed_dict={X: x, Y:y})
Session looks at all trainable variables that loss depends on and update them
32
Optimizer
Session looks at all trainable variables that optimizer depends on and update them
33
Trainable variables
tf.Variable(initial_value=None, trainable=True, collections=None,
validate_shape=True, caching_device=None, name=None, variable_def=None, dtype=None,
expected_shape=None, import_scope=None)
34
List of optimizers in TF
tf.train.GradientDescentOptimizer
tf.train.AdagradOptimizer
tf.train.MomentumOptimizer
tf.train.AdamOptimizer
tf.train.ProximalGradientDescentOptimizer
tf.train.ProximalAdagradOptimizer
tf.train.RMSPropOptimizer
And more
35
Discussion question
1. How to know that our model is correct?
2. How to improve our model?
36
How to improve our model
37
Huber loss
Robust to outliers
Intuition: if the difference between the predicted value and the real value is small,
square it
If it’s large, take its absolute value
38
Implementing Huber loss
Can’t write:
if Y - Y_predicted < delta:
39
Huber loss
def huber_loss(labels, predictions, delta=1.0):
residual = tf.abs(predictions - labels)
condition = tf.less(residual, delta)
small_res = 0.5 * tf.square(residual)
large_res = delta * residual - 0.5 * tf.square(delta)
return tf.select(condition, small_res, large_res)
40
Assignment 1
Out midnight today
Due 1/31
Optional Interactive Grading
41
Logistic Regression
42
Then he separated the
light from the darkness
The first logistic
regression model 43
MNIST Database
Each image is a 28x28 array, flattened out to be a 1-d tensor of size 784
44
MNIST
X: image of a handwritten digit
Y: the digit value
45
Want
X: image of a handwritten digit
Y: the digit value
Recognize the digit in the image
46
Model
logits = X * w + b
Y_predicted = softmax(logits)
loss = cross_entropy(Y, Y_predicted)
47
*Y is a one-hot vector
Batch ‘em up
X = tf.placeholder(tf.float32, [batch_size, 784], name="image")
Y = tf.placeholder(tf.float32, [batch_size, 10], name="label")
48
*Y is a one-hot vector
Process data
from tensorflow.examples.tutorials.mnist import input_data
MNIST = input_data.read_data_sets("/data/mnist", one_hot=True)
49
Process data
from tensorflow.examples.tutorials.mnist import input_data
MNIST = input_data.read_data_sets("/data/mnist", one_hot=True)
MNIST.train: 55,000 examples
MNIST.validation: 5,000 examples
MNIST.test: 10,000 examples
50
Phase 1: Assemble our graph
51
Step 2: Create placeholders for
inputs and labels
X = tf.placeholder(tf.float32, [batch_size, 784], name="image")
Y = tf.placeholder(tf.float32, [batch_size, 10], name="label")
52
Step 3: Create weight and bias
tf.Variable(initial_value=None, trainable=True, collections=None,
name=None, dtype=None, ...)
53
Step 4: Build model to predict Y
logits = X * w + b
54
Step 5: Specify loss function
entropy = tf.nn.softmax_cross_entropy_with_logits(logits, Y)
loss = tf.reduce_mean(entropy)
55
Step 6: Create optimizer
tf.train.GradientDescentOptimizer(learning_rate=0.001).minimize(loss)
56
Phase 2: Train our model
Initialize variables
Run optimizer op
(with data fed into placeholders for inputs and labels)
57
Run our model
Average loss epoch 0: 1.28812279526
Average loss epoch 1: 0.732620414598
Average loss epoch 2: 0.600486441648
Average loss epoch 3: 0.53647331619
Average loss epoch 4: 0.497578099683
...
Average loss epoch 9: 0.41295143427
Total time: 8.83596801758 seconds
Optimization Finished!
Accuracy 0.8977
58
TensorBoard it
59
Next class
Structure your model in TensorFlow
Example: word2vec
Feedback: [email protected]
Thanks!
60
