AI VIETNAM

Free AI Course 2020

Tensorflow (Draft)
Deep Learning Framework

Quang-Vinh Dinh
Ph.D. in Computer Science

Year 2020
 AI VIETNAM
Free AI Course 2020

Outline
 Introduction to Tensorflow
 Tensorflow/Keras
 Model Construction – Linear Regression
 Model Construction – Logistic Regression
 Model Construction – Softmax Regression
 Model Training
 Model Saving and Loading
AI VIETNAM
Free AI Course
Introduction
https://en.wikipedia.org/wiki/TensorFlow

Installation Package Declaration

pip install tensorflow import tensorflow as tf

Example 1 Example 2

1
 AI VIETNAM
Free AI Course
Introduction
 Tensor Important attributes
 ~ ndarray in Numpy
 Run on both CPU and GPU
 All tensors are immutable
 Multi-dimensional arrays with a uniform type

Year 2020 2
 AI VIETNAM
Free AI Course
Introduction
 Tensor
 Broadcasting

data

1 2
Axis 0 + 1
3 4

Axis 1

2 3

4 5

Year 2020 3
 AI VIETNAM
Free AI Course
Introduction
 Tensor
 Broadcasting

data

1 2
Axis 0 + 2
3 4

Axis 1

3 4

5 6

Year 2020 4
 AI VIETNAM
Free AI Course
Introduction
 Tensor
 Broadcasting
0 1 2
0
1 2 3
v
1
4 5 6
2 7 8 9
+ 1 0 1

3 10 11 12

Y
0 1 2 0 1 2
0
1 2 3 1 0 1 0
2 2 4
1
4 5 6 1 0 1 1
5 5 7
2 7 8 9
+ 1 0 1
=
2 8 8 10
3 10 11 12 1 0 1 3 11 11 13
Year 2020 5
 AI VIETNAM
Free AI Course
Introduction
 Tensor
 Important functions

Squared Difference

𝑠𝑑 = 𝒙 − 𝒚 2

x sd

1 16

2 9
2
𝒙−𝑦 =
3 4

4 1

Year 2020 6
 AI VIETNAM
Free AI Course
Introduction
 Tensor
 Important functions

random.normal() random.uniform()

Year 2020 7
 AI VIETNAM
Free AI Course
Introduction
 Tensor
 Important functions concat()
tensor_1 tensor_2

1 2 3 4
Axis 0

3 4 5 6

Axis 1

tensor_3

1 2
tensor_4
3 4
1 2 3 4
3 4
3 4 5 6
5 6

Year 2020 8
 AI VIETNAM
Free AI Course
Introduction
 Images in files
 Important functions
argmin()

1 2
7 9 .argmin(axis=0) = 0 0
8 6

1 2
7 9 .argmin(axis=1) = 0 0 1
8 6

Year 2020 9
 AI VIETNAM
Free AI Course
Introduction
 Images in files
argmax()
 Important functions

1 2
7 9 .argmax(axis=0) = 2 1
8 6

1 2
7 9 .argmax(axis=1) = 1 1 0
8 6

Year 2020 10
6
 AI VIETNAM
Free AI Course
Introduction
 Tensor
 Important functions

X v result

1 2 1 5
=
3 4 2 11

v X result

1 2 1 2 7
=
3 4 10

Year 2020 11
 AI VIETNAM
Free AI Course
Introduction
 Tensor
 Important functions

X Y result

1 2 2 3 6 5
=
3 4 2 1 14 13

Y X result

2 3 1 2 11 16
=
2 1 3 4 5 8

Year 2020 12
5
 AI VIETNAM
Free AI Course
Introduction
 Tensor
 Important functions

X Y result

1 2 2 3 6 5
=
3 4 2 1 14 13

Y X result

2 3 1 2 11 16
=
2 1 3 4 5 8

Year 2020 13
5
 AI VIETNAM
Free AI Course
Introduction
 Variable
 Represents a tensor whose value can be changed

Year 2020 14
5
 AI VIETNAM
Free AI Course
Introduction
 Gradient computation

𝑥1

𝑥2
+ 𝑦  .
2

𝑥3

𝑥4

Year 2020 15
4
 AI VIETNAM
Free AI Course
Introduction
 Gradient computation

Year 2020 16
4
 AI VIETNAM
Free AI Course
Introduction
 Gradient computation

Year 2020 17
4
 AI VIETNAM
Free AI Course
Introduction
 Gradient computation

Year 2020 18
4
 AI VIETNAM
Free AI Course
Introduction
 Gradient computation

Year 2020 19
4
 AI VIETNAM
Free AI Course
Introduction
 Gradient computation

Year 2020 20
4
 Gradient computation

Year 2020 21
 Gradient computation

Year 2020 22
 AI VIETNAM
Free AI Course
Introduction
 Gradient computation

Year 2020 23
 AI VIETNAM
Free AI Course
Introduction
 Gradient computation

Year 2020 24
 AI VIETNAM
AI Insight Course
Linear Regression
Initialize
Demo
b=0.04 and Input 𝑥 = 6.7
w=-0.34
Given
sample Model
data Parameters
𝑏 = 0.04 w = -0.34

House price prediction

Label

𝑧 = 𝑥𝑤 + 𝑏 = -2.238 𝑦 = 9.1

Forward Loss
propagation (1) 2
Year 2020 𝑧−𝑦 = 128.55
25
4
 AI VIETNAM
AI Insight Course
Linear Regression
Demo
Input 𝑥 = 0.67 Input 𝑥 = 0.67
Backpropagation Forward
propagation
Model Model
Parameters 𝜂 = 0.01 Parameters
𝑏 = 0.26676 𝑤 = 1.17929 𝑏 = 0.26676 𝑤 = 1.17929

𝑏 = 𝑏 − 𝜂𝐿′ b w = w − 𝜂𝐿′ w 𝑏 = 𝑏 − 𝜂𝐿′ b w = w − 𝜂𝐿′ w

Label Label

𝑧 = 𝑥𝑤 + 𝑏 = -2.238 𝑦 = 9.1 𝑧 = 𝑥𝑤 + 𝑏 = -2.238 𝑦 = 9.1

𝐿′𝑤 = 2𝑥 𝑧 − 𝑦
= −151.9292
𝐿′𝑏 = 2 𝑧 − 𝑦 Loss Giá trị w, b mới Loss
= −22.676 giúp loss giảm
2 2
Year 2020 𝑧−𝑦 = 128.55 𝑧−𝑦 = 0.868
26
4
 AI VIETNAM
Free AI Course
Introduction
 Gradient computation

Year 2020 27
 AI VIETNAM
Free AI Course
Introduction
 Gradient computation

Year 2020 27
4
 AI VIETNAM
Free AI Course
Class in Python
 Create a class

keyword class __init__() function

self: a reference to the current

instance of the class
Year 2020 28
4
 AI VIETNAM
Free AI Course
Class in Python
 __call__ function

Built-in method

Be called like a function

Year 2020 29
4
 AI VIETNAM
Free AI Course 2020

Outline
 Introduction to Tensorflow
 Tensorflow/Keras
 Model Construction – Linear Regression
 Model Construction – Logistic Regression
 Model Construction – Softmax Regression
 Model Training
 Model Saving and Loading
 AI VIETNAM
Free AI Course
Keras
 Run on top of TensorFlow
 Integrated into Tensorflow

Package Declaration

Year 2020 https://en.wikipedia.org/wiki/Keras 30

7
 AI VIETNAM
Free AI Course
Keras
Model Model
𝑥1 𝑥1
𝑧1 𝑦1
𝑥2 𝑥2
𝑧2 Softmax 𝑦2
𝑧
𝑥3 𝑥3
𝑧3 𝑦3
𝑥4
1

1 keras.activations.softmax

keras.Input(shape=(3,)) keras.Input(shape=(4,))

keras.layers.Dense(units=1) keras.layers.Dense(units=3)

Year 2020 31
8
 AI VIETNAM
Free AI Course
Keras
𝑥1
𝑥1 𝑧1
Model 𝑥2
𝑥2 𝑧2
𝑥3
𝑧 𝑧3
… 𝑥3
𝑥4

1 1

keras.Sequential()

Year 2020
 AI VIETNAM
Free AI Course
Keras
Model
keras.Input(shape=(4,))

𝑥1
… 𝑧1 𝑦1
𝑥2
𝑧2 Softmax 𝑦2
𝑥3
keras.Sequential() 𝑧3 𝑦3
𝑥4

1 keras.layers.Activation()
keras.activations.softmax

keras.layers.Dense(units=3)

Year 2020 33
4
 AI VIETNAM
Free AI Course 2020

Outline
 Introduction to Tensorflow
 Tensorflow/Keras
 Model Construction – Linear Regression
 Model Construction – Logistic Regression
 Model Construction – Softmax Regression
 Model Training
 Model Saving and Loading
 AI VIETNAM
Free AI Course
Model Construction
 Linear regression Input layer a layer
area w price
Feature Label

𝑏
1
keras.Input(shape=(1,)) keras.layers.Dense(units=1)

House price data

Model

price = w ∗ 𝑎𝑟𝑒𝑎 + 𝑏
y = 𝑤𝑥 + 𝑏

Year 2020 34
4
 AI VIETNAM
Free AI Course
Model Construction
Input layer a layer
 Linear regression 𝑥1
Features Label
𝑥2

𝑧
𝑥3

Input(shape=(3,)) layers.Dense(units=1)
Advertising-based sale data

Model

Sale = 𝑤1 ∗ 𝑇𝑉 + 𝑤2 ∗ 𝑅𝑎𝑑𝑖𝑜 + 𝑤3 ∗ 𝑁𝑒𝑤𝑠𝑝𝑎𝑝𝑒𝑟 + 𝑏

y = 𝑤1 𝑥1 + 𝑤2 𝑥2 + 𝑤3 𝑥3 + 𝑏

Year 2020 35
4
 AI VIETNAM
Free AI Course
Model Construction
 Linear regression

Features Label

Boston House
Price Data

Model
medv = 𝑤1 ∗ 𝑥1 + ⋯ + 𝑤13 ∗ 𝑥13 + 𝑏

Year 2020 36
4
 AI VIETNAM
Free AI Course
Model Construction
 Linear regression
Model
medv = 𝑤1 ∗ 𝑥1 + ⋯ + 𝑤13 ∗ 𝑥13 + 𝑏

Input layer a layer

𝑥1
…

…
𝑧
𝑥13

Input(shape=(13,)) layers.Dense(units=1)

Year 2020 37
4
 AI VIETNAM
Free AI Course 2020

Outline
 Introduction to Tensorflow
 Tensorflow/Keras
 Model Construction – Linear Regression
 Model Construction – Logistic Regression
 Model Construction – Softmax Regression
 Model Training
 Model Saving and Loading
 AI VIETNAM
Free AI Course
Model Construction
Feature Label  Logistic regression
Model
z = 𝑤𝑥 + 𝑏
1
𝑦ො =
1 + 𝑒 −𝑧

Input layer A layer

𝑥

Sigmoid
𝑧 function
yො

Input(shape=(1,))
layers.Dense(units=1, activation='sigmoid')
38
 AI VIETNAM
Free AI Course
Model Construction
Feature Label  Logistic regression
Model
𝑧 = 𝜽𝑇 𝒙
1
𝑦ො =
1 + 𝑒 −𝑧

Input layer A layer

𝑥1

𝑥2
Sigmoid
𝑧 function
yො

Input(shape=(2,))
layers.Dense(units=1, activation='sigmoid')
39
 AI VIETNAM
Free AI Course
Model Construction
Feature Label

Model  Logistic regression

𝑧 = 𝜽𝑇 𝒙
1
𝑦ො =
1 + 𝑒 −𝑧

Input layer A layer

𝑥1

𝑥2
𝑧 Sigmoid yො
function
𝑥3

𝑥4
layers.Dense(units=1, activation='sigmoid')
1
Input(shape=(4,))
 AI VIETNAM
Free AI Course 2020

Outline
 Introduction to Tensorflow
 Tensorflow/Keras
 Model Construction – Linear Regression
 Model Construction – Logistic Regression
 Model Construction – Softmax Regression
 Model Training
 Model Saving and Loading
 AI VIETNAM
Free AI Course
Model Construction
Feature Label 𝑧1 = 𝑥𝑤1 + 𝑏1  Softmax regression
𝑧2 = 𝑥𝑤2 + 𝑏2
𝑒 𝑧1
yො 1 = 2
σ𝑗=1 𝑒 𝑧𝑗
𝑒 𝑧1
Iris Classification Data yො 2 = 2
σ𝑗=1 𝑒 𝑧𝑗

Input layer A layer Activation layer

𝑥 𝑤1
𝑧11
𝑤 yො 1
𝑏1 Softmax
𝑤2 function
1 𝑧2 yො 2
𝑏2
layers.Activation()
Input(shape=(1,))
activations.softmax
layers.Dense(units=2)
 AI VIETNAM
Free AI Course
Model Construction
Feature Label 𝑧1 = 𝑥𝑤1 + 𝑏1  Softmax regression
𝑧2 = 𝑥𝑤2 + 𝑏2
𝑒 𝑧1
yො 1 = 2
σ𝑗=1 𝑒 𝑧𝑗
𝑒 𝑧1
Iris Classification Data yො 2 = 2
σ𝑗=1 𝑒 𝑧𝑗

Input layer A layer

𝑥 𝑤1
𝑧11
𝑤 yො 1
𝑏1 Softmax
𝑤2 function
1 𝑧2 yො 2
𝑏2

Input(shape=(1,))
layers.Dense(units=2, activation='softmax') 42
4
 AI VIETNAM
Free AI Course
Model Construction
 Softmax regression

Input layer A layer

𝑥
𝑧1 yො 1

𝑧2 Softmax yො 2

1 𝑧3 yො 3

Input(shape=(1,))
layers.Dense(units=3, activation='softmax') 43
4
 AI VIETNAM
Free AI Course Model Construction
 Softmax regression
Forward computation
𝑒𝒛
𝒛 = 𝜽𝑇 𝒙 𝐲ො = 𝑘
σ𝑖=1 𝑒 𝑧𝑖

Input layer A layer

𝑥1
𝑧1 yො 1
𝑥2
𝑧2 Softmax yො 2
𝑥3
𝑧3 yො 3
𝑥4
layers.Dense(units=3, activation='softmax')
1
Input(shape=(4,))
 AI VIETNAM
Free AI Course 2020

Outline
 Introduction to Tensorflow
 Tensorflow/Keras
 Model Construction – Linear Regression
 Model Construction – Logistic Regression
 Model Construction – Softmax Regression
 Model Training
 Model Saving and Loading
AI VIETNAM
Free AI Course
Training
 Logistic regression Computed automatically

Declare optimizer and loss function

 Tính output 𝑦ො
𝑧 = 𝜽𝑇 𝒙
1 model.compile(optimizer='sgd',
𝑦ො = 𝜎(𝑧) =
1 + 𝑒 −𝑧 loss='binary_crossentropy')
 Tính loss (binary cross-entropy)
𝐿(𝜽) = −y T logොy−(1−y)T log(1−ොy ) Start training

 Tính đạo hàm

model.fit(x-data, y-data, batch-size, epochs)
𝐿′𝜽 = x T (ොy − 𝑦)

 Cập nhật tham số (Stochastic

gradient descent) If batch-size=1  Stochastic training

𝜽 = 𝜽 − 𝜂𝐿′𝜽 If batch-size=N  Batch training

If 1<batch-size<N  Mini-batch training
45
AI VIETNAM
Free AI Course
Training
 Softmax regression Computed automatically

Declare optimizer and loss function

 Tính output 𝑦ො
𝑒𝒛
𝒛 = 𝜽𝑇 𝒙 𝐲ො = 𝑘 model.compile(optimizer='sgd',
σ𝑖=1 𝑒 𝑧𝑖
loss='categorical_crossentropy')
 Tính loss (cross-entropy)
𝑘

𝐿(𝜽) = − ෍ δ 𝑖, 𝑦 log𝑦ො𝑖 Start training

𝑖=1
 Tính đạo hàm
model.fit(x-data, y-data, batch-size, epochs)
𝜕𝐿
= 𝒙 𝑦ො𝑖 − δ 𝑖, 𝑦
𝜕𝜽𝑖
If batch-size=1  Stochastic training
 Cập nhật tham số (Stochastic
gradient descent) If batch-size=m  Batch training
𝜽 = 𝜽 − 𝜂𝐿′𝜽 If 1<batch-size<m  Mini-batch training
46
 AI VIETNAM
Free AI Course
Training
 Linear regression
Feature Label

Model

price = w ∗ 𝑎𝑟𝑒𝑎 + 𝑏
y = 𝑤𝑥 + 𝑏

House price data

Year 2020
 AI VIETNAM
Free AI Course
Training
 Logistic regression
Feature Label

Model
z = 𝑤𝑥 + 𝑏
1
𝑦ො =
1 + 𝑒 −𝑧

Year 2020 48
 AI VIETNAM
Free AI Course
Training
 Softmax regression

Model
𝒛 = 𝜽𝑇 𝒙
𝑒𝒛
𝐲ො = 𝑘
σ𝑖=1 𝑒 𝑧𝑖

Year 2020 49
 AI VIETNAM
Free AI Course 2020

Outline
 Introduction to Tensorflow
 Tensorflow/Keras
 Model Construction – Linear Regression
 Model Construction – Logistic Regression
 Model Construction – Softmax Regression
 Model Training
 Model Saving and Loading
AI VIETNAM
Free AI Course
Model Saving and Loading

Model Saving Testing

Model Loading

50
4
AI VIETNAM
Free AI Course
Model Saving and Loading
Model Saving

Model Loading

51
4
 AI VIETNAM
Free AI Course
Tensorflow
 Demo

Year 2020 52
4
 AI VIETNAM
Free AI Course
Reference

Tensor
https://www.tensorflow.org/guide/tensor

TensorFlow 2 quickstart for beginners

https://www.tensorflow.org/tutorials/quickstart/beginner

Save and load models

https://www.tensorflow.org/tutorials/keras/save_and_load

Gradient tape
https://www.tensorflow.org/guide/autodiff

Year 2020 53
4