CS490 ̶ Advanced Topics in Computing
(Deep Learning)
Lecture 16: Convolutional Neural Networks (CNNs)
Dr. Muhammad Shahzad
[email protected]Department Of Computing (DOC),
School of Electrical Engineering & Computer Science (SEECS),
National University of Sciences & Technology (NUST)
12/04/2021
Fully Connected Layer
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 2
Motivation: Deep Learning on Images
How many entries does
the weight matrix 𝑤 1
has assuming that the
12288-dimensional first hidden layer have
input vector 1000 units?
64 x 64 x 3 3 Billion!
Shape of 𝒘𝟏 is 1000 x 3M
i.e., adding 1000 biases,
we need to train more
1000 x 1000 x 3 3 Million-dimensional input than 3 Billion parameters
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 3
Convolutional Neural Networks
▪ Similar to regular Neural Networks except that they make the
explicit assumption that the inputs are images, which allows us to
encode certain properties into the architecture
▪ These then make the forward function more efficient to implement
and vastly reduce the amount of parameters in the network, e.g.,
using local receptive field and parameter sharing scheme
A ConvNet is made up of Layers
Every Layer has a simple API: It transforms an input 3D volume to an output 3D
volume with some differentiable function that may or may not have parameteras
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 4
Layers used to build ConvNets
▪ A ConvNet architecture is in the simplest case a list of Layers that
transform the image volume into an output volume (e.g. holding
the class scores)
▪ Three main types of layers that are stacked to build ConvNet
architectures:
► Convolutional Layer
► Pooling Layer
► Fully-Connected Layer (exactly as seen in regular Neural
Networks)
▪ Each Layer accepts an input 3D volume and transforms it to an
output 3D volume through a differentiable function
▪ Each Layer may or may not have parameters (e.g. CONV/FC do,
RELU/POOL don’t)
▪ Each Layer may or may not have additional hyperparameters (e.g.
CONV/FC/POOL do, RELU doesn’t)
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 5
How does Convolution work?
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 6
Edge Detection Via Convolution Operation
3x1 + 1x1 + 2x1 + 0x0 + 5x0 + 7x0 + 1x(-1) + 8x(-1) + 2x(-1) = -5
-5
1 0 -1
1 0 -1
1 0 -1
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 7
Edge Detection Via Convolution Operation
0x1 + 5x1 + 7x1 + 1x0 + 8x0 + 2x0 + 2x(-1) + 9x(-1) + 5x(-1) = -4
-5 -4
1 0 -1
1 0 -1
1 0 -1
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 8
Edge Detection Via Convolution Operation
1x1 + 8x1 + 2x1 + 2x0 + 9x0 + 5x0 + 7x(-1) + 3x(-1) + 1x(-1) = 0
-5 -4 0
1 0 -1
1 0 -1
1 0 -1
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 9
Edge Detection Via Convolution Operation
1x1 + 6x1 + 2x1 + 7x0 + 2x0 + 3x0 + 8x(-1) + 8x(-1) + 9x(-1) = -16
-5 -4 0 8
1 0 -1
-10 -2 2 3
1 0 -1
0 -2 -4 -7
1 0 -1
-3 -2 -3 -16
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 10
How does Convolution work?
▪ Convolution of the image with a filter (also called kernel,
window, mask, or template) with different coefficient values
results in a new filtered output image e.g.,
► Image convolved with a filter with positive and equal
coefficients results in smoothed output image
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 11
How does Convolution work?
▪ Convolution of the image with a filter (also called kernel,
window, mask, or template) with different coefficient values
results in a new filtered output image e.g.,
► Image convolved with a filter with positive and equal
coefficients results in smoothed output image
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 12
How does Convolution work?
▪ Convolution of the image with a filter (also called kernel,
window, mask, or template) with different coefficient values
results in a new filtered output image e.g.,
► Image convolved with a filter with positive and equal
coefficients results in smoothed output image
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 13
How does Convolution work?
▪ Convolution of the image with a filter (also called kernel,
window, mask, or template) with different coefficient values
results in a new filtered output image e.g.,
► Similarly we can also compute image derivatives to compute
edges in the input image
Any idea what could be the filter coefficients?
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 14
Edge Detection Via Convolution Operation
The natural derivative operator can be defined as the
difference between the intensity of neighbouring pixels
f
= f ( x + 1) − f ( x)
x
z1 z2 z3
z4 z5 z6
z7 z8 z9
z5 = -1 z6 = -1
z8 = 1 z9 = 1
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 15
Edge Detection Via Convolution Operation
Vertical edges
Horizontal edges
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 16
Edge Detection Via Convolution Operation
10x1 + 10x1 + 10x1 + 0x0 + 0x0 + 0x0 + 0x(-1) + 0x(-1) + 0x(-1) = 30
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 17
Edge Detection Via Convolution Operation
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 18
Learning To Detect Edges
3 -3
2 -2 10 -10
3 -3
Prewitt Sobel Schar
With the rise of deep
learning, it is possible to
automatically learn these
filter coefficients more
robustly via backpropagation
for a specific task e.g., edge
detection
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 19
Edge Detection Via Convolution Operation
Vertical edges
Horizontal edges
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 20
Spatial Dimensions: A Closer Look
7x7 input
(spatially)
assume
3x3 filter
applied with
stride 1
7
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 21
Spatial Dimensions: A Closer Look
7x7 input
(spatially)
assume
3x3 filter
applied with
stride 1
7
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 22
Spatial Dimensions: A Closer Look
7x7 input
(spatially)
assume
3x3 filter
applied with
stride 1
7
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 23
Spatial Dimensions: A Closer Look
7x7 input
(spatially)
assume
3x3 filter
applied with
stride 1
7
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 24
Spatial Dimensions: A Closer Look
Output
dimension?
7
5x5 output
7x7 input
(spatially)
assume
3x3 filter
applied with
stride 1
7
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 25
Spatial Dimensions: A Closer Look
7x7 input
(spatially)
assume
3x3 filter
applied with
stride 2
7
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 26
Spatial Dimensions: A Closer Look
7x7 input
(spatially)
assume
3x3 filter
applied with
stride 2
7
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 27
Spatial Dimensions: A Closer Look
Output
dimension?
7
3x3 output
7x7 input
(spatially)
assume
3x3 filter
applied with
stride 2
7
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 28
Spatial Dimensions: A Closer Look
Doesn’t fit!
7 Cannot apply
3x3 filter on
7x7 input 7x7 input with
(spatially) stride 3
assume
3x3 filter
applied with
stride 3
7
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 29
Spatial Dimensions: A Closer Look
Output size?
(N - F) / stride + 1
E.g., with N = 7, F = 3:
stride 1 => (7 - 3)/1 + 1 = 5
stride 2 => (7 - 3)/2 + 1 = 3
stride 3 => (7 - 3)/3 + 1 = 2.33
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 30
Common Practice: Zero Padding At Borders
(N+2P-F)/stride + 1
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 31
Valid vs Same Convolutions
(N+2P-F)/stride + 1
▪ Valid convolution: The spatial dimensions of the resulting image
after convolution shrinks
▪ Same convolution: The spatial dimensions of the resulting image
after the convolution stays the same
► Acheived via zero-padding
(N+2P-F)/S + 1 = N
For S=1,
N+2P-F + 1 = N
=> P = (F-1)/2
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 32
Convolution Layer
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 33
Convolution Over Volumes
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 34
Convolution Over Volumes
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 35
Convolution Over Volumes
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 36
Convolution Over Volumes
6x6x3 3x3x3 4x4
Note we have now 27 learnable coefficients
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 37
Convolution Over Volumes
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 38
Convolution Over Volumes
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 39
Convolution Over Volumes
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 40
Convolutional Layer: Neuron View
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 41
Receptive Field
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 42
Convolutional Layer: Neuron View
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 43
Single Convolutional Layer
with 6 5x5x3 filters
𝑤1
(75x6 entries)
𝑎0 𝑎1
𝑧 1 = 𝑤 1 𝑎 0 + 𝑏1
𝑎1 = 𝑔(𝑧1 )
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 44
ConvNets
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 45
ConvNets
Flatten the last volume, e.g., 24 x 24 x 10 volume into 5760-d vector of
neurons and feed them to Fully Connected (FC) layer followed by a softmax
unit for prediction
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 46
Example
Input volume: 32x32x3
10 5x5x3 filters with stride 1, pad 2
Output volume size?
(32+2*2-5)/1+1 = 32 spatially, so
32x32x10
Number of parameters in this layer?
each filter has 5*5*3 + 1 = 76 params (+1 for bias)
=> 76*10 = 760
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 47
ConvNet Dimensions
Common settings:
K = (powers of 2, e.g. 32, 64, 128, 512)
- F = 3, S = 1, P = 1
- F = 5, S = 1, P = 2
- F = 5, S = 2, P = ? (whatever fits)
- F = 1, S = 1, P = 0
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 48
1x1 convolution
2x
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 49
1x1 convolution
2x
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 50
1x1 convolution layer
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 51
Pooling Layer
▪ Makes the representations smaller and more manageable
▪ Operates over each activation map independently
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 52
MAX Pooling
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 53
Average-Pooling
3.75 1.25
4 2
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 54
MAX-Pooling
What would be the
results of appliying
Max-POOl using
F=3&S=1?
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 55
MAX-Pooling
9 9 5
9 9 5
8 6 9
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 56
Pooling Dimensions
Common settings:
F = 2, S = 2
F = 3, S = 2
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 57
Example: ConvNets
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 58
Summary of Typical ConvNet Design
▪ ConvNets stack CONV,POOL,FC layers
▪ Trend towards smaller filters and deeper architectures
▪ Trend towards getting rid of POOL/FC layers (just CONV)
▪ Historically architectures looked like
[(CONV-RELU)*N-POOL?]*M - (FC-RELU)*K, SOFTMAX
where N is usually up to ~5, M is large, 0 <= K <= 2
▪ However, recent advances such as ResNet/GoogLeNet have
challenged this paradigm
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 59
CNNs vs FC Neural Networks
Two major advantages of CNNs over FC neural networks
▪ Parameter sharing
► A feature detector (such as vertical edge detector) that is
useful in one part of the image is probably useful in another
part of the image (translational invariance)
For a regular neural network with 32 x 32 x 3(= 3072) convolved
dense connections, this means you with 6 filters 5 x 5 x 3 resulting
have 3072 x 4704 ≈ 14 Million weights in 28 x 28 x 6 volume (= 4704)
How many parameters do we
need for Conv layer?
32 x 32 x 3 28 x 28 x 6
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 60
CNNs vs FC Neural Networks
Two major advantages of CNNs over FC neural networks
▪ Parameter sharing
► A feature detector (such as vertical edge detector) that is
useful in one part of the image is probably useful in another
part of the image (translational invariance)
32 x 32 x 3(= 3072) convolved
(75 + 1) x 6 = 456 only with 6 filters 5 x 5 x 3 resulting
in 28 x 28 x 6 volume (= 4704)
How many parameters do we
need for Conv layer?
32 x 32 x 3 28 x 28 x 6
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 61
CNNs vs FC Neural Networks
Two major advantages of CNNs over FC neural networks
▪ Parameter sharing
► A feature detector (such as vertical edge detector) that is
useful in one part of the image is probably useful in another
part of the image (translational invariance)
▪ Sparsity of connections (i.e., Local receptive field)
► In each layer, each output value depends only on a small
number of inputs
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 62
Acknowledgements
Various contents in this presentation have been taken from
different books, lecture notes (particularly CS231n Stanford, MIT
6.S191, deeplearning.ai & neuralnetworksanddeeplearning.com),
and the web. These solely belong to their owners and are here used
only for clarifying various educational concepts. Any copyright
infringement is not intended.
CS490 – Advanced Topics in Computing (Deep Learning) Lecture 16: Convolutional Neural Networks (CNNs) 63
