AI VIETNAM

All-in-One Course

Extra Class

Advanced CNN Architecture

Nguyen Quoc Thai

1
Year 2023
 CONTENT
(1) – ResNet
(2) – Feature Extraction
(3) – Image Similarity

2
 1 – ResNet
! Review: Skip Connection

target

block 1 block 2 block 3 block 4

Loss
input

+ + + +
output

3
 1 – ResNet
! Review: Skip Connection
[128, 10, 64, 64] [128, 3, 64, 64]

[128, 3, 64, 64] Conv2d Conv2d

in_channels=3 [128, 10, 64, 64] in_channels=10 [128, 3, 64, 64] [128, 3, 64, 64]
input
out_channels=10 out_channels=3
feature ReLU + ReLU
kernel_size=3 kernel_size=3
map
stride=1 stride=1
padding=1 padding=1 Identity Shortcut

[128, 10, 64, 64] [128, 3, 32, 32]

[128, 3, 64, 64] Conv2d Conv2d
in_channels=3 [128, 10, 64, 64] in_channels=10 [128, 3, 32, 32] [128, 3, 32, 32]
input
out_channels=10 out_channels=3
feature ReLU + ReLU
kernel_size=3 kernel_size=3
map
stride=1 [128, 3, 32, 32] stride=2
padding=1 padding=1 Projection Shortcut
Conv2d
in_channels=3
out_channels=3
kernel_size=3
stride=2
padding=1 4
 1 – ResNet
! Review: Skip Connection

+ +

Short Skip Connection Long Skip Connection

(ResNet, …) (UNet, …)

5
 1 – ResNet
! ResNet18
1x1 conv, 128 1x1 conv, 256 1x1 conv, 512
Stride: 2 Stride: 2 Stride: 2

64-d 2 strides to 2 strides to 2 strides to

3x3 conv, 64 down-sample down-sample down-sample Classifier
relu
3x3 conv, 64
+ relu Source 6
 1 – ResNet
! ResNet18

7
 1 – ResNet
! ResNet (18 – 34 – 50 – 101 – 152)

Source 8
 1 – ResNet
! CIFAR10 Classification using ResNet18

❖ Training: 50,000 images. Testing: 10,000 images

❖ Classes: 10

9
 1 – ResNet
! CIFAR10 Classification using ResNet18 – Demo

❖ Load Dataset

10
 1 – ResNet
! CIFAR10 Classification using ResNet18 – Demo

❖ Preprocessing

11
 1 – ResNet
! CIFAR10 Classification using ResNet18 – Demo

❖ Model

12
 1 – ResNet
! CIFAR10 Classification using ResNet18 – Demo

❖ Training

13
 2 – Pretrained Models
! ImageNet

❖ Training: 1,281,167 images. Validation: 50,000 images. Testing: 100,000 images

❖ Object classes: 1,000

14
 2 – Pretrained Models
! ImageNet

❖ Training: 1,281,167 images. Validation: 50,000 images. Testing: 100,000 images

❖ Object classes: 1,000

MODEL
(ResNet, VGG)

Large Dataset Trained

(ImageNet,..)
Pretrained Model

15
 2 – Pretrained Models
! ImageNet

❖ Training: 1,281,167 images. Validation: 50,000 images. Testing: 100,000 images

❖ Object classes: 1,000

Source 16
 2 – Pretrained Models
! ImageNet

❖ Training: 1,281,167 images. Validation: 50,000 images. Testing: 100,000 images

❖ Object classes: 1,000

Source 17
 2 – Pretrained Models
! Transfer Learing

❖ Traditional Learning: Isolated, single task ❖ Transfer Learning: Learning of a new tasks relies
learning on the previous learned tasks

Model Model

Dataset #1 Dataset #1 Trained

Knowledge
(Weights)

Model Model

Dataset #2 Dataset #2 18
 2 – Pretrained Models
! Transfer Learing

❖ Transfer Learning: Feature Extractor

Model (ResNet18)

Feature Extraction Classifier

Large Dataset
(ImageNet)
Trained Copy Weights

Model (ResNet18)

Feature Extraction New Task

Small Dataset
(CIFAR10,…) 19
 2 – Pretrained Models
! Feature Extraction using Pretrained Models

Vector
Feature Extractor Matrix New Task

Pretraied Model (ResNet18)

20
 2 – Pretrained Models
! Feature Extraction using Pretrained Models - Demo

21
 2 – Pretrained Models
! Feature Extraction using Pretrained Models - Demo

22
 3 – Image Similarity
1 Similarity between two images

MSE
L1 – Distance
Cosine Similarity
Normalized Cross-Correlation
Feature Extractor

Similarity/ Similarity Score /

Distance Function Distance

Feature Extractor

23
 3 – Image Similarity
1 Similarity between two images - MSE

𝑌 = [𝑦!, 𝑦", 𝑦#, …, 𝑦$ ]

𝑌' = [𝑦(!, 𝑦(", 𝑦(#, … , 𝑦($ ]

24
 3 – Image Similarity
1 Similarity between two images - MSE

1.0 2.0 4.0 5.0

2
- = ( -0.2 -0.4 0.6 -0.6 ) 0.04 0.16 0.36 0.36 0.23
mean
1.2 2.4 3.4 5.6

100 105 98 110

2
- = ( -2 -2 1 ? ) 4 4 1 ? ?
mean
102 107 97 108

25
 3 – Image Similarity
5 Similarity between two images – on raw images

MSE: 0.4063 MSE: 0.3265

26
 3 – Image Similarity
3 Similarity between two images –L1 Distance

'
𝑌 = [𝑦!, 𝑦", 𝑦#, …, 𝑦$ ]
𝐿1 = + |𝑝% − 𝑞% |
𝑌' = [𝑦(!, 𝑦(", 𝑦(#, … , 𝑦($ ]
%&!

27
 3 – Image Similarity
3 Similarity between two images –L1 Distance

100 105 98 110

sum
- = | -2 -2 1 2 | 2 2 1 2 7

102 107 97 108

5 8 4 2
sum
- = | 2 1 -6 ? | 2 1 6 ? 13

3 7 10 6
28
 3 – Image Similarity
5 Similarity between two images – on raw images

L1: 51128.43 L1: 40339.5

29
 3 – Image Similarity
5 Similarity between two images – on raw images

Problem?

L1: 46423.3 L1: 40339.5

MSE: 0.3477 MSE: 0.3265 30
 3 – Image Similarity
2 Similarity between two images – Cosine Similarity

𝐴 = [𝑎!, 𝑎", 𝑎#, … , 𝑎$ ]

𝐵( = 𝑏!, 𝑏", 𝑏#, … , 𝑏$

𝐴. 𝐵 = 𝑎!𝑏! + 𝑎"𝑏" + ⋯ 𝑎$ 𝑏$
𝐴 = 𝑎!" + 𝑎"" + ⋯ + 𝑎$"

𝐵 = 𝑏!" + 𝑏"" + ⋯ + 𝑏$"

31
 3 – Image Similarity
2 Similarity between two images – Cosine Similarity

5.0

6.0
1∗5+2∗6+3∗7+4∗8
1.0 2.0 3.0 4.0
* 7.0
= 1! + 2! + 3! + 4! 5! + 6! + 7! + 8!
= 0.97

8.0

3 ?
3 2 4 6 * = ?∗ ?
= ?
2

6 32
 3 – Image Similarity
5 Similarity between two images – on raw images

Cosine sim: 1.0 Cosine sim: 0.83

33
 3 – Image Similarity
4 Similarity between two images – Normalized Cross-Correlation

Formula

34
 3 – Image Similarity
4 Similarity between two images – Normalized Cross-Correlation

35
 3 – Image Similarity
4 Similarity between two images – Normalized Cross-Correlation

2
1.0 2.0 4.0 9.0 - 4 = ( -3 -2 0 5 ) 9 4 0 25

2
4.0 3.0 1.0 2.0 - 2.5 = ( 1.5 0.5 -1.5 -0.5 ) 2.25 ? 2.25 ?

36
 3 – Image Similarity
4 Similarity between two images – Normalized Cross-Correlation

-3 -2 0 5

* = -4.5 ? 0 -2.5

1.5 0.5 -1.5 -0.5 −8

= ?
38 ∗ 5
Sum
9 4 0 25 38

Sum
2.25 0.25 2.25 0.25 5
37
 3 – Image Similarity
5 Similarity between two images – on raw images

NCC: 0.99 NCC: 0.3937

38
 3 – Image Similarity
5 Similarity between two images – on raw images

Method Similarity

L1 Distance 51128.43

MSE 0.4063

Cosine similarity 0.7620

Normalized Cross Correlation -0.3564

39
 3 – Image Similarity
5 Similarity between two images – on raw images

Method Similarity

L1 Distance 40339.5

MSE 0.3265

Cosine similarity 0.8294

Normalized Cross Correlation 0.3937

40
 3 – Image Similarity
5 Similarity between two images – on raw images

Method Similarity

L1 Distance 46423.3

MSE 0.3477

Cosine similarity 1.0000

Normalized Cross Correlation 0.9999

41
 3 – Image Similarity
5 Similarity between two images – on raw images

Method Similarity

L1 Distance 30073.23

MSE 0.2542

Cosine similarity 0.8691

Normalized Cross Correlation 0.3812

Problem?

42
 3 – Image Similarity
6 Similarity between two images – using resnet18

Feature Extractor

Feature Extractor

43
 3 – Image Similarity
6 Similarity between two images – using resnet18

Load the model

Compute feature

44
 3 – Image Similarity
6 Similarity between two images – using resnet18

Feature Extractor

Similarity/ Similarity Score /

Distance Function Distance

Feature Extractor

45
 3 – Image Similarity
6 Similarity between two images – using resnet18

Method Similarity

L1 Distance 53.34

MSE 0.1329

Cosine similarity 0.8999

Normalized Cross Correlation 0.4541

46
 3 – Image Similarity
6 Similarity between two images – using resnet18

Method Similarity

L1 Distance 49.47

MSE 0.123

Cosine similarity 0.912

Normalized Cross Correlation 0.595

47
 3 – Image Similarity
6 Similarity between two images – using resnet18

Method Similarity

L1 Distance 4.344

MSE 0.011

Cosine similarity 0.9993

Normalized Cross Correlation 0.9961

48
 3 – Image Similarity
5 Similarity between two images – on raw images

Method Similarity

L1 Distance 22.1067

MSE 0.0549

Cosine similarity 0.9983

Normalized Cross Correlation 0.7959

49
Thanks!
Any questions?

50