Deep Learning with MATLAB
Cagliari (Italy), 19 Sep. 2019
Giuseppe Ridinò
Senior Application Engineer
MathWorks Italy
© 2019 The MathWorks, Inc.
1
MATLAB & Simulink in Research
2
TNG Data Project and Jupyterlab Interface
Project: [Link]
Jupyterlab: [Link]/data/lab/
Scripts: [Link]
[Link]/data/docs/scripts/
3
MATLAB Community Tools for Astronomy
[Link]
4
LIGO: Spot 1st-Ever Gravitational Waves with MATLAB
Article
Blog post
5
NASA Build Kepler Pipeline tools with MATLAB
download: [Link]/nasa/kepler-pipeline
6
Agenda
Deep Learning Overview
Deep Learning in 6 Lines of MATLAB Code
Deep Learning with…
Images Signals
Labeling Ground Truth Data
Working with Other Frameworks
Performance, Deployment, and Next Steps
7
Artificial Intelligence
▪ Development of computer systems to perform tasks that
normally require human intelligence
Machine Learning
Decision Deep Logistic
Trees Learning Regression
K-means SVM
Nearest Reinforcement Learning
Neighbor Gaussian
Mixture
8
Machine Learning and Deep Learning
Machine Learning
Unsupervised
Learning
[No Labeled Data]
Clustering
9
Machine Learning and Deep Learning
Machine Learning
Unsupervised Supervised Learning
Learning [Labeled Data]
[No Labeled Data]
Clustering Classification Regression
10
Machine Learning and Deep Learning
Machine Learning
Unsupervised Supervised Learning
Learning [Labeled Data]
[No Labeled Data] Supervised learning typically involves
feature extraction
Clustering Classification Regression
Deep Learning
Deep learning typically does not
involve feature extraction
11
What is Deep Learning?
12
Deep Learning
▪ Subset of machine learning with automatic feature extraction
– Learns features and tasks directly from data
– More Data = better model
Deep Learning
Machine
Learning
Deep
Learning
13
Deep Learning uses a neural network architecture
Input Output
Hidden Layers (n)
Layer Layer
14
Deep Learning datatypes
Image Signal
Text
Numeric
15
Example 1: Detection and localization using deep learning
YOLO v2 (You Only Look Once) Semantic Segmentation using SegNet
16
Example 2: Analyzing signal data using deep learning
Signal Classification using LSTMs Speech Recognition using CNNs
17
Machine Learning vs Deep Learning
Deep learning performs end-to-end learning by learning features, representations and tasks directly
from images, text and sound
Deep learning algorithms also scale with data – traditional machine learning saturates
Machine Learning
Deep Learning
18
Deep Learning workflow
PREPARE DATA TRAIN MODEL DEPLOY SYSTEM
Data access and Model design, Multiplatform code
preprocessing Hyperparameter generation (CPU,
tuning GPU)
Ground truth labeling Model exchange Embedded
across frameworks deployment
Hardware-
Simulation-based Enterprise
accelerated
data generation Deployment
training
I Iteration and Refinement
19
Agenda
Deep Learning Overview
Deep Learning in 6 Lines of MATLAB Code
Deep Learning with…
Images Signals
Labeling Ground Truth Data
Working with Other Frameworks
Performance, Deployment, and Next Steps
20
Deep Learning in 6 lines of MATLAB code
21
Using Apps for designing Deep Learning networks
22
Different Neural Networks for Different Data
23
Types of Data
Numeric Time Series/ Image
Data Text Data Data
Machine Learning or CNN
LSTM LSTM or CNN
24
Convolutional Neural Networks (CNN)
Edges
Shapes
Objects
25
CNN Visualization methods
Layer Activations Class Activations DeepDream Images
26
Another network for signals - LSTM
▪ LSTM = Long Short Term Memory (Networks)
– Signal, text, time-series data
– Use previous data to predict new information
▪ I live in France. I speak ___________.
c0 C1 Ct
27
Combining convolution and recurrent layers
Learn spatial and temporal features simultaneously
Response
CNN LSTM Class 1
class label
CNN LSTM CNN learns features
from each frame
individually
CNN LSTM
LSTM learns
relationships
Video Frames
CNN LSTM between frames
28
Agenda
Deep Learning Overview
Deep Learning in 6 Lines of MATLAB Code
Deep Learning with…
Images Signals
Labeling Ground Truth Data
Working with Other Frameworks
Performance, Deployment, and Next Steps
29
Agenda
Deep Learning Overview
Deep Learning in 6 Lines of MATLAB Code
Deep Learning with…
Images Signals
Labeling Ground Truth Data
Working with Other Frameworks
Performance, Deployment, and Next Steps
30
Two approaches for Deep Learning
1. Train a Deep Neural Network from Scratch
2. Fine-tune a pre-trained model (transfer learning)
31
Semantic Segmentation
CamVid Dataset
1. Segmentation and Recognition Using Structure from Motion Point Clouds, ECCV 2008
2. Semantic Object Classes in Video: A High-Definition Ground Truth Database ,Pattern Recognition Letters
32
Semantic Segmentation Network
Boat
Airplane
Other classes
33
Semantic Segmentation Network
34
Example: Semantic Segmentation for Free Road
Detection
35
1. Use CamVid dataset
▪ Download the dataset (images and labels)
▪ Resize images to fit the input sizes of our neural network
36
2. Balance classes using class weighting
Ideally, all classes would have an
equal number of observations.
However, the classes in CamVid are
imbalanced.
We use the pixel label counts and
calculate the median frequency
class weights.
37
3. Split dataset
Training Validation Test
60% 20% 20%
• Trains the model • Checks accuracy • Tests model accuracy
• Computer “learns” of model during • Not used until validation
from this data training accuracy is good
38
4. Define network
91x1 Layer array with layers:
1 'inputImage' Image Input 360x480x3 images with 'zerocenter' normalization
2 'conv1_1' Convolution 64 3x3x3 convolutions with stride [1 1] and padding [1 1 1 1]
3 'bn_conv1_1' Batch Normalization Batch normalization
4 'relu1_1' ReLU ReLU
5 'conv1_2' Convolution 64 3x3x64 convolutions with stride [1 1] and padding [1 1 1 1]
6 'bn_conv1_2' Batch Normalization Batch normalization
7 'relu1_2' ReLU ReLU
8 'pool1' Max Pooling 2x2 max pooling with stride [2 2] and padding [0 0 0 0]
9 'conv2_1' Convolution 128 3x3x64 convolutions with stride [1 1] and padding [1 1 1 1]
10 'bn_conv2_1' Batch Normalization Batch normalization
11 'relu2_1' ReLU ReLU
...
• Use segnetLayers to create a SegNet network
initialized from a pre-trained VGG-16.
• Another option: Deeplab v3+ network with weights
initialized from a pre-trained Resnet-18.
Last 9 layers: the last one provides
also the Class Weighting
39
5. Train network
40
Useful tools for Semantic Segmentation
▪ Automatically create network structures
– Using segnetLayers and fcnLayers
▪ Handle pixel labels
– Using the pixelLabelImageDatastore and
pixelLabelDatastore
▪ Evaluate network performance
– Using evaluateSemanticSegmentation
▪ Examples and tutorials to learn concepts
41
Agenda
Deep Learning Overview
Deep Learning in 6 Lines of MATLAB Code
Deep Learning with…
Images Signals
Labeling Ground Truth Data
Working with Other Frameworks
Performance, Deployment, and Next Steps
42
Common network architectures - signal processing
Convolutional Neural Networks (CNN)
Time-Frequency Transformation
Long Short Term Memory (LSTM) Networks
Feature Engineering
43
Example: Speech Recognition
44
1. Create Datastore
▪ Datastore creates
reference for data
▪ Do not have to load in
all objects into memory
45
2. Compute speech spectrograms
Amplitude
Frequency
Time
46
3. Split datastores
Training Validation Test
70% 15% 15%
• Trains the model • Checks accuracy • Tests model accuracy
• Computer “learns” of model during • Not used until validation
from this data training accuracy is good
47
4. Define architecture and parameters
Model Parameters
Neural Network Architecture
48
5. Train network
49
Training is an iterative process
Parameters adjusted according to performance
50
Agenda
Deep Learning Overview
Deep Learning in 6 Lines of MATLAB Code
Deep Learning with…
Images Signals
Labeling Ground Truth Data
Working with Other Frameworks
Performance, Deployment, and Next Steps
51
Labeling for deep learning is repetitive,
tedious, and time-consuming…
but necessary
52
Ground truth labeling for images and video
“How do I label
my data?”
App for
Ground Truth
Labeling
Label regions
and pixels for
semantic
segmentation
53
Ground truth labeling for signals and audio
“How do I label
my data?”
App for Signal
Analysis and
Labeling
Label signal
waveforms,
attributes, and
samples for
machine
learning
54
MATLAB Apps help you speed up labeling work
For labeling images and videos:
1. Image Labeler (Computer Vision Toolbox)
▪ [Link]
2. Video Labeler (Computer Vision Toolbox)
▪ [Link]
3. Ground Truth Labeler (Automated Driving Toolbox)
▪ [Link]
How to choose the correct app for your project?
– [Link]
55
MATLAB Apps help you speed up labeling work
For labeling signals and audio:
1. Audio Labeler
(Audio Toolbox)
[Link]
[Link]
2. Signal Labeler
(Signal Processing Toolbox)
[Link]
[Link]
56
Agenda
Deep Learning Overview
Deep Learning in 6 Lines of MATLAB Code
Deep Learning with…
Images Signals
Labeling Ground Truth Data
Working with Other Frameworks
Performance, Deployment, and Next Steps
57
Two ways to work with TensorFlow and PyTorch
MATLAB Python
Co-execution (Python and C++)
Model Exchange
58
Three use cases for model exchange
1. Enable teams to use different frameworks and tools
2. Access to code generation and other downstream tools in MATLAB
3. MATLAB users gain access to models in research/DL ecosystem
59
Model exchange with MATLAB
PyTorch Keras-
Tensorflow
Caffe2
MXNet ONNX MATLAB
Core ML
Caffe
CNTK (…)
Open Neural Network Exchange
60
ONNX – Industry Standard for Model Exchange
Source: [Link]
61
Transfer Learning with Pretrained Models
Inception-v3 ResNet-101 VGG-16
Inception-
ResNet-v2
ResNet-18 GoogLeNet DenseNet-201
VGG-19
SqueezeNet AlexNet ResNet-50
Import & Export Models Between Frameworks
Keras-Tensorflow Caffe Model ONNX Model
Importer Importer Converter
More comprehensive list here: [Link]
62
Agenda
Deep Learning Overview
Deep Learning in 6 Lines of MATLAB Code
Deep Learning with…
Images Signals
Labeling Ground Truth Data
Working with Other Frameworks
Performance, Deployment, and Next Steps
63
Training Performance and Scalability
64
Deep Learning on CPU, GPU, Multi-GPU and Clusters
HOW TO TA R G E T ?
Single Single CPU
CPU Single GPU
Single CPU, Multiple GPUs
On-prem server with GPUs Cloud GPUs
65
MATLAB parallel capabilities for cloud are evolving
Option Parallel Computing Toolbox MATLAB Parallel Server (formally called MDCS)
Desktop Custom Cloud User-managed Cloud Pre-configured Cloud Custom Cloud On Premise
Reference Cloud Center Hosting Providers Reference
Architecture Architecture
Description Explicit desktop Explicit desktop Preconfigured clusters in Cloud solutions from Custom infrastructure Scale to clusters in your
scaling scaling Amazon Web Services MathWorks partners for AWS, Azure, and organization
others
When to Prototyping, small Data in cloud, access User-centric workflows, Managed environments with Internally-managed Traditional scaling on
use scaling high-end machines, small studies operating expense rather environments with assets you manage at
especially with GPUs than capital expense operating expense your facilities
rather than capital
expense
Maximum Physical cores in Physical cores in 1024 per cluster Defined by Hosting Provider No limit Physical cores in cluster
workers machine machine
License • Network • Online • Online only • Network • Online • Network
Options • Online • Network (requires • Online • Network (requires • Online
support) support)
[Link] 66
Cloud Reference Architecture: MATLAB & Simulink
Virtual Network
Use cases:
Remote • Data analytics on cloud-stored data
Desktop • Access to high-end hardware:
Client − multi-core
− GPUs
− FPGAs
• Prototyping parallel algorithms
• On-server pre-production testing
67
Reference Architecture: MATLAB Distributed Computing Server
Virtual Network
Head node VM
with MATLAB job scheduler
Use cases:
• Parameter sweeps
• Monte Carlo runs
• Optimization
Client with MATLAB and • Distributed array calculations
Parallel Computing Toolbox
Compute node
VMs
MathWorks Hosted
License Manager
68
NVIDIA NGC & DGX Supports MATLAB for Deep Learning
▪ GPU-accelerated MATLAB Docker container for deep learning
– Leverage multiple GPUs on NVIDIA DGX Systems and in the Cloud
▪ Cloud providers include: AWS, Azure, Google, Oracle, and Alibaba
▪ NVIDIA DGX System / Station
– Interconnects 4/8/16 Volta GPUs in one box
▪ Containers available for R2018a through R2019a
– New Docker container with every major release (a/b)
▪ Download MATLAB container from NGC Registry
– [Link]
69
Inference Performance and Deployment
70
Deployment process
PREPARE DATA TRAIN MODEL DEPLOY SYSTEM
Data access and Model design, Multiplatform code
preprocessing Hyperparameter generation (CPU,
tuning GPU)
Ground truth labeling Model exchange Edge deployment
across frameworks
Hardware-
Simulation-based Enterprise
accelerated
data generation Deployment
training
I Iteration and Refinement
71
Deploying Deep Learning models for inference
Intel
MKL-DNN
Library
NVIDIA
Coder GPU Coder TensorRT &
Products cuDNN
Libraries
Deep Learning
Networks
ARM
Compute
Library
72
Deploying to HW targets
Desktop CPU
NVIDIA
Coder GPU Coder TensorRT &
Products cuDNN
Libraries
Deep Learning
Networks
Raspberry Pi board
73
With GPU Coder, MATLAB is fast
GPU Coder is faster
than TensorFlow,
MXNet and PyTorch
TensorFlow
MXNet
GPU Coder
PyTorch
Intel® Xeon® CPU 3.6 GHz - NVIDIA libraries: CUDA10 - cuDNN 7 - Frameworks: TensorFlow 1.13.0, MXNet 1.4.0 PyTorch 1.0.0 74
Semantic Segmentation speedup
Running in MATLAB Generated Code from GPU Coder
75
Blog: Deep Learning
[Link]
76
Introduction to Deep Learning
[Link]
77
Deep Learning with MATLAB
[Link]
78
Deep Learning Onramp
[Link]
79
Thank you!
80
