An Overview of
Machine Learning
Speaker: Yi-Fan Chang
Adviser: Prof. J. J. Ding
Date: 2011/10/21
Outline & Content
What is machine learning?
Learning system model
Training and testing
Performance
Algorithms
Machine learning structure
What are we seeking?
Learning techniques
Applications
Conclusion
What is machine learning?
A branch of artificial intelligence, concerned wit
h the design and development of algorithms that al
low computers to evolve behaviors based on empiric
al data.
As intelligence requires knowledge, it is necessar
y for the computers to acquire knowledge.
Learning system model
Testing
Input Learning
Samples Method
System
Training
Training and testing
Data acquisition Practical usage
Universal
set
(unobserved
)
Training set Testing set
(observed) (unobserved
)
Training and testing
Training is the process of making the system able
to learn.
No free lunch rule:
Training set and testing set come from the same distribut
ion
Need to make some assumptions or bias
Performance
There are several factors affecting the performanc
e:
Types of training provided
The form and extent of any initial background knowledge
The type of feedback provided
The learning algorithms used
Two important factors:
Modeling
Optimization
Algorithms
The success of machine learning system also depend
s on the algorithms.
The algorithms control the search to find and buil
d the knowledge structures.
The learning algorithms should extract useful info
rmation from training examples.
Algorithms
Supervised learning (
)
Prediction
Classification (discrete labels), Regression (real value
s)
Unsupervised learning ( )
Clustering
Probability distribution estimation
Finding association (in features)
Dimension reduction
Semi-supervised learning
Reinforcement learning
Decision making (robot, chess machine)
Algorithms
Supervised learning Unsupervised learning
10 Semi-supervised learning
Machine learning structure
Supervised learning
Machine learning structure
Unsupervised learning
What are we seeking?
Supervised: Low E-out or maximize probabilistic te
rms
E-in: for training set
E-out: for testing set
Unsupervised: Minimum quantization error, Minimum
distance, MAP, MLE(maximum likelihood estimation)
What are we seeking?
Under-fitting VS. Over-fitting (fixed N)
error
(model = hypothesis + loss
functions)
Learning techniques
Supervised learning categories and techniques
Linear classifier (numerical functions)
Parametric (Probabilistic functions)
Naïve Bayes, Gaussian discriminant analysis (GDA), Hi
dden Markov models (HMM), Probabilistic graphical mod
els
Non-parametric (Instance-based functions)
K-nearest neighbors, Kernel regression, Kernel densit
y estimation, Local regression
Non-metric (Symbolic functions)
Classification and regression tree (CART), decision t
ree
Aggregation
Bagging (bootstrap + aggregation), Adaboost, Random f
orest
Learning techniques
• Linear classifier
, where w is an d-dim vector (learned)
Techniques:
Perceptron
Logistic regression
Support vector machine (SVM)
Ada-line
Multi-layer perceptron (MLP)
Learning techniques
Using perceptron learning algorithm(PLA)
Trainin Testing
g rate: 0.10
Error Error rate: 0.156
Learning techniques
Using logistic regression
Trainin Testing
g rate: 0.11
Error Error rate: 0.145
Learning techniques
• Non-linear case
Support vector machine (SVM):
Linear to nonlinear: Feature transform and kernel function
Learning techniques
Unsupervised learning categories and techniques
Clustering
K-means clustering
Spectral clustering
Density Estimation
Gaussian mixture model (GMM)
Graphical models
Dimensionality reduction
Principal component analysis (PCA)
Factor analysis
Applications
Face detection
Object detection and recognition
Image segmentation
Multimedia event detection
Economical and commercial usage
Conclusion
We have a simple overview of some t
echniques and algorithms in machine lea
rning. Furthermore, there are more and
more techniques apply machine learning
as a solution. In the future, machine l
earning will play an important role in
our daily life.
Reference
[1] W. L. Chao, J. J. Ding, “Integrate
d Machine Learning Algorithms for Human
Age Estimation”, NTU, 2011.
