Lecture 6
Feature Selection
and Extraction
Machine Learning
Ivan Smetannikov
15.06.2016
Lecture plan
• Dimensionality Reduction
• Feature Selection
• Feature Extraction
Machine learning. Lecture 6. Feature Selection and Extraction. 15.06.2016. 2
Lecture plan
• Dimensionality Reduction
• Feature Selection
• Feature Extraction
Machine learning. Lecture 6. Feature Selection and Extraction. 15.06.2016. 3
Dimensionality Reduction
Why should we look at dimensionality
reduction?
• Speeds up algorithms
• Reduces space used by data for them
Machine learning. Lecture 6. Feature Selection and Extraction. 15.06.2016. 4
Dimensionality Reduction
What is dimensionality reduction?
• You’ve collected many features – maybe
more than you need. Can you ”simply”
your data set in a rational and useful way?
Machine learning. Lecture 6. Feature Selection and Extraction. 15.06.2016. 5
Dimensionality Reduction
Example:
• Redundant data set –
different units for
same attribute
• Reduce data to 1D
(2D -> 1D)
Machine learning. Lecture 6. Feature Selection and Extraction. 15.06.2016. 6
Dimensionality Reduction
Another Example
• Helicopter flying - do a survey of pilots
(x1 = skill, x2 = pilot enjoyment) These
features may be highly correlated
• This correlation can be combined into a
single attribute called aptitude (for
example)
Machine learning. Lecture 6. Feature Selection and Extraction. 15.06.2016. 7
Dimensionality Reduction
So what does
dimensionality
reduction mean?
• Let plot a line
• Take exact example
and record
position on that
line
• So we can present
x1 as 1D number
Machine learning. Lecture 6. Feature Selection and Extraction. 15.06.2016. 8
Dimensionality Reduction
Another example 3D -> 2D
Machine learning. Lecture 6. Feature Selection and Extraction. 15.06.2016. 9
Dimensionality Reduction
Motivation:
Collect a large data set (50 dimensions)
Machine learning. Lecture 6. Feature Selection and Extraction. 15.06.2016. 10
Dimensionality Reduction
Using dimensionality reduction come up with a
different feature representation
Machine learning. Lecture 6. Feature Selection and Extraction. 15.06.2016. 11
Lecture plan
• Dimensionality Reduction
• Feature Selection
• Feature Extraction
Machine learning. Lecture 6. Feature Selection and Extraction. 15.06.2016. 12
Feature Selection
Goals of feature selection:
• Avoiding retraining and improving the
quality of classification
• Best understanding of models
• Boosting of classifying models
Machine learning. Lecture 6. Feature Selection and Extraction. 15.06.2016. 13
Feature Selection
Type of elected attributes:
• Redundant attributes - do not carry any
additional information
• Irrelevant attributes - are not generally
informative
Machine learning. Lecture 6. Feature Selection and Extraction. 15.06.2016. 14
Feature Selection
Evaluation methods of feature selection:
• At various datasets
• With different classifiers (if possible)
• By adding to datasets noise and target
vectors
Machine learning. Lecture 6. Feature Selection and Extraction. 15.06.2016. 15
Feature Selection
Feature selection types:
• Filter methods
a. Univariate
b. Multivariate
• Wrapper methods
a. Deterministic
b. Randomized
• Embedded methods
Machine learning. Lecture 6. Feature Selection and Extraction. 15.06.2016. 16
Feature Selection
Filter methods:
Evaluate the quality of certain attributes and
remove the worst of them.
+ Simple to compute, easy to scale
- Ignore the relationships between attributes
or features used by classifier
Machine learning. Lecture 6. Feature Selection and Extraction. 15.06.2016. 17
Feature Selection
Examples of filter methods:
• Univariate:
o Euclidian distance
o Information gain
• Spearman corellation coefficient
o Multivariate:
o CFS
o MBF
Machine learning. Lecture 6. Feature Selection and Extraction. 15.06.2016. 18
Feature Selection
Spearman corellation coefficient
Python SciPy: x : (N,) array_like
scipy.stats.pearsonr(x, y) Input
Parameters:
y : (N,) array_like
Input
(Pearson’s correlation coefficient,
Returns:
2-tailed p-value)
Machine learning. Lecture 6. Feature Selection and Extraction. 15.06.2016. 19
Feature Selection
Weka:
ASEvaluation evaluator = new CorrelationAttributeEval();
Ranker ranker = new Ranker();
// ranker.setThreshold(0.05); or ranker.setNumToSelect(10);
AttributeSelection selection = new AttributeSelection();
selection.setInputFormat(heavyInstances);
selection.setEvaluator(evaluator);
selection.setSearch(ranker);
Instances lightInstances = Filter.useFilter(heavyInstances, selection);
Machine learning. Lecture 6. Feature Selection and Extraction. 15.06.2016. 20
Feature Selection
Wrapper methods:
Get a subset of attributes of the source
+ Higher accuracy than Filtering
+ Consider the relationships between
attributes
+ Direct interaction with the classifier
- Long computing time
- The probability of re-education
Machine learning. Lecture 6. Feature Selection and Extraction. 15.06.2016. 21
Feature Selection
Examples of Wrapper methods:
• Deterministic:
o SFS (sequential forward selection?)
o SBE (sequential backward elimination?)
o SVM-RFE
• Randomized:
o Randomized Hill Climbing
o Genetic Algorithms
Machine learning. Lecture 6. Feature Selection and Extraction. 15.06.2016. 22
Feature Selection
SVM-RFE
• Train SVM on training subset
• Rank features by received weights
• Throw out last features
• Repeat until the necessary amount of
features will left
Machine learning. Lecture 6. Feature Selection and Extraction. 15.06.2016. 23
Feature Selection
SVM-RFE (Python example)
x = [1, 5, 1.5, 8, 1, 9]
y = [2, 8, 1.8, 8, 0.6, 11]
Machine learning. Lecture 6. Feature Selection and Extraction. 15.06.2016. 24
Feature Selection
SVM-RFE (Python example)
X = np.array([[1,2], [5,8], [1.5,1.8], [8,8],
[1,0.6], [9,11]])
y = [0,1,0,1,0,1]
Let use SVM:
clf = svm.SVC(kernel='linear', C = 1.0)
Let fit our model:
clf.fit(X,y)
Let predict predict something:
print(clf.predict([0.58,0.76]))
Machine learning. Lecture 6. Feature Selection and Extraction. 15.06.2016. 25
Feature Selection
Embedded
• Take into account the particular classifier
• Use individual method for each classifier
Machine learning. Lecture 6. Feature Selection and Extraction. 15.06.2016. 26
Feature Selection
Random Forest:
Machine learning. Lecture 6. Feature Selection and Extraction. 15.06.2016. 27
Feature Selection
Random Forest:
• Select a subsample of size N for each tree
with replacement
• Build decision trees. To select next feature
to split the considered
• Choose the best for a given criteria
Machine learning. Lecture 6. Feature Selection and Extraction. 15.06.2016. 28
Feature Selection
Random Forest (Python example):
# Import the random forest package
from sklearn.ensemble import RandomForestClassifier
# Create the random forest object which will include all the
parameters for the fit
forest = RandomForestClassifier(n_estimators = 100)
# Fit the training data to the Survived labels and create the
decision trees
forest = forest.fir(train_data[0::, 1::],
train_data[0::, 0])
# Take the same decision trees and run it on the test data
output = forest.predict(test_data)
Machine learning. Lecture 6. Feature Selection and Extraction. 15.06.2016. 29
Feature Selection
Random Forest (Weka):
int numFolds = 10;
br = new BufferedReader(new FileReader(“data.arff"));
Instances trainData = new Instances(br);
trainData.setClassIndex(trainData.numAttributes() - 1);
RandomForest rf = new RandomForest();
rf.setNumTrees(100);
rf.buildClassifier(trainData);
Evaluation evaluation = new Evaluation(trainData);
evaluation.crossValidateModel(rf, trainData, numFolds, new Random(1));
System.out.println("F-measure= " + evaluation.fMeasure(0));
Machine learning. Lecture 6. Feature Selection and Extraction. 15.06.2016. 30
Feature Selection
IG and IG
Machine learning. Lecture 6. Feature Selection and Extraction. 15.06.2016. 31
Feature Selection
Redundancy
Machine learning. Lecture 6. Feature Selection and Extraction. 15.06.2016. 32
Feature Selection
Regularization
Machine learning. Lecture 6. Feature Selection and Extraction. 15.06.2016. 33
Lecture plan
• Dimensionality Reduction
• Feature Selection
• Feature Extraction
Machine learning. Lecture 6. Feature Selection and Extraction. 15.06.2016. 34
Feature Extraction
Machine learning. Lecture 6. Feature Selection and Extraction. 15.06.2016. 35
Feature Extraction
Feature Extraction
• Reducing the amount of resources
required to describe a large set of data
• New features
• Linear and nonlinear
Machine learning. Lecture 6. Feature Selection and Extraction. 15.06.2016. 36
Feature Extraction
Feature Extraction
• Reducing the amount of resources
required to describe a large set of data
• New features
• Linear and nonlinear
Machine learning. Lecture 6. Feature Selection and Extraction. 15.06.2016. 37
Feature Extraction
Linear and nonlinear
Maniford Sculpting PCA
Machine learning. Lecture 6. Feature Selection and Extraction. 15.06.2016. 38
Feature Extraction
PCA
We have 2D dataset which we wish to
reduce to 1D
Machine learning. Lecture 6. Feature Selection and Extraction. 15.06.2016. 39
Feature Extraction
PCA tries to find the surface (a straight line
in this case) which has
the minimum projection error
Machine learning. Lecture 6. Feature Selection and Extraction. 15.06.2016. 40
Feature Extraction
PCA (Python example)
Let use Iris-data and import PCA
from sklearn.decomposition import PCA as sklearnPCA
sklearn_pca = sklearnPCA(n_components=2)
Y_sklearn = sklearn_pca.fit_transform(X_std)
Machine learning. Lecture 6. Feature Selection and Extraction. 15.06.2016. 41
Feature Extraction
PCA (Python example)
Let plot PCA-results
Machine learning. Lecture 6. Feature Selection and Extraction. 15.06.2016. 42
Feature Extraction
PCA (Weka)
PrincipalComponents pca = new PrincipalComponents();
pca.setInputFormat(trainingData);
pca.setMaximumAttributes(100);
newData = Filter.useFilter(newData, pca);
Machine learning. Lecture 6. Feature Selection and Extraction. 15.06.2016. 43
