Scribd
Upload
29 views68 pages

TensorFlow Classification

The document discusses classification as a fundamental problem in machine learning, detailing various classifiers and their evaluation metrics such as accuracy, precision, and recall. It highlights the importance of understanding these metrics, especially in skewed datasets, and emphasizes the need for careful model selection and threshold tuning to optimize performance. Additionally, it introduces concepts like confusion matrices and ROC curves to aid in model evaluation and decision-making.

Uploaded by

Surya Bhoi
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
29 views68 pages

TensorFlow Classification

The document discusses classification as a fundamental problem in machine learning, detailing various classifiers and their evaluation metrics such as accuracy, precision, and recall. It highlights the importance of understanding these metrics, especially in skewed datasets, and emphasizes the need for careful model selection and threshold tuning to optimize performance. Additionally, it introduces concepts like confusion matrices and ROC curves to aid in model evaluation and decision-making.

Uploaded by

Surya Bhoi
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 68

Classification as a Machine Learning Problem

Over view

Classification is a canonical problem in Machine Learning

Classifiers can be measured using accuracy, precision and


recall

Traditional ML models for classification include SVM and


Naive Bayes

Neural networks perform very well on classification problems


Classification and Classifiers
Machine Learning

Work with a huge maze of Make intelligent decisions


Find patterns
data
Machine Learning

Emails on a server Spam or Ham? Trash or Inbox


Types of Machine Learning Problems

Classification Regression Clustering Rule-extraction


Types of Machine Learning Problems

Classification Regression Clustering Rule-extraction


Whales: Fish or Mammals?

Mammals Fish
Members of the infraorder Cetacea Look like fish, swim like fish, move with
fish
Whales: Fish or Mammals?

ML-based Classifier
ML-based Classifier

Training Prediction
Feed in a large corpus of data classified Use it to classify new instances which it
correctly has not seen before
Training the ML-based Classifier

Classification

ML-based Classifier
Corpus

Feedback - loss
Improves model parameters function or cost
function
An algorithm might have high accuracy but
still be a poor machine learning model

Its predictions are useless


Accuracy, Precision, Recall
All-is-well Binary Classifier

Medical reports
Always classify as No Cancer
“normal”

Here, accuracy for rare cancer may be 99.9999%, but…


Accuracy

Some labels maybe much more common/rare


than others

Such a dataset is said to be skewed

Accuracy is a poor evaluation metric here


Confusion Matrix
Predicted Labels
No Cancer
Cancer
Actual Label
10 instances 4 instances
Cancer

No Cancer 5 instances 1000 instances


Confusion Matrix
Predicted Labels
No Cancer
Cancer
Actual Label
10 4
Cancer

No Cancer 5 1000
True Positive
Predicted Labels
No Cancer
Cancer
Actual Label
10 4
Cancer

No Cancer 5 1000

Actual Label = Predicted Label


True Positive
Predicted Labels
No Cancer
Cancer
Actual Label

Cancer
10 TP 4

No Cancer 5 1000

Actual Label = Predicted Label


False Positive
Predicted Labels
No Cancer
Cancer
Actual Label
10 4
Cancer

No Cancer 5 1000

Actual Label =/ Predicted Label


False Positive
Predicted Labels
No Cancer
Cancer
Actual Label
10 4
Cancer

No Cancer 5
FP 1000

Actual Label =/ Predicted Label


True Positive
Predicted Labels
No Cancer
Cancer
Actual Label
10 4
Cancer

No Cancer 5 1000

Actual Label = Predicted Label


True Negative
Predicted Labels
No Cancer
Cancer
Actual Label
10 4
Cancer

No Cancer 5 1000 TN

Actual Label = Predicted Label


False Negative
Predicted Labels
No Cancer
Cancer
Actual Label
10 4
Cancer

No Cancer 5 1000

Actual Label =/ Predicted Label


False Negative
Predicted Labels
No Cancer
Cancer
Actual Label

Cancer
10 4 FN

No Cancer 5 1000

Actual Label =/ Predicted Label


Confusion Matrix
Predicted Labels
No Cancer
Cancer
Actual Label

Cancer
10 TP 4 FN

No Cancer 5 FP 1000 TN
Accuracy
Predicted Labels
No Cancer
Cancer
Actual Label

Cancer
10 TP 4 FN

No Cancer 5 FP 1000 TN
Accuracy
Predicted Labels
No Cancer
Cancer
Actual Label

Cancer
10 TP 4 FN

No Cancer 5 FP 1000 TN

Actual Label = Predicted Label


Accuracy
Predicted Labels
No Cancer
Cancer
Actual Label

Cancer
10 TP 4 FN

No Cancer 5 FP 1000 TN

Accuracy =
TP + TN
=
1010
= 99.12%
Num Instances 1019
Accuracy

Accuracy = 99.12%

Classifier gets it right 99.12% of the time

But…
Accuracy
Predicted Labels
No Cancer
Cancer
Actual Label

Cancer
10 TP 4 FN

No Cancer 5 FP 1000 TN

People on chemotherapy, radiation when not required


Accuracy
Predicted Labels
No Cancer
Cancer
Actual Label

Cancer
10 TP 4 FN

No Cancer 5 FP 1000 TN

Cancer not detected, no treatment prescribed


Accuracy is not a good metric to evaluate
whether this model performs well
Precision
Predicted Labels
No Cancer
Cancer
Actual Label

Cancer
10 TP 4 FN

No Cancer 5 FP 1000 TN
Precision
Predicted Labels
No Cancer
Cancer
Actual Label

Cancer
10 TP 4 FN

No Cancer 5 FP 1000 TN

Precision = Accuracy when classifier flags cancer


Precision
Predicted Labels
No Cancer
Cancer
Actual Label

Cancer
10 TP 4 FN

No Cancer 5 FP 1000 TN

TP 10
Precision = TP + FP = 15 = 66.67%
Precision = 66.67%
Precision
1 in 3 cancer diagnoses is incorrect
Recall
Predicted Labels
No Cancer
Cancer
Actual Label

Cancer
10 TP 4 FN

No Cancer 5 FP 1000 TN
Recall
Predicted Labels
No Cancer
Cancer
Actual Label

Cancer
10 TP 4 FN

No Cancer 5 FP 1000 TN

Recall = Accuracy when cancer actually present


Recall
Predicted Labels
No Cancer
Cancer
Actual Label

Cancer
10 TP 4 FN

No Cancer 5 FP 1000 TN

TP 10
Recall = TP + FN = 14 = 71.42%
Recall = 71.42%
Recall
2 in 7 cancer cases missed
Choosing a Machine Learning Model
ML-based Binary Classifier

Breathes like a mammal


Mammal
Gives birth like a mammal
ML-based Classifier

Corpus
ML-based Binary Classifier

Breathes like a mammal


P(fish) = 0.45
Gives birth like a mammal
ML-based Classifier

Corpus
Applying Logistic Regression
Probability of
animal being (95%)
fish Lives in water, breathes with gills, lays
eggs
(60%)

Lives in water, breathes with lungs,does not lay


eggs
Lives on land, breathes with lungs,does not lay
eggs
(5%) (40%)

Whales: Fish or Mammals?


Choosing Decision Threshold
(50%)
Probability of
animal being
fish
Pthreshold (80%)
(95%)

(60%)

(5%) (20%) (40%)


Choosing Decision Threshold
Probability of
animal being
fish
Pthreshold (80%)
(95%)

(60%)

(5%) (20%) (40%)

If probability < Pthreshold, it’s a mammal


Applying Logistic Regression
Probability of
animal being
fish
Pthreshold (80%)
(95%)

(60%)

(5%) (20%) (40%)

If probability > Pthreshold, it’s a fish


Predicted
No Cancer
Cancer
Actual TP FN

Cancer 0 14

FP TN
“Always No Cancer
0 1005
Negative”
Pthreshold =1 - Recall = 0%

- Precision = Infinite

- Classifier too conservative


Precision vs.“Conservativeness”
1.0 Precision

0
1.0
“Conservativeness” of Decision Threshold
Predicted
No Cancer
Cancer
Actual TP FN

Cancer 14 0

FP TN
“Always No Cancer
1005 0
Positive”
Pthreshold = 0 - Recall = 100%

- Precision = 14/1019 = 13.7%

- Classifier not conservative enough


Recall vs.“Conser vativeness”
1.0

Recall
0
1.0
“Conservativeness” of Decision Threshold
Precision-Recall Tradeoff
1.0 Precision

Recall
0
1.0
“Conservativeness” of Decision Threshold
Precision-Recall Tradeoff
1.0

Precision

Recall
Heuristics to Choose a Model

ROC Curve
F1 Score
Plot a curve to maximize true positives,
Harmonic mean of precision and recall
minimize false positives
Heuristics to Choose a Model

ROC Curve
F1 Score
Plot a curve to maximize true positives,
Harmonic mean of precision and recall
minimize false positives
Precision x Recall
F1 = 2x
Precision + Recall
F1 Score - Harmonic mean of precision, recall

- Closer to lower of two

- Favors even tradeoff


Choosing Pthreshold

Tweak threshold values


Calculate F1 Score
Run training by changing threshold values for
Each training run produces a model, calculate F1 score for each model
each execution

High F1 score better


Calculate precision, recall
Choose threshold which results in the highest F1
Find values for each training run
score
Heuristics to Choose a Model

ROC Curve
F1 Score
Plot a curve to maximize true positives,
Harmonic mean of precision and recall
minimize false positives
Choosing Pthreshold

True
Positive
Rate

False Positive
Rate
Choosing Pthreshold

Should be as high as
True possible
Positive
Rate

False Positive
Rate
Choosing Pthreshold

Should be as low as
True possible
Positive
Rate

False Positive
Rate
Choosing Pthreshold
ROC Curve

(Receiver Operating
Characteristic)

True
Positive
Rate

False Positive
Rate
Choosing Pthreshold
1.0

True
Positive Different values of Pthreshold
(Hyperparameter tuning)
Rate

False Positive
Rate
Choosing Pthreshold
1.0

True
Positive Fit ROC curve from different
Rate values of Pthreshold

False Positive
Rate
ROC Cur ve
1.0

True
Positive Pick top-left corner point as Pthreshold
Rate Why? Maximises True Positive Rate,
minimises False Positive Rate
0

False Positive
Rate
ROC of Perfect Classifier
1.0

TP = 100%

FP = 0%

True
Positive
Rate

False Positive
Rate
ROC of Random Classifier
1.0

TP = FP
True
Positive
Rate

False Positive
Rate

You might also like