1
Confusion Matrix & Performance Measurement
Metrics
Al Amin Biswas
Lecturer, CSE, DIU
Confusion Matrix
2
A confusion matrix for a two classes (+, -) is shown below.
There are four quadrants in the confusion matrix, which are symbolized as
below.
True Positive (TP: f++) : The number of instances that were positive (+) and
correctly classified as positive (+).
False Negative (FN: f+-): The number of instances that were positive (+) and
incorrectly classified as negative (-).
False Positive (FP: f-+): The number of instances that were negative (-) and
incorrectly classified as (+).
True Negative (TN: f--): The number of instances that were negative (-)
and correctly classified as (-).
Confusion Matrix
3
Note:
Np = TP (f++) + FN (f+-)
= is the total number of positive instances.
Nn = FP(f-+) + Tn(f--)
= is the total number of negative instances.
N = Np + Nn
= is the total number of instances.
(TP + TN) denotes the number of correct classification
(FP + FN) denotes the number of errors in classification.
4
Confusion Matrix Example
For example,
Class + -
+ 52 (TP) 18 (FN)
- 21 (FP) 123 (TN)
Calculate the performance evaluation metrics
5 Accuracy
It is defined as the fraction of the number of examples that are correctly
classified by the classifier to the total number of instances.
Accuracy
Performance Evaluation Metrics
6
We now define a number of metrics for the measurement of a classifier.
In our discussion, we shall make the assumptions that there are only two
classes: + (positive) and – (negative)
True Positive Rate (TPR): It is defined as the fraction of the positive
examples predicted correctly by the classifier.
This metrics is also known as Recall, Sensitivity or Hit rate.
False Positive Rate (FPR): It is defined as the fraction of negative examples
classified as positive class by the classifier.
Performance Evaluation Metrics
7
False Negative Rate (FNR): It is defined as the fraction of positive
examples classified as a negative class by the classifier.
True Negative Rate (TNR): It is defined as the fraction of negative
examples classified correctly by the classifier
This metric is also known as Specificity.
8
Performance Evaluation Metrics
Both, Precision and Recall are defined by
9
Performance Evaluation Metrics
F1 Score (F1): Recall (r) and Precision (p) are two
widely used metrics employed in analysis..
It is defined in terms of (r or Recall) and (p or Precision) as
follows.
Note
F1 represents the harmonic mean between recall and precision
High value of F1 score ensures that both Precision and Recall
are reasonably high.
Analysis with Performance Measurement Metrics
10
Based on the various performance metrics, we can characterize a classifier.
We do it in terms of TPR, FPR, Precision and Recall and Accuracy
Case 1: Perfect Classifier
When every instance is correctly classified, it is called the perfect classifier. In this case,
TP = P, TN = N and CM is
Predicted Class
TPR = TP/(TP+FN)= =1 + -
+ P 0
FPR = =0
Actual
class
Precision = = 1 - 0 N
F1 Score = = 1
Accuracy = = 1
Analysis with Performance Measurement Metrics
11
Case 2: Worst Classifier
When every instance is wrongly classified, it is called the worst classifier. In this
case, TP = 0, TN = 0 and the CM is
Predicted Class
+ -
TPR = =0
+ 0 P
Actual
FPR = = 1
class
- N 0
Precision = = 0
F1 Score = Not applicable
as Recall + Precision = 0
Accuracy = =0
Analysis with Performance Measurement Metrics
12
Case 3: Ultra-Liberal Classifier
The classifier always predicts the + class correctly. Here, the False Negative
(FN) and True Negative (TN) are zero. The CM is
Predicted Class
+ -
TPR = = 1
+ P 0
Actual
FPR = = 1
class
- N 0
Precision =
F1 Score =
Accuracy = =0
Analysis with Performance Measurement Metrics
13
Case 4: Ultra-Conservative Classifier
This classifier always predicts the - class correctly. Here, the False Negative
(FN) and True Negative (TN) are zero. The CM is
Predicted Class
+ -
TPR = = 0
+ 0 p
Actual
FPR = = 0
class
- 0 N
Precision =
(as TP + FP = 0)
F1 Score =
Accuracy = =0
