.
BDIJOF-FBSOJOH
1FSGPSNBODF.FUSJDT ityof Bergamo
Performance metrics
Outline
1. Metrics
2. Precision and recall
3. Receiver Operating Characteristic (ROC) curves
4. Worked example
2 /31
Outline
1. Metrics
2. Precision and recall
3. Receiver Operating Characteristic (ROC) curves
4. Worked example
3 /31
Metrics
It is extremely important to use quantitative metrics for evaluating a machine learning
model
Until now, we relied on the cost function value for regression and classification
Other metrics can be used to better evaluate and understand the model
For classification
Accuracy/Precision/Recall/F1-score, ROC curves,…
For regression
Normalized RMSE, Normalized Mean Absolute Error (NMAE),…
4 /31
Accuracy
Accuracy is a measure of how close a given set of guessing from our model are closed
to their true value.
(
If a classifier make 10 predictions and 9 of them are correct, the accuracy is 90%.
Accuracy is a measure of how well a binary classifier correctly identifies or excludes
a condition.
It’s the proportion of correct predictions among the total number of cases
examined.
5 /31
Classification case: metrics for skewed classes
Disease dichotomic classification example
Train logistic regression model " , with ! ( if disease, ! ( otherwise.
Find that you got error on test set ( correct diagnoses)
The " class has very few samples with
Only of patients actually have disease
respect to the " class
If I use a classifier that always classifies the observations to the % class, I get of
accuracy!!
For skewed classes, the accuracy metric can be deceptive
6 /31
Outline
1. Metrics
2. Precision and recall
3. Receiver Operating Characteristic (ROC) curves
4. Worked example
7 /31
Precision and recall
Suppose that ! ( in presence of a rare class that we want to detect
Precision (How much we are precise in the detection)
Of all patients where we classified , Confusion matrix
what fraction actually has the disease?
Actual class
"
!
Estiamted class
1 (p) 0 (n)
True positive False positive
Recall (How much we are good at detecting) 1 (Y)
(TP) (FP)
Of all patients that actually have the disease, what
fraction did we correctly detect as having the disease? False negative True negative
0 (N)
(FN) (TN)
"
!
8 /31
F1-score
It is usually better to compare models by means of one number only. The & ' can
be used to combine precision and recall
Precision(P) Recall (R) Average F1 Score
Algorithm 1 0.5 0.4 0.45 0.444 The best is Algorithm 1
Algorithm 2 0.7 0.1 0.4 0.175
Algorithm 3 0.02 1.0 0.51 0.0392
Algorithm 3 classifies always Average says not correctly
that Algorithm 3 is the best
10 /31
Summaries of the confusion matrix
Different metrics can be computed from the confusion matrix, depending on the class of
interest (https://en.wikipedia.org/wiki/Precision_and_recall)
11 /31
Outline
1. Metrics
2. Precision and recall
3. Receiver Operating Characteristic (ROC) curves
4. Worked example
12 /31
Ranking instead of classifying
Classifiers such as logistic regression can output a probability of belonging to a class (or
something similar)
We can use this to rank the different istances and take actions on the cases at top of
the list
We may have a budget, so we have to target most promising individuals
Ranking enables to use different techniques for visualizing model performance
13 /31
Ranking instead of classifying
p n
Y 0 0 p n
Instance 1 0
True class Score N 100 100 Y
description
99 100
…………… 0,99 N
…………… 0,98 Different confusion
…………… 0,96 p n matrices by changing
…………… 0,90 Y
2 0 the threshold
…………… 0,88 N 98 100
p n
…………… 0,87 Y
2 1
…………… 0,85 98 99
N
…………… 0,80 p n
…………… 0,70 Y
6 4
94 96
N
14 /31
Ranking instead of classifying
ROC curves are a very general way to represent and compare the performance of
different models (on a binary classification task)
Perfection Observations
classify always negative
Recall (True Positive Rate)
Random classify always positive
Better guessing
classifier : random classifier
Worse
classifier : worse than random classifier
Different classifiers can be compared
Area Under the Curve (AUC): probability that a
randomly chosen positive instance will be ranked
1 – specificity (False Positive Rate) ahead of randomly chosen negative instance
15 /31
Outline
1. Metrics
2. Precision and recall
3. Receiver Operating Characteristic (ROC) curves
4. Worked examples
16 /31
Breast cancer detection
Breast cancer is the most common cancer amongst women in the world.
It accounts for 25% of all cancer cases, and affected over 2.1 Million people in 2015
alone.
It starts when cells in the breast begin to grow out of control. These cells usually
form tumors that can be seen via X-ray or felt as lumps in the breast area.
The key challenges against it’s detection is how to classify tumors into malignant
(cancerous) or benign(non cancerous).
Goal: classifying these tumors using machine learning and the Breast Cancer
Wisconsin (Diagnostic) Dataset.
17 /31
Breast cancer Wisconsin dataset Output:
This dataset has been referred from Kaggle. Class 4 stands for malignant cancer
Class 2 stands for benign cancer
Uniformity Single
Clump Uniformity of Cell Marginal Epithelial Bare Bland Normal
id_num Thickness of Cell Size Shape Adhesion Cell Size Nuclei Chromatin Nucleoli Mitoses Class
1041801 5 3 3 3 2 3 4 4 1 4
1043999 1 1 1 1 2 3 3 1 1 2
1044572 8 7 5 10 7 9 5 5 4 4
1047630 7 4 6 4 6 1 4 3 1 4
1048672 4 1 1 1 2 1 2 1 1 2
1049815 4 1 1 1 2 1 3 1 1 2
1050670 10 7 7 6 4 10 4 1 2 4
……… …… …… …… …… …… …… ……. ……. ……. …….
18 /31
Breast cancer detection
We will use the dataset to compare differente logistic regression models by means
of the ROC curve associated to each of them.
To this aim we will work with 4 different dataset (plus an extra one)
1. Case 1: the whole dataset
2. Case 2: the first group of 5 features
3. Case 3: the second group of 5 features
4. Case 4: only the first two features
Extra: after learning the model of CASE 1, take only the features with the smallest
p-value.
19 /31
%% Load and clean data
Matlab code data = readtable('breast_cancer_w.xlsx'); %load our data as a table
Phi=table2array(data(:,1:end-1));
y=table2array(data(:,end));
Output:
y(y==4)=1; % in the original date 4 stands for malignant cancer
Class 4 stands for malignant y(y==2)=0; % in the original date 2 stands for benign cancer
cancer and it is for us the positive % Setup the data matrix appropriately, and add ones for the intercept
output. We set it to 1 term
[N, d] = size(Phi);
Class 2 stands for benign cancer Phi = [ones(N, 1) Phi]; % Add intercept term
and it is for us the negative %% Train and test data
output. We set it to 0. mdl = fitglm(Phi,y,'Distribution','binomial','Link','logit')
%% ============ Part 2: Compute the ROC curve ============
scores = mdl.Fitted.Probability;
[X,Y,T,AUC] = perfcurve(y,scores,1);
%Plot the ROC curve.
perfcurve compute the points in figure
plot(X,Y)
the ROC curve as well as the AUC xlabel('False positive rate')
ylabel('True positive rate')
title('ROC for Classification by Logistic Regression')
20 /31
Results
Comparison of case 1, 2, 3 and 4
Using only the first 2 features is nott
a smart choice.
21 /31
Results
Comparison of case 1, 4 and best
Using only the best features
provides a model that performs
almost as well as using all the
features
22 /31
Pneumonia detection
Suppose to have at disposal X-ray images of lungs: Healthy people - Covid-19 disease
patients
23 /31
Acknowledgments
The COVID-19 X-ray image is curated by Dr. Joseph Cohen, a postdoctoral fellow at
the University of Montreal, see https://josephpcohen.com/w/public-covid19-dataset/
The previous data contain only X-ray images of people with a disease. To collect
images of healthy people, we can download another X-ray dataset on the platform
Kaggle https://www.kaggle.com/paultimothymooney/chest-xray-pneumonia
The analysis is inspired from a tutorial by Adrian Rosebrock:
https://www.pyimagesearch.com/2020/03/16/detecting-covid-19-in-x-ray-images-with-keras-
tensorflow-and-deep-learning/
24 /31
Acknowledgments
We want to use a classifier to perform classification:
Healthy patients: class
Patients with a disease: class
The input data are directly the X-ray images
For these computer vision tasks, the state of the art algorithm are the Convolutional
Neural Networks:
we can use them to classify the images into healthy and disease
25 /31
Estimated covid label
Pneumonia detection True label
Estimated healthy label
26 /31
Pneumonia detection
Classification results on test set
Actual class
Estimated class
1 (p) 0 (n)
Sensitivity (recall, true positive rate)
True positive False positive
1 (Y)
11 0
False negative True negative
0 (N)
1 11
Specificity (true negative rate)
Accuracy: )
27 /31
Pneumonia detection
Classification results on test set
Sensitivity (recall, true positive rate) Specificity (true negative rate)
Sensitivity: of patients that do have COVID-19 (i.e., true positives), we could
accurately identify them as “COVID-19 positive” 92% of the time using our model
Specificity: of patients that do not have COVID-19 (i.e., true negatives), we could
accurately identify them as “COVID-19 negative” 100% of the time using our model.
28 /31
Pneumonia detection
Classification results on test set
Sensitivity (recall, true positive rate) Specificity (true negative rate)
Being able to accurately detect healthy patients with 100% accuracy is great. We do
not want to quarantine someone for nothing
…but we don’t want to classify someone as «healthy» when they are «COVID-19
positive», since it could infect other people without knowing
29 /31
Summary
Balancing sensitivity and specificity is incredibly challenging when it comes to medical
applications
The results should always be validated with another pool of people
Furthermore, we need to be concerned of what the model is actually learning:
Does the results align with the medical knowledge?
Was the dataset well representative of the population or there was selection bias?
30 /31
Summary
Furthermore, we need to be concerned of
what the model is actually learning:
Do we accounted for all external factors
(confounding) that could interfere with the
response?
31 /31
