Chapter 3
Logistic Regression
MSc. Nguyen Khanh Loi
[email protected]
8/2023
MSc Nguyen Khanh Loi
Content
Ø Classification
Ø Hypothesis Representation
Ø Decision boundary
Ø Cost function
Ø Simplified cost function and gradient descent
Ø Advanced optimization
Ø Multi-class classification: One-vs-all
2
MSc Nguyen Khanh Loi
Classification
3
MSc Nguyen Khanh Loi
Classification
Email: Spam / Not Spam?
Online Transactions: Fraudulent (Yes / No)?
Tumor: Malignant / Benign ?
0: “Negative Class” (e.g., benign tumor)
1: “Positive Class” (e.g., malignant tumor)
Y can only one of two values -> Binary classfication
4
MSc Nguyen Khanh Loi
Classification
(Yes) 1
Malignant ?
(No) 0
Tumor Size Tumor Size
Threshold classifier output at 0.5:
If , predict “y = 1”
If , predict “y = 0”
5
MSc Nguyen Khanh Loi
Classification
Classification: y = 0 or 1
can be > 1 or < 0
Logistic Regression:
6
MSc Nguyen Khanh Loi
Hypothesis Representation
7
MSc Nguyen Khanh Loi
Hypothesis Representation
Logistic Regression Model
Want
Sigmoid function
Logistic function
8
MSc Nguyen Khanh Loi
Interpretation of Hypothesis Output
= estimated probability that y = 1 on input x
Example: If
Tell patient that 70% chance of tumor being malignant
“probability that y = 1, given x,
parameterized by ”
9
MSc Nguyen Khanh Loi
Decision boundary
10
MSc Nguyen Khanh Loi
Decision boundary
Logistic regression
Suppose predict “ “ if
predict “ “ if
11
MSc Nguyen Khanh Loi
Decision boundary
Logistic regression
When is ?
𝑔 𝑧 ≥ 0.5
𝑧≥0
𝜃 !𝑥 ≥ 0 𝜃 !𝑥 < 0
12
MSc Nguyen Khanh Loi
Decision boundary
2 features: x1, x2
Example: x2
𝜃! = −3 3
2
𝜃" = 1
1
𝜃# = 1
1 2 3 x1
−3 + 𝑥! + 𝑥" = 0
Predict “ “ if
13
MSc Nguyen Khanh Loi
Non-linear decision boundaries
x2
Choose: 𝜃! = −1, 𝜃" = 0, 𝜃# = 0, 𝜃$ = 1,
𝜃% = 1 1
ℎ& 𝑥 = 𝑔(−1 + 𝑥"# + 𝑥## ) -1 1 x1
-1
Decision boundary: 𝑧 = −1 + 𝑥"# + 𝑥## = 0
Predict “ “ if
14
MSc Nguyen Khanh Loi
Non-linear decision boundaries
x2 x2
x1 x1
15
MSc Nguyen Khanh Loi
Cost function
16
MSc Nguyen Khanh Loi
Cost function
m examples
Training set:
How to choose parameters ?
17
MSc Nguyen Khanh Loi
Cost function
Linear regression:
Linear regression Logistic regression
“convex” “non-convex”
18
MSc Nguyen Khanh Loi
Logistic regression cost function
If y = 1
0 1
19
MSc Nguyen Khanh Loi
Logistic regression cost function
If y = 0
0 1
20
MSc Nguyen Khanh Loi
Simplified cost function and gradient descent
21
MSc Nguyen Khanh Loi
Logistic regression cost function
maximum likelihood
To fit parameters :
To make a prediction given new :
Output
22
MSc Nguyen Khanh Loi
Gradient Descent
Want :
Repeat
(simultaneously update all )
23
MSc Nguyen Khanh Loi
Gradient Descent
Want :
Repeat
(simultaneously update all )
Algorithm looks identical to linear regression!
24
MSc Nguyen Khanh Loi
Advanced optimization
25
MSc Nguyen Khanh Loi
Optimization algorithm
Cost function . Want .
Given , we have code that can compute
-
- (for )
Gradient descent:
Repeat
26
MSc Nguyen Khanh Loi
Optimization algorithm
Given , we have code that can compute
-
- (for )
Optimization algorithms: Advantages:
- Gradient descent - No need to manually pick
- Conjugate gradient - Often faster than gradient
- BFGS descent.
- L-BFGS Disadvantages:
- More complex
27
MSc Nguyen Khanh Loi
Multi-class classification: One-vs-all
28
MSc Nguyen Khanh Loi
Multiclass classification
Email foldering/tagging: Work, Friends, Family, Hobby
Medical diagrams: Not ill, Cold, Flu
Weather: Sunny, Cloudy, Rain, Snow
29
MSc Nguyen Khanh Loi
Multiclass classification
Binary classification: Multi-class classification:
x2 x2
x1 x1
30
MSc Nguyen Khanh Loi
One-vs-all (one-vs-rest):
x2
x1
x2 x2
x1 x1
x2
Class 1:
Class 2:
Class 3:
x1
31
MSc Nguyen Khanh Loi
One-vs-all
Train a logistic regression classifier for each
class to predict the probability that .
On a new input , to make a prediction, pick the
class that maximizes
32
MSc Nguyen Khanh Loi
Practice 1
As an example of simple logistic regression, Suzuki et al. (2006) measured
sand grain size on 28 beaches in Japan and observed the presence or
absence of the burrowing wolf spider Lycosa ishikariana on each beach.
Grain size (mm) Spiders Grain size (mm) Spiders
0.245 absent 0.432 absent
0.247 absent 0.473 present
0.285 present 0.509 present
0.299 present 0.529 present
0.327 present 0.561 absent
0.347 present 0.569 absent
0.356 absent 0.594 present
0.36 present 0.638 present
0.363 absent 0.656 present
0.364 present 0.816 present
0.398 absent 0.853 present
0.4 present 0.938 present
0.409 absent 1.036 present
0.421 present 1.045 present
33
MSc Nguyen Khanh Loi
