Chapter 2

Linear regression
MSc. Nguyen Khanh Loi
[email protected]
8/2023
MSc Nguyen Khanh Loi
Content

Ø Linear regression with one variable

Ø Cost function
Ø Gradient descent
Ø Linear Regression with multiple variables

2 Linear regression
MSc Nguyen Khanh Loi
Linear regression with one variable

3 Linear regression
MSc Nguyen Khanh Loi
Linear regression with one variable

Supervised Learning Regression Problem

Given the “right answer” for Predict real-valued output
each example in the data.

4 Linear regression
MSc Nguyen Khanh Loi
Linear regression with one variable

Training set of Size in feet2 (x) Price ($) in 1000's (y)

housing prices 2104 460
(Portland, OR) 1416 232
1534 315
852 178
… …
Notation:
m = Number of training examples
x’s = “input” variable / features
y’s = “output” variable / “target” variable

5 Linear regression
MSc Nguyen Khanh Loi
 Linear regression with one variable

Training Set How do we represent h ?

Learning Algorithm

Size of h Estimated
house price

Linear regression with one variable.

Univariate linear regression.

6 Linear regression
MSc Nguyen Khanh Loi
Cost function

7 Linear regression
MSc Nguyen Khanh Loi
Cost function

Size in feet2 (x) Price ($) in 1000's (y)

Training Set
2104 460
1416 232
1534 315
852 178
… …

Hypothesis:
‘s: Parameters
How to choose ‘s ?

8 Linear regression
MSc Nguyen Khanh Loi
 Cost function

3 3 3

2 2 2

1 1 1

0 0 0
0 1 2 3 0 1 2 3 0 1 2 3

9 Linear regression
MSc Nguyen Khanh Loi
Cost function

Idea: Choose so that

is close to for our
training examples

10 Linear regression
MSc Nguyen Khanh Loi
 Cost function

Simplified
Hypothesis:

Parameters:

Cost Function:

Goal:

11 Linear regression
MSc Nguyen Khanh Loi
 Cost function - Simplified

(for fixed , this is a function of x) (function of the parameter )

3 3

2 2
y
1 1

0 0
0 1 2 3 -0,5 0 0,5 1 1,5 2 2,5
x

12 Linear regression
MSc Nguyen Khanh Loi
 Cost function - Simplified

(for fixed , this is a function of x) (function of the parameter )

3 3

2 2
y
1 1

0 0
0 1 2 3 -0,5 0 0,5 1 1,5 2 2,5
x

13 Linear regression
MSc Nguyen Khanh Loi
 Cost function - Simplified

(for fixed , this is a function of x) (function of the parameter )

3 3

2 2
y
1 1

0 0
0 1 2 3 -0,5 0 0,5 1 1,5 2 2,5
x

14 Linear regression
MSc Nguyen Khanh Loi
Cost function

Hypothesis:

Parameters:

Cost Function:

Goal:

15 Linear regression
MSc Nguyen Khanh Loi
 Cost function

(for fixed , this is a function of x) (function of the parameters )

500

400
Price ($) 300
in 1000’s
200

100

0
0 1000 2000 3000
Size in feet2 (x)

16 Linear regression
MSc Nguyen Khanh Loi
Cost function

17 Linear regression
MSc Nguyen Khanh Loi
Cost function

(for fixed , this is a function of x) (function of the parameters )

18 Linear regression
MSc Nguyen Khanh Loi
Cost function

(for fixed , this is a function of x) (function of the parameters )

19 Linear regression
MSc Nguyen Khanh Loi
Cost function

(for fixed , this is a function of x) (function of the parameters )

20 Linear regression
MSc Nguyen Khanh Loi
Cost function

(for fixed , this is a function of x) (function of the parameters )

21 Linear regression
MSc Nguyen Khanh Loi
Gradient descent

22 Linear regression
MSc Nguyen Khanh Loi
 Gradient descent

How to find x*?

23 Linear regression
MSc Nguyen Khanh Loi
 Gradient descent

How to find x*?

- If then xt is
to the right x*

η: eta – Learning rate

24 Linear regression
MSc Nguyen Khanh Loi
Gradient descent

It is not as easy to find the solution of a zero derivative

import math
import numpy as np
import matplotlib.pyplot as plt

def grad(x):
return 2*x+ 5*np.cos(x) (x1, it1) = myGD1(.1, -5) #x0 = -5
(x2, it2) = myGD1(.1, 5) #x0 = 5
print('x1 = %f, cost = %f, after %d'%(x1[-1], cost(x1[-1]), it1))
def cost(x):
print('x2 = %f, cost = %f, after %d'%(x2[-1], cost(x2[-1]), it2))
return x**2 + 5*np.sin(x)
x1 = -1.110667, cost = -3.246394, after 11
def myGD1(eta, x0): x2 = -1.110341, cost = -3.246394, after 29
x = [x0]
for it in range(100):
x_new = x[-1] - eta*grad(x[-1])
if abs(grad(x_new)) < 1e-3:
break
x.append(x_new)
return (x, it)

25 Linear regression
MSc Nguyen Khanh Loi
Gradient descent

It is not as easy to find the solution of a zero derivative

26 Linear regression
MSc Nguyen Khanh Loi
Gradient descent

It is not as easy to find the solution of a zero derivative

Change learning rate

27 Linear regression
MSc Nguyen Khanh Loi
Gradient descent

Summary:
- If is too small: slow convergence.
- If is too large: may not decrease on every iteration;
may not converge.

To choose learning rate, try:

….., 0.001, …., 0.01, ….., 0.1, ….., 1

28 Linear regression
MSc Nguyen Khanh Loi
Gradient descent

Have some function

Want

Outline:
• Start with some
• Keep changing to reduce
until we hopefully end up at a minimum

29 Linear regression
MSc Nguyen Khanh Loi
Gradient descent

J(q0,q1)

q1
q0

30 Linear regression
MSc Nguyen Khanh Loi
Gradient descent

J(q0,q1)

q1
q0

31 Linear regression
MSc Nguyen Khanh Loi
Gradient descent

Gradient descent algorithm

Correct: Simultaneous update Incorrect:

32 Linear regression
MSc Nguyen Khanh Loi
Gradient descent

Gradient descent algorithm Linear Regression Model

33 Linear regression
MSc Nguyen Khanh Loi
Gradient descent

Gradient descent algorithm

update
and
simultaneously

34 Linear regression
MSc Nguyen Khanh Loi
Gradient descent

(for fixed , this is a function of x) (function of the parameters )

35 Linear regression
MSc Nguyen Khanh Loi
Gradient descent

(for fixed , this is a function of x) (function of the parameters )

36 Linear regression
MSc Nguyen Khanh Loi
Gradient descent

(for fixed , this is a function of x) (function of the parameters )

37 Linear regression
MSc Nguyen Khanh Loi
Gradient descent

(for fixed , this is a function of x) (function of the parameters )

38 Linear regression
MSc Nguyen Khanh Loi
Gradient descent

(for fixed , this is a function of x) (function of the parameters )

39 Linear regression
MSc Nguyen Khanh Loi
Gradient descent

(for fixed , this is a function of x) (function of the parameters )

40 Linear regression
MSc Nguyen Khanh Loi
Gradient descent

(for fixed , this is a function of x) (function of the parameters )

41 Linear regression
MSc Nguyen Khanh Loi
Gradient descent

(for fixed , this is a function of x) (function of the parameters )

42 Linear regression
MSc Nguyen Khanh Loi
Gradient descent

(for fixed , this is a function of x) (function of the parameters )

43 Linear regression
MSc Nguyen Khanh Loi
Linear Regression with multiple variables

44 Linear regression
MSc Nguyen Khanh Loi
Multiple features

Multiple features (variables).

Size (feet2) Price ($1000)

2104 460
1416 232
1534 315
852 178
… …

45 Linear regression
MSc Nguyen Khanh Loi
Multiple features

Multiple features (variables).

Size (feet2) Number of Number of Age of home Price ($1000)
bedrooms floors (years)

2104 5 1 45 460
1416 3 2 40 232
1534 3 2 30 315
852 2 1 36 178
… … … … …

Notation:
= number of features
= input (features) of training example.
= value of feature in training example.

46 Linear regression
MSc Nguyen Khanh Loi
Multiple features

Hypothesis:
Previously:

For convenience of notation, define .

𝑥! 𝜃!
𝑥" 𝜃"
𝑋 = 𝑥# 𝜃 = 𝜃#
⋮ ⋮
𝑥$ 𝜃$

47 Linear regression
MSc Nguyen Khanh Loi
Gradient descent for multiple variables

Hypothesis:
Parameters:
Cost function:

Gradient descent:
Repeat

(simultaneously update for every )

48 Linear regression
MSc Nguyen Khanh Loi
 Gradient descent for multiple variables

New algorithm :
Gradient Descent
Repeat
Previously (n=1):
Repeat
(simultaneously update for
)

(simultaneously update )

49 Linear regression
MSc Nguyen Khanh Loi
Feature Scaling

50 Linear regression
MSc Nguyen Khanh Loi
 Feature Scaling

E.g. = size (0-2000 feet2)

= number of bedrooms (1-5)

House: 𝑥! = 2000, 𝑥" = 5, 𝑝𝑟𝑖𝑐𝑒 = $500𝑘 size of the prameters 𝜃# , 𝜃! , 𝜃"

E.g. 𝜃# = 50, 𝜃! = 50, 𝜃" = 0.1 E.g. 𝜃# = 50, 𝜃! = 0.1, 𝜃" = 50

51 Linear regression
MSc Nguyen Khanh Loi
 Feature Scaling

E.g. = size (0-2000 feet2) , = number of bedrooms (1-5)

size of feature 𝑥! size of parameter 𝜃!

size in feet2
#bedrooms

𝜃" Parameters

100

10
0 1 𝜃!

53 Linear regression
MSc Nguyen Khanh Loi
Feature Scaling
Feature size and gradient descent

𝜃" Parameters

100

10
0 1 𝜃!

54 Linear regression
MSc Nguyen Khanh Loi
 Feature Scaling
Feature size and gradient descent
Feature Scaling
Idea: Make sure features are on a similar scale.
Parameters
rescaled 𝜃"

0 𝜃!

rescaled

55 Linear regression
MSc Nguyen Khanh Loi
Feature Scaling

Feature Scaling
Idea: Make sure features are on a similar scale.
E.g. = size (0-2000 feet2) size (feet2)
= number of bedrooms (1-5)
number of bedrooms

Mean normalization
Replace with to make features have approximately zero mean
(Do not apply to ).
E.g.

57 Linear regression
MSc Nguyen Khanh Loi
Practice 1

Thời gian học (h) Điểm thi Thời gian học (h) Điểm thi

155 51 171 63
180 52 170 76
164 54 181 64
162 53 182 66
181 55 189 69
182 59 184 72
173 61 209 70
190 59 210 80

58 Linear regression
MSc Nguyen Khanh Loi
Practice 2

Build a model which predicts sales based on the money spent on

different platforms for marketing.

59 Linear regression
MSc Nguyen Khanh Loi