Multiple Linear Regression

Multiple Features (variables)

The original linear regression model uses a single feature (e.g., size of a house) to predict an outcome
(e.g., price).

By adding more features (e.g., number of bedrooms, floors, and age of the house), the model can make
more accurate predictions

Notation:

xj = j th feature
​

n=number of features
x(i) =features of ith training example
xj (i) =value of feature j in ith training example
​

💡 Examples:
x(2) = [1416 3 2 40]
(2) = 2
x3 ​

Model
Previous single variable model Updated multivariable model
fw,b (x) = wx + b
​
fw,b (x) = w1 x1 + … + wn xn + b
​ ​ ​ ​ ​

The model can be simplified using vectors of parameters and features:

w = [w1 w2 w3 … wn ]
​ ​ ​ ​

x = [x1 x2 x3 … xn ]
​ ​ ​ ​

b, however, is included in the complete model, which is described below.

Multiple Linear Regression 1

 fw,b (x) = w ⋅ x + b = w1 x1 + w2 x2 + … + wn xn + b
​ ​ ​ ​ ​ ​ ​

multiple linear regression (not multivariate function)

💡 NOTE: “⋅” is dot product.

Vectorization
Vectorization simplifies code by allowing operations on entire arrays or vectors at once, rather than using
loops.

It enhances performance by leveraging modern numerical libraries like NumPy, which can utilize parallel
processing capabilities of CPUs and GPUs.

Notice that the linear algebra is 1-based indexing, while the Python code is 0-based indexing.

Vector Python code

Parameters and features

w = [w1 w2 w3 ] ​ ​ ​

x = [x1 x2 x3 ] ​ ​ ​

b = 4

with w1 ​
= 1.0, w2 = 2.5, w3 = −3.3 ​ ​

and x1 ​ = 10, x2 = 20, x3 = 30 ​ ​

and bis a number.

Without vectorization

fw,b (x) = w1 x1 + w2 x2 + w3 x3 + b
​ ​ ​ ​ ​ ​ ​

Multiple Linear Regression 2

 fw,b (x) = (∑nj=1 wj xj ) + b
​ ​ ​ ​

Vectorization

fw,b (x) = w ⋅ x + b
​

Gradient Descent for Multiple Linear Regression

The cost function J is now a function of the vector w and the scalar b.

Pre-derived gradient descent algorithm

repeat until convergence:
∂
w = w − α ∂w J(w, b) ​


∂
b = b − α ∂b J(w, b)
​ 

Final gradient descent algorithm

wj = wj − α m1 ∑i=1 (fw,b (xi ) − y i )xj (i) 
m
​ ​ ​ ​ ​ ​

simultaneously update
m
b=b− α m1 ​ ∑i=1 (fw,b (xi )
​ ​ − y )
i
wj (for j = 1, … ,n) and b
​

Normal Equation Method

An alternative to gradient descent

Normal Equation:

Only for Linear Regression.

Solves for w and bwithout iterations.

source: GeeksForGeeks 30/05/2025

Disadvantage:

Doesn’t generalize to other learning algorithms.

Slow when the number of features is large (> 10,000)

An advanced linear algebra library is required, so engineers hardly utilize the normal equation directly.

It is usually used on the backend of machine learning libraries that implement linear regression.

For most learning algorithms, gradient descent is often the better way to get the job done

Multiple Linear Regression 3

Multiple Linear Regression 4