What is Linear Regression?
Linear Regression is a supervised learning algorithm in machine
learning, which is widely used for solving regression problems.
Regression is a type of machine learning problem where the goal is to
predict a continuous output variable based on one or more input
variables.
In Linear Regression, the goal is to find the best-fitting linear equation
to describe the relationship between the input variables (also known
as predictors or features) and the output variable (also known as the
response variable).
The equation for a simple linear regression model can be written as
follows:
y = b0 + b1 * x
Here, y is the dependent variable (the variable we are trying to
predict), x is the independent variable (the predictor or feature), b0 is
the intercept term (the value of y when x is zero), and b1 is the slope
coefficient (the change in y for a unit change in x).
The goal of Linear Regression is to find the best values for b0 and b1
such that the line best fits the data points, minimizing the errors or
the difference between the predicted values and the actual values.
Types of Linear Regression?
There are two main types of Linear Regression models: Simple Linear
Regression and Multiple Linear Regression.
Simple Linear Regression: In simple linear regression, there is only
one independent variable (also known as the predictor or feature) and
one dependent variable (also known as the response variable). The
goal of simple linear regression is to find the best-fitting line to
describe the relationship between the independent and dependent
variable. The equation for a simple linear regression model can be
written as:
Y = b0 + b1 * X
Here, Y is the dependent variable, X is the independent variable, b0 is
the intercept term, and b1 is the slope coefficient.
Multiple Linear Regression: In multiple linear regression, there are
multiple independent variables and one dependent variable. The goal
of multiple linear regression is to find the best-fitting line to describe
the relationship between the independent variables and the
dependent variable. The equation for a multiple linear regression
model can be written as:
Y = b0 + b1 * X1 + b2 * X2 + … + bn * Xn
Here, Y is the dependent variable, X1, X2, …, Xn are the independent
variables, b0 is the intercept term, and b1, b2, …, bn are the slope
coefficients.
In both types of linear regression, the goal is to find the best values
for the intercept and slope coefficients that minimize the difference
between the predicted values and the actual values. Linear regression
is widely used in many real-world applications, such as finance,
marketing, and healthcare, for predicting outcomes such as stock
prices, customer behavior, and patient outcomes.
Linear Regression Line
In machine learning, a regression line can show two types of
relationships between the input variables (also known as predictors or
features) and the output variable (also known as the response
variable) in a linear regression model.
● Positive Relationship: A positive relationship exists
between the input variables and the output variable when
the slope of the regression line is positive. In other words, as
the values of the input variables increase, the value of the
output variable also increases. This can be seen as an
upward slope on a scatter plot of the data.
● Negative Relationship: A negative relationship exists
between the input variables and the output variable when
the slope of the regression line is negative. In other words,
as the values of the input variables increase, the value of the
output variable decreases. This can be seen as a downward
slope on a scatter plot of the data.
Finding the best fit line
In machine learning, finding the best-fitting line is crucial in linear
regression, as it determines the accuracy of the predictions made by
the model. The best-fitting line is the line that has the smallest
difference between the predicted values and the actual values.
To find the best-fitting line in a linear regression model, we use a
process called “ordinary least squares (OLS) regression”. This process
involves calculating the sum of the squared differences between the
predicted values and the actual values for each data point, and then
finding the line that minimizes this sum of squared errors.
The best-fitting line is found by minimizing the residual sum of
squares (RSS), which is the sum of the squared differences between
the predicted values and the actual values. This is achieved by
adjusting the values of the intercept and slope coefficients, also
known as c and m, respectively.
Once the values of c and m are determined, we can use the linear
regression equation to make predictions for new data points. The
equation for a simple linear regression model can be written as:
y=c+m*x
Here, y is the dependent variable (the variable we are trying to
predict), x is the independent variable (the predictor or feature), c is
the intercept term (the value of y when x is zero), and m is the slope
coefficient (the change in y for a unit change in x).
In multiple linear regression, the equation would have more
independent variables, and the slope coefficients for each variable
would be included in the equation.
Overall, finding the best-fitting line in a linear regression model is
critical for accurate predictions and is achieved by minimizing the
residual sum of squares using the OLS regression method.
Gradient Descent : Linear Regression
In this tutorial you can learn how the gradient descent algorithm
works and implement it from scratch in python. First we look at what
linear regression is, then we define the loss function. We learn how
the gradient descent algorithm works and finally we will implement it
on a given data set and make predictions.
Linear Regression
In statistics, linear regression is a linear approach to modeling
the relationship between a dependent variable and one or more
independent variables. Let X be the independent variable and Y
be the dependent variable. We will define a linear relationship
between these two variables as follows:
This is the equation for a line that you studied in high school. m
is the slope of the line and c is the y intercept. Today we will use
this equation to train our model with a given dataset and predict
the value of Y for any given value of X. Our challenge today is to
determine the value of m and c, such that the line corresponding
to those values is the best fitting line or gives the minimum
error.
Loss Function
The loss is the error in our predicted value of m and c. Our goal
is to minimize this error to obtain the most accurate value of m
and c.
We will use the Mean Squared Error function to calculate the
loss. There are three steps in this function:
1. Find the difference between the actual y and predicted
y value(y = mx + c), for a given x.
2. Square this difference.
3. Find the mean of the squares for every value in X.
Mean Squared Error Equation
Here yᵢ is the actual value and ȳᵢ is the predicted
value. Lets substitute the value of ȳᵢ:
Substituting the value of ȳᵢ
So we square the error and find the mean. hence the name Mean
Squared Error. Now that we have defined the loss function, lets
get into the interesting part — minimizing it and finding m and
c.
The Gradient Descent Algorithm
Gradient descent is an iterative optimization algorithm to find
the minimum of a function. Here that function is our Loss
Function.
Understanding Gradient Descent
Imagine a valley and a person with no sense of direction who
wants to get to the bottom of the valley. He goes down the slope
and takes large steps when the slope is steep and small steps
when the slope is less steep. He decides his next position based
on his current position and stops when he gets to the bottom of
the valley which was his goal.
Let’s try applying gradient descent to m and c and approach it
step by step:
1. Initially let m = 0 and c = 0. Let L be our learning rate.
This controls how much the value of m changes with
each step. L could be a small value like 0.0001 for good
accuracy.
2. Calculate the partial derivative of the loss function with
respect to m, and plug in the current values of x, y, m
and c in it to obtain the derivative value D.
Derivative with respect to m
Dₘ is the value of the partial derivative with respect to m.
Similarly lets find the partial derivative with respect to c, Dc :
Derivative with respect to c
3. Now we update the current value of m and c using the
following equation:
4. We repeat this process until our loss function is a very small
value or ideally 0 (which means 0 error or 100% accuracy). The
value of m and c that we are left with now will be the optimum
values.
Now going back to our analogy, m can be considered the current
position of the person. D is equivalent to the steepness of the
slope and L can be the speed with which he moves. Now the new
value of m that we calculate using the above equation will be his
next position, and L×D will be the size of the steps he will take.
When the slope is more steep (D is more) he takes longer steps
and when it is less steep (D is less), he takes smaller steps.
Finally he arrives at the bottom of the valley which corresponds
to our loss = 0.
Now with the optimum value of m and c our model is ready to
make predictions !
Implementing the Model
Now let’s convert everything above into code and see our model
in action !
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
plt.rcParams['figure.figsize'] = (12.0,
9.0)
# Preprocessing Input data
data = pd.read_csv('data.csv')
X = data.iloc[:, 0]
Y = data.iloc[:, 1]
plt.scatter(X, Y)
plt.show()
# Building the model
m = 0
c = 0
L = 0.0001 # The learning Rate
epochs = 1000 # The number of iterations to perform
gradient descent
n = float(len(X)) # Number of elements in X
# Performing Gradient Descent
for i in range(epochs):
Y_pred = m*X + c # The current predicted value of Y
D_m = (-2/n) * sum(X * (Y - Y_pred)) # Derivative
wrt m
D_c = (-2/n) * sum(Y - Y_pred) # Derivative wrt c
m = m - L * D_m # Update m
c = c - L * D_c # Update c
print (m, c)
1.4796491688889395 0.10148121494753726
# Making predictions
Y_pred = m*X + c
plt.scatter(X, Y)
plt.plot([min(X), max(X)], [min(Y_pred), max(Y_pred)],
color='red') # regression line
plt.show()
Gradient descent is one of the simplest and widely used
algorithms in machine learning, mainly because it can be applied
to any function to optimize it. Learning it lays the foundation to
mastering machine learning.
