Linear Regression: A Comprehensive Guide

Author: ChatGPT
Introduction to Linear Regression
Linear regression is a fundamental statistical method that models the relationship between a dependent
variable and one or more independent variables by fitting a linear equation to observed data. The
simplest form, simple linear regression, involves one independent variable and one dependent variable,
while multiple linear regression extends this to multiple predictors. The goal of linear regression is to
estimate the coefficients of the linear equation that best predict the dependent variable by minimizing
the difference between observed and predicted values.

Dating back to Francis Galton's work on heredity in the 19th century, linear regression has evolved
through rigorous mathematical formalization and computational advances. Early methods relied heavily
on manual calculations, but the advent of digital computers and sophisticated algorithms has enabled
researchers and practitioners to apply linear regression to large and complex datasets. Today, linear
regression remains both a teaching tool and a practical workhorse across fields such as economics,
engineering, and the natural sciences.

Linear regression is prized for its interpretability, computational efficiency, and ease of implementation.
It provides clear insights into how changes in predictor variables are associated with changes in the
response variable. Despite its simplicity, linear regression serves as the foundation for more advanced
modeling approaches, including generalized linear models and various machine learning algorithms.
Mathematical Formulation
At its core, the linear regression model posits that the dependent variable y can be expressed as a
linear combination of independent variables x and an error term ε. In the simplest case of simple linear
regression with a single predictor x, the model is written as:

y = β■ + β■ x + ε,

where β■ is the intercept, β■ is the slope coefficient, and ε represents the random error term assumed
to have zero mean. The parameters β■ and β■ are estimated from data to best fit the observed points.

Multiple linear regression extends this framework to p predictors. The model becomes:

y = β■ + β■ x■ + β■ x■ + ... + β■ x■ + ε.

Each coefficient β■ measures the effect on y of a one-unit change in the corresponding predictor x■,
holding other variables constant. This generalization allows for modeling complex relationships
involving several explanatory factors.

The matrix form of the linear regression model provides a compact representation. Let y be an n×1
vector of responses, X an n×(p+1) design matrix with a column of ones for the intercept, β a (p+1)×1
vector of coefficients, and ε an n×1 vector of errors. The model is written as:

y = Xβ + ε.

This notation facilitates derivations and computational implementations of estimation and inference
procedures.
Ordinary Least Squares Estimation
The most common method for estimating the parameters β in the linear regression model is Ordinary
Least Squares (OLS). OLS chooses β■ to minimize the sum of squared residuals:

SSR(β) = Σ■ (y■ - x■■β)²,

where y■ is the observed response and x■■ is the i-th row of the design matrix X. Minimizing SSR
leads to a set of normal equations that can be solved analytically.

The OLS solution in matrix form is given by:

β■ = (X■X)■¹ X■y,

provided that X■X is invertible. This closed-form solution is computationally efficient for moderate-sized
datasets and forms the basis for statistical inference in regression analysis.

In practice, numerical methods such as QR decomposition or singular value decomposition (SVD) are
often used to compute β■ in a numerically stable manner, especially when predictors are highly
correlated or when the design matrix is ill-conditioned.
Assumptions of OLS
For OLS estimates to have desirable properties (unbiasedness, efficiency, consistency), several key
assumptions must hold:

1. Linearity: The relationship between the predictors and the response is linear in parameters. 2.
Independence: The residuals are independent of each other. 3. Homoscedasticity: The residuals have
constant variance across observations. 4. No multicollinearity: Predictors are not perfectly collinear. 5.
Normality (for inference): Residuals are normally distributed.

Violation of these assumptions can lead to biased estimates, incorrect standard errors, and unreliable
hypothesis tests. Diagnostic checks and remedial measures should be applied when assumptions are
suspect.

Common techniques for addressing assumption violations include transforming variables, adding
interaction terms, or using weighted least squares when variance is non-constant.
Interpretation of Coefficients
The intercept β■ represents the expected value of y when all predictors are zero, provided that zero is
within the range of the data. The slope coefficient β■ quantifies the marginal effect of a one-unit change
in predictor x■ on the response variable y, holding other variables constant.

The coefficient of determination, R², measures the proportion of variance in the dependent variable that
is predictable from the independent variables:

R² = 1 - SSR/SST,

where SST is the total sum of squares. An R² close to 1 indicates a model that explains a large portion
of the variability in the data, while an R² near 0 suggests poor explanatory power.

Adjusted R² accounts for the number of predictors in the model and penalizes the addition of
uninformative variables, providing a more reliable measure when comparing models with different
numbers of predictors.
Inference and Hypothesis Testing
Statistical inference in linear regression involves testing hypotheses about the model parameters and
constructing confidence intervals. The t-statistic for testing H■: β■ = 0 is computed as:

t = β■■ / SE(β■■),

where SE(β■■) is the standard error of the estimated coefficient. Under the null hypothesis and OLS
assumptions, the t-statistic follows a t-distribution with n - p - 1 degrees of freedom.

The overall significance of the regression model is assessed using the F-test, which compares the fit of
the full model to a reduced model containing only the intercept:

F = [(SSR_reduced - SSR_full)/p] / [SSR_full/(n - p - 1)].

A large F-value indicates that the model provides a significantly better fit than the null model.

Confidence intervals for coefficients are constructed as:

β■■ ± t_{α/2, n-p-1} × SE(β■■),

providing a range of plausible values for the true parameter β■ at a specified confidence level.
Diagnostics and Model Selection
Model diagnostics are crucial for validating the assumptions and performance of a linear regression
model. Residual plots, such as residuals vs. fitted values or Q-Q plots, help detect non-linearity,
heteroscedasticity, and departures from normality.

Multicollinearity among predictors can inflate standard errors and destabilize coefficient estimates. The
Variance Inflation Factor (VIF) quantifies multicollinearity; VIF values exceeding 5 or 10 warrant
investigation and potential remedies such as removing or combining variables.

Model selection criteria, including Akaike Information Criterion (AIC) and Bayesian Information Criterion
(BIC), balance goodness of fit with model complexity. Lower AIC or BIC values indicate more
parsimonious models that avoid overfitting.
Multiple Linear Regression
Multiple linear regression generalizes simple linear regression to include multiple predictor variables.
This allows for modeling complex relationships and controlling for confounding factors.

The estimation of coefficients in multiple regression follows the same OLS principles, using the matrix
solution β■ = (X■X)■¹ X■y. Interpretation of individual coefficients requires holding other predictors
constant.

Interaction terms and polynomial expansions can be incorporated to model non-linear relationships.
Care must be taken to avoid overfitting by using cross-validation and ensuring that the model
complexity is appropriate for the available data.
Regularization Techniques
When predictors are highly correlated or when overfitting is a concern, regularization techniques
introduce penalty terms to the OLS objective. Ridge regression penalizes the sum of squared
coefficients:

β■_ridge = argmin (SSR + λ Σ■ β■²),

where λ ≥ 0 controls the strength of the penalty.

Lasso regression applies an L1 penalty, encouraging sparsity:

β■_lasso = argmin (SSR + λ Σ■ |β■|),

which can set some coefficients exactly to zero, facilitating variable selection.

Elastic Net combines L1 and L2 penalties, balancing between ridge and lasso to handle correlated
predictors and enforce sparsity simultaneously.
Applications and Case Study
To illustrate linear regression, consider a case study predicting house prices based on features such as
square footage, number of bedrooms, and age of the property. Data is collected, cleaned, and split into
training and testing sets. The model is fitted using OLS on the training data, and performance is
evaluated on the test set.

Key steps include feature scaling, handling categorical variables via one-hot encoding, and diagnosing
model fit using residual analysis. Performance metrics such as Mean Squared Error (MSE) and R²
quantify the model’s predictive accuracy.

Conclusions drawn from the case study demonstrate how linear regression provides interpretable
relationships between predictors and response, guiding decision-making in real estate valuation and
beyond.