NAME: AYANAVA
CHATTERJEE
ROLL: 14400120027
SUBJECT: MACHINE
LEARNING
TOPIC: Sparse modelling and
estimation
Introduction
In machine learning, a sparse model is a model that assumes that most
of the data is zero, or that the amount of relevant data is small. This
assumption allows you to get results even with a small amount of data.
Sparse modeling
The definition of sparse is come from the distribution. Sparse means
something that is small number or amount and spread out over an
area(or distribution). For example, sparse vectors and matrix have
most of zeros and only a few number of non-zero valued elements.
When creating a polynomial regression model(meaning that have
many terms of coefficients), the model can be expressed as a
vector of polynomial coefficients.
Sparse representation
y D x d
m
Overcomplete
Sparse Vector
Example
From Figure in previous slide:
y is the object we want to interpret. And DD is dictionary matrix to make yy,
and xx is sparse vector for deciding which vector is used from DD. So our
purpose is that minimizing the difference between YY and the matrix
multiplication of WW and XX. Also for the regularization, we can add
regularization term. As a result, criteria of this example can be expressed like
this:
minD,X∥Y−DX∥22+λ∥X∥0,∞colD,Xmin∥Y−DX∥22+λ∥X∥0,∞col
Example part 1
Previous formula, first term is the square of L2-norm(also known as Euclidean
distance),and second term is L0-norm(cardinality) of X. If the the number of
zeros in the sparse vector is large, the regularization term has small value. That
is, sparse vector becomes more sparse.
Let’s look at more precisely about the definition of norm. Actually, norm is the
length of vector in terms of pp. The mathematical expression of vector norm is
like this:
∥x∥p=(∑i=1d∣xi∣p)1/p=(∣x1∣p+⋯+∣xd∣p)1/p∥x∥p=(i=1∑d∣xi∣p)1/p=(∣x1∣p+⋯+∣xd∣p)1
/p
SPARSE MODELLING TECHNIQUES
L1 Regularization (Lasso): Adds a penalty proportional to the absolute values
of the coefficients to the loss function, encouraging sparsity by driving some
coefficients to zero.
Sparse Coding: Aims to represent data as a sparse combination of basis
functions, often used in unsupervised learning for feature extraction.
Compressed Sensing: A technique used in signal processing and machine
learning to recover signals from fewer measurements than traditionally
required, leveraging sparsity in the signal.
Feature Selection: Involves choosing a subset of relevant features for model
building, often guided by techniques such as statistical tests, recursive
feature elimination, or embedded methods.
Problem of Sparse Modelling
We already expressed the criteria of specific sparse modeling, but we
can generalize its form,
(D,X)min L(Y,D,X)+Ψ(X,D)
The first term is loss function for quantifying the goodness of
approximation. So it tries to minimize the error between YY and DXDX.
And second term is regularization term as explained, it evaluates
sparseness of coefficients(or dictionary). If this value is small, we can
say that the model is more sparse and explainable.
MODEL INTERPRETABILITY
Another advantage from sparsity is variable
selection. If we define a model as sparse, it means
that some dependent variables represented by 0
are not used to predict. So we can analyse the
performance of model from selected variable.
That is called model interpretability.
Benefits of sparse modelling
Here are some benefits of sparse models in machine learning:
Fast and energy efficient: Sparse models can be lightweight, with low
power consumption and high speeds.
Explainable: Sparse models can provide explainable AI solutions.
Applicable to a variety of tasks: Sparse models can be used for tasks
like image denoising, segmentation, restoration, and superresolution.
Can be integrated into existing systems: Sparse models can be
integrated into existing systems because of their low processing loads.
Challenges
Choosing the Right Sparsity Level: Too much sparsity can lead to
underfitting, while too little might not achieve the desired efficiency
Computational Complexity: While sparse models can be efficient,
some techniques for finding sparse solutions can be
computationally intensive.
Application of sparse modelling
Sparse modeling is a rapidly developing area at the intersection of statistical
learning and signal processing, motivated by the age-old statistical problem of
selecting a small number of predictive variables in high-dimensional datasets.
This collection describes key approaches in sparse modeling, focusing on its
applications in fields including neuroscience, computational biology, and
computer vision. Sparse modeling methods can improve the interpretability of
predictive models and aid efficient recovery of high-dimensional unobserved
signals from a limited number of measurements. Yet despite significant
advances in the field, a number of open issues remain when sparse modeling
meets real-life applications. The book discusses a range of practical
applications and state-of-the-art approaches for tackling the challenges
presented by these applications. Topics considered include the choice of
method in genomics applications; analysis of protein mass-spectrometry data;
the stability of sparse models in brain imaging applications; sequential testing
approaches; algorithmic aspects of sparse recovery; and learning sparse
latent models.
THANK YOU
