Motivation Dimension reduction Refresher on Linear Algebra Principal component analysis PCA with Python Limits of PCA
Principal component analysis
Motivation Dimension reduction Refresher on Linear Algebra Principal component analysis PCA with Python Limits of PCA
High dimensional data in data analysis?
Words embeddings in NLP
Motivation Dimension reduction Refresher on Linear Algebra Principal component analysis PCA with Python Limits of PCA
High dimensional data in data analysis?
Brain activity
Motivation Dimension reduction Refresher on Linear Algebra Principal component analysis PCA with Python Limits of PCA
High dimensional data in data analysis?
Challenges ?
Motivation Dimension reduction Refresher on Linear Algebra Principal component analysis PCA with Python Limits of PCA
High dimensional data in data analysis?
Challenges ?
Visualize
Group in relevant clusters
Difficult with high dimensional data!
A classical dimension reduction approach Principal Component
Analysis
Motivation Dimension reduction Refresher on Linear Algebra Principal component analysis PCA with Python Limits of PCA
Dimension reduction
Dimension reduction without loss of information?
Motivation Dimension reduction Refresher on Linear Algebra Principal component analysis PCA with Python Limits of PCA
Dimension reduction
Dimension reduction without loss of information?
Motivation Dimension reduction Refresher on Linear Algebra Principal component analysis PCA with Python Limits of PCA
Dimension reduction
Scientific questions
How can we reduce dimension to separate observations?
Possible answer : Principal Component Analysis
Motivation Dimension reduction Refresher on Linear Algebra Principal component analysis PCA with Python Limits of PCA
High dimensional data in data analysis?
Challenges ?
Motivation Dimension reduction Refresher on Linear Algebra Principal component analysis PCA with Python Limits of PCA
High dimensional data in data analysis?
Challenges ?
Visualize
Group in relevant clusters
Difficult with high dimensional data!
Motivation Dimension reduction Refresher on Linear Algebra Principal component analysis PCA with Python Limits of PCA
Dimension reduction
Main features of Principal Component Analysis (PCA)
preserves the global structure of data.
maps all the clusters as a whole
potential applications : noise filtering, feature extractions, stock
market predictions, and gene data analysis.
Motivation Dimension reduction Refresher on Linear Algebra Principal component analysis PCA with Python Limits of PCA
Refresher on Linear Algebra
Vectors of Rp
Rp is the set of vectors with p components
1
For e.g. X = −3 is a 3-component vector.
4
We can also say that X belongs to R3
Concept of basis
The family (X1 , · · · , Xp ) is a basis of Rp if each vector of Rp can be
expressed in a unique way as a linear combination of X1 · · · , Xp
Motivation Dimension reduction Refresher on Linear Algebra Principal component analysis PCA with Python Limits of PCA
Refresher on Linear Algebra
Example 1
! !!
1 0
, is a basis of R2
0 1
!
x
Indeed every X = 1 can be expressed in a unique way as
x2
! !
1 0
X = x1 · + x2 ·
0 1
Motivation Dimension reduction Refresher on Linear Algebra Principal component analysis PCA with Python Limits of PCA
Refresher on Linear Algebra
!
2
Example with X =
3
Motivation Dimension reduction Refresher on Linear Algebra Principal component analysis PCA with Python Limits of PCA
Refresher on Linear Algebra
Example 2
! !!
1 1
, is a basis of R2
1 −1
!
x1
Indeed, every X = can be expressed in a unique way as
x2
x1 + x2 1
! !
x1 − x2 1
· X= + ·
2 1 2 −1
!
3
Example with X =
2
! !
1 1
X = 2.5 · + 0.5 ·
1 −1
Motivation Dimension reduction Refresher on Linear Algebra Principal component analysis PCA with Python Limits of PCA
Refresher on Linear Algebra
!
3
Example with X =
2
Motivation Dimension reduction Refresher on Linear Algebra Principal component analysis PCA with Python Limits of PCA
Refresher on Linear Algebra
Matrices
A matrix with p rows and p columns is an array of reals with p
rows nd p columns
A matrix maps vectors of Rp to vectors of Rp
It can be interpretated as a linear transformation of the plane in
the case p = 2
Motivation Dimension reduction Refresher on Linear Algebra Principal component analysis PCA with Python Limits of PCA
Refresher on Linear Algebra
Matrices
If we are given a matrix M, the transformation may be not so
simple to identify!
!
2 1
What is the transformation associated to M = ?
1 2
! !
x1 y1
Y = M · X with X = and Y = means
x2 y2
y1 = 2x1 + x2
y2 = x1 + 2x2
The transformation is eplicit but the geometric interpretation is
not so clear
Motivation Dimension reduction Refresher on Linear Algebra Principal component analysis PCA with Python Limits of PCA
Refresher on Linear Algebra
Simpler with a smartchange of coordinate?
Motivation Dimension reduction Refresher on Linear Algebra Principal component analysis PCA with Python Limits of PCA
Refresher on Linear Algebra
Eigenvalues and eigenvectors
Let A ∈ Md (R).
The vector X ∈ Rd \ {0} is said to be a eigenvector of matrix A
associated to the eigenvalue λ if AX = λX.
Motivation Dimension reduction Refresher on Linear Algebra Principal component analysis PCA with Python Limits of PCA
Refresher on Linear Algebra
Example 3
! !
2 1 1
Let A = and X =
1 2 1
!
3
Since AX = , X is an eigenvector of A with associated
3
eigenvalue 3
Motivation Dimension reduction Refresher on Linear Algebra Principal component analysis PCA with Python Limits of PCA
Refresher on Linear Algebra
Motivation Dimension reduction Refresher on Linear Algebra Principal component analysis PCA with Python Limits of PCA
Refresher on Linear Algebra
Diagonalizable matrices
The square matrix A with p columns and p rows is said to be
diagonalizable if there exists (X1 , · · · , Xp ) such that
Condition 1 : (X1 , · · · , Xp ) is a basis of Rp
Condition 2 : for each i, Xi is an eigenvector of A
Motivation Dimension reduction Refresher on Linear Algebra Principal component analysis PCA with Python Limits of PCA
Principal Component Analysis
Principle
How perform dimension reduction in a linear way?
Mathematical tool : linear projection on a low-dimensional space
Motivation Dimension reduction Refresher on Linear Algebra Principal component analysis PCA with Python Limits of PCA
Principal Component Analysis
Principle
How can we find the low-dimensional space H?
Motivation Dimension reduction Refresher on Linear Algebra Principal component analysis PCA with Python Limits of PCA
Principal Component Analysis
Principle
Principal component analysis : how does it work?
The k dimensional space H that we are looking for is generated
by the k eigenvectors uα associated to the k largest eigenvalues
λα of the matrix X T X
We have several possible choices for the matrix X :
General PCA : the raw data matrix X = R
Centered PCA: the centered data matrix. the matrix X T X is then
the matrix of empirical covariances
Normed PCA : the normed and centered data matrix. The matrix
X T X is then the matrix of empirical correlations
Projection of the observation oi on the axis α
Motivation Dimension reduction Refresher on Linear Algebra Principal component analysis PCA with Python Limits of PCA
Principal Component Analysis
Principle
Principal component analysis : how does it work?
In general n ≫ d (number of observations ≫ number of initial
variables)
It is the reason why we deal with the matrix X T X with dimension
d × d rather than XX T with dimension n × n
Existence of some links between these two analysis
Motivation Dimension reduction Refresher on Linear Algebra Principal component analysis PCA with Python Limits of PCA
PCA with Python
An example
To illustrate PCA we consider a dataset containing gene
expression profiles for 105 breast tumour samples measured
using Swegene Human 27K RAP UniGene188 arrays
Within the population of cells, one can focus focused on the
expression of GATA3 and XBP1, whose expression was known
to correlate with estrogen receptor status1
1
Breast cancer cells may be estrogen receptor positive, ER +,
or negative, ER , indicating capacity to respond to estrogen
signalling, which can therefore influence treatment
Motivation Dimension reduction Refresher on Linear Algebra Principal component analysis PCA with Python Limits of PCA
PCA with Python
An example
We plot the expression levels of GATA3 and XBP1 against one
another to visualise the data in the two-dimensional space
Motivation Dimension reduction Refresher on Linear Algebra Principal component analysis PCA with Python Limits of PCA
PCA with Python
An example
We perform PCA and visualize by plotting the original data
side-by-side with the transformed data
Original data versus transformed ones
Motivation Dimension reduction Refresher on Linear Algebra Principal component analysis PCA with Python Limits of PCA
PCA with Python
An example
We have simply rotated the original data, so that the greatest
variance aligns along the x-axis and so forth
We can find out how much of the variance each of the principle
components explains
Motivation Dimension reduction Refresher on Linear Algebra Principal component analysis PCA with Python Limits of PCA
PCA with Python
An example
PC1 explains the vast majority of the variance in the observations
The dimensionality reduction step of PCA occurs when we
choose to discard the later PCs.
We visualise the data using only PC1.
Motivation Dimension reduction Refresher on Linear Algebra Principal component analysis PCA with Python Limits of PCA
Limits of PCA
An example
Principle component analysis is not always appropriate for
complex datasets, particularly when dealing with nonlinearities
To illustrate this, let’s consider an simulated expression set
containing 8 genes, with 10 timepoints/conditions.
Motivation Dimension reduction Refresher on Linear Algebra Principal component analysis PCA with Python Limits of PCA
Limits of PCA
An example
The data can be separated out by a single direction
The data from time/condition 1 through to time/condition 10 can
ordered
Intuitively, the data can be represented by a single dimension
We run PCA as we would normally, and visualise the result,
plotting the first two PCs
Motivation Dimension reduction Refresher on Linear Algebra Principal component analysis PCA with Python Limits of PCA
Limits of PCA
An example
Motivation Dimension reduction Refresher on Linear Algebra Principal component analysis PCA with Python Limits of PCA
Limits of PCA
An example
We see that the PCA plot has placed the datapoints in a
horseshoe shape, with condition/time point 1 very close to
condition/time point 10.
From the earlier plots of gene expression profiles we can see that
the relationships between the various genes are not entirely
straightforward.
For example, gene 1 is initially correlated with gene 2, then
negatively correlated, and finally uncorrelated, whilst no
correlation exists between gene 1 and genes 5 - 8.
These nonlinearities make it difficult for PCA which, in general,
attempts to preserve large pairwise distances, leading to the well
known horseshoe effect
Motivation Dimension reduction Refresher on Linear Algebra Principal component analysis PCA with Python Limits of PCA
Limits of PCA
Pro and cons of PCA
Main advantages of PCA
Simple to implement, no tuning
Highly interpretable. We can find decide on how much variance
to preserve using eigenvalues.
Main drawbacks of PCA
It is a global transform which may not preserve local structure
(clusters)
It is sensitive to outliers
