Models Lineals
2-The L2 models
Pere Puig
Department de Matemtiques
Universitat Autnoma de Barcelona
[email protected]
Models Lineals
The size of the Earth
On 19 March 1791 the meter
was theoretically defined by
the French Academy of Science
as being equal to the ten
millionth part of one quarter
of the terrestrial meridian.
However, the length of the
meridian still had to be set up.
The persons in charge of this
mission were Pierre-Franois
Mechain and Jean-Baptiste
Delambre. The length of the
meridian would be determined
by triangulation between
Dunkirk and Barcelona.
Models Lineals
Models Lineals
Measurements of the French meridian arc, made in 1795
between Barcelona and Dunkirk:
Using this data set, Legendre in 1805 solved an
overdetermined system of equations by the method of least
squares.
Models Lineals
The L2 solutions
A general problem: Consider an overdetermined system of
equations
, where A is a nxm matrix, n>m, and
The L2 solution of the system is a vector
which minimizes
the quadratic difference between b and Ax:
where ai is the i-th row of the matrix A.
Theorem: Any solution of the mxm system of equations
(normal equations)
solves the L2 minimization problem.
Models Lineals
Proof: b is a vector in Rn and the columns of A define a vector
space C where Ax is an arbitrary vector in C. The minimum
distance b Ax is attained when b-Ax is orthogonal to Ax.
b Ax
C
Ax
Models Lineals
Example 1: fitting a simple line (simple linear regression).
Suppose there are n data points {yi, xi}, where i = 1, 2, , n.
The goal is to find the equation of the straight line
which would provide a "best" (L2) fit for the data points.
Here,
Models Lineals
Consequently, the normal equations are,
and the solutions,
They are usually expressed as,
where Sxy is the sample covariance of x and y, and S2x is the
sample variance of x.
Models Lineals
Example 2: fitting a simple line without intercept.
Now there are n data points {yi, xi}, where i = 1, 2, , n, and
the goal is to find the equation of the line passing through the
origin
which would provide a "best" (L2) fit.
The normal equation is,
and the solution:
Models Lineals
Example 3: the two variables predicting model.
Now there are n data points {zi, yi, xi}, where i = 1, 2, , n, and
the goal is to find the equation of the plane
which would provide a "best" (L2) fit for the data points.
Here,
Models Lineals
Consider the example of the sales staff :
Models Lineals
Now the predicting model is:
This is different from the L1 solution:
Note that the L2 solution has been expressed as,
but is
always invertible?
Models Lineals
Proposition: Let A a nxm matrix,
is invertible
Proof: Remember that a square matrix B is invertible
Remember also that rank(A) + dim(Ker(A)) = m, and therefore
rank(A) =m is equivalent to say that
The proof will be finished if we prove that
-
Models Lineals
Property: Consider an overdetermined system of equations
, where A is a nxm matrix, n>m, rank(A)=m, with a first column of
1s. Then, the L2 solution
, satisfies
where indicates the mean of the coefficients of the i-th column
of the matrix A, and is the mean of the components of vector b.
Proof: (Exercise)
How can it be interpreted?
Models Lineals
An application: the best local polynomial fit
Consider 2k+1 consecutive values of a time series, yt-k,...,yt,..., yt+k.
A local polynomial estimator of order p<2k+1 is the minimizer of
the least squares function,
Then, choosing u=0 we obtain a predictor of the central
observation yt . The local polynomial approach consists in
replacing yt by y*t=0
Models Lineals
The normal equations of this multiple regression
model are determined by,
However we are only interested in the value of 0
because we want to evaluate the polynomial at
u=0.
Models Lineals
Notice that,
Moreover, the coefficients (cu) do not depend of yt+u
This is a kind of linear filter.
17
Models Lineals
Example: Finding the local polynomial smoothing with p=2
and k=2 (5 consecutive points).
1
1
1
1
1
-2
-1
0
1
2
4
1
0
1
4
1
-2
4
1
-1
1
1
0
0
1
1
1
Matrix At
Matrix A
5
0
10
0
10
0
10
0
34
0,48571
0
-0,14286
At*A
0 -0,14286
0,1
0
0 0,071429
inv(At*A)
Les multiplicacions de matrius es fan amb MMULT
L'inversi d'una matriu es fa amb MINVERSA
-0,08571 0,3428571 0,48571 0,34286 -0,08571
-0,2
-0,1
0
0,1
0,2
0,142857 -0,071429 -0,14286 -0,0714 0,142857
inv(At*A)*At
18
1
2
4
Models Lineals
This linear filter is,
yt*=(-3 yt-2 + 12 yt-1 + 17 yt + 12 yt+1 -3 yt+2 )/35
For instance, it can be applied to the following time
series,
Lung diseases mortality for women from January 1974 to
December 1979.
19
Models Lineals
It can also be compared with the centered (simple) 5 points
moving average CMA(5).
1200
1000
800
600
400
200
en
e74
ju
l-7
4
en
e75
ju
l-7
5
en
e76
ju
l-7
6
en
e77
ju
l-7
7
en
e78
ju
l-7
8
en
e79
ju
l-7
9
Data
CMA(5)
20
LP2(5)
