Least-Squares Regression
ChEn 2450
Concept: Given N data points (xi,yi), nd parameters in the function f(x) that minimize the error between f(xi) and yi.
f(x)
x
Wednesday, September 28, 11 1
Hoffman 4.10.3
Linear Least-Squares Regression
Concept: Given N data points (xi,yi), nd parameters in the function f(x) that minimize the error between f(xi) and yi.
N
S=
i=1
[yi
f (xi )]
S - Sum of the squared errors.
To minimize S (error between function & data), we take partial derivatives w.r.t. the function parameters and set to zero. ASSUME f(x) is an f (x) = np-order polynomial:
np
ak x
k=0
S=
N i=1
yi
k=0
np
ak xk i
i - data point index k - polynomial coefcient index
S = ak
N i=1
Normal Equations Only for polynomials
2xk yi i
np j=0
aj xj = 0 k = 0 . . . np i
i - data point index j - dummy index k - polynomial coefcient index
np+1 equations for np+1 unknowns (ak). Equations are linear w.r.t. ak.
Wednesday, September 28, 11
Linear Least Squares regression.
2
Example: np=1 (Linear Polynomial)
S = ak
Linear polynomial, np=1:
N i=1
xk yi i
np j=0
aj xj i
N
=0
k = 0 . . . np
Here we divided the entire equation by -2. (why?)
f (x) = a0 + a1 x
S=
i=1
[yi
N
a0
a1 xi ]
S=
i=1
2 gi
n
gi = yi
a0
N
a1 xi
n
Apply chain rule.
X S gi X S = = (2gi )( 1) a0 g i a0 i=1 i=1 =
n X i=1
X S gi X S = = (2gi )( xi ) a1 g i a1 i=1 i=1 =
n X i=1
2 (yi
a0
a1 xi )
2xi (yi
N
a0
a1 xi )
N
S a0 S a1
=
i=1 N
2( 1) [yi 2( xi ) [yi
i=1
a0 a0
a1 xi ] a1 xi ]
0 0
=
i=1 N
[yi [yi
i=1
a0 a0
a1 xi ]
i=1 N
yi yi xi
i=1
=
i=1 N
(a0 + a1 xi )
a1 xi ] xi
=
i=1
(a0 + a1 xi ) xi
2 equations, 2 unknowns (a0, a1)
Wednesday, September 28, 11
Example (contd.)
N N
yi
i=1 N
=
i=1 N
(a0 + a1 xi ) (a0 + a1 xi ) xi
i=1
2 eqns, 2 unknowns. Lets put these in Matrix form...
yi xi
i=1
For N=4 points,
y1 + y2 + y 3 + y4 y 1 x 1 + y 2 x 2 + y3 x 3 + y 4 x 4
= (a0 + a1 x1 ) + (a0 + a1 x2 ) + (a0 + a1 x3 ) + (a0 + a1 x4 ) = (a0 + a1 x1 ) x1 + (a0 + a1 x2 ) x2 + (a0 + a1 x3 ) x3 + (a0 + a1 x4 ) x4
Step 1: dene the solution variable vector.
Step 2: dene the matrix and RHS vector.
"
a0 a1
PN
xi
i=1
"
Wednesday, September 28, 11
PN
N xi
i=1
# PN a0 i=1 xi PN 2 = a1 i=1 xi
! PN i=1 yi PN i=1 xi yi
# PN i=1 xi PN 2 i=1 xi
Linear least squares regression for a linear polynomial.
! PN i=1 yi PN i=1 xi yi
Example - Reaction Rate Constant
Pre-exponential factor Activation energy
Cl N=N Cl + N2
Benzene diazonium chloride Chlorobenzene
k = A exp
rate constant
Ea RT
Gas constant R=8.314 J/mol-K
Temperature
4.5 5
ln(k) = ln(A)
log(k) (1/s)
Ea RT
recall: ln(ab)=ln(a)+ln(b)
ln(A)=38.9246 Ea=121481.5094
data best fit
T (K) 313 319 323 328
k (1/s) 0.00043 0.00103 0.00180 0.00355 0.00717
5.5 6 6.5 7 7.5 8 3.9 3.8 3.7 1/RT (mol/J) 3.6 x 10
4
y = a0 + a1 x
y = ln(k), a0 = ln(A), 1 x= , RT a1 = Ea
333
"
PN
N xi
i=1
# PN a0 b0 i=1 xi PN 2 = a1 b1 i=1 xi
Note: we need to calculate A (pre-exponential factor) from a0.
Now we are ready to go to the computer to determine a0 and a1.
5
Wednesday, September 28, 11
Polynomial Regression & Normal Equations
(Alternative Formulation for Linear Least Squares)
p = a0 + a1 x + a2 x + a3 x + + an x
2 3
Given N>np observations (xi,yi), and a np order polynomial, nd aj.
. . . x2 N
A
One equation for each observation (N equations)
1 1 . . .
1 xN
x1 x2 . . .
x2 1 x2 2
. . .
n x1 p n x2 p n
. . .
xNp
NOTE: this is an overdetermined (more equations than unknowns) linear problem for the coefcients, ai.
Example - linear polynomial: p(x) = a0 + a1 x
a0 a1 = . . . anp
y1 . . . yN
b
AT A = AT b
Normal Equations
Another form of linear least-squares regression.
1 x1 1 x2 . . . . . . 1 xN
y1 y2 a0 = . a1 . . yN
1 x1
1 x2
1 1 1 . . xN .
x1 x2 . . . 1 xN
a0 a1
1 x1
1 x2
y1 y 1 2 . . xN . yN
6
Wednesday, September 28, 11
The Two are One...
Consider each of the previous approaches for a rst order polynomial.
S = ak
N i=1
Direct Least Squares
xk yi i
np j=0
N
aj xj = 0 i
a0 a0
k = 0 . . . np
1 1 . . . 1
A A =A b
T T
A= x1 x2 . . . xN
Matrix Transpose Approach
k=0
i=1 N
(yi xi (yi
i=1
N N
a1 xi ) a1 xi )
N
= =
0, 0
a0 a1
k=1
AT b =
xi
i=1 N
a0
i=1 N
1 + a1 xi + a1
=
i=1 N
yi , xi yi
i=1
N i=1 yi N i=1 xi yi
N N
b=
y1 y2 y3 . . . yN
a0
i=1
x2 i
i=1
N xi
i=1
N i=1 xi N 2 i=1 xi
a0 a1
N i=1 yi N = i=1 xi yi
A A=
T
i=1
xi
N i=1 xi N 2 i=1 xi
Typically most convenient for linear regression problems.
7
Wednesday, September 28, 11
Example - Reaction Rate Constant
Pre-exponential factor Activation energy
Cl N=N Cl + N2
Benzene diazonium chloride Chlorobenzene
k = A exp
rate constant
Ea RT
Gas constant R=8.314 J/mol-K
Temperature
4.5 5
ln(k) = ln(A)
log(k) (1/s)
Ea RT
recall: ln(ab)=ln(a)+ln(b)
ln(A)=38.9246 Ea=121481.5094
data best fit
T (K) 313 319 323 328
k (1/s) 0.00043 0.00103 0.00180 0.00355 0.00717
5.5 6 6.5 7 7.5 8 3.9 3.8 3.7 1/RT (mol/J) 3.6 x 10
4
y = a0 + a1 x
y = ln(k), a0 = ln(A),
1 1 RT1 1 1 RT2 . . . . . . 1 RT1 N A
1 x= , RT a1 = Ea
333
Wednesday, September 28, 11
ln(k1 ) ln(k2 ) a0 = . . a1 . ln(kN )
b
A A =A b
T T
Note: need to calculate A (preexponential factor) from a0.
8
Linear Least Squares Regression in MATLAB
BEFORE you go to the computer, formulate the problem as a polynomial regression problem!
Do it manually - the way that we just showed.
See MATLAB code for previous example posted on class website. This is my favored method, and provides maximum exibility!
Polynomial regression: p=polyfit(x,y,n)
polyval(p,xi) evaluates the resulting polynomial at xi. gives the best t for a polynomial of order n through the data. if n==(length(x)-1) then you get an interpolant. if n<(length(x)-1) then you get a least-squares t. You still must get the problem into a polynomial form.
9
Wednesday, September 28, 11
The
R =1
2
2 Value R
N i=1 (yi N i=1 (yi
How well does the regressed line t the data?
f (xi )) y)
2 2
(xi,yi) - data points that we are regressing. f(x) - function we are regressing to. f(xi) - regression function value at xi.
N
y=
1 N i=1
yi
Average value of yi
R2=1
Wednesday, September 28, 11
Perfect t
10
Nonlinear Least Squares Regression
Assume f(x) has n parameters ak that we want to determine via regression.
N
S=
i=1
[yi
f (xi )]
S = 0, ak
k = 1...n
Same approach as before, but now the parameters of f(x) may enter nonlinearly! Example:
f (x) = ax
b
N
S=
S=
N X i=1
2 gi , i=1
N
g i = yi
N
axb i
xb i axb ln(xi ) i
yi
b 2 axi
X @S @gi X @S = = 2gi @a @gi @a i=1 i=1 X @S @gi X @S = = 2gi @b @gi @b i=1 i=1
N N
S = a S = b
N X i=1 N X i=1
2xb yi i 2axb i
axb i axb i
0=
N X i=1
x b yi i
axb i axb i
ln(xi ) yi
0=
N X i=1
xb ln(xi ) yi i
Two nonlinear equations to solve for a and b.
Can we reformulate this as a linear problem?
Wednesday, September 28, 11 11
Kinetics Example Revisited
Sum of squared errors.
Minimize S w.r.t. A and Ea.
rate constant
N S= yi i=1
A exp
Ea RTi
Pre-exponential factor
Activation energy
k = A exp
Ea RT
Gas constant R=8.314 J/mol-K
Temperature
S=
N i=1
2 gi ,
N X
gi = yi
A exp
Ea RTi
S A S Ea
@S = @A
@S = @Ea
@S @gi @gi @A i=1 N X = 2gi exp
i=1 N X
Ea RTi
0 0 = =
N 2A = exp RTi i=1
N i=1
2 exp
Ea yi A exp RTi Ea yi A exp RTi
Ea RTi
Ea RTi
2 nonlinear equations with 2 unknowns A, Ea.
@S @gi @gi @Ea i=1 N X A Ea = 2gi exp RTi RTi i=1
N i=1
exp
Ea RTi
yi
A exp A exp
Ea RTi
N 1 exp Ti i=1
Ea RTi
yi
Ea RTi
We will show how to solve this soon!
12
Wednesday, September 28, 11
