Least square fit of a line/polynomial to

input/output data

Dr. Madhu N. Belur

Control & Computing group

Department of Electrical Engineering
Indian Institute of Technology Bombay
Email: [email protected]

30th Nov, 2010

Outline Scilab Least squares Scilab commands

Outline

1 Scilab

2 Least squares

3 Scilab commands

Madhu Belur, CC group, EE, IITB Least square fit

Outline Scilab Least squares Scilab commands

Today’s focus

Scilab is free.

Madhu Belur, CC group, EE, IITB Least square fit

Outline Scilab Least squares Scilab commands

Today’s focus

Scilab is free.
Matrix/loops syntax is same as for Matlab.

Madhu Belur, CC group, EE, IITB Least square fit

Outline Scilab Least squares Scilab commands

Today’s focus

Scilab is free.
Matrix/loops syntax is same as for Matlab.
Scilab provides all basic and many advanced tools.

Madhu Belur, CC group, EE, IITB Least square fit

Outline Scilab Least squares Scilab commands

Today’s focus

Scilab is free.
Matrix/loops syntax is same as for Matlab.
Scilab provides all basic and many advanced tools.
Today: best fit: line and polynomial : reglin command

Madhu Belur, CC group, EE, IITB Least square fit

Outline Scilab Least squares Scilab commands

Linear fit

Given n samples of (x, z) pairs:

xi and zi for i = 1, . . . , n, we expect following equation is satisfied

zi = a1 xi + a0 for i = 1, . . . , n (1)

for some constants a1 and a0 .

Madhu Belur, CC group, EE, IITB Least square fit

Outline Scilab Least squares Scilab commands

Linear fit

Given n samples of (x, z) pairs:

xi and zi for i = 1, . . . , n, we expect following equation is satisfied

zi = a1 xi + a0 for i = 1, . . . , n (1)

for some constants a1 and a0 .

xi and zi fall on some line with slope a1 and ‘z-intercept’=a0 .
The ‘line fit’ problem:
Find these constants a1 and a0 .
‘Best’ fit?

Madhu Belur, CC group, EE, IITB Least square fit

Outline Scilab Least squares Scilab commands

Best fit

Real situtation:
Equation (1) will not be satisfied ‘exactly’.
Least-square-fit problem:
Given n samples of (xi , zi ) pairs,

Madhu Belur, CC group, EE, IITB Least square fit

Outline Scilab Least squares Scilab commands

Best fit

Real situtation:
Equation (1) will not be satisfied ‘exactly’.
Least-square-fit problem:
Given n samples of (xi , zi ) pairs,
find constants a1 and a0 such that the ‘total square error’
n
X
(zi − a1 xi − a0 )2 (2)
i=1

is least.

Madhu Belur, CC group, EE, IITB Least square fit

Outline Scilab Least squares Scilab commands

Best fit

Real situtation:
Equation (1) will not be satisfied ‘exactly’.
Least-square-fit problem:
Given n samples of (xi , zi ) pairs,
find constants a1 and a0 such that the ‘total square error’
n
X
(zi − a1 xi − a0 )2 (2)
i=1

is least.
Least square error

Madhu Belur, CC group, EE, IITB Least square fit

Outline Scilab Least squares Scilab commands

Higher order polynomial best fit

Least-square-fit problem (of higher order):

Madhu Belur, CC group, EE, IITB Least square fit

Outline Scilab Least squares Scilab commands

Higher order polynomial best fit

Least-square-fit problem (of higher order):

Suppose we expect zi satisfies the following equation:

zi = a2 xi2 + a1 xi + a0

Madhu Belur, CC group, EE, IITB Least square fit

Outline Scilab Least squares Scilab commands

Higher order polynomial best fit

Least-square-fit problem (of higher order):

Suppose we expect zi satisfies the following equation:

zi = a2 xi2 + a1 xi + a0

Points (xi , zi ) are sitting on a parabola.

Madhu Belur, CC group, EE, IITB Least square fit

Outline Scilab Least squares Scilab commands

Higher order polynomial best fit

Least-square-fit problem (of higher order):

Suppose we expect zi satisfies the following equation:

zi = a2 xi2 + a1 xi + a0

Points (xi , zi ) are sitting on a parabola.

Problem (more generally):

Madhu Belur, CC group, EE, IITB Least square fit

Outline Scilab Least squares Scilab commands

Higher order polynomial best fit

Least-square-fit problem (of higher order):

Suppose we expect zi satisfies the following equation:

zi = a2 xi2 + a1 xi + a0

Points (xi , zi ) are sitting on a parabola.

Problem (more generally):
Given n samples of (xi , zi ) pairs and some choice of degree d.

Madhu Belur, CC group, EE, IITB Least square fit

Outline Scilab Least squares Scilab commands

Higher order polynomial best fit

Least-square-fit problem (of higher order):

Suppose we expect zi satisfies the following equation:

zi = a2 xi2 + a1 xi + a0

Points (xi , zi ) are sitting on a parabola.

Problem (more generally):
Given n samples of (xi , zi ) pairs and some choice of degree d.
Find constants ad , . . . a1 and a0 such that the ‘total square error’
n
X
(ad xid + ad−1 xid−1 + · · · + a1 xi + a0 − zi )2 (3)
i=1

is least.

Madhu Belur, CC group, EE, IITB Least square fit

Outline Scilab Least squares Scilab commands

Higher order polynomial best fit

Least-square-fit problem (of higher order):

Suppose we expect zi satisfies the following equation:

zi = a2 xi2 + a1 xi + a0

Points (xi , zi ) are sitting on a parabola.

Problem (more generally):
Given n samples of (xi , zi ) pairs and some choice of degree d.
Find constants ad , . . . a1 and a0 such that the ‘total square error’
n
X
(ad xid + ad−1 xid−1 + · · · + a1 xi + a0 − zi )2 (3)
i=1

is least.

Madhu Belur, CC group, EE, IITB Least square fit

Outline Scilab Least squares Scilab commands

reglin

Linear regression problem.

The unknowns ai enter the problem linearly.

Madhu Belur, CC group, EE, IITB Least square fit

Outline Scilab Least squares Scilab commands

reglin

Linear regression problem.

The unknowns ai enter the problem linearly.
(i.e. ai ’s are not getting squared, or multiplied to each other.)
[a1, a0] = reglin(x,z)
where x and z are vectors with same number of columns.
X
(zi − a1 xi − a0 )2
i

is minimized.

Madhu Belur, CC group, EE, IITB Least square fit

Outline Scilab Least squares Scilab commands

reglin

Linear regression problem.

The unknowns ai enter the problem linearly.
(i.e. ai ’s are not getting squared, or multiplied to each other.)
[a1, a0] = reglin(x,z)
where x and z are vectors with same number of columns.
X
(zi − a1 xi − a0 )2
i

is minimized.
[a1, a0, sig] = reglin(x,z)
sig : standard deviation of the residual.

Madhu Belur, CC group, EE, IITB Least square fit

Outline Scilab Least squares Scilab commands

reglin

Linear regression problem.

The unknowns ai enter the problem linearly.
(i.e. ai ’s are not getting squared, or multiplied to each other.)
[a1, a0] = reglin(x,z)
where x and z are vectors with same number of columns.
X
(zi − a1 xi − a0 )2
i

is minimized.
[a1, a0, sig] = reglin(x,z)
sig : standard deviation of the residual.
(Smaller sig means better fit.)

Madhu Belur, CC group, EE, IITB Least square fit

Outline Scilab Least squares Scilab commands

More than one independent variables

Suppose z depends on independent variables x1 , x2 , etc.

[a1,a0]=reglin(x,z)
where x and z are matrix/vector with same number of columns.

Madhu Belur, CC group, EE, IITB Least square fit

Outline Scilab Least squares Scilab commands

More than one independent variables

Suppose z depends on independent variables x1 , x2 , etc.

[a1,a0]=reglin(x,z)
where x and z are matrix/vector with same number of columns.
(but x has many rows.)

Madhu Belur, CC group, EE, IITB Least square fit

Outline Scilab Least squares Scilab commands

More than one independent variables

Suppose z depends on independent variables x1 , x2 , etc.

[a1,a0]=reglin(x,z)
where x and z are matrix/vector with same number of columns.
(but x has many rows.)
If x has more than one rows:
components in a1 = number of rows of x
(number of independent variables.)

Madhu Belur, CC group, EE, IITB Least square fit

Outline Scilab Least squares Scilab commands

Exercise plan

Generate data using known (actual) values of a0 and a1.

Madhu Belur, CC group, EE, IITB Least square fit

Outline Scilab Least squares Scilab commands

Exercise plan

Generate data using known (actual) values of a0 and a1.

Add noise.

Madhu Belur, CC group, EE, IITB Least square fit

Outline Scilab Least squares Scilab commands

Exercise plan

Generate data using known (actual) values of a0 and a1.

Add noise.
Use noisy data to estimate a0 and a1:

Madhu Belur, CC group, EE, IITB Least square fit

Outline Scilab Least squares Scilab commands

Random numbers

rand(3) generates a 1 × 1 random number (uniformly distributed

between 0 and 1).

Madhu Belur, CC group, EE, IITB Least square fit

Outline Scilab Least squares Scilab commands

Random numbers

rand(3) generates a 1 × 1 random number (uniformly distributed

between 0 and 1).
If P is an n × p matrix, then

Madhu Belur, CC group, EE, IITB Least square fit

Outline Scilab Least squares Scilab commands

Random numbers

rand(3) generates a 1 × 1 random number (uniformly distributed

between 0 and 1).
If P is an n × p matrix, then
B=rand(P)
defines a random matrix B of the size of P.

Madhu Belur, CC group, EE, IITB Least square fit

Outline Scilab Least squares Scilab commands

Random numbers

rand(3) generates a 1 × 1 random number (uniformly distributed

between 0 and 1).
If P is an n × p matrix, then
B=rand(P)
defines a random matrix B of the size of P.
(P is not overwritten. A new matrix B is defined.)

Madhu Belur, CC group, EE, IITB Least square fit

Outline Scilab Least squares Scilab commands

Exercises:

Generate data, add noise, and estimate the parameters back:

One independent variable (first).
a0a = 3;
a1a = 6;
(actual)
x = 1:10;

Madhu Belur, CC group, EE, IITB Least square fit

Outline Scilab Least squares Scilab commands

Exercises:

Generate data, add noise, and estimate the parameters back:

One independent variable (first).
a0a = 3;
a1a = 6;
(actual)
x = 1:10;
dev = 0.5;

Madhu Belur, CC group, EE, IITB Least square fit

Outline Scilab Least squares Scilab commands

Exercises:

Generate data, add noise, and estimate the parameters back:

One independent variable (first).
a0a = 3;
a1a = 6;
(actual)
x = 1:10;
dev = 0.5;
y = a1a*x + a0a + dev*(rand(x)-0.5)
Use this x and y to estimate a0 and a1
Compare a0 & a1 with actual a0a & a1a.

Madhu Belur, CC group, EE, IITB Least square fit

Outline Scilab Least squares Scilab commands

Exercises:

Generate data, add noise, and estimate the parameters back:

One independent variable (first).
a0a = 3;
a1a = 6;
(actual)
x = 1:10;
dev = 0.5;
y = a1a*x + a0a + dev*(rand(x)-0.5)
Use this x and y to estimate a0 and a1
Compare a0 & a1 with actual a0a & a1a.
For sig to compare with dev, note that sig is standard deviation
(most relevant for normal distribution, and we used uniform
distribution).

Madhu Belur, CC group, EE, IITB Least square fit

Outline Scilab Least squares Scilab commands

Fit 2nd order polynomial

Find a2 , a1 and a0 such that

y = a1 x + a2 x 2 + a0
Suppose a0a = 3; a1a = 6; a2a = 2; (actual)

Madhu Belur, CC group, EE, IITB Least square fit

Outline Scilab Least squares Scilab commands

Fit 2nd order polynomial

Find a2 , a1 and a0 such that

y = a1 x + a2 x 2 + a0
Suppose a0a = 3; a1a = 6; a2a = 2; (actual)
x = 1:10; dev = 0.5;

Madhu Belur, CC group, EE, IITB Least square fit

Outline Scilab Least squares Scilab commands

Fit 2nd order polynomial

Find a2 , a1 and a0 such that

y = a1 x + a2 x 2 + a0
Suppose a0a = 3; a1a = 6; a2a = 2; (actual)
x = 1:10; dev = 0.5;
y = a1a∗x + a2a∗x•ˆ2 + a0a + dev∗(rand(x)-0.5)
x = 1:10; x2 = x.ˆ2;
X=[x;x2];
reglin(X,y)

Madhu Belur, CC group, EE, IITB Least square fit

Outline Scilab Least squares Scilab commands

Read/write csv files

csv ≡ comma separated value

Madhu Belur, CC group, EE, IITB Least square fit

Outline Scilab Least squares Scilab commands

Read/write csv files

csv ≡ comma separated value

r = read csv(’lsquare data.csv’,ascii(9)); // read csv data into r

r = strsubst(r,’,’,’.’); // string substitute
r = evstr(r); // convert string to numerical values

Madhu Belur, CC group, EE, IITB Least square fit

Outline Scilab Least squares Scilab commands

Read/write csv files

csv ≡ comma separated value

r = read csv(’lsquare data.csv’,ascii(9)); // read csv data into r

r = strsubst(r,’,’,’.’); // string substitute
r = evstr(r); // convert string to numerical values

xval = r(1,:); //first row (time)

yval = r(2,:); //second row (displacement)

Madhu Belur, CC group, EE, IITB Least square fit

Outline Scilab Least squares Scilab commands

Displacement under gravity

Constant gravity: g = 9.8 m/s 2 downwards.

Initial velocity: v0
Initial displacement: x0
1
x(t) = x0 + v0 t − gt 2
2

Madhu Belur, CC group, EE, IITB Least square fit

Outline Scilab Least squares Scilab commands

Spoken tutorial?

More spoken tutorials?

Madhu Belur, CC group, EE, IITB Least square fit

Outline Scilab Least squares Scilab commands

Spoken tutorial?

More spoken tutorials?

For least square fit: and remaining topics:

Madhu Belur, CC group, EE, IITB Least square fit

Outline Scilab Least squares Scilab commands

Spoken tutorial?

More spoken tutorials?

For least square fit: and remaining topics:
Enthusiasts/volunteers required for spoken tutorials creation.

Madhu Belur, CC group, EE, IITB Least square fit

Outline Scilab Least squares Scilab commands

Spoken tutorial?

More spoken tutorials?

For least square fit: and remaining topics:
Enthusiasts/volunteers required for spoken tutorials creation.
Be in touch

Madhu Belur, CC group, EE, IITB Least square fit

Outline Scilab Least squares Scilab commands

Spoken tutorial?

More spoken tutorials?

For least square fit: and remaining topics:
Enthusiasts/volunteers required for spoken tutorials creation.
Be in touch

Madhu Belur, CC group, EE, IITB Least square fit