Goldsman ISyE 6739 Linear Regression
REGRESSION
12.1 Simple Linear Regression Model
12.2 Fitting the Regression Line
12.3 Inferences on the Slope Parameter
1
Goldsman ISyE 6739 12.1 Simple Linear Regression Model
Suppose we have a data set with the following paired
observations:
(x1, y1), (x2, y2), . . . , (xn, yn)
Example:
xi = height of person i
yi = weight of person i
Can we make a model expressing yi as a function of
xi ?
2
Goldsman ISyE 6739 12.1 Simple Linear Regression Model
Estimate yi for fixed xi. Lets model this with the
simple linear regression equation,
yi = 0 + 1xi + i,
where 0 and 1 are unknown constants and the error
terms are usually assumed to be
iid
1, . . . , n N (0, 2)
yi N (0 + 1xi, 2).
3
Goldsman ISyE 6739 12.1 Simple Linear Regression Model
y = 0 + 1x
with high 2
y = 0 + 1x
with low 2
4
Goldsman ISyE 6739 12.1 Simple Linear Regression Model
Warning! Look at data before you fit a line to it:
doesnt look very linear!
5
Goldsman ISyE 6739 12.1 Simple Linear Regression Model
xi yi
Production Electric Usage
($ million) (million kWh)
Jan 4.5 2.5
Feb 3.6 2.3
Mar 4.3 2.5
Apr 5.1 2.8
May 5.6 3.0
Jun 5.0 3.1
Jul 5.3 3.2
Aug 5.8 3.5
Sep 4.7 3.0
Oct 5.6 3.3
Nov 4.9 2.7
Dec 4.2 2.5
6
Goldsman ISyE 6739 12.1 Simple Linear Regression Model
3.4
yi 3.0
2.6
2.2
3.5 4.0 4.5 5.0 5.5 6.0
xi
Great... but how do you fit the line?
7
Goldsman ISyE 6739 12.2 Fitting the Regression Line
Fit the regression line y = 0 + 1x to the data
(x1, y1), . . . , (xn, yn)
by finding the best match between the line and the
data. The bestchoice of 0, 1 will be chosen to
minimize
n n
))2 2
X X
Q= (yi (0 + 1xi = i.
i=1 i=1
8
Goldsman ISyE 6739 12.2 Fitting the Regression Line
This is called the least square fit. Lets solve...
Q P
0 = 2 (yi (0 + 1xi)) = 0
Q P
1 = 2 xi(yi (0 + 1xi)) = 0
P P
yi = n0 + 1 xi
P P
xiyi = 2 xi(yi (0 + 1xi)) = 0
After a little algebra, get
P P P
n xiyi( xi)( yi)
1 =
n x2
P P
i ( xi)2
0 = y 1x, where y 1 Py and x 1 Px .
n i n i
9
Goldsman ISyE 6739 12.2 Fitting the Regression Line
Lets introduce some more notation:
x)2 x2 2
P P
Sxx = (xi = i nx
xi)2
P
(
x2
P
= i n
P P
Sxy = (xi x)(yi y) = xiyi nxy
P P
P ( xi)( yi )
= xiyi n
These are called sums of squares.
10
Goldsman ISyE 6739 12.2 Fitting the Regression Line
Then, after a little more algebra, we can write
Sxy
1 =
Sxx
Fact: If the is are iid N (0, 2), it can be shown that
0 and 1 are the MLEs for 0 and 1, respectively.
(See text for easy proof).
Anyhow, the fitted regression line is:
y = 0 + 1x.
11
Goldsman ISyE 6739 12.1 Simple Linear Regression Model
Fix a specific value of the explanatory variable x, the
equation gives a fitted value y|x = 0 + 1x for the
dependent variable y.
12
y
y = 0 + 1x
y|x
x
x xi
Goldsman ISyE 6739 12.2 Fitting the Regression Line
For actual data points xi, the fitted values are yi =
0 + 1xi.
observed values : yi = 0 + 1xi + i
fitted values : yi = 0 + 1xi
Lets estimate the error variation 2 by considering
the deviations between yi and yi.
SSE = (yi yi = (yi (0 + 1xi))2
)2
P P
= yi2 0 yi 1 xiyi.
P P P
13
Goldsman ISyE 6739 12.2 Fitting the Regression Line
Turns out that 2 SSE
n2 is a good estimator for 2.
P12
Example: Car plant energy usage n = 12, i=1 xi =
x2 yi2 = 98.697,
P P P
58.62, yi = 34.15, i = 291.231,
P
xiyi = 169.253
1 = 0.49883, 0 = 0.4090
fitted regression line is
y = 0.409 + 0.499x y|5.5 = 3.1535
What about something like y|10.0?
14
Goldsman ISyE 6739 12.3 Inferences on Slope Parameter 1
Sxy
(xi x)2 and
P
1 = Sxx , where Sxx =
P P P
Sxy = (xi x)(yi y) = (xi x)yi y (xi x)
P
= (xi x)yi
15
Goldsman ISyE 6739 12.3 Inferences on Slope Parameter 1
Since the yis are independent with yi N(0+1xi, 2)
(and the xis are constants), we have
E1 = S1xx ESxy = S1xx (xi x)Eyi = X1xx
P P
(xi x)(0 + 1x
= 1 [ X(x x) + P(x x)x ]
Sxx 0 | i
{z } 1 i i
0
1 P 2 1 X 2 2)
= Sxx (x i x i x) = (
Sxx | x
i {z nx = 1
}
Sxx
1 is an unbiased estimator of 1.
16
Goldsman ISyE 6739 12.3 Inferences on Slope Parameter 1
Further, since 1 is a linear combination of indepen-
dent normals, 1 is itself normal. We can also derive
1 1 X 2
Var(1) = 2 Var(Sxy ) = 2 (xi x)2Var(yi) = .
Sxx Sxx Sxx
2
Thus, 1 N(1, Sxx )
17
Goldsman ISyE 6739 12.3 Inferences on Slope Parameter 1
While were at it, we can do the same kind of thing
with the intercept parameter, 0:
0 = y 1x
Thus, E0 = Ey xE1 = 0 + 1x x1 = 0 Similar
to before, since 0 is a linear combination of indepen-
dent normals, it is also normal. Finally,
x2
P
Var(0) = i 2.
nSxx
18
Goldsman ISyE 6739 12.3 Inferences on Slope Parameter 1
Proof:
Cov(y, 1) = 1 Cov(y, P(x x)y )
Sxx i i
P
(xix
= Sxx )Cov(y, yi)
P
(xix) 2
= Sxx n = 0
Var(0) = Var(y 1x)
= Var(y) + x2Var1 2x Cov(y,
| {z
1)}
0
2
= n + x2 Sxx 2
2
= 2 Sxx nx .
nSxx
P 2
xi 2
Thus, 0 N(0, nSxx ).
19
Goldsman ISyE 6739 12.3 Inferences on Slope Parameter 1
Back to 1 N(1, 2/Sxx) . . .
1 1
q N(0, 1)
2/Sxx
Turns out:
SSE 22(n2)
(1) 2
= n2 n2 ;
(2) 2 is independent of 1.
20
Goldsman ISyE 6739 12.3 Inferences on Slope Parameter 1
1
1
/ Sxx N(0, 1)
s t(n 2)
/ 2(n2)
n2
1 1
t(n 2).
/ Sxx
21
Goldsman ISyE 6739 12.3 Inferences on Slope Parameter 1
t(n 2)
t/2,n2 t/2,n2
22
Goldsman ISyE 6739 12.3 Inferences on Slope Parameter 1
2-sided Confidence Intervals for 1:
1 = Pr(t/2,n2 1 1
t/2,n2)
/ Sxx
= Pr(1 t/2,n2 1 1 + t/2,n2 )
Sxx Sxx
1-sided CIs for 1:
1 (, 1 + t,n2 )
Sxx
1 (1 t,n2 , )
Sxx
23
