Analysis of Variance Approach
to Regression Analysis
Rationale
The Analysis of Variance (ANOVA) is a
statistical principle that is based on
partitioning total observed variation into
several components with the aim of trying
to explain the sources of such variation.
Total observed variation is often measured
by the total of the squared deviations of
each observation from the mean.
2
Rationale
In the context of regression analysis in
which we presume that the observations on
the response variable can be expressed as a
(linear) function of the independent
variables in the form of
yi 0 1 xi ei
Rationale
Based on sample data, and assuming that
such a relation is true, the line that best fits
the observed values is obtained as
yi 0 1 xi
After fitting the said regression line, we
now gather some evidence if indeed such
model really holds in describing such
relationship.
4
Rationale
y
yi
( yi yi )
( yi y )
yi 0 1 xi
( yi y )
yi
( yi y ) ( yi yi ) ( yi y )
x
5
Total Deviation
( yi y ) ( yi yi ) ( yi y )
TOTAL DEVIATION
Deviation of fitted regression
value around the mean
Deviation around fitted
regression line
Sum of Squares
2
2
(
y
y
)
[(
y
y
)
(
y
y
)]
i
i
i
i
( yi y ) 2 ( yi yi ) 2 2 ( yi yi )( yi y )
( yi y ) 2 ( yi y ) 2 ( yi yi ) 2
Total Sum of Squares
(TSS)
Sum of Squares due
to the Regression of y
on x (SSR)
Sum of Squares Error
(SSE)
TSS = SSR + SSE
7
Degrees of Freedom
Total Degrees of Freedom (associated with
TSS) is (n-1). One degree of freedom is
lost because:
The deviations ( yi y ) is subject to one
constraint: sum=0; or,
The sample mean is used to estimate the
population mean
Degrees of Freedom
Degrees of Freedom due to Error : n-2.
Two degrees of freedom are lost because
we are estimating two parameters 0 and 1
in obtaining the fitted value yi
Degrees of Freedom
Degreed of Freedom due to Regression: 1.
Although there are n deviations ( yi y ) , all
fitted values yi are calculated from the same
regression line.
Two df is associated with regression line but
1 df is lost because the deviations ( yi y )
are subject to one constraint: sum is zero
Thus, df Total df Regression df Error
10
Mean Squares
In a general ANOVA, the mean squares
are obtained by dividing the SS with it
corresponding df. That is
MSTot = SSTot/(n-1)
MSR = SSR/1; MSE = SSE/(n-2)
Note: MSTot MSR + MSE
11
ANOVA Table
Results of the Analysis of Variance are
summarized in an ANOVA table:
Source of
Variation
Regression
Error
Total
df
SS
MS
1
n-2
n-1
SSR
SSE
MSR
MSE
12
Expected Mean Squares (EMS)
The EMS are useful quantities that:
Tells us what parametric function is being
estimated by the MS [Method of Moments
Estimator]
In some instances, this will suggest how the
test-statistic will be defined to test specific
hypotheses.
13
Expected Mean Squares (EMS)
2
E
[
MSE
]
The mean of the sampling distribution of
MSE is 2 whether or not X and Y are
linearly related ( whether or not 1=0)
( SSE ) / 2 ~
(2n2) E[ SSE / 2 ] n 2
SSE
2
E
E
[
MSE
]
n 2
14
Expected Mean Squares (EMS
E[ MSR ] 1
2
2
(
x
x
)
i
The mean of the sampling distribution of
MSR is also 2 when 1=0. In this case,
MSR and MSE will tend to be of the same
magnitude.
When, 10, MSR > MSE. Thus, a
comparison of MSR and MSE may be
used to determined whether or not 1=0.
15
Test of Hypothesis:
H0:1=0 vs H1: 10
From the EMS, it appears to be logical that
to test this hypothesis, one can compare
MSR and MSE.
From statistical theory and assuming
normality of the error terms (Cochrans
Theorem):
16
Test of Hypothesis:
H0:1=0 vs H1: 10
MSR and MSE are independent
MSR ~
2
(1)
, MSE ~ (2n 2)
Thus, a logical test-statistic (GLRT) would
be
MSR
Fc
MSE
~ F(1,n 2)
Reject H0 for large values of Fc or if
Fc F ,(1,n2)
17
General Linear Test Approach
Another approach to test the hypothesis
concerning regression parameters (or a
function of such parameters).
First Fit the Full Model. In SLR case,
yi 0 1 xi ei
Compute:
SSE ( F ) ( yi yi ) 2 ( yi [ 0 1 xi ]) 2 SSE
18
General Linear Test Approach
Fit the Reduced Model under H0
(assuming H0 is true). In the SLR case,
H0: 1=0. This means, we fit the (reduced)
model: yi 0 ei
In this case, the value of 0 that minimizes
2
) 2 is y
e
(
y
i i 0
0
19
General Linear Test Approach
Compute the SSE for the reduced model
as: SSE ( R) ( yi 0 ) 2 ( yi y ) 2 SST
Note that in general, SSE ( F ) SSE ( R) due to
the fact that the more parameters are
employed in model fitting, the better the
fit.
20
General Linear Test Approach
Test Statistic:
SSE ( R ) SSE ( F )
df R df F
*
F
~ F[ df R df F ,df F ]
SSE ( F )
df F
Note that in the case of the SLR and
testing H0:1=0,
21
General Linear Test Approach
SST SSE
SSR
MSR
(n 1) (n 2)
*
1
F
SSE
MSE MSE
n2
Thus, the two tests are equivalent.
This approach can be extended for more
complex tests.
22
