A Tool for Linear Prediction
What is regression?
refers to the method of predicting the value of one variable on the basis of the known value/s of other variable/s done by establishing a simplified statement, or a model, of the relationship between a variable and another variable or set of variables
Example of a Statement
Y = + 1X1 + 2X2 + 3X3 + This statement is called a regression equation This implies that the value of Y, except for a random error, is determined by the values of the variables X1, X2 and X3. When the observed variations in the values of Y can largely be accounted for by the values of X1, X2 and X3 then the statement is acceptable.
What is a regression equation?
A regression equation is the equation of the line which is defined by the path of the means of Ys for fixed Xs, or the regression equation of Y on X. Assumptions: equation is linear Y is normally distributed for each value of X Var(Y) is the same for each value of X
A Graphical Presentation
Y
Income
X1
X2
X3
X4
X5
X6
X7
Education Fig. 1. General form of regression of Y on X or the path of the means of Y values for fixed values of X
Method for finding the regression line
Least squares procedure - method for finding the straight line that minimizes the squared deviations of points around it
Assumptions: 1. Random sampling 2. E( i) = 0 3. Xi and i are statistically independent
A Graphical Presentation
Y
vertical distances between the line and the points X
Fig. 1. Least-squares equation minimizing sum of squares of vertical distances and estimating the regression of Y on X
Components of deviations
SST = sum of squared deviations between the mean and the observed values (d.f. = n -1) SSR = sum of squared deviations between the observed values and the corresponding fitted values (d.f. = p - 1, or the number of explanatory variables) SSE = unexplained deviations; = SST - SSR with d.f. = n - p
Statistics for Evaluating Regression Equation
F =
MSR MSE
where, MSR = SSR/(p - 1) MSE = SSE/(n - p) Thus, F is a measure of the relative size of the variations accounted for by the fitted regression line to the unexplained remaining variations.
Statistics for Evaluating Regression Equation
Coefficient of determination, R2 R2 = SSR/SST R2 = 1.0 implies that the regression line is a perfect fit R2 = 0.0 implies that the regression line fails to explain the deviations in the observed values; ie, all deviations are accounted for by SSE
Statistics for Evaluating Regression Equation
Regression coefficient - quantitative measure of the change in the value of the dependent variable (Y) that is effected by a unit change in the value of the explanatory variable (X).
Y = a + b1X1 + b2X2 + b3X3 + b4X4
regression coefficients
Statistics for Evaluating Regression Equation
Partial Fi - measure of the significance of the contribution of the regression coefficient bi , given that the other variables are included in the model.
Y = a + b1X1 + b2X2 + b3X3 + b4X4
F1 F2 F3 F4
Stepwise Regression
Used when there is no definite model to be tested, or when from a set of variables a subset is to be identified as the explanatory variables
Variables are entered into (or step in) or removed from (or step out) the model one at a time
Stepping Method Criteria:
significance of the associated F-value
The explanatory variable with the highest F-value is
entered first
A variable is not entered if the contribution to the overall
F-value is not significant
Regression with Dummy Variables
Regression procedure when at least one of the explanatory variables are categorical (either nominal or ordinal) A categorical variable with k levels is converted into (k - 1) indicator variables Example: Levels of the original variable, X4 : X41, X42, X43, X44 Dummy variables : DX41, DX42, DX43
Regression with Dummy Variables
The dummies are defined as follows: DX41 = 1, if the unit belongs to X41 level of variable X4 0, otherwise DX42 = 1, if the unit belongs to X42 level of variable X4 0, otherwise DX43 = 1, if the unit belongs to X43 level of variable X4 0, otherwise Note that when the three dummies are all zero, the unit belongs to the omitted level of variable X4. Thus, there is no need for another dummy variable.
Regression with Dummy Variables
The Model: Y = a + b1X1 + b2X2 + b3X3 + b4X4 If X4 is a categorical variable, the adjusted model is: Y = a + b1X1 + b2X2 + b3X3 + b4DX41 + b42DX42 + b43DX43
Regression with Dummy Variables
Interpretation of the Coefficients of Dummy Variables Value of X4 X41 X42 X43 X44 Mean value of Y Y = a + b1X1 + b2X2 + b3X3 + b41 Y = a + b1X1 + b2X2 + b3X3 + b42 Y = a + b1X1 + b2X2 + b3X3 + b43 Y = a + b1X1 + b2X2 + b3X3
