Data Mining

Model Selection
Bob Stine
Dept of Statistics, Wharton School
University of Pennsylvania

Wharton
Department of Statistics

From Last Time

Review from prior class

Calibration
Missing data procedures

Missing at random vs. informative missing

Problems of greedy model selection

Problems with stepwise regression.
So then why be greedy?

Questions

Missing data procedure: Why not impute?

Add an indicator is fast, suited to problems with many missing.
Imputation more suited to small, well-specified models.
EG. Suppose every X has missing values. How many imputation
models do you need to build, and which cases should you use?

Wharton
Department of Statistics

Topics for Today

Over-fitting

Model promises more than it delivers

Model selection procedures

Subset selection
Regularization (aka, shrinkage)
Averaging

Cross-validation
Wharton
Department of Statistics

Model Validation

Narrow interpretation

A predictive model is valid if its predictions

have the properties advertised by model
Calibrated, right on average
Correct uncertainty, at least variance

mean
&
variance

Must know process that selected model

Cannot validate a model from a static,

published perspective
Stepwise model for S&P 500 looks okay, but...

Wharton
Department of Statistics

Model Validation

Fails miserably (as it should) when used to

predict future returns

Predictors are simply random noise

Greedy selection overfits, finding coincidental
patterns

2 RMSE

training

Wharton
Department of Statistics

test
5

Over-Fitting

Critical problem in data mining

Caused by an excess of potential explanatory

variables (predictors)

error rate
Claimed
steadily falls with

size of the model

Over-confident

over-fitting

Model claims to
predict new cases
better than it will.

Challenge
Wharton
Department of Statistics

Select predictors that produce a model that

minimizes the prediction error without over-fitting.

Multiplicity

Why is overfitting common?

Classical model comparison

Test statistic, like the usual t-statistic

Special case of likelihood ratio test

Designed for testing one, a priori hypothesis

Reject if |t| > 2, p-value < 0.05

Problem of multiple testing (multiplicity)

What is the chance
that the largest of p
z-statistics is greater
than 2?

Wharton
Department of Statistics

P(max |z|>1.96)

0.05

0.23

25

0.72

100

0.99

Model Selection

Approaches

Find predictive model without overfitting

Three broad methods

Subset selection

Greedy L0 methods like forward stepwise

Penalized likelihood (AIC, BIC, RIC)

Shrinkage

Regularized: L1 (lasso) and L2 (ridge regression)

Bayesian connections, shrink toward prior

Model averaging
Wharton
Department of Statistics

Dont pick one; rather, average several

Next week
8

Subset Solution

Bonferroni procedure

If testing p hypotheses, then test each at level

/p rather than testing each at level .
Pr(Error in p tests)
= Pr(E1 or E2 or Ep)







Pr(Error ith test)
If test each at level /p, then
Pr(Error in p tests) p(/p) =

Not very popular easy to see why

Loss of power

Bonferroni z

Cost of data-driven

2.6

25
100

3.1
3.5

100000

5.0

Wharton
Department of Statistics

hypothesis testing

Discussion

Bonferroni is pretty tight

Inequality is almost equality if tests are

independent and threshold /p is small

Flexible

Dont have to test every H0 at same level

Allocate more to interesting tests
Split =0.05 with to p linear terms and to all interactions

Process matters

Look at model for stock market in prior class

Many predictors in model pass Bonferroni!
The selection process produces biased estimate of error
Use Bonferroni from the start, not at the end

Wharton
Department of Statistics

10

Popular Alternative Rules

Model selection criteria

AIC (Akaike information criterion, Cp)

BIC (Bayesian information criterion, SIC)
RIC (risk inflation criterion)

Designed to solve different problems

Equivalent to varying p-to-enter threshold

AIC, Cp: Accept variable if
z2 > 2
Equivalent to putting p-to-enter 0.16

BIC:





z2 > log n
Aims to identify the true model

RIC:





z2 > 2 log p Bonferroni
Wharton
Department of Statistics

The more you consider, the stiffer the penalty

11

Penalized Likelihood

Alternative characterization of criteria

Maximum likelihood in LS regression

Find model that minimizes -2 log likelihood

Problem: always adds more variables (max R2)

Penalized methods

Add predictors so long as

-2 log likelihood + (model size)
decreases

Criteria vary in choice of

2 for AIC, (log n) for BIC, (2 log p) for RIC

Wharton
Department of Statistics

12

 JMP output

Example

Osteo example

Results

Add variables so
long as BIC
decreases
Fit extra then
reverts back to
best

AIC vs BIC

AIC: less penalty,

larger model
What happens if try either with stock market model?
Wharton
Department of Statistics

13

Shrinkage Solution

Saturated model

Rather than pick a subset, consider models that

contain all possible features
p = # possible Xs
Good start (and maybe finished) if p << n

Shrinkage allows fitting all if p > n

Shrinkage maximizes penalized likelihood
RSS analogous to
-2 log likelihood

Wharton
Department of Statistics

Penalize by size of the coefficients

Fit has to improve by enough (RSS decrease)
to compensate for size of coefficients
= regularization
Ridge regression: min RSS + 2 bb
parameter,
a tuning parameter
that must be chosen
LASSO regression: min RSS + 1 |bj|
14

Lasso vs Ridge Regression

L1

min RSS, |bj|<c

L2

min RSS, bj2<c

Corners produce selection

Wharton
Department of Statistics

Interpret as Lagrange multiplier.

15

Cross-Validation Solution
Common sense alternative to criteria

Apply the model to new data

Estimate hidden curve plot of over-fitting

No free lunches
Trade-off

More data for testing means less for fitting:

Good estimate of the fit of a poorly estimated model.

Poor estimate of the fit of a well estimated model.

Highly variable
Results depend which group was excluded for testing
Multi-fold cross-validation has become common

Optimistic
Only place I know of a random sample from same population

Wharton
Department of Statistics

Multi-fold: leave out different subsets

1
2
3
4
5
16

Variability of CV

Example

Compare simple and complex osteo models

Need to fit both to the same CV samples Not so easy in JMP

Evaluate one model

Method of validation

Exclude some of the cases

Fit the model to others
Predict the held-back cases
Repeat, allowing missing data to affect results
Compare out-of-sample errors to model claims

Is assessment correct?

Under what conditions?

Wharton
Department of Statistics

17

Osteo Example

CV 50 times, split sample

Variability

Training
Wharton
Department of Statistics

SD of pred errors

SD of residuals

If only did one CV sample, might think model

would be 20% better or 15% worse than claimed!
Test cases
look worse
Test cases
look better
Test
18

CV in Data Mining

DM methods often require a three-way CV

Training sample to fit model
Tuning sample to pick special constants
Test sample to see how well final model does

Methods without tuning sample have advantage

Use all of the data to pick the model, without having

to reserve a portion for the choice of constants
Example: method that has honest p-values, akin to
regression model with Bonferroni

Caution

Software not always clear how the CV is done

Be sure CV includes the choice of form of model

Wharton
Department of Statistics

19

Lasso

Regularized regression model

Find regression that minimizes

Residual SS + |i|
where is a tuning constant
Bayesian: double exponential prior on
Scaling issues
What happens if the s are not on a common scale?

shrinkage

Shrink estimated parameters toward zero

Penalty determines amount of shrinkage
Larger penalty (), fewer variable effects in model

Equivalent to constrained optimization

Wharton
Department of Statistics

20

Lasso Example

How to set the tuning parameter ?

Empirical: Vary to see how fit changes

Cross-validation, typically 10-fold CV

Large values of lead to very sparse models
Shrinks all the way back to zero

Small values of produce dense models

CV compares prediction errors for choices

Implementations

Generalized regression in JMP Pro

glmnet package in R (See James et al, Ch 6)
More naked software than JMP or Stata

Wharton
Department of Statistics

21

 Fit L

Lasso Example
1

regression, Lasso

Plot estimated coefficients as relax penalty

Implemented in JMP as generalized regression

osteo
model
Where to stop
adding features?

Wharton
Department of Statistics

22

Lasso Example in R

Follow script from James

See on-line document Glmnet Vignette

Similar output

Less formatting, but more accessible details

Wharton
Department of Statistics

Repeated 10-fold CV

23

Discussion of CV

Use in model selection vs model validation

Shrinkage methods use CV to pick model
Validation reserves data to test final model

Comments on use in validation

Cannot do selection and validation at same time

Flexible: models do not have to be nested
Optimistic
Splits in CV are samples from one population
Real test in practice often collected later than training data

Population drift

Populations often change over time; CV considers a shapshot

Alternatives?

Bootstrap methods

Wharton
Department of Statistics

24

Take-Aways

Overfitting

Increased model complexity often claims to

produce a better fit, but in fact it got worse

Model selection methods

Criteria such as AIC or p-value thresholds

Shrinkage methods such as lasso

Cross validation

Multiple roles: validation vs model selection

Flexible and intuitive, but highly variable

Wharton
Department of Statistics

25

Some questions to ponder...

If you fit a regression model with 10

coefficients, whats the chance that one is

statistically significant by chance alone?
How can you avoid this problem?

If you have a coefficient in your model that

has a t2, what is going to happen to its
significance if you apply split-sample CV?

Why is cross-validation used to pick lasso

models?

Is further CV needed to validation a lasso fit?

Wharton
Department of Statistics

26

 Thursday

Next Time
Newberry Lab

Hands-on time with JMP, R, and data

Fit models to the ANES data
You can come to class, but I wont be here!

Friday

Wharton
Department of Statistics

July 4th holiday

27