IDRE Statistical Consulting
[Link]
1
• Introduction
• Motivating example: The SAQ
• Pearson correlation
• Partitioning the variance in factor analysis
• Extracting factors
• Principal components analysis
• Running a PCA with 8 components in SPSS
• Running a PCA with 2 components in SPSS
• Common factor analysis
• Principal axis factoring (2-factor PAF)
• Maximum likelihood (2-factor ML)
• Rotation methods
• Simple Structure
• Orthogonal rotation (Varimax)
• Oblique (Direct Oblimin)
• Generating factor scores
2
• Motivating example: The SAQ
• Pearson correlation
• Partitioning the variance in
factor analysis
3
Observed variables
Latent variable
Assumption: the correlations among all observed variables can be explained by latent variable
4
1. I dream that Pearson is attacking me with correlation coefficients
2. I don’t understand statistics
3. I have little experience with computers
4. All computers hate me
5. I have never been good at mathematics
6. My friends are better at statistics than me
7. Computers are useful only for playing games
8. I did badly at mathematics at school
5
There exist varying
magnitudes of
correlation among
variables
Large negative Large positive
6
• Common variance
• variance that is shared among a set of items
• Communality (h2)
• common variance that ranges between 0 and 1
• Unique variance
• variance that’s not common
• Specific variance
• variance that is specific to a particular item
• Item 4 “All computers hate me” → anxiety about computers in addition to anxiety about SPSS
• Error variance
• anything unexplained by common or specific variance
• e.g., a mother got a call from her babysitter that her two-year old son ate her favorite lipstick).
7
In PCA, there is no unique variance. Common variance across
a set of items makes up total variance.
8
Common variance =
Due to factor(s)
Unique variance =
Total variance is Due to items
made up of common
and unique variance
9
• Factor Extraction
• Type of model (e.g., PCA or EFA?)
• Estimation method (e.g., Principal Axis Factoring or Maximum Likelihood?)
• Number of factors or components to extract (e.g., 1 or 2?)
• Factor Rotation
• Achieve simple structure
• Orthogonal or oblique?
10
• Principal components analysis
• PCA with 8 / 2 components
• Common factor analysis
• Principal axis factoring (2-factor PAF)
• Maximum likelihood (2-factor ML)
11
• Principal Components Analysis (PCA)
• Goal: to replicate the correlation matrix using a set of components that are fewer in
number than the original set of items
Recall communality in PCA
PC1
PC1
8 variables 2 components
12
• Eigenvalues
• Total variance explained by given principal component
• Eigenvalues > 0, good
• Negative eigenvalues → ill-conditioned
• Eigenvalues close to zero → multicollinearity
• Eigenvectors
• weight for each eigenvalue
• eigenvector times the square root of the eigenvalue → component loadings
• Component loadings
• correlation of each item with the principal component
• Eigenvalues are the sum of squared component loadings across all items for each
component
13
Analyze – Dimension Reduction – Factor
Note: Factors are NOT the same as Components
8 components is NOT what you typically want to use
14
Component loadings
Sum of squared loadings across
correlation of each item with the principal
components is the communality
component
1 0.6592 = 0.434
1
43.4% of the variance
explained by first
component (think R-square)
1
0.1362 = 0.018
1
1.8% of the variance
explained by second
1
component
1 Q: why is it 1?
1
Excel demo
1
Sum squared loadings down each
3.057 1.067 0.958 0.736 0.622 0.571 0.543 0.446 column (component) = eigenvalues 15
3.057 1.067 0.958 0.736 0.622 0.571 0.543 0.446
Why is the left column the same as the right?
Look familiar? Extraction Sums of Squared Loadings = Eigenvalues
16
3.057 Why is eigenvalue greater than 1 a
criteria?
Recall eigenvalues represent total
variance explained by a component
Since the communality is 1 in a PCA
Look for the for a single item, if the eigenvalue is
elbow greater than 1, it explains the
communality of more than 1 item
1.067
0.958
0.736 0.622
0.571 0.543
0.446
17
Analyze – Dimension Reduction – Factor
Goal of PCA is
dimension reduction
This is more realistic
than an 8-component
solution
18
Notice communalities not equal 1
Recall these numbers from the 8-component solution
3.057 1.067 0.958 0.736 0.622 0.571 0.543 0.446
Notice only two eigenvalues
84.0% of the total variance in Item 2 is explained by Comp 1. How would you derive and interpret these communalities? 19
20
• Principal components analysis
• PCA with 8 / 2 components
• Common factor analysis
• Principal axis factoring (2-factor PAF)
• Maximum likelihood (2-factor ML)
21
• Factor Analysis (EFA)
• Goal: also to reduce dimensionality, BUT assume total variance can be divided into
common and unique variance
• Makes more sense to define a construct
with measurement error
8 variables
1 variable = factor
22
Analyze – Dimension Reduction – Factor
Make note of the word SPSS does not change
eigenvalue it will come its menu to reflect
back to haunt us later changes in your
analysis. You have to
know the
idiosyncrasies yourself.
23
Initial communalities are
the squared multiple
correlation coefficients
controlling for all other
items in your model
Q: what was the initial
communality for PCA?
Sum of communalities across items = 3.01
24
Unlike the PCA model, the
sum of the initial
eigenvalues do not equal
the sums of squared
loadings
Sum eigenvalues = 4.124
2.510 0.499
Sum of squared loadings Factor 1 = 2.51
Sum of squared loadings Factor 2 = 0.499
The reason is because (recall) Sum of communalities across items = 3.01
Eigenvalues are for PCA
not for factor analysis!
(SPSS idiosyncrasies)
25
Analyze – Dimension Reduction – Factor
Caution!
Eigenvalues are only for
PCA, yet SPSS uses the
eigenvalue criteria for EFA
When you look at the
scree plot in SPSS, you are
making a conscious
decision to use the PCA
solution as a proxy for
your EFA
26
27
Sum of squared loadings across
factors is the communality Communalities
These are analogous to component loadings in PCA
Squaring the 0.438 0.5882 = 0.346
loadings and
summing up 0.052 34.5% of the variance in
gives you either Item 1 explained by first
the factor
Communality or 0.319
(-0.303)2 = 0.091
the Extraction
Sums of 0.461 9.1% of the variance in Item
Squared
1 explained by second
Loadings
factor
0.344
Summing down
the 0.309 0.345 + 0.091 = 0.437
communalities or 43.7% of the variance in
across the 0.850 Item 1 explained by both
eigenvalues gives factors = COMMUNALITY!
0.236
you total Sum squared loadings down each
variance column = Extraction Sums of Square
2.510 0.499 3.01
explained (3.01) Loadings (not eigenvalues) 28
Communalities
Caution when interpreting unrotated
loadings. Most of total variance
explained by first factor.
Which item has the least total variance explained by both factors? 29
EFA Communalities
or components
Excel demo PCA 8 3.01 EFA 30
31
Analyze – Dimension Reduction – Factor New output
A significant chi-
square means you
reject the current
hypothesized model
This is telling us we reject
the two-factor model
32
Number Chi- Iterations
of Factors square Df p-value needed
Chi-square and 1 553.08 20 <0.05 4
degrees of freedom
goes down
2 198.62 13 < 0.05 39 Want NON-
3 13.81 7 0.055 57 significant chi-
4 1.386 2 0.5 168 square
The three factor 5 NS -2 NS NS
model is preferred
from chi-square 6 NS -5 NS NS
Iterations
7 NS -7 NS NS needed
8 N/A N/A N/A N/A goes up
An eight factor model is not possible in SPSS
33
34
EFA: Total Variance PCA: Total Variance
Explained = Total Explained = Total
Communality Variance
Explained NOT
Total Variance
For both models, Communalities are
communality is the item specific
total proportion of
variance due to all
factors or
components in the
model
35
36
(across all items)
37
• Simple Structure
• Orthogonal rotation (Varimax)
• Oblique (Direct Oblimin)
38
1. Each item has high loadings on one factor only
2. Each factor has high loadings for only some of the items.
Pedhazur and Schemlkin (1991)
Item Factor 1 Factor 2 Factor 3 The goal of rotation
is to achieve simple
1 0.8 0 0 structure
2 0.8 0 0
3 0.8 0 0
4 0 0.8 0
5 0 0.8 0
6 0 0.8 0
7 0 0 0.8
8 0 0 0.8
39
1. Most items have high loadings on more than one factor
2. Factor 3 has high loadings on 5/8 items
Item Factor 1 Factor 2 Factor 3
1 0.8 0 0.8
2 0.8 0 0.8
3 0.8 0 0
4 0.8 0 0
5 0 0.8 0.8
6 0 0.8 0.8
7 0 0.8 0.8
8 0 0.8 0
40
Without rotation, first factor is the most general factor onto which most items load and explains the largest amount of variance
Varimax: Orthogonal means the
orthogonal rotation factors are uncorrelated
maximizes
variances of the
loadings within the
factors while
maximizing
differences
between high and
low loadings on a
particular factor
41
The factor
transformation matrix
turns the regular factor The amount of rotation
matrix into the rotated is the angle of rotation
factor matrix
42
43
Unrotated solution Communalities Varimax rotation Communalities
0.438 0.437
maximizes sum of
0.052 0.052 the variance of
squared loadings
within each factor
0.319 0.319
0.461 0.461
0.344 0.344
0.309 0.309
0.850 0.850
0.236 0.236
communalities are
the same
44
Unrotated Solution Varimax Solution
Higher absolute loadings in Varimax solution for Tech Anxiety 45
True or False: Rotation changes how the variances are distributed but not the total communality
maximizes
variances of the
loadings
3.01 3.01
Even though the distribution of the variance is different the total
sum of squared loadings is the same
Answer: T
46
Quartimax: maximizes the squared loadings so that
each item loads most strongly onto a single factor.
Good for generating a single factor.
The difference between Quartimax and
unrotated solution is that maximum
Varimax: good for distributing among more than one factor variance can be in a factor that is not the
first
47
• factor pattern matrix
• partial standardized regression coefficients of each item with a particular factor
• Think (P)artial = Pattern
• factor structure matrix
• simple zero order correlations of each item with a particular factor
• Think (S)imple = Structure
• factor correlation matrix
• matrix of intercorrelations among factors
48
When Delta =0 → Oblique rotation
Direct Quartimin means the factors
are correlated
Larger delta
increases
correlation among
factors
Negative delta
increases makes
factors more
orthogonal
49
50
angle of correlation ϕ
determines whether the factors
are orthogonal or oblique
angle of axis rotation θ
how the axis rotates in relation to
the data points (analogous to
rotation in orthogonal rotation)
51
The more correlated If the factors are
the factors, the orthogonal, the
greater the correlations between
difference between them would be zero,
pattern and structure then the factor
matrix pattern matrix would
EQUAL the factor
structure matrix.
52
Partial standardized regression coefficients Simple zero order correlations
(can exceed one) (can’t exceed one)
0.566 0.537 0.653 is the simple
0.740 is the correlation of Factor 1
effect of 0.037 0.082 on Item 1
Factor 1 on
Item 1 0.489 Note that the sum of
controlling 0.252 squared loadings do NOT
for Factor 2 0.661 match communalities
0.436
There IS a way 0.489
to make the
0.337
sum of squared 0.464
loadings equal
to the 0.260
communality. 1.185
Think back to 0.871
Orthogonal 0.215 0.344
Rotation.
53
Note: now the sum
of the squared
loadings is HIGHER
This is exactly the than the unrotated
same as the solution
unrotated 2-factor
PAF solution SPSS uses the
structure matrix to
calculate this
SPSS uses the -factor contributions
structure matrix to will overlap and
calculate this become greater
-factor contributions than the total
will overlap and variance
become greater
than the total
variance 3.01 4.25
54
Partial loadings Zero-order loadings
Correlations same
Lower absolute loadings of Items 4,8 onto Tech Anxiety for Pattern Matrix
55
Structure Matrix Pattern Matrix
Why do you think
the second
loading is lower in
the Pattern
Matrix compared
to the Structure
Matrix?
56
• There is no consensus about which one to use in the literature
• Hair et al. (1995)
• Better to interpret the pattern matrix because it gives the unique contribution of the factor
on a particular item
• Pett et al. (2003)
• Structure matrix should be used for interpretation
• Pattern matrix for obtaining factor scores
• My belief: I agree with Hair
Hair, J. F. J., Anderson, R. E., Tatham, R. L., & Black, W. C. (1995). Multivariate data analysis . Saddle River.
Pett, M. A., Lackey, N. R., & Sullivan, J. J. (2003). Making sense of factor analysis: The use of factor analysis for instrument development in health care research. Sage.
57
58
• Regression
• Bartlett
• Anderson-Rubin
59
Analyze – Dimension Reduction – Factor – Factor Scores
What it looks like in SPSS Data View
60
-0.452 -0.452
This is how the factor scores are generated
-0.733 -0.733
SPSS takes the standardized scores for each item 1.32 1.32
Then multiply each score
-0.829 -0.829
-0.749 -0.749
-0.203 -0.203
0.0692 0.0692
-1.42 -1.42
-0.880 -0.113
61
Covariance matrix of the true factor scores Covariance matrix of the estimated factor scores
Notice that for Direct Regression method has
Quartimin, the raw factor score mean of zero,
covariances do not match and variance equal to the
squared multiple
correlation of estimated
and true factor scores
62
Notice that for Direct Quartimin, the raw
correlations do match (property of
Regression method)
However, note that the factor scores are
still correlated even though we did
Varimax
63
• 1. Regression Method
• Variance equals the square multiple correlation between factors and variables
• Maximizes correlation between estimated and true factor scores but can be biased
• 2. Bartlett
• Factor scores highly correlate with own true factor and not with others
• Unbiased estimate of true factor scores
• 3. Anderson-Rubin
• Estimated factor scores become uncorrelated with other true factors and uncorrelated with
other estimated factor scores
• Biased especially if factors are actually correlated, not for oblique rotations
64
Direct Quartimin
65
66
