Mixed Methods Research Design

and Analysis With Validity: A Primer

R. Burke Johnson
Version 3.0, Copyright 2014
 Mixed Methods Research Design and
Analysis With Validity: A Primer
• This is a free document to be shared with people
interested in learning about mixed methods
research.

• To cite this document, use the following:

Johnson, R. B. (2014). Mixed methods research

design and analysis with validity: A primer.
Department of Professional Studies, University of
South Alabama, USA.
 Table of Contents
• Introduction, Definition, and Major MMR Types
• Research Questions
• Major Types of Data Collection
• Sampling in MMR
• Research Validity or Legitimation in MMR
• Research Designs in MMR
• Data Analysis in MMR
• Report Writing in MMR
Some of the material in this primer is based on
the following research methods textbook:
“Educational Research: Quantitative,
Qualitative, and Mixed Approaches” 5th ed.)
by
Burke Johnson and Larry Christensen
(2014, Sage Publications)
 Mixed Methods Research
Introduction
• Mixed methods research (MMR)—(also called
mixed research or mixed methodology) is the
type of research in which a researcher or team
of researchers mixes or combines qualitative
and quantitative research
philosophies/paradigms, methodologies,
methods, techniques, approaches, concepts,
or language into a single research study or a
set of related studies. The use of multiple
disciplines is also integral to MMR.
 MMR Introduction (cont.)
• Greene (2014) contends, and I agree, that mixed
methods research encourages interactive
combination or mixing at 3 levels (or more):
1. Method (or what I call method of data
collection)
2. Methodology (or what I call research
method in quantitative research and theoretical
framework in qualitative research)
3. Paradigm.
 MMR Introduction (cont.)
Three major types of MMR:
• Qualitatively driven or qualitative dominant
• Quantitatively driven or quantitative dominant
• Interactive or equal status.
(Read: Johnson, Onwuegbuzie, & Turner’s 2007
JMMR article. Click here)
Note: Teddlie and Tashakkori call the first two quasi
mixed research.
 MMR Introduction (cont.)
Proponents of mixed research adhere to some form of
the compatibility thesis (i.e., “it’s okay to thoughtfully
mix”) and, often (but not always), to the philosophy
of pragmatism (provides ontology, epistemology,
axiology, and philosophical methodology) (Note:
some prefer critical realism)
• Epistemologically, goal is to make warranted
assertions
• See Johnson and Onwuegbuzie (2004) article (click
here) and Johnson and Gray (2010) chapter (click
here)
 MMR Introduction (cont.)
• Mixing usually is done to obtain breadth and
depth of understanding, perspective,
complexity, and difference and/or
corroboration.
• Example: would be important to study the
quantitative and qualitative aspects of an
organization such as a school or a city
government.
 MMR Introduction (cont.)
• Mixed methods research is one of the three
major research methodology paradigms:
– Qualitative research
– Quantitative research
– Mixed research.

• Click here for an overview of quantitative,

qualitative, and mixed research
• Click here for Comparison Table
 MMR Introduction (cont.)
• Example: study the quantitative and qualitative
aspects of a school.

• Fundamental principle of mixed research: advises

researchers to thoughtfully and strategically mix
or combine qualitative and quantitative research
methods, approaches, procedures, concepts, and
other paradigm characteristics in a way that
produces an overall design with multiple
(divergent and convergent) and complementary
strengths (broadly viewed) and nonoverlapping
weaknesses.
 MMR Introduction (cont.)
In my fundamental principle, I view the
“complementary strengths” condition broadly
to include all of the purposes for mixing
identified by Jennifer Greene et al. (1989) to
produce a better “whole”:
• Triangulation (convergence, corroboration)
• Complementarity (elaboration, enhancement)
• Development (one method informs the other)
• Initiation (find contradictions, perspectives)
• Expansion (expand the breadth of study)
 Purposes for mixing
There are perhaps hundreds of more specific reasons for mixing
in addition to the five identified by Greene et al. (1989). Here
are a few:
• Explaining complexity,
• Juxtaposition-dialogue/comparison-synthesis,
• Explaining interaction between/among natural and human
systems,
• Determining what works for whom and the
relevance/importance of context,
• Describing/explaining process and outcomes,
• Sequentially generating and testing theory,
• Continued on next page
 Purposes for Mixing (cont.)
• Producing interdisciplinary substantive theory,
including/comparing multiple perspectives and data
regarding a phenomenon;
• Breaking down binaries/dualisms (some of both);
• Iteratively/sequentially connecting local/idiographic
knowledge with national/general/nomothetic
knowledge;
• Learning from different perspectives on teams and in
the field and literature;
• Achieving multiple participation, social justice, and
action . . .
• and the list continues.
 Mixed Methods Research Questions
In all research, the research questions should drive the
study.
• Your goal is to answer your research questions (which
can be thoughtfully modified during your study if
needed)
• Methods are tools that help us obtain data to answer
our substantive research questions
• Methods also can suggest new ways of looking at
issues/questions, but must be careful to not be
method-centric (e.g., I only use qualitative research
methods or I only use any one kind of research
method)
Flowchart of topic, problem, purpose
RQs, hypotheses.
 Mixed Methods Research Questions
Multiple perspectives about MM RQs:
• In QUAN or QUAL driven, you have (a) primary
questions and (b) supplemental questions that add to
the overall topic
• Can use separate QUAN and QUAL questions and then
combine in analysis and interpretation
• QUAL and QUAN can address one question. Robert Yin
(and I) claims QUAL or QUAN can virtually always add
something to the RQ answer. Recommends that you
examine your question both ways. Look at the
“research object” quantitatively and qualitatively.
 Mixed Methods Data Collection
(For Burke’s overview of the major methods of
data collection click here)(For the 2003
Johnson and Turner chapter on data collection
in MMR, click here)

I contend there are six major methods of data

collection (and many additional smaller types
and creative combinations/constructions). . .
 Data collection (cont.)
Here are the six major methods of data collection:
1. Tests
2. Questionnaires
3. Interviews
4. Observation
5. Focus groups
6. Constructed and secondary-or-existing data.

Note: technological advancements can be used for each

of these (e.g., internet for questionnaires, GIS for
observation).
 Data collection (cont.)
The six major methods can be mixed with each other;
--Called inter-method mixing
and
The six major methods have mixed versions;
--Called intra-method mixing.
Note: The terms inter- and intra-method also can be
used at the level of methodology and paradigm.
(Terms were coined in chapter by Johnson and Turner,
2003)
 Data collection (cont.)
Before I list some MMR data collection
approaches, an important point:
• Method of data collection does not logically
entail epistemology (and vice versa) but there
is mutual influence
• A qualitative researcher and a quantitative
researcher will instantiate quite different
forms of, e.g., observations or interviews
 Data collection (cont.)
• In the Handbook of Mixed Methods, our
chapter (Johnson & Turner, 2003) outlines
intra- and inter-data collection method mixing
• There are quantitative and qualitative versions
of the six major methods of data collection
• And, there are mixed versions of each of the
six major methods of data collection (intra-
method mixing)
 Data collection (cont.)
• Mixed questionnaires
• Includes a mixture of open-ended and closed-
ended items on one or more questionnaires
• Note: a single open-ended question at the end
of a questionnaire, practically speaking, does
not produce a “mixed questionnaire.”
Technically speaking, it does.
 Data collection (cont.)
• Mixed interviews
• Includes a mixture of depth-interviewing and
systematic/targeted/variable-oriented
interviewing.
• Can be done in a single interview or
• Can be done in separate interviews in a study
 Data collection (cont.)
• Mixed focus groups
• Includes a mixture of a priori and
emergent/flowing focus group strategies.
• Usually done in single focus group because
focus groups are primarily qualitative
 Data collection (cont.)
• Mixed testing
• Includes a mixture of standardized open-
ended and closed-ended pre-made tests or
test components
• Testing tends to be primarily qualitative, but
even this can be complemented via QUAL
 Data collection (cont.)
• Mixed observation
• Includes mixture of standardized/confirmatory
and less structured/exploratory observation
• Often alternates between participatory and
nonparticipatory researcher roles
 Data collection (cont.)
• Mixed constructed and secondary or existing
data
• Includes mixture of non-numeric and numeric
information, documents, and archived data
based on narrative/stories/pictures and
numbers, open-ended and closed-ended
information
 Sampling Overview
(For Burke’s notes on sampling in quantitative,
qualitative, and mixed research, click here)
 Sampling in Mixed Research
Typically, researcher selects quantitative sample
using quantitative sampling technique and
qualitative using qualitative sampling
technique.
A question is, how do they interrelate?
 MMR Sampling (cont.)
• Mixed research relies on both QUAN and
QUAL sampling methods

• Mixed research sampling classified into

“mixed sampling designs.”

• The following typology was developed in

collaboration with Tony Onwuegbuzie
 MMR Sampling (cont.)
Mixed sampling designs are classified on two
major criteria:
1. Time orientation: “Do the quantitative and
qualitative phases/components occur
concurrently or sequentially?”
– Concurrent time orientation: data collected for
quantitative and qualitative at approximately
same time.
– Sequential time orientation: data obtained in
stages.
 MMR Sampling (cont.)
2. Sample relationship “Is relationship between
quantitative and qualitative samples identical,
parallel, nested, or multilevel?”:
– Identical : same people participate in quantitative and
qualitative phases of study.
– Parallel : separate quantitative and qualitative samples
drawn from the population.
– Nested : participants selected for one phase/component
are subset of participants selected for other.
– Multilevel : quantitative and qualitative samples selected
from different levels of a hierarchical population.
 MMR Sampling (cont.)
Combine these two criteria—time orientation (which has
two types) and sample relationship (which has four
types)—to form eight mixed sampling designs:
(1) identical concurrent
(2) identical sequential
(3) parallel concurrent
(4) parallel sequential
(5) nested concurrent
(6) nested sequential
(7) multilevel concurrent
(8) multilevel sequential.
 MMR Sampling (cont.)
Examples:
• In identical concurrent sampling design,
quantitative and qualitative data are collected from
same people (identical) at approximately the same
time (i.e., concurrently).
• In identical sequential sampling design,
quantitative and qualitative data are collected from
same people (identical) in stages (sequential).
 MMR Sampling (cont.)
Once the mixed sampling design has been determined
(or concurrently)…
• The researcher determines the specific qualitative
sampling method, the specific quantitative sampling
method, and determines the QUAN and QUAL
sample sizes.
• Then the samples (of people, sites) are located and
data are collected.
 Validity of Research Findings
• For an overview of validity of research results
in quantitative, qualitative, and mixed
methods research, click here.
• (The term validity also is used in
measurement: For an overview of
measurement validity, click here.)
 Multiple Validities
I am reviewing validity in quantitative and
qualitative research because of the requirement
of multiple validities in MMR (Johnson &
Onwuegbuzie, 2006).
Multiple validities: the extent to which all of the
pertinent validities (quantitative, qualitative, and
mixed) are addressed and resolved successfully.
--this important idea is currently ignored in some
mixed methods textbooks.
 Review: QUAN Validity
If you are not familiar with validation in QUAN, go
here to read the relevant lecture.
Cook and Campbell’s four QUAN types include
1. Statistical conclusion validity (the ability to infer
that the independent and dependent variables
are related and the strength of that relationship)
2. Construct validity (the extent to which a higher-
order construct is accurately represented in a
particular study)
 Review: QUAN Validity
3. Internal (causation) validity (the ability to
infer that a causal relationship exists between
two variables)
4. External (generalization) validity (the extent
to which the study results can be generalized
to and across populations of persons, settings,
times, outcomes, and treatment variations)
 Review: QUAN Validity
Because of the importance of causation in
social/behavioral/health research, I want to
review the criteria that must be met for
causation.
 Internal (Causation) Validity

NOTE: To fully understand causation, you must also

determine any relevant mediating variables and
moderating variables. See next slides for definitions.
Review: QUAN Validity
Review: QUAN Validity
Review: QUAN Validity: Mediating
Variables
 Review: QUAN Validity:
For causation as used in epidemiology, click
here.
 Review: QUAL Validity
If you’re not familiar with validation in qualitative
research, go here for overview.

Joseph Maxwell’s (1992) QUAL validity types include

1. Descriptive validity (the factual accuracy of an

account as reported by the researcher)

2. Interpretative validity (accurately portraying the

participants’ perspectives and meanings, and
providing the insider’s viewpoint)
 Review: QUAL Validity
3. Theoretical validity (the degree to which a theoretical
explanation fits the data)
4. And internal (causation) and external (generalizing)
validity. Same as with quan except that (according to
Johnson and Christensen, 2014) you can view internal
validity as including both “local/idiographic causation”
(particular causes, including intentions, of specific or
local attitudes, conditions, and events) and
“nomological/general causation” (the standard view of
causation in science; it refers to causation among
variables)
 Review: QUAL Validity (cont.)

On the next slide I list some useful strategies for

conducting defensible qualitative research…
Click here for Table on Strategies to
Promote Qualitative Validity

Click here for

application/companion-table.
 Research Validity or Legitimation in
Mixed Research
• Goal is to make justified qualitative,
quantitative, and integrated claims.
• Often want to make meta-inferences,
which are inferences or conclusions that
build on or integrates quan and qual
findings
• Only a subset of the following typically
will be relevant/pertinent for a particular
research study, and multiple validities
legitimation is the most important.
 MMR Research Validity (cont.)
1. Inside-outside validity – The extent to which the
researcher accurately understands, uses, and presents
the participants’ subjective insider or “native” views
and the researcher’s objective outsider view
– These also are called the emic and etic viewpoints
– Strategy: Try to move back-and-forth, step-in-step-
out
– Represent both viewpoints and create a third
MMR viewpoint; make meta inferences.
 MMR Research Validity (cont.)
2. Paradigmatic/philosophical validity – the
degree to which the mixed researcher clearly
explains his or her philosophical beliefs about
research
• The most popular paradigms or worldviews
are pragmatism, critical realism,
transformative, and dialecticalism
• This will enable the conduct of MMR.
– This set of beliefs should be logical and defensible
 MMR Research Validity (cont.)
3. Commensurability approximation validity – the
degree to which a mixed researcher can make
Gestalt switches between the lenses of a
qualitative researcher and a quantitative
researcher and integrate the two views into an
“integrated” or third viewpoint
– Need to become a QUAL researcher, a QUAN, and by
moving back-and-forth become a MMR researcher.
– Requires extensive training
– More easily/quickly done by team of a QUAN, a QUAL,
and a mediator/integrator MMR researcher
 MMR Research Validity (cont.)
4. Weakness minimization validity – extent to which
the weakness from one research method or
approach is compensated by the strengths from
another method or approach.
– MMR researcher designs/combines QUAN & QUAL
methods/approaches to have nonoverlapping
weaknesses.
– Use QUAL or QUAN help you to see what you would
have missed had you only used one
method/approach.
– E.g., use depth interviews to pick up on what the
standardized test failed to measure.
 MMR Research Validity (cont.)
5. Sequential validity – the degree to which a
mixed researcher appropriately addresses
and/or builds on effects, understandings,
knowledge, or findings from earlier qualitative
and quantitative phases
– Ask: would results have been different, in a
negative way, if sequencing had been reversed?
– Ask: was second stage appropriately informed by
first stage (and third, fourth, etc. by prior stages)
 MMR Research Validity (cont.)
6. Conversion validity – accuracy/quality of data
transformations (quantitizing qualitative data
and qualitizing quantitative data) and appropriate
interpretations made on transformed data.
– Only relevant in studies where data are converted
• Example of quantitizing is counting QUAL data
(e.g., words, categories, themes); converting to
quantitative codes for statistical analysis.
• Examples of qualitizing are labeling factors and
developing categories and types from QUAN data
 MMR Research Validity (cont.)
7. Sample integration validity – the degree to which
a mixed researcher makes appropriate
conclusions, generalizations, and meta-inferences
from mixed samples (combination of QUAN and
QUAL samples).
• E.g., sample to population “statistical”
generalizations are better with large random
samples; “meaning” and experiential statements
are better justified with purposive samples
studied in depth.
 MMR Research Validity (cont.)
8. Integrative/integration validity – degree to
which the researcher has achieved integration
of data, analysis, and conclusions. Integrative
conclusions are sometimes called meta-
inferences.
 MMR Research Validity (cont.)
9. Socio-Political validity – the degree to which a
mixed researcher addresses the interests,
values, and viewpoints of multiple standpoints
and stakeholders in the research process.
– Need to understand the value positions and
viewpoints in order to effectively address them
– Try to be extra sensitive to needs of stakeholders
with minimal power and voice
– Ask: Is the study defensible to multiple
stakeholders?
 MMR Research Validity (cont.)
10. Multiple validities – the extent to which all of
the pertinent validities (quantitative, qualitative,
and mixed) are addressed and resolved
successfully.
– Valid/legitimate MMR requires the conduct of good
qualitative and good quantitative research.
– Two poorly designed and executed components does
not = good design with defensible findings.
– Need to identify and use combination of relevant
QUAL, QUAN, and Mixed validity types and strategies.
 Research Designs in Mixed Methods
Research
I will examine the typologies and approaches of
• John Creswell (2014)
• Jennifer Greene (2007)
• Burke Johnson and Tony Onwuegbuzie (equal
order)(2004, 2014)
• Joseph Maxwell (2003)
• Jan Morse (1991) (Just her notation system)
• Charles Teddlie and Abbas Tashakkori (2009)
 MMR Designs (cont.)
Creswell (2014) has revised his typology to the following:

Three “basic” designs:

• Convergent parallel design (qual and quan data collection and
analysis is done concurrently)
• Explanatory sequential design (quanqual)
• Exploratory sequential design (qualquan)

Three “advanced” designs:

• Experimental intervention design (collect qual data before, during,
and after experiment)
• Social Justice/participatory design (quanqual)
• Program evaluation design (qualquanqualquan)
Figure 1
Basic Mixed Methods Designs

Convergent Parallel Design

Quan ta ve Data Quan ta ve

Collec on and Results
Analysis
Merge
Results Interpret or Explain
For Comparison Convergence/Divergence
Qualita ve Data Qualita ve
Collec on and Results
Analysis

Explanatory Sequen al Design

Quan ta ve Data Determine Qualita ve Data Interpret How

Quan ta ve Quan ta ve Qualita ve Qualita ve Data
Collec on and Collec on and
Results Results to Results Explains
Analysis Analysis Quan ta ve Results
Explain

Exploratory Sequen al Design

Quan ta ve Data
Use Results to Collec on and Interpret How
Qualita ve Data Qualita ve Form Variables, Analysis Based on Quan ta ve Quan ta ve Results
Collec on and Results Instruments, Variables, Results Provide New Results,
New, Be er Instruments,
Analysis Interven ons Instruments, and Be er Interven ons
Interven ons
 Figure 2
Advanced Mixed Methods Designs
Experimental Mixed Methods Design
Experimental Study

Qualita ve
Data Collec on, Experiment Group Pre-test Interven on Post-Test Qualita ve
Analysis, and Data Collec on,
Results Control Group Pre-test Post-Test Analysis, and Results
(Exploratory) (Explanatory)
Qualita ve Data Collec on,
Analysis, and Results (Convergent)

Social Jus ce Design (using an Explanatory Sequen al Design example)

Qualita ve Data Interpret How

Theory Research Quan ta ve Quan ta ve Qualita ve Qualita ve Data
Data Collec on Collec on and Results
Ques ons Results Analysis
Explains
e.g., survey Quan ta ve Results
And Calls for Ac on

Promote Social Jus ce

Mul stage Evalua on Design (using an Exploratory Sequen al Design example)
Single Program Objec ve
Summa ve
Forma ve Program
Forma ve Needs Theory/ Instrument Program
Program Revision
Assessment Conceptual Framework Development Evalua on
Assessment
(qualita ve (based on qualita ve (based on (pre-post
(qualita ve
data collec on) results) quan ta ve tests) quan ta ve
data collec on)
tests)
 T&T’s MMR Designs
Here are the major MMR designs in the Teddlie
& Tashakkori typology (cell 4 in next slide):
• Parallel mixed designs
• Sequential mixed designs
• Conversion mixed designs
• Multilevel mixed designs
• Fully integrated mixed designs.
Teddlie and Tashakkori’s Designs
Parallel Mixed Design (Teddlie and
Tashakkori)
Sequential Mixed Design (Teddlie and
Tashakkori)
Fully Integrated Mixed Design (Teddlie
and Tashakkori)
Conversion Mixed Design (Teddlie and
Tashakkori)
 MMR Designs (cont.)
Ultimately, design should be based on multiple
dimensions in a way to best answer your
research question.
Here are 7 design dimensions discussed by
Teddlie & Tashakkori (2009):
1. Number of methodological approaches
(monomethods vs. mixed methods);
2. Number of strands or phases (monostrand vs.
multistrand);
 MMR Designs (cont.)
3. Type of implementation process (parallel,
sequential, conversion, multilevel,
combination);
4. Stage of integration of approaches (across all
stages, within experiential stage only, other
variants possible);
5. Priority of methodological approach
(QUAL+quan, QUAN+qual, QUANqual,
QUALquan);
 MMR Designs (cont.)
6. Functions of the research study (triangulation,
complementarity, development, initiation,
expansion, other functions possible);
7. Theoretical or ideological perspective (some
variant of transformative perspective or other
perspectives vs. no theoretical or ideological
perspective).
 MMR Designs (cont.)
6. Functions of the research study
7. Theoretical or ideological perspective
 MMR Design Criteria (Greene)
Jennifer Greene also has a list of design
dimensions (2007):
1. Paradigms (relative weight of paradigm);
2. Phenomena (addressing same or different
phenomena?);
3. Methods (how similar or different are the
methods selected?);
4. Status (relative weight of methods);
 MMR Design Criteria (Greene)
5. Implementation independence
(conceptualized and implemented
interactively or independently);
6. Implementation timing (concurrent or
sequential);
7. Study (essentially one research study or
two?).
 Greene’s MMR Designs
Greene (2007):
• Has two broad categories of designs
– Component designs (QUAL and QUAN kept
distinct)
– Integrated designs (QUAL and QUAN mixed,
blended, iterated, nested, nuanced, complex)
• She does not believe design typologies are
especially useful.
 MMR Designs (cont.)
Greene, Teddlie &Tashakkori, Guest, Hesse-
Biber, Morse, Onwuegbuzie, and I, and many
others make an important point:
• Do not limit yourself to designs provided in
any one current design typology.
• Learn how to construct the design you need!
• Prepackaged/canned designs are very useful
starting points that you can build on.
 MMR Designs (cont.)
Click here for a new, more advanced approach
to mixed design that I am currently
developing. I call it the multiple-dimension
approach to mixed design, and it allows
researchers to deal with many technical
complexities in MMR design and construct a
justified design.
 MMR Designs (cont.)
Now, I present some design notation and a resulting
typology developed by Johnson and
Onwuegbuzie (2004) (and based on the Morse,
Patton, Teddlie, Tashakkori, and Morgan).
• It provides a common notation used in MMR
from Morse, 1991.
• It provides a way of visualizing the relationship of
the components in your design.
• Like the other typologies, however, it is only a
starting point for constructing your design to
answer your research questions.
 MMR Designs (cont.)
Mixed research designs - classified on two major
dimensions:
– Time order (concurrent, sequential)
– Method/methodology/paradigm emphasis
(interactive or equal status versus qualitatively
driven or quantitatively driven where one
methodology is the core that is supplemented by
qual or quan).
These dimensions produce a 2-by-2 matrix, with 9
basic mixed methods research designs that you
can build on.
 MMR Designs (cont.)
To use design Figure provided in the next slide,
you answer two questions:
1. Do your research questions suggest that
you operate largely within one
method/methodology/paradigm or not?
2. Should you conduct the QUAL and QUAN
phases/components concurrently or
sequentially?
MMR Designs (cont.)
 MMR Designs (cont.)
Notation:
• QUAL and qual stand for qualitative research.
• QUAN and quan stand for quantitative research.
• Capital letters denote priority or increased
weight.
• Lowercase letters denote lower priority.
• Plus sign (+) denotes concurrent collection of
data.
• Arrow (→) denotes sequential collection of data.
 MMR Designs (cont.)
Example: qual→QUAN
• Quantitatively driven, sequential design
• Quantitative core design, preceded by
supplemental qualitative phase.
• E.g.: Phase one: open-ended interviews on
dropping out of on-line .
• Phase two: quantitative study of predictors of
dropping out, with random sample and using
statistical methods.
• Qualitative part plays important but supportive
role.
 MMR Designs (cont.)
More complexity:
On the next slide I show “Mixed Model” designs.
• This terminology has largely been dropped
• Now called “conversion” designs
• Usually part of more complex MMR design
• The point is that mixing can take place across
stages (e.g., QUAL data collection followed by
QUAN data analysis).
 MMR Designs (cont.)
In case you like steps or process, the next slide
shows the nonlinear/recursive process (or
tries to)…
 MMR Designs (cont.)
I reemphasize “stage 6” and “stage 7”:

(6) Continually validate data.

(7) Continually interpret data and findings.

The process of conducting a defensible (“valid”)

research study is continual. It must be
considered and addressed at every point in what
the researcher does!
 MMR Designs (cont.)
One last approach to design (Joseph Maxwell and D. Loomis,
2003) is focused on the interactive process of design rather
than selecting or constructing an a priori design.
Design is viewed as the continual interaction of
• Research questions (the substantive questions to be
answered via the research)
• Purposes (intellectual, practical, personal)
• Conceptual model (the theory to be generated or tested)
• Methods (research methods and methods of data
collection)
• Validity (addressing threats to validity; ruling out
alternative explanations)
MMR Designs (cont.)
 Data Analysis in Mixed Methods
Research

Mixed data analysis – use of both quantitative

and qualitative analytical procedures in a
research study.

• First, I will provide a classification of mixed

methods data analysis (based on work of
Onwuegbuzie)
 MMR Data Analysis (cont.)
Onwuegbuzie and I and colleagues have
developed a Mixed Analysis Matrix.

• It’s shown in the next slide.

 MMR Data Analysis (cont.)

To find your cell in the matrix, answer two

questions:
 MMR Data Analysis (cont.)
1. What type(s) of data do you have?

• Monodata – have one data type.

• Multidata –have both qualitative and

quantitative data.
 MMR Data Analysis (cont.)
2. How many data analysis approaches will you
use?

• Use of only one type of analysis (qualitative

analysis or quantitative analysis) called
monoanalysis.

• Use of both types of analysis called

multianalysis.
 MMR Data Analysis (cont.)
Based on answers to the two questions, your
analysis fits into one of four possible types:

1. Monodata-monoanalysis.
– This is not type of mixed data analysis.
 MMR Data Analysis (cont.)
2. Monodata-multianalysis – analysis of one type of
data using both qualitative and quantitative analysis.
• First: analyze data with standard approach.
• Second, either qualitize or quantitize data for
additional analysis.
– Qualitize – transform quantitative data into qualitative
data.
– Quantitize data – transform qualitative data into
quantitative data.
 MMR Data Analysis (cont.)
3. Multidata-monoanalysis – analysis of both
data types (QUAL and QUAN) using only one
analysis type.
• Results in:
– Only QUAN analysis of QUAN and QUAL data, or
– Only QUAL analysis of QUAN and QUAL data.
• Recommend avoiding this approach.
 MMR Data Analysis (cont.)
4. Multitype mixed analysis – analysis of both
types of data (QUAL data and QUAN data)
using both types of analysis (QUAL and QUAN
analysis).
• Includes many specific approaches to mixed
data analysis.
• This is recommended type of mixed analysis.
 MMR Data Analysis (cont.)
Now, I will provide several data analytic
strategies
• They also can be viewed as “steps” in MMR
data analysis; Onwuegbuzie & Teddlie, 2003)…
 MMR Data Analysis (cont.)
1. Data reduction—reduce number of
dimensions in QUAN and QUAL data
• QUAN via descriptive statistics, exploratory
factor analysis
• QUAL via thematic analysis, memoing
 “You know what you display”
(Miles and Huberman, 1994)
2. Data display—visually describe and depict
your QUAN and QUAL data
• QUAN via tables, graphs
• QUAL via graphs, charts, matrices, checklists,
rubrics, networks, Venn diagrams
• And QUAN and QUAL combined in the same
tables, graphs, matrices
 MMR Data Analysis (cont.)
3. Data transformation—quantitizing and
qualitizing the data.
• Quantitizing (convert QUAN to QUAL)
• Qualitizing (convert QUAL to QUAN)
 MMR Data Analysis (cont.)
4. Data correlation—Correlate or cross-classify
different data types.
• Convert one or both QUAL into “variables”
and correlate.
• For example, point-biserial correlation, cross-
tabulation tables, and use of multiple-
response variables.
 MMR Data Analysis (cont.)
5. Data consolidation—combine QUAN and
QUAL data (in data analysis software: SPSS,
SAS, MAXQDA, QDA-Miner, etc.)
• Create consolidated
– codes,
– variables,
– data sets.
 MMR Data Analysis (cont.)
6. Data comparison—compare research findings
from QUAN and QUAL data and analysis.

7. Data integration—integrate QUAN and QUAL

findings into coherent whole;
--make meta-inferences (inference or
conclusion that builds on or integrates QUAL
and QUAN findings)
 MMR Data Analysis (cont.)
On the next two slides, I show two examples of
analyses from QUAL that work well with
MMR…
 More on Integration
Construct matrices to analyze, compare, and present
results (this strategy originally suggested by Miles &
Huberman, 1994). Be creative!
For example:
• Rows=research question; columns=quan result, qual
result, integrated statement (called a joint display)
• Rows=research question; columns=quan result, qual
result, difference, explanation for difference
• Rows=independent/predictor variables;
columns=dependent variable outcomes (this is an
input-output matrix)
 More on Integration
More matrices (remember be creative):
• Rows=cases; columns=outcomes on quan and
qual measures
• Rows=issues, locations, events, situations;
columns=stakeholder groups on quan and
qual measures (e.g., students, teachers,
parents, administrators)
• Rows=(IVs, location, or whatever you like);
columns=time-ordered outcomes
 Writing Mixed Research Reports
General suggestions:
• Overall, consider using major sections of APA
report as broad divisions (Intro, Methods,
Results/Findings, Discussion).
• However, be creative, and provide multiple
perspectives in these sections.
• One creative strategy I’ve seen is to alternate
between emic and etic perspectives when
discussing results.
 MMR Reports (cont.)
• One effective organization for Results section is
list qualitative, quantitative, and integrated
results for each research question.
– That is, answer one question at a time using qual and
quan data/results/findings/interpretations.
• Integration is the key, throughout (e.g., in data
analysis, results, discussion, and theoretical
understanding/explanation).
• At a minimum, integration is needed in the
discussion section of the report.
 MMR Reports (cont.)
For more guidance on mixed methods research
reports, see
• Pat Bazeley (in press) (click here) and
• Nancy Leech (2012)(click here)
 References
For references, click here.

Or, here they are…

 References
Bazeley, P. (in press). Writing up mixed and multi methods research for
diverse audiences. In S. Hesse-Biber & R. B. Johnson (Eds.) Oxford
handbook of multiple and mixed methods research. New York, NY:
Oxford.

Christensen, L. B., Johnson, R. B., & Turner, L. A. (2014). Research

methods, design, and analysis (12th ed.). Boston, MA: Pearson.

Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design

and analysis issues for field settings. Chicago, IL: Rand McNally.

Creswell, J. W. (in press). Revisiting mixed methods and advancing

scientific practices. In S. Hesse-Biber & R. B. Johnson (Eds.) Oxford
handbook of multiple and mixed methods research. New York, NY:
Oxford.
 References
Greene, J. C. (2007). Mixed methods in social inquiry. San
Francisco: Jossey-Bass.

Greene, J. C. (in press). Preserving distinctions within the

MMR merger. In S. Hesse-Biber & R. B. Johnson (Eds.)
Oxford handbook of multiple and mixed methods research.
New York, NY: Oxford.

Greene, J. C., Caracelli, V. J., & Graham, W. F. (1989). Toward a

conceptual framework for mixed-method evaluation
designs. Educational Evaluation and Policy Analysis,
11, 255–274.
 References
Guest, G. (2012). Describing mixed methods
research: An alternative to typologies. Journal of
Mixed Methods Research, 7, 141-151.

Hesse-Biber, S. N. (2010). Mixed methods research:

Merging theory with practice. New York, NY:
Guilford Press.

Johnson, R. B. (2011). Do we need paradigms? A

mixed methods perspective. Mid-Western
Educational Researcher, 24(2), 31-40.
 References
Johnson, R. B. (2012). Guest Editor’s Editorial: Dialectical
pluralism and mixed research. American Behavioral
Scientist, 56, 751-754

Johnson, R. B., & Christensen, L. B. (2014). (5th edition).

Educational research methods: Quantitative, qualitative,
and mixed approaches. Los Angeles, CA: Sage.

Johnson, R. B., & Gray, R. (2010). A history of philosophical

and theoretical issues for mixed methods research. In A.
Tashakkori & C. Teddlie, Handbook of mixed methods in
social and behavioral research (2nd ed.) (pp. 69-94).
Thousand Oaks, CA: Sage.
 References
Johnson, R. B., McGowan, M. W., & Turner, L. A. (2010).
Grounded theory in practice: Is it inherently a mixed
method? Research in the Schools, 17(2), 65-78.

Johnson, R. B., & Onwuegbuzie, A. J. (2004). Mixed

methods research: A research paradigm whose time
has come. Educational Researcher, 33(7), 14-26.

Johnson, R. B., Onwuegbuzie, A. J., & Turner, L. A. (2007).

Toward a definition mixed methods research. Journal
of Mixed Methods Research, 1, 112-133.
 References
Johnson, R. B., Onwuegbuzie, A. J., Tucker, S., & Icenogle, M. L. (in
press). Conducting mixed methods research using dialectical
pluralism and social psychological strategies. In P. Leavy, The Oxford
Handbook of Qualitative Research, New York, NY: Oxford.

Johnson, R. B., & Stefurak, T. (2013). Considering the evidence-and-

credibility discussion in evaluation through the lens of dialectical
pluralism. In D. Mertens and S. Hesse-Biber (Eds.), Mixed methods
and credibility of evidence in evaluation. New Directions for
Evaluation, 138, 103-109.

Johnson, R. B., & Turner, L. A. (2003). Data collection strategies in

mixed methods research. In A. Tashakkori & C. Teddlie (Eds.),
Handbook of Mixed Methods in Social and Behavioral Research (pp.
297-319). Newbury Park: Sage.
 References
Leech, N. L. (2012). Writing mixed research reports. American
Behavioral Scientist, 56, 866-881.

Maxwell, J. A. (1992). Understanding and validity in qualitative

research. Harvard Educational Review, 62(3), 279–299.

Maxwell, J. A., & Loomis, D. M. (2003). Mixed methods design: An

alternative approach. In A. Tashakkori & C. Teddlie (Eds.), SAGE
handbook of mixed methods in social & behavioral research.
Thousand Oaks, CA: Sage.

Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: An

expanded source book. Thousand Oaks, CA: Sage.
 References
Morse, J. M. (1991). Approaches to qualitative-quantitative
methodological triangulation. Nursing Research, 40, 120–
123.

Onwuegbuzie, A. J., & Johnson, R. B. (2006). The “validity”

issue in mixed methods research. Research in the Schools,
13(1), 48-63. (Link: http://www.msera.org/rits_131.htm)

Onwuegbuzie, A. J., & Johnson, R. B. (2004). Mixed methods

research. In R. B. Johnson & L. B. Christensen, Educational
research: Quantitative, qualitative, and mixed approaches
(pp. 408-431). Boston, MA: Allyn and Bacon.
 References
Onwuegbuzie, A. J., & Teddlie, C. (2003). A framework for analyzing data in
mixed methods research. In A. Tashakkori & C. Teddlie (Eds.), Handbook of
mixed methods in social and behavioral research (pp. 351–383). Thousand
Oaks, CA: Sage.

Teddlie, C., & Johnson, R. B. (2009). Chapter 4: Methodological thought since

the 20th century. In C. Teddlie & A. Tashakkori, Foundations of mixed
methods research: Integrating quantitative and qualitative techniques in
the social and behavioral sciences(pp. 62-82). Thousand Oaks, CA: Sage.

Teddlie, C., & Tashakkori, A. (2009). Foundations of mixed methods research:

Integrating qualitative and quantitative approaches in the social and
behavioral sciences. Thousand Oaks, CA: Sage.