CHAPTER 2

BACKGROUND AND DEFINITION

OF CONCEPTS

2.1 INTRODUCTION

In this chapter, we lay out the general foundation for case study research in software
engineering. We characterize the case study strategy and compare it with other em-
pirical research strategies, primarily survey, experiment, and action research. Aspects
of empirical research strategies are elaborated, for example, their primary purpose,
whether they have a fixed or flexible design, whether data are quantitative or qualita-
tive, and the roles which triangulation and replication play. We discuss, on the basis
of different sources within and outside software engineering, what constitutes an ex-
emplary case study and summarize criteria or good case study research. We set out a
scheme to help decide when case study is a feasible research strategy, and we define
a general research process for case studies, which is used throughout the book.

2.2 RESEARCH STRATEGIES

Let us start with three different general definitions of the term case study, one by
Robson [162], one by Yin [217], both in the social science field, and one definition
by Benbasat et al. [19] in the information systems field.

Case Study Research in Software Engineering: Guidelines and Examples, First Edition.
Per Runeson, Martin Höst, Austen Rainer, and Björn Regnell.
© 2012 John Wiley & Sons, Inc. Published 2012 by John Wiley & Sons, Inc.

11
12 BACKGROUND AND DEFINITION OF CONCEPTS

Robson. Case study is a strategy for doing research that involves an empirical
investigation of a particular contemporary phenomenon within its context using
multiple sources of evidence.
Yin. Case study is an empirical inquiry that investigates a contemporary phe-
nomenon within its real-life context, especially when the boundaries between
the phenomenon and its context are not clearly evident.
Benbasat. A case study examines a phenomenon in its natural setting, employing
multiple methods of data collection to gather information from one or a few
entities (people, groups, or organization). The boundaries of the phenomenon
are not clearly evident at the outset of the research and no experimental control
or manipulation is used.

The three definitions agree on that case study is an empirical method aimed at
investigating contemporary phenomena in their context. Robson calls it a research
strategy and stresses the use of multiple sources of evidence, Yin denotes it an in-
quiry and remarks that the boundary between the phenomenon and its context may be
unclear, while Benbasat et al. make the definitions somewhat more specific, mention-
ing information gathering from few entities (people, groups, and organizations) and
the lack of experimental control. The three definitions together emphasize important
characteristics of case studies.
We derive from these general definitions, specifically for software engineering, the
conclusion that

Case study in software engineering is an empirical enquiry that draws on multiple

sources of evidence to investigate one instance (or a small number of instances)
of a contemporary software engineering phenomenon within its real-life con-
text, especially when the boundary between phenomenon and context cannot
be clearly specified.

There are three other major research strategies that are related to case studies,
survey, experiment, and action research:

Survey is the “collection of standardized information from a specific population, or

some sample from one, usually, but not necessarily by means of a questionnaire
or interview” [162]. Surveys provide an overview rather than depth in the studied
field.
Experiment or controlled experiment is characterized by “measuring the effects
of manipulating one variable on another variable” [162] and that “subjects
are assigned to treatments by random” [215]. The effect of one specific
variable is studied in experiments. Quasi-experiments are similar to controlled
experiments, except that subjects are not randomly assigned to treatments
[32]. Quasi-experiments conducted in an industry setting may have many
characteristics in common with case studies.
CHARACTERISTICS OF RESEARCH STRATEGIES 13

Action research with its purpose to “influence or change some aspect of whatever
is the focus of the research” [162] is closely related to case study. In some
definitions, a case study is purely observational while action research is fo-
cused on and involved in the change process. In software process improvement
[44, 75] and technology transfer studies [64], the research method has clear
characteristics of action research, although it is sometimes referred to as an
“iterative case study” [7]. In IS, where action research is widely used, there is
a discussion on finding the balance between action and research, see for ex-
ample, Avison et al. [10] and Baskerville and Wood-Harper [16]. We prefer
including action research in the wider notion of case study, and for the research
part, these guidelines in this book apply as well. For the action part, guidelines
on software process improvement may be useful [201], as well as literature on
design science [71].
Easterbrook et al. [47] also count ethnographic studies among the major research
strategies. We prefer to consider ethnographic studies as a specialized type of case
studies with focus on cultural practices [47] or long duration studies with large
amounts of participant-observer data [111]. Zelkowitz and Wallace define four differ-
ent “observational methods” in software engineering [218]: project monitoring, case
study, assertion, and field study. The guidelines in this book apply to all these, except
assertion that we do not consider a proper research method. We also prefer to see
project monitoring as part of a case study and field studies as multiple case studies.
Yin includes archival analysis and history studies, as distinct types of research
methods [217, p. 5]. We prefer including the archives and historical data as sources for
information in case studies, rather than distinct research methods. Yin also recognizes
that multiple strategies may be appropriate for a given study and we hold the same
view. For example, a survey may be conducted within a case study to get a broad
overview of the studied object, literature search often precedes a case study to identify
the foundations for the studied object, and archival analyses may be a part of its data
collection. Ethnographic methods, such as interviews and observations, are mostly
used for data collection in case studies.
In general, the borderline between the types of study is not always distinct. Robson
summarizes his view, which we consider applies also to software engineering research,
“Many flexible design studies, although not explicitly labeled as such, can be usefully
viewed as case studies” [162, p. 185].

2.3 CHARACTERISTICS OF RESEARCH STRATEGIES

2.3.1 Purpose
Different research strategies serve different purposes; one type of research strategy
does not fit all purposes. We distinguish between the following four general types of
purposes for research, tailored from Robson’s classification [162]:
Exploratory – finding out what is happening, seeking new insights, and generating
ideas and hypotheses for new research.
14 BACKGROUND AND DEFINITION OF CONCEPTS

Descriptive – portraying the current status of a situation or phenomenon.

Explanatory – seeking an explanation for a situation or a problem, mostly but not
necessarily, in the form of a causal relationship.
Improving – trying to improve a certain aspect of the studied phenomenon.

Case study strategy was originally used primarily for exploratory purposes, and
some researchers still limit case studies to this purpose, as discussed by Flyvbjerg [56].
However, it is also used for descriptive purposes if the generality of the situation or
phenomenon is of secondary importance. Case studies may also be used for explana-
tory purposes, for example, in interrupted time series design (pre- and postevent
studies), although the isolation of factors may be a problem. This involves testing
of existing theories in confirmatory studies. Finally, as indicated above, case studies
in the software engineering discipline often take an improvement approach, similar
to action research; see, for example, the iterative case study of quality assurance in
Chapter 12.
Klein and Myers define three types of case studies depending on the research per-
spective: positivist, critical, and interpretive [111]. A positivist case study searches
evidence for formal propositions, measures variables, tests hypotheses, and draws
inferences from a sample to a stated population, which is close to the natural science
research model [115] and related to Robson’s explanatory category. A critical case
study aims at social critique and at being emancipatory, that is, identifying different
forms of social, cultural, and political domination that may hinder human ability.
Improving case studies may have a character of being critical to the current practice
and contribute to change. An interpretive case study attempts to understand phe-
nomena through the participants’ interpretation of their context, which is similar to
Robson’s exploratory and descriptive types. Software engineering case studies tend
to lean toward a positivist perspective, especially for explanatory-type studies. This
is related to the pragmatic nature of empirical software engineering research, where
the practical implications of a certain practice is more relevant than the questions on
abstract philosophical principles.

2.3.2 Control and Data

Conducting research on real-world phenomena implies a constant trade-off between
level of control and degree of realism. The realistic situation is often complex and
nondeterministic, which hinders the understanding of what is happening, especially
for studies with explanatory purposes. On the other hand, increasing the control
reduces the degree of realism, sometimes leading to the underlying causal factors
and structures being set outside the scope of the study. Case studies are by definition
conducted in real-world settings, and thus have a high degree of realism, mostly at
the expense of the level of control. Experiments on the other hand mostly isolate a
certain part of reality, for example, the inspection process [9] for better control of the
situation, but at the expense of realism.
CHARACTERISTICS OF RESEARCH STRATEGIES 15

TABLE 2.1 Overview of Research Strategy Characteristics

Experiment Survey Case Study Action Research
Primary objective Explanatory Descriptive Exploratory Improving
Primary data Quantitative Quantitative Qualitative Qualitative
Design type Fixed Fixed Flexible Flexible

The data collected in an empirical study may be quantitative or qualitative.

Quantitative data involve numbers and classes, while qualitative data involve words,
descriptions, pictures, diagrams, and so on. Quantitative data are analyzed using
statistics, while qualitative data are analyzed using categorization and sorting. Quan-
titative data are more exact, while qualitative data are ‘richer’ in what they may
express. Case studies tend mostly to be based on qualitative data, but combinations
of qualitative and quantitative data often provide better understanding of the studied
phenomenon [176], what is sometimes called ‘mixed methods’ [162].
The research process may be characterized as a fixed or flexible design according
to Anastas and MacDonald [4] and Robson [162]. In a fixed design process, all
parameters are defined at the launch of the study, while in a flexible design process
key parameters of the study may be changed during the course of the study. Case
studies are typically flexible design studies as they may be adjusted to findings during
the course of the study. Experiments and surveys are fixed design studies, which cannot
be changed once they are launched. Other literature use the terms quantitative and
qualitative design studies for fixed and flexible design studies, respectively, [34]. We
prefer to adhere to the fixed/flexible terminology since it reduces the risk for confusion
that a study with, in its terminology, “qualitative design” may collect both qualitative
and quantitative data. Otherwise, it may be unclear whether the term qualitative refers
to the data or the design of the study.
Table 2.1 shows an overview of the primary characteristics of the above-discussed
research strategies. Note that there may be other secondary characteristics of each of
the strategies.

2.3.3 Triangulation
Triangulation is important to increase the precision and strengthen the validity of
empirical research. Triangulation means taking multiple perspectives towards the
studied object and thus providing a broader picture. The need for triangulation is
obvious when relying primarily on qualitative data, which is broader and richer, but
less precise than quantitative data. However, it is relevant also for quantitative data,
for example, to compensate for measurement or modeling errors. Four different types
of triangulation may be applied [196]:

Data (Source) Triangulation – using more than one data source or collecting the
same data at different occasions.
Observer Triangulation – using more than one observer in the study.
16 BACKGROUND AND DEFINITION OF CONCEPTS

Methodological Triangulation – combining different types of data collection

methods, for example, qualitative and quantitative methods.
Theory Triangulation – using alternative theories or viewpoints.

In an example case study, multiple interviewees may be interviewed (data trian-

gulation) by more than one interviewer (observer triangulation), complemented with
data from time and defect reporting systems (methodological and data source triangu-
lation), using two different theories as a basis for the analysis (theory triangulation).
A case study will never provide conclusions of statistical significance. On the con-
trary, many kinds of evidence, figures, statements, and documents are linked together
to support a strong and relevant conclusion. Bratthall and Jørgensen investigate this
issue related to a specific software engineering case study [27]. They conclude on
the basis of empirical evidence from the case study that “a multiple data source case
study is more trustworthy than a comparable single data source case study.”

2.3.4 Replication
An important characteristic of a good and trustworthy research, in general, is trans-
parency that enables replication by others. The replications as such should add to the
validity of the research findings. This is also true for case study research. However,
the type of replication in flexible design studies, based on qualitative data, is very
different from fixed design studies, based on quantitative data.
The quantitative replication is based on sampling logic, for example, an experiment
is replicated with new subjects or artifacts, assuming the subjects and artifacts are
sampled from a population. There are many different types of replication in different
fields of research, and terminology is not yet defined for software engineering [63].
Whether the replications should be exact or varied is debated. Basili et al. argue that
replications should be as exact as possible [14], while Miller argues for learning by
variations in replications [129], which is supported by Kitchenham [94].
Replication in qualitative studies follows a different logic. In general, cases are not
sampled from a population but selected for a certain purpose. Selecting a replication
case is either aimed at finding similar results, confirming earlier findings (literal
replication), or aimed at finding contrasting results for predictable reasons (theoretical
replication).
Conclusions can be drawn across replicated quantitative studies using meta-
analysis [127]. Regarding synthesis of evidence from multiple case studies with
qualitative studies, the area is less mature and needs further development [36].

2.3.5 Inductive and Deductive Enquiries

Empirical research may be inductive, meaning that theory is induced from the observa-
tions. In inductive research, the researcher first observes with an open mind, identifies
patterns in the observations, sets up tentative hypotheses, and finally relates them to
existing theory or develops new theory (see Figure 2.1a). This is the original principle
WHAT MAKES A GOOD CASE STUDY? 17

FIGURE 2.1 Inductive (a) and deductive (b) approaches to empirical research.

of grounded theory research [34], although the two founders of the method, Strauss and
Glaser, developed it in different directions [205]. The Straussian approach is the most
pragmatic of the two, which makes it more feasible to software engineering research.
The deductive research starts with existing theory, sets out hypothesis for the
research, and finally makes observations (see Figure 2.1b). During the analysis, ob-
servations either confirm or reject the hypothesis, which leads to either confirmed or
rejected theories.
Case study may have characteristics of both paradigms. An exploratory case
study has inductive characteristics, while an explanatory case study has deductive
characteristics.

2.4 WHAT MAKES A GOOD CASE STUDY?

The label “case study” is used for many kinds of studies, ranging from small demon-
strations of toy size to full-scale industrial situations. By definition, a case study
“investigates a contemporary phenomenon within its real-life context” [217]. How-
ever, the size and the context are not sufficient to characterize a good case study.
Yin has speculated on the characteristics of exemplary case studies, and it is worth
considering these here as suggestions to further improve the standards of software
engineering case studies. The characteristics are as follows [217, pp. 160–165]:
18 BACKGROUND AND DEFINITION OF CONCEPTS

• The study is of a significant topic. The significance of a topic could be deter-

mined by, for example, reference to the extant literature on the topic or through
consultation with stakeholders and participants in the prospective case study.
• The study must be complete in that
– the boundary of the case (i.e., the distinction between the phenomenon of
interest and its context) is made explicit;
– there is a comprehensive collection of appropriate evidence;
– there are no significant constraints on the conduct of the study, for example,
the study does not run short of time, budget, or resources.
• The study must consider alternative perspectives on the topic.
• The study must present sufficient evidence when reporting the results and dis-
seminating the artifacts of the case study.
• The reports of the study must be engaging to the reader; in other words, the
reports are well written.
• The case study must respect the ethical, professional, and legal standards relevant
to that study.

Where a replication is being performed, and hence an existing case study protocol
is available for reference, it may be easier to design for an exemplary study. Implied
within Yin’s characteristics are case study design and case study workplan issues, to
which we have added the recognition of ethical, professional, and legal standards.
Kyburz-Graber [113] summarizes Yin’s quality criteria, stating that an exemplary
case study should have

• A theoretical basis including research questions is described.

• Triangulation is ensured by using multiple sources of evidence (data collection
and interpretation).
• A chain of evidence is designed with traceable reasons and arguments.
• The case study research is fully documented.
• The case study report is compiled through an iterative review and rewriting
process.

These criteria cover the foundation for the study, including the theory it is based
on and the research questions it is set out to answer. The criteria relate to the data
collection and the use of multiple sources of evidence. They address the analysis
procedure and its ability to clearly define which evidence backs up the conclusions.
They concern the reporting and the overall study process, including requirements on
transparency to enable external evaluation of the procedures. All these aspects are
important for the quality of the case study, and no single criterion is sufficient to
judge the quality of a study.
Perry et al. define similar criteria for a case study [134] in the domain of software
engineering, but still they are very general. It is expected that
WHEN IS THE CASE STUDY STRATEGY FEASIBLE? 19

• A case study has research questions set out from the beginning of the study.
• Collects data in a planned and consistent manner.
• Draws inferences from the data to answer the research question.
• Explores a phenomenon or produces an explanation, description, or causal anal-
ysis of the phenomenon.
• Addresses threats to validity in a systematic way.

These characteristics stress the need for planning and order in a case study, which shall
not be considered to be in conflict with the flexible nature of case studies. Instead, it
is stated to ensure a transparent path from observations to conclusions in the study.
In summary, the key characteristics of a good case study are that

1. it is of flexible type, coping with the complex and dynamic characteristics of

real-world phenomena, like software engineering;
2. its conclusions are based on a clear chain of evidence, whether qualitative
or quantitative, collected from multiple sources in a planned and consistent
manner; and
3. it adds to existing knowledge by being based on previously established theory,
if such theory exists, or by building a theory.

2.5 WHEN IS THE CASE STUDY STRATEGY FEASIBLE?

First and foremost, the case study strategy is feasible when studying “a contemporary
phenomenon within its real-life context, especially when the boundaries between the
phenomenon and context are not clearly evident" [217]. This is valid for many, if
not most, research studies in software engineering. The contemporaneousness is a
necessity to allow data collection from the case. The fuzzy borderline between the
phenomenon and the context may be handled within a case study, and its flexible
design principles.
Second, the type of research question is an important criterion for strategy selec-
tion. For exploratory research questions, the case study strategy is a perfect match.
However, also for descriptive research questions, the case study may be feasible if
representativeness of a sampling based study may be sacrificed for better realism in a
case study. If representativeness is critical, the survey is a better option. Explanatory
research questions may be addressed in case studies, but the evidence is not a statisti-
cally significant quantitative analysis of a representative sample, rather a qualitative
understanding of how phenomena function in their context. If quantitative evidence
is critical, the experiment strategy is the better option. For improving the type of
research purposes, the action research strategy is a natural choice, which we consider
as a variant of case study research.
Third, case studies may be launched when the degree of control is not a critical
issue. The realism achieved in case studies is in stark conflict with the ability to control
the situation. Studies that strictly aim at comparing different options under the same
20 BACKGROUND AND DEFINITION OF CONCEPTS

TABLE 2.2 When to Use Different Research Methods?

Criteria Experiment Survey Case Study
Yin criteria [217]
Contemporaneous event Yes Yes Yes
Type of research question How, Who, How,
Why What Why
Where,
How many,
How much
Requires control Yes No No
Fenton and Pfleeger criteria [54]
Level of control High Low
Difficulty of control Low High
Level of replication High Low
Cost of replication Low High

circumstances are not in line with case study strategy, although there are means to
use case studies of, for example, tools selection [109]. Experiments are primarily
designed for that type of research questions, but normally less realistic, at least for in
vitro studies.
Yin [217] uses these three criteria, originating from the COSMOS Corporation1 , to
determine when a case study is the more suitable research strategy: the degree to which
the phenomenon under study is a contemporary phenomenon, the type of research
question, and the degree of control required. The criteria of Yin are summarized in
the upper part of Table 2.2.
Fenton and Pfleeger [54] identify four factors that affect the choice between experi-
ment and case study: level of control, difficulty of control, level of replication, and cost
of replication. The last two factors are not covered by Yin’s criteria. When statistical
replication is important, experiment is a better choice than case study. However, there
are also other types of replication in case study designs, which may provide general-
ization, also from case studies. The criteria of Fenton and Pfleeger are summarized
in the lower part of Table 2.2.

2.6 CASE STUDY RESEARCH PROCESS

When conducting a case study, there are five major process steps to be considered,
as defined in Figure 2.2. Each of the steps is elaborated in detail in Chapters 3–6,
respectively. Chapters 7 and 8 also relate these steps on matters of scaling up and
using case studies.

1 [Link]
CONCLUSION 21

1. Case study design – objectives are defined and the case study is
planned.
2. Preparation for data collection – procedures and protocols for
data collection are defined.
3. Collecting evidence – data collection procedures are executed on
the studied case.
4. Analysis of collected data – data analysis procedures are applied
to the data.
5. Reporting – the study and its conclusions are packaged in feasible
formats for reporting.

FIGURE 2.2 Case study process.

This process is almost the same for any kind of empirical study, compare, for ex-
ample, to the processes proposed by Wohlin et al. [215] and Kitchenham et al. [105].
However, as the case study strategy is a flexible design strategy, there is a significant
amount of iteration over the steps [6], which is explicitly modeled on a process by
Verner et al. [208]. The data collection and analysis may be conducted incrementally.
If insufficient data are collected for the analysis, more data collection may be planned.
However, there is a limit to the flexibility; the case study should have specific objec-
tives set out from the beginning. If the objectives change, it is a new case study rather
than the existing one, though this is a matter of judgment like all other classifications.
Eisenhardt adds two steps between 4 and 5 above in her process for building theories
from case study research [48], (a) shaping hypotheses and (b) enfolding literature,
while the rest, except for terminological variations, are the same as above.
Hence, the process steps are very general, and form only a framework for presenting
the guidelines.

2.7 CONCLUSION

Since the case study strategy originates from different fields of research, it is no
surprise that terminology and definitions vary somewhat. In this chapter we define
basic concepts for case studies in software engineering, which are used throughout
the book. We hope that defining a set of basic concepts can help establish a standard
of terminology in the empirical software engineering research community. We also
identify characteristics of the case study strategy, to help researchers choose a feasible
research strategy for a specific research situation.