SENG 421:
Software Metrics
Empirical Investigation
(Chapter 4)
Department of Electrical & Computer Engineering, University of Calgary
B.H. Far （far@[Link]）
[Link]
Contents
Software engineering investigation
Investigation principles
Investigation techniques
Formal experiments: Planning
Formal experiments: Principles
Formal experiments: Types
Formal experiments: Selection
Guidelines for empirical research
far@[Link] 2
Empirical SE
Fill the gap between research and practice by:
Developing methods for studying SE practice
Building a body of knowledge of SE practice
Validating research before deployment in
industrial settings
far@[Link] 3
SE Investigation
What is software engineering investigation?
Applying “scientific” principles and techniques to
investigate properties of software and software
related tools and techniques.
Why talking about software engineering
investigation?
Because the standard of empirical software
engineering research is quite poor.
far@[Link] 4
SE Investigation: Examples
Experiment to confirm rules-of-thumb
Should the LOC in a module be less than 200?
Should the number of branches in any functional decomposition be
less than 7?
Experiment to explore relationships
How does the project team experience with the application affect the
quality of the code?
How does the requirements quality affect the productivity of the
designer?
How does the design structure affect maintainability of the code?
Experiment to initiate novel practices
Would it be better to start OO design with UML?
Would the use of SRE improve software quality?
far@[Link] 5
SE Investigation: Why?
To improve (process and/or product)
To evaluate (process and/or product)
To prove a theory or hypothesis
To disprove a theory or hypothesis
To understand (a scenario, a situation)
To compare (entities, properties, etc.)
far@[Link] 6
SE Investigation: What?
Person’s performance
Tool’s performance
Person’s perceptions
Tool’s usability
Document’s understandability
Program’s complexity
etc.
far@[Link] 7
SE Investigation: Where & When?
In the field
In the lab
In the classroom
Anytime depending on what questions you
are asking
far@[Link] 8
SE Investigation: How?
Hypothesis/question generation
Data collection
Data evaluation
Data interpretation
Feed back into iterative process
far@[Link] 9
SE Investigation: Characteristics
Data sources come from industrial settings
This may include people, program code, etc.
Usually
Surveys
Case-studies ( hypothesis generation)
Experiments ( hypothesis testing)
far@[Link] 10
Where Data Come From?
First Degree Contact
Direct access to participants
Example:
Brainstorming
Interviews
Questionnaires
System illustration
Work diaries
Think-aloud protocols
Participant observation
far@[Link] 12
Where Data Come From?
Second Degree Contact
Access to work environment during work time,
but not necessarily participants
Example:
Instrumenting systems
Real time monitoring
far@[Link] 13
Where Data Come From?
Third Degree Contact
Access to work artifacts, such as source code,
documentation
Example:
Problem report analysis
Documentation analysis
Analysis of tool logs
Off-line monitoring
far@[Link] 14
Practical Considerations
Hidden Aspects of Performing Studies
Negotiations with industrial partners
Obtaining ethics approval and informed consent
from participants
Adapting “ideal” research designs to fit with
reality
Dealing with the unexpected
Staffing of project
far@[Link] 15
Investigation Principles
There are 4 main principles of investigation:
1. Stating the hypothesis: What should be investigated?
2. Selecting investigation technique: conducting
surveys, case studies, formal experiments
3. Maintaining control over variables: dependent and
independent variables
4. Making meaningful investigation: verification of
theories, evaluating accuracy of models, validating
measurement results.
far@[Link] 16
SE Investigation Techniques
Three ways to investigate:
Formal experiment: A controlled investigation of an
activity, by identifying, manipulating and documenting
key factors of that activity.
Case study: Document an activity by identifying key
factors (inputs, constraints and resources) that may affect
the outcomes of that activity.
Survey: A retrospective study of a situation to try to
document relationships and outcomes.
far@[Link] 17
Case-study or Experiment?
How to decide whether conduct an experiment or
perform a case-study?
Factor Experiment Case-study
Retrospective Yes (usually) No (usually)
Level of control High Low
Difficulty of control Low High
Level of replication High Low
Cost of replication Low High
Can Generalize? Yes (may be) No
Control is the key factor
far@[Link] 18
Hypothesis
The first step is deciding what to investigate.
The goal for the research can be expressed as a
hypothesis in quantifiable terms that is to be tested.
The test result (the collected data) will confirm or
refute the hypothesis.
Example:
Can Software Reliability Engineering (SRE) help us
to achieve an overall improvements in software
development practice in our company?
far@[Link] 19
Examples /1
Experiment: research in the small
You have heard about software reliability
engineering (SRE) and its advantages and may
want to investigate whether to use SRE in your
company. You may design a controlled (dummy)
project and apply the SRE technique to it. You
may want to experiment with the various phases
of application (defining operational profile,
developing test-cases and decision upon
adequacy of test run) and document the results
for further investigation.
far@[Link] 20
Examples /2
Case study: research in the typical
You may have used software reliability
engineering (SRE) for the first time in a project
in your company. After the project is completed,
you may perform a case-study to capture the
effort involved (budget, personnel), the number
of failures investigated, and the project duration.
far@[Link] 21
Examples /3
Survey: investigate in the large
After you have used SRE in many projects in
your company, you may conduct a survey to
capture the effort involved (budget, personnel),
the number of failures investigated, and the
project duration for all the projects. Then, you
may compare these figures with those from
projects using conventional software test
technique to see if SRE could lead to an overall
improvements in practice.
far@[Link] 22
Hypothesis (cont’d)
Other Examples:
Can integrated development and testing tools
improve our productivity?
Does Cleanroom software development produce
better-quality software than using the
conventional development methods?
Does code produced using Agile software
development have a lower number of defects per
KLOC than code produced using the
conventional methods?
far@[Link] 23
Control /1
What variables may affect truth of a hypothesis?
How do they affect it?
Variable:
Independent (values are set by the experiment or initial
conditions)
Dependent (values are affected by change of other
variables)
Example: Effect of “programming language” on
the “quality” of resulting code.
Programming language is an independent and quality is a
dependent variable.
far@[Link] 24
Control /2
A common mistake: ignoring other variables that may affect
the values of a dependent variable.
Example:
Suppose you want to determine whether a change in programming
language (independent variable) can affect the productivity
(dependent variable) of your project. For instance, you currently use
FORTRAN and you want to investigate the effects of changing to
Ada. The values of all other variables should stay the same (e.g.,
application experience, programming environment, type of problem,
etc.)
Without this you cannot be sure that the difference in productivity is
attributable to the change in language.
But list of other variables may grow beyond control!
far@[Link] 25
Control /3
How to identify the dependent and
independent variables?
Example:
A→ D
F &B→Z
D&C →F
Given : { A, B, C}
Using causal ordering:
{ A, B, C} ⇒ {D} ⇒ {F } ⇒ {Z }
far@[Link] 26
Formal Experiments: Planning
1. Conception
Defining the goal of investigation
2. Design
Generating quantifiable (and manageable)
hypotheses to be tested
Defining experimental objects or units
Identifying experimental subject
Identifying the response variable(s)
far@[Link] 27
Formal Experiments: Planning
3. Preparation
Getting ready to start, e.g., purchasing tools,
hardware, training personnel, etc.
4. Execution
5. Review and analysis
Review the results for soundness and validity
6. Dissemination & decision making
Documenting conclusions
far@[Link] 28
Formal Experiments: Principles
1. Replication
Experiment under identical conditions should
be repeatable.
Confounded results (unable to separate the
results of two or more variables) should be
avoided.
2. Randomization
The experimental trials must be organized in a
way that the effects of uncontrolled variables
are minimized
far@[Link] 29
Formal Experiments: Principles
3. Local control
Blocking: allocating experimental units to blocks or
groups so the units within a block are relatively
homogeneous. The blocks are designed so that the
experimental design captures the anticipated variation
in the blocks by grouping like varieties, so that the
variation does not contribute to the experimental error.
Balancing: is the blocking and assigning of treatments
so that an equal number of subjects is assigned to each
treatment. Balancing is desirable because it simplifies
the statistical analysis.
far@[Link] 30
Example: Blocking & Balancing
You are investigating the comparative effects of three design techniques
on the quality of the resulting code.
The experiment involves teaching the techniques to 12 developers and
measuring the number of defects found per 1000 LOC to assess the code
quality.
It may be the case that the twelve developers graduated from three
universities. It is possible that the universities trained the developers in
very different ways, so that being from a particular university can affect
the way in which the design technique is understood or used.
To eliminate this possibility, three blocks can be defined so that the first
block contains all developers from university X, the second block from
university Y, and the third block from university Z. Then, the treatments
are assigned at random to the developers from each block. If the first
block has six developers, two are assigned to design method A, two to B,
and two to C.
far@[Link] 31
Formal Experiments: Principles
3. Local control
Correlation: the most
popular technique to
assess relationships
among observational
data
Linear and nonlinear
correlation.
Nonlinear correlation
is hard to be measured
and may stay hidden.
far@[Link] 32
Formal Experiments: Types
Factorial design:
Crossing (each level of each
factor appears with each level of
the other factor
Nesting (each level of one
occurs entirely in conjunction
with one level of another)
Proper nested or crossed design
may reduce the number of cases
to be tested.
far@[Link] 33
Formal Experiments: Types
Advantages of factorial design
Resources can be used more efficiently
Coverage (completeness) of the target variables’ range of
variation
Implicit replication
Disadvantages of factorial design
Higher costs of preparation, administration and analysis
Number of combinations will grow rapidly
Some of the combinations may be worthless
far@[Link] 34
Formal Experiments: Selection
Selecting the number of
variables:
Single variable
Multiple variables
Example: Measuring time to
code a program module with or
without using a reusable
repository
Without considering the effects of
experience of programmers
With considering the effects of
experience of programmers
far@[Link] 35
Formal Experiments: Baselines
Baseline is an “average”
treatment of a variable in a
number of experiments (or
case studies).
It provides a measure to
identify whether the value
is within an acceptable
range.
It may help checking the
validity of measurement.
far@[Link] 36
Empirical Research Guidelines
far@[Link] 37
Contents
1. Experimental context
2. Experimental design
3. Data collection
4. Analysis
5. Presentation of results
6. Interpretation of results
far@[Link] 38
1. Experimental Context
Goals:
Ensure that the objectives of the experiment
have been properly defined
Ensure that the description of the experiment
provides enough details for the practitioners
far@[Link] 39
1. Experimental Context
C1: Be sure to specify as much of the context as possible. In
particular, clearly define the entities, attributes and measures
that are capturing the contextual information.
C2: If a specific hypothesis is being tested, state it clearly
prior to performing the study, and discuss the theory from
which it is derived, so that its implications are apparent.
C3: If the target is exploratory, state clearly and, prior to
data analysis, what questions the investigation is intended to
address, and how it will address them.
far@[Link] 40
2. Experimental Design
Goal:
Ensure that the design is appropriate for the
objectives of the experiment
Ensure that the objective of the experiment
can be reached using the techniques specified
in the design
far@[Link] 41
2. Experimental Design /1
D1: Identify the population from which the subjects
and objects are drawn.
D2: Define the process by which the subjects and
objects were selected (inclusion/exclusion criteria).
D3: Define the process by which subjects and
objects are assigned to treatments.
D4: Restrict yourself to simple study designs or, at
least, to designs that are fully analyzed in the
literature.
D5: Define the experimental unit.
far@[Link] 42
2. Experimental Design /2
D6: For formal experiments, perform a pre-
experiment or pre-calculation to identify or estimate
the minimum required sample size.
D7: Use appropriate levels of blinding.
D8: Avoid the use of controls unless you are sure
the control situation can be unambiguously defined.
D9: Fully define all treatments (interventions).
D10: Justify the choice of outcome measures in
terms of their relevance to objectives of the
empirical study.
far@[Link] 43
3. Data Collection
Goal
Ensure that the data collection process is well
defined
Monitor the data collection and watch for
deviations from the experiment design
far@[Link] 44
3. Data Collection
DC1: Define all software measures fully, including the
entity, attribute, unit and counting rules.
DC2: Describe any quality control method used to ensure
completeness and accuracy of data collection.
DC3: For observational studies and experiments, record data
about subjects who drop out from the studies.
DC4: For observational studies and experiments, record data
about other performance measures that may be adversely
affected by the treatment, even if they are not the main focus
of the study.
far@[Link] 45
4. Analysis
Goal
Ensure that the collected data from the
experiment is analyzed correctly
Monitor the data analysis and watch for
deviations from the experiment design
far@[Link] 46
4. Analysis
A1: Specify any procedures used to control for
multiple testing.
A2: Consider using blind analysis (avoid “fishing
for results”).
A3: Perform sensitivity analysis.
A4: Ensure that the data do not violate the
assumptions of the tests used on them.
A5: Apply appropriate quality control procedures to
verify the results.
far@[Link] 47
5. Presentation of Results
Goal
Ensure that the reader of the results can
understand the objective, the process and the
results of experiment
far@[Link] 48
5. Presentation of Results
P1: Describe or cite a reference for all procedures used.
Report or cite the statistical package used.
P2: Present quantitative results as well as significance levels.
Quantitative results should show the magnitude of effects
and the confidence limits.
P3: Present the raw data whenever possible. Otherwise,
confirm that they are available for review by the reviewers
and independent auditors.
P4: Provide appropriate descriptive statistics.
P5: Make appropriate use of graphics.
far@[Link] 49
6. Interpretation of Results
Goal
Ensure that the conclusions are derived
merely from the results of the experiment
far@[Link] 50
6. Interpretation of Results
I1: Define the population to which inferential
statistics and predictive models apply.
I2: Differentiate between statistical significance and
practical importance.
I3: Specify any limitations of the study.
far@[Link] 51
far@[Link] 52
