Software Testing and Analysis: Process, Principles

and Techniques
byMauro PezzandMichal Young
John Wiley & Sons 2008 (510 pages)
ISBN:9780471455936
Providing students and professionals with strategies for reliable,
cost-effective software development, this guide covers a full
spectrum of topics from basic principles and underlying theory to
organizational and process issues in real-world application.

Table of Contents
Software Testing and Analysis-Process, Principles, and Techniques
Preface
Part I - Fundamentals of Test and Analysis
Chapter 1 - Software Test and Analysis in a Nutshell
Chapter 2 - A Framework for Test and Analysis
Chapter 3 - Basic Principles
Chapter 4 - Test and Analysis Activities Within a Software Process
Part II - Basic Techniques
Chapter 5 - Finite Models
Chapter 6 - Dependence and Data Flow Models
Chapter 7 - Symbolic Execution and Proof of Properties
Chapter 8 - Finite State Verification
Part III - Problems and Methods
Chapter 9 - Test Case Selection and Adequacy
Chapter 10 - Functional Testing
Chapter 11 - Combinatorial Testing
Chapter 12 - Structural Testing
Chapter 13 - Data Flow Testing
Chapter 14 - Model-Based Testing
Chapter 15 - Testing Object-Oriented Software
Chapter 16 - Fault-Based Testing
Chapter 17 - Test Execution

Chapter 18 - Inspection
Chapter 19 - Program Analysis
Part IV - Process
Chapter 20 - Planning and Monitoring the Process
Chapter 21 - Integration and Component-based Software Testing
Chapter 22 - System, Acceptance, and Regression Testing
Chapter 23 - Automating Analysis and Test
Chapter 24 - Documenting Analysis and Test
Bibliography
Index
List of Figures
List of Tables
List of Sidebars

Back Cover
You can't test quality into a software product, but neither can you
build a quality software product without test and analysis. Software
test and analysis is increasingly recognized, in research and in
industrial practice, as a core challenge in software engineering and
computer science. Software Testing and Analysis: Process,
Principles, and Techniques is the first book to present a range of
complementary software test and analysis techniques in an
integrated, coherent fashion. It covers a full spectrum of topics from
basic principles and underlying theory to organizational and process
issues in real-world application. The emphasis throughout is on
selecting a complementary set of practical techniques to achieve an
acceptable level of quality at an acceptable cost.
Highlights of the book include
Interplay among technical and non-technical issues in crafting
an approach to software quality, with chapters devoted to
planning and monitoring the software quality process.
A selection of practical techniques ranging from inspection to
automated program and design analyses to unit, integration,
system, and regression testing, with technical material set in
the context of real-world problems and constraints in software
development.
A coherent view of the state of the art and practice, with
technical and organizational approaches to push the state of
practice toward the state of the art.
Throughout, the text covers techniques that are suitable for nearterm application, with sufficient technical background to help you
know how and when to apply them. Exercises reinforce the
instruction and ensure that you master each topic before
proceeding.
By incorporating software testing and analysis techniques into
modern practice, Software Testing and Analysis: Process, Principles,

and Techniques provides both students and professionals with

realistic strategies for reliable and cost-effective software
development.
About the Authors
Mauro Pezz, PhD, is Professor of Software Engineering at the
University of Milano Bicocca and visiting Professor of Software
Engineering at the University of Lugano. He received his PhD
degree in computer science from Politecnico di Milano (Italy). Dr.
Pezz is serving as associate editor of ACM Transactions on Software
Engineering, and has served as technical program chair of the 2006
ACM International Symposium on Software Testing and Analysis. He
has been technical lead for several multi-year international research
and development projects in close collaboration with leading
European information and communication companies. He also has
an active consulting practice.
Michal Young, PhD, is Associate Professor of Computer Science at
University of Oregon. He earned a doctorate in information and
computer science from University of California. He has formerly
served on the faculty of Purdue University, the Tecnopadova Master
of Software Engineering Program in Padua, Italy, and the Oregon
Master of Software Engineering. He has served as the technical
program chair of the 1998 ACM International Symposium on
Software Testing and Analysis, program co-chair of the 2002
International conference on Software Engineering, general chair of
the ACM Sigsoft 2006 Symposium on Foundations of Software
Engineering, and as associate editor of IEEE Transactions on
Software Engineering.

Software Testing and Analysis-Process, Principles, and

Techniques
Mauro Pezz
Universit di Milano Bicocca
Michal Young
University of Oregon
PUBLISHER Daniel Sayre
SENIOR PRODUCTION EDITOR Lisa Wojcik
EDITORIAL ASSISTANT Lindsay Murdock
COVER DESIGNER Madelyn Lesure
COVER PHOTO Rick Fischer/Masterfile
WILEY 200TH ANNIVERSARY LOGO DESIGN Richard J. Pacifico
This book was typeset by the authors using pdf LATEX and printed and bound by Malloy
Lithographing. The cover was printed by Phoenix Color Corp. This book is printed on
acid free paper.
2008 John Wiley & Sons, Inc.
All rights reserved. No part of this publication may be reproduced, stored in a retrieval
system or transmitted in any form or by any means, electronic, mechanical,
photocopying, recording, scanning or otherwise, except as permitted under Sections 107
or 108 of the 1976 United States Copyright Act, without either the prior written
permission of the Publisher, or authorization through payment of the appropriate percopy fee to the Copyright Clearance Center, Inc. 222 Rosewood Drive, Danvers, MA
01923, website http://www.copyright.com. Requests to the Publisher for permission
should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111
River Street, Hoboken, NJ 07030-5774, (201) 748-6011, fax (201) 748-6008, website
http://www.wiley.com/go/permissions.
To order books or for customer service please, call 1-800-CALL WILEY (225-5945).
978-0-471-45593-6
10 9 8 7 5 6 4 3 2 1

Preface

Overview
This book addresses software test and analysis in the context of an
overall effort to achieve quality. It is designed for use as a primary
textbook for a course in software test and analysis or as a
supplementary text in a software engineering course, and as a
resource for software developers.
The main characteristics of this book are:
It assumes that the reader's goal is to achieve a suitable balance of cost, schedule,
and quality. It is not oriented toward critical systems for which ultra-high reliability
must be obtained regardless of cost, nor will it be helpful if one's aim is to cut cost or
schedule regardless of consequence.
It presents a selection of techniques suitable for near-term application, with sufficient
technical background to understand their domain of applicability and to consider
variations to suit technical and organizational constraints. Techniques of only
historical interest and techniques that are unlikely to be practical in the near future
are omitted.
It promotes a vision of software testing and analysis as integral to modern software
engineering practice, equally as important and technically demanding as other
aspects of development. This vision is generally consistent with current thinking on
the subject, and is approached by some leading organizations, but is not universal.
It treats software testing and static analysis techniques together in a coherent
framework, as complementary approaches for achieving adequate quality at
acceptable cost.

Why This Book?

One cannot "test quality into" a badly constructed software product, but neither can one
build quality into a product without test and analysis. The goal of acceptable quality at
acceptable cost is both a technical and a managerial challenge, and meeting the goal
requires a grasp of both the technical issues and their context in software development.
It is widely acknowledged today that software quality assurance should not be a phase
between development and deployment, but rather a set of ongoing activities interwoven
with every task from initial requirements gathering through evolution of the deployed
product. Realization of this vision in practice is often only partial. It requires careful
choices and combinations of techniques fit to the organization, products, and processes,
but few people are familiar with the full range of techniques, from inspection to testing to
automated analyses. Those best positioned to shape the organization and its processes
are seldom familiar with the technical issues, and vice versa. Moreover, there still
persists in many organizations a perception that quality assurance requires less skill or
background than other aspects of development.
This book provides students with a coherent view of the state of the art and practice,
and provides developers and managers with technical and organizational approaches to
push the state of practice toward the state of the art.

Who Is This Book For?

Students who read portions of this book will gain a basic understanding of principles and
issues in software test and analysis, including an introduction to process and
organizational issues. Developers, including quality assurance professionals, will find a
variety of techniques with sufficient discussion of technical and process issues to support
adaptation to the particular demands of their organization and application domain.
Technical managers will find a coherent approach to weaving software quality
assurance into the overall software process. All readers should obtain a clearer view of
the interplay among technical and nontechnical issues in crafting an approach to
software quality.
Students, developers, and technical managers with a basic background in computer
science and software engineering will find the material in this book accessible without
additional preparation. Some of the material is technically demanding, but readers may
skim it on a first reading to get the big picture, and return to it at need.
A basic premise of this book is that effective quality assurance is best achieved by
selection and combination of techniques that are carefully woven into (not grafted onto)
a software development process for a particular organization. A software quality
engineer seeking technical advice will find here encouragement to consider a wider
context and participate in shaping the development process. A manager whose faith lies
entirely in process, to the exclusion of technical knowledge and judgment, will find here
many connections between technical and process issues, and a rationale for a more
comprehensive view.

How to Read This Book

This book is designed to permit selective reading. Most readers should begin with Part I,
which presents fundamental principles in a coherent framework and lays the groundwork
for understanding the strengths and weaknesses of individual techniques and their
application in an effective software process. Part II brings together basic technical
background for many testing and analysis methods. Those interested in particular
methods may proceed directly to the relevant chapters in Part III of the book. Where
there are dependencies, the Required Background section at the beginning of a chapter
indicates what should be read in preparation. Part IV discusses how to design a
systematic testing and analysis process and incorporates it into an overall development
process, and may be read either before or after Part III.
Readers new to the field of software test and analysis can obtain an overview by
reading Chapters
1

Software Test and Analysis in a nutshell

A Framework for Test and Analysis

4 Test and Analysis Activities within a Software Process 10 Functional Testing

11

Combinatorial Testing

14

Model-Based Testing

15

Testing Object-Oriented Software

17

Test Execution

18

Inspection

19

Program Analysis

20
Open table as spreadsheet

Planning and Monitoring the Process

Notes for Instructors

This book can be used in an introductory course in software test and analysis or as a
supplementary text in an undergraduate software engineering course.
An introductory graduate-level or an undergraduate level course in software test and
analysis can cover most of the book. In particular, it should include
All of Part I (Fundamentals of Test and Analysis), which provides a complete
overview.
Most of Part II (Basic Techniques), which provides fundamental background,
possibly omitting the latter parts of Chapters 6 (Dependence and Data Flow
Models) and 7 (Symbolic Execution and Proof of Properties). These chapters
are particularly suited for students who focus on theoretical foundations and
those who plan to study analysis and testing more deeply.
A selection of materials from Parts III (Problems and Methods) and IV
(Process).
For a course with more emphasis on techniques than process, we recommend
Chapter 10 (Functional Testing), to understand how to approach black-box
testing.
The overview section and at least one other section of Chapter 11
(Combinatorial Testing) to grasp some combinatorial techniques.
Chapter 12 (Structural Testing), through Section 12.3, to introduce the basic
coverage criteria.
Chapter 13 (Data Flow Testing), through Section 13.3, to see an important
application of data flow analysis to software testing.
The overview section and at least one other section of Chapter 14 (Model-based
Testing) to grasp the interplay between models and testing.
Chapter 15 (Testing Object-Oriented Software) to appreciate implications of the
object-oriented paradigm on analysis and testing.
Chapter 17 (Test Execution), to manage an easily overlooked set of problems
and costs.
Chapter 18 (Inspection) to grasp the essential features of inspection and
appreciate the complementarity of analysis and test.

Chapter 19 (Program Analysis) to understand the role of automated program

analyses and their relation to testing and inspection techniques.
Chapters 20 (Planning and Monitoring the Process), 21 (Integration and
Component based Software Testing), and 22 (System, Acceptance, and
Regression Testing) to widen the picture of the analysis and testing process.
For a stronger focus on software process and organizational issues, we recommend
Chapter 10 (Functional Testing), a selection from Chapters 11 and 14
(Combinatorial Testing and Model-Based Testing), and Chapters 15 (Testing
Object- Oriented Software), 17 (Test Execution), 18 (Inspection), and 19
(Program Analysis) to provide a basic overview of techniques.
Part IV, possibly omitting Chapter 23 (Automating Analysis and Test), for a
comprehensive view of the quality process.
When used as a supplementary text in an undergraduate software engineering course,
Chapters 1 (Software Test and Analysis in a Nutshell), and 2 (A Framework for Test and
Analysis) can provide a brief overview of the field. We recommend completing these two
essential chapters along with either Chapter 4, or a selection of chapters from Part III,
or both, depending on the course schedule. Chapter 4 (Test and Analysis Activities
within a Software Process) can be used to understand the essential aspects of a quality
process. The following chapters from Part III will help students grasp essential
techniques:
Chapter 10 (Functional Testing) and a selection of techniques from Chapters 11
(Combinatorial Testing) and 14 (Model-Based Testing), to grasp basic black-box
testing techniques.
Chapter 12 (Structural Testing), through Section 12.3, to introduce basic
coverage criteria.
Chapter 15 (Testing Object-Oriented Software), through Section 15.3, to
appreciate implications of the object oriented paradigm on analysis and testing.
Chapter 17 (Test Execution), to manage an easily overlooked set of problems
and costs.
Chapter 18 (Inspection), to grasp the essential features of inspection.

Figure 1: Selecting core material by need

In addition, Chapter 20 (Planning and Monitoring the Process) is useful to gain a deeper
appreciation of the interplay between software quality activities and other aspects of a
software process.
If the computer science graduate curriculum does not include a course devoted to
analysis and testing, we recommend that a graduate software engineering course also
cover Chapters 5 (Finite Models), 8 (Finite State Verification), and 19 (Program
Analysis) to provide essential technical background.
Supplementary material and a discussion forum are available on the book Web site,
http://www.wiley.com/college/pezze

Part I: Fundamentals of Test and Analysis

Chapter List
Chapter 1: Software Test and Analysis in a Nutshell
Chapter 2: A Framework for Test and Analysis
Chapter 3: Basic Principles
Chapter 4: Test and Analysis Activities Within a Software Process

Chapter 1: Software Test and Analysis in a Nutshell

Before considering individual aspects and techniques of software analysis and testing, it
is useful to view the "big picture" of software quality in the context of a software
development project and organization. The objective of this chapter is to introduce the
range of software verification and validation (V&V) activities and a rationale for selecting
and combining them within a software development process. This overview is
necessarily cursory and incomplete, with many details deferred to subsequent chapters.

1.1 Engineering Processes and Verification

Engineering disciplines pair design and construction activities with activities that check
intermediate and final products so that defects can be identified and removed. Software
engineering is no exception: Construction of high-quality software requires
complementary pairing of design and verification activities throughout development.
Verification and design activities take various forms ranging from those suited to highly
repetitive construction of noncritical items for mass markets to highly customized or
highly critical products. Appropriate verification activities depend on the engineering
discipline, the construction process, the final product, and quality requirements.
Repetition and high levels of automation in production lines reduce the need for
verification of individual products. For example, only a few key components of products
like screens, circuit boards, and toasters are verified individually. The final products are
tested statistically. Full test of each individual product may not be economical, depending
on the costs of testing, the reliability of the production process, and the costs of field
failures.
Even for some mass market products, complex processes or stringent quality
requirements may require both sophisticated design and advanced product verification
procedures. For example, computers, cars, and aircraft, despite being produced in
series, are checked individually before release to customers. Other products are not
built in series, but are engineered individually through highly evolved processes and
tools. Custom houses, race cars, and software are not built in series. Rather, each
house, each racing car, and each software package is at least partly unique in its design
and functionality. Such products are verified individually both during and after production
to identify and eliminate faults.
Verification of goods produced in series (e.g., screens, boards, or toasters) consists of
repeating a predefined set of tests and analyses that indicate whether the products
meet the required quality standards. In contrast, verification of a unique product, such as
a house, requires the design of a specialized set of tests and analyses to assess the
quality of that product. Moreover, the relationship between the test and analysis results
and the quality of the product cannot be defined once for all items, but must be
assessed for each product. For example, the set of resistance tests for assessing the
quality of a floor must be customized for each floor, and the resulting quality depends on
the construction methods and the structure of the building.
Verification grows more difficult with the complexity and variety of the products. Small
houses built with comparable technologies in analogous environments can be verified
with standardized procedures. The tests are parameterized to the particular house, but
are nonetheless routine. Verification of a skyscraper or of a house built in an extreme
seismic area, on the other hand, may not be easily generalized, instead requiring

specialized tests and analyses designed particularly for the case at hand.
Software is among the most variable and complex of artifacts engineered on a regular
basis. Quality requirements of software used in one environment may be quite different
and incompatible with quality requirements of a different environment or application
domain, and its structure evolves and often deteriorates as the software system grows.
Moreover, the inherent nonlinearity of software systems and uneven distribution of faults
complicates verification. If an elevator can safely carry a load of 1000 kg, it can also
safely carry any smaller load, but if a procedure correctly sorts a set of 256 elements, it
may fail on a set of 255 or 53 or 12 elements, as well as on 257 or 1023.
The cost of software verification often exceeds half the overall cost of software
development and maintenance. Advanced development technologies and powerful
supporting tools can reduce the frequency of some classes of errors, but we are far
from eliminating errors and producing fault-free software. In many cases new
development approaches introduce new subtle kinds of faults, which may be more
difficult to reveal and remove than classic faults. This is the case, for example, with
distributed software, which can present problems of deadlock or race conditions that are
not present in sequential programs. Likewise, object-oriented development introduces
new problems due to the use of polymorphism, dynamic binding, and private state that
are absent or less pronounced in procedural software.
The variety of problems and the richness of approaches make it challenging to choose
and schedule the right blend of techniques to reach the required level of quality within
cost constraints. There are no fixed recipes for attacking the problem of verifying a
software product. Even the most experienced specialists do not have pre-cooked
solutions, but need to design a solution that suits the problem, the requirements, and the
development environment.

1.2 Basic Questions

To start understanding how to attack the problem of verifying
software, let us consider a hypothetical case. The Board of
Governors of Chipmunk Computers, an (imaginary) computer
manufacturer, decides to add new online shopping functions to the
company Web presence to allow customers to purchase individually
configured products. Let us assume the role of quality manager. To
begin, we need to answer a few basic questions:
When do verification and validation start? When are they
complete?
What particular techniques should be applied during
development of the product to obtain acceptable quality at an
acceptable cost?
How can we assess the readiness of a product for release?

How can we control the quality of successive releases?

How can the development process itself be improved over the course of the current
and future projects to improve products and make verification more costeffective?

1.3 When Do Verification and Validation Start and End?

Although some primitive software development processes concentrate testing and
analysis at the end of the development cycle, and the job title "tester" in some
organizations still refers to a person who merely executes test cases on a complete
product, today it is widely understood that execution of tests is a small part of the
verification and validation process required to assess and maintain the quality of a
software product.
Verification and validation start as soon as we decide to build a software product, or
even before. In the case of Chipmunk Computers, when the Board of Governors asks
the information technology (IT) manager for a feasibility study, the IT manager considers
not only functionality and development costs, but also the required qualities and their
impact on the overall cost.
The Chipmunk software quality manager participates with other key designers in the
feasibility study, focusing in particular on risk analysis and the measures needed to
assess and control quality at each stage of development. The team assesses the impact
of new features and new quality requirements on the full system and considers the
contribution of quality control activities to development cost and schedule. For example,
migrating sales functions into the Chipmunk Web site will increase the criticality of
system availability and introduce new security issues. A feasibility study that ignored
quality could lead to major unanticipated costs and delays and very possibly to project
failure.
The feasibility study necessarily involves some tentative architectural design, for
example, a division of software structure corresponding to a division of responsibility
between a human interface design team and groups responsible for core business
software ("business logic") and supporting infrastructure, and a rough build plan breaking
the project into a series of incremental deliveries. Opportunities and obstacles for
costeffective verification are important considerations in factoring the development effort
into subsystems and phases, and in defining major interfaces.
Overall architectural design divides work and separates qualities that can be verified
independently in the different subsystems, thus easing the work of the testing team as
well as other developers. For example, the Chipmunk design team divides the system
into a presentation layer, back-end logic, and infrastructure. Development of the three
subsystems is assigned to three different teams with specialized experience, each of
which must meet appropriate quality constraints. The quality manager steers the early
design toward a separation of concerns that will facilitate test and analysis.
In the Chipmunk Web presence, a clean interface between the presentation layer and
back end logic allows a corresponding division between usability testing (which is the
responsibility of the human interface group, rather than the quality group) and verification

of correct functioning. A clear separation of infrastructure from business logic serves a

similar purpose. Responsibility for a small kernel of critical functions is allocated to
specialists on the infrastructure team, leaving effectively checkable rules for consistent
use of those functions throughout other parts of the system.
Taking into account quality constraints during early breakdown into subsystems allows
for a better allocation of quality requirements and facilitates both detailed design and
testing. However, many properties cannot be guaranteed by one subsystem alone. The
initial breakdown of properties given in the feasibility study will be detailed during later
design and may result in "cross-quality requirements" among subsystems. For example,
to guarantee a given security level, the infrastructure design team may require
verification of the absence of some specific security holes (e.g., buffer overflow) in other
parts of the system.
The initial build plan also includes some preliminary decisions about test and analysis
techniques to be used in development. For example, the preliminary prototype of
Chipmunk on-line sales functionality will not undergo complete acceptance testing, but
will be used to validate the requirements analysis and some design decisions.
Acceptance testing of the first release will be based primarily on feedback from selected
retail stores, but will also include complete checks to verify absence of common security
holes. The second release will include full acceptance test and reliability measures.
If the feasibility study leads to a project commitment, verification and validation (V&V)
activities will commence with other development activities, and like development itself will
continue long past initial delivery of a product. Chipmunk's new Web- based functions will
be delivered in a series of phases, with requirements reassessed and modified after
each phase, so it is essential that the V&V plan be cost-effective over a series of
deliveries whose outcome cannot be fully known in advance. Even when the project is
"complete," the software will continue to evolve and adapt to new conditions, such as a
new version of the underlying database, or new requirements, such as the opening of a
European sales division of Chipmunk. V&V activities continue through each small or large
change to the system.
Why Combine Techniques?
No single test or analysis technique can serve all purposes. The primary reasons for
combining techniques, rather than choosing a single "best" technique, are
Effectiveness for different classes of faults. For example, race conditions are
very difficult to find with conventional testing, but they can be detected with
static analysis techniques.
Applicability at different points in a project. For example, we can apply
inspection techniques very early to requirements and design representations

that are not suited to more automated analyses.

Differences in purpose. For example, systematic (nonrandom) testing is
aimed at maximizing fault detection, but cannot be used to measure reliability;
for that, statistical testing is required.
Trade-offs in cost and assurance. For example, one may use a relatively
expensive technique to establish a few key properties of core components
(e.g., a security kernel) when those techniques would be too expensive for
use throughout a project.

1.4 What Techniques Should Be Applied?

The feasibility study is the first step of a complex development process that should lead
to delivery of a satisfactory product through design, verification, and validation activities.
Verification activities steer the process toward the construction of a product that
satisfies the requirements by checking the quality of intermediate artifacts as well as the
ultimate product. Validation activities check the correspondence of the intermediate
artifacts and the final product to users' expectations.
The choice of the set of test and analysis techniques depends on quality, cost,
scheduling, and resource constraints in development of a particular product. For the
business logic subsystem, the quality team plans to use a preliminary prototype for
validating requirements specifications. They plan to use automatic tools for simple
structural checks of the architecture and design specifications. They will train staff for
design and code inspections, which will be based on company checklists that identify
deviations from design rules for ensuring maintainability, scalability, and correspondence
between design and code.
Requirements specifications at Chipmunk are written in a structured, semiformal format.
They are not amenable to automated checking, but like any other software artifact they
can be inspected by developers. The Chipmunk organization has compiled a checklist
based on their rules for structuring specification documents and on experience with
problems in requirements from past systems. For example, the checklist for inspecting
requirements specifications at Chipmunk asks inspectors to confirm that each specified
property is stated in a form that can be effectively tested.
The analysis and test plan requires inspection of requirements specifications, design
specifications, source code, and test documentation. Most source code and test
documentation inspections are a simple matter of soliciting an off-line review by one
other developer, though a handful of critical components are designated for an additional
review and comparison of notes. Component interface specifications are inspected by
small groups that include a representative of the "provider" and "consumer" sides of the
interface, again mostly off-line with exchange of notes through a discussion service. A
larger group and more involved process, including a moderated inspection meeting with
three or four participants, is used for inspection of a requirements specification.
Chipmunk developers produce functional unit tests with each development work
assignment, as well as test oracles and any other scaffolding required for test execution.
Test scaffolding is additional code needed to execute a unit or a subsystem in isolation.
Test oracles check the results of executing the code and signal discrepancies between
actual and expected outputs.
Test cases at Chipmunk are based primarily on interface specifications, but the extent to
which unit tests exercise the control structure of programs is also measured. If less than

90% of all statements are executed by the functional tests, this is taken as an indication
that either the interface specifications are incomplete (if the missing coverage
corresponds to visible differences in behavior), or else additional implementation
complexity hides behind the interface. Either way, additional test cases are devised
based on a more complete description of unit behavior.
Integration and system tests are generated by the quality team, working from a catalog
of patterns and corresponding tests. The behavior of some subsystems or components
is modeled as finite state machines, so the quality team creates test suites that exercise
program paths corresponding to each state transition in the models.
Scaffolding and oracles for integration testing are part of the overall system architecture.
Oracles for individual components and units are designed and implemented by
programmers using tools for annotating code with conditions and invariants. The
Chipmunk developers use a home-grown test organizer tool to bind scaffolding to code,
schedule test runs, track faults, and organize and update regression test suites.
The quality plan includes analysis and test activities for several properties distinct from
functional correctness, including performance, usability, and security. Although these are
an integral part of the quality plan, their design and execution are delegated in part or
whole to experts who may reside elsewhere in the organization. For example, Chipmunk
maintains a small team of human factors experts in its software division. The human
factors team will produce look-and-feel guidelines for the Web purchasing system, which
together with a larger body of Chipmunk interface design rules can be checked during
inspection and test. The human factors team also produces and executes a usability
testing plan.
Parts of the portfolio of verification and validation activities selected by Chipmunk are
illustrated in Figure 1.1. The quality of the final product and the costs of the quality
assurance activities depend on the choice of the techniques to accomplish each activity.
Most important is to construct a coherent plan that can be monitored. In addition to
monitoring schedule progress against the plan, Chipmunk records faults found during
each activity, using this as an indicator of potential trouble spots. For example, if the
number of faults found in a component during design inspections is high, additional
dynamic test time will be planned for that component.

Figure 1.1: Main analysis and testing activities through the software life
cycle.

1.5 How Can We Assess the Readiness of a

Product?
Analysis and testing activities during development are intended
primarily to reveal faults so that they can be removed. Identifying
and removing as many faults as possible is a useful objective during
development, but finding all faults is nearly impossible and seldom a
cost-effective objective for a nontrivial software product. Analysis
and test cannot go on forever: Products must be delivered when
they meet an adequate level of functionality and quality. We must
have some way to specify the required level of dependability and to
determine when that level has been attained.
Different measures of dependability are appropriate in different
contexts. Availability measures the quality of service in terms of
running versus down time; mean time between failures (MTBF)
measures the quality of the service in terms of time between
failures, that is, length of time intervals during which the service is
available. Reliability is sometimes used synonymously with
availability or MTBF, but usually indicates the fraction of all
attempted operations (program runs, or interactions, or sessions)
that complete successfully.
Both availability and reliability are important for the Chipmunk Web
presence. The availability goal is set (somewhat arbitrarily) at an
average of no more than 30 minutes of down time per month. Since
30 one-minute failures in the course of a day would be much worse
than a single 30-minute failure, MTBF is separately specified as at
least one week. In addition, a reliability goal of less than 1 failure
per 1000 user sessions is set, with a further stipulation that certain
critical failures (e.g., loss of data) must be vanishingly rare.
Having set these goals, how can Chipmunk determine when it has met them? Monitoring
systematic debug testing can provide a hint, but no more. A product with only a single fault
can have a reliability of zero if that fault results in a failure on every execution, and there is
no reason to suppose that a test suite designed for finding faults is at all representative of
actual usage and failure rate.
From the experience of many previous projects, Chipmunk has empirically determined that

in its organization, it is fruitful to begin measuring reliability when debug testing is yielding
less than one fault ("bug") per day of tester time. For some application domains, Chipmunk
has gathered a large amount of historical usage data from which to define an operational
profile, and these profiles can be used to generate large, statistically valid sets of randomly
generated tests. If the sample thus tested is a valid model of actual executions, then
projecting actual reliability from the failure rate of test cases is elementary. Unfortunately, in
many cases such an operational profile is not available.
Chipmunk has an idea of how the Web sales facility will be used, but it cannot construct and
validate a model with sufficient detail to obtain reliability estimates from a randomly
generated test suite. They decide, therefore, to use the second major approach to verifying
reliability, using a sample of real users. This is commonly known as alpha testing if the tests
are performed by users in a controlled environment, observed by the development
organization. If the tests consist of real users in their own environment, performing actual
tasks without interference or close monitoring, it is known as beta testing. The Chipmunk
team plans a very small alpha test, followed by a longer beta test period in which the
software is made available only in retail outlets. To accelerate reliability measurement after
subsequent revisions of the system, the beta test version will be extensively instrumented,
capturing many properties of a usage profile.

1.6 How Can We Ensure the Quality of Successive

Releases?
Software test and analysis does not stop at the first release. Software products often
operate for many years, frequently much beyond their planned life cycle, and undergo
many changes. They adapt to environment changes-for example, introduction of new
device drivers, evolution of the operating system, and changes in the underlying
database. They also evolve to serve new and changing user requirements. Ongoing
quality tasks include test and analysis of new and modified code, reexecution of system
tests, and extensive record-keeping.
Chipmunk maintains a database for tracking problems. This database serves a dual
purpose of tracking and prioritizing actual, known program faults and their resolution and
managing communication with users who file problem reports. Even at initial release, the
database usually includes some known faults, because market pressure seldom allows
correcting all known faults before product release. Moreover, "bugs" in the database are
not always and uniquely associated with real program faults. Some problems reported
by users are misunderstandings and feature requests, and many distinct reports turn out
to be duplicates which are eventually consolidated.
Chipmunk designates relatively major revisions, involving several developers, as "point
releases," and smaller revisions as "patch level" releases. The full quality process is
repeated in miniature for each point release, including everything from inspection of
revised requirements to design and execution of new unit, integration, system, and
acceptance test cases. A major point release is likely even to repeat a period of beta
testing.
Patch level revisions are often urgent for at least some customers. For example, a patch
level revision is likely when a fault prevents some customers from using the software or
when a new security vulnerability is discovered. Test and analysis for patch level
revisions is abbreviated, and automation is particularly important for obtaining a
reasonable level of assurance with very fast turnaround. Chipmunk maintains an
extensive suite of regression tests. The Chipmunk development environment supports
recording, classification, and automatic re-execution of test cases. Each point release
must undergo complete regression testing before release, but patch level revisions may
be released with a subset of regression tests that run unattended overnight.
When fixing one fault, it is all too easy to introduce a new fault or re-introduce faults that
have occurred in the past. Chipmunk developers add new regression test cases as
faults are discovered and repaired.

1.7 How Can the Development Process Be Improved?

As part of an overall process improvement program, Chipmunk has implemented a
quality improvement program. In the past, the quality team encountered the same
defects in project after project. The quality improvement program tracks and classifies
faults to identify the human errors that cause them and weaknesses in test and analysis
that allow them to remain undetected.
Chipmunk quality improvement group members are drawn from developers and quality
specialists on several project teams. The group produces recommendations that may
include modifications to development and test practices, tool and technology support,
and management practices. The explicit attention to buffer overflow in networked
applications at Chipmunk is the result of failure analysis in previous projects.
Fault analysis and process improvement comprise four main phases: Defining the data
to be collected and implementing procedures for collecting it; analyzing collected data to
identify important fault classes; analyzing selected fault classes to identify weaknesses
in development and quality measures; and adjusting the quality and development
process.
Collection of data is particularly crucial and often difficult. Earlier attempts by Chipmunk
quality teams to impose fault data collection practices were a dismal failure. The quality
team possessed neither carrots nor sticks to motivate developers under schedule
pressure. An overall process improvement program undertaken by the Chipmunk
software division provided an opportunity to better integrate fault data collection with
other practices, including the normal procedure for assigning, tracking, and reviewing
development work assignments. Quality process improvement is distinct from the goal of
improving an individual product, but initial data collection is integrated in the same bug
tracking system, which in turn is integrated with the revision and configuration control
system used by Chipmunk developers.
The quality improvement group defines the information that must be collected for
faultiness data to be useful as well as the format and organization of that data.
Participation of developers in designing the data collection process is essential to
balance the cost of data collection and analysis with its utility, and to build acceptance
among developers.
Data from several projects over time are aggregated and classified to identify classes of
faults that are important because they occur frequently, because they cause particularly
severe failures, or because they are costly to repair. These faults are analyzed to
understand how they are initially introduced and why they escape detection. The
improvement steps recommended by the quality improvement group may include specific
analysis or testing steps for earlier fault detection, but they may also include design
rules and modifications to development and even to management practices. An

important part of each recommended practice is an accompanying recommendation for

measuring the impact of the change.

Summary
The quality process has three distinct goals: improving a software product (by
preventing, detecting, and removing faults), assessing the quality of the software
product (with respect to explicit quality goals), and improving the long-term quality and
cost- effectiveness of the quality process itself. Each goal requires weaving quality
assurance and improvement activities into an overall development process, from product
inception through deployment, evolution, and retirement.
Each organization must devise, evaluate, and refine an approach suited to that
organization and application domain. A well-designed approach will invariably combine
several test and analysis techniques, spread across stages of development. An array of
fault detection techniques are distributed across development stages so that faults are
removed as soon as possible. The overall cost and cost-effectiveness of techniques
depends to a large degree on the extent to which they can be incrementally re-applied
as the product evolves.

Further Reading
This book deals primarily with software analysis and testing to improve and assess the
dependability of software. That is not because qualities other than dependability are
unimportant, but rather because they require their own specialized approaches and
techniques. We offer here a few starting points for considering some other important
properties that interact with dependability. Norman's The Design of Everyday Things
[Nor90] is a classic introduction to design for usability, with basic principles that apply to
both hardware and software artifacts. A primary reference on usability for interactive
computer software, and particularly for Web applications, is Nielsen's Designing Web
Usability [Nie00]. Bishop's text Computer Security: Art and Science [Bis02] is a good
introduction to security issues. The most comprehensive introduction to software safety
is Leveson's Safeware [Lev95].

Exercises
Philip has studied "just-in-time" industrial production methods and is convinced that
they should be applied to every aspect of software development. He argues that
1.1 test case design should be performed just before the first opportunity to execute
the newly designed test cases, never earlier. What positive and negative
consequences do you foresee for this just-in-time test case design approach?
A newly hired project manager at Chipmunk questions why the quality manager is

involved in the feasibility study phase of the project, rather than joining the team
1.2 only when the project has been approved, as at the new manager's previous
company. What argument(s) might the quality manager offer in favor of her
involvement in the feasibility study?
Chipmunk procedures call for peer review not only of each source code module,
but also of test cases and scaffolding for testing that module. Anita argues that
inspecting test suites is a waste of time; any time spent on inspecting a test case
designed to detect a particular class of fault could more effectively be spent
1.3 inspecting the source code to detect that class of fault. Anita's project manager, on
the other hand, argues that inspecting test cases and scaffolding can be costeffective when considered over the whole lifetime of a software product. What
argument(s) might Anita's manager offer in favor of this conclusion?
The spiral model of software development prescribes sequencing incremental
prototyping phases for risk reduction, beginning with the most important project
risks. Architectural design for testability involves, in addition to defining testable
1.4 interface specifications for each major module, establishing a build order that
supports thorough testing after each stage of construction. How might spiral
development and design for test be complementary or in conflict?
You manage an online service that sells downloadable video recordings of classic
movies. A typical download takes one hour, and an interrupted download must be
restarted from the beginning. The number of customers engaged in a download at
1.5 any given time ranges from about 10 to about 150 during peak hours. On average,
your system goes down (dropping all connections) about two times per week, for
an average of three minutes each time. If you can double availability or double
mean time between failures, but not both, which will you choose? Why?
Having no a priori operational profile for reliability measurement, Chipmunk will
depend on alpha and beta testing to assess the readiness of its online purchase
functionality for public release. Beta testing will be carried out in retail outlets, by
1.6 retail store personnel, and then by customers with retail store personnel looking on.
How might this beta testing still be misleading with respect to reliability of the
software as it will be used at home and work by actual customers? What might
Chipmunk do to ameliorate potential problems from this reliability misestimation?

1.7

The junior test designers of Chipmunk Computers are annoyed by the procedures
for storing test cases together with scaffolding, test results, and related
documentation. They blame the extra effort needed to produce and store such data
for delays in test design and execution. They argue for reducing the data to store

to the minimum required for reexecuting test cases, eliminating details of test
documentation, and limiting test results to the information needed for generating
oracles. What argument(s) might the quality manager use to convince the junior
test designers of the usefulness of storing all this information?

Chapter 2: A Framework for Test and Analysis

The purpose of software test and analysis is either to assess software qualities or else
to make it possible to improve the software by finding defects. Of the many kinds of
software qualities, those addressed by the analysis and test techniques discussed in this
book are the dependability properties of the software product.
There are no perfect test or analysis techniques, nor a single "best" technique for all
circumstances. Rather, techniques exist in a complex space of trade-offs, and often
have complementary strengths and weaknesses. This chapter describes the nature of
those trade-offs and some of their consequences, and thereby a conceptual framework for understanding and better integrating material from later chapters on individual
techniques.
It is unfortunate that much of the available literature treats testing and analysis as
independent or even as exclusive choices, removing the opportunity to exploit their
complementarities. Armed with a basic understanding of the trade-offs and of strengths
and weaknesses of individual techniques, one can select from and combine an array of
choices to improve the cost-effectiveness of verification.

2.1 Validation and Verification

While software products and processes may be judged on several properties ranging
from time-to-market to performance to usability, the software test and analysis
techniques we consider are focused more narrowly on improving or assessing
dependability.
Assessing the degree to which a software system actually fulfills its requirements, in the
sense of meeting the user's real needs, is called validation. Fulfilling requirements is not
the same as conforming to a requirements specification. A specification is a statement
about a particular proposed solution to a problem, and that proposed solution may or
may not achieve its goals. Moreover, specifications are written by people, and therefore
contain mistakes. A system that meets its actual goals is useful, while a system that is
consistent with its specification is dependable.[1]
"Verification" is checking the consistency of an implementation with a specification. Here,
"specification" and "implementation" are roles, not particular artifacts. For example, an
overall design could play the role of "specification" and a more detailed design could play
the role of "implementation"; checking whether the detailed design is consistent with the
overall design would then be verification of the detailed design. Later, the same detailed
design could play the role of "specification" with respect to source code, which would be
verified against the design. In every case, though, verification is a check of consistency
between two descriptions, in contrast to validation which compares a description
(whether a requirements specification, a design, or a running system) against actual
needs.
Figure 2.1 sketches the relation of verification and validation activities with respect to
artifacts produced in a software development project. The figure should not be
interpreted as prescribing a sequential process, since the goal of a consistent set of
artifacts and user satisfaction are the same whether the software artifacts
(specifications, design, code, etc.) are developed sequentially, iteratively, or in parallel.
Verification activities check consistency between descriptions (design and specifications)
at adjacent levels of detail, and between these descriptions and code.[2] Validation
activities attempt to gauge whether the system actually satisfies its intended purpose.

Figure 2.1: Validation activities check work products against actual user
requirements, while verification activities check consistency of work
products.
Validation activities refer primarily to the overall system specification and the final code.
With respect to overall system specification, validation checks for discrepancies
between actual needs and the system specification as laid out by the analysts, to ensure
that the specification is an adequate guide to building a product that will fulfill its goals.
With respect to final code, validation aims at checking discrepancies between actual
need and the final product, to reveal possible failures of the development process and to
make sure the product meets end-user expectations. Validation checks between the
specification and final product are primarily checks of decisions that were left open in the
specification (e.g., details of the user interface or product features). Chapter 4 provides
a more thorough discussion of validation and verification activities in particular software
process models.
We have omitted one important set of verification checks from Figure 2.1 to avoid
clutter. In addition to checks that compare two or more artifacts, verification includes
checks for self-consistency and well-formedness. For example, while we cannot judge
that a program is "correct" except in reference to a specification of what it should do, we
can certainly determine that some programs are "incorrect" because they are ill- formed.
We may likewise determine that a specification itself is ill-formed because it is
inconsistent (requires two properties that cannot both be true) or ambiguous (can be
interpreted to require some property or not), or because it does not satisfy some other
well-formedness constraint that we impose, such as adherence to a standard imposed
by a regulatory agency.
Validation against actual requirements necessarily involves human judgment and the
potential for ambiguity, misunderstanding, and disagreement. In contrast, a specification
should be sufficiently precise and unambiguous that there can be no disagreement about

whether a particular system behavior is acceptable. While the term testing is often used
informally both for gauging usefulness and verifying the product, the activities differ in
both goals and approach. Our focus here is primarily on dependability, and thus primarily
on verification rather than validation, although techniques for validation and the relation
between the two is discussed further in Chapter 22.
Dependability properties include correctness, reliability, robustness, and safety.
Correctness is absolute consistency with a specification, always and in all
circumstances. Correctness with respect to nontrivial specifications is almost never
achieved. Reliability is a statistical approximation to correctness, expressed as the
likelihood of correct behavior in expected use. Robustness, unlike correctness and
reliability, weighs properties as more and less critical, and distinguishes which properties
should be maintained even under exceptional circumstances in which full functionality
cannot be maintained. Safety is a kind of robustness in which the critical property to be
maintained is avoidance of particular hazardous behaviors. Dependability properties are
discussed further in Chapter 4.
[1]A

good requirements document, or set of documents, should include both a

requirements analysis and a requirements specification, and should clearly distinguish
between the two. The requirements analysis describes the problem. The specification
describes a proposed solution. This is not a book about requirements engineering, but
we note in passing that confounding requirements analysis with requirements
specification will inevitably have negative impacts on both validation and verification.
[2]This

part of the diagram is a variant of the well-known "V model" of verification and
validation.

2.2 Degrees of Freedom

Given a precise specification and a program, it seems that one ought to be able to arrive
at some logically sound argument or proof that a program satisfies the specified
properties. After all, if a civil engineer can perform mathematical calculations to show
that a bridge will carry a specified amount of traffic, shouldn't we be able to similarly
apply mathematical logic to verification of programs?
For some properties and some very simple programs, it is in fact possible to obtain a
logical correctness argument, albeit at high cost. In a few domains, logical correctness
arguments may even be cost-effective for a few isolated, critical components (e.g., a
safety interlock in a medical device). In general, though, one cannot produce a complete
logical "proof" for the full specification of practical programs in full detail. This is not just
a sign that technology for verification is immature. It is, rather, a consequence of one of
the most fundamental properties of computation.
Even before programmable digital computers were in wide use, computing pioneer Alan
Turing proved that some problems cannot be solved by any computer program. The
universality of computers - their ability to carry out any programmed algorithm, including
simulations of other computers - induces logical paradoxes regarding programs (or
algorithms) for analyzing other programs. In particular, logical contradictions ensue from
assuming that there is some program P that can, for some arbitrary program Q and
input I, determine whether Q eventually halts. To avoid those logical contradictions, we
must conclude that no such program for solving the "halting problem" can possibly exist.
Countless university students have encountered the halting problem in a course on the
theory of computing, and most of those who have managed to grasp it at all have
viewed it as a purely theoretical result that, whether fascinating or just weird, is
irrelevant to practical matters of programming. They have been wrong. Almost every
interesting property regarding the behavior of computer programs can be shown to
"embed" the halting problem, that is, the existence of an infallible algorithmic check for
the property of interest would imply the existence of a program that solves the halting
problem, which we know to be impossible.
In theory, undecidability of a property S merely implies that for each verification
technique for checking S, there is at least one "pathological" program for which that
technique cannot obtain a correct answer in finite time. It does not imply that verification
will always fail or even that it will usually fail, only that it will fail in at least one case. In
practice, failure is not only possible but common, and we are forced to accept a
significant degree of inaccuracy.
Program testing is a verification technique and is as vulnerable to undecidability as other
techniques. Exhaustive testing, that is, executing and checking every possible behavior
of a program, would be a "proof by cases," which is a perfectly legitimate way to

construct a logical proof. How long would this take? If we ignore implementation details
such as the size of the memory holding a program and its data, the answer is "forever."
That is, for most programs, exhaustive testing cannot be completed in any finite amount
of time.
Suppose we do make use of the fact that programs are executed on real machines with
finite representations of memory values. Consider the following trivial Java class:
1 class Trivial{
2
static int sum(int a, int b) { return a+b; }
3 }
The Java language definition states that the representation of an int is 32 binary digits,
and thus there are only 232 232 = 264 1021 different inputs on which the method
Trivial.sum() need be tested to obtain a proof of its correctness. At one nanosecond
(109 seconds) per test case, this will take approximately 1012 seconds, or about
30,000 years.
A technique for verifying a property can be inaccurate in one of two directions (Figure
2.2). It may be pessimistic, meaning that it is not guaranteed to accept a program even
if the program does possess the property being analyzed, or it can be optimistic if it
may accept some programs that do not possess the property (i.e., it may not detect all
violations). Testing is the classic optimistic technique, because no finite number of tests
can guarantee correctness. Many automated program analysis techniques for properties
of program behaviors[3] are pessimistic with respect to the properties they are designed
to verify. Some analysis techniques may give a third possible answer, "don't know." We
can consider these techniques to be either optimistic or pessimistic depending on how
we interpret the "don't know" result. Perfection is unobtainable, but one can choose
techniques that err in only a particular direction.

Figure 2.2: Verification trade-off dimensions

A software verification technique that errs only in the pessimistic direction is called a
conservative analysis. It might seem that a conservative analysis would always be
preferable to one that could accept a faulty program. However, a conservative analysis
will often produce a very large number of spurious error reports, in addition to a few
accurate reports. A human may, with some effort, distinguish real faults from a few
spurious reports, but cannot cope effectively with a long list of purported faults of which
most are false alarms. Often only a careful choice of complementary optimistic and
pessimistic techniques can help in mutually reducing the different problems of the
techniques and produce acceptable results.
In addition to pessimistic and optimistic inaccuracy, a third dimension of compromise is
possible: substituting a property that is more easily checked, or constraining the class of
programs that can be checked. Suppose we want to verify a property S, but we are not
willing to accept the optimistic inaccuracy of testing for S, and the only available static
analysis techniques for S result in such huge numbers of spurious error messages that
they are worthless. Suppose we know some property S that is a sufficient, but not
necessary, condition for S (i.e., the validity of S implies S, but not the contrary). Maybe
S is so much simpler than S that it can be analyzed with little or no pessimistic
inaccuracy. If we check S rather than S, then we may be able to provide precise error
messages that describe a real violation of S rather than a potential violation of S.
A Note on Terminology
Many different terms related to pessimistic and optimistic inaccuracy appear in the

literature on program analysis. We have chosen these particular terms because it is

fairly easy to remember which is which. Other terms a reader is likely to encounter
include:
Safe A safe analysis has no optimistic inaccuracy; that is, it accepts only correct
programs. In other kinds of program analysis, safety is related to the goal of the
analysis. For example, a safe analysis related to a program optimization is one that
allows that optimization only when the result of the optimization will be correct.
Sound Soundness is a term to describe evaluation of formulas. An analysis of a
program P with respect to a formula F is sound if the analysis returns True only when
the program actually does satisfy the formula. If satisfaction of a formula F is taken
as an indication of correctness, then a sound analysis is the same as a safe or
conservative analysis.
If the sense of F is reversed (i.e., if the truth of F indicates a fault rather than
correctness) then a sound analysis is not necessarily conservative. In that case it is
allowed optimistic inaccuracy but must not have pessimistic inaccuracy. (Note,
however, that use of the term sound has not always been consistent in the software
engineering literature. Some writers use the term unsound as we use the term
optimistic.)
Complete Completeness, like soundness, is a term to describe evaluation of
formulas. An analysis of a program P with respect to a formula F is complete if the
analysis always returns True when the program actually does satisfy the formula. If
satisfaction of a formula F is taken as an indication of correctness, then a complete
analysis is one that admits only optimistic inaccuracy. An analysis that is sound but
incomplete is a conservative analysis.

Many examples of substituting simple, checkable properties for actual properties of

interest can be found in the design of modern programming languages. Consider, for
example, the property that each variable should be initialized with a value before its
value is used in an expression. In the C language, a compiler cannot provide a precise
static check for this property, because of the possibility of code like the following:
1
2
3
4
5
6
7
8

int i, sum;
int first=1;
for (i=0; i<10; ++i) {
if (first) {
sum=0; first=0;
}
sum += i;
}

It is impossible in general to determine whether each control flow path can be executed,
and while a human will quickly recognize that the variable sum is initialized on the first
iteration of the loop, a compiler or other static analysis tool will typically not be able to
rule out an execution in which the initialization is skipped on the first iteration. Java neatly
solves this problem by making code like this illegal; that is, the rule is that a variable
must be initialized on all program control paths, whether or not those paths can ever be
executed.
Software developers are seldom at liberty to design new restrictions into the
programming languages and compilers they use, but the same principle can be applied
through external tools, not only for programs but also for other software artifacts.
Consider, for example, the following condition that we might wish to impose on
requirements documents:
1. Each significant domain term shall appear with a definition in the glossary of
the document.
This property is nearly impossible to check automatically, since determining whether a
particular word or phrase is a "significant domain term" is a matter of human judgment.
Moreover, human inspection of the requirements document to check this requirement will
be extremely tedious and error-prone. What can we do? One approach is to separate
the decision that requires human judgment (identifying words and phrases as
"significant") from the tedious check for presence in the glossary.
Each significant domain term shall be set off in the requirements document by the
1.a use of a standard style term. The default visual representation of the term style is a
single underline in printed documents and purple text in on-line displays.
Each word or phrase in the term style shall appear with a definition in the glossary
1.b of the document.
Property (1a) still requires human judgment, but it is now in a form that is much more
amenable to inspection. Property (1b) can be easily automated in a way that will be
completely precise (except that the task of determining whether definitions appearing in
the glossary are clear and correct must also be left to humans).
As a second example, consider a Web-based service in which user sessions need not
directly interact, but they do read and modify a shared collection of data on the server.
In this case a critical property is maintaining integrity of the shared data. Testing for this
property is notoriously difficult, because a "race condition" (interference between writing
data in one process and reading or writing related data in another process) may cause
an observable failure only very rarely.
Fortunately, there is a rich body of applicable research results on concurrency control
that can be exploited for this application. It would be foolish to rely primarily on direct

testing for the desired integrity properties. Instead, one would choose a (well- known,
formally verified) concurrency control protocol, such as the two-phase locking protocol,
and rely on some combination of static analysis and program testing to check
conformance to that protocol. Imposing a particular concurrency control protocol
substitutes a much simpler, sufficient property (two-phase locking) for the complex
property of interest (serializability), at some cost in generality; that is, there are
programs that violate two-phase locking and yet, by design or dumb luck, satisfy
serializability of data access.
It is a common practice to further impose a global order on lock accesses, which again
simplifies testing and analysis. Testing would identify execution sequences in which data
is accessed without proper locks, or in which locks are obtained and relinquished in an
order that does not respect the two-phase protocol or the global lock order, even if data
integrity is not violated on that particular execution, because the locking protocol failure
indicates the potential for a dangerous race condition in some other execution that might
occur only rarely or under extreme load.
With the adoption of coding conventions that make locking and unlocking actions easy to
recognize, it may be possible to rely primarily on flow analysis to determine
conformance with the locking protocol, with the role of dynamic testing reduced to a
"back-up" to raise confidence in the soundness of the static analysis. Note that the
critical decision to impose a particular locking protocol is not a post-hoc decision that
can be made in a testing "phase" at the end of development. Rather, the plan for
verification activities with a suitable balance of cost and assurance is part of system
design.
[3]Why

do we bother to say "properties of program behaviors" rather than "program

properties?" Because simple syntactic properties of program text, such as declaring
variables before they are used or indenting properly, can be decided efficiently and
precisely.

2.3 Varieties of Software

The software testing and analysis techniques presented in the main
parts of this book were developed primarily for procedural and
object-oriented software. While these "generic" techniques are at
least partly applicable to most varieties of software, particular
application domains (e.g., real-time and safety-critical software) and
construction methods (e.g., concurrency and physical distribution,
graphical user interfaces) call for particular properties to be verified,
or the relative importance of different properties, as well as
imposing constraints on applicable techniques. Typically a software
system does not fall neatly into one category but rather has a
number of relevant characteristics that must be considered when
planning verification.
As an example, consider a physically distributed (networked) system
for scheduling a group of individuals. The possibility of concurrent
activity introduces considerations that would not be present in a
single-threaded system, such as preserving the integrity of data.
The concurrency is likely to introduce nondeterminism, or else
introduce an obligation to show that the system is deterministic,
either of which will almost certainly need to be addressed through
some formal analysis. The physical distribution may make it
impossible to determine a global system state at one instant, ruling
out some simplistic approaches to system test and, most likely,
suggesting an approach in which dynamic testing of design
conformance of individual processes is combined with static analysis
of their interactions. If in addition the individuals to be coordinated
are fire trucks, then the criticality of assuring prompt response will
likely lead one to choose a design that is amenable to strong
analysis of worst-case behavior, whereas an average- case analysis
might be perfectly acceptable if the individuals are house painters.
As a second example, consider the software controlling a "soft" dashboard display in an
automobile. The display may include ground speed, engine speed (rpm), oil pressure, fuel
level, and so on, in addition to a map and navigation information from a global positioning
system receiver. Clearly usability issues are paramount, and may even impinge on safety
(e.g., if critical information can be hidden beneath or among less critical information). A
disciplined approach will not only place a greater emphasis on validation of usability

throughout development, but to the extent possible will also attempt to codify usability
guidelines in a form that permits verification. For example, if the usability group determines
that the fuel gauge should always be visible when the fuel level is below a quarter of a tank,
then this becomes a specified property that is subject to verification. The graphical interface
also poses a challenge in effectively checking output. This must be addressed partly in the
architectural design of the system, which can make automated testing feasible or not
depending on the interfaces between high-level operations (e.g., opening or closing a
window, checking visibility of a window) and low-level graphical operations and
representations.

Summary
Verification activities are comparisons to determine the consistency of two or more
software artifacts, or self-consistency, or consistency with an externally imposed criterion.
Verification is distinct from validation, which is consideration of whether software fulfills its
actual purpose. Software development always includes some validation and some
verification, although different development approaches may differ greatly in their relative
emphasis.
Precise answers to verification questions are sometimes difficult or impossible to obtain, in
theory as well as in practice. Verification is therefore an art of compromise, accepting some
degree of optimistic inaccuracy (as in testing) or pessimistic inaccuracy (as in many static
analysis techniques) or choosing to check a property that is only an approximation of what
we really wish to check. Often the best approach will not be exclusive reliance on one
technique, but careful choice of a portfolio of test and analysis techniques selected to obtain
acceptable results at acceptable cost, and addressing particular challenges posed by
characteristics of the application domain or software.

Further Reading
The "V" model of verification and validation (of which Figure 2.1 is a variant) appears in
many software engineering textbooks, and in some form can be traced at least as far back
as Myers' classic book [Mye79]. The distinction between validation and verification as given
here follow's Boehm [Boe81], who has most memorably described validation as "building
the right system" and verification as "building the system right."

The limits of testing have likewise been summarized in a famous

aphorism, by Dijkstra [Dij72] who pronounced that "Testing can
show the presence of faults, but not their absence." This phrase has
sometimes been interpreted as implying that one should always
prefer formal verification to testing, but the reader will have noted
that we do not draw that conclusion. Howden's 1976 paper [How76]
is among the earliest treatments of the implications of computability

theory for program testing.

A variant of the diagram in Figure 2.2 and a discussion of pessimistic
and optimistic inaccuracy were presented by Young and Taylor
[YT89]. A more formal characterization of conservative abstractions
in static analysis, called abstract interpretation, was introduced by
Cousot and Cousot in a seminal paper that is, unfortunately, nearly
unreadable [CC77]. We enthusiastically recommend Jones's lucid
introduction to abstract interpretation [JN95], which is suitable for
readers who have a firm general background in computer science
and logic but no special preparation in programming semantics.
There are few general treatments of trade-offs and combinations of software testing and
static analysis, although there are several specific examples, such as work in
communication protocol conformance testing [vBDZ89, FvBK+91]. The two-phase locking
protocol mentioned in Section 2.2 is described in several texts on databases; Bernstein et
al. [BHG87] is particularly thorough.

Exercises

2.1

The Chipmunk marketing division is worried about the start-up time of the new
version of the RodentOS operating system (an (imaginary) operating system of
Chipmunk). The marketing division representative suggests a software requirement
stating that the start-up time shall not be annoying to users.
Explain why this simple requirement is not verifiable and try to reformulate the
requirement to make it verifiable.

2.2

Consider a simple specification language SL that describes

systems diagrammatically in terms of functions, which
represent data transformations and correspond to nodes of the
diagram, and flows, which represent data flows and correspond
to arcs of the diagram.[4] Diagrams can be hierarchically
refined by associating a function F (a node of the diagram)
with an SL specification that details function F. Flows are
labeled to indicate the type of data.
Suggest some checks for self-consistency for SL.

2.3

A calendar program should provide timely reminders; for

example, it should remind the user of an upcoming event early
enough for the user to take action, but not too early.
Unfortunately, "early enough" and "too early" are qualities
that can only be validated with actual users. How might you
derive verifiable dependability properties from the timeliness
requirement?

It is sometimes important in multi-threaded applications to ensure that a sequence of

accesses by one thread to an aggregate data structure (e.g., some kind of table)
appears to other threads as an atomic transaction. When the shared data structure
is maintained by a database system, the database system typically uses
concurrency control protocols to ensure the atomicity of the transactions it manages.
No such automatic support is typically available for data structures maintained by a
program in main memory.
Among the options available to programmers to ensure serializability (the illusion of
atomic access) are the following:
The programmer could maintain very coarse-grain locking, preventing any
interleaving of accesses to the shared data structure, even when such
interleaving would be harmless. (For example, each transaction could be
encapsulated in an single synchronized Java method.) This approach can
cause a great deal of unnecessary blocking between threads, hurting
performance, but it is almost trivial to verify either automatically or manually.
2.4

Automated static analysis techniques can sometimes verify serializability with

finer-grain locking, even when some methods do not use locks at all. This
approach can still reject some sets of methods that would ensure
serializability.
The programmer could be required to use a particular concurrency control
protocol in his or her code, and we could build a static analysis tool that
checks for conformance with that protocol. For example, adherence to the
common two-phase-locking protocol, with a few restrictions, can be checked
in this way.
We might augment the data accesses to build a serializability graph
structure representing the "happens before" relation among transactions in
testing. It can be shown that the transactions executed in serializable manner

if and only if the serializability graph is acyclic.

Compare the relative positions of these approaches on the three axes of verification
techniques: pessimistic inaccuracy, optimistic inaccuracy, and simplified properties.

When updating a program (e.g., for removing a fault, changing or adding a

functionality), programmers may introduce new faults or expose previously hidden
faults. To be sure that the updated version maintains the functionality provided by the
previous version, it is common practice to reexecute the test cases designed for the
former versions of the program. Reexecuting test cases designed for previous
versions is called regression testing.
When testing large complex programs, the number of regression test cases may be
2.5 large. If updated software must be expedited (e.g., to repair a security vulnerability
before it is exploited), test designers may need to select a subset of regression test
cases to be reexecuted.
Subsets of test cases can be selected according to any of several different criteria.
An interesting property of some regression test selection criteria is that they do not
to exclude any test case that could possibly reveal a fault.
How would you classify such a property according to the sidebar of page 21?
[4]Readers expert in Structured Analysis may have noticed that SL

resembles a simple Structured Analysis specification

Chapter 3: Basic Principles

Mature engineering disciplines are characterized by basic principles. Principles provide a
rationale for defining, selecting, and applying techniques and methods. They are valid
beyond a single technique and over a time span in which techniques come and go, and can
help engineers study, define, evaluate, and apply new techniques.

Analysis and testing (A&T) has been common practice since the
earliest software projects. A&T activities were for a long time based
on common sense and individual skills. It has emerged as a distinct
discipline only in the last three decades.
This chapter advocates six principles that characterize various
approaches and techniques for analysis and testing: sensitivity,
redundancy, restriction, partition, visibility, and feedback. Some of
these principles, such as partition, visibility, and feedback, are quite
general in engineering. Others, notably sensitivity, redundancy, and
restriction, are specific to A&T and contribute to characterizing A&T
as a discipline.

3.1 Sensitivity
Human developers make errors, producing faults in software. Faults
may lead to failures, but faulty software may not fail on every
execution. The sensitivity principle states that it is better to fail
every time than sometimes.
Consider the cost of detecting and repairing a software fault. If it is detected immediately
(e.g., by an on-the-fly syntactic check in a design editor), then the cost of correction is very
small, and in fact the line between fault prevention and fault detection is blurred. If a fault is
detected in inspection or unit testing, the cost is still relatively small. If a fault survives initial
detection efforts at the unit level, but triggers a failure detected in integration testing, the
cost of correction is much greater. If the first failure is detected in system or acceptance
testing, the cost is very high indeed, and the most costly faults are those detected by
customers in the field.
A fault that triggers a failure on every execution is unlikely to survive past unit testing. A
characteristic of faults that escape detection until much later is that they trigger failures only
rarely, or in combination with circumstances that seem unrelated or are difficult to control.
For example, a fault that results in a failure only for some unusual configurations of
customer equipment may be difficult and expensive to detect. A fault that results in a failure
randomly but very rarely - for example, a race condition that only occasionally causes data
corruption - may likewise escape detection until the software is in use by thousands of
customers, and even then be difficult to diagnose and correct.

The small C program in Figure 3.1 has three faulty calls to string
copy procedures. The call to strcpy, strncpy, and stringCopy all pass
a source string "Muddled," which is too long to fit in the array
middle. The vulnerability of strcpy is well known, and is the culprit in
the by-now-standard buffer overflow attacks on many network
services. Unfortunately, the fault may or may not cause an
observable failure depending on the arrangement of memory (in this
case, it depends on what appears in the position that would be
middle[7], which will be overwritten with a newline character). The
standard recommendation is to use strncpy in place of strcpy. While
strncpy avoids overwriting other memory, it truncates the input
without warning, and sometimes without properly null-terminating
the output. The replacement function stringCopy, on the other hand,
uses an assertion to ensure that, if the target string is too long, the
program always fails in an observable manner.

1 /**
2 * Worse than broken: Are you feeling lucky?
3 */
4
5 #include <assert.h>
6
7
char before[ ] = "=Before=";
8
char middle[ ] = "Middle";
9
char after[ ] = "=After=";
10
11 void show() {
12
printf("%s\n%s\n%s\n", before, middle, after);
13 }
14
15 void stringCopy(char *target, const char *source, int howBi
16
17 int main(int argc, char *argv) {
18
show();
19
strcpy(middle, "Muddled"); /* Fault, but may not fail */
20
show();
21
strncpy(middle, "Muddled", sizeof(middle)); /* Fault, may
22
show();
23
stringCopy(middle, "Muddled",sizeof(middle)); /* Guarante
24
show();
25 }
26
27 /* Sensitive version of strncpy; can be counted on to fail
28 * in an observable way EVERY time the source is too large
29 * for the target, unlike the standard strncpy or strcpy.
30 */
31 void stringCopy(char *target, const char *source, int howBi
32
assert(strlen(source) < howBig);
33
strcpy(target, source);
34 }

Figure 3.1: Standard C functions strcpy and strncpy may or may

not fail when the source string is too long. The procedure
stringCopy is sensitive: It is guaranteed to fail in an observable
way if the source string is too long.
The sensitivity principle says that we should try to make these faults

easier to detect by making them cause failure more often. It can be

applied in three main ways: at the design level, changing the way in
which the program fails; at the analysis and testing level, choosing a
technique more reliable with respect to the property of interest; and
at the environment level, choosing a technique that reduces the
impact of external factors on the results.
Replacing strcpy and strncpy with stringCopy in the program of Figure 3.1 is a simple
example of application of the sensitivity principle in design. Run-time array bounds checking
in many programming languages (including Java but not C or C++) is an example of the
sensitivity principle applied at the language level. A variety of tools and replacements for the
standard memory management library are available to enhance sensitivity to memory
allocation and reference faults in C and C++.
The fail-fast property of Java iterators is another application of the sensitivity principle. A
Java iterator provides a way of accessing each item in a collection data structure. Without
the fail-fast property, modifying the collection while iterating over it could lead to unexpected
and arbitrary results, and failure might occur rarely and be hard to detect and diagnose. A
fail-fast iterator has the property that an immediate and observable failure (throwing
ConcurrentModificationException) occurs when the illegal modification occurs. Although failfast behavior is not guaranteed if the update occurs in a different thread, a fail-fast iterator
is far more sensitive than an iterator without the fail-fast property.

So far, we have discussed the sensitivity principle applied to design

and code: always privilege design and code solutions that lead to
consistent behavior, that is, such that fault occurrence does not
depend on uncontrolled execution conditions that may mask faults,
thus resulting in random failures. The sensitivity principle can also
be applied to test and analysis techniques. In this case, we privilege
techniques that cause faults to consistently manifest in failures.
Deadlock and race conditions in concurrent systems may depend on
the relative speed of execution of the different threads or processes,
and a race condition may lead to an observable failure only under
rare conditions. Testing a concurrent system on a single
configuration may fail to reveal deadlocks and race conditions.
Repeating the tests with different configurations and system loads
may help, but it is difficult to predict or control the circumstances
under which failure occurs. We may observe that testing is not
sensitive enough for revealing deadlocks and race conditions, and
we may substitute other techniques that are more sensitive and less

dependent on factors outside the developers' and testers' control.

Model checking and reachability analysis techniques are limited in
the scope of the faults they can detect, but they are very sensitive
to this particular class of faults, having the advantage that they
attain complete independence from any particular execution
environment by systematically exploring all possible interleavings of
processes.
Test adequacy criteria identify partitions of the input domain of the
unit under test that must be sampled by test suites. For example,
the statement coverage criterion requires each statement to be
exercised at least once, that is, it groups inputs according to the
statements they execute. Reliable criteria require that inputs
belonging to the same class produce the same test results: They all
fail or they all succeed. When this happens, we can infer the
correctness of a program with respect to the a whole class of inputs
from a single execution. Unfortunately, general reliable criteria do
not exist[1].
Code inspection can reveal many subtle faults. However, inspection teams may produce
completely different results depending on the cohesion of the team, the discipline of the
inspectors, and their knowledge of the application domain and the design technique. The
use of detailed checklists and a disciplined review process may reduce the influence of
external factors, such as teamwork attitude, inspectors' discipline, and domain knowledge,
thus increasing the predictability of the results of inspection. In this case, sensitivity is
applied to reduce the influence of external factors.

Similarly, skilled test designers can derive excellent test suites, but
the quality of the test suites depends on the mood of the designers.
Systematic testing criteria may not do better than skilled test
designers, but they can reduce the influence of external factors,
such as the tester's mood.
[1]Existence of a general, reliable test coverage criterion would allow

us to prove the equivalence of programs. Readers interested in this

topic will find more information in Chapter 9.

3.2 Redundancy
Redundancy is the opposite of independence. If one part of a software artifact
(program, design document, etc.) constrains the content of another, then they are not
entirely independent, and it is possible to check them for consistency.
The concept and definition of redundancy are taken from information theory. In
communication, redundancy can be introduced into messages in the form of errordetecting and error-correcting codes to guard against transmission errors. In software
test and analysis, we wish to detect faults that could lead to differences between
intended behavior and actual behavior, so the most valuable form of redundancy is in the
form of an explicit, redundant statement of intent.
Where redundancy can be introduced or exploited with an automatic, algorithmic check
for consistency, it has the advantage of being much cheaper and more thorough than
dynamic testing or manual inspection. Static type checking is a classic application of this
principle: The type declaration is a statement of intent that is at least partly redundant
with the use of a variable in the source code. The type declaration constrains other parts
of the code, so a consistency check (type check) can be applied.
An important trend in the evolution of programming languages is introduction of additional
ways to declare intent and automatically check for consistency. For example, Java
enforces rules about explicitly declaring each exception that can be thrown by a method.
Checkable redundancy is not limited to program source code, nor is it something that
can be introduced only by programming language designers. For example, software
design tools typically provide ways to check consistency between different design views
or artifacts. One can also intentionally introduce redundancy in other software artifacts,
even those that are not entirely formal. For example, one might introduce rules quite
analogous to type declarations for semistructured requirements specification documents,
and thereby enable automatic checks for consistency and some limited kinds of
completeness.
When redundancy is already present - as between a software specification document
and source code - then the remaining challenge is to make sure the information is
represented in a way that facilitates cheap, thorough consistency checks. Checks that
can be implemented by automatic tools are usually preferable, but there is value even in
organizing information to make inconsistency easier to spot in manual inspection.
Of course, one cannot always obtain cheap, thorough checks of source code and other
documents. Sometimes redundancy is exploited instead with run-time checks. Defensive
programming, explicit run-time checks for conditions that should always be true if the
program is executing correctly, is another application of redundancy in programming.

3.3 Restriction
When there are no acceptably cheap and effective ways to check a property,
sometimes one can change the problem by checking a different, more restrictive
property or by limiting the check to a smaller, more restrictive class of programs.
Consider the problem of ensuring that each variable is initialized before it is used, on
every execution. Simple as the property is, it is not possible for a compiler or analysis
tool to precisely determine whether it holds. See the program in Figure 3.2 for an
illustration. Can the variable k ever be uninitialized the first time i is added to it? If
someCondition(0) always returns true, then k will be initialized to zero on the first time
through the loop, before k is incremented, so perhaps there is no potential for a run-time
error - but method someCondition could be arbitrarily complex and might even depend
on some condition in the environment. Java's solution to this problem is to enforce a
stricter, simpler condition: A program is not permitted to have any syntactic control paths
on which an uninitialized reference could occur, regardless of whether those paths could
actually be executed. The program in Figure 3.2 has such a path, so the Java compiler
rejects it.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

/** A trivial method with a potentially uninitialized v

* Maybe someCondition(0) is always true, and therefor
* always initialized before use ... but it's impossib
* general, to know for sure. Java rejects the method.
*/
static void questionable() {
int k;
for (int i=0; i < 10; ++i) {
if (someCondition(i)) {
k=0;
} else {
k+=i;
}
}
System.out.println(k);
}
}

Figure 3.2: Can the variable k ever be uninitialized the first time i is added to it? The
property is undecidable, so Java enforces a simpler, stricter property.
Java's rule for initialization before use is a program source code restriction that enables
precise, efficient checking of a simple but important property by the compiler. The

choice of programming language(s) for a project may entail a number of such

restrictions that impact test and analysis. Additional restrictions may be imposed in the
form of programming standards (e.g., restricting the use of type casts or pointer
arithmetic in C), or by tools in a development environment. Other forms of restriction can
apply to architectural and detailed design.
Consider, for example, the problem of ensuring that a transaction consisting of a
sequence of accesses to a complex data structure by one process appears to the
outside world as if it had occurred atomically, rather than interleaved with transactions of
other processes. This property is called serializability: The end result of a set of such
transactions should appear as if they were applied in some serial order, even if they
didn't.
One way to ensure serializability is to make the transactions really serial (e.g., by putting
the whole sequence of operations in each transaction within a Java synchronized block),
but that approach may incur unacceptable performance penalties. One would like to
allow interleaving of transactions that don't interfere, while still ensuring the appearance
of atomic access, and one can devise a variety of locking and versioning techniques to
achieve this. Unfortunately, checking directly to determine whether the serializability
property has been achieved is very expensive at run-time, and precisely checking
whether it holds on all possible executions is impossible. Fortunately, the problem
becomes much easier if we impose a particular locking or versioning scheme on the
program at design time. Then the problem becomes one of proving, on the one hand,
that the particular concurrency control protocol has the desired property, and then
checking that the program obeys the protocol. Database researchers have completed
the first step, and some of the published and well-known concurrency control protocols
are trivial to check at run-time and simple enough that (with some modest additional
restrictions) they can be checked even by source code analysis.
From the above examples it should be clear that the restriction principle is useful mainly
during design and specification; it can seldom be applied post hoc on a complete
software product. In other words, restriction is mainly a principle to be applied in design
for test. Often it can be applied not only to solve a single problem (like detecting
potential access of uninitialized variables, or nonserializable execution of transactions)
but also at a more general, architectural level to simplify a whole set of analysis
problems.
Stateless component interfaces are an example of restriction applied at the architectural
level. An interface is stateless if each service request (method call, remote procedure
call, message send and reply) is independent of all others; that is, the service does not
"remember" anything about previous requests. Stateless interfaces are far easier to test
because the correctness of each service request and response can be checked
independently, rather than considering all their possible sequences or interleavings. A
famous example of simplifying component interfaces by making them stateless is the

Hypertext Transport Protocol (HTTP) 1.0 of the World-Wide-Web, which made Web
servers not only much simpler and more robust but also much easier to test.

3.4 Partition
Partition, often also known as "divide and conquer," is a general engineering principle.
Dividing a complex problem into subproblems to be attacked and solved independently is
probably the most common human problem-solving strategy. Software engineering in
particular applies this principle in many different forms and at almost all development
levels, from early requirements specifications to code and maintenance. Analysis and
testing are no exception: the partition principle is widely used and exploited.
Partitioning can be applied both at process and technique levels. At the process level,
we divide complex activities into sets of simple activities that can be attacked
independently. For example, testing is usually divided into unit, integration, subsystem,
and system testing. In this way, we can focus on different sources of faults at different
steps, and at each step, we can take advantage of the results of the former steps. For
instance, we can use units that have been tested as stubs for integration testing. Some
static analysis techniques likewise follow the modular structure of the software system
to divide an analysis problem into smaller steps.
Many static analysis techniques first construct a model of a system and then analyze the
model. In this way they divide the overall analysis into two subtasks: first simplify the
system to make the proof of the desired properties feasible and then prove the property
with respect to the simplified model. The question "Does this program have the desired
property?" is decomposed into two questions, "Does this model have the desired
property?" and "Is this an accurate model of the program?"
Since it is not possible to execute the program with every conceivable input, systematic
testing strategies must identify a finite number of classes of test cases to execute.
Whether the classes are derived from specifications (functional testing) or from program
structure (structural testing), the process of enumerating test obligations proceeds by
dividing the sources of information into significant elements (clauses or special values
identifiable in specifications, statements or paths in programs), and creating test cases
that cover each such element or certain combinations of elements.

3.5 Visibility
Visibility means the ability to measure progress or status against goals. In software
engineering, one encounters the visibility principle mainly in the form of process visibility,
and then mainly in the form of schedule visibility: ability to judge the state of development
against a project schedule. Quality process visibility also applies to measuring achieved
(or predicted) quality against quality goals. The principle of visibility involves setting goals
that can be assessed as well as devising methods to assess their realization.
Visibility is closely related to observability, the ability to extract useful information from a
software artifact. The architectural design and build plan of a system determines what
will be observable at each stage of development, which in turn largely determines the
visibility of progress against goals at that stage.
A variety of simple techniques can be used to improve observability. For example, it is
no accident that important Internet protocols like HTTP and SMTP (Simple Mail
Transport Protocol, used by Internet mail servers) are based on the exchange of simple
textual commands. The choice of simple, human-readable text rather than a more
compact binary encoding has a small cost in performance and a large payoff in
observability, including making construction of test drivers and oracles much simpler.
Use of human-readable and human-editable files is likewise advisable wherever the
performance cost is acceptable.
A variant of observability through direct use of simple text encodings is providing readers
and writers to convert between other data structures and simple, human- readable and
editable text. For example, when designing classes that implement a complex data
structure, designing and implementing also a translation from a simple text format to the
internal structure, and vice versa, will often pay back handsomely in both ad hoc and
systematic testing. For similar reasons it is often useful to design and implement an
equality check for objects, even when it is not necessary to the functionality of the
software product.

3.6 Feedback
Feedback is another classic engineering principle that applies to analysis and testing.
Feedback applies both to the process itself (process improvement) and to individual
techniques (e.g., using test histories to prioritize regression testing).
Systematic inspection and walkthrough derive part of their success from feedback.
Participants in inspection are guided by checklists, and checklists are revised and
refined based on experience. New checklist items may be derived from root cause
analysis, analyzing previously observed failures to identify the initial errors that lead to
them.

Summary
Principles constitute the core of a discipline. They form the basis of methods,
techniques, methodologies and tools. They permit understanding, comparing, evaluating
and extending different approaches, and they constitute the lasting basis of knowledge
of a discipline.
The six principles described in this chapter are
Sensitivity: better to fail every time than sometimes,
Redundancy: making intentions explicit,
Restriction: making the problem easier,
Partition: divide and conquer,
Visibility: making information accessible, and
Feedback: applying lessons from experience in process and techniques.
Principles are identified heuristically by searching for a common denominator of
techniques that apply to various problems and exploit different methods, sometimes
borrowing ideas from other disciplines, sometimes observing recurrent phenomena.
Potential principles are validated by finding existing and new techniques that exploit the
underlying ideas. Generality and usefulness of principles become evident only with time.
The initial list of principles proposed in this chapter is certainly incomplete. Readers are
invited to validate the proposed principles and identify additional principles.

Further Reading
Analysis and testing is a relatively new discipline. To our knowledge, the principles
underlying analysis and testing have not been discussed in the literature previously.

Some of the principles advocated in this chapter are shared with other software
engineering disciplines and are discussed in many books. A good introduction to
software engineering principles is the third chapter of Ghezzi, Jazayeri, and Mandrioli's
book on software engineering [GJM02].

Exercises
Indicate which principles guided the following choices:
1. Use an externally readable format also for internal files, when possible.
2. Collect and analyze data about faults revealed and removed from the
code.

3.1

3. Separate test and debugging activities; that is, separate the design and
execution of test cases to reveal failures (test) from the localization and
removal of the corresponding faults (debugging).
4. Distinguish test case design from execution.
5. Produce complete fault reports.
6. Use information from test case design to improve requirements and
design specifications.
7. Provide interfaces for fully inspecting the internal state of a class.

A simple mechanism for augmenting fault tolerance consists of replicating

3.2 computation and comparing the obtained results. Can we consider redundancy for
fault tolerance an application of the redundancy principle?
A system safety specification describes prohibited behaviors (what the system
3.3 must never do). Explain how specified safety properties can be viewed as an
implementation of the redundancy principle.
Process visibility can be increased by extracting information about the progress of
3.4 the process. Indicate some information that can be easily produced to increase
process visibility.

Chapter 4: Test and Analysis Activities Within a Software

Process
Dependability and other qualities of software are not ingredients that can be added in a
final step before delivery. Rather, software quality results from a whole set of
interdependent activities, among which analysis and testing are necessary but far from
sufficient. And while one often hears of a testing "phase" in software development, as if
testing were a distinct activity that occurred at a particular point in development, one
should not confuse this flurry of test execution with the whole process of software test
and analysis any more than one would confuse program compilation with programming.
Testing and analysis activities occur throughout the development and evolution of
software systems, from early in requirements engineering through delivery and
subsequent evolution. Quality depends on every part of the software process, not only
on software analysis and testing; no amount of testing and analysis can make up for
poor quality arising from other activities. On the other hand, an essential feature of
software processes that produce high-quality products is that software test and analysis
is thoroughly integrated and not an afterthought.

4.1 The Quality Process

One can identify particular activities and responsibilities in a software development
process that are focused primarily on ensuring adequate dependability of the software
product, much as one can identify other activities and responsibilities concerned primarily
with project schedule or with product usability. It is convenient to group these quality
assurance activities under the rubric "quality process," although we must also recognize
that quality is intertwined with and inseparable from other facets of the overall process.
Like other parts of an overall software process, the quality process provides a
framework for selecting and arranging activities aimed at a particular goal, while also
considering interactions and trade-offs with other important goals. All software
development activities reflect constraints and trade-offs, and quality activities are no
exception. For example, high dependability is usually in tension with time to market, and
in most cases it is better to achieve a reasonably high degree of dependability on a tight
schedule than to achieve ultra-high dependability on a much longer schedule, although
the opposite is true in some domains (e.g., certain medical devices).
The quality process should be structured for completeness, timeliness, and costeffectiveness. Completeness means that appropriate activities are planned to detect
each important class of faults. What the important classes of faults are depends on the
application domain, the organization, and the technologies employed (e.g., memory
leaks are an important class of faults for C++ programs, but seldom for Java programs).
Timeliness means that faults are detected at a point of high leverage, which in practice
almost always means that they are detected as early as possible. Cost-effectiveness
means that, subject to the constraints of completeness and timeliness, one chooses
activities depending on their cost as well as their effectiveness. Cost must be considered
over the whole development cycle and product life, so the dominant factor is likely to be
the cost of repeating an activity through many change cycles.
Activities that one would typically consider as being in the domain of quality assurance or
quality improvement, that is, activities whose primary goal is to prevent or detect faults,
intertwine and interact with other activities carried out by members of a software
development team. For example, architectural design of a software system has an
enormous impact on the test and analysis approaches that will be feasible and on their
cost. A precise, relatively formal architectural model may form the basis for several
static analyses of the model itself and of the consistency between the model and its
implementation, while another architecture may be inadequate for static analysis and, if
insufficiently precise, of little help even in forming an integration test plan.
The intertwining and mutual impact of quality activities on other development activities
suggests that it would be foolish to put off quality activities until late in a project. The
effects run not only from other development activities to quality activities but also in the
other direction. For example, early test planning during requirements engineering
typically clarifies and improves requirements specifications. Developing a test plan during

architectural design may suggest structures and interfaces that not only facilitate testing
earlier in development, but also make key interfaces simpler and more precisely defined.
There is also another reason for carrying out quality activities at the earliest opportunity
and for preferring earlier to later activities when either could serve to detect the same
fault: The single best predictor of the cost of repairing a software defect is the time
between its introduction and its detection. A defect introduced in coding is far cheaper to
repair during unit test than later during integration or system test, and most expensive if
it is detected by a user of the fielded system. A defect introduced during requirements
engineering (e.g., an ambiguous requirement) is relatively cheap to repair at that stage,
but may be hugely expensive if it is only uncovered by a dispute about the results of a
system acceptance test.

4.2 Planning and Monitoring

Process visibility is a key factor in software process in general, and
software quality processes in particular. A process is visible to the
extent that one can answer the question, "How does our progress
compare to our plan?" Typically, schedule visibility is a main
emphasis in process design ("Are we on schedule? How far ahead or
behind?"), but in software quality process an equal emphasis is
needed on progress against quality goals. If one cannot gain
confidence in the quality of the software system long before it
reaches final testing, the quality process has not achieved adequate
visibility.
A well-designed quality process balances several activities across the
whole development process, selecting and arranging them to be as
cost-effective as possible, and to improve early visibility. Visibility is
particularly challenging and is one reason (among several) that
quality activities are usually placed as early in a software process as
possible. For example, one designs test cases at the earliest
opportunity (not "just in time") and uses both automated and
manual static analysis techniques on software artifacts that are
produced before actual code.
Early visibility also motivates the use of "proxy" measures, that is, use of quantifiable
attributes that are not identical to the properties that one really wishes to measure, but that
have the advantage of being measurable earlier in development. For example, we know that
the number of faults in design or code is not a true measure of reliability. Nonetheless, one
may count faults uncovered in design inspections as an early indicator of potential quality
problems, because the alternative of waiting to receive a more accurate estimate from
reliability testing is unacceptable.

Quality goals can be achieved only through careful planning of

activities that are matched to the identified objectives. Planning is
integral to the quality process and is elaborated and revised through
the whole project. It encompasses both an overall strategy for test
and analysis, and more detailed test plans.
The overall analysis and test strategy identifies company- or project-wide standards that
must be satisfied: procedures for obtaining quality certificates required for certain classes
of products, techniques and tools that must be used, and documents that must be

produced. Some companies develop and certify procedures following international

standards such as ISO 9000 or SEI Capability Maturity Model, which require detailed
documentation and management of analysis and test activities and well-defined phases,
documents, techniques, and tools. A&T strategies are described in detail in Chapter 20, and
a sample strategy document for the Chipmunk Web presence is given in Chapter 24.
The initial build plan for Chipmunk Web-based purchasing functionality includes an analysis
and test plan. A complete analysis and test plan is a comprehensive description of the
quality process and includes several items: It indicates objectives and scope of the test and
analysis activities; it describes documents and other items that must be available for
performing the planned activities, integrating the quality process with the software
development process; it identifies items to be tested, thus allowing for simple completeness
checks and detailed planning; it distinguishes features to be tested from those not to be
tested; it selects analysis and test activities that are considered essential for success of the
quality process; and finally it identifies the staff involved in analysis and testing and their
respective and mutual responsibilities.
The final analysis and test plan includes additional information that illustrates constraints,
pass and fail criteria, schedule, deliverables, hardware and software requirements, risks,
and contingencies. Constraints indicate deadlines and limits that may be derived from the
hardware and software implementation of the system under analysis and the tools available
for analysis and testing. Pass and fail criteria indicate when a test or analysis activity
succeeds or fails, thus supporting monitoring of the quality process. The schedule describes
the individual tasks to be performed and provides a feasible schedule. Deliverables specify
which documents, scaffolding and test cases must be produced, and indicate the quality
expected from such deliverables. Hardware, environment and tool requirements indicate the
support needed to perform the scheduled activities. The risk and contingency plan identifies
the possible problems and provides recovery actions to avoid major failures. The test plan
is discussed in more detail in Chapter 20.

4.3 Quality Goals

Process visibility requires a clear specification of goals, and in the case of quality
process visibility this includes a careful distinction among dependability qualities. A team
that does not have a clear idea of the difference between reliability and robustness, for
example, or of their relative importance in a project, has little chance of attaining either.
Goals must be further refined into a clear and reasonable set of objectives. If an
organization claims that nothing less than 100% reliability will suffice, it is not setting an
ambitious objective. Rather, it is setting no objective at all, and choosing not to make
reasoned trade-off decisions or to balance limited resources across various activities. It
is, in effect, abrogating responsibility for effective quality planning, and leaving trade-offs
among cost, schedule, and quality to an arbitrary, ad hoc decision based on deadline
and budget alone.
The relative importance of qualities and their relation to other project objectives varies.
Time-to-market may be the most important property for a mass market product,
usability may be more prominent for a Web based application, and safety may be the
overriding requirement for a life-critical system.
Product qualities are the goals of software quality engineering, and process qualities are
means to achieve those goals. For example, development processes with a high degree
of visibility are necessary for creation of highly dependable products. The process goals
with which software quality engineering is directly concerned are often on the "cost" side
of the ledger. For example, we might have to weigh stringent reliability objectives against
their impact on time-to-market, or seek ways to improve time-to-market without
adversely impacting robustness.
Software product qualities can be divided into those that are directly visible to a client
and those that primarily affect the software development organization. Reliability, for
example, is directly visible to the client. Maintainability primarily affects the development
organization, although its consequences may indirectly affect the client as well, for
example, by increasing the time between product releases. Properties that are directly
visible to users of a software product, such as dependability, latency, usability, and
throughput, are called external properties. Properties that are not directly visible to end
users, such as maintainability, reusability, and traceability, are called internal properties,
even when their impact on the software development and evolution processes may
indirectly affect users.
The external properties of software can ultimately be divided into dependability (does
the software do what it is intended to do?) and usefulness. There is no precise
dependability way to distinguish these, but a rule of thumb is that when software is not
dependable, we say it has a fault, or a defect, or (most often) a bug, resulting in an
undesirable behavior or failure.

It is quite possible to build systems that are very reliable, relatively free from usefulness
hazards, and completely useless. They may be unbearably slow, or have terrible user
interfaces and unfathomable documentation, or they may be missing several crucial
features. How should these properties be considered in software quality? One answer is
that they are not part of quality at all unless they have been explicitly specified, since
quality is the presence of specified properties. However, a company whose products are
rejected by its customers will take little comfort in knowing that, by some definitions,
they were high-quality products.
We can do better by considering quality as fulfillment of required and desired properties,
as distinguished from specified properties. For example, even if a client does not
explicitly specify the required performance of a system, there is always some level of
performance that is required to be useful.
One of the most critical tasks in software quality analysis is making desired properties
explicit, since properties that remain unspecified (even informally) are very likely to
surface unpleasantly when it is discovered that they are not met. In many cases these
implicit requirements can not only be made explicit, but also made sufficiently precise
that they can be made part of dependability or reliability. For example, while it is better
to explicitly recognize usability as a requirement than to leave it implicit, it is better yet to
augment[1] usability requirements with specific interface standards, so that a deviation
from the standards is recognized as a defect.
[1]Interface

standards augment, rather than replace, usability requirements because

conformance to the standards is not sufficient assurance that the requirement is met.
This is the same relation that other specifications have to the user requirements they are
intended to fulfill. In general, verifying conformance to specifications does not replace
validating satisfaction of requirements.

4.4 Dependability Properties

The simplest of the dependability properties is correctness: A program or system is
correct if it is consistent with its specification. By definition, a specification divides all
possible system behaviors[2] into two classes, successes (or correct executions) and
failures. All of the possible behaviors of a correct system are successes.
A program cannot be mostly correct or somewhat correct or 30% correct. It is
absolutely correct on all possible behaviors, or else it is not correct. It is very easy to
achieve correctness, since every program is correct with respect to some (very bad)
specification. Achieving correctness with respect to a useful specification, on the other
hand, is seldom practical for nontrivial systems. Therefore, while correctness may be a
noble goal, we are often interested in assessing some more achievable level of
dependability.
Reliability is a statistical approximation to correctness, in the sense that 100% reliability
is indistinguishable from correctness. Roughly speaking, reliability is a measure of the
likelihood of correct function for some "unit" of behavior, which could be a single use or
program execution or a period of time. Like correctness, reliability is relative to a
specification (which determines whether a unit of behavior is counted as a success or
failure). Unlike correctness, reliability is also relative to a particular usage profile. The
same program can be more or less reliable depending on how it is used.
Particular measures of reliability can be used for different units of execution and different
ways of counting success and failure. Availability is an appropriate measure when a
failure has some duration in time. For example, a failure of a network router may make it
impossible to use some functions of a local area network until the service is restored;
between initial failure and restoration we say the router is "down" or "unavailable." The
availability of the router is the time in which the system is "up" (providing normal service)
as a fraction of total time. Thus, a network router that averages 1 hour of down time in
each 24-hour period would have an availability of 2324, or 95.8%.
Mean time between failures (MTBF) is yet another measure of reliability, also using time
as the unit of execution. The hypothetical network switch that typically fails once in a 24hour period and takes about an hour to recover has a mean time between failures of 23
hours. Note that availability does not distinguish between two failures of 30 minutes each
and one failure lasting an hour, while MTBF does.
The definitions of correctness and reliability have (at least) two major weaknesses. First,
since the success or failure of an execution is relative to a specification, they are only as
strong as the specification. Second, they make no distinction between a failure that is a
minor annoyance and a failure that results in catastrophe. These are simplifying
assumptions that we accept for the sake of precision, but in some circumstances particularly, but not only, for critical systems - it is important to consider dependability

properties that are less dependent on specification and that do distinguish among
failures depending on severity.
Software safety is an extension of the well-established field of system safety into
software. Safety is concerned with preventing certain undesirable behaviors, called
hazards. It is quite explicitly not concerned with achieving any useful behavior apart from
whatever functionality is needed to prevent hazards. Software safety is typically a
concern in "critical" systems such as avionics and medical systems, but the basic
principles apply to any system in which particularly undesirable behaviors can be
distinguished from run-of-the-mill failure. For example, while it is annoying when a word
processor crashes, it is much more annoying if it irrecoverably corrupts document files.
The developers of a word processor might consider safety with respect to the hazard of
file corruption separately from reliability with respect to the complete functional
requirements for the word processor.
Just as correctness is meaningless without a specification of allowed behaviors, safety
is meaningless without a specification of hazards to be prevented, and in practice the
first step of safety analysis is always finding and classifying hazards. Typically, hazards
are associated with some system in which the software is embedded (e.g., the medical
device), rather than the software alone. The distinguishing feature of safety is that it is
concerned only with these hazards, and not with other aspects of correct functioning.
The concept of safety is perhaps easier to grasp with familiar physical systems. For
example, lawn-mowers in the United States are equipped with an interlock device,
sometimes called a "dead-man switch." If this switch is not actively held by the operator,
the engine shuts off. The dead-man switch does not contribute in any way to cutting
grass; its sole purpose is to prevent the operator from reaching into the mower blades
while the engine runs.
One is tempted to say that safety is an aspect of correctness, because a good system
specification would rule out hazards. However, safety is best considered as a quality
distinct from correctness and reliability for two reasons. First, by focusing on a few
hazards and ignoring other functionality, a separate safety specification can be much
simpler than a complete system specification, and therefore easier to verify. To put it
another way, while a good system specification should rule out hazards, we cannot be
confident that either specifications or our attempts to verify systems are good enough to
provide the degree of assurance we require for hazard avoidance. Second, even if the
safety specification were redundant with regard to the full system specification, it is
important because (by definition) we regard avoidance of hazards as more crucial than
satisfying other parts of the system specification.
Correctness and reliability are contingent upon normal operating conditions. It is not
reasonable to expect a word processing program to save changes normally when the
file does not fit in storage, or to expect a database to continue to operate normally when

the computer loses power, or to expect a Web site to provide completely satisfactory
service to all visitors when the load is 100 times greater than the maximum for which it
was designed. Software that fails under these conditions, which violate the premises of
its design, may still be "correct" in the strict sense, yet the manner in which the software
fails is important. It is acceptable that the word processor fails to write the new file that
does not fit on disk, but unacceptable to also corrupt the previous version of the file in
the attempt. It is acceptable for the database system to cease to function when the
power is cut, but unacceptable for it to leave the database in a corrupt state. And it is
usually preferable for the Web system to turn away some arriving users rather than
becoming too slow for all, or crashing. Software that gracefully degrades or fails "softly"
outside its normal operating parameters is robust.
Software safety is a kind of robustness, but robustness is a more general notion that
concerns not only avoidance of hazards (e.g., data corruption) but also partial
functionality under unusual situations. Robustness, like safety, begins with explicit
consideration of unusual and undesirable situations, and should include augmenting
software specifications with appropriate responses to undesirable events.
Figure 4.1 illustrates the relation among dependability properties.
Quality analysis should be part of the feasibility study. The sidebar on page 47 shows an
excerpt of the feasibility study for the Chipmunk Web presence. The primary quality
requirements are stated in terms of dependability, usability, and security. Performance,
portability and interoperability are typically not primary concerns at this stage, but they
may come into play when dealing with other qualities.

Figure 4.1: Relation among dependability properties

[2]We

are simplifying matters somewhat by considering only specifications of behaviors.

A specification may also deal with other properties, such as the disk space required to
install the application. A system may thus also be "incorrect" if it violates one of these
static properties.

4.5 Analysis
Analysis techniques that do not involve actual execution of program source code play a
prominent role in overall software quality processes. Manual inspection techniques and
automated analyses can be applied at any development stage. They are particularly well
suited at the early stages of specifications and design, where the lack of executability of
many intermediate artifacts reduces the efficacy of testing.
Inspection, in particular, can be applied to essentially any document including
requirements documents, architectural and more detailed design documents, test plans
and test cases, and of course program source code. Inspection may also have
secondary benefits, such as spreading good practices and instilling shared standards of
quality. On the other hand, inspection takes a considerable amount of time and requires
meetings, which can become a scheduling bottleneck. Moreover, re-inspecting a
changed component can be as expensive as the initial inspection. Despite the versatility
of inspection, therefore, it is used primarily where other techniques are either
inapplicable or where other techniques do not provide sufficient coverage of common
faults.
Automated static analyses are more limited in applicability (e.g., they can be applied to
some formal representations of requirements models but not to natural language
documents), but are selected when available because substituting machine cycles for
human effort makes them particularly cost-effective. The cost advantage of automated
static analyses is diminished by the substantial effort required to formalize and properly
structure a model for analysis, but their application can be further motivated by their
ability to thoroughly check for particular classes of faults for which checking with other
techniques is very difficult or expensive. For example, finite state verification techniques
for concurrent systems requires construction and careful structuring of a formal design
model, and addresses only a particular family of faults (faulty synchronization structure).
Yet it is rapidly gaining acceptance in some application domains because that family of
faults is difficult to detect in manual inspection and resists detection through dynamic
testing.
Excerpt of Web Presence Feasibility Study
Purpose of this document
This document was prepared for the Chipmunk IT management team. It describes the
results of a feasibility study undertaken to advise Chipmunk corporate management
whether to embark on a substantial redevelopment effort to add online shopping
functionality to the Chipmunk Computers' Web presence.
Goals

The primary goal of a Web presence redevelopment is to add online shopping

facilities. Marketing estimates an increase of 15% over current direct sales within 24
months, and an additional 8% savings in direct sales support costs from shifting
telephone price inquiries to online price inquiries. [ ]
Architectural Requirements
The logical architecture will be divided into three distinct subsystems: human
interface, business logic, and supporting infrastructure. Each major subsystem must
be structured for phased development, with initial features delivered 6 months from
inception, full features at 12 months, and a planned revision at 18 months from project
inception. [ ]
Quality Requirements
Dependability With the introduction of direct sales and customer relationship
management functions, dependability of Chipmunk's Web services becomes
businesscritical. A critical core of functionality will be identified, isolated from less
critical functionality in design and implementation, and subjected to the highest level of
scrutiny. We estimate that this will be approximately 20% of new development and
revisions, and that the V&V costs for those portions will be approximately triple the
cost of V&V for noncritical development.
Usability The new Web presence will be, to a much greater extent than before, the
public face of Chipmunk Computers. [ ]
Security Introduction of online direct ordering and billing raises a number of security
issues. Some of these can be avoided initially by contracting with one of several
service companies that provide secure credit card transaction services. Nonetheless,
order tracking, customer relationship management, returns, and a number of other
functions that cannot be effectively outsourced raise significant security and privacy
issues. Identifying and isolating security concerns will add a significant but
manageable cost to design validation. [ ]

Sometimes the best aspects of manual inspection and automated static analysis can be
obtained by carefully decomposing properties to be checked. For example, suppose a
desired property of requirements documents is that each special term in the application
domain appear in a glossary of terms. This property is not directly amenable to an
automated static analysis, since current tools cannot distinguish meaningful domain
terms from other terms that have their ordinary meanings. The property can be checked
with manual inspection, but the process is tedious, expensive, and error-prone. A hybrid
approach can be applied if each domain term is marked in the text. Manually checking
that domain terms are marked is much faster and therefore less expensive than

manually looking each term up in the glossary, and marking the terms permits effective
automation of cross-checking with the glossary.

4.6 Testing
Despite the attractiveness of automated static analyses when they
are applicable, and despite the usefulness of manual inspections for
a variety of documents including but not limited to program source
code, dynamic testing remains a dominant technique. A closer look,
though, shows that dynamic testing is really divided into several
distinct activities that may occur at different points in a project.
Tests are executed when the corresponding code is available, but
testing activities start earlier, as soon as the artifacts required for
designing test case specifications are available. Thus, acceptance
and system test suites should be generated before integration and
unit test suites, even if executed in the opposite order.
Early test design has several advantages. Tests are specified
independently from code and when the corresponding software
specifications are fresh in the mind of analysts and developers,
facilitating review of test design. Moreover, test cases may highlight
inconsistencies and incompleteness in the corresponding software
specifications. Early design of test cases also allows for early repair
of software specifications, preventing specification faults from
propagating to later stages in development. Finally, programmers
may use test cases to illustrate and clarify the software
specifications, especially for errors and unexpected conditions.
No engineer would build a complex structure from parts that have not themselves been
subjected to quality control. Just as the "earlier is better" rule dictates using inspection to
reveal flaws in requirements and design before they are propagated to program code, the
same rule dictates module testing to uncover as many program faults as possible before
they are incorporated in larger subsystems of the product. At Chip- munk, developers are
expected to perform functional and structural module testing before a work assignment is
considered complete and added to the project baseline. The test driver and auxiliary files
are part of the work product and are expected to make reexecution of test cases, including
result checking, as simple and automatic as possible, since the same test cases will be
used over and over again as the product evolves.

4.7 Improving the Process

While the assembly-line, mass production industrial model is inappropriate for software,
which is at least partly custom-built, there is almost always some commonality among
projects undertaken by an organization over time. Confronted by similar problems,
developers tend to make the same kinds of errors over and over, and consequently the
same kinds of software faults are often encountered project after project. The quality
process, as well as the software development process as a whole, can be improved by
gathering, analyzing, and acting on data regarding faults and failures.
The goal of quality process improvement is to find cost-effective countermeasures for
classes of faults that are expensive because they occur frequently, or because the
failures they cause are expensive, or because, once detected, they are expensive to
repair. Countermeasures may be either prevention or detection and may involve either
quality assurance activities (e.g., improved checklists for design inspections) or other
aspects of software development (e.g., improved requirements specification methods).
The first part of a process improvement feedback loop, and often the most difficult to
implement, is gathering sufficiently complete and accurate raw data about faults and
failures. A main obstacle is that data gathered in one project goes mainly to benefit
other projects in the future and may seem to have little direct benefit for the current
project, much less to the persons asked to provide the raw data. It is therefore helpful to
integrate data collection as well as possible with other, normal development activities,
such as version and configuration control, project management, and bug tracking. It is
also essential to minimize extra effort. For example, if revision logs in the revision control
database can be associated with bug tracking records, then the time between checking
out a module and checking it back in might be taken as a rough guide to cost of repair.
Raw data on faults and failures must be aggregated into categories and prioritized.
Faults may be categorized along several dimensions, none of them perfect. Fortunately,
a flawless categorization is not necessary; all that is needed is some categorization
scheme that is sufficiently fine-grained and tends to aggregate faults with similar causes
and possible remedies, and that can be associated with at least rough estimates of
relative frequency and cost. A small number of categories - maybe just one or two - are
chosen for further study.
The analysis step consists of tracing several instances of an observed fault or failure
back to the human error from which it resulted, or even further to the factors that led to
that human error. The analysis also involves the reasons the fault was not detected and
eliminated earlier (e.g., how it slipped through various inspections and levels of testing).
This process is known as "root cause analysis," but the ultimate aim is for the most costeffective countermeasure, which is sometimes but not always the ultimate root cause.
For example, the persistence of security vulnerabilities through buffer overflow errors in
network applications may be attributed at least partly to widespread use of

programming languages with unconstrained pointers and without array bounds checking,
which may in turn be attributed to performance concerns and a requirement for
interoperability with a large body of legacy code. The countermeasure could involve
differences in programming methods (e.g., requiring use of certified "safe" libraries for
buffer management), or improvements to quality assurance activities (e.g., additions to
inspection checklists), or sometimes changes in management practices.

4.8 Organizational Factors

The quality process includes a wide variety of activities that require specific skills and
attitudes and may be performed by quality specialists or by software developers.
Planning the quality process involves not only resource management but also
identification and allocation of responsibilities.
A poor allocation of responsibilities can lead to major problems in which pursuit of
individual goals conflicts with overall project success. For example, splitting
responsibilities of development and quality-control between a development and a quality
team, and rewarding high productivity in terms of lines of code per person-month during
development may produce undesired results. The development team, not rewarded to
produce high-quality software, may attempt to maximize productivity to the detriment of
quality. The resources initially planned for quality assurance may not suffice if the initial
quality of code from the "very productive" development team is low. On the other hand,
combining development and quality control responsibilities in one undifferentiated team,
while avoiding the perverse incentive of divided responsibilities, can also have unintended
effects: As deadlines near, resources may be shifted from quality assurance to coding,
at the expense of product quality.
Conflicting considerations support both the separation of roles (e.g., recruiting quality
specialists), and the mobility of people and roles (e.g, rotating engineers between
development and testing tasks).
At Chipmunk, responsibility for delivery of the new Web presence is distributed among a
development team and a quality assurance team. Both teams are further articulated into
groups. The quality assurance team is divided into the analysis and testing group,
responsible for the dependability of the system, and the usability testing group,
responsible for usability. Responsibility for security issues is assigned to the
infrastructure development group, which relies partly on external consultants for final
tests based on external attack attempts.
Having distinct teams does not imply a simple division of all tasks between teams by
category. At Chipmunk, for example, specifications, design, and code are inspected by
mixed teams; scaffolding and oracles are designed by analysts and developers;
integration, system, acceptance, and regression tests are assigned to the test and
analysis team; unit tests are generated and executed by the developers; and coverage
is checked by the testing team before starting integration and system testing. A
specialist has been hired for analyzing faults and improving the process. The process
improvement specialist works incrementally while developing the system and proposes
improvements at each release.

Summary

Test and analysis activities are not a late phase of the development process, but rather
a wide set of activities that pervade the whole process. Designing a quality process with
a suitable blend of test and analysis activities for the specific application domain,
development environment, and quality goals is a challenge that requires skill and
experience.
A well-defined quality process must fulfill three main goals: improving the software
product during and after development, assessing its quality before delivery, and
improving the process within and across projects. These challenging goals can be
achieved by increasing visibility, scheduling activities as early as practical, and
monitoring results to adjust the process. Process visibility - that is, measuring and
comparing progress to objectives - is a key property of the overall development
process. Performing A&T activities early produces several benefits: It increases control
over the process, it hastens fault identification and reduces the costs of fault removal, it
provides data for incrementally tuning the development process, and it accelerates
product delivery. Feedback is the key to improving the process by identifying and
removing persistent errors and faults.

Further Reading
Qualities of software are discussed in many software engineering textbooks; the
discussion in Chapter 2 of Ghezzi, Jazayeri, and Mandrioli [GJM02] is particularly useful.
Process visibility is likewise described in software engineering textbooks, usually with an
emphasis on schedule. Musa [Mus04] describes a quality process oriented particularly
to establishing a quantifiable level of reliability based on models and testing before
release. Chillarege et al. [CBC+92] present principles for gathering and analyzing fault
data, with an emphasis on feedback within a single process but applicable also to quality
process improvement.

Exercises
We have stated that 100% reliability is indistinguishable from correctness, but they
are not quite identical. Under what circumstance might an incorrect program be
4.1 100% reliable? Hint: Recall that a program may be more or less reliable depending
on how it is used, but a program is either correct or incorrect regardless of usage.
We might measure the reliability of a network router as the fraction of all packets
4.2 that are correctly routed, or as the fraction of total service time in which packets
are correctly routed. When might these two measures be different?
If I am downloading a very large file over a slow modem, do I care more about the
4.3 availability of my internet service provider or its mean time between failures?

4.4 Can a system be correct and yet unsafe?

4.5 Under what circumstances can making a system more safe make it less reliable?
Software application domains can be characterized by the relative importance of
schedule (calendar time), total cost, and dependability. For example, while all three
4.6 are important for game software, schedule (shipping product in September to be
available for holiday purchases) has particular weight, while dependability can be
somewhat relaxed. Characterize a domain you are familiar with in these terms.
Consider responsiveness as a desirable property of an Internet chat program. The
informal requirement is that messages typed by each member of a chat session
4.7 appear instantaneously on the displays of other users. Refine this informal
requirement into a concrete specification that can be verified. Is anything lost in the
refinement?
4.8 Identify some correctness, robustness and safety properties of a word processor.

Part II: Basic Techniques

Chapter List
Chapter 5: Finite Models
Chapter 6: Dependence and Data Flow Models
Chapter 7: Symbolic Execution and Proof of Properties
Chapter 8: Finite State Verification

Chapter 5: Finite Models

From wind-tunnels to Navier-Stokes equations to circuit diagrams to finite-element models
of buildings, engineers in all fields of engineering construct and analyze models.
Fundamentally, modeling addresses two problems in engineering. First, analysis and test
cannot wait until the actual artifact is constructed, whether that artifact is a building or a
software system. Second, it is impractical to test the actual artifact as thoroughly as we
wish, whether that means subjecting it to all foreseeable hurricane and earthquake forces,
or to all possible program states and inputs. Models permit us to start analysis earlier and
repeat it as a design evolves, and allows us to apply analytic methods that cover a much
larger class of scenarios than we can explicitly test. Importantly, many of these analyses
may be automated.

This chapter presents some basic concepts in models of software

and some families of models that are used in a wide variety of
testing and analysis techniques. Several of the analysis and testing
techniques described in subsequent chapters use and specialize
these basic models. The fundamental concepts and trade-offs in the
design of models is necessary for a full understanding of those test
and analysis techniques, and is a foundation for devising new
techniques and models to solve domain-specific problems.

5.1 Overview
A model is a representation that is simpler than the artifact it represents but preserves (or
at least approximates) some important attributes of the actual artifact. Our concern in this
chapter is with models of program execution, and not with models of other (equally
important) attributes such as the effort required to develop the software or its usability. A
good model of (or, more precisely, a good class of models) must typically be:

Compact A model must be representable and manipulable in a

reasonably compact form. What is "reasonably compact" depends
largely on how the model will be used. Models intended for human
inspection and reasoning must be small enough to be
comprehensible. Models intended solely for automated analysis may
be far too large and complex for human comprehension, but must
still be sufficiently small or regular for computer processing.
Predictive A model used in analysis or design must represent some salient characteristics
of the modeled artifact well enough to distinguish between "good" and "bad" outcomes of
analysis, with respect to those characteristics.
Typically, no single model represents all characteristics well enough to be useful for all
kinds of analysis. One does not, for example, use the same model to predict airflow over an
aircraft fuselage and to design internal layout for efficient passenger loading and safe
emergency exit.

Semantically meaningful Beyond distinguishing between

predictions of success and failure, it is usually necessary to interpret
analysis results in a way that permits diagnosis of the causes of
failure. If a finite-element model of a building predicts collapse in a
category five hurricane, we want to know enough about that
collapse to suggest revisions to the design. Likewise, if a model of an
accounting system predicts a failure when used concurrently by
several clients, we need a description of that failure sufficient to
suggest possible revisions.
Sufficiently general Models intended for analysis of some important characteristic (e.g.,
withstanding earthquakes or concurrent operation by many clients) must be general enough
for practical use in the intended domain of application.
We may sometimes tolerate limits on design imposed by limitations of our modeling and
analysis techniques. For example, we may choose a conventional bridge design over a
novel design because we have confidence in analysis techniques for the former but not the

latter, and we may choose conventional concurrency control protocols over novel
approaches for the same reason. However, if a program analysis technique for C programs
is applicable only to programs without pointer variables, we are unlikely to find much use for
it.
Since design models are intended partly to aid in making and evaluating design decisions,
they should share these characteristics with models constructed primarily for analysis.
However, some kinds of models - notably the widely used UML design notations - are
designed primarily for human communication, with less attention to semantic meaning and
prediction.
Models are often used indirectly in evaluating an artifact. For example, some models are
not themselves analyzed, but are used to guide test case selection. In such cases, the
qualities of being predictive and semantically meaningful apply to the model together with
the analysis or testing technique applied to another artifact, typically the actual program or
system.

Graph Representations
We often use directed graphs to represent models of programs.
Usually we draw them as "box and arrow" diagrams, but to reason
about them it is important to understand that they have a welldefined mathematical meaning, which we review here.
A directed graph is composed of a set of nodes N and a relation E on the set (that is, a set
of ordered pairs), called the edges. It is conventional to draw the nodes as points or
shapes and to draw the edges as arrows. For example:

Typically, the nodes represent entities of some kind, such as procedures or classes or
regions of source code. The edges represent some relation among the entities. For
example, if we represent program control flow using a directed graph model, an edge (a,b)
would be interpreted as the statement "program region a can be directly followed by
program region b in program execution."
We can label nodes with the names or descriptions of the entities they represent. If nodes a
and b represent program regions containing assignment statements, we might draw the two
nodes and an edge (a,b) connecting them in this way:

Sometimes we draw a single diagram to represent more than one

directed graph, drawing the shared nodes only once. For example,
we might draw a single diagram in which we express both that class
B extends (is a subclass of) class A and that class B has a field that
is an object of type C. We can do this by drawing edges in the
"extends" relation differently than edges in the "includes" relation.

Drawings of graphs can be refined in many ways, for example, depicting some relations as
attributes rather than directed edges. Important as these presentation choices may be for
clear communication, only the underlying sets and relations matter for reasoning about
models.

5.2 Finite Abstractions of Behavior

A single program execution can be viewed as a sequence of states
alternating with actions (e.g., machine operations).[1] The possible
behaviors of a program are a set of such sequences. If we abstract
from the physical limits of a particular machine, for all but the most
trivial programs the set of possible execution sequences is infinite.
That whole set of states and transitions is called the state space of
the program. Models of program execution are abstractions of that
space.
States in the state space of program execution are related to states
in a finite model of execution by an abstraction function. Since an
abstraction function suppresses some details of program execution,
it lumps together execution states that differ with respect to the
suppressed details but are otherwise identical. Figure 5.1 illustrates
two effects of abstraction: The execution model is coarsened
(sequences of transitions are collapsed into fewer execution steps),
and nondeterminism is introduced (because information required to
make a deterministic choice is sacrificed).

Figure 5.1: Abstraction elides details of execution states and in so

doing may cause an abstract model execution state to represent
more than one concrete program execution state. In the illustration,
program state is represented by three attributes, each with two
possible values, drawn as light or dark circles. Abstract model states
retain the first two attributes and elide the third. The relation
between (1a) and (1b) illustrates coarsening of the execution model,

since the first and third program execution steps modify only the
omitted attribute. The relation between (2a) and (2b) illustrates
introduction of nondeterminism, because program execution states
with different successor states have been merged.
Finite models of program execution are inevitably imperfect. Collapsing the potentially
infinite states of actual execution into a finite number of representative model states
necessarily involves omitting some information. While one might hope that the omitted
information is irrelevant to the property one wishes to verify, this is seldom completely true.
In Figure 5.1, parts 2(a) and 2(b) illustrate how abstraction can cause a set of deterministic
transitions to be modeled by a nondeterministic choice among transitions, thus making the
analysis imprecise. This in turn can lead to "false alarms" in analysis of models.
[1]We put aside, for the moment, the possibility of parallel or

concurrent execution. Most but not all models of concurrent

execution reduce it to an equivalent serial execution in which
operation by different procedures are interleaved, but there also
exist models for which our treatment here is insufficient.

5.3 Control Flow Graphs

It is convenient and intuitive to construct models whose states are closely related to
locations in program source code. In general, we will associate an abstract state with a
whole region (that is, a set of locations) in a program. We know that program source
code is finite, so a model that associates a finite amount of information with each of a
finite number of program points or regions will also be finite.
Control flow of a single procedure or method can be represented as an intraprocedural
control flow graph, often abbreviated as control flow graph or CFG. The intraprocedural
control flow graph is a directed graph in which nodes represent regions of the source
code and directed edges represent the possibility that program execution proceeds from
the end of one region directly to the beginning of another, either through sequential
execution or by a branch. Figure 5.2 illustrates the representation of typical control flow
constructs in a control flow graph.

Figure 5.2: Building blocks for constructing intraprocedural control flow graphs. Other
control constructs are represented analogously. For example, the for construct of C,
C++, and Java is represented as if the initialization part appeared before a while
loop, with the increment part at the end of the while loop body.
In terms of program execution, we can say that a control flow graph model retains some
information about the program counter (the address of the next instruction to be
executed), and elides other information about program execution (e.g., the values of
variables). Since information that determines the outcome of conditional branches is
elided, the control flow graph represents not only possible program paths but also some
paths that cannot be executed. This corresponds to the introduction of nondeterminism
illustrated in Figure 5.1.
The nodes in a control flow graph could represent individual program statements, or
even individual machine operations, but it is desirable to make the graph model as
compact and simple as possible. Usually, therefore, nodes in a control flow graph model
of a program represent not a single point but rather a basic block, a maximal program

region with a single entry and single exit point.

A basic block typically coalesces adjacent, sequential statements of source code, but in
some cases a single syntactic program statement is broken across basic blocks to
model control flow within the statement. Figures 5.3 and 5.4 illustrate construction of a
control flow graph from a Java method. Note that a sequence of two statements within
the loop has been collapsed into a single basic block, but the for statement and the
complex predicate in the if statement have been broken across basic blocks to model
their internal flow of control.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

/**
* Remove/collapse multiple newline characters.
*
* @param String string to collapse newlines in.
* @return String
*/
public static String collapseNewlines(String argStr)
{
char last = argStr.charAt(0);
StringBuffer argBuf = new StringBuffer();

for (int cIdx=0; cIdx < argStr.length(); cIdx++

{
char ch = argStr.charAt(cIdx);
if (ch != '\n' || last != '\n')
{
argBuf.append(ch);
last = ch;
}
}
return argBuf.toString();
}

Figure 5.3: A Java method to collapse adjacent newline characters, from the
StringUtilities class of the Velocity project of the open source Apache project. (c)
2001 Apache Software Foundation, used with permission.

Figure 5.4: A control flow graph corresponding to the Java method in Figure 5.3. The
for statement and the predicate of the if statement have internal control flow
branches, so those statements are broken across basic blocks.
Some analysis algorithms are simplified by introducing a distinguished node to represent
procedure entry and another to represent procedure exit. When these distinguished start
and end nodes are used in a CFG, a directed edge leads from the start node to the
node representing the first executable block, and a directed edge from each procedure
exit (e.g., each return statement and the last sequential block in the program) to the
distinguished end node. Our practice will be to draw a start node identified with the
procedure or method signature, and to leave the end node implicit.
The intraprocedural control flow graph may be used directly to define thoroughness
criteria for testing (see Chapters 9 and 12). Often the control flow graph is used to
define another model, which in turn is used to define a thoroughness criterion. For
example, some criteria are defined by reference to linear code sequences and jumps
(LCSAJs), which are essentially subpaths of the control flow graph from one branch to
another. Figure 5.5 shows the LCSAJs derived from the control flow graph of Figure 5.4.
From Sequence of Basic Blocks To
entry b1 b2 b3

jX

entry b1 b2 b3 b4

jT

entry b1 b2 b3 b4 b5

jE

entry b1 b2 b3 b4 b5 b6 b7

jL

jX
jL

b8 return
b3 b4

jT

jL

b3 b4 b5

jE

jL

b3 b4 b5 b6 b7

jL

Figure 5.5: Linear code sequences and jumps (LCSAJs) corresponding to the Java
method in Figure 5.3 and the control flow graph in Figure 5.4. Note that proceeding to
the next sequential basic block is not considered a "jump" for purposes of identifying
LCSAJs.
For use in analysis, the control flow graph is usually augmented with other information.
For example, the data flow models described in the next chapter are constructed using a
CFG model augmented with information about the variables accessed and modified by
each program statement.
Not all control flow is represented explicitly in program text. For example, if an empty
string is passed to the collapseNewlines method of Figure 5.3, the exception
java.lang.StringIndexOutOfBoundsException will be thrown by String.charAt, and
execution of the method will be terminated. This could be represented in the CFG as a
directed edge to an exit node. However, if one includes such implicit control flow edges
for every possible exception (for example, an edge from each reference that might lead
to a null pointer exception), the CFG becomes rather unwieldy.
More fundamentally, it may not be simple or even possible to determine which of the
implicit control flow edges can actually be executed. We can reason about the call to
argStr.charAt(cIdx) within the body of the for loop and determine that cIdx must always
be within bounds, but we cannot reasonably expect an automated tool for extracting
control flow graphs to perform such inferences. Whether to include some or all implicit
control flow edges in a CFG representation therefore involves a trade-off between
possibly omitting some execution paths or representing many spurious paths. Which is
preferable depends on the uses to which the CFG representation will be put.
Even the representation of explicit control flow may differ depending on the uses to
which a model is put. In Figure 5.3, the for statement has been broken into its
constituent parts (initialization, comparison, and increment for next iteration), each of
which appears at a different point in the control flow. For some kinds of analysis, this
breakdown would serve no useful purpose. Similarly, a complex conditional expression in
Java or C is executed by "short-circuit" evaluation, so the single expression i > 0&&i < 10
can be broken across two basic blocks (the second test is not executed if the first
evaluates to false). If this fine level of execution detail is not relevant to an analysis, we
may choose to ignore short-circuit evaluation and treat the entire conditional expression
as if it were fully evaluated.

5.4 Call Graphs

The intraprocedural control flow graph represents possible execution paths through a
single procedure or method. interprocedural control flow can also be represented as a
directed graph. The most basic model is the call graph, in which nodes represent
procedures (methods, C functions, etc.) and edges represent the "calls" relation. For
example, a call graph representation of the program that includes the collapseNewlines
method above would include a node for StringUtils. collapseNewlines with a directed
edge to method String.charAt.
Call graph representations present many more design issues and trade-offs than
intraprocedural control flow graphs; consequently, there are many variations on the
basic call graph representation. For example, consider that in object-oriented languages,
method calls are typically made through object references and may be bound to
methods in different subclasses depending on the current binding of the object. A call
graph for programs in an object-oriented language might therefore represent the calls
relation to each of the possible methods to which a call might be dynamically bound.
More often, the call graph will explicitly represent only a call to the method in the
declared class of an object, but it will be part of a richer representation that includes
inheritance relations. Constructing an abstract model of executions in the course of
analysis will involve interpreting this richer structure.
Figure 5.6 illustrates overestimation of the calls relation due to dynamic dispatch. The
static call graph includes calls through dynamic bindings that never occur in execution.
A.foo() calls b.bar(), and b's declared class is C, and S inherits from C and overrides
bar(). The call graph includes a possible call from A.foo() to S.bar(). It might very well
include that call even if a more precise analysis could show that b can never actually be
bound to an object of subclass S, because in general such analysis is very expensive or
even impossible.

Figure 5.6: Overapproximation in a call graph. Although the method A.check() can
never actually call C.foo(), a typical call graph construction will include it as a possible
call.
If a call graph model represents different behaviors of a procedure depending on where
the procedure is called, we call it context-sensitive. For example, a context-sensitive
model of collapseNewlines might distinguish between one call in which the argument
string cannot possibly be empty, and another in which it could be. Contextsensitive
analyses can be more precise than context-insensitive analyses when the model includes
some additional information that is shared or passed among procedures. Information not
only about the immediate calling context, but about the entire chain of procedure calls
may be needed, as illustrated in Figure 5.7. In that case the cost of context-sensitive
analysis depends on the number of paths from the root (main program) to each lowest
level procedure. The number of paths can be exponentially larger than the number of
procedures, as illustrated in Figure 5.8.

Figure 5.7: The Java code above can be represented by the context-insensitive call
graph at left. However, to capture the fact that method depends never attempts to
store into a nonexistent array element, it is necessary to represent parameter values
that differ depending on the context in which depends is called, as in the contextsensitive call graph on the right.

Figure 5.8: The number of paths in a call graph - and therefore the number of calling
contexts in a context-sensitive analysis - can be exponentially larger than the number
of procedures, even without recursion.

1 public class C {
2
3
public static C cFactory(String kind) {
4
if (kind == "C") return new C();
5
if (kind == "S") return new S();
6
return null;
7
}
8
9
void foo() {
10
System.out.println("You called the parent's method");
11
}
12
13
public static void main(String args[]) {
14
(new A()).check();
15
}
16 }
17
18 class S extends C {
19
void foo() {
20
System.out.println("You called the child's method"
21
}
22 }
23
24 class A {
25
void check() {
26
C myC = C.cFactory("S");
27
myC.foo();
28
}
29 }
The Java compiler uses a typical call graph model to enforce the language rule that all
checked exceptions are either handled or declared in each method. The throws clauses
in a method declaration are provided by the programmer, but if they were not, they
would correspond exactly to the information that a context insensitive analysis of
exception propagation would associate with each procedure (which is why the compiler
can check for completeness and complain if the programmer omits an exception that can
be thrown).

5.5 Finite State Machines

Most of the models discussed above can be extracted from programs. Often, though,
models are constructed prior to or independent of source code, and serve as a kind of
specification of allowed behavior. Finite state machines of various kinds are particularly
widely used.
In its simplest form, a finite state machine (FSM) is a finite set of states and a set of
transitions among states, that is, a directed graph in which nodes represent program
states and edges represent operations that transform one program state into another.
Since there may be infinitely many program states, the finite set of state nodes must be
an abstraction of the concrete program states.
A transition from one state node a to another state node b denotes the possibility that a
concrete program state corresponding to a can be followed immediately by a concrete
program state corresponding to b. Usually we label the edge to indicate a program
operation, condition, or event associated with the transition. We may label transitions
with both an external event or a condition (what must happen or be true for the program
to make a corresponding state change) and with a program operation that can be
thought of as a "response" to the event. Such a finite state machine with event /
response labels on transitions is called a Mealy machine.
Figure 5.9 illustrates a specification for a converter among Dos, Unix, and Macintosh line
end conventions in the form of a Mealy machine. An "event" for this specification is
reading a character or encountering end-of-file. The possible input characters are
divided into four categories: carriage return, line feed, end-of-file, and everything else.
The states represent both program control points and some information that may be
stored in program variables.

Figure 5.9: Finite state machine (Mealy machine) description of line-end conversion
procedure, depicted as a state transition diagram (top) and as a state transition table
(bottom). An omission is obvious in the tabular representation, but easy to overlook in
the state transition diagram.
There are three kinds of correctness relations that we may reason about with respect to
finite state machine models, illustrated in Figure 5.10. The first is internal properties,
such as completeness and determinism. Second, the possible executions of a model,
described by paths through the FSM, may satisfy (or not) some desired property. Third,
the finite state machine model should accurately represent possible behaviors of the
program. Equivalently, the program should be a correct implementation of the finite state
machine model. We will consider each of the three kinds of correctness relation in turn
with respect to the FSM model of Figure 5.9.

Figure 5.10: Correctness relations for a finite state machine model. Consistency and
completeness are internal properties, independent of the program or a higher-level
specification. If, in addition to these internal properties, a model accurately represents
a program and satisfies a higher-level specification, then by definition the program
itself satisfies the higher-level specification.

1 public class Context {

2
public static void main(String args[]) {
3
Context c = new Context();
4
c.foo(3);
5
c.bar(17);
6
}
7
8
void foo(int n) {
9
int[] myArray = new int[n];
10
depends( myArray, 2) ;
11
}
12
13
void bar(int n) {
14
int[] myArray = new int[n];
15
depends( myArray, 16) ;
16
}
17
18
void depends( int[] a, int n) {
19
a[n] = 42;
20
}
21 }
Duals
In a control flow graph, nodes are associated with program regions, that is, with
blocks of program code that perform computation. In a finite state machine
representation, computations are associated with edges rather than nodes. This
difference is unimportant, because one can always exchange nodes with edges
without any loss of information, as illustrated by the following CFG and FSM
representations:

The graph on the right is called the dual of the graph on the left. Taking the dual of
the graph on the right, one obtains again the graph on the left.
The choice between associating nodes or edges with computations performed by a
program is only a matter of convention and convenience, and is not an important
difference between CFG and FSM models. In fact, aside from this minor difference in
customary presentation, the control flow graph is a particular kind of finite state
machine model in which the abstract states preserve some information about control
flow (program regions and their execution order) and elide all other information about
program state.

Many details are purposely omitted from the FSM model depicted in Figure 5.9, but it is
also incomplete in an undesirable way. Normally, we require a finite state machine
specification to be complete in the sense that it prescribes the allowed behavior(s) for
any possible sequence of inputs or events. For the line-end conversion specification, the
state transition diagram does not include a transition from state l on carriage return; that
is, it does not specify what the program should do if it encounters a carriage return
immediately after a line feed.
An alternative representation of finite state machines, including Mealy machines, is the
state transition table, also illustrated in Figure 5.9. There is one row in the transition
table for each state node and one column for each event or input. If the FSM is
complete and deterministic, there should be exactly one transition in each table entry.
Since this table is for a Mealy machine, the transition in each table entry indicates both
the next state and the response (e.g., d / emit means "emit and then proceed to state
d"). The omission of a transition from state l on a carriage return is glaringly obvious
when the state transition diagram is written in tabular form.

Analysis techniques for verifying properties of models will be presented in subsequent

chapters. For the current example, we illustrate with informal reasoning. The desired
property of this program and of its FSM models is that, for every possible execution, the
output file is identical to the input file except that each line ending is replaced by the lineend convention of the target format. Note, however, that the emit action is responsible
for emitting a line ending along with whatever text has been accumulated in a buffer.
While emit is usually triggered by a line ending in the input, it is also used to reproduce
any text in the buffer when end-of-file is reached. Thus, if the last line of an input file is
not terminated with a line ending, a line ending will nonetheless be added. This
discrepancy between specification and implementation is somewhat easier to detect by
examining the FSM model than by inspecting the program text.
To consider the third kind of correctness property, consistency between the model and
the implementation, we must define what it means for them to be consistent. The most
general way to define consistency is by considering behaviors. Given a way to compare
a sequence of program actions to a path through the finite state machine (which in
general will involve interpreting some program events and discarding others), a program
is consistent with a finite state machine model if every possible program execution
corresponds to a path through the model.[2]
Matching sequences of program actions to paths through a finite state machine model is
a useful notion of consistency if we are testing the program, but it is not a practical way
to reason about all possible program behaviors. For that kind of reasoning, it is more
helpful to also require a relation between states in the finite state machine model and
concrete program execution states.
It should be possible to describe the association of concrete program states with
abstract FSM states by an abstraction function. The abstraction function maps each
concrete program state to exactly one FSM state. Moreover, if some possible step op in
program execution takes the concrete program state from some state before to some
state after, then one of two conditions must apply: If the FSM model does not include
transitions corresponding to op, then program state before and program state after must
be associated with the same abstract state in the model. If the FSM does include
transitions corresponding to op, then there must be a corresponding transition in the
FSM model that connects program state before to program state after.
Using the second notion of conformance, we can reason about whether the
implementation of the line-end conversion program of Figure 5.11 is consistent with the
FSM of Figure 5.9 or Figure 5.12. Note that, in contrast to the control flow graph models
considered earlier, most of the interesting "state" is in the variables pos and atCR.We
posit that the abstraction function might be described by the following table:
Open table as spreadsheet
Abstract state

Concrete state

Lines atCR pos

e (Empty buffer) 212

w (Within line)

12

0 >0

l (Looking for LF)

12

d (Done)

35

1 /** Convert each line from standard input */

2 void transduce() {
3
4
#define BUFLEN 1000
5
char buf[BUFLEN]; /* Accumulate line into this buffer */
6
int pos=0;
/* Index for next character in buffer */
7
8
char inChar; /* Next character from input */
9
10 int atCR = 0; /* 0="within line", 1="optional DOS LF" */
11
12 while ((inChar = getchar()) != EOF ) {
13
switch (inChar) {
14
case LF:
15
if (atCR) { /* Optional DOS LF */
16
atCR = 0;
17
} else {
/* Encountered CR within line */
18
emit(buf, pos);
19
pos=0;
20
}
21
break;
22
case CR:
23
emit(buf, pos);
24
pos=0;
25
atCR = 1;
26
break;
27
default:
28
if (pos >= BUFLEN-2) fail("Buffer overflow");
29
buf[pos++] = inChar;
30
}/* switch */
31 }
32 if (pos > 0) {
33
emit(buf, pos);

34
35

}
}

Figure 5.11: Procedure to convert among Dos, Unix, and Macintosh line
ends.
Open table as spreadsheet
LF

CR

e e / emit l / emit

EOF

other

d/ w / append

w e / emit l / emit d / emit w / append

l

e/ l / emit

d/ w / append

Figure 5.12: Completed finite state machine (Mealy machine) description of line-end
conversion procedure, depicted as a state-transition table (bottom). The omitted
transition in Figure 5.9 has been added.
With this state abstraction function, we can check conformance between the source
code and each transition in the FSM. For example, the transition from state e to state l
is interpreted to mean that, if execution is at the head of the loop with pos equal to zero
and atCR also zero (corresponding to state e), and the next character encountered is a
carriage return, then the program should perform operations corresponding to the emit
action and then enter a state in which pos is zero and atCR is 1 (corresponding to state
l). It is easy to verify that this transition is implemented correctly. However, if we
examine the transition from state l to state w, we will discover that the code does not
correspond because the variable atCR is not reset to zero, as it should be. If the
program encounters a carriage return, then some text, and then a line feed, the line feed
will be discarded - a program fault.
The fault in the conversion program was actually detected by the authors through
testing, and not through manual verification of correspondence between each transition
and program source code. Making the abstraction function explicit was nonetheless
important to understanding the nature of the error and how to repair it.

Summary
Models play many of the same roles in software development as in engineering of other
kinds of artifacts. Models must be much simpler than the artifacts they describe, but
must preserve enough essential detail to be useful in making choices. For models of
software execution, this means that a model must abstract away enough detail to
represent the potentially infinite set of program execution states by a finite and suitably
compact set of model states.

Some models, such as control flow graphs and call graphs, can be extracted from
programs. The key trade-off for these extracted models is precision (retaining enough
information to be predictive) versus the cost of producing and storing the model. Other
models, including many finite state machine models, may be constructed before the
program they describe, and serve as a kind of intermediate-level specification of
intended behavior. These models can be related to both a higher-level specification of
intended behavior and the actual program they are intended to describe.
The relation between finite state models and programs is elaborated in Chapter 6.
Analysis of models, particularly those involving concurrent execution, is described further
in Chapter 8.

Further Reading
Finite state models of computation have been studied at least since the neural models of
McColloch and Pitts [MP43], and modern finite state models of programs remain close
to those introduced by Mealy [Mea55] and Moore [Moo56]. Lamport [Lam89] provides
the clearest and most accessible introduction the authors know regarding what a finite
state machine model "means" and what it means for a program to conform to it. Guttag
[Gut77] presents an early explication of the abstraction relation between a model and a
program, and why the abstraction function goes from concrete to abstract and not vice
versa. Finite state models have been particularly important in development of reasoning
and tools for concurrent (multi-threaded, parallel, and distributed) systems; Pezz,
Taylor, and Young [PTY95] overview finite models of concurrent programs.

Exercises
We construct large, complex software systems by breaking them into manageable
pieces. Likewise, models of software systems may be decomposed into more
5.1 manageable pieces. Briefly describe how the requirements of model compactness,
predictiveness, semantic meaningfulness, and sufficient generality apply to
approaches for modularizing models of programs. Give examples where possible.
Models are used in analysis, but construction of models from programs often
requires some form of analysis. Why bother, then? If one is performing an initial
analysis to construct a model to perform a subsequent analysis, why not just
5.2 merge the initial and subsequent analysis and dispense with defining and
constructing the model? For example, if one is analyzing Java code to construct a
call graph and class hierarchy that will be used to detect overriding of inherited
methods, why not just analyze the source code directly for method overriding?
Linear code sequence and jump (LCSAJ) makes a distinction between "sequential"
control flow and other control flow. Control flow graphs, on the other hand, make

5.3 no distinction between sequential and nonsequential control flow. Considering the
criterion of model predictiveness, is there a justification for this distinction?

What upper bound can you place on the number of basic blocks in a program,
5.4 relative to program size?
A directed graph is a set of nodes and a set of directed edges. A mathematical
relation is a set of ordered pairs.
1. If we consider a directed graph as a representation of a relation, can we
ever have two distinct edges from one node to another?
5.5

2. Each ordered pair in the relation corresponds to an edge in the graph. Is

the set of nodes superfluous? In what case might the set of nodes of a
directed graph be different from the set of nodes that appear in the
ordered pairs?

We have described how abstraction can introduce nondeterminism by discarding

some of the information needed to determine whether a particular state transition is
possible. In addition to introducing spurious transitions, abstraction can introduce
5.6 states that do not correspond to any possible program execution state - we say
such states are infeasible. Can we still have an abstraction function from concrete
states to model states if some of the model states are infeasible?
Can the number of basic blocks in the control flow graph representation of a
5.7 program ever be greater than the number of program statements? If so, how? If
not, why not?
[2]As

with other abstraction functions used in reasoning about programs, the mapping is
from concrete representation to abstract representation, and not from abstract to
concrete. This is because the mapping from concrete to abstract is many-to-one, and its
inverse is therefore not a mathematical function (which by definition maps each object in
the domain set into a single object in the range).

Chapter 6: Dependence and Data Flow Models

The control flow graph and state machine models introduced in the previous chapter
capture one aspect of the dependencies among parts of a program. They explicitly
represent control flow but deemphasize transmission of information through program
variables. Data flow models provide a complementary view, emphasizing and making
explicit relations involving transmission of information.
Models of data flow and dependence in software were originally developed in the field of
compiler construction, where they were (and still are) used to detect opportunities for
optimization. They also have many applications in software engineering, from testing to
refactoring to reverse engineering. In test and analysis, applications range from
selecting test cases based on dependence information (as described in Chapter 13) to
detecting anomalous patterns that indicate probable programming errors, such as uses
of potentially uninitialized values. Moreover, the basic algorithms used to construct data
flow models have even wider application and are of particular interest because they can
often be quite efficient in time and space.

6.1 Definition-Use Pairs

The most fundamental class of data flow model associates the point in a program where
a value is produced (called a "definition") with the points at which the value may be
accessed (called a "use"). Associations of definitions and uses fundamentally capture
the flow of information through a program, from input to output.
Definitions occur where variables are declared or initialized, assigned values, or received
as parameters, and in general at all statements that change the value of one or more
variables. Uses occur in expressions, conditional statements, parameter passing, return
statements, and in general in all statements whose execution extracts a value from a
variable. For example, in the standard greatest common divisor (GCD) algorithm of
Figure 6.1, line 1 contains a definition of parameters x and y, line 3 contains a use of
variable y, line 6 contains a use of variable tmp and a definition of variable y, and the
return in line 8 is a use of variable x.
1
2
3
4
5
6
7
8
9

public int gcd(int x, int y)

int tmp;
while (y != 0) {
tmp=x%y;
x=y;
y=tmp;
}
return x;
}

{ /* A: def x,y */
/* def tmp
/* B: use
/*
/*
/*
/* F: use

Figure 6.1: Java implementation of Euclid's algorithm for calculating the greatest
common denominator of two positive integers. The labels AF are provided to relate
statements in the source code to graph nodes in subsequent figures.
Each definition-use pair associates a definition of a variable (e.g., the assignment to y in
line 6) with a use of the same variable (e.g., the expression y!=0 in line 3). A single
definition can be paired with more than one use, and vice versa. For example, the
definition of variable y in line 6 is paired with a use in line 3 (in the loop test), as well as
additional uses in lines 4 and 5. The definition of x in line 5 is associated with uses in
lines 4 and 8.
A definition-use pair is formed only if there is a program path on which the value
assigned in the definition can reach the point of use without being overwritten by another
value. If there is another assignment to the same value on the path, we say that the first
definition is killed by the second. For example, the declaration of tmp in line 2 is not
paired with the use of tmp in line 6 because the definition at line 2 is killed by the

definition at line 4. A definition-clear path is a path from definition to use on which the
definition is not killed by another definition of the same variable. For example, with
reference to the node labels in Figure 6.2, path E,B,C,D is a definition-clear path from
the definition of y in line 6 (node E of the control flow graph) to the use of y in line 5
(node D). Path A,B,C,D,E is not a definition-clear path with respect to tmp because of
the intervening definition at node C.

Figure 6.2: Control flow graph of GCD method in Figure 6.1.

Definition-use pairs record a kind of program dependence, sometimes called direct data
dependence. These dependencies can be represented in the form of a graph, with a
directed edge for each definition-use pair. The data dependence graph representation of
the GCD method is illustrated in Figure 6.3 with nodes that are program statements.
Different levels of granularity are possible. For use in testing, nodes are typically basic
blocks. Compilers often use a finer-grained data dependence representation, at the level
of individual expressions and operations, to detect opportunities for performanceimproving transformations.

Figure 6.3: Data dependence graph of GCD method in Figure 6.1, with nodes for
statements corresponding to the control flow graph in Figure 6.2. Each directed edge
represents a direct data dependence, and the edge label indicates the variable that
transmits a value from the definition at the head of the edge to the use at the tail of
the edge.
The data dependence graph in Figure 6.3 captures only dependence through flow of
data. Dependence of the body of the loop on the predicate governing the loop is not
represented by data dependence alone. Control dependence can also be represented
with a graph, as in Figure 6.5, which shows the control dependencies for the GCD
method. The control dependence graph shows direct control dependencies, that is,
where execution of one statement controls whether another is executed. For example,
execution of the body of a loop or if statement depends on the result of a predicate.
Control dependence differs from the sequencing information captured in the control flow
graph. The control flow graph imposes a definite order on execution even when two
statements are logically independent and could be executed in either order with the
same results. If a statement is control- or data-dependent on another, then their order of
execution is not arbitrary. Program dependence representations typically include both
data dependence and control dependence information in a single graph with the two
kinds of information appearing as different kinds of edges among the same set of nodes.
A node in the control flow graph that is reached on every execution path from entry point
to exit is control dependent only on the entry point. For any other node N, reached on
some but not all execution paths, there is some branch that controls execution of N in the
sense that, depending on which way execution proceeds from the branch, execution of N
either does or does not become inevitable. It is this notion of control that control
dependence captures.
The notion of dominators in a rooted, directed graph can be used to make this intuitive
notion of "controlling decision" precise. Node M dominates node N if every path from the
root of the graph to N passes through M. A node will typically have many dominators,
but except for the root, there is a unique immediate dominator of node N, which is
closest to N on any path from the root and which is in turn dominated by all the other
dominators of N. Because each node (except the root) has a unique immediate
dominator, the immediate dominator relation forms a tree.

The point at which execution of a node becomes inevitable is related to paths from a
node to the end of execution - that is, to dominators that are calculated in the reverse of
the control flow graph, using a special "exit" node as the root. Dominators in this
direction are called post-dominators, and dominators in the normal direction of execution
can be called pre-dominators for clarity.
We can use post-dominators to give a more precise definition of control dependence.
Consider again a node N that is reached on some but not all execution paths. There
must be some node C with the following property: C has at least two successors in the
control flow graph (i.e., it represents a control flow decision); C is not post-dominated by
N (N is not already inevitable when C is reached); and there is a successor of C in the
control flow graph that is post-dominated by N. When these conditions are true, we say
node N is control-dependent on node C. Figure 6.4 illustrates the control dependence
calculation for one node in the GCD example, and Figure 6.5 shows the control
dependence relation for the method as a whole.

Figure 6.4: Calculating control dependence for node E in the control flow graph of the
GCD method. Nodes C, D, and E in the gray region are post-dominated by E; that is,
execution of E is inevitable in that region. Node B has successors both within and
outside the gray region, so it controls whether E is executed; thus E is
controldependent on B.

Figure 6.5: Control dependence tree of the GCD method. The loop test and the
return statement are reached on every possible execution path, so they are controldependent only on the entry point. The statements within the loop are controldependent on the loop test.

6.2 Data Flow Analysis

Definition-use pairs can be defined in terms of paths in the program control flow graph.
As we have seen in the former section, there is an association (d,u) between a definition
of variable v at d and a use of variable v at u if and only if there is at least one control
flow path from d to u with no intervening definition of v. We also say that definition vd
reaches u, and that vd is a reaching definition at u. If, on the other hand, a control flow
path passes through another definition e of the same variable v, we say that ve kills vd
at that point.
It would be possible to compute definition-use pairs by searching the control flow graph
for individual paths of the form described above. However, even if we consider only
loop-free paths, the number of paths in a graph can be exponentially larger than the
number of nodes and edges. Practical algorithms therefore cannot search every
individual path. Instead, they summarize the reaching definitions at a node over all the
paths reaching that node.
An efficient algorithm for computing reaching definitions (and several other properties, as
we will see below) is based on the way reaching definitions at one node are related to
reaching definitions at an adjacent node. Suppose we are calculating the reaching
definitions of node n, and there is an edge (p,n) from an immediate predecessor node p.
We observe:
If the predecessor node p can assign a value to variable v, then the definition vp
reaches n. We say the definition vp is generated at p.
If a definition vd of variable v reaches a predecessor node p, and if v is not
redefined at that node (in which case we say the vd is killed at that point), then
the definition is propagated on from p to n.
These observations can be stated in the form of an equation describing sets of reaching
definitions. For example, reaching definitions at node E in Figure 6.2 are those at node
D, except that D adds a definition of y and replaces (kills) an earlier definition of y:

This rule can be broken down into two parts to make it a little more intuitive and more
efficient to implement. The first part describes how node E receives values from its
predecessor D, and the second describes how it modifies those values for its
successors:

In this form, we can easily express what should happen at the head of the while loop
(node B in Figure 6.2), where values may be transmitted both from the beginning of the
procedure (node A) and through the end of the body of the loop (node E). The beginning
of the procedure (node A) is treated as an initial definition of parameters and local
variables. (If a local variable is declared but not initialized, it is treated as a definition to
the special value "uninitialized.")

In general, for any node n with predecessors pred(n),

Remarkably, the reaching definitions can be calculated simply and efficiently, first
initializing the reaching definitions at each node in the control flow graph to the empty
set, and then applying these equations repeatedly until the results stabilize. The
algorithm is given as pseudocode in Figure 6.6.
Algorithm Reaching definitions
Input:

A control flow graph G =(nodes,edges)

pred(n)= {m nodes | (m,n) edges}
succ(m)= {n nodes | (m,n) edges}
gen(n)= {vn} if variable v is defined at n, otherwise
kill(n)= all other definitions of v if v is defined a

Output: Reach(n)= the reaching definitions at node n

for n nodes loop
ReachOut(n)= {} ;
end loop;
workList = nodes ;
while (workList = {}) loop
// Take a node from worklist (e.g., pop from stack or queue)
n = any node in workList ;
workList = workList \{n} ;
oldVal = ReachOut(n) ;

// Apply flow equations, propagating values from predecessar

Reach(n)= mpred(n)ReachOut(m);
ReachOut(n)=(Reach(n) \ kill(n)) gen(n) ;
if ( ReachOut(n) = oldVal ) then
// Propagate changed value to successor nodes
workList = workList succ(n)
end if;
end loop;
Figure 6.6: An iterative work-list algorithm to compute reaching definitions by
applying each flow equation until the solution stabilizes.

6.3 Classic Analyses: Live and Avail

Reaching definition is a classic data flow analysis adapted from compiler construction to
applications in software testing and analysis. Other classical data flow analyses from
compiler construction can likewise be adapted. Moreover, they follow a common pattern
that can be used to devise a wide variety of additional analyses.
Available expressions is another classical data flow analysis, used in compiler
construction to determine when the value of a subexpression can be saved and reused
rather than recomputed. This is permissible when the value of the subexpression
remains unchanged regardless of the execution path from the first computation to the
second.
Available expressions can be defined in terms of paths in the control flow graph. An
expression is available at a point if, for all paths through the control flow graph from
procedure entry to that point, the expression has been computed and not subsequently
modified. We say an expression is generated (becomes available) where it is computed
and is killed (ceases to be available) when the value of any part of it changes (e.g.,
when a new value is assigned to a variable in the expression).
As with reaching definitions, we can obtain an efficient analysis by describing the relation
between the available expressions that reach a node in the control flow graph and those
at adjacent nodes. The expressions that become available at each node (the gen set)
and the expressions that change and cease to be available (the kill set) can be
computed simply, without consideration of control flow. Their propagation to a node from
its predecessors is described by a pair of set equations:

The similarity to the set equations for reaching definitions is striking. Both propagate
sets of values along the control flow graph in the direction of program execution (they
are forward analyses), and both combine sets propagated along different control flow
paths. However, reaching definitions combines propagated sets using set union, since a
definition can reach a use along any execution path. Available expressions combines
propagated sets using set intersection, since an expression is considered available at a
node only if it reaches that node along all possible execution paths. Thus we say that,
while reaching definitions is a forward, any-path analysis, available expressions is a
forward, all-paths analysis. A work-list algorithm to implement available expressions
analysis is nearly identical to that for reaching definitions, except for initialization and the
flow equations, as shown in Figure 6.7.

Algorithm Available expressions

Input:

A control flow graph G =(nodes,edges), with a distin

pred(n)= {m nodes | (m,n) edges}
succ(m)= {n nodes | (m,n) edges}
gen(n)= all expressions e computed at node n
kill(n)= expressions e computed anywhere, whose valu
kill(start) is the set of all e.
Output: Avail(n)= the available expressions at node n
for n nodes loop
AvailOut(n)= set of all e defined anywhere ;
end loop;
workList = nodes ;
while (workList = {}) loop
// Take a node from worklist (e.g., pop from stack or queue)
n = any node in workList ;
workList = workList \{n} ;
oldVal = AvailOut(n) ;
// Apply flow equations, propagating values from predeces
Avail(n)= mpred(n)AvailOut(m);
AvailOut(n)=(Avail(n) \ kill(n)) gen(n) ;
if ( AvailOut(n) oldVal ) then
// Propagate changes to successors
workList = workList succ(n)
end if;
end loop;
Figure 6.7: An iterative work-list algorithm for computing available
expressions.
Applications of a forward, all-paths analysis extend beyond the common subexpression
detection for which the Avail algorithm was originally developed. We can think of
available expressions as tokens that are propagated from where they are generated
through the control flow graph to points where they might be used. We obtain different
analyses by choosing tokens that represent some other property that becomes true (is
generated) at some points, may become false (be killed) at some other points, and is
evaluated (used) at certain points in the graph. By associating appropriate sets of
tokens in gen and kill sets for a node, we can evaluate other properties that fit the
pattern
"G occurs on all execution paths leading to U, and there is no intervening
occurrence of K between the last occurrence of G and U."

G, K, and U can be any events we care to check, so long as we can mark their
occurrences in a control flow graph.
An example problem of this kind is variable initialization. We noted in Chapter 3 that Java
requires a variable to be initialized before use on all execution paths. The analysis that
enforces this rule is an instance of Avail. The tokens propagated through the control flow
graph record which variables have been assigned initial values. Since there is no way to
"uninitialize" a variable in Java, the kill sets are empty. Figure 6.8 repeats the source
code of an example program from Chapter 3. The corresponding control flow graph is
shown with definitions and uses in Figure 6.9 and annotated with gen and kill sets for the
initialized variable check in Figure 6.10.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

/** A trivial method with a potentially uninitialized v

* Java compilers reject the program. The compiler uses
* data flow analysis to determine that there is a pote
* (syntactic) execution path on which k is used before
* has been assigned an initial value.
*/
static void questionable() {
int k;
for (int i=0; i < 10; ++i) {
if (someCondition(i)) {
k=0;
} else {
k+=i;
}
}
System.out.println(k);
}
}

Figure 6.8: Function questionable (repeated from Chapter 3) has a potentially

uninitialized variable, which the Java compiler can detect using data flow
analysis.

Figure 6.9: Control flow graph of the source code in Figure 6.8, annotated with
variable definitions and uses.

Figure 6.10: Control flow graph of the source code in Figure 6.8, annotated with gen
and kill sets for checking variable initialization using a forward, all-paths Avail
analysis. (Empty gen and kill sets are omitted.) The Avail set flowing from node G to
node C will be {i,k}, but the Avail set flowing from node B to node C is {i}. The allpaths analysis intersects these values, so the resulting Avail (C) is {i}. This value
propagates through nodes C and D to node F, which has a use of k as well as a
definition. Since k Avail(F), a possible use of an uninitialized variable is
detected.
Reaching definitions and available expressions are forward analyses; that is, they
propagate values in the direction of program execution. Given a control flow graph
model, it is just as easy to propagate values in the opposite direction, backward from

nodes that represent the next steps in computation. Backward analyses are useful for
determining what happens after an event of interest. Live variables is a backward
analysis that determines whether the value held in a variable may be subsequently used.
Because a variable is considered live if there is any possible execution path on which it is
used, a backward, any-path analysis is used.
A variable is live at a point in the control flow graph if, on some execution path, its
current value may be used before it is changed. Live variables analysis can be
expressed as set equations as before. Where Reach and Avail propagate values to a
node from its predecessors, Live propagates values from the successors of a node. The
gen sets are variables used at a node, and the kill sets are variables whose values are
replaced. Set union is used to combine values from adjacent nodes, since a variable is
live at a node if it is live at any of the succeeding nodes.

These set equations can be implemented using a work-list algorithm analogous to those
already shown for reaching definitions and available expressions, except that successor
edges are followed in place of predecessors and vice versa.
Like available expressions analysis, live variables analysis is of interest in testing and
analysis primarily as a pattern for recognizing properties of a certain form. A backward,
any-paths analysis allows us to check properties of the following form:
"After D occurs, there is at least one execution path on which G occurs with no
intervening occurrence of K."
Again we choose tokens that represent properties, using gen sets to mark occurrences
of G events (where a property becomes true) and kill sets to mark occurrences of K
events (where a property ceases to be true).
One application of live variables analysis is to recognize useless definitions, that is,
assigning a value that can never be used. A useless definition is not necessarily a
program error, but is often symptomatic of an error. In scripting languages like Perl and
Python, which do not require variables to be declared before use, a useless definition
typically indicates that a variable name has been misspelled, as in the common gateway
interface (CGI) script of Figure 6.11.
1
2
3

class SampleForm(FormData):
""" Used with Python cgi module
to hold and validate data

4
5
6
7
8
9
10
11
12
13
14
15

from HTML form """

fieldnames = ('name', 'email', 'comment')
# Trivial example of validation. The bug would be
# harder to see in a real validation method.
def validate(self):
valid = 1;
if self.name == "" : valid = 0
if self.email == "" :vald=0
if self.comment == "" : valid = 0
return valid

Figure 6.11: Part of a CGI program (Web form processing) in Python. The
misspelled variable name in the data validation method will be implicitly declared and
will not be rejected by the Python compiler or interpreter, which could allow invalid
data to be treated as valid. The classic live variables data flow analysis can show that
the assignment to valid is a useless definition, suggesting that the programmer
probably intended to assign the value to a different variable.
We have so far seen a forward, any-path analysis (reaching definitions), a forward, allpaths analysis (available definitions), and a backward, any-path analysis (live variables).
One might expect, therefore, to round out the repertoire of patterns with a backward,
all-paths analysis, and this is indeed possible. Since there is no classical name for this
combination, we will call it "inevitability" and use it for properties of the form
"After D occurs, G always occurs with no intervening occurrence of K"
or, informally,
"D inevitably leads to G before K"
Examples of inevitability checks might include ensuring that interrupts are reenabled after
executing an interrupt-handling routine in low-level code, files are closed after opening
them, and so on.

6.4 From Execution to Conservative Flow Analysis

Data flow analysis algorithms can be thought of as a kind of simulated execution. In
place of actual values, much smaller sets of possible values are maintained (e.g., a
single bit to indicate whether a particular variable has been initialized). All possible
execution paths are considered at once, but the number of different states is kept small
by associating just one summary state at each program point (node in the control flow
graph). Since the values obtained at a particular program point when it is reached along
one execution path may be different from those obtained on another execution path, the
summary state must combine the different values. Considering flow analysis in this light,
we can systematically derive a conservative flow analysis from a dynamic (that is, runtime) analysis.
As an example, consider the "taint-mode" analysis that is built into the programming
language Perl. Taint mode is used to prevent some kinds of program errors that result
from neglecting to fully validate data before using it, particularly where invalidated data
could present a security hazard. For example, if a Perl script wrote to a file whose name
was taken from a field in a Web form, a malicious user could provide a full path to
sensitive files. Taint mode detects and prevents use of the "tainted" Web form input in a
sensitive operation like opening a file. Other languages used in CGI scripts do not
provide such a monitoring function, but we will consider how an analogous static analysis
could be designed for a programming language like C.
When Perl is running in taint mode, it tracks the sources from which each variable value
was derived, and distinguishes between safe and tainted data. Tainted data is any input
(e.g., from a Web form) and any data derived from tainted data. For example, if a
tainted string is concatenated with a safe string, the result is a tainted string. One
exception is that pattern matching always returns safe strings, even when matching
against tainted data - this reflects the common Perl idiom in which pattern matching is
used to validate user input. Perl's taint mode will signal a program error if tainted data is
used in a potentially dangerous way (e.g., as a file name to be opened).
Perl monitors values dynamically, tagging data values and propagating the tags through
computation. Thus, it is entirely possible that a Perl script might run without errors in
testing, but an unanticipated execution path might trigger a taint mode program error in
production use. Suppose we want to perform a similar analysis, but instead of checking
whether "tainted" data is used unsafely on a particular execution, we want to ensure that
tainted data can never be used unsafely on any execution. We may also wish to perform
the analysis on a language like C, for which run-time tagging is not provided and would
be expensive to add. So, we can consider deriving a conservative, static analysis that is
like Perl's taint mode except that it considers all possible execution paths.
A data flow analysis for taint would be a forward, any-path analysis with tokens
representing tainted variables. The gen set at a program point would be a set containing

any variable that is assigned a tainted value at that point. Sets of tainted variables would
be propagated forward to a node from its predecessors, with set union where a node in
the control flow graph has more than one predecessor (e.g., the head of a loop).
There is one fundamental difference between such an analysis and the classic data flow
analyses we have seen so far: The gen and kill sets associated with a program point are
not constants. Whether or not the value assigned to a variable is tainted (and thus
whether the variable belongs in the gen set or in the kill set) depends on the set of
tainted variables at that program point, which will vary during the course of the analysis.
There is a kind of circularity here - the gen set and kill set depend on the set of tainted
variables, and the set of tainted variables may in turn depend on the gen and kill set.
Such circularities are common in defining flow analyses, and there is a standard
approach to determining whether they will make the analysis unsound. To convince
ourselves that the analysis is sound, we must show that the output values computed by
each flow equation are monotonically increasing functions of the input values. We will
say more precisely what "increasing" means below.
The determination of whether a computed value is tainted will be a simple function of the
set of tainted variables at a program point. For most operations of one or more
arguments, the output is tainted if any of the inputs are tainted. As in Perl, we may
designate one or a few operations (operations used to check an input value for validity)
as taint removers. These special operations always return an untainted value regardless
of their inputs.
Suppose we evaluate the taintedness of an expression with the input set of tainted
variables being {a,b}, and again with the input set of tainted variables being {a,b,c}. Even
without knowing what the expression is, we can say with certainty that if the expression
is tainted in the first evaluation, it must also be tainted in the second evaluation, in which
the set of tainted input variables is larger. This also means that adding elements to the
input tainted set can only add elements to the gen set for that point, or leave it the
same, and conversely the kill set can only grow smaller or stay the same. We say that
the computation of tainted variables at a point increases monotonically.
To be more precise, the monotonicity argument is made by arranging the possible
values in a lattice. In the sorts of flow analysis framework considered here, the lattice is
almost always made up of subsets of some set (the set of definitions, or the set of
tainted variables, etc.); this is called a powerset lattice because the powerset of set A is
the set of all subsets of A. The bottom element of the lattice is the empty set, the top is
the full set, and lattice elements are ordered by inclusion as in Figure 6.12. If we can
follow the arrows in a lattice from element x to element y (e.g., from {a} to {a,b,c}), then
we say y > x. A function f is monotonically increasing if

Figure 6.12: The powerset lattice of set {a,b,c}. The powerset contains all subsets of
the set and is ordered by set inclusion.
Not only are all of the individual flow equations for taintedness monotonic in this sense,
but in addition the function applied to merge values where control flow paths come
together is also monotonic:

If we have a set of data flow equations that is monotonic in this sense, and if we begin
by initializing all values to the bottom element of the lattice (the empty set in this case),
then we are assured that an iterative data flow analysis will converge on a unique
minimum solution to the flow equations.
The standard data flow analyses for reaching definitions, live variables, and available
expressions can all be justified in terms of powerset lattices. In the case of available
expressions, though, and also in the case of other all-paths analyses such as the one we
have called "inevitability," the lattice must be flipped over, with the empty set at the top
and the set of all variables or propositions at the bottom. (This is why we used the set of
all tokens, rather than the empty set, to initialize the Avail sets in Figure 6.7.)

6.5 Data Flow Analysis with Arrays and Pointers

The models and flow analyses described in the preceding section have been limited to
simple scalar variables in individual procedures. Arrays and pointers (including object
references and procedure arguments) introduce additional issues, because it is not
possible in general to determine whether two accesses refer to the same storage
location. For example, consider the following code fragment:
1
2

a[i] = 13;
k = a[j];

Are these two lines a definition-use pair? They are if the values of i and j are equal,
which might be true on some executions and not on others. A static analysis cannot, in
general, determine whether they are always, sometimes, or never equal, so a source of
imprecision is necessarily introduced into data flow analysis.
Pointers and object references introduce the same issue, often in less obvious ways.
Consider the following snippet:
1
2

a[2] = 42;
i = b[2];

It seems that there cannot possibly be a definition-use pair involving these two lines,
since they involve none of the same variables. However, arrays in Java are dynamically
allocated objects accessed through pointers. Pointers of any kind introduce the
possibility of aliasing, that is, of two different names referring to the same storage
location. For example, the two lines above might have been part of the following
program fragment:
1
2
3
4

int []a= new int[3];

int []b=a;
a[2] = 42;
i = b[2];

Here a and b are aliases, two different names for the same dynamically allocated array
object, and an assignment to part of a is also an assignment to part of b.
The same phenomenon, and worse, appears in languages with lower-level pointer
manipulation. Perhaps the most egregious example is pointer arithmetic in C:
1
2

p=&b;
*(p+i)=k;

It is impossible to know which variable is defined by the second line. Even if we know
the value of i, the result is dependent on how a particular compiler arranges variables in
memory.

Dynamic references and the potential for aliasing introduce uncertainty into data flow
analysis. In place of a definition or use of a single variable, we may have a potential
definition or use of a whole set of variables or locations that could be aliases of each
other. The proper treatment of this uncertainty depends on the use to which the analysis
will be put. For example, if we seek strong assurance that v is always initialized before it
is used, we may not wish to treat an assignment to a potential alias of v as initialization,
but we may wish to treat a use of a potential alias of v as a use of v.
A useful mental trick for thinking about treatment of aliases is to translate the uncertainty
introduced by aliasing into uncertainty introduced by control flow. After all, data flow
analysis already copes with uncertainty about which potential execution paths will
actually be taken; an infeasible path in the control flow graph may add elements to an
any-paths analysis or remove results from an all-paths analysis. It is usually appropriate
to treat uncertainty about aliasing consistently with uncertainty about control flow. For
example, considering again the first example of an ambiguous reference:
1
2

a[i] = 13;
k = a[j];

We can imagine replacing this by the equivalent code:

1
2
3
4
5
6

a[i] = 13;
if (i == j) {
k = a[i];
} else {
k = a[j];
}

In the (imaginary) transformed code, we could treat all array references as distinct,
because the possibility of aliasing is fully expressed in control flow. Now, if we are using
an any-path analysis like reaching definitions, the potential aliasing will result in creating
a definition-use pair. On the other hand, an assignment to a[j] would not kill a previous
assignment to a[i]. This suggests that, for an any-path analysis, gen sets should include
everything that might be referenced, but kill sets should include only what is definitely
referenced.
If we were using an all-paths analysis, like available expressions, we would obtain a
different result. Because the sets of available expressions are intersected where control
flow merges, a definition of a[i] would make only that expression, and none of its
potential aliases, available. On the other hand, an assignment to a[j] would kill a[i]. This
suggests that, for an all-paths analysis, gen sets should include only what is definitely
referenced, but kill sets should include all the possible aliases.
Even in analysis of a single procedure, the effect of other procedures must be
considered at least with respect to potential aliases. Consider, for example, this
fragment of a Java method:

1
2
3
4
5
6
7

public void transfer (CustInfo fromCust, CustInfo toCust)

PhoneNum fromHome = fromCust.gethomePhone();
PhoneNum fromWork = fromCust.getworkPhone();
PhoneNum toHome = toCust.gethomePhone();
PhoneNum toWork = toCust.getworkPhone();

We cannot determine whether the two arguments fromCust and toCust are references
to the same object without looking at the context in which this method is called.
Moreover, we cannot determine whether fromHome and fromWork are (or could be)
references to the same object without more information about how CustInfo objects are
treated elsewhere in the program.
Sometimes it is sufficient to treat all nonlocal information as unknown. For example, we
could treat the two CustInfo objects as potential aliases of each other, and similarly treat
the four PhoneNum objects as potential aliases. Sometimes, though, large sets of
aliases will result in analysis results that are so imprecise as to be useless. Therefore
data flow analysis is often preceded by an interprocedural analysis to calculate sets of
aliases or the locations that each pointer or reference can refer to.

6.6 Interprocedural Analysis

Most important program properties involve more than one procedure, and as mentioned
earlier, some interprocedural analysis (e.g., to detect potential aliases) is often required
as a prelude even to intraprocedural analysis. One might expect the interprocedural
analysis and models to be a natural extension of the intraprocedural analysis, following
procedure calls and returns like intraprocedural control flow. Unfortunately, this is
seldom a practical option.
If we were to extend data flow models by following control flow paths through procedure
calls and returns, using the control flow graph model and the call graph model together
in the obvious way, we would observe many spurious paths. Figure 6.13 illustrates the
problem: Procedure foo and procedure bar each make a call on procedure sub. When
procedure call and return are treated as if they were normal control flow, in addition to
the execution sequences (A,X,Y,B) and (C,X,Y,D), the combined graph contains the
impossible paths (A,X,Y,D) and (C,X,Y,B).

Figure 6.13: Spurious execution paths result when procedure calls and returns are
treated as normal edges in the control flow graph. The path (A,X,Y,D) appears in the
combined graph, but it does not correspond to an actual execution
order.
It is possible to represent procedure calls and returns precisely, for example by making
a copy of the called procedure for each point at which it is called. This would result in a
context-sensitive analysis. The shortcoming of context sensitive analysis was already
mentioned in the previous chapter: The number of different contexts in which a
procedure must be considered could be exponentially larger than the number of
procedures. In practice, a context-sensitive analysis can be practical for a small group of
closely related procedures (e.g., a single Java class), but is almost never a practical
option for a whole program.
Some interprocedural properties are quite independent of context and lend themselves
naturally to analysis in a hierarchical, piecemeal fashion. Such a hierarchical analysis can
be both precise and efficient. The analyses that are provided as part of normal
compilation are often of this sort. The unhandled exception analysis of Java is a good
example: Each procedure (method) is required to declare the exceptions that it may
throw without handling. If method M calls method N in the same or another class, and if
N can throw some exception, then M must either handle that exception or declare that it,

too, can throw the exception. This analysis is simple and efficient because, when
analyzing method M, the internal structure of N is irrelevant; only the results of the
analysis at N (which, in Java, is also part of the signature of N) are needed.
Two conditions are necessary to obtain an efficient, hierarchical analysis like the
exception analysis routinely carried out by Java compilers. First, the information needed
to analyze a calling procedure must be small: It must not be proportional either to the
size of the called procedure, or to the number of procedures that are directly or
indirectly called. Second, it is essential that information about the called procedure be
independent of the caller; that is, it must be context-independent. When these two
conditions are true, it is straightforward to develop an efficient analysis that works
upward from leaves of the call graph. (When there are cycles in the call graph from
recursive or mutually recursive procedures, an iterative approach similar to data flow
analysis algorithms can usually be devised.)
Unfortunately, not all important properties are amenable to hierarchical analysis.
Potential aliasing information, which is essential to data flow analysis even within
individual procedures, is one of those that are not. We have seen that potential aliasing
can depend in part on the arguments passed to a procedure, so it does not have the
context-independence property required for an efficient hierarchical analysis. For such
an analysis, additional sacrifices of precision must be made for the sake of efficiency.
Even when a property is context-dependent, an analysis for that property may be
context-insensitive, although the context-insensitive analysis will necessarily be less
precise as a consequence of discarding context information. At the extreme, a linear
time analysis can be obtained by discarding both context and control flow information.
Context- and flow-insensitive algorithms for pointer analysis typically treat each
statement of a program as a constraint. For example, on encountering an assignment
1

x=y;

where y is a pointer, such an algorithm simply notes that x may refer to any of the same
objects that y may refer to. References(x) References(y) is a constraint that is
completely independent of the order in which statements are executed. A procedure call,
in such an analysis, is just an assignment of values to arguments. Using efficient data
structures for merging sets, some analyzers can process hundreds of thousands of lines
of source code in a few seconds. The results are imprecise, but still much better than
the worst-case assumption that any two compatible pointers might refer to the same
object.
The best approach to interprocedural pointer analysis will often lie somewhere between
the astronomical expense of a precise, context- and flow-sensitive pointer analysis and
the imprecision of the fastest context- and flow-insensitive analyses. Unfortunately, there
is not one best algorithm or tool for all uses. In addition to context and flow sensitivity,

important design trade-offs include the granularity of modeling references (e.g., whether
individual fields of an object are distinguished) and the granularity of modeling the
program heap (that is, which allocated objects are distinguished from each other).

Summary
Data flow models are used widely in testing and analysis, and the data flow analysis
algorithms used for deriving data flow information can be adapted to additional uses.
The most fundamental model, complementary to models of control flow, represents the
ways values can flow from the points where they are defined (computed and stored) to
points where they are used.
Data flow analysis algorithms efficiently detect the presence of certain patterns in the
control flow graph. Each pattern involves some nodes that initiate the pattern and some
that conclude it, and some nodes that may interrupt it. The name "data flow analysis"
reflects the historical development of analyses for compilers, but patterns may be used
to detect other control flow patterns.
An any-path analysis determines whether there is any control flow path from the initiation
to the conclusion of a pattern without passing through an interruption. An all- paths
analysis determines whether every path from the initiation necessarily reaches a
concluding node without first passing through an interruption. Forward analyses check
for paths in the direction of execution, and backward analyses check for paths in the
opposite direction. The classic data flow algorithms can all be implemented using simple
work-list algorithms.
A limitation of data flow analysis, whether for the conventional purpose or to check other
properties, is that it cannot distinguish between a path that can actually be executed and
a path in the control flow graph that cannot be followed in any execution. A related
limitation is that it cannot always determine whether two names or expressions refer to
the same object.
Fully detailed data flow analysis is usually limited to individual procedures or a few
closely related procedures (e.g., a single class in an object-oriented program). Analyses
that span whole programs must resort to techniques that discard or summarize some
information about calling context, control flow, or both. If a property is independent of
calling context, a hierarchical analysis can be both precise and efficient. Potential
aliasing is a property for which calling context is significant. There is therefore a tradeoff between very fast but imprecise alias analysis techniques and more precise but much
more expensive techniques.

Further Reading
Data flow analysis techniques were originally developed for compilers, as a systematic

way to detect opportunities for code-improving transformations and to ensure that those
transformations would not introduce errors into programs (an all-too-common experience
with early optimizing compilers). The compiler construction literature remains an
important source of reference information for data flow analysis, and the classic "Dragon
Book" text [ASU86] is a good starting point.
Fosdick and Osterweil recognized the potential of data flow analysis to detect program
errors and anomalies that suggested the presence of errors more than two decades ago
[FO76]. While the classes of data flow anomaly detected by Fosdick and Osterweil's
system has largely been obviated by modern strongly typed programming languages,
they are still quite common in modern scripting and prototyping languages. Olender and
Osterweil later recognized that the power of data flow analysis algorithms for
recognizing execution patterns is not limited to properties of data flow, and developed a
system for specifying and checking general sequencing properties [OO90, OO92].
Interprocedural pointer analyses - either directly determining potential aliasing relations,
or deriving a "points-to" relation from which aliasing relations can be derived - remains
an area of active research. At one extreme of the cost-versus-precision spectrum of
analyses are completely context- and flow-insensitive analyses like those described by
Steensgaard [Ste96]. Many researchers have proposed refinements that obtain
significant gains in precision at small costs in efficiency. An important direction for future
work is obtaining acceptably precise analyses of a portion of a large program, either
because a whole program analysis cannot obtain sufficient precision at acceptable cost
or because modern software development practices (e.g., incorporating externally
developed components) mean that the whole program is never available in any case.
Rountev et al. present initial steps toward such analyses [RRL99]. A very readable
overview of the state of the art and current research directions (circa 2001) is provided
by Hind [Hin01].

Exercises
For a graph G =(N,V ) with a root r N, node m dominates node n if every path
from r to n passes through m. The root node is dominated only by itself.
The relation can be restated using flow equations.
1. When dominance is restated using flow equations, will it be stated in the
form of an any-path problem or an all-paths problem? Forward or
backward? What are the tokens to be propagated, and what are the gen
and kill sets?
2. Give a flow equation for Dom(n).
3. If the flow equation is solved using an iterative data flow analysis, what
should the set Dom(n) be initialized to at each node n?

4. Implement an iterative solver for the dominance relation in a programming

language of your choosing.
6.1

The first line of input to your program is an integer between 1 and 100
indicating the number k of nodes in the graph. Each subsequent line of
input will consist of two integers, m and n, representing an edge from
node m to node n. Node 0 designates the root, and all other nodes are
designated by integers between 0 and k 1. The end of the input is
signaled by the pseudo-edge (1,1).
The output of your program should be a sequences of lines, each
containing two integers separated by blanks. Each line represents one
edge of the Dom relation of the input graph.
5. The Dom relation itself is not a tree. The immediate dominators relation is
a tree. Write flow equations to calculate immediate dominators, and then
modify the program from part (d) to compute the immediate dominance
relation.

Write flow equations for inevitability, a backward, all-paths intraprocedural analysis.

6.2 Event (or program point) q is inevitable at program point p if every execution path
from p to a normal exit point passes through q.
The Java language automatically initializes fields of objects, in contrast to local
variables of methods that the programmer is responsible for initializing. Given what
6.3 you know of intra- and interprocedural data flow analysis, explain why the language
designers may have made these design choices.
Show the data and control dependence graphs for the binary search program of
6.4 Figure 7.1 on page 103.

Chapter 7: Symbolic Execution and Proof of

Properties

Overview
Symbolic execution builds predicates that characterize the conditions
under which execution paths can be taken and the effect of the
execution on program state. Extracting predicates through symbolic
execution is the essential bridge from the complexity of program
behavior to the simpler and more orderly world of logic. It finds
important applications in program analysis, in generating test data,
and in formal verification[1] (proofs) of program correctness.
Conditions under which a particular control flow path is taken can be
determined through symbolic execution. This is useful for identifying
infeasible program paths (those that can never be taken) and paths
that could be taken when they should not. It is fundamental to
generating test data to execute particular parts and paths in a
program.
Deriving a logical representation of the effect of execution is essential in methods that
compare a program's possible behavior to a formal specification. We have noted in earlier
chapters that proving the correctness of a program is seldom an achievable or useful goal.
Nonetheless the basic methods of formal verification, including symbolic execution, underpin
practical techniques in software analysis and testing. Symbolic execution and the techniques
of formal verification find use in several domains:
Rigorous proofs of properties of (small) critical subsystems, such as a safety kernel
of a medical device;

Formal verification of critical properties (e.g., security

properties) that are particularly resistant to dynamic testing;
Formal verification of algorithm descriptions and logical designs that are much less
complex than their implementations in program code.
More fundamentally, the techniques of formal reasoning are a conceptual foundation for a
variety of analysis techniques, ranging from informal reasoning about program behavior and
correctness to automated checks for certain classes of errors.
[1]Throughout this book we use the term verification in the broad

sense of checking whether a program or system is consistent with

some form of specification. The broad sense of verification includes,

for example, inspection techniques and program testing against

informally stated specifications. The term formal verification is used
in the scientific literature in a much narrower sense to denote
techniques that construct a mathematical proof of consistency
between some formal representation of a program or design and a
formal specification.

7.1 Symbolic State and Interpretation

Tracing execution is familiar to any programmer who has attempted to understand the
behavior of source code by simulating execution. For example, one might trace a single
statement in the binary search routine of Figure 7.1 as shown on the left side of Figure
7.2. One can just as easily use symbolic values like L and H in place of concrete values,
as shown on the right side of Figure 7.2. Tracing execution with symbolic values and
expressions is the basis of symbolic execution.

1
2 /** Binary search for key in sorted array dictKeys, returni
3 * corresponding value from dictValues or null if key does
4 * not appear in dictKeys. Standard binary search algorithm
5 * as described in any elementary text on data structures an
6 **/
7
8 char * binarySearch( char *key, char *dictKeys[ ], char *di
9
int dictSize) {
10
11
int low=0;
12
int high = dictSize - 1;
13
int mid;
14
int comparison;
15
16
while (high >=low) {
17
mid = (high + low) / 2;
18
comparison = strcmp( dictKeys[mid], key );
19
if (comparison < 0) {
20
/* dictKeys[mid] too small; look higher */
21
low=mid+1;
22
} else if ( comparison > 0) {
23
/* dictKeys[mid] too large; look lower */
24
high=mid-1;
25
} else {
26
/* found */
27
return dictValues[mid];
28
}
29
}
30
return 0; /* null means not found */
31 }
32

Figure 7.1: Binary search procedure.

Figure 7.2: Hand-tracing an execution step with concrete values (left) and symbolic
values (right).
When tracing execution with concrete values, it is clear enough what to do with a branch
statement, for example, an if or while test: The test predicate is evaluated with the
current values, and the appropriate branch is taken. If the values bound to variables are
symbolic expressions, however, both the True and False outcomes of the decision may
be possible. Execution can be traced through the branch in either direction, and
execution of the test is interpreted as adding a constraint to record the outcome. For
example, consider

Suppose the symbolic state after one loop iteration is

If we trace execution of the test assuming a True outcome (leading to a second iteration
of the loop), the loop condition becomes a constraint in the symbolic state immediately
after the while test:

Later, when we consider the branch assuming a False outcome of the test, the new
constraint is negated and becomes

or, equivalently,

.
Execution can proceed in this way down any path in the program. One can think of
"satisfying" the predicate by finding concrete values for the symbolic variables that make
it evaluate to True; this corresponds to finding data values that would force execution of
that program path. If no such satisfying values are possible, then that execution path
cannot be executed with any data values; we say it is an infeasible path.

7.2 Summary Information

If there were only a finite number of execution paths in a program,
then in principle a symbolic executor could trace each of them and
obtain a precise representation of a predicate that characterizes
each one. From even a few execution steps in the preceding small
example, one can see that the representation of program state will
quickly become unwieldy. Moreover, there are a potentially infinite
number of program execution paths to consider. An automated
symbolic executor can cope with much more complex symbolic
expressions than a human, but even an automated tool will not get
far with brute force evaluation of every program path.
Since the representation of program state is a logical predicate,
there is an alternative to keeping a complete representation of the
state at every point: a weaker predicate can always be substituted
for the complete representation. That is, if the representation of the
program state at some point in execution is P, and if W P, then
substituting W for P will result in a predicate that still correctly
describes the execution state, but with less precision. We call W a
summary of P.
Consider the computation of mid in line 17 of the binary search example from Figure 7.1. If
we are reasoning about the performance of binary search, the fact that the value of mid lies
halfway between the values of low and high is important, but if we are reasoning about
functional correctness it matters only that mid lies somewhere between them. Thus, if we
had low = L high = H mid = M, and if we could show L H, we could replace M =(L +
H)/2 by the weaker condition L M H.
Note that the weaker predicate L mid H is chosen based on what must be true for the
program to execute correctly. This is not information that can be derived automatically from
source code; it depends as well on our understanding of the code and our rationale for
believing it to be correct. A predicate stating what should be true at a given point can be
expressed in the form of an assertion. When we assert that predicate W is true at a point in
a program, we mark our intention both to verify it at that point (by showing that W is implied
by the predicates that describe the program state at that point) and to replace part of the
program state description P by W at that point.
One of the prices of weakening the predicate in this way will be that satisfying the predicate
is no longer sufficient to find data that forces the program execution along that path. If the
complete predicate P is replaced by a weaker predicate W, then test data that satisfies W

is necessary to execute the path, but it may not be sufficient. Showing that W cannot be
satisfied is still tantamount to showing that the execution path is infeasible.

7.3 Loops and Assertions

The number of execution paths through a program with one or more loops is potentially
infinite, or at least unimaginably huge. This may not matter for symbolic execution along
a single, relatively simple execution path. It becomes a major obstacle if symbolic
execution is used to reason about a path involving several iterations of a loop, or to
reason about all possible program executions.
To reason about program behavior in a loop, we can place within the loop an assertion
that states a predicate that is expected to be true each time execution reaches that
point. Such an assertion is called an invariant. Each time program execution reaches the
invariant assertion, we can weaken the description of program state. If the program
state is represented by P, and the assertion is W, we must first ascertain W P (the
assertion is satisfied along that path), and then we can substitute W for P.
Suppose every loop contained such an assertion, and suppose in addition there was an
assertion at the beginning of the program (perhaps just the trivial predicate True) and a
final assertion at the end. In that case, every possible execution path would consist of a
sequence of segments from one assertion to the next. The assertion at the beginning of
a segment is the precondition for that segment, and the assertion at the end of the
segment is the postcondition. If we were able to execute each such segment
independently, starting with only the precondition and then checking that the assertion at
the end of the segment is satisfied, we would have shown that every assertion is
satisfied on every possible program execution - that is, we would have verified correct
execution on an infinite number of program paths by verifying the finite number of
segments from which the paths are constructed.
We illustrate the technique by using assertions to check the logic of the binary search
algorithm implemented by the program in Figure 7.1. The first precondition and the final
postcondition serve as a specification of correct behavior as a kind of contract: If the
client ensures the precondition, the program will ensure the postcondition.
The binary search procedure depends on the array dictKeys being sorted. Thus we
might have a precondition assertion like the following:

Here we interpret s t for strings as indicating lexical order consistent with the C library
strcmp; that is, we assume that s t whenever strcmp(s,t) 0. For convenience we
will abbreviate the predicate above as sorted.
We can associate the following assertion with the while statement at line 16:

In other words, we assert that the key can appear only between low and high,ifit
appears anywhere in the array. We will abbreviate this condition as inrange.
Inrange must be true when we first reach the loop, because at that point the range
lowhigh is the same as 0size 1. For each path through the body of the loop, the
symbolic executor would begin with the invariant assertion above, and determine that it
is true again after following that path. We say the invariant is preserved.
While the inrange predicate should be true on each iteration, it is not the complete loop
invariant. The sorted predicate remains true and will be used in reasoning. In principle it
is also part of the invariant, although in informal reasoning we may not bother to write it
down repeatedly. The full invariant is therefore sorted inrange.
Let us consider the path from line 16 through line 21 and back to the loop test. We begin
by assuming that the loop invariant assertion holds at the beginning of the segment.
Where expressions in the invariant refer to program variables whose values may
change, they are replaced by symbols representing the initial values of those variables.
The variable bindings will be

We need not introduce symbols to represent the values of dictKeys, dictVals, key,or
size. Since those variables are not changed in the procedure, we can use the variable
names directly. The condition, instantiated with symbolic values, will be

Passing through the while test into the body of the loop adds the clause H L to this
condition. Execution of line 17 adds a binding of (H + L)/2 to variable mid, where x
is the integer obtained by rounding x toward zero. As we have discussed, this can be
simplified with an assertion so that the bindings and condition become

Tracing the execution path into the first branch of the if statement to line 21, we add the
constraint that strcmp(dictKeys[mid], key) returns a negative value, which we interpret
as meaning the probed entry is lexically less than the string value of the key. Thus we
arrive at the symbolic constraint

The assignment in line 21 then modifies a variable binding without otherwise disturbing
the conditions, giving us

Finally, we trace execution back to the while test at line 16. Now our obligation is to
show that the invariant still holds when instantiated with the changed set of variable
bindings. The sorted condition has not changed, and showing that it is still true is trivial.
The interesting part is the inrange predicate, which is instantiated with a new value for
low and thus becomes

Now the verification step is to show that this predicate is a logical consequence of the
predicate describing the program state. This step requires purely logical and
mathematical reasoning, and might be carried out either by a human or by a theoremproving tool. It no longer depends in any way upon the program. The task performed by
the symbolic executor is essentially to transform a question about a program (is the
invariant preserved on a particular path?) into a question of logic alone.
The path through the loop on which the probed key is too large, rather than too small,
proceeds similarly. The path on which the probed key matches the sought key returns
from the procedure, and our obligation there (trivial in this case) is to verify that the
contract of the procedure has been met.
The other exit from the procedure occurs when the loop terminates without locating a
matching key. The contract of the procedure is that it should return the null pointer
(represented in the C language by 0) only if the key appears nowhere in
dictKeys[0..size-1]. Since the null pointer is returned whenever the loop terminates, the
postcondition of the loop is that key is not present in dictKeys.

The loop invariant is used to show that the postcondition holds when the loop terminates.
What symbolic execution can verify immediately after a loop is that the invariant is true
but the loop test is false. Thus we have

Knowing that presence of the key in the array implies L H, and that in fact L > H, we
can conclude that the key is not present. Thus the postcondition is established, and the
procedure fulfills its contract by returning the null pointer in this case.
Finding and verifying a complete set of assertions, including an invariant assertion for
each loop, is difficult in practice. Even the small example above is rather tedious to verify
by hand. More realistic examples can be quite demanding even with the aid of symbolic
execution tools. If it were easy or could be fully automated, we might routinely use this
method to prove the correctness of programs. Writing down a full set of assertions
formally, and rigorously verifying them, is usually reserved for small and extremely
critical modules, but the basic approach we describe here can also be applied in a much
less formal manner and is quite useful in finding holes in an informal correctness
argument.

7.4 Compositional Reasoning

The binary search procedure is very simple. There is only one loop, containing a single if
statement. It was not difficult to reason about individual paths through the control flow. If
the procedure contained nested loops or more conditional branches, we could in
principle still proceed in that manner as long as each cycle in the control flow graph were
broken by at least one assertion. It would, however, be very difficult to think about
programs in this manner and to choose appropriate assertions. It is better if our
approach follows the hierarchical structure of the program, both at a small scale (e.g.,
control flow within a single procedure) and at larger scales (across multiple procedures,
classes, subsystems, etc.).
The steps for verifying the binary search procedure above already hint at a hierarchical
approach. The loop invariant was not placed just anywhere in the loop. We associated it
with the beginning of the loop so that we could follow a standard style of reasoning that
allows us to compose facts about individual pieces of a program to derive facts about
larger pieces. In this hierarchical or compositional style, the effect of any program block
is described by a Hoare triple:

The meaning of this triple is that if the program is in a state satisfying the precondition
pre at entry to the block, then after execution of the block it will be in a state satisfying
the postcondition post.
There are standard templates, or schemata, for reasoning with triples. In the previous
section we were following this schema for reasoning about while loops:

The formula above the line is the premise of an inference, and the formula below the line
is the conclusion. An inference rule states that if we can verify the premise, then we can
infer the conclusion. The premise of this inference rule says that the loop body preserves
invariant I: If the invariant I is true before the loop, and if the condition C governing the
loop is also true, then the invariant is established again after executing the loop body S.
The conclusion says that the loop as a whole takes the program from a state in which
the invariant is true to a state satisfying a postcondition composed of the invariant and
the negation of the loop condition.
The important characteristic of these rules is that they allow us to compose proofs about
small parts of the program into proofs about larger parts. The inference rule for while
allows us to take a triple about the body of a loop and infer a triple about the whole
loop. There are similar rules for building up triples describing other kinds of program

blocks. For example:

This style of reasoning essentially lets us summarize the effect of a block of program
code by a precondition and a postcondition. Most importantly, we can summarize the
effect of a whole procedure in the same way. The contract of the procedure is a
precondition (what the calling client is required to provide) and a postcondition (what the
called procedure promises to establish or return). Once we have characterized the
contract of a procedure in this way, we can use that contract wherever the procedure is
called. For example, we might summarize the effect of the binary search procedure this
way:

7.5 Reasoning about Data Structures and Classes

The contract of the binary search procedure can be specified in a relatively simple, selfcontained manner. Imagine, though, that it is part of a module that maintains a dictionary
structure (e.g., the relation between postal codes and the nearest airport with air-freight
capability). In that case, the responsibility for keeping the table in sorted order would
belong to the module itself, and not to its clients. If implemented in a modern objectoriented language, the data structure would not even be visible to the client, but would
rather be encapsulated within a class.
Modular reasoning about programs must follow the modular structure of program
designs, with the same layering of design secrets. We must have ways of specifying
contracts for classes and other modules that do not expose what the program
constructs encapsulate. Fortunately there are well-developed methods for modular
specification and verification of modules that encapsulate data structures.
A data structure module provides a collection of procedures (methods) whose
specifications are strongly interrelated. Their contracts with clients are specified by
relating them to an abstract model of their (encapsulated) inner state. For example, the
behavior of a dictionary object can be abstractly modeled as a set of key,value
pairs. Reflecting the desired encapsulation and information hiding, the abstract model of
the value of a dictionary structure is the same whether the structure is implemented
using sorted arrays, a hash table, or a tree.
A module may be required to establish and preserve certain structural characteristics of
the data structure it maintains. For example, if the dictionary structure is maintained as a
pair of sorted arrays, then it is the responsibility of the dictionary module to maintain the
arrays in sorted order. If the structure is a balanced search tree, then the responsibility
is to properly initialize and maintain the tree structure. This is called a structural
invariant, and it is directly analogous to a loop invariant. When reasoning about a loop
invariant, we begin by showing that it is established when execution first reaches the
loop; this corresponds to showing that the data structure is properly initialized. The
methods of the data structure module correspond to paths through the body of the loop.
Each method must preserve the structural invariant; that is, if the invariant holds before
invocation of the method, then it must still hold when the method returns.
The second responsibility of a class or other data structure module is that its behavior
must faithfully reflect the abstract model. To make this precise, one posits an
abstraction function that maps concrete object states to abstract model states. The
abstraction function for a dictionary object would map the object to a set of
key,value pairs. Using the conventional notation for an abstraction function, the
contract of the get method of java.util.Map might include a pre- and postcondition that
can be expressed as the Hoare triple

Explicit consideration of the abstract model, abstraction function, and structural invariant
of a class or other data structure model is the basis not only of formal or informal
reasoning about correctness, but also of designing test cases and test oracles.

Summary
Symbolic execution is a bridge from an operational view of program execution to logical
and mathematical statements. The basic symbolic execution technique is like hand
execution using symbols rather than concrete values. To use symbolic execution for
loops, procedure calls, and data structures encapsulated in modules (e.g., classes), it is
necessary to proceed hierarchically, composing facts about small parts into facts about
larger parts. Compositional reasoning is closely tied to strategies for specifying intended
behavior.
Symbolic execution is a fundamental technique that finds many different applications.
Test data generators use symbolic execution to derive constraints on input data. Formal
verification systems combine symbolic execution to derive logical predicates with
theorem provers to prove them. Many development tools use symbolic execution
techniques to perform or check program transformations, for example, unrolling a loop
for performance or refactoring source code.
Human software developers can seldom carry out symbolic execution of program code
in detail, but often use it (albeit informally) for reasoning about algorithms and data
structure designs. The approach to specifying preconditions, postconditions, and
invariants is also widely used in programming, and is at least partially supported by tools
for run-time checking of assertions.

Further Reading
The techniques underlying symbolic execution were developed by Floyd [Flo67] and
Hoare [Hoa69], although the fundamental ideas can be traced all the way back to Turing
and the beginnings of modern computer science. Hantler and King [HK76] provide an
excellent clear introduction to symbolic execution in program verification. Kemmerer and
Eckman [KE85] describe the design of an actual symbolic execution system, with
discussion of many pragmatic details that are usually glossed over in theoretical
descriptions.
Generation of test data using symbolic execution was pioneered by Clarke [Cla76], and
Howden [How77, How78] described an early use of symbolic execution to test

programs. The PREfix tool described by Bush, Pincus, and Sielaff [BPS00] is a modern
application of symbolic testing techniques with several refinements and simplifications for
adequate performance on large programs.

Exercises
We introduce symbols to represent variables whose value may change, but we do
not bother to introduce symbols for variables whose value remains unchanged in
7.1 the code we are symbolically executing. Why are new symbols necessary in the
former case but not in the latter?
Demonstrate that the statement return dictValues[mid] at line 27 of the binary
7.2 search program of Figure 7.1 always returns the value of the input key.
Compute an upper bound to the number of iterations through the while loop of the
7.3 binary search program of Figure 7.1.
The body of the loop of the binary search program of Figure 7.1 can be modified
as follows:

7.4

1
2
3
4
5
6
7
8
9
10
11
12

if (comparison < 0) {
/* dictKeys[mid] too small; look higher */
low=mid+1;
}
if ( comparison > 0) {
/* dictKeys[mid] too large; look lower */
high=mid-1;
}
if (comparison=0) {
/* found */
return dictValues[mid];
}

Demonstrate that the path that traverses the false branch of all three statements is
infeasible.
Write the pre- and postconditions for a program that finds the index of the
7.5 maximum element in a nonempty set of integers.

Chapter 8: Finite State Verification

Finite state verification techniques are intermediate in power and cost between
construction of simple control and data flow models, on the one hand, and reasoning
with the full strength of symbolic execution and theorem proving on the other. They
automatically explore finite but potentially very large representations of program
behavior to address important properties. They are particularly useful for checking
properties for which testing is inadequate. For example, synchronization faults in multithreaded programs may trigger failures very rarely, or under conditions that are nearly
impossible to re-create in testing, but finite state verification techniques can detect them
by exhaustively considering all possible interleavings of concurrent processes. Finite
state verification can similarly be used to systematically explore possible instantiations of
a data model.

8.1 Overview
Most important properties of program execution are undecidable in general, but finite
state verification can automatically prove some significant properties of a finite model of
the infinite execution space. Of course, there is no magic: We must carefully reconcile
and balance trade-offs among the generality of the properties to be checked, the class
of programs or models that can be checked, computational effort, and human effort to
use the techniques.
Symbolic execution and formal reasoning can prove many properties of program
behavior, but the power to prove complex properties is obtained at the cost of devising
complex conditions and invariants and expending potentially unbounded computational
effort. Construction of control and data flow models, on the other hand, can be fully and
efficiently automated, but is typically limited to very simple program properties. Finite
state verification borrows techniques from symbolic execution and formal verification, but
like control and data flow analysis, applies them to models that abstract the potentially
infinite state space of program behavior into finite representations. Finite state
verification techniques fall between basic flow analyses and full-blown formal verification
in the richness of properties they can address and in the human guidance and
computational effort they require.
Since even simple properties of programs are undecidable in general, one cannot expect
an algorithmic technique to provide precise answers in all cases. Often finite state
verification is used to augment or substitute for testing when the optimistic inaccuracy of
testing (due to examining only a sample of the program state space) is unacceptable.
Techniques are therefore often designed to provide results that are tantamount to formal
proofs of program properties. In trade for this assurance, both the programs and
properties that can be checked are severely restricted. Restrictions on program
constructs typically appear in procedures for deriving a finite state model from a
program, generating program code from a design model, or verifying consistency
between a program and a separately constructed model.
Finite state verification techniques include algorithmic checks, but it is misleading to
characterize them as completely automated. Human effort and considerable skill are
usually required to prepare a finite state model and a suitable specification for the
automated analysis step. Very often there is an iterative process in which the first
several attempts at verification produce reports of impossible or unimportant faults,
which are addressed by repeatedly refining the specification or the model.
The automated step can be computationally costly, and the computational cost can
impact the cost of preparing the model and specification. A considerable amount of
manual effort may be expended just in obtaining a model that can be analyzed within
available time and memory, and tuning a model or specification to avoid combinatorial
explosion is itself a demanding task. The manual task of refining a model and

specification to obtain either assurance or useful reports of real faults in design or

coding is much less expensive if the analysis step is near-interactive than if it requires
several minutes or hours.
Some analysis techniques perform quite tolerably on small models, but their
computational demands grow very rapidly with model size. These may be perfectly
acceptable for a simple model of a critical component, such as a protocol whose
description does not depend on the size of the system in which it is implemented. In
other cases, scalability of the finite state verification technique is likely to be a limiting
factor in its useful application.
Finite state verification techniques vary widely in the balance they strike on issues of
generality, precision, automation, computational effort, and scalability. A core idea
shared by all is that a question about a program is translated into a simpler question
about a finite state model of the program, as illustrated in Figure 8.1. Ultimately, one
question about the program (Does it conform to the property we want to check?) is
divided into two (Does the model conform to the simpler property we can check? Is it an
accurate model of the program?)

Figure 8.1: The finite state verification framework.

The model may be derived from an actual program, like the control flow and data flow
models described in prior chapters, or from some other design artifact (e.g., a program
specification). Restrictions on the program may be required to derive a model
automatically from a program. It is also possible to derive program code from annotated
models.[1] If either the model or the program is derived automatically from the other, we
may be able to do so in a way that guarantees consistency between the two.
We may also be able to check consistency automatically even if the derivation is not
automatic. Alternatively, the accuracy of the model may be assessed by conformance

testing, treating the model as a kind of specification. The combination of finite state
verification and conformance testing is often more effective than directly testing for the
property of interest, because a discrepancy that is easily discovered in conformance
testing may very rarely lead to a run-time violation of the property (e.g., it is much easier
to detect that a particular lock is not held during access to a shared data structure than
to catch the occasional data race that the lock protects against).
A property to be checked can be implicit in a finite state verification tool (e.g., a tool
specialized just for detecting potential null pointer references), or it may be expressed in
a specification formalism that is purposely limited to a class of properties that can be
effectively verified using a particular checking technique. Often the real property of
interest is not amenable to efficient automated checking, but a simpler and more
restrictive property is. That is, the property checked by a finite state verification tool may
be sufficient but not necessary for the property of interest. For example, verifying
freedom from race conditions on a shared data structure is much more difficult than
verifying that some lock is always held by threads accessing that structure; the latter is
a sufficient but not necessary condition for the former. This means that we may exclude
correct software that we are not able to verify, but we can be sure that the accepted
software satisfies the property of interest.
[1]Note

that one may independently derive several different models from one program,
but deriving one program from several different models is much more difficult.

8.2 State Space Exploration

While some finite state models of program execution can be derived rather directly from
syntactic program structure (e.g., control flow graph models of individual procedures),
this is not always so. In particular, an adequate finite state machine model of a program
or system with multiple threads of control (Java threads, Ada tasks, operating system
processes, etc.) must include all the possible ways execution of the individual threads
can be interleaved. A global model of the reachable system states and transitions can
be systematically explored by tracing all the possible sequences of interactions.
Let us begin with several simplifying assumptions. We assume that we can determine in
advance how many threads of control, or processes make up the system, and that we
can obtain a finite state machine model of each. We assume also that we can identify
the points at which processes can interact and all the ways that execution of one
process may affect another. A state of the whole system model, then, is a tuple
representing the state of each individual process model, and a transition in the system
model is a transition of one or more of the individual processes, acting individually or in
concert.
From one global system state, several different individual or joint transitions of the
component processes may be possible. That is, execution in the global model is
nondeterministic. This should be no surprise, as it reflects the real situation in multithreaded software, with execution dependent on uncontrolled factors like the arrival of
asynchronous interrupts, process scheduler decisions, and the relative execution speed
of different processes. It is these unpredictable and uncontrollable factors that make
effectively testing programs and systems with multiple threads of control difficult. A test
case may run correctly a million times in a test configuration and fail the first time a client
uses it.
Given an appropriate model and an execution rule, exploring all the possible states
reachable by the system is a completely mechanical process. If "good" states can be
easily distinguished from "bad" states, then the whole process of exploring and checking
the state space model can be automatic. Even the simplest and most brute- force state
space exploration tools can systematically check many times more states in a minute
than a person could in a month.
We illustrate with a simple and somewhat contrived example. In a certain multi- threaded
module of the Chipmunk on-line purchasing system, there is an in-memory data structure
that is initialized by reading configuration tables at system start-up. Initialization of the
data structure must appear atomic (the structure should not be accessed while
initialization is underway). Moreover, it must be reinitialized on occasion. The structure is
kept in memory, rather than read from a database on each use, because it is small,
changes rarely, and is accessed very frequently. A Chipmunk programmer has noticed
that obtaining a monitor lock for each and every access (which is what a Java

"synchronized" method does) substantially reduces concurrency and slows user

response time. The programmer has recently learned of the double-checked locking
idiom to avoid unnecessary locking during data structure initialization. Unfortunately, the
programmer does not fully comprehend the double-check idiom and its underlying
assumptions, and produces the faulty implementation excerpted in Figure 8.2.

1 /** A singleton class with mis-application of double-check p

2 class Table1 {
3
private static Table1 ref = null; // Singleton instance
4
private boolean needsInit = true; // To trigger lazy re5
private ElementClass [ ] theValues;
6
7
private Table1() {} // Initialization is separate
8
9
/** Initialization with double-check pattern. */
10
public static Table1 getTable1() {
11
if (ref == null) { synchedInitialize(); }
12
return ref; 13 }
14
15
private static synchronized void synchedInitialize() {
16
if (ref == null) {
17
Table1 tmp = new Table1();
18
tmp.initialize();
19
ref=tmp; }
20
}
21
22
/** Trigger re-initialization on next access */
23
public void reinit() { needsInit = true; }
24
25
/** Initialize or re-initialize. Must appear atomic to l
26
private synchronized void initialize() {
32 ...
33
needsInit = false;
34
}
35
36
/** Lookup value, lazily re-init. (WRONG!) */
37
public int lookup(int i) {
38
if (needsInit) {
39
synchronized(this) {
40
if (needsInit) {
41
this.initialize();
42
}

43
44
45
46
}
47
60 ...
61 }

}
}
return theValues[i].getX() + theValues[i].getY();

Figure 8.2: Double-check pattern, misapplied to reinitialization.

The fault in this example is simple: The double-check idiom is applicable only to
initialization, not to modification of a structure after initialization.[2] However, it is not easy
for a person to comprehend all the possible ways that multiple threads could interleave
while concurrently executing these methods, and it is surprisingly easy to convince
oneself that the faulty implementation avoids race conditions. Moreover, it is extremely
difficult to find them with conventional testing. Even under heavy load, the potential race
condition in the code of Figure 8.2 very rarely leads to run-time failure and may not
appear at all depending on the scheduling policies and resources of a particular Java
run-time system.
A potential failure is simple to find by systematically tracing through all the possible
interleavings of two threads. We begin by constructing a finite state machine model of
each individual thread. For method lookup in Figure 8.2, the state machines in Figure 8.3
describe the actions of an individual thread executing methods lookup and reInit, but we
do not know in advance how many distinct threads might be executing concurrently.

Figure 8.3: Finite state models of individual threads executing the lookup and reInit
methods from Figure 8.2. Each state machine may be replicated to represent
concurrent threads executing the same method.
Java threading rules ensure that in a system state in which one thread has obtained a
monitor lock, the other thread cannot make a transition to obtain the same lock. We can
observe that the locking prevents both threads from concurrently calling the initialize
method. However, another race condition is possible, between two concurrent threads
each executing the lookup method.
Tracing possible executions by hand - "desk checking" multi-threaded execution - is
capable in principle of finding the race condition between two concurrent threads
executing the lookup method, but it is at best tedious and in general completely
impractical. Fortunately, it can be automated, and many state space analysis tools can
explore millions of states in a short time. For example, a model of the faulty code from
Figure 8.2 was coded in the Promela modeling language and submitted to the Spin
verification tool. In a few seconds, Spin systematically explored the state space and
reported a race condition, as shown in Figure 8.5.

Depth=
10 States=
51 Transitions=
92 Memory= 2.30
pan: assertion violated !(modifying) (at depth 17)
pan: wrote pan_in.trail
(Spin Version 4.2.5 -- 2 April 2005)
...
0.16 real
0.00 user
0.03 sys
Figure 8.5: Excerpts of Spin verification tool transcript. Spin has performed a depthfirst search of possible executions of the model, exploring 10 states and 51 state
transitions in 0.16 seconds before finding a sequence of 17 transitions from the initial
state of the model to a state in which one of the assertions in the model evaluates to
False.
A few seconds of automated analysis to find a critical fault that can elude extensive
testing seems a very attractive option. Indeed, finite state verification should be a key
component of strategies for eliminating faults in multi-threaded and distributed programs,
as well as some kinds of security problems (which are similarly resistant to systematic
sampling in conventional program testing) and some other domains. On the other hand,
we have so far glossed over several limitations and problems of state space exploration,
each of which also appears in other forms of finite state verification. We will consider
two fundamental and related issues in the following sections: the size of the state space
to be explored, and the challenge of obtaining a model that is sufficiently precise without
making the state space explosion worse.

The Promela Modeling Language

The Promela language for finite state models of communicating processes, which is
interpreted by the verification tool Spin, is described in a book and on-line references
(see Further Reading at the end of this chapter). Here we present a very brief and
partial introduction to aid in reading the example code of Figure 8.4.

1 bool needsInit = true, /* Models variable by same name */

2
locked = false, /* To model synchronized block */
3
modifying = false; /* To test for race condition */
4
5 proctype Lookup(int id ) {
6
if :: (needsInit) ->
7
/* "synchonized(this) {"*/
8
atomic { ! locked -> locked = true; };
9
if
10
:: (needsInit) ->
11
/* Body of "intialize()" modeled here */
12
assert (! modifying); /* Test for write/write r
13
modifying = true;
14
/* The actual modification happens here */
15
modifying = false ;
16
needsInit = false;
17
:: (! needsInit) ->
18
skip;
19
fi;
20
/* "}" (end synchronized block) */
21
locked = false ;
22
fi;
23
/* Return a value from lookup() */
24
assert (! modifying); /* Test for read/write race */
25 }
26
27 proctype reInit() {
28
needsInit = true;
29 }
30
31 init {
32
run reInit();
33
run Lookup(1);
34
run Lookup(2);
35 }

Figure 8.4: Promela finite state model of faulty double-check

implementation.
A Promela program describes a set of processes, roughly analogous to threads in
Java. A single process type (proctype) can be instantiated more than once with run
statements to create multiple instances of a process, much as thread objects can be
created from a class in a Java program. A Promela model consists of some global
data type and variable declarations, followed by some process type declarations, and
finally a "main" process init.
Many lexical conventions of Promela are borrowed from the C language, and should
be familiar to C and Java programmers. Comments are enclosed in /* and */,
syntactic nesting is indicated by braces { and }, and assignment is indicated by a
single = while an equality comparison is indicated by ==. As in C, nonzero values are
interpreted as True and zero is Boolean False.
Promela borrows syntax and semantics for "guarded commands" from
Communicating Sequential Processes (CSP), a formal notation for describing
communicating processes. A guarded command in Promela is written expression ->
statements and means that the statements can be executed only when the guarding
expression is true. If the expression evaluates to zero or is otherwise disabled,
execution of the guarded statement is blocked. Thus, the statement

in Figure 8.4 can be used to represent acquiring a monitor lock, because execution
blocks at this point until locked has the value False. The guard is enclosed in an
atomic block to prevent another process taking the lock between evaluation of the
guard condition and execution of the statement.
The concept of enabling or blocking in guarded commands is used in conditional and
looping constructs. Alternatives in an if fi construct, marked syntactically with ::,
begin with guarded commands. If none of the alternatives is enabled (all of the guards
evaluate to False), then the whole if construct blocks. If more than one of the guarded
alternatives is enabled, the if construct does not necessarily choose the first among
them, as a programmer might expect from analogous if else if constructs in
conventional programming languages. Any of the enabled alternatives can be
nondeterministically chosen for execution; in fact the Spin tool will consider the
possible consequences of each choice. The do od construct similarly chooses
nondeterministically among enabled alternatives, but repeats until a break or goto is
evaluated in one of the guarded commands.
The simplest way to check properties of a Promela model is with assertions, like the

two assert statements in Figure 8.4. Spin searches for any possible execution
sequence in which an assertion can be violated. Sequencing properties can also be
specified in the form of temporal logic formulas, or encoded as state machines.

preparing trail, please wait...done

Starting :init: with pid 0
spin: warning, "pan_in", proctype Lookup,
'int
id' variable is never used
Starting reInit with pid 1
1: proc
0 (:init:) line
33 "pan_in" (state 1) [(run reIni
Starting Lookup with pid 2
2: proc
0 (:init:) line
34 "pan_in" (state 2) [(run Looku
Starting Lookup with pid 3
3: proc
0 (:init:) line
35 "pan_in" (state 3) [(run Looku
4: proc 3 (Lookup) line
7 "pan_in" (state 1) [(needsInit)]
5: proc 3 (Lookup) line
9 "pan_in" (state 2) [(!(locked))]
<merge 0 now @3>
5: proc 3 (Lookup) line
9 "pan_in" (state 3) [locked = 1]
6: proc
3 (Lookup) line
11 "pan_in" (state 5) [(needsInit
7: proc
3 (Lookup) line
13 "pan_in" (state 6) [assert(!(m
8: proc
3 (Lookup) line 14 "pan_in" (state 7) [modifying =
9: proc
3 (Lookup) line 16 "pan_in" (state 8) [modifying =
10: proc
3 (Lookup) line
17 "pan_in" (state 9) [needsInit =
11: proc
3 (Lookup) line
22 "pan_in" (state 14) [locked = 0
12: proc
1 (reInit) line
29 "pan_in" (state 1) [needsInit =
13: proc
2 (Lookup) line
7 "pan_in" (state 1) [(needsInit)]
14: proc
2 (Lookup) line
9 "pan_in" (state 2) [(!(locked))]
<merge 0 now @3>
14: proc
2 (Lookup) line
9 "pan_in" (state 3) [locked = 1]
15: proc
2 (Lookup) line
11 "pan_in" (state 5) [(needsInit)
16: proc
2 (Lookup) line
13 "pan_in" (state 6) [assert(!(m
17: proc
2 (Lookup) line
14 "pan_in" (state 7) [modifying =
spin: trail ends after 17 steps
#processes: 4
17: proc
3 (Lookup) line
25 "pan_in" (state 17)
17: proc
2 (Lookup) line
16 "pan_in" (state 8)
17: proc
1 (reInit) line
30 "pan_in" (state 2)
17: proc
0 (:init:) line
36 "pan_in" (state 4)
4 processes created
Exit-Status 0

Figure 8.6: A Spin guided simulation trace describes each of the 17 steps from the
initial model state to the state in which the assertion !(modifying) is violated. For
example, in step 8, one of the two processes (threads) simulating execution of the
Lookup method sets the global variable modifying to True, represented as the integer
value 1. A graphical representation of this trace is presented in Figure 8.7.

Figure 8.7: A graphical interpretation of Spin guided simulation output (Figure 8.6) in
terms of Java source code (Figure 8.2) and state machines (Figure
8.3).
Safety and Liveness Properties
Properties of concurrent systems can be divided into simple safety properties,
sequencing safety properties, and liveness properties.
Simple safety properties divide states of the system into "good" (satisfying the
property) and "bad" (violating the property). They are easiest to specify, and least
expensive to check, because we can simply provide a predicate to be evaluated at
each state. Often simple safety properties relate the local state of one process to
local states of other processes. For example, the assertion assert(! modifying) in the
Promela code of Figure 8.4 states a mutual exclusion property between two
instances of the lookup process. When simple safety properties are expressed in
temporal logic, they have the form p, where p is a simple predicate with no
temporal modalities.
Safety properties about sequences of events are similar, but treat the history of

events preceding a state as an attribute of that state. For example, an assertion that
two operations a and b strictly alternate is a safety property about the history of
those events; a "bad" state is one in which a or b is about to be performed out of
order. Sequencing properties can be specified in temporal logic, but do not require it:
They are always equivalent to simple safety properties embedded in an "observer"
process. Checking a sequencing property adds the same degree of complexity to the
verification process as adding an explicit observer process, whether there is a real
observer (which is straightforward to encode for some kinds of model, and nearly
impossible for others) or whether the observer is implicit in the checking algorithm (as
it would be using a temporal logic predicate with the Spin tool).
True liveness properties, sometimes called "eventuality" properties, are those that
can only be violated by an infinite length execution. For example, if we assert that p
must eventually be true (p), the assertion is violated only by an execution that runs
forever with p continuously false. Liveness properties are useful primarily as a way of
abstracting over sequences of unknown length. For example, fairness properties are
an important class of liveness properties. When we say, for example, that a mutual
exclusion protocol must be fair, we do not generally mean that all processes have an
equal chance to obtain a resource; we merely assert that no process can be starved
forever. Liveness properties (including fairness properties) must generally be stated in
temporal logic, or encoded directly in a Bchi automaton that appears similar to a
deterministic finite state acceptor but has different rules for acceptance. A finite state
verification tool finds violations of liveness properties by searching for execution loops
in which the predicate that should eventually be true remains false; this adds
considerably to the computational cost of verification.
A common mnemonic for safety and liveness is that safety properties say "nothing
bad happens," while liveness properties say "something good eventually happens."
Properties involving real time (e.g., "the stop signal is sent within 5 seconds of
receiving the damage signal") are technically safety properties in which the "bad thing"
is expiration of a timer. However, naive models involving time are so expensive that it
is seldom practical to simply add a clock to a model and use simple safety properties.
Usually it is best to keep reasoning about time separate from verifying untimed
properties with finite state verification.

[2]In

fact even a correctly implemented double-check pattern can fail in Java due to
properties of the Java memory model, as discussed below.

8.3 The State Space Explosion Problem

The finite state model of faulty code described in the previous section is very simple: two
processes concurrently executing the lookup method, another executing the trivial reInit
method, and an even more trivial administrative process to start them. While it is quite
tedious to trace out all the potential interleavings of these processes by hand,[3] an
automated verification tool can do so almost instantaneously.
Unfortunately, larger and more complex models may cause the same tools to grind for
hours or days without producing a result, typically ending by exhausting all available
memory. The number of states in a concurrent system with P processes, each with K
individual states, is at most the number of possible P-tuples of K values, that is, KP.
Synchronization and other dependencies among processes will limit the number of
reachable states to a somewhat smaller number. Nonetheless, the number of reachable
states does typically grow exponentially with the number of processes.
Figure 8.8 and the sidebar on page 127 illustrate state space explosion with the
classical dining philosophers problem. This exponential blow-up in the number of
reachable states is not just an artifact of a naive modeling methodology. It has been
proved, in a variety of models of concurrent execution, that decision procedures even for
very simple properties like freedom from deadlock or race conditions is PSPACEcomplete. This means that in the worst case, exponential complexity is almost certainly
unavoidable in any procedure that can answer the kinds of questions we use state space
exploration to answer.

1 mtype = { Up, Down, /* Fork operations */

2 Thinking, Hungry, Eating /* What philosophers do */ }
3
4 proctype fork(chan opChannel) {
5
do
6
:: opChannel?Up; /* First I can be lifted ... */
7
opChannel?Down; /* Then I can be set do
8
od; /* Then lifted again, and so on */
9
}
10
11 proctype philosopher(chan leftFork, rightFork) {
12
show mtype myState = Thinking;
13
do
14
:: myState = Hungry;
15
leftFork!Up;
16
rightFork!Up;
17
myState = Eating;

18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

rightFork!Down;
leftFork!Down;
myState = Thinking;
od;
}

#define NumSeats 10
chan forkInterface[NumSeats] = [0] of {mtype} ;
init {
int i=0;
do :: i < NumSeats ->
run fork( forkInterface[i] );
i=i+1;
:: i >= NumSeats -> break;
od;
i=0;
do :: i < NumSeats ->
run philosopher( forkInterface[i], forkInterface[
i=i+1;
:: i >= NumSeats-1 -> break;
od;
}

Figure 8.8: The classic dining philosophers problem in Promela. The number of
unique states explored before finding the potential deadlock (with default settings)
grows from 145 with 5 philosophers, to 18,313 with 10 philosophers, to 148,897 with
15 philosophers.
The known complexity results strongly imply that, in the worst case, no finite state
verification technique can be practical. Worst case complexity results, however, say
nothing about the performance of verification techniques on typical problems. Experience
with a variety of automated techniques tells us a fair amount about what to expect: Many
techniques work very well when applied on well-designed models, within a limited
domain, but no single finite state verification technique gives satisfactory results on all
problems. Moreover, crafting a model that accurately and succinctly captures the
essential structure of a system, and that can be analyzed with reasonable performance
by a given verification tool, requires creativity and insight as well as understanding of the
verification approach used by that tool.
An Illustration of State Space Explosion

Consider the classic dining philosophers problem, in which an equal number of

philosophers and forks are arranged around a table. A philosopher must lift both
adjacent forks before eating. A Promela model of the dining philosophers problem is
shown in Figure 8.8. With 5 philosophers and 5 forks, Spin finds the potential
deadlock in less than a second of search, exploring only 145 unique states of the
system. With 10 philosophers and 10 forks, Spin with default settings begins to cut off
the search at a depth of 9999 execution steps, but still finds the deadlock at a depth
of 9995 steps, generating 18,313 unique states while executing a depth-first search.
With 15 philosophers and 15 forks, Spin explores 148,897 states before finding a
deadlock, and again the error trace it creates is too long to be useful in diagnosis.
Spin can be instructed to use a breadth-first search or iterate to find a shorter error
trace, but these options cause it to generate over half a million unique states and
exhaust its default allocation of memory. A version of the model with 10 forks and
only 9 philosophers generates 404,796 unique states with the default settings, with an
inconclusive result since it finds no errors but terminates the search at depth 9999
(after 195 minutes on the same computer that analyzed the first example in a few
seconds). One can increase the allocation of memory and wait longer for a result, but
from the rate of growth it is evident that an approach of buying bigger and faster
machines will not scale to a much larger model.
Fortunately, the deadlock produced by a system of just three philosophers is a
perfectly good representation of the potential deadlock in a system of 10 or 15 or 100
philosopher processes. State space enumeration is most effective when the essential
structure of a system can be represented by a much smaller model.

[3]It

is a useful exercise to try this, because even though the number of reachable states
is quite small, it is remarkably difficult to enumerate them by hand without making
mistakes. Programmers who attempt to devise clever protocols for concurrent operation
face the same difficulty, and if they do not use some kind of automated formal
verification, it is not an exaggeration to say they almost never get it right.

8.4 The Model Correspondence Problem

In the simple examples above, we have written Promela models by hand to verify
concurrent execution in Java programs. One may ask how we can be sure that the
Promela models accurately represent the possible behaviors of the Java programs,
particularly if there are conceptual errors in the design of the Java programs. This is a
serious problem, and it has no fully satisfactory solution.
We could verify correspondence between a finite state model and a program in one of
three ways. First, we could automatically extract a model from the program source code
(or compiled code, e.g., Java byte code), using procedures that we have verified once
and for all. Second, we could turn the derivation relation around, producing program
source code automatically from a model, treating the model as a kind of design
document. The third option is to apply some combination of static analysis and testing to
verify correspondence between model and program.
Automatically extracting models from programs is an attractive option, with the important
advantage that the correctness of the extraction tool can be verified once and for all. In
this approach, sophisticated and expensive verification can be justified and carried out
by tool developers who are much more expert in finite state verification than users of the
tool. The previous section strongly hints at the chief obstacle to model extraction: A
model that blindly mirrors all details of program execution is likely to suffer from a much
worse state space explosion than a model that has been carefully crafted to capture just
the relevant essence of synchronization structure. A model that omits some crucial
detail, on the other hand, can produce so many "false alarm" reports (failures that are
possible in the model but not in the program) that the results are useless. The challenge
for automatic model extraction, then, is to capture just enough of the relevant detail to be
accurate, while abstracting enough to keep state space explosion under control.
Some abstraction of program details can be completely automated. For example,
dependence analysis can be used to identify portions of the program that are irrelevant
to checking a particular property. For this reason, it is often worthwhile to extract
different models from the same program, to check different properties of interest.
Where the required level of detail cannot be determined a priori by program analysis,
sometimes a coarse initial model can be iteratively refined until either a verification or a
counter-example is achieved. This is discussed further in Section 8.7.
Human cleverness in model design and automated support for model extraction are not
mutually exclusive. For example, an important tactic in building finite state models is
abstracting data values. It would be far too expensive to represent all the possible
states of a queue of integers, for instance, but one might be able to capture enough
information in the predicate isEmpty(Q). Sometimes a choice of predicates is strongly
suggested by control structure of the program, and may even be found automatically by
a model extraction tool. In other cases the user may be able to provide much better

predicates to guide automated model extraction.

One can also reverse the model extraction process, starting with a finite state model
and generating program code. Usually what can be generated is not the whole
application, but it may be a component or skeleton in which the relevant behavior is
localized. Essentially, this is equivalent to requiring the developer to manually distinguish
the finite state model from other aspects of the application, but it can be much easier to
specify how the finite state model is combined with other application details than to
specify how the finite state model is extracted from the completed application. Program
generation from (verifiable) finite state models, like program generation in general, is
most applicable within constrained or well-understood application domains.
If a model is automatically extracted, or a program is automatically generated from a
model, then correspondence between model and program can be verified once and for
all by verifying the method of derivation. If, however, the derivation method is at least
partly manual, then it will be necessary to gain confidence in their consistency by some
other approach. Static program analysis can be helpful, but in the worst case a static
analysis that verifies consistency between a model and a program can be as complex as
a static analysis for extracting a model. More typically, conformance is verified by
testing.
The details of an approach to conformance testing depend primarily on the form of the
model and on what can be observed from program execution. A typical scenario is that
the program model is equivalent to a deterministic finite state machine (FSM), and the
only relevant observable aspect of program execution is a set of events (e.g., system
calls or instrumented points) that correspond to event labels in the FSM model. A single
execution is then consistent with the model if the observed sequence of execution events
corresponds to a sequence of state transitions in a traversal of the model. The basic
approach can be extended in several ways, for example, by testing against each of
several communicating state machines separately or in parallel, by checking portions of
program state against model state, or by considering multiple possible traversals in
parallel if the model is inherently nondeterministic or the correspondence between
observed program events and model state transitions is ambiguous. There is a
welldeveloped body of testing techniques based on state machine models, some of
which are discussed further in Chapter 14.
One may ask what advantage finite state verification has over simply testing the
program for the property of interest, if we must still resort to conformance testing to
verify the accuracy of a model. For example, if we are using finite state verification to
show absence of race conditions, and then testing the program for conformance to the
verified model, why not simply use testing to check for race conditions directly in the
program?
In fact, the combination of finite state verification with testing can be both less expensive

and more effective than testing alone. Consider again our simple example of
misapplication of the double-check pattern in Figure 8.2. Tens of thousands of test
executions can fail to reveal the race condition in this code, depending on the way
threads are scheduled on a particular hardware platform and Java virtual machine
implementation. Testing for a discrepancy between model and program, on the other
hand, is fairly straightforward because the model of each individual state machine can be
checked independently (in fact all but one are trivial). The complexity that stymies testing
comes from nondeterministic interleaving of their execution, but this interleaving is
completely irrelevant to conformance testing.

8.5 Granularity of Modeling

Showing that each thread or process in a program performs actions in an order
consistent with its FSM model, and that the effect of each sequence of actions is
modeled correctly, is not quite enough. We also need to consider the granularity of
those actions - the points at which actions from one thread can be interrupted by actions
of another.
Consider the trivial program of Figure 8.9. The race condition is apparent: Both threads
RacerP and RacerQ increment shared variable i. The possible ending values of i depend
on whether i=i+1 is an atomic (indivisible) action, or a sequence of smaller operations.
The coarse-grain FSM of Figure 8.10 treats each statement as an atomic action, while
the fine-grain FSM in the same figure breaks the increment operation into separate load,
add, and store steps. Only the finer grain FSM can reveal the "lost update" problem
illustrated in Figure 8.11.

1 /** Trivial race between two increments. A version of this p

2 * appears in many books on concurrency or operating systems;
3 * the "hello world" of race conditions.
4 */
5 class Unsafe implements Runnable {
6
static int i=1; /* Before increments, value is 1. And af
7
8
/** Each thread increments i by 1 */
9
public void run() {
10
i=i+1; 11 }
12
13
/** Two threads interleave their updates */
14
public static void main(String[] argv) {
15
Unsafe unsafe = new Unsafe();
16
Thread racerP = new Thread(unsafe);
17
racerP.start();
18
Thread racerQ = new Thread(unsafe);
19
racerQ.start();
20
21
/* Wait for both to finish */
22
try {
23
racerP.join(); racerQ.join();
24
} catch (InterruptedException e) {
25
System.err.println("Unexpected interruption");
26
}
27

28
29
30
31
32 }

/* What values could i possibly have? */

System.out.println("i: " + i);
}

Figure 8.9: A simple data race in Java. The possible ending values of i depend on
how the statement i = i+1 in one thread is interleaved with the same sequence in the
other thread.

Figure 8.10: Coarse and fine-grain models of the same program from Figure 8.9. In
the coarse-grain model, i will be increased by 2, but other outcomes are possible in
the finer grain model in which the shared variable i is loaded into temporary variable
or register, updated locally, and then stored.

Figure 8.11: The lost update problem, in which only one of the two increments
affects the final value of i. The illustrated sequence of operations from the program of
Figure 8.9 can be found using the finer grain model of Figure 8.10, but is not revealed
by the coarser grain model.
Even representing each memory access as an individual action is not always sufficient.
Programming language definitions usually allow compilers to perform some
rearrangements in the order of instructions. What appears to be a simple store of a
value into a memory cell may be compiled into a store into a local register, with the
actual store to memory appearing later (or not at all, if the value is replaced first). Two
loads or stores to different memory locations may also be reordered for reasons of
efficiency. Moreover, when a machine instruction to store a value into memory is
executed by a parallel or distributed computer, the value may initially be placed in the
cache memory of a local processor, and only later written into a memory area accessed
by other processors. These reorderings are not under programmer control, nor are they
directly visible, but they can lead to subtle and unpredictable failures in multi-threaded
programs.
As an example, consider once again the flawed program of Figure 8.2. Suppose we
corrected it to use the double-check idiom only for lazy initialization and not for updates
of the data structure. It would still be wrong, and unfortunately it is unlikely we would
discover the flaw through finite state verification. Our model in Promela assumes that

memory accesses occur in the order given in the Java program, but Java does not
guarantee that they will be executed in that order. In particular, while the programmer
may assume that initialization invoked in line 18 of the Java program is completed before
field ref is set in line 19, Java makes no such guarantee.
Breaking sequences of operations into finer pieces exacerbates the state explosion
problem, but as we have seen, making a model too coarse risks failure to detect some
possible errors. Moreover, conformance testing may not be much help in determining
whether a model depends on unjustified assumptions of atomicity. Interruptions in a
sequence of program operations that are mistakenly modeled as an atomic action may
not only be extremely rare and dependent on uncontrolled features of the execution
environment, such as system load or the activity of connected devices, but may also
depend on details of a particular language compiler.
Conformance testing is not generally effective in detecting that a finite state model of a
program relies on unwarranted assumptions of atomicity and ordering of memory
accesses, particularly when those assumptions may be satisfied by one compiler or
machine (say, in the test environment) and not by another (as in the field). Tools for
extracting models, or for generating code from models, have a potential advantage in
that they can be constructed to assume no more than is actually guaranteed by the
programming language.
Many state space analysis tools will attempt to dynamically determine when a sequence
of operations in one process can be treated as if it were atomic without affecting the
results of analysis. For example, the Spin verification tool uses a technique called partial
order reduction to recognize when the next event from one process can be freely
reordered with the next event from another, so only one of the orders need be checked.
Many finite state verification tools provide analogous facilities, and though they cannot
completely compensate for the complexity of a model that is more fine-grained than
necessary, they reduce the penalty imposed on the cautious model-builder.

8.6 Intensional Models

The computational cost of enumerating reachable states, particularly the storage
required to recognize states that have already been explored, is often a limiting factor in
applying finite state verification tools. Sometimes (but not always) this expense can be
significantly reduced by using intensional (symbolic) representations that describe sets
of reachable states without enumerating each one individually.
The idea of symbolic or intensional representations can be illustrated with sets of
integers. Consider the set

The extensional representation, given above, lists the elements of the set. The same set
can be represented intensionally as

The predicate x mod 2 = 0 0 < x < 20, which is true for elements included in the set
and false for excluded elements, is called a characteristic function. The length of the
representation of the characteristic function does not necessarily grow with the size of
the set it describes. For example, the set

contains four times as many elements as the one above, and yet the length of the
representation is the same.
It could be advantageous to use similarly compact representations for sets of reachable
states and transitions among them. For example, ordered binary decision diagrams
(OBDDs) are a representation of Boolean functions that can be used to describe the
characteristic function of a transition relation. Transitions in the model state space are
pairs of states (the state before and the state after executing the transition), and the
Boolean function represented by the OBDD takes a pair of state descriptions and
returns True exactly if there is a transition between such a pair of states. The OBDD is
built by an iterative procedure that corresponds to a breadth-first expansion of the state
space (i.e., creating a representation of the whole set of states reachable in k + 1 steps
from the set of states reachable in k steps). If the OBDD representation does not grow
too large to be manipulated in memory, it stabilizes when all the transitions that can
occur in the next step are already represented in the OBDD form.
Finding a compact intensional representation of the model state space is not, by itself,
enough. In addition we must have an algorithm for determining whether that set satisfies
the property we are checking. For example, an OBDD can be used to represent not only
the transition relation of a set of communicating state machines, but also a class of

temporal logic specification formulas. The OBDD representations of model and

specification can be combined to produce a representation of just the set of transitions
leading to a violation of the specification. If that set is empty, the property has been
verified. This approach is known as symbolic model checking, and has been
spectacularly successful in dealing with some models of concurrent system (primarily for
hardware, but sometimes also for software).
Encoding transition relations as OBDDs can be divided into two parts: representing
transition relations as Boolean functions, and representing Boolean functions as OBDDs.
Representing Boolean functions as OBDDs is straightforward, as illustrated in Figure
8.12. Essentially the BDD is a decision tree that has been transformed into an acyclic
graph by merging nodes leading to identical subtrees. The merging is made efficient by
ordering the decisions in the same way on all paths from the root of the decision tree to
the leaves, which represent outcomes. Constructing the representation of transition
relations as Boolean functions, on the other hand, can be quite involved. Figure 8.13
illustrates some of the basic ideas.

Figure 8.12: Ordered binary decision diagram (OBDD) encoding of the Boolean
proposition a b c, which is equivalent to a (b c). The formula and OBDD
structure can be thought of as a function from the Boolean values of a, b, and c to a
single Boolean value True or False.

Figure 8.13: Ordered binary decision diagram (OBDD) representation of a transition

relation, in three steps. In part (A), each state and symbol in the state machine is
assigned a Boolean label. For example, state s0 is labeled 00. In part (B), transitions
are encoded as tuples sym,from,to indicating a transition from state from to state
to on input symbol sym. In part (C), the transition tuples correspond to paths leading
to the True leaf of the OBDD, while all other paths lead to False. The OBDD
represents a characteristic function that takes valuations of x0 x4 and returns True
only if it corresponds to a state transition.
In the worst case, intensional representations are no more compact than listing the
elements of a set. In fact, information theory tells us that if we have a large set S of
states, a representation capable of distinguishing each subset of S (all elements of 2S)
cannot be more compact on average than the representation that simply lists elements
of the chosen subset. When intensional representations work well, it is because we do
not produce arbitrary sets of reachable states; rather, there is a good deal of structure
and regularity in the state space, and that regularity is exploited in symbolic
representations.
A good rule of thumb is that finite state verification tools that use intensional
representations (typically called symbolic model checkers) are more effective, the more
regularity is captured in the model, while an explicit model checker (like Spin) is apt to
be at least as effective where little regularity can be captured, or where the kinds of
regularity that can be captured can also be exploited in explicit state space exploration
(e.g., the partial order reductions used by Spin). Unfortunately, this advice is rather
vague, because we do not know a precise way to describe or measure the kinds of
regularity that affect verification tool performance.
Whether a finite state verification tool performs explicit state enumeration or manipulates
an intensional representation can be partly hidden from the tool user, and it is possible
for a single tool "front end" for building or extracting models to be connected to multiple

"back end" verification engines.

8.7 Model Refinement

Because construction of finite state models requires a delicate balance between
precision and efficiency, often the first model we construct will be unsatisfactory - either
the verification tool will produce reports of potential failures that are obviously
impossible, or it will exhaust resources before producing any result at all. Minor
differences in the model can have large effects on tractability of the verification
procedure, so in practice finite state verification is often an iterative process of
constructing a model, attempting verification, and then either abstracting the model
further (if the verification exhausts computational resources or the user's patience before
obtaining a conclusive result) or making the model more precise to eliminate spurious
results (i.e., a report of a potential error that cannot actually occur).
An iterative process of model refinement can be at least partly automated. We begin
with a very coarse model that can be efficiently constructed and analyzed, and then we
add detail specifically aimed at ruling out spurious error reports. There are two main
approaches: adding detail directly to the model, or adding premises to the property to
be checked.
Initially, we try to verify that a very coarse model M1 satisfies property P:

However, M is only an approximation of the real system, and we find that the verification
finds a violation of P because of some execution sequences that are possible in M1 but
not in the real system. In the first approach, we examine the counter-example (an
execution trace of M1 that violates P but is impossible in the real system) and create a
new model M2 that is more precise in a way that will eliminate that particular execution
trace (and many similar traces). We attempt verification again with the refined model:

If verification fails again, we repeat the process to obtain a new model M3, and so on,
until verification succeeds with some "good enough" model Mk or we obtain a counterexample that corresponds to an execution of the actual program.
One kind of model that can be iteratively refined in this way is Boolean programs. The
initial Boolean program model of an (ordinary) program omits all variables; branches (if,
while, etc.) refer to a dummy Boolean variable whose value is unknown. Boolean
programs are refined by adding variables, with assignments and tests - but only Boolean
variables. For instance, if a counter-example produced by trying to verify a property of a
pump controller shows that the waterLevel variable cannot be ignored, a Boolean
program might be refined by adding a Boolean variable corresponding to a predicate in

which waterLevel is tested (say, waterLevel < highLimit), rather than adding the variable
waterLevel itself. For some kinds of interprocedural control flow analysis, it is possible
to completely automate the step of choosing additional Boolean variables to refine Mi
into Mi+1 and eliminate some spurious executions.
In the second approach, M remains fixed,[4] but premises that constrain executions to be
checked are added to the property P. When bogus behaviors of M violate P,we add a
constraint C1 to rule them out and try the modified verification problem:

If the modified verification problem fails because of additional bogus behaviors, we try
again with new constraints C2:

so on until verification either succeeds or produces a valid counter-example.

The FLAVERS finite state verification tool is an example of the second approach, adding
constraints to refine a model of concurrent execution. A FLAVERS model approximates
concurrent execution with a pairwise "may immediately precede" (MIP) relation among
operations in different threads. Because MIP relates only pairs of individual process
states, rather than k-tuples for a model with k processes, its size is only quadratic in the
size of the state machine model, rather than exponential in the number of processes.
Moreover, a reasonably good approximation of the MIP relation can be obtained in cubic
time.[5]
If one thinks of each MIP edge in the program model as representing possible
interruption of one thread and continuation of another, it is apparent that paths combining
transitions within individual processes and MIP transitions between processes can
represent all paths through the global state space. Many additional paths, which would
not appear in a more precise global model of possible executions, are also represented.
The overapproximation leads to spurious error reports involving impossible execution
paths.
Additional spurious error reports result from eliding details of data variables. In the
Boolean programs approach to model refinement, we would refine the model by
expanding the finite state representation of the process. With FLAVERS, in contrast,
information about the variable value is represented in a separate constraint state
machine, which may be provided by the user or extracted automatically from the
program to be verified. Only violations of property P that satisfy all the constraints Ci are
reported. The same approach of adding constraints is used to eliminate spurious error
reports resulting from the MIP overestimation of possible concurrency.

[4]In

practice the model M may be augmented slightly to facilitate observing significant

events in the constraint, but the augmentation does not restrict or change the possible
behaviors of the model M.
[5]Published

algorithms for computing the "may immediately precede" relation, or the

closely related "may happen in parallel" (MHP) relation, range from O(n3) to O(n6)
where n is the sum of the sizes of the individual state machine models or control flow
graphs. They differ depending on the thread interactions under consideration (e.g., a
MIP calculation for Ada tasks would use diffferent constraints than a MIP calculation for
Java threads) as well as algorithmic approach.

8.8 Data Model Verification with Relational Algebra

Many information systems have relatively simple logic and algorithms, with much of their
complexity in the structure of the data they maintain. A data model is a key design
description for such systems. It is typically described, for example, in the class and
object diagrams of a Unified Modeling Language (UML) design document, possibly
augmented by assertions in the Object Constraint Language (OCL). The finite state
verification techniques we have described are suited to reasoning about complex or
subtle program logic, but are quite limited in dealing with complex data. Fortunately,
suitable finite state verification techniques can also be devised for reasoning about data
models.
The data model consists of sets of data and relations among them. Often a data model
describes many individual relations and constraints; the challenge is in knowing whether
all of the individual constraints are consistent, and whether together they ensure the
desired properties of the system as a whole. Constructing and testing a portion or
partial version of the system may provide some increased confidence in the realizability
of the system, but even with incremental development it can happen that a fundamental
problem in the data model is discovered only after a great deal of development effort
has been invested in the flawed model. Reasoning about the model itself is a more
timely and cost-effective way to find and correct these flaws.
Let us consider, for example, a simple Web site with a data model described as sets
and relations as follows:
A set of pages, divided among restricted, unrestricted, and maintenance pages.
Unrestricted pages are freely accessible, while restricted pages are accessible
only to registered users, and pages in maintenance are currently inaccessible to
both sets of users.
A set of users, classified as administrator, registered, and unregistered users.
A set of links relations among pages. Different relations describe different kinds
of links. Private links lead to restricted pages, public links lead to unrestricted
pages, and maintenance links lead to pages undergoing maintenance.
A set of access rights relations between users and pages, relating different
classes of users to the pages they can access. Unregistered users can access
only unrestricted pages, registered users can access both restricted and
unrestricted pages, and an administrator can access all pages, including pages
under maintenance.
So far we have identified the sets involved in the relations, which we call their signature.
To complete the description we need to indicate the rules that constrain relations among
specific elements. For example we may:

Exclude self loops from "links" relations; that is, specify that a page should not
be directly linked to itself.
Allow at most one type of link between two pages. Note that relations need not
be symmetric; that is, the relation between A and B is distinct from the relation
between B and A, so there can be a link of type private from A to B and a link of
type public from B back to A.
Require the Web site to be connected; that is, require that there be at least one
way of following links from the home page to each other page of the site.
A data model can be visualized as a diagram with nodes corresponding to sets and
edges representing relations, as in Figure 8.14.

Figure 8.14: The data model of a simple Web site.

We can reason about sets and relations using mathematical laws. For example, set
union and set intersection obey many of the same algebraic laws as addition and
subtraction of integers:

A B = B A
commutative law
A B = B A
" "
(A B) C = A (B C)
associative law
(A B) C = A (B C)
" "
A (B C)=(A B) (A C)
distributive law etc.
These and many other laws together make up relational algebra, which is used
extensively in database processing and has many other uses.
It would be inconvenient to write down a data model directly as a collection of
mathematical formulas. Instead, we use some notation whose meaning is the same as
the mathematical formulas, but is easier to write, maintain, and comprehend. Alloy is
one such modeling notation, with the additional advantage that it can be processed by a
finite state verification tool.

The definition of the data model as sets and relations can be formalized and verified with
relational algebra by specifying signatures and constraints. Figure 8.15 presents a
formalization of the data model of the Web site in Alloy. Keyword sig (signature)
identifies three sets: Pages, User, and Site. The definition of set Pages also defines
three disjoint relations among pages: linksPriv (private links), linksPub (public links), and
linksMain (maintenance links). The definition of User also defines a relation between
users and pages. User is partitioned into three disjoint sets (Administrator, Registered,
and Unregistered). The definition of Site aggregates pages into the site and identifies the
home page. Site is defined static since it is a fixed classification of objects.

1 module WebSite
2
3 // Pages include three disjoint sets of links
4 sig Page{ disj linksPriv, linksPub, linksMain: set Page }
5 // Each type of link points to a particular class of page
6 fact connPub{ all p: Page, s: Site | p.linksPub in s.unres }
7 fact connPriv{ all p: Page, s: Site | p.linksPriv in s.res }
8 fact connMain{ all p: Page, s: Site | p.linksMain in s.main
9 // Self loops are not allowed
10 fact noSelfLoop{ no p: Page| p in p.linksPriv+p.linksPub+p.
11
12 // Users are characterized by the set of pages that they ca
13 sig User{ pages: set Page }
14 // Users are partitioned into three sets
15 part sig Administrator, Registered, Unregistered extends Us
16 // Unregistered users can access only the home page, and un
17 fact accUnregistered{
18
all u: Unregistered, s: Site| u.pages = (s.home+s.unres)
19 // Registered users can access home, restricted and unrestr
20 fact accRegistered{
21
all u: Registered, s: Site|
22
u.pages = (s.home+s.res+s.unres)
23 }
24 // Administrators can access all pages
25 fact accAdministrator{
26
all u: Administrator, s: Site|
27
u.pages = (s.home+s.res+s.unres+s.main)
28 }
29
30 // A web site includes one home page and three disjoint set
31 // of pages: restricted, unrestricted and maintenance

32
33
34
35
36
37
38
39

static sig Site{

home: Page,
disj res, unres, main: set Page
}{
// All pages are accessible from the home page ('^' is tr
all p: (res+unres+main)| p in home.^(linksPub+linksPriv+lin
}

Figure 8.15: Alloy model of a Web site with different kinds of pages, users, and
access rights (data model part). Continued in Figure 8.16.

1 module WebSite
39 ...
40 // We consider one Web site that includes one home page
41 // and some other pages
42 fun initSite() {
43
one s: Site| one s.home and
44
some s.res and
45
some s.unres and
46
some s.main
47 }
48
49 // We consider one administrator and some registered and un
50 fun initUsers() {one Administrator and
51
some Registered and
52
some Unregistered}
53
54 fun init() {
55
initSite() and initUsers()
56 }
57
58 // ANALYSIS
59
60 // Verify if there exists a solution
61 // with sets of cardinality at most 5
62 run init for 5
63
64 // check if unregistered users can visit all unrestrited pa
65 // i.e., all unrestricted pages are connected to the home p
66 // at least a path of public links.

67
68
69
70
71
72
73

// Perform analysis with sets of at most 3 objects.

// '*' indicates the transtivie closure including the sourc

assert browsePub{
all p: Page, s: Site| p in s.unres implies s.home in p.
}
check browsePub for 3

Figure 8.16: Alloy model of a Web site with different kinds of pages, users, and
access rights, continued from Figure 8.15.
The keyword facts introduces constraints.[6] The constraints connPub, connPriv and
connMain restrict the target of the links relations, while noSelfLoop excludes links from a
page to itself. The constraints accAdministrator, accRegistered, and accUnregistered
map users to pages. The constraint that follows the definition of Site forces the Web site
to be connected by requiring each page to belong to the transitive closure of links
starting from the Web page (operator ).
A relational algebra specification may be over- or underconstrained. Overconstrained
specifications are not satisfiable by any implementation, while underconstrained
specifications allow undesirable implementations; that is, implementations that violate
important properties.
In general, specifications identify infinite sets of solutions, each characterized by a
different set of objects and relations (e.g., the infinite set of Web sites with different sets
of pages, users and correct relations among them). Thus in general, properties of a
relational specification are undecidable because proving them would require examining
an infinite set of possible solutions. While attempting to prove absence of a solution may
be inconclusive, often a (counter) example that invalidates a property can be found within
a finite set of small models.
We can verify a specification over a finite set of solutions by limiting the cardinality of the
sets. In the example, we first verify that the model admits solutions for sets with at most
five elements (run init for 5 issued after an initialization of the system.) A positive
outcome indicates that the specification is not overconstrained - there are no logical
contradictions. A negative outcome would not allow us to conclude that no solution
exists, but tells us that no "reasonably small" solution exists.
We then verify that the example is not underconstrained with respect to property
browsePub that states that unregistered users must be able to visit all unrestricted
pages by accessing the site from the home page. The property is asserted by requiring
that all unrestricted pages belong to the reflexive transitive closure of the linkPub relation
from the home page (here we use operator * instead of because the home page is
included in the closure). If we check whether the property holds for sets with at most

three elements (check browsePub for 3) we obtain a counter-example like the one
shown in Figure 8.17, which shows how the property can be violated.

Figure 8.17: A Web site that violates the "browsability" property, because public
page Page_2 is not reachable from the home page using only unrestricted links. This
diagram was generated by the Alloy tool.
The simple Web site in the example consists of two unrestricted pages (page_1, the
home page, and Page_2), one restricted page (page_0), and one unregistered user
(user_2). User_2 cannot visit one of the unrestricted pages (Page_2) because the only
path from the home page to Page_2 goes through the restricted page page_0. The
property is violated because unrestricted browsing paths can be "interrupted" by
restricted pages or pages under maintenance, for example, when a previously
unrestricted page is reserved or disabled for maintenance by the administrator.
The problem appears only when there are public links from maintenance or reserved
pages, as we can check by excluding them:
1
2
3

fact descendant{
all p: Page, s: Site| p in s.main+s.res implies no p.linksP
}

This new specification would not find any counter-example in a space of cardinality 3.
We cannot conclude that no larger counter-example exists, but we may be satisfied that
there is no reason to expect this property to be violated only in larger models.

Summary

Finite state verification techniques fill an important niche in verifying critical properties of
programs. They are particularly crucial where nondeterminism makes program testing
ineffective, as in concurrent execution. In principle, finite state verification of concurrent
execution and of data models can be seen as systematically exploring an enormous
space of possible program states. From a user's perspective, the challenge is to
construct a suitable model of the software that can be analyzed with reasonable
expenditure of human and computational resources, captures enough significant detail
for verification to succeed, and can be shown to be consistent with the actual software.

Further Reading
There is a large literature on finite state verification techniques reaching back at least to
the 1960s, when Bartlett et al. [BSW69] employed what is recognizably a manual
version of state space exploration to justify the corrrectness of a communication
protocol. A number of early state space verification tools were developed initially for
communication protocol verification, including the Spin tool. Holzmann's journal
description of Spin's design and use [Hol97], though now somewhat out of date, remains
an adequate introduction to the approach, and a full primer and reference manual
[Hol03] is available in book form.
The ordered binary decision diagram representation of Boolean functions, used in the
first symbolic model checkers, was introduced by Randal Bryant [Bry86]. The
representation of transition relations as OBDDs in this chapter is meant to illustrate
basic ideas but is simplified and far from complete; Bryant's survey paper [Bry92] is a
good source for understanding applications of OBDDs, and Huth and Ryan [HR00]
provide a thorough and clear step-by-step description of how OBDDs are used in the
SMV symbolic model checker.
Model refinement based on iterative refinements of an initial coarse model was
introduced by Ball and Rajamani in the tools Slam [BR01a] and Bebop [BR01b], and by
Henzinger and his colleagues in Blast [HJMS03]. The complementary refinement
approach of FLAVERS was introduced by Dwyer and colleagues [DCCN04].
Automated analysis of relational algebra for data modeling was introduced by Daniel
Jackson and his students with the Alloy notation and associated tools [Jac02].

Exercises
We stated, on the one hand, that finite state verification falls between basic flow
analysis and formal verification in power and cost, but we also stated that finite
state verification techniques are often designed to provide results that are
8.1 tantamount to formal proofs of program properties. Are these two statements
contradictory? If not, how can a technique that is less powerful than formal
verification produce results that are tantamount to formal proofs?

Construct an ordered binary decision diagram (OBDD) for the proposition

8.2

1. How does the size of the OBDD representation of

8.3

differ depending on which variable (x, y,or z) is first in the variable

ordering (i.e., appears in the root node of the OBDD representation)? Is
the size of the OBDD equivalent for some different orderings of the
variables? Why or why not?
2. Predict whether the order of variables would make a difference for

A property like "if the button is pressed, then eventually the elevator will come" is
classified as a liveness property. However, the stronger real-time version "if the
8.4 button is pressed, then the elevator will arrive within 30 seconds" is technically a
safety property rather than a liveness property. Why?
[6]The

order in which relations and constraints are given is irrelevant. We list constraints
after the relations they refer to.

Part III: Problems and Methods

Chapter List
Chapter 9: Test Case Selection and Adequacy
Chapter 10: Functional Testing
Chapter 11: Combinatorial Testing
Chapter 12: Structural Testing
Chapter 13: Data Flow Testing
Chapter 14: Model-Based Testing
Chapter 15: Testing Object-Oriented Software
Chapter 16: Fault-Based Testing
Chapter 17: Test Execution
Chapter 18: Inspection
Chapter 19: Program Analysis

Chapter 9: Test Case Selection and Adequacy

A key problem in software testing is selecting and evaluating test cases. This chapter
introduces basic approaches to test case selection and corresponding adequacy criteria.
It serves as a general introduction to the problem and provides a conceptual framework
for functional and structural approaches described in subsequent chapters.

Required Background
Chapter 2
The fundamental problems and limitations of test case selection are a
consequence of the undecidability of program properties. A grasp of the basic
problem is useful in understanding Section 9.3.

9.1 Overview
Experience suggests that software that has passed a thorough set of
systematic tests is likely to be more dependable than software that
has been only superficially or haphazardly tested. Surely we should
require that each software module or subsystem undergo thorough,
systematic testing before being incorporated into the main product.
But what do we mean by thorough testing? What is the criterion by
which we can judge the adequacy of a suite of tests that a software
artifact has passed?
Ideally, we should like an "adequate" test suite to be one that
ensures correctness of the product. Unfortunately, that goal is not
attainable. The difficulty of proving that some set of test cases is
adequate in this sense is equivalent to the difficulty of proving that
the program is correct. In other words, we could have "adequate"
testing in this sense only if we could establish correctness without
any testing at all.
In practice we settle for criteria that identify inadequacies in test
suites. For example, if the specification describes different treatment
in two cases, but the test suite does not check that the two cases
are in fact treated differently, then we may conclude that the test
suite is inadequate to guard against faults in the program logic. If no
test in the test suite executes a particular program statement, we
might similarly conclude that the test suite is inadequate to guard
against faults in that statement. We may use a whole set of
(in)adequacy criteria, each of which draws on some source of
information about the program and imposes a set of obligations that
an adequate set of test cases ought to satisfy. If a test suite fails to
satisfy some criterion, the obligation that has not been satisfied may
provide some useful information about improving the test suite. If a
set of test cases satisfies all the obligations by all the criteria, we still
do not know definitively that it is a well-designed and effective test
suite, but we have at least some evidence of its thoroughness.

9.2 Test Specifications and Cases

A test case includes not only input data but also any relevant execution conditions and
procedures, and a way of determining whether the program has passed or failed the
test on a particular execution. The term input is used in a very broad sense, which may
include all kinds of stimuli that contribute to determining program behavior. For example,
an interrupt is as much an input as is a file. The pass/fail criterion might be given in the
form of expected output, but could also be some other way of determining whether a
particular program execution is correct.
A test case specification is a requirement to be satisfied by one or more actual test
cases. The distinction between a test case specification and a test case is similar to the
distinction between a program specification and a program. A test case specification
might be met by several different test cases, and vice versa. Suppose, for example, we
are testing a program that sorts a sequence of words. "The input is two or more words"
would be a test case specification, while test cases with the input values "alpha beta"
and "Milano Paris London" would be two among many test cases satisfying the test
case specification. A test case with input "Milano Paris London" would satisfy both the
test case specification "the input is two or more words" and the test case specification
"the input contains a mix of lower- and upper-case alphabetic characters."
Characteristics of the input are not the only thing that might be mentioned in a test case
specification. A complete test case specification includes pass/fail criteria for judging
test execution and may include requirements, drawn from any of several sources of
information, such as system, program, and module interface specifications; source code
or detailed design of the program itself; and records of faults encountered in other
software systems.
Test specifications drawn from system, program, and module interface specifications
often describe program inputs, but they can just as well specify any observable behavior
that could appear in specifications. For example, the specification of a database system
might require certain kinds of robust failure recovery in case of power loss, and test
specifications might therefore require removing system power at certain critical points in
processing. If a specification describes inputs and outputs, a test specification could
prescribe aspects of the input, the output, or both. If the specification is modeled as an
extended finite state machine, it might require executions corresponding to particular
transitions or paths in the state-machine model. The general term for such test
specifications is functional testing, although the term black-box testing and more specific
terms like specification-based testing and model-based testing are also used.
Testing Terms
While the informal meanings of words like "test" may be adequate for everyday
conversation, in this context we must try to use terms in a more precise and

consistent manner. Unfortunately, the terms we will need are not always used
consistently in the literature, despite the existence of an IEEE standard that defines
several of them. The terms we will use are defined as follows.
Test case A test case is a set of inputs, execution conditions, and a pass/fail
criterion. (This usage follows the IEEE standard.)
Test case specification A test case specification is a requirement to be satisfied by
one or more actual test cases. (This usage follows the IEEE standard.)
Test obligation A test obligation is a partial test case specification, requiring some
property deemed important to thorough testing. We use the term obligation to
distinguish the requirements imposed by a test adequacy criterion from more
complete test case specifications.
Test suite A test suite is a set of test cases. Typically, a method for functional
testing is concerned with creating a test suite. A test suite for a program, system, or
individual unit may be made up of several test suites for individual modules,
subsystems, or features. (This usage follows the IEEE standard.)
Test or test execution We use the term test or test execution to refer to the activity
of executing test cases and evaluating their results. When we refer to "a test," we
mean execution of a single test case, except where context makes it clear that the
reference is to execution of a whole test suite. (The IEEE standard allows this and
other definitions.)
Adequacy criterion A test adequacy criterion is a predicate that is true (satisfied) or
false (not satisfied) of a program, test suite pair. Usually a test adequacy
criterion is expressed in the form of a rule for deriving a set of test obligations from
another artifact, such as a program or specification. The adequacy criterion is then
satisfied if every test obligation is satisfied by at least one test case in the suite.

Test specifications drawn from program source code require coverage of particular
elements in the source code or some model derived from it. For example, we might
require a test case that traverses a loop one or more times. The general term for testing
based on program structure is structural testing, although the term white-box testing or
glass-box testing is sometimes used.
Previously encountered faults can be an important source of information regarding useful
test cases. For example, if previous products have encountered failures or security
breaches due to buffer overflows, we may formulate test requirements specifically to
check handling of inputs that are too large to fit in provided buffers. These fault-based
test specifications usually draw also from interface specifications, design models, or

source code, but add test requirements that might not have been otherwise considered.
A common form of fault-based testing is fault-seeding, purposely inserting faults in
source code and then measuring the effectiveness of a test suite in finding the seeded
faults, on the theory that a test suite that finds seeded faults is likely also to find other
faults.
Test specifications need not fall cleanly into just one of the categories. For example, test
specifications drawn from a model of a program might be considered specificationbased if the model is produced during program design, or structural if it is derived from
the program source code.
Consider the Java method of Figure 9.1. We might apply a general rule that requires
using an empty sequence wherever a sequence appears as an input; we would thus
create a test case specification (a test obligation) that requires the empty string as
input.[1] If we are selecting test cases structurally, we might create a test obligation that
requires the first clause of the if statement on line 15 to evaluate to true and the second
clause to evaluate to false, and another test obligation on which it is the second clause
that must evaluate to true and the first that must evaluate to false.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

/**
* Remove/collapse multiple spaces.
*
* @param String string to remove multiple spaces from.
* @return String
*/
public static String collapseSpaces(String argStr)
{
char last = argStr.charAt(0);
StringBuffer argBuf = new StringBuffer();
for (int cIdx=0; cIdx < argStr.length(); cIdx++)
{
char ch = argStr.charAt(cIdx);
if (ch != '' || last != '')
{
argBuf.append(ch);
last = ch;
}
}
return argBuf.toString();
}

Figure 9.1: A Java method for collapsing sequences of blanks, excerpted from the
StringUtils class of Velocity version 1.3.1, an Apache Jakarta project. Apache
Group, used by permission.
[1]Constructing

and using catalogs of general rules like this is described in Chapter 10.

9.3 Adequacy Criteria

We have already noted that adequacy criteria are just imperfect but
useful indicators of inadequacies, so we may not always wish to use
them directly to generate test specifications from which actual test
cases are drawn. We will use the term test obligation for test
specifications imposed by adequacy criteria, to distinguish them from
test specifications that are actually used to derive test cases. Thus,
the usual situation will be that a set of test cases (a test suite) is
created using a set of test specifications, but then the adequacy of
that test suite is measured using a different set of test obligations.
We say a test suite satisfies an adequacy criterion if all the tests
succeed and if every test obligation in the criterion is satisfied by at
least one of the test cases in the test suite. For example, the
statement coverage adequacy criterion is satisfied by a particular
test suite for a particular program if each executable statement in
the program (i.e., excluding comments and declarations) is executed
by at least one test case in the test suite. A fault-based adequacy
criterion that seeds a certain set of faults would be satisfied if, for
each of the seeded faults, there is a test case that passes for the
original program but fails for the program with (only) that seeded
fault.
It is quite possible that no test suite will satisfy a particular test
adequacy criterion for a particular program. For example, if the
program contains statements that can never be executed (perhaps
because it is part of a sanity check that can be executed only if
some other part of the program is faulty), then no test suite can
satisfy the statement coverage criterion. Analogous situations arise
regardless of the sources of information used in devising test
adequacy criteria. For example, a specification-based criterion may
require combinations of conditions drawn from different parts of the
specification, but not all combinations may be possible.
One approach to overcoming the problem of unsatisfiable test
obligations is to simply exclude any unsatisfiable obligation from a
criterion. For example, the statement coverage criterion can be

modified to require execution only of statements that can be

executed. The question of whether a particular statement or
program path is executable, or whether a particular combination of
clauses in a specification is satisfiable, or whether a program with a
seeded error actually behaves differently from the original program,
are all provably undecidable in the general case. Thus, while tools
may be some help in distinguishing feasible from infeasible test
obligations, in at least some cases the distinction will be left to
fallible human judgment.
If the number of infeasible test obligations is modest, it can be practical to identify each of
them, and to ameliorate human fallibility through peer review. If the number of infeasible test
obligations is large, it becomes impractical to carefully reason about each to avoid excusing
an obligation that is feasible but difficult to satisfy. A common practice is to measure the
extent to which a test suite approaches an adequacy criterion. For example, if an adequacy
criterion based on control flow paths in a program unit induced 100 distinct test obligations,
and a test suite satisfied 85 of those obligations, then we would say that we had reached
85% coverage of the test obligations.
Quantitative measures of test coverage are widely used in industry. They are simple and
cheap to calculate, provide some indication of progress toward thorough testing, and
project an aura of objectivity. In managing software development, anything that produces a
number can be seductive. One must never forget that coverage is a rough proxy measure
for the thoroughness and effectiveness of test suites. The danger, as with any proxy
measure of some underlying goal, is the temptation to improve the proxy measure in a way
that does not actually contribute to the goal. If, for example, 80% coverage of some
adequacy criterion is required to declare a work assignment complete, developers under
time pressure will almost certainly yield to the temptation to design tests specifically to that
criterion, choosing the simplest test cases that achieve the required coverage level. One
cannot entirely avoid such distortions, but to the extent possible one should guard against
them by ensuring that the ultimate measure of performance is preventing faults from
surviving to later stages of development or deployment.

9.4 Comparing Criteria

It would be useful to know whether one test adequacy criterion was more effective than
another in helping find program faults, and whether its extra effectiveness was
worthwhile with respect to the extra effort expended to satisfy it. One can imagine two
kinds of answers to such a question, empirical and analytical. An empirical answer would
be based on extensive studies of the effectiveness of different approaches to testing in
industrial practice, including controlled studies to determine whether the relative
effectiveness of different testing methods depends on the kind of software being tested,
the kind of organization in which the software is developed and tested, and a myriad of
other potential confounding factors. The empirical evidence available falls short of
providing such clear-cut answers. An analytical answer to questions of relative
effectiveness would describe conditions under which one adequacy criterion is
guaranteed to be more effective than another, or describe in statistical terms their
relative effectiveness.
Analytic comparisons of the strength of test coverage depends on a precise definition of
what it means for one criterion to be "stronger" or "more effective" than another. Let us
first consider single test suites. In the absence of specific information, we cannot
exclude the possibility that any test case can reveal a failure. A test suite TA that does
not include all the test cases of another test suite TB may fail revealing the potential
failure exposed by the test cases that are in TB but not in TA. Thus, if we consider only
the guarantees that a test suite provides, the only way for one test suite TA to be
stronger than another suite TB is to include all test cases of TB plus additional ones.
Many different test suites might satisfy the same coverage criterion. To compare
criteria, then, we consider all the possible ways of satisfying the criteria. If every test
suite that satisfies some criterion A is a superset of some test suite that satisfies
criterion B, or equivalently, every suite that satisfies A also satisfies B, then we can say
that A "subsumes" B.
Test coverage criterion A subsumes test coverage criterion B iff, for every program P,
every test set satisfying A with respect to P also satisfies B with respect to P.
In this case, if we satisfy criterion C1, there is no point in measuring adequacy with
respect to C2. For example, a structural criterion that requires exploring all outcomes of
conditional branches subsumes statement coverage. Likewise, a specification-based
criterion that requires use of a set of possible values for attribute A and, independently,
for attribute B, will be subsumed by a criterion that requires all combinations of those
values.
Consider again the example of Figure 9.1. Suppose we apply an adequacy criterion that
imposes an obligation to execute each statement in the method. This criterion can be

met by a test suite containing a single test case, with the input value (value of argStr)
being "doesn'tEvenHaveSpaces." Requiring both the true and false branches of each
test to be taken subsumes the previous criterion and forces us to at least provide an
input with a space that is not copied to the output, but it can still be satisfied by a suite
with just one test case. We might add a requirement that the loop be iterated zero
times, once, and several times, thus requiring a test suite with at least three test cases.
The obligation to execute the loop body zero times would force us to add a test case
with the empty string as input, and like the specification-based obligation to consider an
empty sequence, this would reveal a fault in the code.
Should we consider a more demanding adequacy criterion, as indicated by the
subsumes relation among criteria, to be a better criterion? The answer would be "yes" if
we were comparing the guarantees provided by test adequacy criteria: If criterion A
subsumes criterion B, and if any test suite satisfying B in some program is guaranteed to
find a particular fault, then any test suite satisfying A is guaranteed to find the same fault
in the program. This is not as good as it sounds, though. Twice nothing is nothing.
Adequacy criteria do not provide useful guarantees for fault detection, so comparing
guarantees is not a useful way to compare criteria.
A better statistical measure of test effectiveness is whether the probability of finding at
least one program fault is greater when using one test coverage criterion than another.
Of course, such statistical measures can be misleading if some test coverage criteria
require much larger numbers of test cases than others. It is hardly surprising if a
criterion that requires at least 300 test cases for program P is more effective, on
average, than a criterion that requires at least 50 test cases for the same program. It
would be better to know, if we have 50 test cases that satisfy criterion B, is there any
value in finding 250 test cases to finish satisfying the "stronger" criterion A, or would it
be just as profitable to choose the additional 250 test cases at random?
Although theory does not provide much guidance, empirical studies of particular test
adequacy criteria do suggest that there is value in pursuing stronger criteria, particularly
when the level of coverage attained is very high. Whether the extra value of pursuing a
stronger adequacy criterion is commensurate with the cost almost certainly depends on
a plethora of particulars, and can only be determined by monitoring results in individual
organizations.

Open Research Issues

A good deal of theoretical research has been done on what one can conclude about test
effectiveness from test adequacy criteria. Most of the results are negative. In general,
one cannot be certain that a test suite that meets any practical test adequacy criterion
ensures correctness, or even that it is more effective at finding faults than another test
suite that does not meet the criterion. While theoretical characterization of test adequacy
criteria and their properties was once an active research area, interest has waned, and

it is likely that future theoretical progress must begin with a quite different conception of
the fundamental goals of a theory of test adequacy.
The trend in research is toward empirical, rather than theoretical, comparison of the
effectiveness of particular test selection techniques and test adequacy criteria. Empirical
approaches to measuring and comparing effectiveness are still at an early stage. A
major open problem is to determine when, and to what extent, the results of an empirical
assessment can be expected to generalize beyond the particular programs and test
suites used in the investigation. While empirical studies have to a large extent displaced
theoretical investigation of test effectiveness, in the longer term useful empirical
investigation will require its own theoretical framework.

Further Reading
Goodenough and Gerhart made the original attempt to formulate a theory of "adequate"
testing [GG75]; Weyuker and Ostrand extended this theory to consider when a set of
test obligations is adequate to ensure that a program fault is revealed [WO80].
Gourlay's exposition of a mathematical framework for adequacy criteria is among the
most lucid developments of purely analytic characterizations [Gou83]. Hamlet and Taylor
show that, if one takes statistical confidence in (absolute) program correctness as the
goal, none of the standard coverage testing techniques improve on random testing
[HT90], from which an appropriate conclusion is that confidence in absolute correctness
is not a reasonable goal of systematic testing. Frankl and Iakounenko's study of test
effectiveness [FI98] is a good example of the development of empirical methods for
assessing the practical effectiveness of test adequacy criteria.

Related Topics
Test adequacy criteria and test selection techniques can be categorized by the sources
of information they draw from. Functional testing draws from program and system
specifications, and is described in Chapters 10, 11, and 14. Structural testing draws
from the structure of the program or system, and is described in Chapters 12 and 13.
The techniques for testing object-oriented software described in Chapter 15 draw on
both functional and structural approaches. Selection and adequacy criteria based on
consideration of hypothetical program faults are described in Chapter 16.

Exercises
Deterministic finite state machines (FSMs), with states representing classes of
program states and transitions representing external inputs and observable
program actions or outputs, are sometimes used in modeling system requirements.
We can design test cases consisting of sequences of program inputs that trigger
FSM transitions and the predicted program actions expected in response. We can

also define test coverage criteria relative to such a model. Which of the following
coverage criteria subsume which others?
State coverage For each state in the FSM model, there is a test case that visits
9.1 that state.
Transition coverage For each transition in the FSM model, there is a test case
that traverses that transition.
Path coverage For all finite-length subpaths from a distinguished start state in the
FSM model, there is at least one test case that includes a corresponding subpath.
State-pair coverage For each state r in the FSM model, for each state s
reachable from r along some sequence of transitions, there is at least one test
case that passes through state r and then reaches state s.
Adequacy criteria may be derived from specifications (functional criteria) or code
(structural criteria). The presence of infeasible elements in a program may make it
impossible to obtain 100% coverage. Since we cannot possibly cover infeasible
elements, we might define a coverage criterion to require 100% coverage of
feasible elements (e.g., execution of all program statements that can actually be
reached in program execution). We have noted that feasibility of program elements
9.2 is undecidable in general. Suppose we instead are using a functional test adequacy
criterion, based on logical conditions describing inputs and outputs. It is still
possible to have infeasible elements (logical condition A might be inconsitent with
logical condition B, making the conjunction A B infeasible). Would you expect
distinguishing feasible from infeasible elements to be easier or harder for functional
criteria, compared to structural criteria? Why?
Suppose test suite A satisfies adequacy criterion C1. Test suite B satisfies
9.3 adequacy criterion C2, and C2 subsumes C1. Can we be certain that faults revealed
by A will also be revealed by B?

Chapter 10: Functional Testing

A functional specification is a description of intended program[1] behavior, distinct from
the program itself. Whatever form the functional specification takes - whether formal or
informal - it is the most important source of information for designing tests. Deriving test
cases from program specifications is called functional testing.
Functional testing, or more precisely, functional test case design, attempts to answer
the question "What test cases shall I use to exercise my program?" considering only the
specification of a program and not its design or implementation structure. Being based
on program specifications and not on the internals of the code, functional testing is also
called specification-based or black-box testing.
Functional testing is typically the base-line technique for designing test cases, for a
number of reasons. Functional test case design can (and should) begin as part of the
requirements specification process, and continue through each level of design and
interface specification; it is the only test design technique with such wide and early
applicability. Moreover, functional testing is effective in finding some classes of fault that
typically elude so-called white-box or glass-box techniques of structural or fault-based
testing. Functional testing techniques can be applied to any description of program
behavior, from an informal partial description to a formal specification, and at any level
of granularity from module to system testing. Finally, functional test cases are typically
less expensive to design and execute than white-box tests.

10.1 Overview
In testing and analysis aimed at verification[2] - that is, at finding any discrepancies
between what a program does and what it is intended to do - one must obviously refer
to requirements as expressed by users and specified by software engineers. A
functional specification, that is, a description of the expected behavior of the program, is
the primary source of information for test case specification.
Functional testing, also known as black-box or specification-based testing, denotes
techniques that derive test cases from functional specifications. Usually functional testing
techniques produce test case specifications that identify classes of test cases and are
instantiated to produce individual test cases.
The core of functional test case design is partitioning[3] the possible behaviors of the
program into a finite number of homogeneous classes, where each such class can
reasonably be expected to be consistently correct or incorrect. In practice, the test case
designer often must also complete the job of formalizing the specification far enough to
serve as the basis for identifying classes of behaviors. An important side benefit of test
design is highlighting the weaknesses and incompleteness of program specifications.
Deriving functional test cases is an analytical process that decomposes specifications
into test cases. The myriad aspects that must be taken into account during functional
test case specification makes the process error prone. Even expert test designers can
miss important test cases. A methodology for functional test design helps by
decomposing the functional test design process into elementary steps. In this way, it is
possible to control the complexity of the process and to separate human intensive
activities from activities that can be automated.
Sometimes, functional testing can be fully automated. This is possible, for example,
when specifications are given in terms of some formal model, such as a grammar or an
extended state machine specification. In these (exceptional) cases, the creative work is
performed during specification and design of the software. The test designer's job is
then limited to the choice of the test selection criteria, which defines the strategy for
generating test case specifications. In most cases, however, functional test design is a
human intensive activity. For example, when test designers must work from informal
specifications written in natural language, much of the work is in structuring the
specification adequately for identifying test cases.
[1]We

use the term program generically for the artifact under test, whether that artifact is
a complete application or an individual unit together with a test harness. This is
consistent with usage in the testing research literature.
[2]Here

we focus on software verification as opposed to validation (see Chapter 2). The

problems of validating the software and its specifications, that is, checking the program

behavior and its specifications with respect to the users' expectations, is treated in
Chapter 22.
[3]We

are using the term partition in a common but rather sloppy sense. A true partition
would form disjoint classes, the union of which is the entire space. Partition testing
separates the behaviors or input space into classes whose union is the entire space, but
the classes may not be disjoint.

10.2 Random versus Partition Testing

Strategies
With few exceptions, the number of potential test cases for a given
program is unimaginably huge - so large that for all practical
purposes it can be considered infinite. For example, even a simple
function whose input arguments are two 32-bit integers has 264
1054 legal inputs. In contrast to input spaces, budgets and
schedules are finite, so any practical method for testing must select
an infinitesimally small portion of the complete input space.
Some test cases are better than others, in the sense that some
reveal faults and others do not.[4] Of course, we cannot know in
advance which test cases reveal faults. At a minimum, though, we
can observe that running the same test case again is less likely to
reveal a fault than running a different test case, and we may
reasonably hypothesize that a test case that is very different from
the test cases that precede it is more valuable than a test case that
is very similar (in some sense yet to be defined) to others.
Functional vs. Structural Testing
Test cases and test suites can be derived from several sources of information, including
specifications (functional and model-based testing), detailed design and source code
(structural testing), and hypothesized defects (fault-based testing). Functional test case
design is an indispensable base of a good test suite, complemented but never replaced by
structural and fault-based testing, because there are classes of faults that only functional
testing effectively detects. Omission of a feature, for example, is unlikely to be revealed by
techniques that refer only to the code structure.
Consider a program that is supposed to accept files in either plain ASCII text, or HTML, or
PDF formats and generate standard Postscript. Suppose the programmer overlooks the
PDF functionality, so that the program accepts only plain text and HTML files. Intuitively, a
functional testing criterion would require at least one test case for each item in the
specification, regardless of the implementation; that is, it would require the program to be
exercised with at least one ASCII, one HTML, and one PDF file, thus easily revealing the
failure due to the missing code. In contrast, criteria based solely on the code would not
require the program to be exercised with a PDF file, since each part of the code can be
exercised without attempting to use that feature. Similarly, fault- based techniques, based
on potential faults in design or coding, would not have any reason to indicate a PDF file as a

potential input even if "missing case" were included in the catalog of potential faults.
Functional specifications often address semantically rich domains, and we can use domain
information in addition to the cases explicitly enumerated in the program specification. For
example, while a program may manipulate a string of up to nine alphanumeric characters,
the program specification may reveal that these characters represent a postal code, which
immediately suggests test cases based on postal codes of various localities. Suppose the
program logic distinguishes only two cases, depending on whether they are found in a table
of U.S. zip codes. A structural testing criterion would require testing of valid and invalid U.S.
zip codes, but only consideration of the specification and richer knowledge of the domain
would suggest test cases that reveal missing logic for distinguishing between U.S.-bound
mail with invalid U.S. zip codes and mail bound for other countries.
Functional testing can be applied at any level of granularity where some form of
specification is available, from overall system testing to individual units, although the level of
granularity and the type of software influence the choice of the specification styles and
notations, and consequently the functional testing techniques that can be used.
In contrast, structural and fault-based testing techniques are invariably tied to program
structures at some particular level of granularity and do not scale much beyond that level.
The most common structural testing techniques are tied to fine-grain program structures
(statements, classes, etc.) and are applicable only at the level of modules or small
collections of modules (small subsystems, components, or libraries).

As an extreme example, suppose we are allowed to select only three test cases for a
program that breaks a text buffer into lines of 60 characters each. Suppose the first test
case is a buffer containing 40 characters, and the second is a buffer containing 30
characters. As a final test case, we can choose a buffer containing 16 characters or a
buffer containing 100 characters. Although we cannot prove that the 100-character buffer is
the better test case (and it might not be; the fact that 16 is a power of 2 might have some
unforeseen significance), we are naturally suspicious of a set of tests that is strongly biased
toward lengths less than 60.

Accidental bias may be avoided by choosing test cases from a

random distribution. Random sampling is often an inexpensive way
to produce a large number of test cases. If we assume absolutely no
knowledge on which to place a higher value on one test case than
another, then random sampling maximizes value by maximizing the
number of test cases that can be created (without bias) for a given
budget. Even if we do possess some knowledge suggesting that
some cases are more valuable than others, the efficiency of random
sampling may in some cases outweigh its inability to use any

knowledge we may have.

Consider again the line-break program, and suppose that our budget
is one day of testing effort rather than some arbitrary number of
test cases. If the cost of random selection and actual execution of
test cases is small enough, then we may prefer to run a large
number of random test cases rather than expending more effort on
each of a smaller number of test cases. We may in a few hours
construct programs that generate buffers with various contents and
lengths up to a few thousand characters, as well as an automated
procedure for checking the program output. Letting it run
unattended overnight, we may execute a few million test cases. If
the program does not correctly handle a buffer containing a
sequence of more than 60 nonblank characters (a single "word" that
does not fit on a line), we are likely to encounter this case by sheer
luck if we execute enough random tests, even without having
explicitly considered this case.
Even a few million test cases is an infinitesimal fraction of the complete input space of most
programs. Large numbers of random tests are unlikely to find failures at single points
(singularities) in the input space. Consider, for example, a simple procedure for returning
the two roots of a quadratic equation ax2 +bx +c = 0 and suppose we choose test inputs
(values of the coefficients a, b, and c) from a uniform distribution ranging from 10.0 to
10.0. While uniform random sampling would certainly cover cases in which b2 4ac > 0
(where the equation has no real roots), it would be very unlikely to test the case in which a
= 0 and b = 0, in which case a naive implementation of the quadratic formula

will divide by zero (see Figure 10.1).

1 /** Find the two roots of ax^2 + bx + c,
2 * that is, the values of x for which the result is 0.
3 */
4 class Roots {
5
double root one, root two;
6
int num roots;
7
public roots(double a, double b, double c) {
8
double q;
9
double r;
10
// Apply the textbook quadratic formula:

11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34 }

// Roots = -b +- sqrt(b^2 - 4ac) / 2a

q = b*b - 4*a*c;
if (q > 0&&a!=0) {
// If b^2 > 4ac, there are two distinct roo
num roots = 2;
r=(double) Math.sqrt(q) ;
root one = ((0-b) + r)/(2*a);
root two = ((0-b) - r)/(2*a);
} else if(q==0) { // (BUG HERE)
// The equation has exactly one root
num roots = 1;
root one = (0-b)/(2*a);
root two = root one;
} else {
// The equation has no roots if b^2 < 4ac
num roots = 0;
root one = -1;
root two = -1;
}
}
public int num roots() { return num roots; }
public double first root() { return root one; }
public double second root() { return root two; }

Figure 10.1: The Java class roots, which finds roots of a quadratic
equation. The case analysis in the implementation is incomplete: It
does not properly handle the case in which b2 4ac = 0 and a = 0.
We cannot anticipate all such faults, but experience teaches that
boundary values identifiable in a specification are disproportionately
valuable. Uniform random generation of even large numbers of test
cases is ineffective at finding the fault in this program, but selection
of a few "special values" based on the specification quickly uncovers
it.
Of course, it is unlikely that anyone would test only with random values. Regardless of the
overall testing strategy, most test designers will also try some "special" values. The test
designer's intuition comports with the observation that random sam-pling is an ineffective
way to find singularities in a large input space. The observation about singularities can be
generalized to any characteristic of input data that defines an infinitesimally small portion of
the complete input data space. If again we have just three real-valued inputs a, b, and c,

there is an infinite number of choices for which b = c, but random sampling is unlikely to
generate any of them because they are an infinitesimal part of the complete input data
space.
The observation about special values and random samples is by no means limited to
numbers. Consider again, for example, breaking a text buffer into lines. Since line breaks
are permitted at blanks, we would consider blanks a "special" value for this problem. While
random sampling from the character set is likely to produce a buffer containing a sequence
of at least 60 nonblank characters, it is much less likely to produce a sequence of 60
blanks.

The reader may justifiably object that a reasonable test designer

would not create text buffer test cases by sampling uniformly from
the set of all characters. The designer would instead classify
characters depending on their treatment, lumping alphabetic
characters into one class and white space characters into another.
In other words, a test designer will partition the input space into
classes and will then generate test data in a manner that is likely to
choose data from each partition. Test designers seldom use pure
random sampling; usually they exploit some knowledge of
application semantics to choose samples that are more likely to
include "special" or trouble-prone regions of the input space.
Partition testing separates the input space into classes whose union is the entire space, but
the classes may not be disjoint (and thus the term partition is not mathematically accurate,
although it has become established in testing terminology). Figure 10.2 illustrates a
desirable case: All inputs that lead to a failure belong to at least one class that contains
only inputs that lead to failures. In this case, sampling each class in the quasi-partition
selects at least one input that leads to a failure, revealing the fault. We could easily turn the
quasi-partition of Figure 10.2 into a true partition, by considering intersections among the
classes, but sampling in a true partition would not improve the efficiency or effectiveness of
testing.

Figure 10.2: A quasi-partition of a program's input space. Black

circles represent inputs that lead to failures. All elements of the
input domain belong to at least one class, but classes are not
disjoint.
A testing method that divides the infinite set of possible test cases into a finite set of
classes, with the purpose of drawing one or more test cases from each class, is called a
partition testing method. When partitions are chosen according to information in the
specification, rather than the design or implementation, it is called specificationbased
partition testing, or more briefly, functional testing. Note that not all testing of product
functionality is "functional testing." Rather, the term is used specifically to refer to
systematic testing based on a functional specification. It excludes ad hoc and random
testing, as well as testing based on the structure of a design or implementation.

Partition testing typically increases the cost of each test case, since
in addition to generation of a set of classes, creation of test cases
from each class may be more expensive than generating random
test data. In consequence, partition testing usually produces fewer
test cases than random testing for the same expenditure of time
and money. Partitioning can therefore be advantageous only if the
average value (fault detection effectiveness) is greater.
If we were able to group together test cases with such perfect knowledge that the outcome
of test cases in each class were uniform (either all successes or all failures), then partition
testing would be at its theoretical best. In general we cannot do that, nor can we even
quantify the uniformity of classes of test cases. Partitioning by any means, including
specification-based partition testing, is always based on experience and judgment that

leads one to believe that certain classes of test case are "more alike" than others, in the
sense that failure-prone test cases are likely to be concentrated in some classes. When we
appealed earlier to the test designer's intuition that one should try boundary cases and
special values, we were actually appealing to a combination of experience (many failures
occur at boundary and special cases) and knowledge that identifiable cases in the
specification often correspond to classes of input that require different treatment by an
implementation.
Given a fixed budget, the optimum may not lie in only partition testing or only random
testing, but in some mix that makes use of available knowledge. For example, consider
again the simple numeric problem with three inputs, a, b, and c. We might consider a few
special cases of each input, individually and in combination, and we might consider also a
few potentially significant relationships (e.g., a = b). If no faults are revealed by these few
test cases, there is little point in producing further arbitrary partitions - one might then turn
to random generation of a large number of test cases.
[4]Note that the relative value of different test cases would be quite

different if our goal were to measure dependability, rather than

finding faults so that they can be repaired.

10.3 A Systematic Approach

Deriving test cases from functional specifications is a complex analytical process that
partitions the input space described by the program specification. Brute force generation
of test cases, that is, direct generation of test cases from program specifications,
seldom produces acceptable results: Test cases are generated without particular
criteria, and determining the adequacy of the generated test cases is almost impossible.
Brute force generation of test cases relies on test designers' expertise and is a process
that is difficult to monitor and repeat. A systematic approach simplifies the overall
process by dividing it into elementary steps, thus decoupling different activities, dividing
brain-intensive from automatable steps, suggesting criteria to identify adequate sets of
test cases, and providing an effective means of monitoring the testing activity.
Although suitable functional testing techniques can be found for any granularity level, a
particular functional testing technique may be effective only for some kinds of software
or may require a given specification style. For example, a combinatorial approach may
work well for functional units characterized by a large number of relatively independent
inputs, but may be less effective for functional units characterized by complex
interrelations among inputs. Functional testing techniques designed for a given
specification notation, for example, finite state machines or grammars, are not easily
applicable to other specification styles. Nonetheless, we can identify a general pattern of
activities that captures the essential steps in a variety of different functional test design
techniques. By describing particular functional testing techniques as instantiations of this
general pattern, relations among the techniques may become clearer, and the test
designer may gain some insight into adapting and extending these techniques to the
characteristics of other applications and situations.
Figure 10.3 identifies the general steps of systematic approaches. The steps may be
difficult or trivial depending on the application domain and the available program
specifications. Some steps may be omitted depending on the application domain, the
available specifications and the test designers' expertise. Instances of the process can
be obtained by suitably instantiating different steps. Although most techniques are
presented and applied as stand-alone methods, it is also possible to mix and match
steps from different techniques, or to apply different methods for different parts of the
system to be tested.

Figure 10.3: The main steps of a systematic approach to functional program

testing.
Identify Independently Testable Features Functional specifications can be large and
complex. Usually, complex specifications describe systems that can be decomposed into
distinct features. For example, the specification of a Web site may include features for
searching the site database, registering users' profiles, getting and storing information
provided by the users in different forms, and so on. The specification of each of these
features may comprise several functionalities. For example, the search feature may
include functionalities for editing a search pattern, searching the database with a given
pattern, and so on. Although it is possible to design test cases that exercise several
functionalities at once, designing different test cases for different functionalities can
simplify the test generation problem, allowing each functionality to be examined
separately. Moreover, it eases locating faults that cause the revealed failures. It is thus
recommended to devise separate test cases for each functionality of the system,
whenever possible.
The preliminary step of functional testing consists in partitioning the specifications into
features that can be tested separately. This can be an easy step for well-designed,
modular specifications, but informal specifications of large systems may be difficult to
decompose into independently testable features. Some degree of formality, at least to
the point of careful definition and use of terms, is usually required.
Units and Features
Programs and software systems can be decomposed in different ways. For testing,
we may consider externally observable behavior (features), or the structure of the
software system (units, subsystems, and components).

Independently testable feature An independently testable feature (ITF) is a

functionality that can be tested independently of other functionalities of the software
under test. It need not correspond to a unit or subsystem of the software. For
example, a file sorting utility may be capable of merging two sorted files, and it may
be possible to test the sorting and merging functionalities separately, even though
both features are implemented by much of the same source code. (The nearest IEEE
standard term is test item.)
As functional testing can be applied at many different granularity levels, from unit
testing through integration and system testing, so ITFs may range from the
functionality of an individual Java class or C function up to features of an integrated
system composed of many complete programs. The granularity of an ITF depends on
the exposed interface at whichever granularity is being tested. For example, individual
methods of a class are part of the interface of the class, and a set of related
methods (or even a single method) might be an ITF for unit testing, but for system
testing the ITFs would be features visible through a user interface or application
programming interface.
Unit We reserve the term unit, not for any fixed syntactic construct in a particular
programming language, but for the smallest unit of work assignment in a software
project. Defining "unit" in this manner, rather than (for example) equating units with
individual Java classes or packages, or C files or functions, reflects a philosophy
about test and analysis. A work unit is the smallest increment by which a software
system grows or changes, the smallest unit that appears in a project schedule and
budget, and the smallest unit that may reasonably be associated with a suite of test
cases.
It follows from our definition of "unit" that, when we speak of unit testing, we mean the
testing associated with an individual work unit.
We reserve the term function for the mathematical concept, that is, a set of ordered
pairs having distinct first elements. When we refer to "functions" as syntactic
elements in some programming language, we will qualify it to distinguish that usage
from the mathematical concept. A "function" is a set of ordered pairs but a "C
function" is a syntactic element in the C programming language.

Identification of functional features that can be tested separately is different from module
decomposition. In both cases we apply the divide and conquer principle, but in the
former case, we partition specifications according to the functional behavior as
perceived by the users of the software under test,[5] while in the latter, we identify logical
units that can be implemented separately. For example, a Web site may require a sort
function, as a service routine, that does not correspond to an external functionality. The

sort function may be a functional feature at module testing, when the program under test
is the sort function itself, but is not a functional feature at system test, while deriving test
cases from the specifications of the whole Web site. On the other hand, the registration
of a new user profile can be identified as one of the functional features at system-level
testing, even if such functionality is spread across several modules. Thus, identifying
functional features does not correspond to identifying single modules at the design level,
but rather to suitably slicing the specifications to attack their complexity incrementally.
Independently testable features are described by identifying all the inputs that form their
execution environments. Inputs may be given in different forms depending on the notation
used to express the specifications. In some cases they may be easily identifiable. For
example, they can be the input alphabet of a finite state machine specifying the behavior
of the system. In other cases, they may be hidden in the specification. This is often the
case for informal specifications, where some inputs may be given explicitly as
parameters of the functional unit, but other inputs may be left implicit in the description.
For example, a description of how a new user registers at a Web site may explicitly
indicate the data that constitutes the user profile to be inserted as parameters of the
functional unit, but may leave implicit the collection of elements (e.g., database) in which
the new profile must be inserted.
Trying to identify inputs may help in distinguishing different functions. For example, trying
to identify the inputs of a graphical tool may lead to a clearer distinction between the
graphical interface per se and the associated callbacks to the application. With respect
to the Web-based user registration function, the data to be inserted in the database are
part of the execution environment of the functional unit that performs the insertion of the
user profile, while the combination of fields that can be used to construct such data is
part of the execution environment of the functional unit that takes care of the
management of the specific graphical interface.
Identify Representative Classes of Values or Derive a Model The execution
environment of the feature under test determines the form of the final test cases, which
are given as combinations of values for the inputs to the unit. The next step of a testing
process consists of identifying which values of each input should be selected to form test
cases. Representative values can be identified directly from informal specifications
expressed in natural language. Alternatively, representative values may be selected
indirectly through a model, which can either be produced only for the sake of testing or
be available as part of the specification. In both cases, the aim of this step is to identify
the values for each input in isolation, either explicitly through enumeration or implicitly
trough a suitable model, but not to select suitable combinations of such values (i.e., test
case specifications). In this way, we separate the problem of identifying the
representative values for each input from the problem of combining them to obtain
meaningful test cases, thus splitting a complex step into two simpler steps.
Most methods that can be applied to informal specifications rely on explicit enumeration

of representative values by the test designer. In this case, it is very important to

consider all possible cases and take advantage of the information provided by the
specification. We may identify different categories of expected values, as well as
boundary and exceptional or erroneous values. For example, when considering
operations on a nonempty list of elements, we may distinguish the cases of the empty
list (an error value) and a singleton element (a boundary value) as special cases. Usually
this step determines characteristics of values (e.g., any list with a single element) rather
than actual values.
Implicit enumeration requires the construction of a (partial) model of the specifications.
Such a model may be already available as part of a specification or design model, but
more often it must be constructed by the test designer, in consultation with other
designers. For example, a specification given as a finite state machine implicitly identifies
different values for the inputs by means of the transitions triggered by the different
values. In some cases, we can construct a partial model as a means of identifying
different values for the inputs. For example, we may derive a grammar from a
specification and thus identify different values according to the legal sequences of
productions of the given grammar.
Directly enumerating representative values may appear simpler and less expensive than
producing a suitable model from which values may be derived. However, a formal model
may also be valuable in subsequent steps of test case design, including selection of
combinations of values. Also, a formal model may make it easier to select a larger or
smaller number of test cases, balancing cost and thoroughness, and may be less costly
to modify and reuse as the system under test evolves. Whether to invest effort in
producing a model is ultimately a management decision that depends on the application
domain, the skills of test designers, and the availability of suitable tools.
Generate Test Case Specifications Test specifications are obtained by suitably
combining values for all inputs of the functional unit under test. If representative values
were explicitly enumerated in the previous step, then test case specifications will be
elements of the Cartesian product of values selected for each input. If a formal model
was produced, then test case specifications will be specific behaviors or combinations of
parameters of the model, and a single test case specification could be satisfied by many
different concrete inputs. Either way, brute force enumeration of all combinations is
unlikely to be satisfactory.
The number of combinations in the Cartesian product of independently selected values
grows as the product of the sizes of the individual sets. For a simple functional unit with
five inputs each characterized by six values, the size of the Cartesian product is 65 =
7776 test case specifications, which may be an impractical number for test cases for a
simple functional unit. Moreover, if (as is usual) the characteristics are not completely
orthogonal, many of these combinations may not even be feasible.

Consider the input of a procedure that searches for occurrences of a complex pattern in
a Web database. Its input may be characterized by the length of the pattern and the
presence of special characters in the pattern, among other aspects. Interesting values
for the length of the pattern may be zero, one, or many. Interesting values for the
presence of special characters may be zero, one, or many. However, the combination of
value "zero" for the length of the pattern and value "many" for the number of special
characters in the pattern is clearly impossible.
The test case specifications represented by the Cartesian product of all possible inputs
must be restricted by ruling out illegal combinations and selecting a practical subset of
the legal combinations. Illegal combinations are usually eliminated by constraining the set
of combinations. For example, in the case of the complex pattern presented above, we
can constrain the choice of one or more special characters to a positive length of the
pattern, thus ruling out the illegal cases of patterns of length zero containing special
characters.
Selection of a practical subset of legal combination can be done by adding information
that reflects the hazard of the different combinations as perceived by the test designer
or by following combinatorial considerations. In the former case, for example, we can
identify exceptional values and limit the combinations that contain such values. In the
pattern example, we may consider only one test for patterns of length zero, thus
eliminating many combinations that would otherwise be derived for patterns of length
zero. Combinatorial considerations reduce the set of test cases by limiting the number of
combinations of values of different inputs to a subset of the inputs. For example, we can
generate only tests that exhaustively cover all combinations of values for inputs
considered pair by pair.
Depending on the technique used to reduce the space represented by the Cartesian
product, we may be able to estimate the number of generated test cases generated and
modify the selected subset of test cases according to budget considerations. Subsets of
combinations of values (i.e., potential special cases) can often be derived from models
of behavior by applying suitable test selection criteria that identify subsets of interesting
behaviors among all behaviors represented by a model, for example by constraining the
iterations on simple elements of the model itself. In many cases, test selection criteria
can be applied automatically.
Generate Test Cases and Instantiate Tests The test generation process is completed
by turning test case specifications into test cases and instantiating them. Test case
specifications can be turned into test cases by selecting one or more test cases for
each test case specification. Test cases are implemented by creating the scaffolding
required for their execution.
[5]Here

the word "user" designates the individual using the specified service. It can be the
user of the system, when dealing with a system specification, but it can be another

module of the system, when dealing with detailed design specifications.

10.4 Choosing a Suitable Approach

In the next chapters we will see several approaches to functional testing, each applying
to different kinds of specifications. Given a specification, there may be one or more
techniques well suited for deriving functional test cases, while some other techniques
may be hard or even impossible to apply or may lead to unsatisfactory results. Some
techniques can be interchanged; that is, they can be applied to the same specification
and lead to similar results. Other techniques are complementary; that is, they apply to
different aspects of the same specification or at different stages of test case generation.
The choice of approach for deriving functional test cases depends on several factors:
the nature of the specification, form of the specification, expertise and experience of test
designers, structure of the organization, availability of tools, budget and quality
constraints, and costs of designing and implementing scaffolding.
Nature and form of the specification Different approaches exploit different
characteristics of the specification. For example, the presence of several constraints on
the input domain may suggest using a partitioning method with constraints, such as the
category-partition method described in Chapter 11, while unconstrained combinations of
values may suggest a pairwise combinatorial approach. If transitions among a finite set
of system states are identifiable in the specification, a finite state machine approach
may be indicated, while inputs of varying and unbounded size may be tackled with
grammar-based approaches. Specifications given in a specific format (e.g., as decision
structures) suggest corresponding techniques. For example, functional test cases for
SDL[6] specifications of protocols are often derived with finite state machine-based
criteria.
Experience of test designers and organization The experience of testers and
company procedures may drive the choice of the testing technique. For example, test
designers expert in category partition may prefer that technique over a catalog-based
approach when both are applicable, while a company that works in a specific application
area may require the use of domain-specific catalogs.
Tools Some techniques may require the use of tools, whose availability and cost should
be taken into account when choosing a testing technique. For example, several tools are
available for deriving test cases from SDL specifications. The availability of one of these
tools may suggest the use of SDL for capturing a subset of the requirements expressed
in the specification.
Budget and quality constraints Different quality and budget constraints may lead to
different choices. For example, if the primary constraint is rapid, automated testing, and
reliability requirements are not stringent, random test case generation may be
appropriate. In contrast, thorough testing of a safety critical application may require the
use of sophisticated methods for functional test case generation. When choosing an

approach, it is important to evaluate all relevant costs. For example, generating a large
number of random test cases may necessitate design and construction of sophisticated
test oracles, or the cost of training to use a new tool may exceed the advantages of
adopting a new approach.
Scaffolding costs Each test case specification must be converted to a concrete test
case, executed many times over the course of development, and checked each time for
correctness. If generic scaffolding code required to generate, execute, and judge the
outcome of a large number of test cases can be written just once, then a combinatorial
approach that generates a large number of test case specifications is likely to be
affordable. If each test case must be realized in the form of scaffolding code written by
hand - or worse, if test execution requires human involvement - then it is necessary to
invest more care in selecting small suites of test case specifications.
Many engineering activities require careful analysis of trade-offs. Functional testing is no
exception: Successfully balancing the many aspects is a difficult and often
underestimated problem that requires skilled designers. Functional testing is not an
exercise of choosing the optimal approach, but a complex set of activities for finding a
suitable combination of models and techniques that yield a set of test cases to satisfy
cost and quality constraints. This balancing extends beyond test design to software
design for test. Appropriate design not only improves the software development
process, but can greatly facilitate the job of test designers and lead to substantial
savings.

Open Research Issues

Functional testing is by far the most common way of deriving test cases in industry, but
neither industrial practice nor research has established general and satisfactory
methodologies. Research in functional testing is increasingly active and progressing in
many directions.
Deriving test cases from formal models is an active research area. In the past three
decades, formal methods have been studied mainly as a means of proving software
properties. Recently, attention has moved toward using formal methods for deriving test
cases. There are three main open research topics in this area:
Definition of techniques for automatically deriving test cases from particular
formal models. Formal methods present new challenges and opportunities for
deriving test cases. We can both adapt existing techniques borrowed from other
disciplines or research areas and define new techniques for test case
generation. Formal notations can support automatic generation of test cases,
thus opening additional problems and research challenges.
Adaptation of formal methods to be more suitable for test case generation. As

illustrated in this chapter, test cases can be derived in two broad ways, either by
identifying representative values or by deriving a model of the unit under test. A
variety of formal models could be used in testing. The research challenge lies in
identifying a trade-off between costs of creating formal models and savings in
automatically generating test cases.
Development of a general framework for deriving test cases from a range of
formal specifications. Currently research addresses techniques for generating
test cases from individual formal methods. Generalization of techniques will allow
more combinations of formal methods and testing.
Another important research area is fed by interest in different specification and design
paradigms (e.g., software architectures, software design patterns, and service- oriented
applications). Often these approaches employ new graphical or textual notations.
Research is active in investigating different approaches to automatically or
semiautomatically deriving test cases from these artifacts and studying the effectiveness
of existing test case generation techniques.
Increasing size and complexity of software systems is a challenge to testing. Existing
functional testing techniques do not take advantage of test cases available for parts of
the artifact under test. Compositional approaches for deriving test cases for a given
system taking advantage of test cases available for its subsystems is an important open
research problem.

Further Reading
Functional testing techniques, sometimes called black-box testing or specification- based
testing, are presented and discussed by several authors. Ntafos [DN81] makes the case
for random rather than systematic testing; Frankl, Hamlet, Littlewood, and Strigini
[FHLS98] is a good starting point to the more recent literature considering the relative
merits of systematic and statistical approaches.

Related topics
Readers interested in practical technique for deriving functional test specifications from
informal specifications and models may continue with the next two chapters, which
describe several functional testing techniques. Readers interested in the
complementarities between functional and structural testing may continue with Chapters
12 and 13, which describe structural and data flow testing.

Exercises
In the Extreme Programming (XP) methodology (see the sidebar on page 381), a
written description of a desired feature may be a single sentence, and the first

10.1 step to designing the implementation of that feature is designing and implementing
a set of test cases. Does this aspect of the XP methodology contradict our
assertion that test cases are a formalization of specifications?

10.2

1. Compute the probability of selecting a test case that reveals the fault in
line 19 of program Root of Figure 10.1 by randomly sampling the input
domain, assuming that type double has range 231 231 1.
2. Compute the probability of randomly selecting a test case that reveals a
fault if lines 13 and 19 were both missing the condition a 0.

10.3 Identify independently testable units in the following specification.

Desk calculator Desk calculator performs the following algebraic operations: sum,
subtraction, product, division, and percentage on integers and real numbers. Operands
must be of the same type, except for percentage, which allows the first operator to be
either integer or real, but requires the second to be an integer that indicates the
percentage to be computed. Operations on integers produce integer results. Program
Calculator can be used with a textual interface that provides the following commands:
Mx=N, where Mx is a memory location, that is, M0 M9, and N is a number. Integers
are given as nonempty sequences of digits, with or without sign. Real numbers are given
as nonempty sequences of digits that include a dot ".", with or without sign. Real
numbers can be terminated with an optional exponent, that is, character "E" followed by
an integer. The command displays the stored number.
Mx=display, where Mx is a memory location and display indicates the value shown on
the last line.
operand1 operation operand2, where operand1 and operand2 are numbers or
memory locations or display and operation is one of the following symbols: "+", "-", "*",
"/", "%", where each symbol indicates a particular operation. Operands must follow the
type conventions. The command displays the result or the string Error.
or with a graphical interface that provides a display with 12 characters and the following
keys:
, the 10 digits
, the operations
to display the result of a sequence of operations

, to clear display

, where

is pressed before a digit to indicate the target

memory, 09, keys
, pressed after
and a digit indicate the
operation to be performed on the target memory: add display to memory, store display
in memory, retrieve memory; that is, move the value in memory to the display and clear
memory.
Example:
prints 65 (the value 15 is
stored in memory cell 3 and then retrieved to compute 80 15).
[6]SDL

(Specification Description Language) is a formal specification notation based on

extended finite state machines, widely used in telecommunication systems and
standardized by the International Telecommunication Union.

Chapter 11: Combinatorial Testing

Requirements specifications typically begin in the form of natural language statements.
The flexibility and expressiveness of natural language, which are so important for human
communication, represent an obstacle to automatic analysis. Combinatorial approaches
to functional testing consist of a manual step of structuring the specification statement
into a set of properties or attributes that can be systematically varied and an
automatizable step of producing combinations of choices.
Simple "brute force" synthesis of test cases by test designers squanders the intelligence
of expert staff on tasks that can be partly automated. Even the most expert of test
designers will perform suboptimally and unevenly when required to perform the repetitive
and tedious aspects of test design, and quality will vary widely and be difficult to monitor
and control. In addition, estimation of the effort and number of test cases required for a
given functionality will be subjective.
Combinatorial approaches decompose the "brute force" work of the test designers into
steps, to attack the problem incrementally by separating analysis and synthesis activities
that can be quantified and monitored, and partially supported by tools. They identify the
variability of elements involved in the execution of a given functionality, and select
representative combinations of relevant values for test cases. Repetitive activities such
as the combination of different values can be easily automated, thus allowing test
designers to focus on more creative and difficult activities.

Required Background
Chapter 10
Understanding the limits of random testing and the needs of a systematic
approach motivates the study of combinatorial as well as model-based testing
techniques. The general functional testing process illustrated in Section 10.3
helps position combinatorial techniques within the functional testing process.

11.1 Overview
In this chapter, we introduce three main techniques that are successfully used in
industrial environments and represent modern approaches to systematically derive test
cases from natural language specifications: the category-partition approach to identifying
attributes, relevant values, and possible combinations; combinatorial sampling to test a
large number of potential interactions of attributes with a relatively small number of
inputs; and provision of catalogs to systematize the manual aspects of combinatorial
testing.
The category-partition approach separates identification of the values that characterize
the input space from the combination of different values into complete test cases. It
provides a means of estimating the number of test cases early, size a subset of cases
that meet cost constraints, and monitor testing progress.
Pairwise and n-way combination testing provide systematic ways to cover interactions
among particular attributes of the program input space with a relatively small number of
test cases. Like the category-partition method, it separates identification of
characteristic values from generation of combinations, but it provides greater control
over the number of combinations generated.
The manual step of identifying attributes and representative sets of values can be made
more systematic using catalogs that aggregate and synthesize the experience of test
designers in a particular organization or application domain. Some repetitive steps can
be automated, and the catalogs facilitate training for the inherently manual parts.
These techniques address different aspects and problems in designing a suite of test
cases from a functional specification. While one or another may be most suitable for a
specification with given characteristics, it is also possible to combine ideas from each.

11.2 Category-Partition Testing

Category-partition testing is a method for generating functional tests from informal
specifications. The following steps comprise the core part of the category-partition
method:
A. Decompose the specification into independently testable features Test
designers identify features to be tested separately, and identify parameters and any
other elements of the execution environment the unit depends on. Environment
dependencies are treated identically to explicit parameters. For each parameter and
environment element, test designers identify the elementary parameter characteristics,
which in the category-partition method are usually called categories.
B. Identify Representative Values Test designers select a set of representative
classes of values for each parameter characteristic. Values are selected in isolation,
independent of other parameter characteristics. In the category-partition method,
classes of values are called choices, and this activity is called partitioning the categories
into choices.
C. Generate Test Case Specifications Test designers impose semantic constraints on
values to indicate invalid combinations and restrict valid combinations (e.g., limiting
combinations involving exceptional and invalid values).
Categories, choices, and constraints can be provided to a tool to automatically generate
a set of test case specifications. Automating trivial and repetitive activities such as these
makes better use of human resources and reduces errors due to distraction. Just as
important, it is possible to determine the number of test cases that will be generated (by
calculation, or by actually generating them) before investing human effort in test
execution. If the number of derivable test cases exceeds the budget for test execution
and evaluation, test designers can reduce the number of test cases by imposing
additional semantic constraints. Controlling the number of test cases before test
execution begins is preferable to ad hoc approaches in which one may at first create
very thorough test suites and then test less and less thoroughly as deadlines approach.
We illustrate the category-partition method using a specification of a feature from the
direct sales Web site of Chipmunk Computers. Customers are allowed to select and
price custom configurations of Chipmunk Computers. A configuration is a set of selected
options for a particular model of computer. Some combinations of model and options are
not valid (e.g., digital LCD monitor with analog video card), so configurations are tested
for validity before they are priced. The check configuration function (Figure 11.1) is
given a model number and a set of components, and returns the Boolean value True if
the configuration is valid or False otherwise. This function has been selected by the test
designers as an independently testable feature.

Check Configuration Check the validity of a computer configuration. The parameters

of check configuration are:
Model A model identifies a specific product and determines a set of constraints on
available components. Models are characterized by logical slots for components,
which may or may not be implemented by physical slots on a bus. Slots may be
required or optional. Required slots must be assigned a suitable component to obtain
a legal configuration, while optional slots may be left empty or filled depending on the
customer's needs.
Example: The required "slots" of the Chipmunk C20 laptop computer include a
screen, a processor, a hard disk, memory, and an operating system. (Of these, only
the hard disk and memory are implemented using actual hardware slots on a bus.)
The optional slots include external storage devices such as a CD/DVD writer.
Set of Components A set of slot,component pairs, which must correspond to
the required and optional slots associated with the model. A component is a choice
that can be varied within a model and that is not designed to be replaced by the end
user. Available components and a default for each slot is determined by the model.
The special value "empty" is allowed (and may be the default selection) for optional
slots.
In addition to being compatible or incompatible with a particular model and slot,
individual components may be compatible or incompatible with each other.
Example: The default configuration of the Chipmunk C20 includes 20 gigabytes of
hard disk; 30 and 40 gigabyte disks are also available. (Since the hard disk is a
required slot, "empty" is not an allowed choice.) The default operating system is
RodentOS 3.2, personal edition, but RodentOS 3.2 mobile server edition may also be
selected. The mobile server edition requires at least 30 gigabytes of of hard disk.
Figure 11.1: Functional specification of the feature Check configuration of the Web
site of a computer manufacturer.
A. Identify Independently Testable Features and Parameter Characteristics We
assume that step A starts by selecting the Check configuration feature to be tested
independently of other features. This entails choosing to separate testing of the
configuration check per se from its presentation through a user interface (e.g., a Web
form), and depends on the architectural design of the software system.
Step A requires the test designer to identify the parameter characteristics, that is, the
elementary characteristics of the parameters and environment elements that affect the
unit's execution. A single parameter may have multiple elementary characteristics. A
quick scan of the functional specification would indicate model and components as the

parameters of check configuration. More careful consideration reveals that what is

"valid" must be determined by reference to additional information. In fact, the functional
specification assumes the existence of a database of models and components. The
database is an environment element that, though not explicitly mentioned in the functional
specification, is required for executing and thus testing the feature, and partly
determines its behavior. Note that our goal is not to test a particular configuration of the
system with a fixed database, but to test the generic system that may be configured
through different database contents.
Having identified model, components, and product database as the parameters and
environment elements required to test the check configuration functionality, the test
designer would next identify the parameter characteristics of each.
Model may be represented as an integer, but we know that it is not to be used
arithmetically, but rather serves as a key to the database and other tables. The
specification mentions that a model is characterized by a set of slots for required
components and a set of slots for optional components. We may identify model number,
number of required slots, and number of optional slots as characteristics of parameter
model.
Parameter components is a collection of slot,selection pairs. The size of a
collection is always an important characteristic, and since components are further
categorized as required or optional, the test designer may identify number of required
components with nonempty selection and number of optional components with
nonempty selection as characteristics. The matching between the tuple passed to check
configuration and the one actually required by the selected model is important and may
be identified as category correspondence of selection with model slots. The actual
selections are also significant, but for now the test designer simply identifies required
component selection and optional component selection, postponing selection of relevant
values to the next stage in test design.
The environment element product database is also a collection, so number of models in
the database and number of components in the database are parameter
characteristics. Actual values of database entries are deferred to the next step in test
design.
There are no hard-and-fast rules for choosing categories, and it is not a trivial task.
Categories reflect the test designer's judgment regarding which classes of values may
be treated differently by an implementation, in addition to classes of values that are
explicitly identified in the specification. Test designers must also use their experience
and knowledge of the application domain and product architecture to look under the
surface of the specification and identify hidden characteristics. For example, the
specification fragment in Figure 11.1 makes no distinction between configurations of
models with several required slots and models with none, but the experienced test

designer has seen enough failures on "degenerate" inputs to test empty collections
wherever a collection is allowed.
The number of options that can (or must) be configured for a particular model of
computer may vary from model to model. However, the category-partition method
makes no direct provision for structured data, such as sets of slot,selection pairs. A
typical approach is to "flatten" collections and describe characteristics of the whole
collection as parameter characteristics. Typically the size of the collection (the length of
a string, for example, or in this case the number of required or optional slots) is one
characteristic, and descriptions of possible combinations of elements (occurrence of
special characters in a string, for example, or in this case the selection of required and
optional components) are separate parameter characteristics.
Suppose the only significant variation among slot,selection pairs was between pairs
that are compatible and pairs that are incompatible. If we treated each pair as a
separate characteristic, and assumed n slots, the category-partition method would
generate all 2n combinations of compatible and incompatible slots. Thus we might have a
test case in which the first selected option is compatible, the second is compatible, and
the third incompatible, and a different test case in which the first is compatible but the
second and third are incompatible, and so on. Each of these combinations could be
combined in several ways with other parameter characteristics. The number of
combinations quickly explode. Moreover, since the number of slots is not actually fixed,
we cannot even place an upper bound on the number of combinations that must be
considered. We will therefore choose the flattening approach and select possible
patterns for the collection as a whole.
Identifying and Bounding Variation
It may seem that drawing a boundary between a fixed program and a variable set of
parameters would be the simplest of tasks for the test designer. It is not always so.
Consider a program that produces HTML output. Perhaps the HTML is based on a
template, which might be encoded in constants in C or Java code, or might be
provided through an external data file, or perhaps both: it could be encoded in a C or
source code file that is generated at compile time from a data file. If the HTML
template is identified in one case as a parameter to varied in testing, it seems it
should be so identified in all three of these variations, or even if the HTML template is
embedded directly in print statements of the program, or in an XSLT transformation
script.
The underlying principle for identifying parameters to be varied in testing is
anticipation of variation in use. Anticipating variation is likewise a key part of
architectural and detailed design of software. In a well-designed software system,
module boundaries reflect "design secrets," permitting one part of a system to be

modified (and retested) with minimum impact on other parts. The most frequent
changes are facilitated by making them input or configurable options. The best
software designers identify and document not only what is likely to change, but how
often and by whom. For example, a configuration or template file that may be
modified by a user will be clearly distinguished from one that is considered a fixed
part of the system.
Ideally the scope of anticipated change is both clearly documented and consonant
with the program design. For example, we expect to see client-customizable aspects
of HTML output clearly isolated and documented in a configuration file, not embedded
in an XSLT script file and certainly not scattered about in print statements in the code.
Thus, the choice to encode something as "data" rather than "program" should at least
be a good hint that it may be a parameter for testing, although further consideration
of the scope of variation may be necessary. Conversely, defining the parameters for
variation in test design can be part of the architectural design process of setting the
scope of variation anticipated for a given product or release.

Should the representative values of the flattened collection of pairs be one compatible
selection, one incompatible selection, all compatible selections, all incompatible
selections, or should we also include mix of 2 or more compatible and 2 or more
incompatible selections? Certainly the latter is more thorough, but whether there is
sufficient value to justify the cost of this thoroughness is a matter of judgment by the test
designer.
We have oversimplified by considering only whether a selection is compatible with a slot.
It might also happen that the selection does not appear in the database. Moreover, the
selection might be incompatible with the model, or with a selected component of another
slot, in addition to the possibility that it is incompatible with the slot for which it has been
selected. If we treat each such possibility as a separate parameter characteristic, we
will generate many combinations, and we will need semantic constraints to rule out
combinations like there are three options, at least two of which are compatible with the
model and two of which are not, and none of which appears in the database. On the
other hand, if we simply enumerate the combinations that do make sense and are worth
testing, then it becomes more difficult to be sure that no important combinations have
been omitted. Like all design decisions, the way in which collections and complex data
are broken into parameter characteristics requires judgment based on a combination of
analysis and experience.
B. Identify Representative Values This step consists of identifying a list of
representative values (more precisely, a list of classes of values) for each of the
parameter characteristics identified during step A. Representative values should be
identified for each category independently, ignoring possible interactions among values

for different categories, which are considered in the next step.

Representative values may be identified by manually applying a set of rules known as
boundary value testing or erroneous condition testing. The boundary value testing rule
suggests selection of extreme values within a class (e.g., maximum and minimum values
of the legal range), values outside but as close as possible to the class, and "interior"
(non-extreme) values of the class. Values near the boundary of a class are often useful
in detecting "off by one" errors in programs. The erroneous condition rule suggests
selecting values that are outside the normal domain of the program, since experience
suggests that proper handling of error cases is often overlooked.
Table 11.1 summarizes the parameter characteristics and the corresponding value
choices identified for feature Check configuration.[1] For numeric characteristics whose
legal values have a lower bound of 1, i.e., number of models in database and number
of components in database, we identify 0, the erroneous value, 1, the boundary value,
and many, the class of values greater than 1, as the relevant value classes. For numeric
characteristics whose lower bound is zero, i.e., number of required slots for selected
model and number of optional slots for selected model, we identify 0 as a boundary
value, 1 and many as other relevant classes of values. Negative values are impossible
here, so we do not add a negative error choice. For numeric characteristics whose legal
values have definite lower and upper-bounds, i.e., number of optional components with
non-empty selection and number of optional components with non-empty selection, we
identify boundary and (when possible) erroneous conditions corresponding to both lower
and upper bounds.
Table 11.1: Categories and value classes derived with the category-partition
method from the specification of Figure 11.1
Open table as spreadsheet
Parameter: Model
Model number
malformed

[error]

not in database

[error]

valid
Number of required slots for selected
model (#SMRS)

Number of optional slots for selected

model (#SMOS)

[single]

[single]

[property RSNE]
[single]

[property OSNE]
[single]

many

[property RSNE],
[property RSMANY]

many

[property OSNE],
[property OSMANY]

Parameter: Components
Correspondence of
selection with model
slots
omitted slots

[error]

extra slots

[error]

mismatched slots

[error]

complete
correspondence
Number of required components with nonempty selection

Number of optional components with nonempty selection

[if RSNE] [error]

< number of required

slots

[if RSNE] [error]

< number of
required slots

[if OSNE]

= number of required
slots

[if RSMANY]

= number of
required slots

[if OSMANY]

Required component selection

some default

Optional component selection

[single]

some default

all valid

all valid

1 incompatible with
slot

1 incompatible
with slot

1 incompatible with
another selection

1 incompatible
with another
selection

1 incompatible with
model

1 incompatible
with model

1 not in database

[error] 1 not in database

[single]

[error]

Environment element: Product database

Number of models in database (#DBM)

Number of components in database

(#DBC)

0
1
many

[error]
[single]

0
1

[error]
[single]

many

Identifying relevant values is an important but tedious task. Test designers may improve
manual selection of relevant values by using the catalog approach described in Section
11.4, which captures the informal approaches used in this section with a systematic
application of catalog entries.
C. Generate Test Case Specifications A test case specification for a feature is given
as a combination of value classes, one for each identified parameter characteristic.
Unfortunately, the simple combination of all possible value classes for each parameter
characteristic results in an unmanageable number of test cases (many of which are
impossible) even for simple specifications. For example, in the Table 11.1 we find 7
categories with 3 value classes, 2 categories with 6 value classes, and one with four
value classes, potentially resulting in 37 62 4 = 314,928 test cases, which would be
acceptable only if the cost of executing and checking each individual test case were very
small. However, not all combinations of value classes correspond to reasonable test
case specifications. For example, it is not possible to create a test case from a test
case specification requiring a valid model (a model appearing in the database) where
the database contains zero models.
The category-partition method allows one to omit some combinations by indicating value
classes that need not be combined with all other values. The label [error] indicates a
value class that need be tried only once, in combination with non-error values of other
parameters. When [error] constraints are considered in the category-partition
specification of Table 11.1, the number of combinations to be considered is reduced to
1331135522+11 = 2711. Note that we have treated "component not in
database" as an error case, but have treated "incompatible with slot" as a normal case
of an invalid configuration; once again, some judgment is required.
Although the reduction from 314,928 to 2,711 is impressive, the number of derived test
cases may still exceed the budget for testing such a simple feature. Moreover, some
values are not erroneous per se, but may only be useful or even valid in particular
combinations. For example, the number of optional components with non-empty
selection is relevant to choosing useful test cases only when the number of optional slots
is greater than 1. A number of non-empty choices of required component greater than
zero does not make sense if the number of required components is zero.
Erroneous combinations of valid values can be ruled out with the property and if-property
constraints. The property constraint groups values of a single parameter characteristic
to identify subsets of values with common properties. The property constraint is
indicated with label property PropertyName, where PropertyName identifies the property

for later reference. For example, property RSNE (required slots non- empty) in Table
11.1 groups values that correspond to non-empty sets of required slots for the
parameter characteristic Number of Required Slots for Selected Model (#SMRS), i.e.,
values 1 and many. Similarly, property OSNE (optional slots non-empty) groups nonempty values for the parameter characteristic Number of Optional Slots for Selected
Model (#SMOS).
The if-property constraint bounds the choices of values for a parameter characteristic
that can be combined with a particular value selected for a different parameter
characteristic. The if-property constraint is indicated with label if PropertyName, where
PropertyName identifies a property defined with the property constraint. For example,
the constraint if RSNE attached to value 0 of parameter characteristic Number of
required components with non-empty selection limits the combination of this value with
values 1 and many of the parameter characteristics Number of Required Slots for
Selected Model (#SMRS). In this way, we rule out illegal combinations like Number of
required components with non-empty selection = 0 with Number of Required Slots for
Selected Model (#SMRS) = 0.
The property and if-property constraints introduced in Table 11.1 further reduce the
number of combinations to be considered to 1 3 1 1 (3 + 2 + 1) 5 5 2 2
+ 11 = 1811.
The number of combinations can be further reduced by iteratively adding property and ifproperty constraints and by introducing the new single constraint, which is indicated with
label single and acts like the error constraint, i.e., it limits the number of occurrences of
a given value in the selected combinations to 1.
Test designers can introduce new property, if-property, and single constraints to reduce
the total number of combinations when needed to meet budget and schedule limits.
Placement of these constraints reflects the test designer's judgment regarding
combinations that are least likely to require thorough coverage.
The single constraints introduced in Table 11.1 reduces the number of combinations to
be considered to 1 1 1 1 1 3 4 4 1 1 + 19 = 67, which may be a
reasonable balance between cost and quality for the considered functionality. The
number of combinations can also be reduced by applying the pairwise and n-way
combination testing techniques, as explained in the next section.
The set of combinations of value classes for the parameter characteristics can be turned
into test case specifications by simply instantiating the identified combinations. Table
11.2 shows an excerpt of test case specifications. The error tag in the last column
indicates test case specifications corresponding to the error constraint. Corresponding
test cases should produce an error indication. A dash indicates no constraints on the
choice of values for the parameter or environment element.

Table 11.2: An excerpt of test case specifications derived from the value classes give
Open table as spreadsheet

#
#
#
Corr.
Required
Optional
# Required # Optional
required optional w/
components componen
components components
slots
slots
model
selection
selection
slots

Model#

malformed

many

many same

EQR

all valid

all va

Not in DB

many

many same

EQR

all valid

all va

valid

many same

all valid

all va

valid

many

many same

EQR

EQO

in-other

in-m

valid

many

many same

EQR

EQO

in-mod

all va

valid

many

many same

EQR

EQO

in-mod

in-s

valid

many

many same

EQR

EQO

in-mod

in-oth

valid

many

many same

EQR

EQO

in-mod

in-m

Legend
EQR

= # req
slot

EQO

= # opt
slot

in-mod

1 incompat w/
model

in-other

1 incompat w/
another slot

in-slot

1 incompat w/
slot

Choosing meaningful names for parameter characteristics and value classes allows
(semi)automatic generation of test case specifications.
[1]At

this point, readers may ignore the items in square brackets, which indicate
constraints identified in step C of the category-partition method.

11.3 Pairwise Combination Testing

However one obtains sets of value classes for each parameter characteristic, the next
step in producing test case specifications is selecting combinations of classes for
testing. A simple approach is to exhaustively enumerate all possible combinations of
classes, but the number of possible combinations rapidly explodes.
Some methods, such as the category-partition method described in the previous section,
take exhaustive enumeration as a base approach to generating combinations, but allow
the test designer to add constraints that limit growth in the number of combinations. This
can be a reasonable approach when the constraints on test case generation reflect real
constraints in the application domain, and eliminate many redundant combinations (for
example, the "error" entries in category-partition testing). It is less satisfactory when,
lacking real constraints from the application domain, the test designer is forced to add
arbitrary constraints (e.g., "single" entries in the category-partition method) whose sole
purpose is to reduce the number of combinations.
Consider the parameters that control the Chipmunk Web site display, shown in Table
11.3. Exhaustive enumeration produces 432 combinations, which is too many if the test
results (e.g., judging readability) involve human judgment. While the test designer might
hypothesize some constraints, such as observing that monochrome displays are limited
mostly to hand-held devices, radical reductions require adding several "single" and
"property" constraints without any particular rationale.
Exhaustive enumeration of all n-way combinations of value classes for n parameters, on
the one hand, and coverage of individual classes, on the other, are only the extreme
ends of a spectrum of strategies for generating combinations of classes. Between them
lie strategies that generate all pairs of classes for different parameters, all triples, and
so on. When it is reasonable to expect some potential interaction between parameters
(so coverage of individual value classes is deemed insufficient), but covering all
combinations is impractical, an attractive alternative is to generate k-way combinations
for k < n, typically pairs or triples.
How much does generating possible pairs of classes save, compared to generating all
combinations? We have already observed that the number of all combinations is the
product of the number of classes for each parameter, and that this product grows
exponentially with the number of parameters. It turns out that the number of
combinations needed to cover all possible pairs of values grows only logarithmically with
the number of parameters - an enormous saving.
A simple example may suffice to gain some intuition about the efficiency of generating
tuples that c