Software

Testing

Software-Testing_Final.indb 1 31-01-2018 [Link]

 LICENSE, DISCLAIMER OF LIABILITY, AND LIMITED WARRANTY
By purchasing or using this book (the “Work”), you agree that this license grants
permission to use the contents contained herein, but does not give you the
right of ownership to any of the textual content in the book or ownership to
any of the information or products contained in it. This license does not permit
uploading of the Work onto the Internet or on a network (of any kind) without
the written consent of the Publisher. Duplication or dissemination of any text,
code, simulations, images, etc. contained herein is limited to and subject to
licensing terms for the respective products, and permission must be obtained
from the Publisher or the owner of the content, etc., in order to reproduce or
network any portion of the textual material (in any media) that is contained in
the Work.
Mercury Learning and Information (“MLI” or “the Publisher”) and anyone
involved in the creation, writing, or production of the companion disc, accom-
panying algorithms, code, or computer programs (“the software”), and any
accompanying Web site or software of the Work, cannot and do not warrant
the performance or results that might be obtained by using the contents of the
Work. The author, developers, and the Publisher have used their best efforts
to insure the accuracy and functionality of the textual material and/or programs
contained in this package; we, however, make no warranty of any kind, express
or implied, regarding the performance of these contents or programs. The
Work is sold “as is” without warranty (except for defective materials used in
manufacturing the book or due to faulty workmanship).
The author, developers, and the publisher of any accompanying content, and
anyone involved in the composition, production, and manufacturing of this
work will not be liable for damages of any kind arising out of the use of (or the
inability to use) the algorithms, source code, computer programs, or textual
material contained in this publication. This includes, but is not limited to, loss
of revenue or profit, or other incidental, physical, or consequential damages
arising out of the use of this Work.
The sole remedy in the event of a claim of any kind is expressly limited to
replacement of the book, and only at the discretion of the Publisher. The use of
“implied warranty” and certain “exclusions” vary from state to state, and might
not apply to the purchaser of this product.

Software-Testing_Final.indb 2 31-01-2018 [Link]

 Software
Testing
A Self-Teaching Introduction

RAJIV CHOPRA, PhD

MERCURY LEARNING AND INFORMATION

Dulles, Virginia
Boston, Massachusetts
New Delhi

Software-Testing_Final.indb 3 31-01-2018 [Link]

 Copyright ©2018 by Mercury Learning and Information LLC. All rights reserved.

Original Title and Copyright: Software Testing, 4/e. © 2014 by S.K. Kataria & Sons.

This publication, portions of it, or any accompanying software may not be reproduced in any
way, stored in a retrieval system of any type, or transmitted by any means, media, electronic dis-
play or mechanical display, including, but not limited to, photocopy, recording, Internet postings,
or scanning, without prior permission in writing from the publisher.
Publisher: David Pallai
Mercury Learning and Information
22841 Quicksilver Drive
Dulles, VA 20166
info@[Link]
[Link]
(800) 232-0223

R. Chopra. Software Testing: A Self-Teaching Introduction.

ISBN: 978-1-683921-66-0

The publisher recognizes and respects all marks used by companies, manufacturers, and develop-
ers as a means to distinguish their products. All brand names and product names mentioned in
this book are trademarks or service marks of their respective companies. Any omission or misuse
(of any kind) of service marks or trademarks, etc. is not an attempt to infringe on the property
of others.

Library of Congress Control Number: 2017960714

181920321  This book is printed on acid-free paper in the United States of America

Our titles are available for adoption, license, or bulk purchase by institutions, corporations, etc.
For additional information, please contact the Customer Service Dept. at 800-232-0223(toll free).
All of our titles are available in digital format at [Link] and other digital vendors.
The sole obligation of Mercury Learning and Information to the purchaser is to replace the
book, based on defective materials or faulty workmanship, but not based on the operation or
functionality of the product.

Software-Testing_Final.indb 4 31-01-2018 [Link]

 CONTENTS
Chapter 1: Introduction to Software Testing 1
1.0. Introduction 1
1.1. The Testing Process 2
1.2. What is Software Testing? 2
1.3. Why Should We Test? What is the Purpose? 6
1.4. Who Should do Testing? 9
1.5. How Much Should We Test? 9
1.6. Selection of Good Test Cases 9
1.7. Measurement of Testing 10
1.8. Incremental Testing Approach 10
1.9. Basic Terminology Related to Software Testing 11
1.10. Testing Life Cycle 17
1.11. When to Stop Testing? 18
1.12. Principles of Testing 18
1.13. Limitations of Testing 19
1.14. Available Testing Tools, Techniques, and Metrics 20
Summary 20
Multiple Choice Questions 21
Answers 22
Conceptual Short Questions With Answers 22
Review Questions 26

Software-Testing_Final.indb 5 31-01-2018 [Link]

 vi • Contents

Chapter 2: Software Verification and Validation 29

2.0. Introduction 29
2.1. Differences between Verification and Validation 30
2.2. Differences between QA And QC? 31
2.3. Evolving Nature of Area 31
2.4. V&V Limitations 32
2.5. Categorizing V&V Techniques 33
2.6. Role of V&V in SDLC—Tabular Form [IEEE std. 1012] 33
2.7. Proof of Correctness (Formal Verification) 37
2.8. Simulation and Prototyping 38
2.9. Requirements Tracing 38
2.10. Software V&V Planning (SVVP) 39
2.11. Software Technical Reviews (STRs) 43
2.11.1. Rationale for STRs 43
2.11.2. Types of STRs 45
2.11.3. Review Methodologies 46
2.12. Independent V&V Contractor (IV&V) 47
2.13. Positive and Negative Effects of Software V&V on Projects 48
2.14. Standard for Software Test Documentation (IEEE829) 50
Summary57
Multiple Choice Questions 58
Answers 59
Conceptual Short Questions With Answers 59
Review Questions 62
Chapter 3: Black-Box (or Functional) Testing Techniques 65
3.0. Introduction to Black-Box (or Functional Testing) 65
3.1. Boundary Value Analysis (BVA) 66
3.1.1. What is BVA? 66
3.1.2. Limitations of BVA 67
3.1.3. Robustness Testing 67
3.1.4. Worst-Case Testing 68
3.1.5. Examples with Their Problem Domain 69
3.1.6. Guidelines for BVA 74

Software-Testing_Final.indb 6 31-01-2018 [Link]

 Contents • vii

3.2. Equivalence Class Testing 74

3.2.1. Weak Normal Equivalence Class Testing 75
3.2.2. Strong Normal Equivalence Class Testing 76
3.2.3. Weak Robust Equivalence Class Testing 76
3.2.4. Strong Robust Equivalence Class Testing 77
3.2.5. Solved Examples 78
3.2.6. Guidelines for Equivalence Class Testing 85
3.3. Decision Table Based Testing 86
3.3.1. What are Decision Tables? 86
3.3.2. Advantages, Disadvantage, and Applications of
Decision Tables 87
3.3.3. Examples 90
3.3.4. Guidelines for Decision Table Based Testing 96
3.4. Cause-Effect Graphing Technique 97
3.4.1. Causes and Effects 97
3.4.2. Test Cases for the Triangle Problem 98
3.4.3. Test Cases for Payroll Problem 100
3.4.4. Guidelines for the Cause-Effect Functional
Testing Technique 101
3.5. Comparison on Black-Box (or Functional)
Testing Techniques 102
3.5.1. Testing Effort 102
3.5.2. Testing Efficiency 104
3.5.3. Testing Effectiveness 104
3.5.4. Guidelines for Functional Testing 105
3.6. Kiviat Charts 105
3.6.1. The Concept of Balance 107
Summary 115
Multiple Choice Questions 115
Answers 117
Conceptual Short Questions With Answers 117
Review Questions 127

Software-Testing_Final.indb 7 31-01-2018 [Link]

 viii • Contents

Chapter 4: White-Box (or Structural) Testing Techniques 133

4.0. Introduction to White-Box Testing or Structural
Testing or Clear-Box or ­Glass-Box or Open-Box Testing 133
4.1. Static versus Dynamic White-Box Testing 134
4.2. Dynamic White-Box Testing Techniques 135
4.2.1. Unit/Code Functional Testing 135
4.2.2. Code Coverage Testing 136
4.2.3. Code Complexity Testing 141
4.3. Mutation Testing Versus Error Seeding—Differences
in Tabular Form 186
4.4. Comparison of Black-Box and White-Box Testing in
Tabular Form 188
4.5. Practical Challenges in White-Box Testing 190
4.6. Comparison on Various White-Box Testing Techniques 190
4.7. Advantages of White-Box Testing 191
Summary 192
Multiple Choice Questions 192
Answers 194
Conceptual Short Questions With Answers 194
Review Questions 200
Chapter 5: Gray-Box Testing 207
5.0. Introduction to Gray-Box Testing 207
5.1. What is Gray-Box Testing? 208
5.2. Various Other Definitions of Gray-Box Testing 208
5.3. Comparison of White-Box , Black-Box, and
Gray-Box Testing Approaches in Tabular Form 209
Summary 211
Multiple Choice Questions 211
Answers 212
Conceptual Short Questions With Answers 212
Review Questions 213

Software-Testing_Final.indb 8 31-01-2018 [Link]

 Contents • ix

Chapter 6: Reducing the Number of Test Cases 215

6.0. Prioritization Guidelines 215
6.1. Priority Category Scheme 216
6.2. Risk Analysis 217
6.3. Regression Testing—Overview 220
6.3.1. Differences between Regression and
Normal Testing 220
6.3.2. Types of Regression Testing 221
6.4. Prioritization of Test Cases for Regression Testing 224
6.5. Regression Testing Technique—A Case Study 225
6.6. Slice-Based Testing 226
Summary 228
Multiple Choice Questions 228
Answers 230
Conceptual Short Questions With Answers 231
Review Questions 233
Chapter 7: Levels of Testing 235
7.0. Introduction 235
7.1. Unit, Integration, System, and Acceptance
Testing Relationship 236
7.2. Integration Testing 237
7.2.1. Classification of Integration Testing 238
7.2.2. Decomposition-Based Integration 238
7.2.3. Call Graph-Based Integration 241
7.2.4. Path-Based Integration with its Pros and Cons 243
7.2.5. System Testing 246
Summary291
Multiple Choice Questions 292
Answers 293
Conceptual Short Questions With Answers 293
Review Questions 298

Software-Testing_Final.indb 9 31-01-2018 [Link]

 x • Contents

Chapter 8: Object-Oriented Testing 301

8.0. Basic Unit for Testing, Inheritance, and Testing 302
8.1. Basic Concepts of State Machines 310
8.2. Testing Object-Oriented Systems 333
8.2.1. Implementation-Based Class Testing/
White-Box or Structural Testing 333
8.2.2. Responsibility-Based Class Testing/
Black-Box/Functional Specification-Based
Testing of Classes 345
8.3. Heuristics for Class Testing 356
8.4. Levels of Object-Oriented Testing 363
8.5. Unit Testing a Class 364
8.6. Integration Testing of Classes 367
8.7. System Testing (With Case Study ) 371
8.8. Regression and Acceptance Testing 381
8.9. Managing the Test Process 383
8.10. Design for Testability (DFT) 387
8.11. GUI Testing 390
8.12. Comparison of Conventional and Object-Oriented Testing 390
8.13. Testing using Orthogonal Arrays 392
8.14. Test Execution Issues 394
8.15. Case Study—Currency Converter Application 394
Summary 403
Multiple Choice Questions 404
Answers 405
Conceptual Short Questions With Answers 406
Review Questions 408
Chapter 9: Automated Testing 409
9.0. Automated Testing 409
9.1. Consideration during Automated Testing 410
9.2. Types of Testing Tools-Static V/s Dynamic 411
9.3. Problems with Manual Testing 413

Software-Testing_Final.indb 10 31-01-2018 [Link]

 Contents • xi

9.4. Benefits of Automated Testing 414

9.5. Disadvantages of Automated Testing 415
9.6. Skills Needed for Using Automated Tools 416
9.7. Test Automation: “No Silver Bullet” 417
9.8. Debugging 418
9.9. Criteria for Selection of Test Tools 422
9.10. Steps for Tool Selection 424
9.11. Characteristics of Modern Testing Tools 425
9.12. Case Study on Automated Tools, Namely,
Rational Robot, Win Runner, Silk Test, and Load Runner 425
Summary 428
Multiple Choice Questions 429
Answers 430
Conceptual Short Questions With Answers 431
Review Questions 432
Chapter 10: Test Point Analysis (TPA) 435
10.0. Introduction 435
10.1. Methodology 436
10.1.1. TPA Philosophy 436
10.1.2. TPA Model 437
10.2. Case Study 449
10.3. TPA for Case Study 450
10.4. Phase Wise Breakup Over Testing Life Cycle 453
10.5. Path Analysis 453
10.6. Path Analysis Process 454
Summary 474
Multiple Choice Questions 478
Answers 479
Conceptual Short Questions With Answers 479
Review Questions 480

Software-Testing_Final.indb 11 31-01-2018 [Link]

 xii • Contents

Chapter 11: Testing Your Websites—Functional and

Non-Functional Testing 481
11.0. Abstract 481
11.1. Introduction 482
11.2. Methodology 486
11.2.1. Non-Functional Testing (or White-Box Testing) 486
11.2.2. Functional Testing (or Black-Box Testing) 489
Summary 490
Multiple Choice Questions 490
Answers 490
Conceptual Short Questions With Answers 491
Review Questions 492
Chapter 12: Regression Testing of a Relational Database 493
12.0. Introduction 493
12.1. Why Test an RDBMS? 494
12.2. What Should We Test? 495
12.3. When Should We Test? 496
12.4. How Should We Test? 497
12.5. Who Should Test? 500
Summary 501
Multiple Choice Questions 501
Answers 502
Conceptual Short Questions With Answers 502
Review Questions 503
Chapter 13: A Case Study on Testing of E-Learning
Management Systems 505
1 Introduction 506
2 Software Requirement Specifications 507
2.1. Introduction 507
2.1.1. Purpose 507
2.1.2. Scope 507

Software-Testing_Final.indb 12 31-01-2018 [Link]

 Contents • xiii

2.1.3. Definitions, Acronyms, and Abbreviations 508

2.1.4. References Books 508
2.1.5. Overview 509
2.2. Overall Descriptions 509
2.2.1. Product Perspective 509
2.2.2. Product Functions 510
2.2.3. User Characteristics 511
2.2.4. Constraints 511
2.2.5. Assumptions and Dependencies 511
2.2.6. Apportioning of Requirements 511
2.3. Specific Requirements 512
2.3.1. User Interfaces and Validations 512
2.3.2. Functions 521
2.3.3. Modules 521
2.3.4. Performance Requirements 522
2.3.5. Logical Database Requirements 522
2.3.6. Design Constraints 522
2.3.7. Software System Attributes 522
2.4. Change Management Process 523
2.5. Document Approval 523
2.6. Supporting Information 523
3 System Design 524
4 Reports And Testing 525
4.1. Test Report 525
4.2. Testing 525
4.2.1. Types of Testing 525
4.2.2. Levels of Testing 526
5 Test Cases 528
5.1. Return Filed Report 528
5.2. Monthly/Quarterly Tax Paid Form 544
5.3. Monthly/Quarterly Tax Paid Form 546

Software-Testing_Final.indb 13 31-01-2018 [Link]

 xiv • Contents

5.4. Monthly /Quarterly Tax Paid Form 550

5.5. Service Wise Report (Admin Report) 552
5.6. STRPs Wise Report (Admin Report) 556
Conclusion 557
Chapter 14: The Game Testing Process 559
14.1. “Black-Box” Testing 560
14.2. “White-Box” Testing 562
14.3. The Life Cycle of a Build 563
14.4. On Writing Bugs Well 573
Exercises 580
Chapter 15: Basic Test Plan Template 583

Appendix A: Quality Assurance and Testing Tools 591

Appendix B: Suggested Projects 599
Appendix C: Glossary 611
Appendix D: Sample Project Description 647
Appendix E: Bibliography 653

Index 655

Software-Testing_Final.indb 14 31-01-2018 [Link]

 CHAPTER

1
Introduction to Software
Testing
Inside this Chapter:
1.0. Introduction
1.1. The Testing Process
1.2. What is Software Testing?
1.3. Why Should We Test? What is the Purpose?
1.4. Who Should Do Testing?
1.5. What Should We Test?
1.6. Selection of Good Test Cases
1.7. Measurement of the Progress of Testing
1.8. Incremental Testing Approach
1.9. Basic Terminology Related to Software Testing
1.10. Testing Life Cycle
1.11. When to Stop Testing?
1.12. Principles of Testing
1.13. Limitations of Testing
1.14. Available Testing Tools, Techniques, and Metrics

1.0. INTRODUCTION
Testing is the process of executing the program with the intent of finding
faults. Who should do this testing and when should it start are very ­important
questions that are answered in this text. As we know software testing is the

Software-Testing_Final.indb 1 31-01-2018 [Link]

 2 • Software Testing

fourth phase of the software development life cycle (SDLC). About 70% of
development time is spent on testing. We explore this and many other inter-
esting concepts in this chapter.

1.1.  THE TESTING PROCESS

Testing is different from debugging. Removing errors from your programs
is known as debugging but testing aims to locate as yet undiscovered errors.
We test our programs with both valid and invalid inputs and then compare
our expected outputs as well as the observed outputs (after execution of soft-
ware). Please note that testing starts from the requirements analysis phase
only and goes until the last maintenance phase. During requirement analy-
sis and designing we do static testing wherein the SRS is tested to check
whether it is as per user requirements or not. We use techniques of code
reviews, code inspections, walkthroughs, and software technical reviews
(STRs) to do static testing. Dynamic testing starts when the code is ready
or even a unit (or module) is ready. It is dynamic testing as now the code is
tested. We use various techniques for dynamic testing like black-box, gray-
box, and white-box testing. We will be studying these in the subsequent
chapters.

1.2.  WHAT IS SOFTWARE TESTING?

The concept of software testing has evolved from simple program “check-
out” to a broad set of activities that cover the entire software life-cycle.
There are five distinct levels of testing that are given below:
a. Debug: It is defined as the successful correction of a failure.
b. Demonstrate: The process of showing that major features work
with typical input.
c. Verify: The process of finding as many faults in the application under
test (AUT) as possible.
d. Validate: The process of finding as many faults in requirements,
design, and AUT.
e. Prevent: To avoid errors in development of requirements, design,
and implementation by self-checking techniques, including “test
before design.”

Software-Testing_Final.indb 2 31-01-2018 [Link]

 Introduction to Software Testing • 3

There are various definitions of testing that are given below:

“Testing is the process of exercising or evaluating a system or system

component by manual or automated means to verify that it satisfies
specified requirements.”
[IEEE 83a]

OR
“Software testing is the process of executing a program or system
with the intent of finding errors.”

[Myers]

OR
“It involves any activity aimed at evaluating an attribute or capabil-
ity of a program or system and determining that it meets its required
results.”

[Hetzel]

Testing is NOT:
a. The process of demonstrating that errors are not present.
b. The process of showing that a program performs its intended func-
tions correctly.
c. The process of establishing confidence that a program does what it is
supposed to do.
So, all these definitions are incorrect. Because, with these as guidelines,
one would tend to operate the system in a normal manner to see if it works.
One would unconsciously choose such normal/correct test data as would
prevent the system from failing. Besides, it is not possible to certify that a
system has no errors—simply because it is almost impossible to detect all
errors.
So, simply stated: “Testing is basically a task of locating errors.”
It ­may be:
a. Positive testing: Operate the application as it should be ­operated.
Does it behave normally? Use a proper variety of legal test data,
including data values at the boundaries to test if it fails. Check actual
test results with the expected. Are results correct? Does the applica-
tion function correctly?

Software-Testing_Final.indb 3 31-01-2018 [Link]

 4 • Software Testing

b. Negative testing: Test for abnormal operations. Does the system

fail/crash? Test with illegal or abnormal data. Intentionally attempt
to make things go wrong and to discover/detect—“Does the pro-
gram do what it should not do? Does it fail to do what it should?”
c. Positive view of negative testing: The job of testing is to dis-
cover errors before the user does. A good tester is one who is suc-
cessful in making the system fail. Mentality of the tester has to be
destructive—opposite to that of the creator/author, which should be
constructive.
One very popular equation of software testing is:
Software Testing = Software Verification + Software Validation

As per IEEE definition(s):

Software verification: “It is the process of evaluating a system or
component to determine whether the products of a given develop-
ment phase satisfy the conditions imposed at the start of that phase.”
OR
“It is the process of evaluating, reviewing, inspecting and doing desk
checks of work products such as requirement specifications, design
specifications and code.”
OR
“It is a human testing activity as it involves looking at the documents
on paper.”
Whereas software validation: “It is defined as the process of eval-
uating a system or component during or at the end of development
process to determine whether it satisfies the specified requirements.
It involves executing the actual software. It is a computer based test-
ing process.”
Both verification and validation (V&V) are complementary to each ­other.
As mentioned earlier, good testing expects more than just running
a ­program. We consider a leap-year function working on MS SQL (server data
base):

CREATE FUNCTION  f_is_leap_year (@ ai_year small int)

RETURNS small int
AS
BEGIN
- -if year is illegal (null or –ve), return –1
IF (@ ai_year IS NULL) or

Software-Testing_Final.indb 4 31-01-2018 [Link]

 Introduction to Software Testing • 5

(@ ai_year <= 0) RETURN –1

IF ((
(@ ai_year % 4) = 0) AND
((@ ai_year % 100)< > 0)) OR
((@ ai_year % 400) = 0)
RETURN 1 — leap year
RETURN 0 --Not a leap year
END

We execute above program with number of inputs:

TABLE 1.1  Database Table: Test_leap_year

Serial Year Expected Observed Match

no. (year to test) result result
1. –1 –1 –1 Yes
2. –400 –1 –1 Yes
3. 100 0 0 Yes
4. 1000 0 0 Yes
5. 1800 0 0 Yes
6. 1900 0 0 Yes
7. 2010 0 0 Yes
8. 400 1 1 Yes
9. 1600 1 1 Yes
10. 2000 1 1 Yes
11. 2400 1 1 Yes
12. 4 1 1 Yes
13. 1204 1 1 Yes
14. 1996 1 1 Yes
15. 2004 1 1 Yes

In this database table given above there are 15 test cases. But these are
not sufficient as we have not tried with all possible inputs. We have not con-
sidered the trouble spots like:
i.  Removing statement (@ ai_year % 400 = 0) would result in Y2K
­problem.
ii. Entering year in float format like 2010.11.
iii. Entering year as a character or as a string.
iv. Entering year as NULL or zero (0).

Software-Testing_Final.indb 5 31-01-2018 [Link]

 6 • Software Testing

This list can also grow further. These are our trouble spots or critical ­areas.
We wish to locate these areas and fix these problems before our customer
does.

1.3.  WHY SHOULD WE TEST? WHAT IS THE PURPOSE?

Testing is necessary. Why?
1. The Technical Case:
a. Competent developers are not inflallible.
b. The implications of requirements are not always forseeable.
c. T
 he behavior of a system is not necessarily predictable from its
components.
d. Languages, databases, user interfaces, and operating systems
have bugs that can cause application failures.
e. Reusable classes and objects must be trustworthy.
2. The Business Case:
a. If you don’t find bugs your customers or users will.
b. 
Post-release debugging is the most expensive form of
­development.
c. Buggy software hurts operations, sales, and reputation.
d. Buggy software can be hazardous to life and property.
3. The Professional Case:
a. Test case design is a challenging and rewarding task.
b. Good testing allows confidence in your work.
c. Systematic testing allows you to be most effective.
d. Your credibility is increased and you have pride in your efforts.
4. The Economics Case: Practically speaking, defects get introduced
in every phase of SDLC. Pressman has described a defect amplifica-
tion model wherein he says that errors get amplified by a certain fac-
tor if that error is not removed in that phase only. This may increase
the cost of defect removal. This principle of detecting errors as close
to their point of introduction as possible is known as phase contain-
ment of errors.

Software-Testing_Final.indb 6 31-01-2018 [Link]

 Introduction to Software Testing • 7

1. Requirements Analysis; 2. Design;

3. Implementaon; 4. Tesng; 5. Maintenance
FIGURE 1.1  Efforts During SDLC.

5. To Improve Quality: As computers and software are used in critical

applications, the outcome of a bug can be severe. Bugs can cause huge
losses. Bugs in critical systems have caused airplane crashes, allowed
space shuttle systems to go awry, and halted trading on the stock mar-
ket. Bugs can kill. Bugs can cause disasters.
In a computerized embedded world, the quality and reliability of
software is a matter of life and death. This can be achieved only if
thorough testing is done.
6. For Verification and Validation (V&V): Testing can serve as metrics.
It is heavily used as a tool in the V&V process. We can compare the
quality among different products under the same specification based
on results from the same test.
Good testing can provide measures for all relevant quality factors.
7. For Reliability Estimation: Software reliability has important
­relationships with many aspects of software, including the structure
and the amount of testing done to the software. Based on an opera-
tional profile (an estimate of the relative frequency of use) of vari-
ous inputs to the program, testing can serve as a statistical sampling
method to gain failure data for reliability estimation.

Software-Testing_Final.indb 7 31-01-2018 [Link]

 8 • Software Testing

Recent Software Failures

a. May 31st, 2012, HT reports the failure of air traffic management soft-
ware, Auto Trac-III, at Delhi Airport. The system is unreliable. This
ATC software was installed in 2010 as Auto Trac-II (its older ver-
sion). Since then it has faced many problems due to inadequate test-
ing. Some of the snags were:
1. May 28, 2011, snag hits radar system of ATC.
2. Feb. 22, 2011, ATC goes blind for 10 minutes with no data about
arriving or departing flights.
3. July 28, 2010, radar screens at ATC go blank for 25 minutes after
system displaying flight data crashes.
4. Feb. 10, 2010, one of the radar scopes stops working at ATC.
5. Jan. 27, 2010, a screen goes blank at ATC due to a technical glitch.
6. Jan. 15, 2010, radar system collapses due to software glitch. ATC
officials manually handle the aircraft.
b. The case of a 2010 Toyota Prius that had a software bug that caused
braking problems on bumpy roads.
c. In another case of Therac-25, 6 cancer patients were given overdose.
d. A breach on play station network caused a loss of $170 million to Sony
Corp.

Why it happened?
As we know software testing constitutes about 40% of overall effort and
25% of the overall software budget. Software defects are introduced during
SDLC due to poor quality requirements, design, and code. Sometimes due
to the lack of time and inadequate testing, some of the defects are left behind,
only to be found later by users. Software is a ubiquitous product; 90% of
people use software in their everyday life. Software has high failure rates due
to the poor qualify of the software.
Smaller companies that don’t have deep pockets can get wiped out
because they did not pay enough attention to software quality and conduct
the right amount of testing.

Software-Testing_Final.indb 8 31-01-2018 [Link]

 Introduction to Software Testing • 9

1.4.  WHO SHOULD DO TESTING?

As mentioned earlier, testing starts right from the very beginning. This
implies that testing is everyone’s responsibility. By “everyone,” we mean all
project team members. So, we cannot rely on one person only. Naturally, it
is a team effort. We cannot only designate the tester responsible. Even the
developers are responsible. They build the code but do not indicate any
errors as they have written their own code.

1.5.  HOW MUCH SHOULD WE TEST?

Consider that there is a while_loop that has three paths. If this loop is
executed twice, we have (3 × 3) paths and so on. So, the total number of
paths through such code will be:
= 1 + 3 + (3 × 3) + (3 × 3 × 3) + ...
= 1 + ∑3n (where n > 0)
This means an infinite number of test cases. Thus, testing is not 100%
exhaustive.

1.6.  SELECTION OF GOOD TEST CASES

Designing a good test case is a complex art. It is complex because:
a. Different types of test cases are needed for different classes of infor-
mation.
b. All test cases within a test suite will not be good. Test cases may be good
in variety of ways.
People create test cases according to certain testing styles like
c. 
domain testing or risk-based testing. And good domain tests are
different from good risk-based tests.
Brian Marick coined a new term to a lightly documented test case—the
test idea. According to Brian, “A test idea is a brief statement of something
that should be tested.” For example, if we are testing a square-root function,
one test idea would be—“test a number less than zero.” The idea here is
again to check if the code handles an error case.

Software-Testing_Final.indb 9 31-01-2018 [Link]

 10 • Software Testing

Cem Kaner said—“The best test cases are the ones that find bugs.” Our
efforts should be on the test cases that finds issues. Do broad or deep cover-
age testing on the trouble spots.
A test case is a question that you ask of the program. The point of run-
ning the test is to gain information like whether the program will pass or fail
the test.

1.7.  MEASUREMENT OF TESTING

There is no single scale that is available to measure the testing progress.
A good project manager (PM) wants worse conditions to occur in the very
beginning of the project instead of in the later phases. If errors are large in
numbers, we can say either testing was not done thoroughly or it was done so
thoroughly that all errors were covered. So there is no standard way to meas-
ure our testing process. But metrics can be computed at the organizational,
process, project, and product levels. Each set of these measurements has its
value in monitoring, planning, and control.

Metrics is assisted by four core components—schedule quality, ­resources,

NOTE
and size.

1.8.  INCREMENTAL TESTING APPROACH

To be effective, a software tester should be knowledgeable in two key areas:
1. Software testing techniques
2. The application under test (AUT)
For each new testing assignment, a tester must invest time in learning
about the application. A tester with no experience must also learn testing
techniques, including general testing concepts and how to define test cases.
Our goal is to define a suitable list of tests to perform within a tight deadline.
There are 8 stages for this approach:
Stage 1: Exploration
Purpose: To gain familiarity with the application
Stage 2: Baseline test
Purpose: To devise and execute a simple test case
Stage 3: Trends analysis

Software-Testing_Final.indb 10 31-01-2018 [Link]

 Introduction to Software Testing • 11

Purpose: To evaluate whether the application performs as expected

when actual output cannot be predetermined
Stage 4: Inventory
Purpose: To identify the different categories of data and create a test
for each category item
Stage 5: Inventory combinations
Purpose: To combine different input data
Stage 6: Push the boundaries
Purpose: To evaluate application behavior at data boundaries
Stage 7: Devious data
Purpose: To evaluate system response when specifying bad data
Stage 8: Stress the environment
Purpose: To attempt to break the system
The schedule is tight, so we may not be able to perform all of the stages.
The time permitted by the delivery schedule determines how many stages
one person can perform. After executing the baseline test, later stages could
be performed in parallel if more testers are available.

1.9. BASIC TERMINOLOGY RELATED TO SOFTWARE

TESTING
We must define the following terminologies one by one:
1. Error (or mistake or bugs): People make errors. When people make mis-
takes while coding, we call these mistakes bugs. Errors tend to propagate.
A requirements error may be magnified during design and still amplified
during coding. So, an error is a mistake during SDLC.
2. Fault (or defect): A missing or incorrect statement in a program result-
ing from an error is a fault. So, a fault is the representation of an error.
Representation here means the mode of expression, such as a narrative
text, data flow diagrams, hierarchy charts, etc. Defect is a good synonym
for fault. Faults can be elusive. They requires fixes.
3. Failure: A failure occurs when a fault executes. The manifested inability
of a system or component to perform a required function within speci-
fied limits is known as a failure. A failure is evidenced by incorrect out-
put, abnormal termination, or unmet time and space constraints. It is a
dynamic process.

Software-Testing_Final.indb 11 31-01-2018 [Link]

 12 • Software Testing

So, Error (or mistake or bug) Fault (or defect) Failure.

For example,
 rror (e.g., * replaced by /)
E Defect (e.g., C = A/B) (e.g.,
C = 2 instead of 8)
4. Incident: When a failure occurs, it may or may not be readily apparent to
the user. An incident is the symptom associated with a failure that alerts
the user to the occurrence of a failure. It is an unexpected occurrence
that requires further investigation. It may not need to be fixed.
5. Test: Testing is concerned with errors, faults, failures, and incidents. A
test is the act of exercising software with test cases. A test has two distinct
goals—to find failures or to demonstrate correct execution.
6. Test case: A test case has an identity and is associated with program
behavior. A test case also has a set of inputs and a list of expected outputs.
The essence of software testing is to determine a set of test cases for the
item to be tested.
The test case template is shown below.

Test Case ID
Purpose
Preconditions
Inputs
Expected Outputs
Postconditions
Execution History
Date    Result    Version     Run By

FIGURE 1.2  Test Case Template.

There are 2 types of inputs:

a. Preconditions: Circumstances that hold prior to test case execution.
b. Actual inputs: That were identified by some testing method.
Expected outputs are also of two types:
a. Post conditions
b. Actual outputs

Software-Testing_Final.indb 12 31-01-2018 [Link]

 Introduction to Software Testing • 13

The act of testing entails establishing the necessary preconditions, pro-

viding the test case inputs, observing the outputs, and then comparing these
with the expected outputs to determine whether the test passed.
The remaining information in a test case primarily supports testing team.
Test cases should have an identity and a reason for being. It is also useful to
record the execution history of a test case, including when and by whom it
was run, the pass/fail result of each execution, and the version of the soft-
ware on which it was run. This makes it clear that test cases are valuable, at
least as valuable as source code. Test cases need to be developed, reviewed,
used, managed, and saved. So, we can say that test cases occupy a central
position in testing.

Test cases for ATM:

Preconditions: System is started.

Test Test case Test case Test steps Test Test

case ID name description status priority
Step Expected Actual
(P/F)
result result
Session Verify To verify Insert a Card is accepted; High
01 Card whether the ­readable System asks for
system reads card entry of PIN
a customer’s
ATM card.

Insert ­an Card is ejected; High

unreadable System displays
card an error screen;
System is ready
to start a new
session

Validate To verify Enter valid System ­displays High

PIN whether PIN a menu of trans-
the system action types
accepts
customer’s
PIN
Enter Customer is High
­invalid PIN asked to re-enter
PIN

(Continued)

Software-Testing_Final.indb 13 31-01-2018 [Link]

 14 • Software Testing

Test Test case Test case Test steps Test Test

case ID name description status priority
Step Expected Actual
(P/F)
result result
Enter System ­displays High
­incorrect a menu of
PIN the first ­transaction types.
time, then
­correct PIN
the ­second
time

Enter System ­displays High

­incorrect a menu of
PIN the first ­transaction types.
time and
second time,
then ­correct
PIN the
third time

Enter An ­appropriate High

­incorrect message is
PIN three ­displayed;
times Card is retained
by machine;
­Session is
­terminated

Session Validate To verify Perform a System asks High

02 User whether transaction whether
Session the system ­customer
allows wants another
­customer to ­transaction
perform a
transaction

To verify When System ­displays High

whether system asks a menu of
the ­system for another ­transaction types
allows multi- ­transaction,
ple transac- answer is
tions in one yes
session

(Continued)

Software-Testing_Final.indb 14 31-01-2018 [Link]

 Introduction to Software Testing • 15

Test Test case Test case Test steps Test Test

case ID name description status priority
Step Expected Actual
(P/F)
result result

When System ejects High

system asks card and is ready
for another to start a new
transaction, session
answer is no

Verify To verify Choose System displays High

Transac- whether withdrawal a menu of pos-
tions the system transaction sible withdrawal
allows to amounts
withdraw

To verify Choose an System dispenses High

whether amount that this amount of
system the system cash;
­performs currently System prints
a legal has and a correct
withdrawal which is not receipt show-
transaction greater than ing amount and
properly the account correct updated
balance balance

Choose an System displays High

amount an appropriate
greater than message and
what the asks customer
system to choose a
currently ­different
has amount

Press System Medium

“Cancel” displays an
key ­appropriate
message and
offers the
­customer
the option of
­choosing another
transaction

(Continued)

Software-Testing_Final.indb 15 31-01-2018 [Link]

 16 • Software Testing

Test Test case Test case Test steps Test Test

case ID name description status priority
Step Expected Actual
(P/F)
result result
Deposit Verify To verify Choose System displays High
01 Deposit whether deposit a request for the
Transac- system transaction customer to type
tion allows a the amount
deposit
To verify System displays
whether Enter a entered amount High
the system legitimate
performs a amount
legal deposit
transaction
properly
Receipt System prints a High
correct receipt
showing the
amount and
correct updated
balance
Press “Can- System displays Medium
cel” key an appropriate
message and
offers the ­
customer the
option of
choosing another
transaction

  7. Test suite: A collection of test scripts or test cases that is used for validat-
ing bug fixes (or finding new bugs) within a logical or physical area of a
product. For example, an acceptance test suite contains all of the test
cases that were used to verify that the software has met certain prede-
fined acceptance criteria.
  8.  Test script: The step-by-step instructions that describe how a test case is
to be executed. It may contain one or more test cases.

Software-Testing_Final.indb 16 31-01-2018 [Link]

 Introduction to Software Testing • 17

  9.  Test ware: It includes all of testing documentation created during the
testing process. For example, test specification, test scripts, test cases,
test data, the environment specification.
10.  Test oracle: Any means used to predict the outcome of a test.
11.  Test log: A chronological record of all relevant details about the execu-
tion of a test.
12. Test report: A document describing the conduct and results of testing
carried out for a system.

1.10.  TESTING LIFE CYCLE

FIGURE 1.3  A Testing Life Cycle.

In the development phase, three opportunities arise for errors to be

made resulting in faults that propagate through the remainder of the devel-
opment process. The first three phases are putting bugs IN, the testing phase
is finding bugs, and the last three phases are getting bugs OUT. The fault
resolution step is another opportunity for errors and new faults. When a fix
causes formerly correct software to misbehave, the fix is deficient.

Software-Testing_Final.indb 17 31-01-2018 [Link]

 18 • Software Testing

1.11.  WHEN TO STOP TESTING?

Testing is potentially endless. We cannot test until all defects are unearthed
and removed. It is simply impossible. At some point, we have to stop testing
and ship the software. The question is when?
Realistically, testing is a trade-off between budget, time, and quality. It
is driven by profit models.
The pessimistic approach is to stop testing whenever some or any of the
allocated resources—time, budget, or test cases—are exhausted.
The optimistic stopping rule is to stop testing when either reliability
meets the requirement, or the benefit from continuing testing cannot justify
the testing cost. [Yang]

1.12.  PRINCIPLES OF TESTING

To make software testing effective and efficient we follow certain
­principles. These principles are stated below.
1. Testing should be based on user requirements: This is in order to
uncover any defects that might cause the program or system to fail
to meet the client’s requirements.
2. Testing time and resources are limited: Avoid redundant tests.
3. Exhaustive testing is impossible: As stated by Myer, it is impossible
to test everything due to huge data space and the large number of
paths that a program flow might take.
4. U
 se effective resources to test: This represents the most suitable
tools, procedures, and individuals to conduct the tests. The test
team should use tools like:
a. Deja Gnu: It is a testing frame work for interactive or batch-­
oriented applications. It is designed for regression and
embedded system testing. It runs on UNIX platform. It is a
cross-platform operating system.
5. T
 est planning should be done early: This is because test planning
can begin independently of coding and as soon as the client require-
ments are set.

Software-Testing_Final.indb 18 31-01-2018 [Link]

 Introduction to Software Testing • 19

6. T
 esting should begin “in small” and progress toward testing “in
large”: The smallest programming units (or modules) should be
tested first and then expanded to other parts of the system.
7. Testing should be conducted by an independent third party.
8. All tests should be traceable to customer requirements.
9. Assign best people for testing. Avoid programmers.
10. T
 est should be planned to show software defects and not their
absence.
11. P
 repare test reports including test cases and test results to summa-
rize the results of testing.
12. A
 dvance test planning is a must and should be updated in a timely
manner.

1.13.  LIMITATIONS OF TESTING

1. Testing can show the presence of errors—not their absence.
2. N
 o matter how hard you try, you will never find the last bug in an
application.
3. The domain of possible inputs is too large to test.
4. There are too many possible paths through the program to test.
5. In short, maximum coverage through minimum test-cases. That is
the challenge of testing.
6. Various testing techniques are complementary in nature and it is
only through their combined use that one can hope to detect most
errors.

To see some of the most popular testing tools of 2017, visit the following
NOTE
site: [Link]

Software-Testing_Final.indb 19 31-01-2018 [Link]

 20 • Software Testing

1.14. AVAILABLE TESTING TOOLS, TECHNIQUES, AND

METRICS
There are an abundance of software testing tools that exist. Some of the early
tools are listed below:
a. M
 othora: It is an automated mutation testing tool-set devel-
oped at Purdue University. Using Mothora, the tester can cre-
ate and execute test cases, measure test case adequacy, determine
input-output correctness, locate and remove faults or bugs, and
control and document the test.
b. NuMega’s Bounds Checker, Rational’s Purify: They are run-
time checking and debugging aids. They can both check and pro-
tect against memory leaks and pointer problems.
c. B
 allista COTS Software Robustness Testing Harness [Bal-
lista]: It is a full-scale automated robustness testing tool. It gives
quantitative measures of robustness comparisons across operat-
ing systems. The goal is to automatically test and harden commer-
cial off-the-shelf (COTS) software against robustness failures.

SUMMARY

1. Software testing is an art. Most of the testing methods and practices

are not very different from 20 years ago. It is nowhere near maturity,
although there are many tools and techniques available to use. Good
testing also requires a tester’s creativity, experience, and intuition,
together with proper techniques.
2. Testing is more than just debugging. It is not only used to locate
errors and correct them. It is also used in validation, verification pro-
cess, and reliability measurement.
3. Testing is expensive. Automation is a good way to cut down cost and
time. Testing efficiency and effectiveness is the criteria for coverage
based testing techniques.
4. Complete testing is infeasible. Complexity is the root of the prob-
lem.
5. Testing may not be the most effective method to improve software
quality.

Software-Testing_Final.indb 20 31-01-2018 [Link]

 Introduction to Software Testing • 21

MULTIPLE CHOICE QUESTIONS

1. Software testing is the process of

a. Demonstrating that errors are not present
b. Executing the program with the intent of finding errors
c. Executing the program to show that it executes as per SRS
d. All of the above.
2. Programmers make mistakes during coding. These mistakes are
known as
a. Failures b. Defects
c. Bugs d. Errors
3. Software testing is nothing else but
a. Verification only
b. Validation only
c. Both verification and validation
d. None of the above.
4. Test suite is a
a. Set of test cases b. Set of inputs
c. Set of outputs d. None of the above.
5. Which one is not the verification activity?
a. Reviews b. Path testing
c. Walkthroughs d. Acceptance testing
6. A break in the working of a system is called a(n)
a. Defect b. Failure
c. Fault d. Error
7. One fault may lead to
a. One failure b. No failure
c. Many failures d. All of the above.

Software-Testing_Final.indb 21 31-01-2018 [Link]

 22 • Software Testing

8. Verification is
a. Checking product with respect to customer’s expectations
b. Checking product with respect to SRS
c. Checking product with respect to the constraints of the project
d. All of the above.
9. Validation is
a. Checking the product with respect to customer’s expectations
b. Checking the product with respect to specification
c. Checking the product with respect to constraints of the project
d. All of the above.
10. Which one of the following is not a testing tool?
a. Deja Gnu b. TestLink
c. TestRail d. SOLARIS

ANSWERS

1. b. 2. c. 3. c. 4. a.
5. d. 6. b. 7. d. 8. b.
9. a. 10. d.

CONCEPTUAL SHORT QUESTIONS WITH ANSWERS

Q. 1. Are there some myths associated to software testing?
Ans. Some myths related to software testing are as follows:
1. Testing is a structured waterfall idea: Testing may be purely inde-
pendent, incremental, and iterative activity. Its nature depends
upon the context and adapted strategy.
2. Testing is trivial: Adequate testing means a complete under-
standing of application under test (AUT) and appreciating t­ esting
­techniques.

Software-Testing_Final.indb 22 31-01-2018 [Link]

 Introduction to Software Testing • 23

3. Testing is not necessary: One can minimize the programming

errors but cannot eliminate them. So, testing is necessary.
4. Testing is time consuming and expensive: We remember a say-
ing—“Pay me now or pay me much more later” and this is true
in the case of software testing as well. It is better to apply testing
strategies now or else defects may appear in the final product
which is more expensive.
5. Testing is destructive process: Testing a software is a diagnostic
and creative activity which promotes quality.
Q. 2. Give one example of
a. Interface specification bugs
b. Algorithmic bugs
c. Mechanical bugs
Ans. a.  Examples of interface specification bugs are:
i. Mismatch between what the client needs and what the server
offers.
ii. Mismatch between requirements and implementation.
b. Examples of algorithmic bugs are:
i. Missing initialization.
ii. Branching errors.
c. Examples of mechanical bugs are:
i. Documentation not matching with the operating procedures.
Q. 3. Why are developers not good testers?
Ans. A person checking his own work using his own documentation has
some disadvantages:
i. Misunderstandings will not be detected. This is because the
checker will assume that what the other individual heard from
him was correct.
ii. Whether the development process is being followed properly or
not cannot be detected.

Software-Testing_Final.indb 23 31-01-2018 [Link]

 24 • Software Testing

iii. The individual may be “blinded” into accepting erroneous system

specifications and coding because he falls into the same trap dur-
ing testing that led to the introduction of the defect in the first
place.
iv. It may result in underestimation of the need for extensive ­testing.
v. It discourages the need for allocation of time and effort for t­ esting.
Q. 4. Are there any constraints on testing?
Ans. The following are the constraints on testing:
i. Budget and schedule constraints
ii. Changes in technology
iii. Limited tester’s skills
iv. Software risks
Q. 5. What are test matrices?
Ans. A test matrix shows the inter-relationship between functional events
and tests. A complete test matrix defines the conditions that must be
tested during the test process.
The left side of the matrix shows the functional events and the top
identifies the tests that occur in those events. Within the matrix,
cells are the process that needs to be tested. We can even cascade
these test matrices.
Q. 6. What is a process?
Ans. It is defined as a set of activities that represent the way work is per-
formed. The outcome of a process is usually a product or service.
For example,

Examples of IT processes Outcomes

1.  Analyze business needs Needs statement
2.  Run job Executed job
3.  UNIT test Defect-free unit

Q. 7. Explain PDCA view of a process?

Ans. A PDCA cycle is a conceptual view of a process. It is shown in
Figure 1.4.

Software-Testing_Final.indb 24 31-01-2018 [Link]

 Introduction to Software Testing • 25

It has four components:

FIGURE 1.4

i. P-Devise a plan: Define your ­objective and find the condi-

tions and methods required to achieve your objective. Express a
­specific objective numerically.
ii. D-Execute (or Do) the plan: Create the conditions and
­perform the necessary teaching and training to execute the plan.
Make sure everyone thoroughly understands the objectives and
the plan. Teach workers all of the procedures and skills they need
to fulfill the plan and a thorough understanding of job is needed.
Then they can perform the work according to these procedures.
iii. C-Check the results: Check to determine whether work is
­progressing according to the plan and that the expected results
are obtained. Also, compare the results of the work with the
objectives.
iv. A-Take necessary action: If a check finds out an abnormality,
i.e., if the actual value differs from the target value then search
for its cause and try to mitigate the cause. This will prevent the
recurrence of the defect.
Q. 8. Explain the V-model of testing?
Ans. According to the waterfall model, testing is a post-development activ-
ity. The spiral model took one step further by breaking the product
into increments each of which can be tested separately. However,
V-model brings in a new perspective that different types of testing
apply at different levels. The V-model splits testing into two parts—
design and execution. Please note that the test design is done early

Software-Testing_Final.indb 25 31-01-2018 [Link]

 26 • Software Testing

while the test execution is done in the end. This early design of tests
reduces overall delay by increasing parallelism between develop-
ment and testing. It enables better and more timely validation of
individual phases. The V-model is shown in Figure 1.5.

FIGURE 1.5

The levels of testing echo the levels of abstractions found in the

waterfall model of SDLC. Please note here, especially in terms of
functional testing, that the three levels of definition, i.e., specifica-
tion, initial design, and detailed design; correspond directly to three
levels of testing—unit, integration, and system testing.

A practical relationship exists between the levels of testing versus

black- and white-box testing. Most practitioners say that structural
testing is most appropriate at the unit level while functional testing
is most appropriate at the system level.

REVIEW QUESTIONS

1. What is software testing?

2. Distinguish between positive and negative testing?

Software-Testing_Final.indb 26 31-01-2018 [Link]

 Introduction to Software Testing • 27

3. Software testing is software verification plus software validation.

Discuss.
4. What is the need of testing?
5. Who should do testing? Discuss various people and their roles dur-
ing development and testing.
6. What should we test?
7. What criteria should we follow to select test cases?
8. Can we measure the progress of testing?
9. “Software testing is an incremental process.” Justify the statement.
10. Define the following terms:
a. Error b. Fault
c. Failure d. Incident
e. Test f. Test case
g. Test suite h. Test script
i. Testware
11. Explain the testing life cycle?
12. When should we stop testing? Discuss pessimistic and optimistic
approaches.
13. Discuss the principles of testing.
14. What are the limitations of testing?
15. a.  What is debugging?
b.  Why exhaustive testing is not possible?
16. What are modern testing tools?
17. Write a template for a typical test case.
18. Differentiate between error and fault.
19. What is software testing? How is it different from debugging?
20. Differentiate between verification and validation?
21. Explain the concept of a test case and test plan.

Software-Testing_Final.indb 27 31-01-2018 [Link]

 28 • Software Testing

22. a.  Differentiate between positive testing and negative testing.

b. Why is 100% testing not possible through either black-box or
white-box testing techniques?
c.  Name two testing tools used for functional testing.
d. What is static testing? Name two techniques to perform static
testing.
23. “Software testing is an incremental process.” Justify the statement.
24. a.  Why are developers not good testers?
b.  Which approach should be followed to step testing?
25. a. Discuss the role of software testing during the software life cycle
and why it is so difficult?
b. What should we test? Comment on this statement. Illustrate the
importance of testing.
c. Will exhaustive testing (even if possible for very small programs)
guarantee that the program is 100% correct?
d. Define the following terms:
i. Test suite
ii. Bug
iii. Mistake
iv. Software failure

Software-Testing_Final.indb 28 31-01-2018 [Link]

 chaPter

2
Software Verification and
Validation
Inside this Chapter:
2.0. Introduction
2.1. Differences Between Verification and Validation
2.2. Differences Between QA and QC
2.3. Evolving Nature of Area
2.4. V&V Limitations
2.5. Categorizing V&V Techniques
2.6. Role of V&V in SDLC—Tabular Form
2.7. Proof of Correctness (Formal Verification)
2.8. Simulation and Prototyping
2.9. Requirements Tracing
2.10. Software V&V Planning (SVVP)
2.11. Software Technical Reviews (STRs)
2.12. Independent V&V (IV&V) Contractor
2.13. Positive and Negative Effects of Software V&V on Projects
2.14. Standard for Software Test Documentation (IEEE829)

2.0. INTRODUCTION
Software that satisfies its user expectations is a necessary goal of a success-
ful software development organization. To achieve this goal, software engi-
neering practices must be applied throughout the evolution of the software

Software-Testing_Final.indb 29 31-01-2018 [Link]

 30 • Software Testing

product. Most of these practices attempt to create and modify software in

a manner that maximizes the probability of satisfying its user expectations.

2.1. DIFFERENCES BETWEEN VERIFICATION

AND VALIDATION
Software verification and validation (V&V) is a technical discipline of sys-
tems engineering. According to Stauffer and Fuji (1986), software V&V is
“a systems engineering process employing a rigorous methodology for evalu-
ating the correctness and quality of software product through the software
life cycle.”
According to Dr. Berry Boehm (1981), software V&V is performed in
parallel with the software development and not at the conclusion of the soft-
ware development. However, verification and validation are different. Table
shows the differences between them.

Verification Validation
1. It is a static process of verifying 1. It is a dynamic process of validating/
documents, design, and code. testing the actual product.
2. It does not involve executing the 2. It involves executing the code.
code.
3. It is human based checking of 3. It is the computer-based execution
­documents/files. of program.
4. Target is requirements specification, 4. Target is actual product—a unit, a
application architecture, high level module, a set of integrated modules,
and detailed design, and database and the final product.
design.
5. It uses methods like inspections, 5. It uses methods like black-box,
walk throughs, desk-checking, etc. ­gray-box, and white-box testing.
6. It, generally, comes first—before 6. It generally follows verification.
validation.
7. It answers the question—Are we 7. It answers the question—Are we
building the product right? building the right product?
8. It can catch errors that validation 8. It can catch errors that verification
cannot catch. cannot catch.

Software-Testing_Final.indb 30 31-01-2018 [Link]

 Software Verification and Validation • 31

Both of these are essential and complementary. Each provides its own
sets of error filters.
Each has its own way of finding the errors in the software.

2.2.  DIFFERENCES BETWEEN QA AND QC?

Quality assurance: The planned and systematic activities implemented in a
quality system so that quality requirements for a product or service will be
fulfilled is known as quality assurance.
Quality control: The observation techniques and activities used to fulfill
requirements for quality is known as quality control.
Both are, however, different to each other. We tabulate The differences
between them are shown below.

Quality Assurance (QA) Quality Control (QC)

1. It is process related. 1. It is product related.
2. It focuses on the process used to 2. It focuses on testing of a product
develop a product. developed or a product under
­development.
3. It involves the quality of the 3. It involves the quality of the
­processes. ­products.
4. It is a preventive control. 4. It is a detective control.
5. Allegiance is to development. 5. Allegiance is not to ­development.

2.3.  EVOLVING NATURE OF AREA

As the complexity and diversity of software products continue to increase,
the challenge to develop new and more effective V&V strategies continues.
The V&V approaches that were reasonably effective on small batch-oriented
products are not sufficient for concurrent, distributed, or embedded prod-
ucts. Thus, this area will continue to evolve as new research results emerge
in response to new V&V challenges.

Software-Testing_Final.indb 31 31-01-2018 [Link]

 32 • Software Testing

2.4.  V&V LIMITATIONS

The overall objective of software V&V approaches is to ensure that the prod-
uct is free from failures and meets its user’s expectations. There are several
theoretical and practical limitations that make this objective impossible to
obtain for many products. They are discussed below:

1.  Theoretical Foundations

Howden claims the most important theoretical result in program testing and
analysis is that no general purpose testing or analysis procedure can be used
to prove program correctness.

2.  Impracticality of Testing All Data

For most programs, it is impractical to attempt to test the program with all
possible inputs due to a combinational explosion. For those inputs selected,
a testing oracle is needed to determine the correctness of the output for a
particular test input.

3.  Impracticality of Testing All Paths

For most programs, it is impractical to attempt to test all execution paths
through the product due to a combinational explosion. It is also not possible
to develop an algorithm for generating test data for paths in an arbitrary
product due to the mobility to determine path feasibility.

4.  No Absolute Proof of Correctness

Howden claims that there is no such thing as an absolute proof of cor-
rectness. Instead, he suggests that there are proofs of equivalency, i.e.,
proofs that one description of a product is equivalent to another descrip-
tion. Hence, unless a formal specification can be shown to be correct and,
indeed, reflects exactly the user’s expectations, no claims of product cor-
rectness can be made.

Software-Testing_Final.indb 32 31-01-2018 [Link]

 Software Verification and Validation • 33

2.5.  CATEGORIZING V&V TECHNIQUES

Various V&V techniques are categorized below:

Guessing

FIGURE 2.1  Types of V&V Techniques.

The static methods of V&V involves the review processes. Whereas

dynamic methods like black-box testing can be applied at all levels, even at
the system level. While the principle of white-box testing is that it checks for
interface errors at the module level, black-box testing can also be done at
the module level by testing boundary conditions and low-level functions like
correctly displaying error messages.

2.6. ROLE OF V&V IN SDLC—TABULAR FORM

[IEEE STD. 1012]
Traceability Analysis
It traces each software requirement back to the system requirements
established in the concept activity. This is to ensure that each requirement

Software-Testing_Final.indb 33 31-01-2018 [Link]

 34 • Software Testing

c­ orrectly satisfies the system requirements and that no extraneous software

requirements are added. In this technique, we also determine whether any
derived requirements are consistent with the original objectives, physical
laws, and the technologies described in system document.

Interface Analysis
It is the detailed examination of the interface requirements specifications.
The evaluation criteria is the same as that for requirements specification.
The main focus is on the interfaces between software, hardware, user, and
external software.

Criticality Analysis
Criticality is assigned to each software requirement. When requirements are
combined into functions, the combined criticality of requirements form the
criticality for the aggregate function. Criticality analysis is updated periodi-
cally as requirement changes are introduced. This is because such changes
can cause an increase or decrease in a functions criticality which depends on
how the revised requirement impacts system criticality.
Criticality analysis is a method used to locate and reduce high-risk
­problems and is performed at the beginning of the project. It identifies
the functions and modules that are required to implement critical program
functions or quality requirements like safety, security, etc.
Criticality analysis involves the following steps:
Step 1: C onstruct a block diagram or control flow diagram (CFD) of the
system and its elements. Each block will represent one software
function (or module) only.
Step 2: T race each critical function or quality requirement through
CFD.
Step 3: Classify all traced software functions as critical to:
a. Proper execution of critical software functions.
b. Proper execution of critical quality requirements.
Step 4: F ocus additional analysis on these traced critical software
­functions.
Step 5: R epeat criticality analysis for each life cycle process to deter-
mine whether the implementation details shift the emphasis of
the criticality.

Software-Testing_Final.indb 34 31-01-2018 [Link]

 Software Verification and Validation • 35

Hazard and Risk Analysis

It is done during the requirements definition activity. Now hazards or risks
are identified by further refining the system requirements into detailed soft-
ware requirements. These risks are assessed for their impact on the system.
A summary of these activities is given below:

V&V Activity V&V Tasks Key Issues

1. Requirements „„ Traceability analysis „„ Evaluates the
V&V „„ Software requirements correctness,
­evaluation completeness,
„„ Interface analysis accuracy,
„„ Criticality analysis consistency, testabil-
„„ System V&V test plan ity, and readability
­generation of software
„„ Acceptance V&V test plan requirements.
generation „„ Evaluates the
„„ Configuration management ­software interfaces.
assessment „„ Identifies the
„„ Hazard analysis criticality of each
„„ Risk analysis software function.
„„ Initiates the V&V
test planning for the
V&V system test.
„„ Initiates the V&V
test planning for the
V&V acceptance
test.
„„ Ensures complete-
ness and adequacy
of the SCM process.
„„ Identifies potential
hazards based on
the product data
during the speci-
fied development
activity.
„„ Identifies potential
risks based on the
product data during
the specified devel-
opment activity.
(Continued)

Software-Testing_Final.indb 35 31-01-2018 [Link]

 36 • Software Testing

2. Design V&V „„ Traceability analysis „„ Evaluates software

„„ Software design evaluation design modules for
„„ Interface analysis correctness, com-
„„ Criticality analysis pleteness, accuracy,
„„ Component V&V test plan consistency, testabil-
generation and verification ity, and readability.
„„ Integration V&V test plan „„ Initiates the V&V
generation and verification test planning for the
„„ Hazard analysis V&V component
„„ Risk analysis test.
„„ Initiates the V&V
test planning for the
V&V integration
test.
3. Implementation „„ Traceability analysis „„ Verifies the
V&V „„ Source code and source code ­correctness,
documentation evaluation ­completeness,
„„ Interface analysis ­consistency,
„„ Criticality analysis ­accuracy, testability,
„„ V&V test case generation and and readability of
verification source code.
„„ V&V test procedure
­generation and verification
„„ Component V&V test
­execution and verification
„„ Hazard analysis
„„ Risk analysis
4. Test V&V „„ Traceability analysis
„„ Acceptance V&V test
­procedure generation and
verification
„„ Integration V&V test
­execution and verification
„„ System V&V test execution
and verification
„„ Acceptance V&V test
execution and verification

Software-Testing_Final.indb 36 31-01-2018 [Link]

 Software Verification and Validation • 37

5. Maintenance „„ SVVP (software verification „„ Modifies the SVVP.

V&V and validation plan) revision „„ Evaluates the effect
„„ Proposed change assessment on software of the
„„ Anomaly evaluation operation anomalies.
„„ Criticality analysis „„ Verifies the
„„ Migration assessment ­correctness of
„„ Retirement assessment software when
„„ Hazard analysis migrated to a
„„ Risk analysis different operational
environment.
„„ Ensures that the
existing system
­continues to
function correctly
when specific
­software elements
are retired.

2.7. PROOF OF CORRECTNESS (FORMAL VERIFICATION)

A proof of correctness is a mathematical proof that a computer program or
a part thereof will, when executed, yield correct results, i.e., results fulfilling
specific requirements. Before proving a program correct, the theorem to be
proved must, of course, be formulated.
Hypothesis: The hypothesis of such a correctness theorem is typically a con-
dition that the relevant program variables must satisfy immediately “before”
the program is executed. This condition is called the precondition.
Thesis: The thesis of the correctness theorem is typically a condition that the
relevant program variables must satisfy immediately “after” execution of the
program. This latter condition is called the postcondition.
So, the correctness theorem is stated as follows:
“If the condition, V, is true before execution of the program, S, then the
condition, P, will be true after execution of S.”
where V is precondition and P is postcondition.
Notation: Such a correctness theorem is usually written as {V} S {P},
where V, S, and P have been explained above.

Software-Testing_Final.indb 37 31-01-2018 [Link]

 38 • Software Testing

By program variable we broadly include input and output data, e.g., data
entered via a keyboard, displayed on a screen, or printed on paper. Any
externally observable aspect of the program’s execution may be covered by
the precondition and postcondition.

2.8.  SIMULATION AND PROTOTYPING

Simulation and prototying are techniques for analyzing the expected behav-
ior of a product. There are many approaches to constructing simulations and
prototypes that are well documented in literature.
For V&V purposes, simulations and prototypes are normally used to
analyze requirements and specifications to ensure that they reflect the user’s
needs. Because they are executable, they offer additional insight into the
completeness and correctness of these documents.
Simulations and prototypes can also be used to analyze predicted prod-
uct performance, especially for candidate product designs, to ensure that
they conform to the requirements.
It is important to note that the utilization of simulation and prototyping
as a V&V technique requires that the simulations and prototypes themselves
be correct. Thus, the utilization of these techniques requires an additional
level of V&V activity.

2.9.  REQUIREMENTS TRACING

“It is a technique for ensuring that the product, as well as the testing of the
product, addresses each of its requirements.” The usual approach to performing
requirements tracing uses matrices.
 ne type of matrix maps requirements to software modules. Construction
a. O
and analysis of this matrix can help ensure that all requirements are
properly addressed by the product and that the product does not have
any superfluous capabilities. System verification diagrams are another
way of analyzing requirements/modules traceability.
 nother type of matrix maps requirements to test cases. Construction
b. A
and analysis of this matrix can help ensure that all requirements are
properly tested.

Software-Testing_Final.indb 38 31-01-2018 [Link]

 Software Verification and Validation • 39

 third type of matrix maps requirements to their evaluation approach.

c. A
The evaluation approaches may consist of various levels of testing,
reviews, simulations, etc.
The requirements/evaluation matrix ensures that all requirements will
undergo some form of V&V. Requirements tracing can be applied for all the
products of the software evolution process.

2.10.  SOFTWARE V&V PLANNING (SVVP)

The development of a comprehensive V&V plan is essential to the success
of a project. This plan must be developed early in the project. Depending
on the development approach followed, multiple levels of test plans may be
developed corresponding to various levels of V&V activities. IEEE 83b has
documented the guidelines for the contents of system, software, build, and
module test plans.
The following steps for SVVP are listed below:

Step 1: Identification of V&V Goals

V&V goals must be identified from the requirements and specifications.
These goals must address those attributes of the product that correspond
to its user expectations. These goals must be achievable taking into account
both theoretical and practical limitations.

Step 2: Selection of V&V Techniques

Once Step 1 (above) is finished, we must select specific techniques for each
of the products that evolves during SDLC. These are given below:
a. During the Requirements Phase: The applicable techniques for
accomplishing the V&V objectives for requirements are—technical
reviews, prototyping, and simulations. The review process is often
called a system requirement review (SRR).
b. During the Specifications Phase: The applicable techniques for this
phase are technical reviews, requirements tracing, prototyping, and
simulations. The requirements must be traced to the specifications.
c. During the Design Phase: The techniques for accomplishing the
V&V objectives for designs are technical reviews, requirements tracing,

Software-Testing_Final.indb 39 31-01-2018 [Link]

 40 • Software Testing

prototyping, simulation, and proof of correctness. We can go for two

types of design reviews:
i. High-level designs that correspond to an architectural view of
the product are often reviewed in a preliminary design review
(PDR).
ii. Detailed designs are addressed by a critical design review
(CDR).
d. During the Implementation Phase: The applicable techniques
for accomplishing V&V objectives for implementation are technical
reviews, requirements tracing, testing, and proof of correctness. Various
code review techniques such as walk throughs and inspections exist.
At the source-code level, several static analysis techniques are available
for detecting implementation errors. The requirements tracing activity
is concerned with tracing requirements to source-code modules. The
bulk of the V&V activity for source code consists of testing. Multiple
levels of testing are usually performed. At the module-level, proof-of-
correctness techniques may be applied, if applicable.
e. During the Maintenance Phase: Because changes describe
modifications to products, the same techniques used for V&V during
development may be applied during modification. Changes to
implementation require regression testing.
Step 3: Organizational Responsibilities
The organizational structure of a project is a key planning consideration for
project managers. An important aspect of this structure is the delegation of
V&V activities to various organizations.
This decision is based upon the size, complexity, and criticality of the
product. Four types of organizations are addressed. These organizations
reflect typical strategies for partitioning tasks to achieve V&V goals for the
product. These are:
a. Developmental Organization: This type of organization has the
following responsibilities:
1. To participate in technical reviews for all of the evolution products.
2. To construct prototypes and simulations.
3. To prepare and execute test plans for unit and integration levels of
testing. This is called preliminary qualification testing (PQT).
4. To construct any applicable proofs of correctness at the module level.

Software-Testing_Final.indb 40 31-01-2018 [Link]

 Software Verification and Validation • 41

b. Independent Test Organization (ITO): This type of organization has

the following responsibilities:
1. It enables test activities to occur in parallel with those of development.
2. It participates in all of the product’s technical reviews and monitors
PQT effort.
3. The primary responsibility of the ITO is the preparation and execution
of the product’s system test plan. This is sometimes referred to as the
formal qualification test (FQT).
The plan for this must contain the equivalent of a requirements/
evaluation matrix that defines the V&V approach to be applied for
each requirement.
4. If the product must be integrated with other products, this integration
activity is normally the responsibility of the ITO.
c. Software Quality Assurance (SQA) Organizations: The intent here
is to identify some activities for ensuring software quality. Evaluations
are the primary avenue for ensuring software quality. Some evaluation
types are given below:
i. Internal consistency of product
ii. Understandability of product
iii. Traceability to indicated documents
iv. Consistency with indicated documents
v. Appropriate allocation of sizing, timing, and resources
vi. Adequate test coverage of requirements
vii. Completeness of testing
viii. Completeness of regression testing
d. Independent V&V Contractor: An independent contractor may be
selected to do V&V. The scope of activities of this organization varies
from that of an ITO (discussed above) and SQA organization.

Step 4: Integrating V&V Approaches

Once a set of V&V objectives has been identified, an overall integrated
V&V approach must be determined. This approach involves the integration
of techniques applicable to various life cycle phases as well as the delega-
tion of these tasks among the project’s organizations. The planning of this

Software-Testing_Final.indb 41 31-01-2018 [Link]

 42 • Software Testing

i­ntegrated V&V approach is very dependent upon the nature of the product
and the process used to develop it. Earlier the waterfall approach for testing
was used and now incremental approach is used. Regardless of the approach
selected, V&V progress must be tracked. Requirements/ evaluation matrices
play a key role in this tracking by providing a means of insuring that each
requirement of the product is addressed.

Step 5: Problem Tracking

i. It involves documenting the problems encountered during the V&V
effort.
ii. Routing these problems to appropriate persons for correctness.
iii. Ensuring that corrections have been done.
iv. Typical information to be collected includes:
a. When the problem occurred
b. Where the problem occurred
c. State of the system before occurrence
d. Evidence of the problem
e. Priority for solving problem

NOTE This fifth step is very important when we go with OS testing.

Step 6: Tracking Test Activities

SVVP must provide a mechanism for tracking testing effort, testing cost, and
testing quality. To do this the following data is collected:
a. Number of tests executed
b. Number of tests remaining
c. Time used
d. Resources used
e. Number of problems found
These data can be used to compare actual test progress against scheduled
progress.

Software-Testing_Final.indb 42 31-01-2018 [Link]

 Software Verification and Validation • 43

Step 7: Assessment
It is important that the software V&V plan provide for the ability to collect
data that can be used to assess both the product and the techniques used to
develop it. This involves careful collection of error and failure data, as well
as analysis and classification of these data.

2.11.  SOFTWARE TECHNICAL REVIEWS (STRs)

A review process can be defined as a critical evaluation of an object. It
includes techniques such as walk-throughs, inspections, and audits. Most of
these approaches involve a group meeting to assess a work product.
Software technical reviews can be used to examine all of the products of
the software evolution process. In particular, they are especially applicable
and necessary for those products not yet in machine-processable form, such
as requirements or specifications written in natural language.

2.11.1. Rationale for STRs

The main rationale behind STRs are as follows:
a. Error-Prone Software Development and Maintenance Process:
The complexity and error-prone nature of developing and maintaining
software should be demonstrated with statistics depicting error
frequencies for intermediate software products. These statistics must
also convey the message that errors occur throughout the development
process and that the later these errors are detected, the higher the cost
for their repair.

Summary:
i. Complexity of software development and maintenance processes.
ii. Error frequencies for software work products.
iii. Error distribution throughout development phases.
iv. Increasing costs for error removal throughout the life cycle.

b. Inability to Test All Software: It is not possible to test all software.

Clearly exhaustive testing of code is impractical. Current technology

Software-Testing_Final.indb 43 31-01-2018 [Link]

 44 • Software Testing

also does not exist for testing a specification or high level design. The
idea of testing a software test plan is also bewildering. Testing also does
not address quality issues or adherence to standards which are possible
with review processes.

Summary:
i. Exhaustive testing is impossible.
ii. Intermediate software products are largely untestable.
c. Reviews are a Form of Testing: The degree of formalism, scheduling,
and generally positive attitude afforded to testing must exist for software
technical reviews if quality products are to be produced.

Summary:
i. Objectives
ii. Human based versus machine based
iii. Attitudes and norms
d. Reviews are a Way of Tracking a Project: Through identification
of deliverables with well defined entry and exit criteria and successful
review of these deliverables, progress on a project can be followed and
managed more easily [Fagan]. In essence, review processes provide
milestones with teeth. This tracking is very beneficial for both project
management and customers.

Summary:
i. Individual developer tracking
ii. Management tracking
iii. Customer tracking
e. Reviews Provide Feedback: The instructor should discuss and
provide examples about the value of review processes for providing
feedback about software and its development process.

Summary:
i. Product ii. Process

Software-Testing_Final.indb 44 31-01-2018 [Link]

 Software Verification and Validation • 45

f. Educational Aspects of Reviews: It includes benefits like a better

understanding of the software by the review participants that can be
obtained by reading the documentation as well as the opportunity of
acquiring additional technical skills by observing the work of others.

Summary:
i. Project understanding ii. Technical skills

2.11.2.  Types of STRs

A variety of STRs are possible on a project depending upon the developmen-
tal model followed, the type of software product being produced, and the
standards which must be adhered to. These developmental modes may be
i. Waterfall model
ii. Rapid prototyping
iii. Iterative enhancement
iv. Maintenance activity modeling
And the current standards may be
i. Military standards
ii. IEEE standards
iii. NBS standards
Reviews are classified as formal and Informal reviews. We tabulate the
differences between them in tabular form.

Informal reviews Formal reviews

i. It is a type of review that typically i. It is a planned meeting.
occurs spontaneously among peers.
ii. Reviewers have no responsibility. ii. Reviewers are held accountable
for their participation in the
review.
iii. No review reports are generated. iii. Review reports containing action
items are generated and acted
upon.

Software-Testing_Final.indb 45 31-01-2018 [Link]

 46 • Software Testing

2.11.3.  Review Methodologies

There are three approaches to reviews
a. Walkthrough (or presentation reviews)
b. Inspection (or work product reviews)
c. Audits
a. Walkthroughs (or Presentation Reviews) Walkthroughs are well
defined by Yourdon. Walkthroughs can be viewed as presentation
reviews in which a review participant, usually the developer of the
software being reviewed, narrates a description of the software
and the remainder of the review group provides their feedback
throughout the presentation.
They are also known as presentation reviews because the bulk of
the feedback usually occurs for the material actually presented
at the level it is presented. Thus, advance preparation on the
part of reviewers is often not detectable during a walkthrough.
Walkthroughs suffer from these limitations as well as the fact that
they may lead to disorganized and uncontrolled reviews.
Walkthroughs may also be stressful if the developer of the software
is conducting the walkthrough. This has lead to many variations
such as having someone other than the developer perform the
walkthrough. It is also possible to combine multiple reviews into a
single review such as a combined design and code walkthrough.
b. Inspections (or Walk Product Reviews): It is a formal approach.
Inspections were first performed by Fagan at IBM. They require
a high degree of preparation for the review participants but the
benefits include a more systematic review of the software and a
more controlled and less stressed meeting.
There are many variations of inspections, but all include some
form of a checklist or agenda that guides the preparation of the
review participants and serves to organize the review meeting
itself. Inspections are also characterized by rigorous entry and exit
requirements for the work products being inspected.
An inspection process involves the collection of data that can be
used for feedback on the quality of the development and review

Software-Testing_Final.indb 46 31-01-2018 [Link]

 Software Verification and Validation • 47

processes as well as to influence future validation techniques on the

software itself.

Inspections Walkthroughs
1. It is a five-step process that is 1. It has fewer steps than
well formalized. ­inspections and is a less formal
process.
2. It uses checklists for locating 2. It does not use a checklist.
errors.
3. It is used to analyze the quality 3. It is used to improve the quality
of the process. of the product.
4. This process takes a longer time. 4. It is a shorter process.
5. It focuses on training of junior 5. It focuses on finding defects.
staff.

c. Audits: Audits should also be described as an external type of

review process. Audits serve to ensure that the software is properly
validated and that the process is producing its intended results.

2.12.  INDEPENDENT V&V CONTRACTOR (IV&V)

An independent V&V contractor may sometimes be used to ensure inde-
pendent objectivity and evaluation for the customer.
The use of a different organization, other than the software develop-
ment group, for software V&V is called independent verification and valida-
tion (IV&V). Three types of independence are usually required:
i. Technical Independence: It requires that members of the IV&V
team (organization or group) may not be personnel involved in the
development of the software. This team must have some knowledge
about the system design or some engineering background enabling
them to understand the system. The IV&V team must not be influenced
by the development team when the IV&V team is learning about the
system requirements, proposed solutions for building the system,
and problems encountered. Technical independence is crucial in the

Software-Testing_Final.indb 47 31-01-2018 [Link]

 48 • Software Testing

team’s ability to detect the subtle software requirements, software

design, and coding errors that escape detection by development
testing and SQA reviews.

The technical IV&V team may need to share tools from the computer
support environment (e.g., compilers, assemblers, utilities) but
should execute qualification tests on these tools to ensure that the
common tools themselves do not mask errors in the software being
analyzed and tested. The IV&V team uses or develops its own set of
test and analysis tools separate from the developer’s tools whenever
possible.
ii. Managerial Independence: It means that the responsibility for
IV&V belongs to an organization outside the contractor and program
organizations that develop the software. While assurance objectives
may be decided by regulations and project requirements, the IV&V
team independently decides the areas of the software/system to
analyze and test, techniques to conduct the IV&V, schedule of tasks,
and technical issues to act on. The IV&V team provides its findings in
a timely manner simultaneously to both the development team and
the systems management.
iii. Financial Independence: It means that control of the IV&V budget
is retained in an organization outside the contractor and program
organization that develop the software. This independence protects
against diversion of funds or adverse financial pressures or influences
that may cause delay or stopping of IV&V analysis and test tasks and
timely reporting of results.

2.13. POSITIVE AND NEGATIVE EFFECTS OF SOFTWARE

V&V ON PROJECTS
Software V&V has some positive effects on a software project. The following
are given below:
1. Better quality of software. This includes factors like completeness,
consistency, readability, and testablity of the software.
2. More stable requirements.
3. More rigorous development planning, at least to interface with the
software V&V organization.

Software-Testing_Final.indb 48 31-01-2018 [Link]

 Software Verification and Validation • 49

4. Better adherence by the development organization to programming

language and development standards and configuration management
practices.
5. Early error detection and reduced false starts.
6. Better schedule compliance and progress monitoring.
7. Greater project management visibility into interim technical quality and
progress.
8. Better criteria and results for decision making at formal reviews and
audits.
Some negative effects of software V&V on a software development project
include:
1. Additional project cost of software V&V (10–30% extra).
2. Additional interface involving the development team, user, and software
V&V organization. For example, attendance at software V&V status
meetings, anomaly resolution meetings.
3. Additional documentation requirement beyond the deliverable products,
if software V&V is receiving incremental program and documentation
releases.
4. Need to share computing facilities with and access to classified data for
the software V&V organization.
5. Lower development staff productivity if programmers and engineers
spend time explaining the system to software V&V analysts, especially if
explanations are not documented.
6. Increased paperwork to provide written responses to software V&V
error reports and other V&V data requirements. For example, notices of
formal review and audit meetings, updates to software release schedule,
and response to anomaly reports.
7. Productivity of development staff affected adversely in resolving invalid
anomaly reports.
Some steps can be taken to minimize the negative effects and to max-
imize the positive effects of software V&V. To recover much of the costs,
software V&V is started early in the software requirements phase. The inter-
face activities for documentation, data, and software deliveries between
developer and software V&V groups should be considered as an inherently
necessary step required to evaluate intermediate development products.

Software-Testing_Final.indb 49 31-01-2018 [Link]

 50 • Software Testing

To offset unnecessary costs, software V&V must organize its activities

to focus on critical areas of the software so that it uncovers critical errors
for the development group and thereby results in significant cost savings
to the development process. To do this, software V&V must use its criti-
cality analysis to identify critical areas. It must scrutinize each discrepancy
before release to ensure that no false or inaccurate information is released
to prevent the development group from wasting time on inaccurate or trivial
reports.
To eliminate the need to have development personnel train the software
V&V staff, it is imperative that software V&V select personnel who are expe-
rienced and knowledgeable about the software and its engineering appli-
cation. When software V&V engineers and computer scientists reconstruct
the specific details and idiosyncrasies of the software as a method of recon-
firming the correctness of engineering and programming assumptions, they
often find subtle errors. They gain detailed insight into the development
process and an ability to spot critical errors early. The cost of the develop-
ment interface is minimal, and at times nonexistent, when the software V&V
assessment is independent.
Finally, the discrepancies detected in software and the improvement
in documentation quality resulting from error correction suggests that
software V&V costs are offset by having more reliable and maintainable
software. Many companies rely on their software systems for their daily
operations. Failure of the system, loss of data, and release of or tamper-
ing with sensitive information may cause serious work disruptions and
serious financial impact. The costs of software V&V are offset in many
application areas by increased reliability during operation and reduced
costs of maintenance.

2.14. STANDARD FOR SOFTWARE TEST DOCUMENTATION

(IEEE829)
„„ The IEEE829 standard for software test documentation describes a set
of basic software test documents. It defines the content and form of each
test document.
„„ In this addendum, we give a summary of the structure of the most impor-
tant IEEE829 defined test documents.

Software-Testing_Final.indb 50 31-01-2018 [Link]

 Software Verification and Validation • 51

„„ This addendum is based on the course materials by Jukka Paakki (and

the IEEE829 standard).

Test Plan
1. Test-plan Identifier: Specifies the unique identifier assigned to the test
plan.
2. Introduction: Summarizes the software items and features to be tested,
provides references to the documents relevant for testing (for example,
overall project plan, quality assurance plan, configuration management
plan, applicable standards, etc.).
3. Test Items: Identifies the items to be tested including their version/
revision level, provides references to the relevant item documentation
(for example, requirements specification, design specification, user’s
guide, operations guide, installation guide, etc.), and identifies items
which are specifically excluded from testing.
4. Features to be Tested: Identifies all software features and their
combinations to be tested, identifies the test-design specification
associated with each feature and each combination of features.
5. Features not to be Tested: Identifies all features and significant
combinations of features which will not be tested, and the reasons for
this.
6. Approach: Describes the overall approach to testing (the testing activities
and techniques applied, the testing of non functional requirements
such as performance and security, the tools used in testing); specifies
completion criteria (for example, error frequency or code coverage);
identifies significant constraints such as testing-resource availability and
strict deadlines; serves for estimating the testing efforts.
7. Item Pass/Fail Criteria: Specifies the criteria to be used to determine
whether each test item has passed or failed testing.
8. Suspension Criteria and Resumption: Specifies the criteria used to
suspend all or a portion of the testing activity on the test items (for
example, at the end of working day, due to hardware failure or other
external exception, etc.), specifies the testing activities which must be
repeated when testing is resumed.

Software-Testing_Final.indb 51 31-01-2018 [Link]

 52 • Software Testing

9. Test Deliverables: Identifies the deliverable documents, typically

test-design specifications, test-case specifications, test-procedure
specifications, test-item transmittal reports, test logs, test-incident
reports, description of test-input data and test-output data, and
description of test tools.
10. Testing Tasks: Identifies the set of tasks necessary to prepare and
perform testing (for example, description of the main phases in the
testing process, design of verification mechanisms, plan for maintenance
of the testing environment, etc.).
11. Environmental Needs: Specifies both the necessary and desired properties
of the test environment (for example, hardware, communications and
systems software, software libraries, test support tools, level of security
for the test facilities, drivers and stubs to be implemented, office or
laboratory space, etc.).
12. Responsibilities: Identifies the groups of persons responsible for
managing, designing, preparing, executing, witnessing, checking,
and resolving the testing process; identifies the groups responsible
for providing the test items (Section 3) and the environmental needs
(Section 11).
13. Staffing and Training Needs: Specifies the number of testers by skill
level, and identifies training options for providing necessary skills.
14. Schedule: Includes test milestones (those defined in the overall project
plan as well as those identified as internal ones in the testing process),
estimates the time required to do each testing task, identifies the
temporal dependencies between testing tasks, specifies the schedule
over calendar time for each task and milestone.
15. Risks and Contingencies: Identifies the high-risk assumptions of the
test plan (lack of skilled personnel, possible technical problems, etc.),
specifies contingency plans for each risk (for example, employment
of additional testers, increase of night shift, exclusion of some tests of
minor importance, etc.).
16. Approvals: Specifies the persons who must approve this plan.

Software-Testing_Final.indb 52 31-01-2018 [Link]

 Software Verification and Validation • 53

Test-Case Specification
1. Test-case Specification Identifier: Specifies the unique identifier
assigned to this test-case specification.
2. Test Items: Identifies and briefly describes the items and features to
be exercised by this test case, supplies references to the relevant
item documentation (for example, requirements specification, design
specification, user’s guide, operations guide, installation guide, etc.).
3. Input Specifications: Specifies each input required to execute the test
case (by value with tolerance or by name); identifies all appropriate
databases, files, terminal messages, memory resident areas, and external
values passed by the operating system; specifies all required relationships
between inputs (for example, timing).
4. Output Specifications: Specifies all of the outputs and features (for
example, response time) required of the test items, provides the exact
value (with tolerances where appropriate) for each required output or
feature.
5. Environmental Needs: Specifies the hardware and software configuration
needed to execute this test case, as well as other requirements (such as
specially trained operators or testers).
6. Special Procedural Requirements: Describes any special constraints on
the test procedures which execute this test case (for example, special set-
up, operator intervention, etc.).
7. Intercase Dependencies: Lists the identifiers of test cases which must be
executed prior to this test case, describes the nature of the dependencies.

Test-Incident Report (Bug Report)

1. Bug-Report Identifier: Specifies the unique identifier assigned to this
report.
2. Summary: Summarizes the (bug) incident by identifying the test items
involved (with version/revision level) and by referencing the relevant
documents (for example, test-procedure specification, test-case
specification, test log).
3. Bug Description: Provides a description of the incident, so as to correct
the bug, repeat the incident, or analyze it offline.

Software-Testing_Final.indb 53 31-01-2018 [Link]

 54 • Software Testing

„„ Inputs
„„ Expected results
„„ Actual results
„„ Date and time
„„ Test-procedure step
„„ Environment
„„ Repeatability (whether repeated; whether occurring always, occa-
sionally, or just once).
„„ Testers
„„ Other observers
„„ Additional information that may help to isolate and correct the cause
of the incident; for example, the sequence of operational steps or his-
tory of user-interface commands that lead to the (bug) incident.
4. Impact: Priority of solving the incident/correcting the bug (urgent, high,
medium, low).

Test-Summary Report
1. Test-Summary-Report Identifier: Specifies the unique identifier assigned
to this report.
2. Summary: Summarizes the evaluation of the test items, identifies
the items tested (including their version/revision level), indicates
the environment in which the testing activities took place, supplies
references to the documentation over the testing process (for example,
test plan, test-design specifications, test-procedure specifications, test-
item transmittal reports, test logs, test-incident reports, etc.).
3. Variances: Reports any variances/deviations of the test items from
their design specifications, indicates any variances of the actual testing
process from the test plan or test procedures, specifies the reason for
each variance.
4. Comprehensiveness Assessment: Evaluates the comprehensiveness of
the actual testing process against the criteria specified in the test plan,
identifies features or feature combinations which were not sufficiently
tested and explains the reasons for omission.
5. Summary of Results: Summarizes the success of testing (such as
coverage), identifies all resolved and unresolved incidents.

Software-Testing_Final.indb 54 31-01-2018 [Link]

 Software Verification and Validation • 55

6. Evaluation: Provides an overall evaluation of each test item including

its limitations (based upon the test results and the item-level pass/fail
criteria).
7. Summary of Activities: Summarizes the major testing activities and
events, summarizes resource consumption (for example, total staffing
level, total person-hours, total machine time, and total elapsed time used
for each of the major testing activities).
8. Approvals: Specifies the persons who must approve this report (and the
whole testing phase).

Inspection Checklist for Test Plans

1. Have all materials required for a test plan inspection been received?
2. Are all materials in the proper physical format?
3. Have all test plan standards been followed?
4. Has the testing environment been completely specified?
5. Have all resources been considered, both human and hardware/software?
6. Have all testing dependencies been addressed (driver function,
hardware, etc.)?
7. Is the test plan complete, i.e., does it verify all of the requirements? (For
unit testing: does the plan test all functional and structural variations
from the high-level and detailed design?)
8. Is each script detailed and specific enough to provide the basis for test
case generation?
9. Are all test entrance and exit criteria sufficient and realistic?
10. Are invalid as well as valid input conditions tested?
11. Have all pass/fail criteria been defined?
12. Does the test plan outline the levels of acceptability for pass/fail and exit
criteria (e.g., defect tolerance)?
13. Have all suspension criteria and resumption requirements been
identified?
14. Are all items excluded from testing documented as such?

Software-Testing_Final.indb 55 31-01-2018 [Link]

 56 • Software Testing

15. Have all test deliverables been defined?

16. Will software development changes invalidate the plan? (Relevant for
unit test plans only).
17. Is the intent of the test plan to show the presence of failures and not
merely the absence of failures?
18. Is the test plan complete, correct, and unambiguous?
19. Are there holes in the plan or is there overlap in the plan?
20. Does the test plan offer a measure of test completeness and test reliability
to be sought?
21. Are the test strategy and philosophy feasible?

Inspection Checklist for Test Cases

1. Have all materials required for a test case inspection been received?
2. Are all materials in the proper physical format?
3. Have all test case standards been followed?
4. Are the functional variations exercised by each test case required by the
test plan? (Relevant for unit test case documents only.)
5. Are the functional variations exercised by each test case clearly
documented in the test case description? (Relevant for unit test case
documents only.)
6. Does each test case include a complete description of the expected input
and output or result?
7. Have all testing execution procedures been defined and documented?
8. Have all testing dependencies been addressed (driver function,
hardware, etc.)?
9. Do the test cases accurately implement the test plan?
10. Are all data set definitions and setup requirements complete and
accurate?
11. Are operator instructions and status indicators complete, accurate, and
simple?

Software-Testing_Final.indb 56 31-01-2018 [Link]

 Software Verification and Validation • 57

12. Have all intercase dependencies been identified and described?

13. Is each condition tested once and only once?
14. Have all test entrance and exit criteria been observed?
15. Are the test cases designed to show the presence of failure and not
merely the absence of failure?
16. Are the test cases designed to show omissions and extensions?
17. Are the test cases complete, correct, and unambiguous?
18. Are the test cases realistic?
19. Are the test cases documented so as to be 100% reproducible?
20. Has the entire testing environment been documented?
21. Has configuration management been set up, directories established, and
have case data and tools been loaded?

SUMMARY
1. Software engineering technology has matured sufficiently to be addressed
in approved and draft software engineering standards and guidelines.
2. Business, industries, and government agencies spend billions annually
on computer software for many of their functions:
„„ To manufacture their products.
„„ To provide their services.
„„ To administer their daily activities.
„„ To perform their short- and long-term management functions.
3. As with other products, industries and businesses are discovering that
their increasing dependence on computer technology to perform these
functions, emphasizes the need for safe, secure, reliable computer
systems. They are recognizing that software quality and reliability
are vital to their ability to maintain their competitiveness and high
technology posture in the market place. Software V&V is one of several
methodologies that can be used for building vital quality software.

Software-Testing_Final.indb 57 31-01-2018 [Link]

 58 • Software Testing

MULTIPLE CHOICE QUESTIONS

1. Which one of the following is product related?
a. Quality control b. Quality assurance
c. Both (a) and (b) d. None of the above.
2. Howden claims that
a. No general purpose testing can be used to prove program c­ orrectness.
b. There is no such thing as an absolute proof of correctness.
c. Both (a) and (b)
d. None of the above.
3. Static testing involves
a. Symbolic execution b. Code walkthrough
c. Inspections d. All of the above.
4. The role of V&V in SDLC is given in which of the following standards
a. IEEE std. 1012 b. IEEE std. 9012
c. IEEE std. 83b d. None of the above.
5. Detailed designs are addressed by a
a. Preliminary design review b. Critical design review
c. Both (a) and (b) d. None of the above.
6. A planned meeting is also known as a
a. Informal review b. Formal review
c. Technical review d. Dynamic review
7. Structured walkthrough is a
a. Dynamic testing technique
b. Formal static testing technique
c. Informal static testing
d. Acceptance testing technique

Software-Testing_Final.indb 58 31-01-2018 [Link]

 Software Verification and Validation • 59

8. Which of the following is not a validation activity?

a. Unit testing b. System testing
c. Acceptance testing d. Walkthroughs
9. Which of the following is not a verification activity?
a. Acceptance testing b. Inspections
c. Walkthroughs d. Buddy check
10. During validation
a. Process is checked.
b. Product is checked.
c. Developer’s performance is evaluated.
d. The customer checks the product.

ANSWERS
1. a. 2. c. 3. d. 4. a.
5. b. 6. b. 7. c. 8. d.
9. a. 10. d.

CONCEPTUAL SHORT QUESTIONS WITH ANSWERS

Q. 1. What sort of errors are covered by regression testing?
Ans. Regression testing includes mainly four types of errors:
i. Data Corruption Errors: Due to sharing of data, these errors
result in side effects.
ii. Inappropriate Control Sequencing Errors: Due to the changes
in the execution sequences, these errors result in side effects.
For example, an attempt to remove an item from a queue before
it is placed into the queue.
iii. Resource Contention: Potential bottlenecks and deadlocks are
some examples of these types of errors.
iv. Performance Deficiencies: Timing errors and storage utilization
errors are some examples of these types of errors.

Software-Testing_Final.indb 59 31-01-2018 [Link]

 60 • Software Testing

Q. 2. What is criticality analysis?

Ans. It is a method to locate and reduce high-risk problems. It is per-
formed at the beginning of the project. It identifies the functions
and modules that are required to implement critical program func-
tions or quality requirements like safety, security, etc.
The steps of analysis are as follows:
Step 1: Develop a block diagram or control flow diagram of the
system and its software elements. Each block or control flow box
represents a system or software function (module).
 Step 2: Trace each critical function or quality requirements through
the block or control flow diagram.
Step 3: Classify all traced software functions (modules) as critical
to either the proper execution of critical software functions or the
quality requirements.
Step 4: Focus additional analysis on these traced critical software
functions (modules).
 Step 5: Repeat criticality analysis for each life-cycle process/activity
to determine whether the implementation details shift the emphasis
of the criticality.
Q. 3. What is traceability analysis?
Ans. Traceability analysis traces each software requirement back to the
system requirements. This is done to ensure that each requirement
correctly satisfies the system requirements. This analysis will also
determine whether any derived software requirements are consis-
tent with the original objectives, physical laws, and technologies
described in the system document.
Q. 4. What is interface analysis?
Ans. It is a detailed examination of the interface requirements specifica-
tions. The evaluation criteria here is on the interfaces between the
software and other hardware, user, and external software. Critical-
ity analysis is continued and updated for the software. Criticality
is assigned to each software requirement. When requirements are
combined into functions, the combined criticality of requirements
form the criticality for the aggregate function. The criticality anal-
ysis is updated periodically as requirement changes are ­introduced

Software-Testing_Final.indb 60 31-01-2018 [Link]

 Software Verification and Validation • 61

as such changes can cause a functions criticality to increase or

decrease depending on how the revised requirements impact
­system ­criticality.
Q. 5. Explain the tool support for review processes.
Ans. As tools become available to perform some of the tasks previously
done by humans, the cost effectiveness of review processes increases.
For example, utilization of a compiler to detect syntax errors in code
and thus alleviating this task for the reviewers.

Another example is the design and specification consistency
­checkers.
Q. 6. Suggest some techniques to find different types of errors.
Ans. Some of the techniques are discussed below:
i. Algorithm Analysis: It examines the logic and accuracy of
the software requirements by translating algorithms into some
language or structured format. The analysis involves rederiving
equations or evaluating the suitability of specific numerical
techniques. Algorithm analysis examines the correctness of the
equations and numerical techniques, truncation and sounding
effects, numerical precision of word storage and variables, and
data typing influences.
ii. Analytic Modeling: It provides performance evaluation and
capacity planning information on software design. It represents
the program logic and processing of some kind of model and
analyzes it for efficiency.
iii. Control Flow Analysis: It is used to show the hierarchy of main
routines and their subfunctions. It checks that the proposed
control flow is free of problems like unreachable or incorrect
code.
iv. Database Analysis: It ensures that the database structure and
access methods are compatible with the logical design. It is done
on programs with significant data storage to ensure that common
data and variable regions are used consistently between all calling
routines, that data integrity is enforced and no data or variable
can be accidentally overwritten by overflowing data tables, and
that data typing and use are consistent throughout the program.

Software-Testing_Final.indb 61 31-01-2018 [Link]

 62 • Software Testing

Q. 7. What is desk checking?

Ans. It involves the examination of the software design or code by an indi-
vidual. It includes:
„„ Looking over the code for defects.

„„ Checking for correct procedure interfaces.

„„ Reading the comments and comparing it to external specifications

and software design.

Q. 8. What are Petri-nets?
Ans. Petri-nets model system to ensure software design adequacy for cat-
astrophic failure. The system is modeled using conditions and events
represented by STDs. They can be executed to see how the software
design will actually work under certain conditions.
Q. 9. What is program slicing?
Ans. Slicing is a program decomposition technique used to trace an out-
put variable back through the code to identify all code statements
relevant to a computation in the program.
Q. 10. What is test certification?
Ans. It ensures that the reported test results are the actual finding of the
tests. Test related tools, media, and documentation are certified to
ensure maintainability and repeatability of tests. This technique is
also used to show that the delivered software product is identical to
the software product that was subjected to V&V. It is used in critical
software systems to verify that the required tests have been executed
and that the delivered software product is identical to the product
subjected to software V&V.

REVIEW QUESTIONS
1. a. Discuss briefly the V&V activities during the design phase of the
software development process.
b.  Discuss the different forms of IV&V.
2. a. What is the importance of technical reviews in software development
and maintenance life cycle?
b.  Briefly discuss how walkthroughs help in technical reviews.
3. a. Explain why validation is more difficult than verification.
b. Explain validation testing.

Software-Testing_Final.indb 62 31-01-2018 [Link]

 Software Verification and Validation • 63

4. Explain the following with the help of an example: ‘Verification and

Validation.’
5. Write short notes on V&V standards.
6. Write short notes on ‘Independent V&V contractor.’
7. What is an independent test organization? Why is it necessary?
8. Discuss the role of verification and validation in software evolution.
What are verification and validation objective standards?
9. What is the difference between verification and validation?
10. Discuss V&V approaches, standards, and objectives.
11. What is formal testing?
12. a. What is a software audit?
b. Explain code walkthrough.
13. What are the different V&V activities performed at the coding phase of
the software development process?
14. Discuss in brief software verification and validation plan.
15. a. Why are technical reviews conducted?
b. Discuss a formal method for conducting technical reviews.
16. Discuss one testing technique used during the design phase of SDLC.
17. Explain all the phases through which we can approach testing using a
sample software testing problem. Comment on difficulty of testing.
18. Briefly explain code walkthrough and inspection.
19. During code review you detect errors whereas during code testing you
detect failures. Justify.
20. a. What are the testing activities performed during requirement analysis
phase of SDLC.
b. Briefly describe formal proof of correctness.
21. What is a requirements traceability matrix? Give its importance.

Software-Testing_Final.indb 63 31-01-2018 [Link]

Software-Testing_Final.indb 64 31-01-2018 [Link]
 chaPter

3
Black-Box (or Functional)
Testing Techniques
Inside this Chapter:
3.0. Introduction to Black-Box (or Functional Testing)
3.1. Boundary Value Analysis (BVA)
3.2. Equivalence Class Testing
3.3. Decision Table Based Testing
3.4. Cause-Effect Graphing Technique
3.5. Comparison on Black-Box (or Functional) Testing Techniques
3.6. Kiviat Charts

3.0. INTRODUCTION TO BLACK-BOX (OR FUNCTIONAL

TESTING)
The term Black-Box refers to the software which is treated as a black-box.
By treating it as a black-box, we mean that the system or source code is not
checked at all. It is done from the customer’s viewpoint. The test ­engineer
engaged in black-box testing only knows the set of inputs and expected
­outputs and is unaware of how those inputs are transformed into outputs
by the software. We will now discuss various techniques of performing
black‑box testing.

Software-Testing_Final.indb 65 31-01-2018 [Link]

 66 • Software Testing

3.1.  BOUNDARY VALUE ANALYSIS (BVA)

It is a black-box testing technique that believes and extends the concept that
the density of defect is more towards the boundaries. This is done for the
following reasons:
i. Programmers usually are not able to decide whether they have to use
<= operator or < operator when trying to make comparisons.
ii. Different terminating conditions of for-loops, while loops, and repeat
loops may cause defects to move around the boundary conditions.
iii. The requirements themselves may not be clearly understood,
especially around the boundaries, thus causing even the correctly
coded program to not perform the correct way.
Strongly typed languages such as Ada and Pascal permit explicit defini-
tion of variable ranges. Other languages such as COBOL, FORTRAN, and
C are not strongly typed, so boundary value testing is more appropriate for
programs coded in such languages.

3.1.1.  What Is BVA?

The basic idea of BVA is to use input variable values at their minimum,
just above the minimum, a nominal value, just below their maximum, and
at their maximum. That is, {min, min+, nom, max–, max}. This is shown in
Figure 3.1.
BVA is based upon a critical assumption that is known as single fault
assumption theory. According to this assumption, we derive the test cases on
the basis of the fact that failures are not due to a simultaneous occurrence

FIGURE 3.1  BVA Test Cases.

Software-Testing_Final.indb 66 31-01-2018 [Link]

 Black-Box (or Functional) Testing Techniques • 67

of two (or more) faults. So we derive test cases by holding the values of all
but one variable at their nominal values and letting that variable assume its
extreme values.
If we have a function of n-variables, we hold all but one at the nominal
values and let the remaining variable assume the min, min+, nom, max–,
and max values, repeating this for each variable. Thus, for a function of n
variables, BVA yields (4n + 1) test cases.

3.1.2.  Limitations of BVA

1. Boolean and logical variables present a problem for boundary value
analysis.
2. BVA assumes the variables to be truly independent which is not always
possible.
3. BVA test cases have been found to be rudimentary because they are
obtained with very little insight and imagination.

3.1.3.  Robustness Testing

Another variant to BVA is robustness testing. In BVA, we are within the
legitimate boundary of our range. That is, we consider the following values
for testing:
{min, min+, nom, max–, max} whereas in robustness testing, we try to
cross these legitimate boundaries also. So, now we consider these values for
testing:
{min–, min, min+, nom, max–, max, max+}
Again, with robustness testing, we can focus on exception handling.
With strongly typed languages, robustness testing may be very awkward. For
example, in PASCAL, if a variable is defined to be within a certain range,
values outside that range result in run-time errors that abort normal execu-
tion.
For a program with n-variables, robustness testing will yield (6n + 1)
test-cases. So, we can draw a graph now. (Figure 3.2)

Software-Testing_Final.indb 67 31-01-2018 [Link]

 68 • Software Testing

FIGURE 3.2  Robustness Test Cases.

Each dot represents a test value at which the program is to be tested.

In robustness testing, we cross the legitimate boundaries of input domain.
In the graph of Figure 3.2, we show this by dots that are outside the range
[a, b] of variable x1. Similarly, for variable x2, we have crossed its legitimate
boundary of [c, d] of variable x2.
This type of testing is quite common in electric and electronic circuits.
Furthermore, this type of testing also works on single fault assumption
­theory.

3.1.4. Worst-Case Testing

If we reject our basic assumption of single fault assumption theory and focus
on what happens when we reject this theory—it simply means that we want
to see what happens when more than one variable has an extreme value.
This is multiple path assumption theory. In electronic circuit a­ nalysis, this is
called as “worst-case analysis.” We use this idea here to generate worst-case
test cases.
For each variable, we start with the five- element set that contains the
min, min+, nom, max–, and max values. We then take the Cartesian product
of these sets to generate test cases. This is shown in Figure 3.3.
For a program with n-variables, 5n test cases are generated.

NOTE Robust worst-case testing yields 7n test cases.

Software-Testing_Final.indb 68 31-01-2018 [Link]

 Black-Box (or Functional) Testing Techniques • 69

FIGURE 3.3  Worst-Case Test Cases.

3.1.5. Examples with Their Problem Domain

[Link].  Test-Cases for the Triangle Problem

Before we generate the test cases, first we need to give the problem domain:
Problem Domain: “The triangle program accepts three integers, a, b, and
c, as input. These are taken to be the sides of a triangle. The integers a, b,
and c must satisfy the following conditions:
C1: 1 ≤ a ≤ 200 C 4: a < b + c
C2: 1 ≤ b ≤ 200 C 5: b < a + c
C3: 1 ≤ c ≤ 200 C 6: c < a + b
The output of the program may be: Equilateral, Isosceles, Scalene, or
“NOT-A-TRIANGLE.”

How to Generate BVA Test Cases?

We know that our range is [1, 200] where 1 is the lower bound and 200 is the
upper bound. Also, we find that this program has three inputs—a, b, and c.
So, for our case
n=3
BVA yields (4n + 1) test cases, so we can say that the total number of
test cases will be (4 × 3 + 1) = 12 + 1 = 13.
Table 3.1 shows those 13 test cases.

Software-Testing_Final.indb 69 31-01-2018 [Link]

 70 • Software Testing

TABLE 3.1  BVA test cases for triangle problem.

Case ID a b c Expected output

1. 100 100 1 Isosceles
2. 100 100 2 Isosceles
3. 100 100 100 Equilateral
4. 100 100 199 Isosceles
5. 100 100 200 Not a triangle
6. 100 1 100 Isosceles
7. 100 2 100 Isosceles
8. 100 100 100 Equilateral
9. 100 199 100 Isosceles
10. 100 200 100 Not a triangle
11. 1 100 100 Isosceles
12. 2 100 100 Isosceles
13. 100 100 100 Equilateral
14. 199 100 100 Isosceles
15. 200 100 100 Not a triangle

Please note that we explained above that we can have 13 test cases
(4n + 1) for this problem. But instead of 13, now we have 15 test cases. Also,
test case ID number 8 and 13 are redundant. So, we ignore them. However,
we do not ignore test case ID number 3 as we must consider at least one test
case out of these three. Obviously, it is mechanical work!
We can say that these 13 test cases are sufficient to test this program
using BVA technique.

Question for Practice

1. Applying the robustness testing technique, how would you generate the
test cases for the triangle problem given above?

Software-Testing_Final.indb 70 31-01-2018 [Link]

 Black-Box (or Functional) Testing Techniques • 71

[Link].  Test Cases for Next Date Function

Before we generate the test cases for the Next Date function, we must know
the problem domain of this program:
Problem Domain Next Date is a function of three variables: month, date,
and year. It returns the date of next day as output. It reads current date as
input date. The conditions are:
C1: 1 ≤ month ≤ 12
C2: 1 ≤ day ≤ 31
C3: 1900 ≤ year ≤ 2025
If any of conditions C1, C2, or C3 fails, then this function produces an
output “value of month not in the range 1...12.”
Because many combinations of dates exist this function just displays one
message: “Invalid Input Date.”

Complexities in Next Date Function

A very common and popular problem occurs if the year is a leap year. We
have taken into consideration that there are 31 days in a month. But what
happens if a month has 30 days or even 29 or 28 days? A year is called as a
leap year if it is divisible by 4, unless it is a century year. Century years are
leap years only if they are multiples of 400. So, 1992, 1996, and 2000 are leap
years while 1900 is not a leap year.
Furthermore, in this Next Date problem, we find examples of Zipf’s law
also, which states that “80% of the activity occurs in 20% of the space.” Here
also, much of the source-code of the Next Date function is devoted to the
leap year considerations.

How to Generate BVA Test Cases for This Problem?

The Next Date program takes date as input and checks it for validity. If valid,
it returns the next date as its output.
As we know, with single fault assumption theory, (4n + 1) test cases
can be designed. Here, also n = 3. So, the total number of test cases are (4 ×
3 + 1) = 12 + 1 = 13.
The boundary value test cases are

Software-Testing_Final.indb 71 31-01-2018 [Link]

 72 • Software Testing

Case Month Day Year

ID (mm) (dd) (yyyy) Expected Output
1. 6 15 1900 16 June, 1900
2. 6 15 1901 16 June, 1901
3. 6 15 1962 16 June, 1962
4. 6 15 2024 16 June, 2024
5. 6 15 2025 16 June, 2025
6. 6 1 1962 2 June, 1962
7. 6 2 1962 1 June, 1962
8. 6 30 1962 1 July, 1962
9. 6 31 1962 Invalid date as June has 30 days.
10. 1 15 1962 16 January, 1962
11. 2 15 1962 16 February, 1962
12. 11 15 1962 16 November, 1962
13. 12 15 1962 16 December, 1962

So, we have applied BVA on our Next Date problem.

Question for Practice

1. Applying robustness testing, how would you generate the test cases for
the Next Date function given above.

[Link].  Test Cases for the Commission Problem

Before we generate the test cases, we must formulate the problem statement
or domain for commission problem.
Problem Domain: A rifle salesperson sold rifle locks, stocks, and barrels
that were made by a gunsmith. Locks cost $45, stocks cost $30, and barrels
cost $25. This salesperson had to sell at least one complete rifle per month,
and the production limits were such that the most the salesperson could sell
in a month was 70 locks, 80 stocks, and 90 barrels. The salesperson used to
send the details of sold items to the gunsmith. The gunsmith then computed
the salesperson’s commission as follows:
a. 10% on sales up to and including $1000
b. 15% of the next $800
c. 20% on any sales in excess of $1800

Software-Testing_Final.indb 72 31-01-2018 [Link]

 Black-Box (or Functional) Testing Techniques • 73

The commission program produced a monthly sales report that gave the total
number of locks, stocks, and barrels sold, the salesperson’s total dollar sales
and finally, the commission.

How to Generate BVA Test Cases for this Problem?

We have 3 inputs in this program. So, we need (4n + 1) = 4 * 3 + 1 = 12 +
1 = 13 test cases to test this program.
The boundary value test cases are listed below in Table. We can also find
that the monthly sales are limited as follows:
1 ≤ locks ≤ 70
1 ≤ stocks ≤ 80
1 ≤ barrels ≤ 90

Case ID Locks Stocks Barrels Sales

1. 35 40 1 2800
2. 35 40 2 2825
3. 35 40 45 3900
4. 35 40 89 5000
5. 35 40 90 5025
6. 35 1 45 2730
7. 35 2 45 2760
8. 35 40 45 3900
9. 35 79 45 5070
10. 35 80 45 5100
11. 1 40 45 2370
12. 2 40 45 2415
13. 35 40 45 3900
14. 69 40 45 5430
15. 70 40 45 5475

Out of these 15 test cases, 2 are redundant. So, 13 test cases are ­sufficient
to test this program.

Software-Testing_Final.indb 73 31-01-2018 [Link]

 74 • Software Testing

3.1.6. Guidelines for BVA

1. The normal versus robust values and the single-fault versus the multiple-
fault assumption theory result in better testing. These methods can be
applied to both input and output domain of any program.
2. Robustness testing is a good choice for testing internal variables.
3. Keep in mind that you can create extreme boundary results from non-
extreme input values.

3.2.  EQUIVALENCE CLASS TESTING

The use of equivalence classes as the basis for functional testing has two
motivations:
a. We want exhaustive testing
b. We want to avoid redundancy
This is not handled by the BVA technique as we can see massive redun-
dancy in the tables of test cases.

FIGURE 3.4  Equivalence Class Partitioning.

In this technique, the input and the output domain is divided into a
finite number of equivalence classes. Then, we select one representative of
each class and test our program against it. It is assumed by the tester that
if one representative from a class is able to detect error then why should
he consider other cases. Furthermore, if this single representative test case
did not detect any error then we assume that no other test case of this class
can detect error. In this method, we consider both valid and invalid input
domains. The system is still treated as a black-box meaning that we are not
bothered about its internal logic.
The idea of equivalence class testing is to identify test cases by using one
element from each equivalence class. If the equivalence classes are chosen
wisely, the potential redundancy among test cases can be reduced.

Software-Testing_Final.indb 74 31-01-2018 [Link]

 Black-Box (or Functional) Testing Techniques • 75

For example, in our triangle problem, we would certainly have a test

case for an equilateral triangle and we might pick the triple (10, 10, 10) as
inputs for a test case. If this is so then it is obvious that there is no sense in
testing for inputs like (8, 8, 8) and (100, 100, 100). Our intuition tells us that
these would be “treated the same” as the first test case. Thus, they would be
redundant. The key and the craftsmanship lies in the choice of the equiva-
lence relation that determines the classes.
Four types of equivalence class testing are discussed below:
1. Weak normal equivalence class testing
2. Strong normal equivalence class testing
3. Weak robust equivalence class testing
4. Strong robust equivalence class testing
We will discuss these one by one.

3.2.1. Weak Normal Equivalence Class Testing

The word “weak” means single fault assumption. This type of testing is
accomplished by using one variable from each equivalence class in a test
case. We would, thus, end up with the weak equivalence class test cases as
shown in Figure 3.5.
Each dot in Figure 3.5 indicates a test data. From each class we have one
dot meaning that there is one representative element of each test case.

FIGURE 3.5  Weak Normal Equivalence Class Test Cases.

Software-Testing_Final.indb 75 31-01-2018 [Link]

 76 • Software Testing

In fact, we will have, always, the same number of weak equivalence class test
cases as the classes in the partition.

3.2.2. Strong Normal Equivalence Class Testing

This type of testing is based on the multiple fault assumption theory. So,
now we need test cases from each element of the Cartesian product of
the equivalence classes, as shown in Figure 3.6.

FIGURE 3.6  Strong Normal Equivalence Class Test Cases.

Just like we have truth tables in digital logic, we have similarities between
these truth tables and our pattern of test cases. The Cartesian product guar-
antees that we have a notion of “completeness” in two ways:
a. We cover all equivalence classes.
b. We have one of each possible combination of inputs.

3.2.3.  Weak Robust Equivalence Class Testing

The name for this form of testing is counter intuitive and oxymoronic. The
word “weak” means single fault assumption theory and the word “robust”
refers to invalid values. The test cases resulting from this strategy are shown
in Figure 3.7.
Two problems occur with robust equivalence testing. They are listed
below:
1. Very often the specification does not define what the expected output for
an invalid test case should be. Thus, testers spend a lot of time defining
expected outputs for these cases.

Software-Testing_Final.indb 76 31-01-2018 [Link]

 Black-Box (or Functional) Testing Techniques • 77

2. Also, strongly typed languages like Pascal and Ada, eliminate the need
for the consideration of invalid inputs. Traditional equivalence testing
is a product of the time when languages such as FORTRAN, C, and
COBOL were dominant. Thus, this type of error was common.

FIGURE 3.7  Weak Robust Equivalence Class Test Cases.

3.2.4. Strong Robust Equivalence Class Testing

This form of equivalence class testing is neither counter intuitive nor oxymo-
ronic but is redundant. As explained earlier “robust” means consideration of
invalid values and “strong” means multiple fault assumption. We obtain the
test cases from each element of the Cartesian product of all the equivalence
classes. This is shown in Figure 3.8.
We find here that we have 8 robust (invalid) test cases and 12 strong or
valid inputs. Each is represented with a dot. So, totally we have 20 test cases
(represented as 20 dots) using this technique.

FIGURE 3.8  Strong Robust Equivalence Class Test Cases.

Software-Testing_Final.indb 77 31-01-2018 [Link]

 78 • Software Testing

3.2.5. Solved Examples

[Link]. Equivalence Class Test Cases for the Triangle Problem

As stated in the problem definition earlier, we note that in our triangle prob-
lem four possible outputs can occur:
a. NOT-A-TRIANGLE
b. Scalene
c. Isosceles
d. Equilateral
We can use these to identify output (range) equivalence classes as
­follows:
01 = {<a, b, c>: the triangle is equilateral}
02 = {<a, b, c>: the triangle is isosceles}
03 = {<a, b, c>: the triangle is scalene}
04 = {<a, b, c>: sides a, b, and c do not form a triangle}
Now, we apply these four techniques of equivalence class partitioning
one by one to this problem.
a. The four weak normal equivalence class test cases are:

Case ID a b c Expected output

WN1 5 5 5 Equilateral
WN2 2 2 3 Isosceles
WN3 3 4 5 Scalene
WN4 4 1 2 Not a triangle

b. Because no valid subintervals of variables a, b, and c exist, so the strong

normal equivalence class test cases are identical to the weak normal
equivalence class test cases.
c. Considering the invalid values for a, b, and c yields the following
additional weak robust equivalence class test cases:

Software-Testing_Final.indb 78 31-01-2018 [Link]

 Black-Box (or Functional) Testing Techniques • 79

Case ID a b c Expected output

WR1 –1 5 5 Invalid value of a
WR2 5 –1 5 Invalid value of b
WR3 5 5 –1 Invalid value of c
WR4 201 5 5 Out of range value of a
WR5 5 201 5 Out of range value of b
WR6 5 5 201 Out of range value of c

d. While strong robust equivalance class test cases are:

Case ID a b c Expected output

SR1 –1 5 5 Invalid value of a
SR2 5 –1 5 Invalid value of b
SR3 5 5 –1 Invalid value of c
SR4 –1 –1 5 Invalid values of a and b
SR5 5 –1 –1 Invalid values of b and c
SR6 –1 5 –1 Invalid values of a and c
SR7 –1 –1 –1 Invalid values of a, b, and c

Please note that the expected outputs describe the invalid input values
thoroughly.

[Link]. Equivalence Class Test Cases for Next Date Function

The actual craft of choosing the test cases lies in this example. Recall that
Next Date is a function of three variables— month (mm), day (dd), and year
(yyyy). Assume that their ranges are
1 ≤ month ≤ 12
1 ≤ day ≤ 31
1812 ≤ year ≤ 2012
So, based on valid values, the equivalence classes are:
M1 = {month: 1 ≤ month ≤ 12}
D1 = {day: 1 ≤ day ≤ 31}
Y1 = {year: 1812 ≤ year ≤ 2012}

Software-Testing_Final.indb 79 31-01-2018 [Link]

 80 • Software Testing

And the invalid equivalence classes are:

M2 = {month: month < 1}
M3 = {month: month > 12}
D2 = {day: day < 1}
D3 = {day: day > 31}
Y2 = {year: year < 1812}
Y3 = {year: year > 2012}

a. & b. Because the number of valid classes equals the number of

independent variables, only one weak normal equivalence class test
case occurs and it is identical to the strong normal equivalence class
test case:

Case ID Month Day Year Expected output

WN1, SN1 6 15 1912 6/16/1912

So, we get this test case on the basis of valid classes – M1, D1, and Y1
above.
c. Weak robust test cases are given below:

Case ID Month Day Year Expected output

WR1 6 15 1912 6/16/1912
WR2 –1 15 1912 Invalid value of month as
month cannot be –ve.
WR3 13 15 1912 Invalid value of month as
month <12, always.
WR4 6 –1 1912 Invalid value of day as day
­cannot be –ve.
WR5 6 32 1912 Invalid value of day as we
­cannot have 32 days in any
month.
WR6 6 15 1811 Invalid value of year as its range
is 1812 to 2012 only.
WR7 6 15 2013 Invalid value of year.

Software-Testing_Final.indb 80 31-01-2018 [Link]

 Black-Box (or Functional) Testing Techniques • 81

So, we get 7 test cases based on the valid and invalid classes of the input
domain.
d. Strong robust equivalence class test cases are given below:

Case ID Month Day Year Expected output

SR1 –1 15 1912 Invalid value of month as
month cannot be –ve.
SR2 6 –1 1912 Invalid value of day as day
is –ve.
SR3 6 15 1811 Invalid value of year.
SR4 –1 –1 1912 Invalid month and day.
SR5 6 –1 1811 Invalid day and year.
SR6 –1 15 1811 Invalid month and year.
SR7 –1 –1 1811 Invalid month, day, and
year.

Modified Equivalence Class for this Problem

We need the modified classes as we know that at the end of a month, the next
day is 1 and the month is incremented. At the end of a year, both the day
and the month are reset to 1 and the year is also incremented. Finally, the
problem of leap year makes determining the last day of a month interesting.
With all this in mind, we postulate the following equivalence classes:
M1 = {month: month has 30 days}
M2 = {month: month has 31 days}
M3 = {month: month is February}
D1 = {day: 1 ≤ day ≤ 28}
D2 = {day: day = 29}
D3 = {day: day = 30}
D4 = {day: day = 31}
Y1 = {year: year = 2000}
Y2 = {year: year is a leap year}
Y3 = {year: year is a common year}
So, now what will be the weak equivalence class test cases?

Software-Testing_Final.indb 81 31-01-2018 [Link]

 82 • Software Testing

As done earlier, the inputs are mechanically selected from the approxi-
mate middle of the corresponding class:

Case ID Month Day Year Expected output

WN1 6 14 2000 6/15/2000
WN2 7 29 1996 7/30/1996
WN3 2 30 2002 2/31/2002 (Impossible)
WN4 6 31 2000 7/1/2000 (Impossible)

This mechanical selection of input values makes no consideration of our

domain knowledge and thus, we have two impossible dates. This will always
be a problem with “automatic” test case generation because all of our domain
knowledge is not captured in the choice of equivalence classes.
The strong normal equivalence class test cases for the revised classes are:

Case ID Month Day Year Expected output

SN1 6 14 2000 6/15/2000
SN2 6 14 1996 6/15/1996
SN3 6 14 2002 6/15/2002
SN4 6 29 2000 6/30/2000
SN5 6 29 1996 6/30/1996
SN6 6 29 2002 6/30/2002
SN7 6 30 2000 6/31/2000 (Impossible date)
SN8 6 30 1996 6/31/1996 (Impossible date)
SN9 6 30 2002 6/31/2002 (Impossible date)
SN10 6 31 2000 7/1/2000 (Invalid input)
SN11 6 31 1996 7/1/1996 (Invalid input)
SN12 6 31 2002 7/1/2002 (Invalid input)
SN13 7 14 2000 7/15/2000
SN14 7 14 1996 7/15/1996
SN15 7 14 2002 7/15/2002

Software-Testing_Final.indb 82 31-01-2018 [Link]

 Black-Box (or Functional) Testing Techniques • 83

Case ID Month Day Year Expected output

SN16 7 29 2000 7/30/2000
SN17 7 29 1990 7/30/1990
SN18 7 29 2002 7/30/2002
SN19 7 30 2000 7/31/2000
SN20 7 30 1996 7/31/1996
SN21 7 30 2002 7/31/2002
SN22 7 31 2000 8/1/2000
SN23 7 31 1996 8/1/1996
SN24 7 31 2002 8/1/2002
SN25 2 14 2000 2/15/2000
SN26 2 14 1996 2/15/1996
SN27 2 14 2002 2/15/2002
SN28 2 29 2000 3/1/2000 (Invalid input)
SN29 2 29 1996 3/1/1996
SN30 2 29 2002 3/1/2002 (Impossible date)
SN31 2 30 2000 3/1/2000 (Impossible date)
SN32 2 30 1996 3/1/1996 (Impossible date)
SN33 2 30 2002 3/1/2002 (Impossible date)
SN34 6 31 2000 7/1/2000 (Impossible date)
SN35 6 31 1996 7/1/1996 (Impossible date)
SN36 6 31 2002 7/1/2002 (Impossible date)

So, three month classes, four day classes, and three year classes results
in 3 × 4 × 3 = 36 strong normal equivalence class test cases. Furthermore,
adding two invalid classes for each variable will result in 150 strong robust
equivalence class test cases.
It is difficult to show these 150 classes here.

Software-Testing_Final.indb 83 31-01-2018 [Link]

 84 • Software Testing

[Link]. Equivalence Class Test Cases for the Commission Problem

The valid classes of the input variables are as follows:
L1 = {locks: 1 ≤ locks ≤ 70}
L2 = {locks = –1}
S1 = {stocks: 1 ≤ stocks ≤ 80}
B1 = {barrels: 1 ≤ barrels ≤ 90}
The corresponding invalid classes of the input variables are as follows:
L3 = {locks: locks = 0 or locks < –1}
L4 = {locks: locks > 70}
S2 = {stocks: stocks < 1}
S3 = {stocks: stocks > 80}
B2 = {barrels: barrels < 1}
B3 = {barrels: barrels > 90}

a. & b.  As the number of valid classes is equal to the number of

independent variables, so we have exactly one weak normal equivalence
class test case and again, it is identical to the strong normal equivalence
class test case. It is given in the following table.

Case ID Locks Stocks Barrels Expected output

WN1, SN1 35 40 45 3900

c. Also, we have seven weak robust test cases as given below:

Case ID Locks Stocks Barrels Expected output

WR1 35 40 45 3900
WR2 0 40 45 Invalid input
WR3 71 40 45 Invalid input
WR4 35 0 45 Invalid input
WR5 35 81 45 Invalid input
WR6 35 40 0 Invalid input
WR7 35 40 91 Invalid input

Software-Testing_Final.indb 84 31-01-2018 [Link]

 Black-Box (or Functional) Testing Techniques • 85

d. And finally, the strong robust equivalence class test cases are as follows:

Case ID Locks Stocks Barrels Expected output

SR1 –1 40 45 Value of locks not in the range 1–70.
SR2 35 –1 45 Value of stocks not in the range 1–80.
SR3 35 40 –1 Value of barrels not in the range 1–90.
SR4 –1 –1 45 Values of locks and stocks are not in their
ranges.
SR5 –1 40 –1 Values of locks and barrels are not in
their ranges.
SR6 35 –1 –1 Values of stocks and barrels are not in
their ranges.
SR7 –1 –1 –1 Values of locks, stocks, and barrels are not
in their ranges.

3.2.6. Guidelines for Equivalence Class Testing

The following are guidelines for equivalence class testing:
1. The weak forms of equivalence class testing (normal or robust) are not
as comprehensive as the corresponding strong forms.
2. If the implementation language is strongly typed and invalid values cause
run-time errors then it makes no sense to use the robust form.
3. If error conditions are a high priority, the robust forms are appropriate.
4. Equivalence class testing is approximate when input data is defined in
terms of intervals and sets of discrete values. This is certainly the case
when system malfunctions can occur for out-of-limit variable values.
5. Equivalence class testing is strengthened by a hybrid approach with
boundary value testing (BVA).
6. Equivalence class testing is used when the program function is complex.
In such cases, the complexity of the function can help identify useful
equivalence classes, as in the next date problem.
7. Strong equivalence class testing makes a presumption that the variables
are independent and the corresponding multiplication of test cases
raises issues of redundancy. If any dependencies occur, they will often
generate “error” test cases, as shown in the next date function.

Software-Testing_Final.indb 85 31-01-2018 [Link]

 86 • Software Testing

8. Several tries may be needed before the “right” equivalence relation is

established.
9. The difference between the strong and weak forms of equivalence class
testing is helpful in the distinction between progression and regression
testing.

3.3.  DECISION TABLE BASED TESTING

Of all the functional testing methods, those based on decision tables are the
most rigorous because decision tables enforce logical rigor.

3.3.1.  What Are Decision Tables?

Decision tables are a precise and compact way to model complicated logic.
They are ideal for describing situations in which a number of combinations
of actions are taken under varying sets of conditions.

FIGURE 3.9  Structure of Decision Table.

It is another popular black-box testing technique. A decision table

has four portions:
a. Stub portion b. Entry portion
c. Condition portion d. Action portion
A column in the entry portion is a rule. Rules indicate which actions are
taken for the conditional circumstances indicated in the condition portion of
the rule. Decision tables in which all conditions are binary are called limited
entry decision tables. If conditions are allowed to have several values, the
resulting tables are called extended entry decision tables.
To identify test cases with decision tables, we follow certain steps:
Step 1. F
 or a module identify input conditions (causes) and action
(effect).
Step 2. Develop a cause-effect graph.

Software-Testing_Final.indb 86 31-01-2018 [Link]

 Black-Box (or Functional) Testing Techniques • 87

Step 3. T
 ransform this cause-effect graph, so obtained in step 2 to a
decision table.
Step 4. C
 onvert decision table rules to test cases. Each column of the
decision table represents a test case. That is,

Number of Test Cases = Number of Rules

n
For a limited entry decision table, if n conditions exist, there must be 2
rules.

3.3.2. Advantages, Disadvantage, and Applications of Decision Tables

Advantages of Decision Tables
1. This type of testing also works iteratively. The table that is drawn in the
first iteration acts as a stepping stone to derive new decision table(s) if
the initial table is unsatisfactory.
2. These tables guranatee that we consider every possible combination
of condition values. This is known as its “completeness property.” This
property promises a form of complete testing as compared to other
techniques.
3. Decision tables are declarative. There is no particular order for conditions
and actions to occur.

Disadvantages of Decision Tables

Decision tables do not scale up well. We need to “factor” large tables into
smaller ones to remove redundancy.

Applications of Decision Tables

This technique is useful for applications characterized by any of the ­following:
a. Prominent if-then-else logic.
b. Logical relationships among input variables.
c. Calculations involving subsets of the input variables.
d. Cause-and-effect relationships between inputs and outputs.
e. High cyclomatic complexity.

Software-Testing_Final.indb 87 31-01-2018 [Link]

 88 • Software Testing

Technique: To identify test cases with decision tables, we interpret condi-

tions as inputs and actions as output. The rules are interpreted as test cases.
Because the decision table can mechanically be forced to be complete, we
know we have a comprehensive set of test cases.
Example of a Decision Table: The triangle problem

R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11
C1: a < b + c? F T T T T T T T T T T
C2: b < a + c? — F T T T T T T T T T
C3: c < a + b? — — F T T T T T T T T
C4: c = b? — — — T T T T F F F F
C5: a = c? — — — T T F F T T F F
C6: b = c? — — — T F T F T F T F
a1: Not a triangle × × ×
a2: Scalene ×
a3: Isosceles × × ×
a4: Equilateral ×
a5: Impossible × × ×

FIGURE 3.10  Example of Decision Table.

Each “-” (hyphen) in the decision table represents a “don’t care” entry.
Use of such entries has a subtle effect on the way in which complete decision
tables are recognized. For limited entry decision tables, if n conditions exist,
there must be 2n rules. When don’t care entries indicate that the condition is
irrelevant, we can develop a rule count as follows:
Rule 1. Rules in which no “don’t care” entries occur count as one rule.
Note that each column of a decision table represents a rule and the
­number of rules is equal to the number of test cases.
Rule 2.  Each “don’t care” entry in a rule doubles the count of that rule.
Note that in this decision table we have 6 conditions (C1—C6). ­Therefore,
n=6
Also, we can have 2n entries, i.e., 26 = 64 entries. Now we establish the
rule and the rule count for the above decision table.

Software-Testing_Final.indb 88 31-01-2018 [Link]

 Black-Box (or Functional) Testing Techniques • 89

R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11
C1: a < b + c? F T T T T T T T T T T
C2: b < a + c? — F T T T T T T T T T
C3: c < a + b? — — F T T T T T T T T
C4: a = b? — — — T T T T F F F F
C5: a = c? — — — T T F F T T F F
C6: b = c? — — — T F T F T F T F
Rule Count 32 16 8 1 1 1 1 1 1 1 1 = 64
a1: Not a triangle × × ×
a2: Scalene ×
a3: Isosceles × × ×
a4: Equilateral ×
a5: Impossible × × ×

FIGURE 3.11  Decision Table With Rule Counts.

From the previous table we find that the rule count is 64. And we have
already said that 2n = 26 = 64. So, both are 64.
The question, however, is to find out why the rule count is 32 for the
Rule-1 (or column-1)?
We find that there are 5 don’t cares in Rule-1 (or column-1) and hence
2n = 25 = 32. Hence, the rule count for Rule-1 is 32. Similarly, for Rule-2, it
is 24 = 16 and 23 = 8 for Rule-3. However, from Rule-4 through Rule-11, the
number of don’t care entries is 0 (zero). So rule count is 20 = 1 for all these
columns. Summing the rule count of all columns (or R1-R11) we get a total
of 64 rule count.
Many times some problems arise with these decision tables. Let us
see how.
Consider the following example of a redundant decision table:

Conditions 1–4 5 6 7 8 9
C1 T F F F F T
C2 — T T F F F
C3 — T F T F F
a1 × × × — — —
a2 — × × × — ×
a3 × — × × × ×

FIGURE 3.12  Redundant Decision Table.

Software-Testing_Final.indb 89 31-01-2018 [Link]

 90 • Software Testing

Please note that the action entries in Rule-9 and Rules 1–4 are NOT
identical. It means that if the decision table were to process a transaction in
which C1 is true and both C2 and C3 are false, both rules 4 and 9 apply. We
observe two things
1. Rules 4 and 9 are in-consistent because the action sets are different.
2. The whole table is non-deterministic because there is no way to decide
whether to apply Rule-4 or Rule-9.
Also note carefully that there is a bottom line for testers now. They
should take care when don’t care entries are being used in a decision table.

3.3.3. Examples

[Link]. Test Cases for the Triangle Problem Using Decision Table Based
Testing Technique
We have already studied the problem domain for the famous triangle prob-
lem in previous chapters. Next we apply the decision table based technique
on the triangle problem. The following are the test cases:

Case ID a b c Expected output

D1 4 1 2 Not a triangle
D2 1 4 2 Not a triangle
D3 1 2 4 Not a triangle
D4 5 5 5 Equilateral
D5 ? ? ? Impossible
D6 ? ? ? Impossible
D7 2 2 3 Isosceles
D8 ? ? ? Impossible
D9 2 3 2 Isosceles
D10 3 2 2 Isosceles
D11 3 4 5 Scalene
FIGURE 3.13  Test Cases For Triangle Problem.

So, we get a total of 11 functional test cases out of which three are
impossible cases, three fail to satisfy the triangle property, one satisfies the
equilateral triangle property, one satisfies the scalene triangle property, and
three ways to get an isoceles triangle.

Software-Testing_Final.indb 90 31-01-2018 [Link]

 Black-Box (or Functional) Testing Techniques • 91

[Link].  Test Cases for Next Date Function

In the previous technique of equivalence partitioning we found that cer-
tain logical dependencies exist among variables in the input domain. These
dependencies are lost in a Cartesian product. The decision table format allows
us to use the notion of “impossible action” to denote impossible combinations
of conditions. Thus, such dependencies are easily shown in decision tables.
From the problem domain for the next date function, we know that there
are some critical situations like the treatment of leap years, special treatment
to year 2000 (Y2K), and special treatment to December month and a 28 day
month is also to be given. So, we go for 3 tries before we derive its test cases.
First try: Special treatment to leap years.
Second try: Special treatment to year = 2000.
Third try: Special treatment to December month and days = 28.
First try: Special treatment to leap years.
The art of testing and the actual craftsmanship lies in identifying appropriate
conditions and actions. Considering the following equivalence classes again:
M1 = {Month: Month has 30 days}
M2 = {Month: Month has 31 days}
M3 = {Month: Month is February}
D1 = {day: 1 ≤ day ≤ 28}
D2 = {day: day = 29}
D3 = {day: day = 30}
D4 = {day: day = 31}
Y1 = {year: year is a leap year}
Y2 = {year: year is not a leap year}
Based on these classes, we draw the decision table:

Conditions
C1: month in M1 T
C2: month in M2 T
C3: month in M3 T
C4: day in D1
C5: day in D2 :
C6: day in D3 :
C7: day in D4 : : :
C8: year in y1
a1: Impossible : : : : : : : :
a2: Next date

FIGURE 3.14  First Try Decision Table.

Software-Testing_Final.indb 91 31-01-2018 [Link]

 92 • Software Testing

Herein, we have 3 × 4 × 2 = 24 elements and 2n = 28 = 256 entries. Many

of the rules would be impossible (note that the classes Y1 and Y2 collapse into
one condition C8). Some of these rules are impossible because
a. there are too many days in a month.
b. they cannot happen in a non-leap year.
c. they compute the next date.
Second try: Special treatment given to year 2000.
As we know, the year 2000 is a leap year. Hence, we must give special treat-
ment to this year by creating a new class-Y1 as follows:
M1 = {month: month has 30 days}
M2 = {month: month has 31 days}
M3 = {month: month is February}
D1 = {day: 1 ≤ day ≤ 28}
D2 = {day: day = 29}
D3 = {day: day = 30}
D4 = {day: day = 31}
Y1 = {year: year = 2000}
Y2 = {year: year is a leap year}
Y3 = {year: year is a common year}
This results in 3 × 4 × 3 = 36 rules that represents the Cartesian product
of 3 month classes (M1.... M3), 4 day classes (D1....D4), and 3 year classes.
So, after second try, our decision table is
Why is the rule count value 3 in column 1?
In column 1 or Rule 1 of this decision table, we have 3 possibilities with
don’t care:
{M1, D1, Y1}
{M1, D1, Y2}
{M1, D1, Y3}
i.e., with “–” or don’t care we have either Y1 or Y2 or Y3.
Also note that in Rule 8, we have three impossible conditions, shown
as “?” (question mark). Also, we find that this table has certain problems with
December month. We will fix these next in the third try.
Third try: Special treatment to December month and to number of­
days = 28.

Software-Testing_Final.indb 92 31-01-2018 [Link]

Software-Testing_Final.indb 93
Rules →
Conditions 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
↓
C1: month in M1 M1 M1 M1 M2 M2 M2 M2 M3 M3 M3 M3 M3 M3 M3 M3
C2: day in D1 D2 D3 D4 D1 D2 D3 D4 D1 D1 D1 D2 D2 D2 D3 D4
C3: year in – – – – – – – – Y1 Y2 Y3 Y1 Y2 Y3 – –

Rule count 3 3 3 3 3 3 3 3 1 1 1 1 1 1 3 3 36 rule

count
Actions
× × × × ×
a1: impossible
a2: increment
day × × × × × ×
a3: reset day × × × × ×
a4: increment
month × ? × × ×
a5: reset month ?
a6: increment
year ?
FIGURE 3.15  Second Try Decision Table With 36 Rule Count.

31-01-2018 [Link]
 94 • Software Testing

Because we know that we have serious problems with the last day of
last month, i.e., December. We have to change month from 12 to 1. So, we
modify our classes as follows:
M1 = {month: month has 30 days}
M2 = {month: month has 31 days except December}
M3 = {month: month is December}
D1 = {day: 1 ≤ day ≤ 27}
D2 = {day: day = 28}
D3 = {day: day = 29}
D4 = {day: day = 30}
D5 = {day: day = 31}
Y1 = {year: year is a leap year}
Y2 = {year is a common year}
The Cartesian product of these contain 40 elements. Here, we have a
22-rule decision table. This table gives a clearer picture of the Next Date
function than does the 36-rule decision table and is given below:
In this table, the first five rules deal with 30-day months. Notice that the
leap year considerations are irrelevant. Rules (6 – 10) and (11 – 15) deal with
31-day months where the first five with months other than December and
the second five deal with December. No impossible rules are listed in this
portion of the decision table.
Still there is some redundancy in this table. Eight of the ten rules simply
increment the day. Do we really require eight separate test cases for this
sub-function? No, but note the type of observation we get from the decision
table.
Finally, the last seven rules focus on February and leap year. This deci-
sion table analysis could have been done during the detailed design of the
Next Date function.
Further simplification of this decision table can also be done. If the
action sets of two rules in a decision table are identical, there must be at least
one condition that allows two rules to be combined with a don’t care entry.
In a sense, we are identifying equivalence classes of these rules. For exam-
ple, rules 1, 2, and 3 involve day classes as D1, D2, and D3 (30 day classes).
These can be combined together as the action taken by them is the same.
Similarly, for other rules other combinations can be done. The correspond-
ing test cases are shown in the table as in Figure 3.17.

Software-Testing_Final.indb 94 31-01-2018 [Link]

Software-Testing_Final.indb 95
Rules →
Conditions 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
↓
C1: month in M1 M1 M1 M1 M1 M2 M2 M2 M2 M2 M3 M3 M3 M3 M3 M4 M4 M4 M4 M4 M4 M4
C2: day in D1 D2 D3 D4 D5 D1 D2 D3 D4 D5 D1 D2 D3 D4 D5 D1 D2 D2 D3 D3 D4 D5
C3: year in – – – – – – – – – – – – – – – – Y1 Y2 Y1 Y2 – –
Actions
× × × ×
a1: impossible

a2: increment × × × × × × × × × × × × × ×
day
a3: reset day × × × × ×
a4: increment × × × ×
month
a5: reset month ×
a6: increment ×
year
FIGURE 3.16  Decision Table for the Next Date Function.

31-01-2018 [Link]
 96 • Software Testing

Case ID Month Day Year Expected output

1–3 April 15 2001 April 16, 2001
4 April 30 2001 May 1, 2001
5 April 31 2001 Impossible
6–9 January 15 2001 January 16, 2001
10 January 31 2001 February 1, 2001
11–14 December 15 2001 December 16, 2001
15 December 31 2001 January 1, 2002
16 February 15 2001 February 16, 2001
17 February 28 2004 February 29, 2004
18 February 28 2001 March 1, 2001
19 February 29 2004 March 1, 2004
20 February 29 2001 Impossible
21–22 February 30 2001 Impossible
FIGURE 3.17  Test Cases for Next Date Problem Using Decision Table Based Testing.
Because, we have 22 rules there are 22 test cases that are listed above.

[Link]. Test Cases for the Commission Problem

The commission problem is not suitable to be solved using this technique of
decision table analysis because very little decisional logic is used in the prob-
lem. The variables in the equivalence classes are truly independent, there-
fore no impossible rules will occur in a decision table in which conditions
correspond to the equivalence classes. Thus, we will have the same test cases
as we did for equivalence class testing.

3.3.4. Guidelines for Decision Table Based Testing

The following guidelines have been found after studying the previous
­examples:
1. This technique works well where lot of decision making takes place such
as the triangle problem and next date problem.
2. The decision table technique is indicated for applications characterized
by any of the following:
„„ Prominant if-then-else logic.
„„ Logical relationships among input variables.
„„ Calculations involving subsets of the input variables.
„„ Cause-and-effect relationships between inputs and outputs.
„„ High cyclomatic complexity.

Software-Testing_Final.indb 96 31-01-2018 [Link]

 Black-Box (or Functional) Testing Techniques • 97

3. Decision tables do not scale up well. We need to “factor” large tables

into smaller ones to remove redundancy.
4. It works iteratively meaning that the table drawn in the first iteration,
and acts as a stepping stone to design new decision tables, if the initial
table is unsatisfactory.

3.4.  CAUSE-EFFECT GRAPHING TECHNIQUE

Cause-effect graphing is basically a hardware testing technique adapted
to software testing. It is a black-box method. It considers only the desired
external behavior of a system. This is a testing technique that aids in select-
ing test cases that logically relate causes (inputs) to effects (outputs) to pro-
duce test cases.

3.4.1. Causes and Effects

A “cause” represents a distinct input condition that brings about an internal
change in the system. An “effect” represents an output condition, a system
transformation, or a state resulting from a combination of causes.
Myer suggests the following steps to derive test cases:
Step 1. F
 or a module, identify the input conditions (causes) and actions
(effect).
Step 2. Develop a cause-effect graph.
Step 3. Transform the cause-effect graph into a decision table.
Step 4. C
 onvert decision table rules to test cases. Each column of the
decision table represents a test case.
Basic cause-effect graph symbols used are given below:

Software-Testing_Final.indb 97 31-01-2018 [Link]

 98 • Software Testing

Consider each node as having the value 0 or 1 where 0 represents the

“absent state” and 1 represents the “present state.” Then the identity func-
tion states that if c1 is 1, e1 is 1, or we can say if c1 is 0, e1 is 0.
The NOT function states that if C1 is 1, e1 is 0 and vice versa. Similarly,
the OR function states that if C1 or C2 or C3 is 1, e1 is 1 else e1 is 0. The AND
function states that if both C1 and C2 are 1, e1 is 1; else e1 is 0. The AND and
OR functions are allowed to have any number of inputs.

3.4.2. Test Cases for the Triangle Problem

We follow the steps listed in Section 3.4.1 to design the test cases for our
triangle problem:
Step 1. First, we must identify the causes and its effects. The causes are:
C1: Side x is less than sum of y and z
C2: Side y is less than sum of x and z
C3: Side z is less than sum of x and y
C4: Side x is equal to side y
C5: Side x is equal to side z
C6: Side y is equal to side z
The effects are:
e1: Not a triangle
e2: Scalene triangle
e3: Isosceles triangle
e4: Equilateral triangle
e5: Impossible
Step 2. Its cause-effect graph
is shown in Figure 3.18.
Step 3. We transform it into a
decision table: FIGURE 3.18  Cause Effect Graph for
Triangle Problem.

Software-Testing_Final.indb 98 31-01-2018 [Link]

 Black-Box (or Functional) Testing Techniques • 99

Conditions R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11

C1: x < y + z? 0 1 1 1 1 1 1 1 1 1 1
C2: y < x + z? × 0 1 1 1 1 1 1 1 1 1
C3: z < x + y? × × 0 1 1 1 1 1 1 1 1
C3: x = y? × × × 1 1 1 1 0 0 0 0
C4: x = z? × × × 1 1 0 0 1 1 0 0
C5: x = z? × × × 1 1 0 0 1 1 0 0
C6: y = z? × × × 1 0 1 0 1 0 1 0
e1: Not a triangle 1 1 1
e2: Scalene 1
e3: Isosceles 1 1 1
e4: Equilateral 1
e5: Impossible 1 1 1

Step 4. Because there are 11 rules, we get 11 test cases and they are:

Test case x y z Expected output

1 4 1 2 Not a triangle
2 1 4 2 Not a triangle
3 1 2 4 Not a triangle
4 5 5 5 Equilateral
5 ? ? ? Impossible
6 ? ? ? Impossible
7 2 2 3 Isosceles
8 ? ? ? Impossible
9 2 3 2 Isosceles
10 3 2 2 Isosceles
11 3 4 5 Scalene

Software-Testing_Final.indb 99 31-01-2018 [Link]

 100 • Software Testing

3.4.3. Test Cases for Payroll Problem

Problem 1. Consider the payroll system of a person.
a. If the salary of a person is less than $70,000 and expenses do not exceed
$30,000, then a 10% tax is charged by the IT department.
b. If the salary is greater than $60,000 and less than or equal to $3000 and
expenses don’t exceed $40,000, then a 20% tax is charged by the IT
department.
c. For a salary greater than $3000, a 5% additional surcharge is also charged.
d. If expenses are greater than $40,000, the surcharge is 9%.
Design test-cases using decision table based testing technique.
Solution. See the following steps:
Step 1. All causes and their effects are identified:
Causes Effects
C1: Salary < = 70,000 E1: 10% tax is charged.
C2: Salary > 60,000 and Salary < = 3000 E2: 20% tax is charged.
C3: Salary > 3000 E3: (20% tax) + (5% surcharge) is
C4: Expenses < = 30,000 charged.
C5: Expenses < = 40,000 E4: (20% tax) + (9% surcharge) is
C6: Expenses > 40,000 charged.

Step 2. It’s cause-effect graph is drawn.

FIGURE 3.19  Cause Effects Graph.

Software-Testing_Final.indb 100 31-01-2018 [Link]

 Black-Box (or Functional) Testing Techniques • 101

Step 3. We transform this cause-effect graph into a decision table. Please

note that these “causes” and “effects” are nothing else but “conditions” and
“actions” of our decision table. So, we get:

1 2 3 4
Conditions C1 1 0 0 0
(or Causes) C2 0 1 0 0
C3 0 0 1 1
C4 1 0 0 0
C5 0 1 1 0
C6 0 0 0 1
Actions E1 × — — —
(or Effects) E2 — × — —
E3 — — × —
E4 — — — ×

FIGURE 3.20  Decision Table.

That is, if C1 and C4 are 1 (or true) then the effect (or action) is E1. Simi-
larly, if C2 and C5 is 1 (or true), action to be taken is E2, and so on.
Step 4. Because there are 4 rules in our decision table above, we must have
at least 4 test cases to test this system using this technique.
These test cases can be:
1. Salary = 20,000, Expenses = 2000
2. Salary = 100,000, Expenses = 10,000
3. Salary = 300,000, Expenses = 20,000
4. Salary = 300,000, Expenses = 50,000
So we can say that a decision table is used to derive the test cases which
can also take into account the boundary values.

3.4.4. Guidelines for the Cause-Effect Functional Testing Technique

1. If the variables refer to physical quantities, domain testing and
equivalence class testing are indicated.

Software-Testing_Final.indb 101 31-01-2018 [Link]

 102 • Software Testing

2. If the variables are independent, domain testing and equivalence class

testing are indicated.
3. If the variables are dependent, decision table testing is indicated.
4. If the single-fault assumption is warranted, boundary value analysis
(BVA) and robustness testing are indicated.
5. If the multiple-fault assumption is warranted, worst-case testing, robust
worst-case testing, and decision table testing are identical.
6. If the program contains significant exception handling, robustness
testing and decision table testing are indicated.
7. If the variables refer to logical quantities, equivalence class testing and
decision table testing are indicated.

3.5. COMPARISON ON BLACK-BOX (OR FUNCTIONAL)

TESTING TECHNIQUES

3.5.1. Testing Effort
The functional methods that we have studied so far vary both in terms of the
number of test cases generated and the effort to develop these test cases.
To compare the three techniques, namely, boundary value analysis (BVA),
equivalence class partitioning, and decision table based technique, we con-
sider the following curve shown in Figure 3.21.

FIGURE 3.21  Test Cases as Per the Testing Method.

Software-Testing_Final.indb 102 31-01-2018 [Link]

 Black-Box (or Functional) Testing Techniques • 103

The domain-based techniques have no recognition of data or logical

dependencies. They are very mechanical in the way they generate test cases.
Because of this, they are also easy to automate. The techniques like equiv-
alence class testing focus on data dependencies and thus we need to show
our craft. The thinking goes into the identification of the equivalence classes
and after that, the process is mechanical. Also note that from the graph, the
decision table based technique is the most sophisticated because it requires
the tester to consider both data and logical dependencies. As we have seen in
our example of the next date function, we had to go three times but once we
get a good and healthy set of conditions, the resulting test cases are complete
and minimal.
Now, consider another graph to show the effort required to identify the
test cases versus its sophistication.

FIGURE 3.22  Test Case Identification Effort as per Testing Method.

We can say that the effort required to identify test cases is the lowest in
BVA and the highest in decision tables. The end result is a trade-off between
the test case effort identification and test case execution effort. If we shift
our effort toward more sophisticated testing methods, we reduce our test
execution time. This is very important as tests are usually executed several
times. Also note that, judging testing quality in terms of the sheer number
of test cases has drawbacks similar to judging programming productivity in
terms of lines of code.
The examples that we have discussed so far show these trends.

Software-Testing_Final.indb 103 31-01-2018 [Link]

 104 • Software Testing

3.5.2. Testing Efficiency
What we found in all of these functional testing strategies is that either the
functionality is untested or the test cases are redundant. So, gaps do occur in
functional test cases and these gaps are reduced by using more sophisticated
techniques.
We can develop various ratios of the total number of test cases generated
by method-A to those generated by method-B or even ratios on a test case
basis. This is more difficult but sometimes management demands numbers
even when they have little meaning. When we see several test cases with the
same purpose, sense redundancy, detecting the gaps is quite difficult. If we
use only functional testing, the best we can do is compare the test cases that
result from two methods. In general, the more sophisticated method will
help us recognize gaps but nothing is guaranteed.

3.5.3. Testing Effectiveness
How can we find out the effectiveness of the testing techniques?
a. By being dogmatic, we can select a method, use it to generate test
cases, and then run the test cases. We can improve on this by not
being dogmatic and allowing the tester to choose the most appropriate
method. We can gain another incremental improvement by devising
appropriate hybrid methods.
b. The second choice can be the structural testing techniques for the test
effectiveness. This will be discussed in subsequent chapters.
Note, however, that the best interpretation for testing effectiveness is
most difficult. We would like to know how effective a set of test cases is for
finding faults present in a program. This is problematic for two reasons.
1. It presumes we know all the faults in a program.
2. Proving that a program is fault free is equivalent to the famous halting
problem of computer science, which is known to be impossible.

Software-Testing_Final.indb 104 31-01-2018 [Link]

 Black-Box (or Functional) Testing Techniques • 105

What Is the Best Solution?

The best solution is to work backward from fault types. Given a particular
kind of fault, we can choose testing methods (functional and structural) that
are likely to reveal faults of that type. If we couple this with knowledge of
the most likely kinds of faults, we end up with a pragamatic approach to test-
ing effectiveness. This is improved if we track the kinds of faults and their
frequencies in the software we develop.

3.5.4. Guidelines for Functional Testing

1. If the variables refer to physical quantities then domain testing and
equivalence class testing are used.
2. If the variables are independent then domain testing and equivalence
class testing are used.
3. If the variables are dependent, decision table testing is indicated.
4. If the single-fault assumption is warranted then BVA and robustness
testing are used.
5. If the multiple-fault assumption is warranted then worst case testing,
robust worst case testing, and decision table based testing are indicated.
6. If the program has significant exception handling, robustness testing and
decision table based testing are identical.
7. If the variable refers to logical quantities, equivalence class testing and
decision table testing are indicated.

3.6. KIVIAT CHARTS

A kiviat chart visually displays a set of metrics that provides easy viewing of
multiple metrics against minimum and maximum thresholds. Each radial of
a Kiviat chart is a metric. All metrics are scaled so that all maximums are on
a common circle and all minimums are on a common circle.
In the charts below, the circle is the maximum threshold and the band is
the minimum threshold. The band between the two is the acceptable range.

Software-Testing_Final.indb 105 31-01-2018 [Link]

 106 • Software Testing

The chart on the left shows that all metrics are well within the acceptable
range. The chart on the right shows an example where all metrics are above
maximum limits.

FIGURE 3.23  DevCodeMetricsWeb screen shot with Kiviat:

[Link]

The best graphical or visual representation of complex concepts and data

can be communicated through Kiviat charts. In addition, Kiviat charts pro-
vide a quick focus on the areas that are in most need of attention and can
be customized to use different scales to represent various problems. This
deliverable will help you get smart on what a Kiviat chart is and how to use
it properly.

Software-Testing_Final.indb 106 31-01-2018 [Link]

 Black-Box (or Functional) Testing Techniques • 107

For an update on Kiviat charts using R! visit: [Link]

NOTE
com/diagramas-kiviat-spider-charts-em-r/

3.6.1. The Concept of Balance

Why Is It Needed?
The problem – multiple dimensions:
„„ Computer: processor
•• printer
•• disks
•• CDROM
•• modem
„„ Software
„„ Personnel
All these together create multiple dimensions and hence the concept of bal-
ance is needed.

Kiviat Charts: The Method

The following steps are usually followed to draw Kiviat charts:
Step 1: Choose factors to be measured.
Step 2: D
 efine factors so that for half the optimum utilization is 1 and
the other half is 0.
Step 3: M
 ark the factors around the chart so that axes with an optimum
of 0 alternate with those with an optimum of 1.
Step 4: Mark the values on the factor axes.
Step 5: Join the marks to form a “star.”
Step 6: E
 valuate the result.
We shall now consider four different cases to draw Kiviat charts.
Case 1: Kiviat charts—A perfectly balanced system.
Consider a sample data given below. Its Kiviat chart is also shown below:

Software-Testing_Final.indb 107 31-01-2018 [Link]

 108 • Software Testing

FIGURE 3.24

Case 2: Kiviat charts—A well-balanced system.

Consider another sample data for a well-balanced system. Using the steps
listed earlier, we draw its kiviat chart.

FIGURE 3.25

Software-Testing_Final.indb 108 31-01-2018 [Link]

 Black-Box (or Functional) Testing Techniques • 109

Case 3: Kiviat charts—A poorly balanced system.

Now consider a poorly balanced system. Its kiviat chart is shown below:

FIGURE 3.26

Case 4: Kiviat charts—A perfectly balanced system.

Consider another set of sample data. We draw its kiviat chart.

FIGURE 3.27

Software-Testing_Final.indb 109 31-01-2018 [Link]

 110 • Software Testing

Kiviat Charts—A Different Example (CASE STUDY)

Problem: Comparing Internet diffusion in countries.
Measures: pervasiveness
geographic dispersion
sectoral absorption
connectivity infrastructure
organizational infrastructure
sophistication of use
Countries compared:
Finland
Jordan
Israel

Internet Dimensions Compared

Dimensions Jordan Israel Finland

Pervasiveness 3 4 4
Geographic dispersion 1 4 3
Sectoral absorption 2 3 3
Connectivity infrastructure 1 2 3
Organizational infrastructure 2 4 4
Sophistication of use 1 3 4

Internet Dimensions Compared in the Form of a Kiviat Chart

FIGURE 3.28

Software-Testing_Final.indb 110 31-01-2018 [Link]

 Black-Box (or Functional) Testing Techniques • 111

Problem with Kiviat Charts

The Cost/Utilization Method
„„ The problem with Kiviat charts - how to develop a metric for multi-

dimensional data?
„„ The solution: express everything in terms of a common measure -

cost.
„„ There are then two dimensions - utilization and cost - which when

multiplied yield a cost/ utilization factor for each system component.

Illustrative Cost/Utilization Histograms

FIGURE 3.29 FIGURE 3.30

FIGURE 3.31 FIGURE 3.32

Software-Testing_Final.indb 111 31-01-2018 [Link]

 112 • Software Testing

FIGURE 3.33 FIGURE 3.34

Cost/Utilization—The Method
The following steps are shown below:
1. Choose factors to be measured.
2. Determine the cost of each factor as a percent of total system cost.
3. Determine the utilization of each factor.
4. Prepare a chart showing the cost and utilization of each factor.
5. Compute the measure of cost/utilization, F.
6. Compute the measure of balance, B.
7. Evaluate the resulting chart and measures.

Cost/Utilization—The Measures
Cost/Utilization: F = ∑uipl
i
where: ui = percent utilization of factor i
pi = cost contribution of factor i
Balance: B = 1− 2 √ ∑(F − ui)2 × pi

Software-Testing_Final.indb 112 31-01-2018 [Link]

 Black-Box (or Functional) Testing Techniques • 113

FIGURE 3.35  Cost/Utilization— FIGURE 3.36  Cost/Utilization—

The Measures. The Measures.

FIGURE 3.37  Cost/Utilization— FIGURE 3.38  Cost/Utilization—

Interpretation. Interpretation.

FIGURE 3.39  Cost/Utilization—Interpretation.

Software-Testing_Final.indb 113 31-01-2018 [Link]

 114 • Software Testing

FIGURE 3.40  Trace of Cost/Utilization Criteria.

FIGURE 3.41  Composite Cost/Utilization Histogram for Two Real Linked

Systems.

Software-Testing_Final.indb 114 31-01-2018 [Link]

 Black-Box (or Functional) Testing Techniques • 115

Conclusions
It is essential to maintain balance between system components in order to:
„„ reduce costs.

„„ maintain smooth functioning with no bottlenecks.

„„ attain effectiveness AND efficiency.

SUMMARY
We summarize the scenarios under which each of these techniques will be
useful:

When we want to test scenarios The most effective black-

that have box testing technique to use
1. Output values dictated by certain conditions Decision tables
depending on values of input variables.
2. Input values in ranges, with each range Boundary Value Analysis
­showing a particular functionality. (BVA)
3. Input values divided into classes. Equivalence partitioning
4. Checking for expected and unexpected input Positive and negative testing
values.
5. Workflows, process flows, or language Graph based testing
­processors.
6. To ensure that requirements are tested and Requirements based testing
met properly.
7. To test the domain expertise rather than Domain testing
­product specification.
8. To ensure that the documentation is consistent Documentation testing
with the product.

MULTIPLE CHOICE QUESTIONS

1. Which is not a functional testing technique?
a. BVA b. Decision table
c. Regression testing d. None of the above.

Software-Testing_Final.indb 115 31-01-2018 [Link]

 116 • Software Testing

2. One weakness of BVA and equivalence partitioning is

a. They are not effective.
b. They do not explore combinations of input circumstances.
c. They explore combinations of input circumstances.
d. None of the above.
3. Decision tables are useful in a situation where
a. An action is taken under varying sets of corditions.
b. A number of combinations of actions are taken under varying sets of
conditions.
c. No action is taken under varying sets of conditions.
d. None of the above.
4. “Causes” and “Effects” are related to
a. Input and output b. Output and input
c. Destination and source d. None of the above.
5. Functionality of a software is tested by
a. White-box testing. b. Black-box testing.
c. Regression testing. d. None of the above.
6. If n represents the number of variables in a program then BVA yields
how many test cases?
a. 4n + 2 b. 4n + 1
c. n + 2 d. n + 1
7. For a function of n variables, the robustness testing will yield
a. 6n + 1 test cases. b. 6n + 2 test cases.
c. 6n + 4 test cases. d. None of the above.
8. Worst case testing yields
a. 5n test cases. b. 5n + 1.
c. 5n. d. None of the above.
9. BVA is based upon
a. Single fault assumption theory. c. Both of the above.
b. Multiple fault assumption theory. d. None of the above.

Software-Testing_Final.indb 116 31-01-2018 [Link]

 Black-Box (or Functional) Testing Techniques • 117

10. In decision tables, which of the following is true?

a. Number of test cases is equal to number of rules (or columns)
b. Number of test cases is not equal to number of rules (or columns)
c. Both (a) and (b)
d. None of the above.

ANSWERS
1. c. 2. b. 3. b. 4. a.
5. b. 6. b. 7. a. 8. c.
9. a. 10. a.

CONCEPTUAL SHORT QUESTIONS WITH ANSWERS

Q. 1. Why we need to perform both types of testing?
Ans. A functional (Black-box) test case might be taken from the
­documentation description of how to perform a certain function.
For example, accepting the bar code input.
On the other hand, a structural test case might be taken from a tech-
nical documentation manual.
Both methods together validate the entire system and is shown in
Table.

Test Phase Performed by Verification Validation

Requirements Developers, Users ×
Review
Unit Testing Developers ×
Integrated Testing Developers ×
System Testing Developers, Users ×

Software-Testing_Final.indb 117 31-01-2018 [Link]

 118 • Software Testing

Q. 2. What is the source of knowledge for functional testing?

Ans. The following items are the knowledge source of functional (or
black-box) testing:
a. Requirements document
b. Specifications
c. Domain knowledge
d. Defect analysis data
Q. 3. What is special value testing?
Ans. It is a sort of functional testing technique. It is most intuitive and
least uniform. Special value testing occurs when a tester uses his or
her domain knowledge, experience with similar programs, and infor-
mation about soft-spots (i.e., critical areas) to device test cases. It is
also known as adhoc testing. It is dependent on the tester’s ability.
Even though special value/adhoc testing is highly subjective, it often
results in a set of test cases that is more effective in revealing faults
than the test cases generated by other methods.
Q. 4. What is random testing?
Ans. A testing technique in which instead of always choosing the min,
min+, nom, max–, and max values of a bounded variable, we use a
random number generator to pick test case values. This will reduce
bias during testing.
Q. 5. Write down the differences between static and dynamic testing?
Ans. The differences between static and dynamic testing are shown
below:

Static testing Dynamic testing

1. It talks about preven- It talks about cure.
tion.
2. It is more cost It is less cost effective.
­effective.
3. It may achieve 100% It may achieve less than 50%
statement coverage. ­statement coverage as it finds the
errors only in the part of codes that
are actually executed.
4. It is not time It is time consuming as it may
­consuming. involve running several test cases.
5. It can be done before It can be done only after executables
compilation. are ready.

Software-Testing_Final.indb 118 31-01-2018 [Link]

 Black-Box (or Functional) Testing Techniques • 119

Q. 6. Give some advantages and disadvantages of functional test cases?

Ans. Functional test cases have two main advantages:
i. They are independent of how the software is implemented. So,
even if the implementation changes, the test cases are still useful.
ii. Test case development can occur in parallel with the
implementation, thereby reducing overall project development
internal.
Functional test cases have two main disadvantages:
i. Some unavoidable redundancies may exist among test cases.
ii. There exists a possibility of gaps of untesting software.
Both of these problems can be solved if we combine the test cases,
so obtained from both functional and structural testing.
Q. 7. Differentiate between positive and negative testing?
Ans. Let us tabulate the differences between the two:

Positive Testing Negative Testing

1. Positive testing tries to prove that Negative testing is done to show that the
a given product does what it is product does not fail when an unex-
supposed to do. pected input is given.
2. A positive test case is one that A negative test case will have the
verifies the requirements of the input values that may not have been
product with a set of expected ­represented in the SRS. These are
output. unknown conditions for the product.
3. The purpose of positive testing is The purpose of negative testing is to try
to prove that the product works as and break the system.
per the user specifications.
4. Positive testing checks the Negative testing covers scenarios for
­product’s behavior. which the product is not designed and
coded.
5. Positive testing is done to verify Negative testing is done to break the
the known test conditions. product with unknown test conditions,
i.e., test conditions that lie outside SRS.
(Continued)

Software-Testing_Final.indb 119 31-01-2018 [Link]

 120 • Software Testing

6. If all documented requirements There is no end to negative testing and

and test conditions are covered 100% coverage is impractical here.
then it gives 100% coverage.
For example: A product For example: A product not ­delivering
­delivering an error when it is an error when it should or delivering an
expected to give error. error when it should not.

Q. 8. What is domain testing?

Ans. Domain testing is a sort of testing strategy wherein we do not look
even at SRS of a software product but test purely on the basis of
domain knowledge and expertise in the domain of application.
For example, in a banking software, knowing the account opening,
closing, etc. processes, enables a tester to test that functionality
­better.
Q. 9. What is documentation testing?
Ans. A testing done to ensure that the documentation is consistent with
the product.
Q. 10. What are the characteristics of a good test?
Ans. i. A good test has a high probability of finding an error.
ii. A good test is not redundant.
iii. A good test is the “best of breed.”
iv. A good test should be neither too simple nor too complex.

Q. 11. Consider the above use case diagram for coffee maker. Find at least
ten acceptance test cases and black-box test cases and document it.
Ans. Test cases for coffee maker.
Preconditions: Run coffee maker by switching on power supply.

Software-Testing_Final.indb 120 31-01-2018 [Link]

 Test Test case Test case Test steps Actual Test status

Software-Testing_Final.indb 121
case id name description Step Expected result result (P/F)
Acc01 Waiting When the coffee System displays menu as
state maker is not in use, follows:
it waits for user 1. Add recipe
input. 2. Delete recipe
3. Edit a recipe
4. Add inventory
5. Check inventory
6. Purchase beverage
Acc02 Add a Only three recipes Add the recipe. Each recipe name must
recipe may be added to the A recipe consists be unique in the recipe
coffee maker. of a name, price, list.
units of coffee,
units of dairy
creamer, units of
chocolate, water.
Acc03 Delete a A recipe may be Choose the rec- A status message is
recipe deleted from the ipe to be deleted printed and the coffee
coffeemaker if it by its name. maker is returned to the
exists in the list of waiting state.
recipes in the coffee
maker.

31-01-2018 [Link]
 Test Test case Test case Test steps Actual Test status
case id name description Step Expected result result (P/F)

Software-Testing_Final.indb 122
Acc04 Edit a The user will be Enter the Upon completion, a
recipe prompted for the recipe name status message is printed
recipe name they along with vari- and the coffee maker is
wish to edit. ous units. returned to the waiting
state.
Acc05 Add Inventory may Type the inven- Inventory is added to
inventory be added to the tory: coffee, the machine and a status
machine at any dairy creamer, message is printed.
time. (Inventory is water.
measured in integer
units.)
Acc06 Check Inventory may be Enter the units System displays the
inventory checked at any time. of each item. inventory.
Acc07 Purchase The user will not be Enter the units 1. System dispensed the
beverage able to purchase a and amount. change, if user paid
beverage if they do more than the price of
not deposit enough the beverage.
money. 2. System returns user’s
money if there is not
enough inventory.

31-01-2018 [Link]
 Test ID Description/steps Expected results Actual results Test status (P/F)

Software-Testing_Final.indb 123
checkOptions Precondition: Run CoffeeMaker
Enter: 0 Program exits
Enter: 1 Add recipe functionality
Enter: 2 Delete recipe functionality
Enter: 3 Edit recipe functionality
Enter: 4 Add inventory
­functionality
Enter: 5 Inventory displays
Enter: 6 Make coffee functionality
addRecipe1 Precondition: Run CoffeeMaker Coffee successfully added.
Enter: 1 Return to main menu.
Name: Coffee
Price: 10
Coffee: 3
Dairy creamer: 1
Chocolate: 0
addRecipe2 Precondition: Run CoffeeMaker Coffee could not be
Enter: 1 added. (Recipe name
Name: Coffee must be unique in the
Price:10 recipe list.)
Coffee: 3 Return to main menu.
Dairy creamer: 1
Chocolate: 0
(Continued)

31-01-2018 [Link]
 Test ID Description/steps Expected results Actual results Test status (P/F)
addRecipe3 Precondition: Run CoffeeMaker Mocha could not be
Enter: 1 added. Price can not be

Software-Testing_Final.indb 124
Name: Mocha negative. Return to main
Price: –50 menu.
addRecipe4 Precondition: Run CoffeeMaker Mocha could not be
Enter: 1 added. Units of cof-
Name: Mocha fee can not be negative.
Price: 60 Return to main menu.
Coffee: –3
addRecipe5 Precondition: Run CoffeeMaker Mocha could not be
Enter: 1 added. Units of dairy
Name: Mocha creamer can not be nega-
Price: 20 tive.
Coffee: –3 Return to main menu.
Dairy creamer: –2
addRecipe6 Precondition: Run CoffeeMaker Mocha could not be
Enter: 1 added. Units of choco-
Name: Mocha late can not be negative.
Price: 20 Return to main menu.
Coffee: 3
Dairy creamer: 2
Chocolate: –3
addRecipe7 Precondition: Run CoffeeMaker Please input an integer.
Enter: 1 Return to main menu.
Name: Mocha
Price: a

31-01-2018 [Link]
 Test ID Description/steps Expected results Actual results Test status (P/F)

Software-Testing_Final.indb 125
addRecipe8 Precondition: Run CoffeeMaker Please input an integer.
Enter: 1 Return to main menu.
Name: Mocha
Price: 20
Coffee: a
addRecipe9 Precondition: Run CoffeeMaker Please input an integer.
Enter: 1 Return to main menu.
Name: Mocha
Price: 20
Coffee: 3
Dairy creamer: 2
Chocolate: a
Precondition: Run CoffeeMaker Coffee successfully added.
addRecipe10 Enter: 1 Return to main menu.
Name: Hot chocolate
Price: 20
Coffee: 3
Dairy creamer: 2
Chocolate: 3
deleteRecipe1 Precondition: addRecipe1 has run Successfully deleted.
successfully Return to main menu.
Enter: 2
Enter: 3

(Continued)

31-01-2018 [Link]
 Test ID Description/steps Expected results Actual results Test status (P/F)
deleteRecipe2 Precondition: Run CoffeeMaker There are no recipes to
Enter: 2 delete.

Software-Testing_Final.indb 126
Return to main menu.

editRecipe1 Precondition: addRecipe1 has run Coffee successfully edited.

successfully Return to main menu.
Enter: 3
Name: Coffee
Price: 10
Coffee: 3
Dairy creamer: 1
Chocolate: 0
editRecipe2 Precondition: addRecipe1 has run Coffee successfully edited.
successfully Return to main menu.
Enter: 3
Name: Coffee
Price: 10
Coffee: 5
Dairy creamer: 4
Chocolate: 0
editRecipe3 Precondition: addRecipe1 has run Mocha could not be added.
successfully Price can not be negative.
Enter: 3
Name: Mocha
Price: 20

REVIEW QUESTIONS

31-01-2018 [Link]
 Black-Box (or Functional) Testing Techniques • 127

REVIEW QUESTIONS
1. Perform the following:
a. Write a program to find the largest number.
b. Design a test case for program at 2(a) using a decision table.
c. Design equivalence class test cases.
2. Explain any five symbols used in the cause-effect graphing technique?
3. How do you measure:
a. Test effectiveness?
b. Test efficiency?
4. Write a short paragraph:
a. Equivalence testing.
5. Explain the significance of boundary value analysis. What is the purpose
of worst case testing?
6. Describe cause-effect graphing technique with the help of an example.
7. a.  Discuss different types of equivalence class tests cases.
b. Consider a program to classify a triangle. Its input is a triple of the
integers (day x, y, z) and date types or input parameters ensure that
they will be integers greater than zero and less than or equal to 200.
The program output may be any of the following words: scalene,
isosceles, equilateral, right angle triangle, not a triangle. Design the
equivalence class test cases.
8. How can we measure and evaluate test effectiveness? Explain with the
help of 11 step S/W testing process.
9. What is the difference between:
Equivalence partitioning and boundary value analysis methods?
10. Consider the previous date function and design test cases using the
following techniques:
a. Boundary value analysis.
b. Equivalence class partitioning.
The function takes current date as an input and returns the previous
date of the day as the output.

Software-Testing_Final.indb 127 31-01-2018 [Link]

 128 • Software Testing

All variables have integer values subject to conditions as follows:

C1: 1 ≤ month ≤ 2
C2: 1 ≤ day ≤ 31
C3: 1920 ≤ year ≤ 2000.
11. Compare the single/multiple fault assumption theory with boundary
value and equivalence class testing.
12. Explain the cause-effect graphing technique in detail. Why is it different
from other functional testing techniques?
13. Differentiate between positive and negative functional testing.
14. What is domain testing? Illustrate through a suitable example.
15. a. “Software testing is an incremental process.” Justify the statement.
b. What do we mean by functional analysis? Discuss in brief any method
to perform functional analysis.
c. Consider a program to determine whether a number is “odd” or
“even” and print the message “Number is Even” or “Number is Odd.”
The number may be any valid integer. Design boundary value and
equivalence class test cases.
16 a. Consider a program that calculates the roots of a quadratic equation
with a, b, and c in range [1, 100]. Design test cases using boundary
value and robustness testing.
b. What is decision table testing?
17. Write a short paragraph on cause-effect graphing.
18. Discuss any two model-based black-box testing approaches?
19. A program computes the grade of each student based on the average
of marks obtained by them in physics, chemistry, and math. Maximum
marks in each subject is 60. Grades are awarded as follows:

Marks Range Grade

60 A+
59–60 A
49–40 B
39–30 C
29–30 D
19–0 F

Software-Testing_Final.indb 128 31-01-2018 [Link]

 Black-Box (or Functional) Testing Techniques • 129

Design robust test cases and identify equivalence class test cases for
output and input domains for this problem.
20. What is the difference between weak normal and strong normal
equivalence class testing?
21. Consider a program for the determination of previous date. Its input is a
triple of day, month, and year with the values in the range:
1 ≤ month ≤ 12
1 ≤ day ≤ 31
1900 ≤ year ≤ 2025
The possible outputs are “Previous date” and “Invalid date.” Design a
decision table and equivalence classes for input domain.
22. Consider a program given below for the selection of the largest of
numbers.
main ( )
{
float A, B, C;
printf (“Enter 3 values:”);
scanf (“%d%d%d”, &A, &B, &C);
printf (“Largest value is”);
if (A > B)
{
if (A > C)
printf (“%d\n”, A);
else
printf (“%d\n”, C);
}
else
{
if (C > B)
printf (“%d”, C);
else
printf (“%f”, B);
}
}

Software-Testing_Final.indb 129 31-01-2018 [Link]

 130 • Software Testing

a. D
 esign the set of test cases using BVA technique and equivalence
class testing technique.
b. Select a set of test cases that will provide 100% statement coverage.
c. Develop a decision table for this program.
23. Consider the above program and show that why is it practically impossible
to do exhaustive testing?
24. a. Consider the following point-based evaluation system for a trainee
salesman of an organization:

Points earned Management action

0–20 Thank you
21–40 Extend probation
41–60 Confirmation
61–80 Promotion
81–100 Promotion with a letter of recommendation

Generate the test cases using equivalence class testing.

b. Explain functional testing.
25. Design test cases using the functional testing taking any example
program? Explain how and why complete testing is not possible by
highlighting some cases of the example given.
26. Let us consider an example of grading the students in an academic
institution. The grading is done according to the following rules:

Marks obtained Grade

80–100 Distinction
60–79 First division
50–59 Second division
40–49 Third division
0–39 Fail

Generate test cases using the equivalence class testing technique.

Software-Testing_Final.indb 130 31-01-2018 [Link]

 Black-Box (or Functional) Testing Techniques • 131

27. Consider the following point-based evaluation system for a salesman of

an organization.

Points earned Grade Management action

80–100 A+ Raise of $10,000
75–80 A– Raise of $5,000
70–75 A Raise of $3000
65–70 B+ Raise of $1000
60–65 B No increment
50–60 C Warning

Generate the test cases using equivalence class testing.

28. Consider a program for the determination of previous date. Its input is a
triple of day, year, month with values in range:
1 ≤ month ≤ 12
1 ≤ day ≤ 31
1912 ≤ year ≤ 2020
Design a decision table for the given scenario.
29. Consider the program for the determination of next date in a calender.
Its input is a triple of day, month, and year with the following range:
1 ≤ month ≤ 12
1 ≤ day ≤ 31
1900 ≤ year ≤ 2025
The possible outputs would be next date or invalid date. Design boundary
value, robust, and worst test cases for this program.
30. a. Explain cause-effect graphing technique in detail. Why is it different
from other functional techniques?
b. Design test cases using functional testing taking any example program.
Explain how and why complete testing is not possible by highlighting
some cases of the example given.

Software-Testing_Final.indb 131 31-01-2018 [Link]

 132 • Software Testing

31. Consider an example of grading a student in a university. The grading is

done as below:

Average marks Grade

90–100 A+
75–89 A
60–74 B
50–59 C
Below 50 Fail

The marks of any three subjects are considered for the calculation of
average marks. Scholarships of $1000 and $500 are given to students
securing more than 90% and 85% marks, respectively. Develop a
decision table, cause effect graph, and generate test cases for the above
scenario.

Software-Testing_Final.indb 132 31-01-2018 [Link]

 chaPter

4
White-Box (or Structural)
Testing Techniques
Inside this Chapter:
4.0. Introduction to White-Box Testing or Structural Testing or
Clear-Box or Glass-Box or Open-Box Testing
4.1. Static Versus Dynamic White-Box Testing
4.2. Dynamic White-Box Testing Techniques
4.3. Mutation Testing Versus Error Seeding—Differences in Tabular
Form
4.4. Comparison of Black-Box and White-Box Testing in Tabular
Form
4.5. Practical Challenges in White-Box Testing
4.6. Comparison on Various White-Box Testing Techniques
4.7. Advantages of White-Box Testing

4.0. INTRODUCTION TO WHITE-BOX TESTING

OR STRUCTURAL TESTING OR CLEAR-BOX OR ­
GLASS-BOX OR OPEN-BOX TESTING
White-box testing is a way of testing the external functionality of the code
by examining and testing the program code that realizes the external func-
tionality. It is a methodology to design the test cases that uses the control

Software-Testing_Final.indb 133 31-01-2018 [Link]

 134 • Software Testing

structure of the application to design test cases. White-box testing is used to

test the program code, code structure, and the internal design flow.

4.1.  STATIC VERSUS DYNAMIC WHITE-BOX TESTING

A number of defects get amplified because of incorrect translation of
requirements and design into program code. We shall now study different
techniques of white-box testing. As shown in Figure 4.1, white-box texting is
classified as static and dynamic.

FIGURE 4.1  Classification of White-Box Testing.

As already discussed in Chapter 2, static testing is a type of testing in

which the program source code is tested without running it (not the binaries
or executable). We only need to examine and review the code. It means that
we need not execute the code. We need to find out whether
„„ The code works according to the functional requirements.
„„ The code has been written in accordance with the design developed ear-
lier in the project life cycle.
„„ The code for any functionality has been missed.
„„ The code handles errors properly.

Software-Testing_Final.indb 134 31-01-2018 [Link]

 White-Box (or Structural) Testing Techniques • 135

Static testing can be done by humans or with the help of specialized

tools. So, static white-box testing is the process of carefully and methodically
reviewing the software design, architecture, or code for bugs without execut-
ing it. It is sometimes referred to as structural analysis [PATT01].
We next discuss some of the dynamic white-box testing techniques one
by one.

4.2.  DYNAMIC WHITE-BOX TESTING TECHNIQUES

In dynamic testing, we test a running program. So, now binaries and
­executables are desired. We try to test the internal logic of the program now.
It entails running the actual product against some pre-designed test cases to
exercise as much of the code as possible.

4.2.1. Unit/Code Functional Testing

It is the process of testing in which the developer performs some quick
checks prior to subjecting the code to more extensive code coverage testing
or code complexity testing. It can be performed in many ways:
1. At the initial stages, the developer or tester can perform certain tests
based on the input variables and the corresponding expected output
variables. This can be a quick test. If we repeat these tests for multiple
values of input variables then the confidence level of the developer to go
to the next level increases.
2. For complex modules, the tester can insert some print statements
in between to check whether the program control passes through all
statements and loops. It is important to remove the intermediate print
statements after the defects are fixed.
3. Another method is to run the product under a debugger or an integrated
development environment (IDE). These tools involve single stepping of
instructions and setting break points at any function or instruction.
All of these initial tests actually fall under “debugging” category rather
than under “testing” category of activities. We put them under “white-box
testing” head as all are related to the knowledge of code structure.

Software-Testing_Final.indb 135 31-01-2018 [Link]

 136 • Software Testing

4.2.2.  Code Coverage Testing

Code coverage testing involves designing and executing test cases and find-
ing out the percentage of code that is covered by testing. The percentage of
code covered by a test is found by adopting a technique called the instru-
mentation of code. These tools rebuild the code, do product linking with a
set of libraries provided by the tool, monitor the portions of code covered,
and report on the portions of the code that are covered frequently, so that
the critical or most used portions of code can be identified. We will next dis-
cuss some basic testing techniques based on code coverage.

[Link]. Statement Coverage
In most of the programming languages, the program construct may be a
sequential control flow, a two-way decision statement like if-then-else, a
multi-way decision statement like switch, or even loops like while, do, repeat
until and for.
Statement coverage refers to writing test cases that execute each of the
program statements. We assume that the more the code is covered, the bet-
ter the testing of the functionality.
For a set of sequential statements (i.e., with no conditional branches),
test cases can be designed to run through from top to bottom. However, this
may not always be true in two cases:
1. If there are asynchronous exceptions in the code, like divide by zero,
then even if we start a test case at the beginning of a section, the test case
may not cover all the statements in that section. Thus, even in the case of
sequential statements, coverage for all statements may not be achieved.
2. A section of code may be entered from multiple points.
In case of an if-then-else statement, if we want to cover all the state-
ments then we should also cover the “then” and “else” parts of the if state-
ment. This means that we should have, for each if-then-else, at least one test
case to test the “then” part and at least one test case to test the “else” part.
The multi-way, switch statement can be reduced to multiple two-way if
statements. Thus, to cover all possible switch cases, there would be multiple
test cases.

Software-Testing_Final.indb 136 31-01-2018 [Link]

 White-Box (or Structural) Testing Techniques • 137

A good percentage of the defects in programs come about because of

loops that do not function properly. More often, loops fail in what are called
“boundary conditions.” So, we must have test cases that
1. Skip the loop completely so that the situation of the termination condition
being true before starting the loop is tested.
2. Check the loop for n = 1.
3. Try covering the loop at the boundary values of n, i.e., just below n and
just above n.
The statement coverage for a program, which is an indication of the per-
centage of statements actually executed in a set of tests, can be calculated
as follows:

f g
Total Statements Exercised
Statement Coverage =

Total Number of Executable Statements in Program
× 100

For example, if the total number of statements exercised = 08

Total number of executable statements in program = 10

∴ Statement coverage = 08 × 100 = 80%

10
Please note from the above discussion that as the type of statement pro-
gresses from a simple sequential statement to if-then-else and through loops,
the number of test cases required to achieve statement coverage increases. As
we have already seen, exhaustive testing (100%) is not possible, so exhaus-
tive coverage of all statements in a program will be impossible for practical
purposes.
Even if we get a high level of the statement coverage, it does not mean
that the program is defect free.
Consider a program that implements wrong requirements and, if such
a code is fully tested with say, 100% coverage, it is still a wrong program.
Hence 100% code coverage does not mean anything. Consider another
example.

Software-Testing_Final.indb 137 31-01-2018 [Link]

 138 • Software Testing

i = 0 ;
if (code = = “y”)
{
statement –1 ;
statement–2 ;
:
:
statement – n ;
}
else
result = {marks/ i} * 100 ;

In this program, when we test with code = “y,” we will get 80% code
coverage. But if the data distribution in the real world is such that 90% of the
time the value of code is not = “y,” then the program will fail 90% of the time
because of the exception-divide by zero. Thus, even with a code coverage of
80%, we are left with a defect that hits the users 90% of the time. The path
coverage technique, discussed next, overcomes this problem.

[Link].  Path Coverage

In the path coverage technique, we split a program into a number of distinct
paths. A program or a part of a program can start from the beginning and
take any of the paths to its completion. The path coverage of a program may
be calculated based on the following formula:

Path Coverage =
f Total Path Exercised
Total Number of paths in Program g × 100

For example, if the total path exercised = 07

Total number of paths in p

­ rogram = 10

∴ Path coverage = 07 × 100 = 70%

10
Consider the following flow graph in Figure 4.2.

There are different paths in this particular flow graph.

Software-Testing_Final.indb 138 31-01-2018 [Link]

 White-Box (or Structural) Testing Techniques • 139

FIGURE 4.2  Flow Graph Example.

Some of the paths are:

Path – 1: 1→2→6
Path – 2: 1→3→5→6
Path – 3: 1→3→4→5→6
Path – 4: 1→3→4→2→6
Regardless of the number of statements in each of these paths, if we can
­execute these paths, then we would have covered most of the typical sce-
narios.
Path coverage provides a stronger condition of coverage than statement
coverage as it relates to the various logical paths in the program rather than
just program statements.

[Link].  Condition Coverage

In the above example, even if we have covered all the paths possible, it does
not mean that the program is fully tested. Path testing is not sufficient as it
does not exercise each part of the Boolean expressions, relational expres-
sions, and so on. This technique of condition coverage or predicate monitors
whether every operand in a complex logical expression has taken on every
TRUE/FALSE value. Obviously, this will mean more test cases and the num-
ber of test cases will rise exponentially with the number of conditions and
Boolean expressions. For example, in if-then-else, there are 22 or 4 p­ ossible

Software-Testing_Final.indb 139 31-01-2018 [Link]

 140 • Software Testing

true/false conditions. The condition coverage which indicates the percent-

age of conditions covered by a set of test cases is defined by the formula:

Condition Coverage =
f Total Decisions Exercised
g
Total Number of Decisions in Program
× 100

Please note that this technique of condition coverage is much stronger

criteria than path coverage, which in turn is much stronger criteria than
statement coverage.

[Link]. Function Coverage
In this white-box testing technique, we try to identify how many program
functions are covered by test cases. So, while providing function coverage,
test cases can be written so as to exercise each of different functions in the
code.
The following are the advantages of this technique:
1. Functions (like functions in C) are easier to identify in a program and,
hence, it is easier to write test cases to provide function coverage.
2. Because functions are at a higher level of abstraction than code, it is
easier to achieve 100% function coverage.
3. It is easier to prioritize the functions for testing.
4. Function coverage provides a way of testing traceability, that is, tracing
requirements through design, coding, and testing phases.
5. Function coverage provides a natural transition to black-box testing.
Function coverage can help in improving the performance as well as
the quality of the product. For example, if in a networking software, we find
that the function that assembles and disassembles the data packets is being
used most often, it is appropriate to spend extra effort in improving the qual-
ity and performance of that function. Thus, function coverage can help in
improving the performance as well as the quality of the product.
Better code coverage is the result of better code flow understanding and
writing effective test cases. Code coverage up to 40–50% is usually achiev-
able. Code coverage of more than 80% requires an enormous amount of
effort and understanding of the code.
The multiple code coverage techniques discussed so far are not mutu-
ally exclusive. They supplement and augment one another. While statement
coverage can provide a basic comfort factor, path, decision, and function
coverage provide more confidence by exercising various logical paths and
functions.

Software-Testing_Final.indb 140 31-01-2018 [Link]

 White-Box (or Structural) Testing Techniques • 141

4.2.3.  Code Complexity Testing

Two questions that come to mind are:
1. Which of the paths are independent? If two paths are not independent,
then we may be able to minimize the number of tests.
2. Is there any limit to the number of tests that must be run to ensure that
all the statements have been executed at least once?
The answer to the above questions is a metric that quantifies the com-
plexity of a program and is known as cyclomatic complexity. It is also known
as structural complexity because it gives the internal view of the code. It is
a number which provides us with an upper bound for the number of tests
that must be conducted to ensure that all statements have been executed at
least once.

[Link]. Cyclomatic Complexity, Its Properties and Its Meaning

in Tabular Form

McCabe IQ covers about 146 different counts and measures. These met-
rices are grouped according to six main “collections” each of which provides
a different level of granularity and information about the code being ana-
lyzed. The collections are given below:
i. McCabe metrics based on cyclomatic complexity, V(G).
ii. Execution coverage metrics based on any of branch, path, or Boolean
coverage.
iii. Code grammar metrics based around line counts and code structure
counts such as nesting.
iv. OO metrics based on the work of Chidamber and Kemerer.
v. Derived metrics based on abstract concepts such as understability,
maintainability, comprehension, and testability.
vi. Custom metrics imported from third-party software/systems, e.g.,
defect count.
McCabe IQ provides for about 100 individual metrics at the method,
procedure, function, control, and section/paragraph level. Also, there are 40
metrices at the class/file and program level.

Software-Testing_Final.indb 141 31-01-2018 [Link]

 142 • Software Testing

Categories of Metrics
There are three categories of metrics:
1. McCabe metrics
2. OO metrics
3. Grammar metrics
Please remember that when collecting metrics, we rely upon subordinates
who need to “buy into” the metrics program. Hence, it is important to only
collect what you intend to use.
We should keep in mind, the Hawthorne Effect which states that when
you collect metrics on people, the people being measured will change their
behavior. Either of these practices will destroy the efficiency of any metrics
program.
The three metrics categories are explained below.

I.  McCabe metrics

a. Cyclomatic complexity, V(G): It is the measure of the amount of

logic in a code module of 3rd and 4th generation languages. If V(G) is
excessively high then it leads to impenetrable code, i.e., a code that is
at a higher risk due to difficulty in testing. The threshold value is 10.
When V(G) > 10, then the likelihood of code being unreliable is much
higher. Please remember that a high V(G) shows a decreased quality
in the code resulting in higher defects that become costly to fix.
b. Essential complexity: It is a measure of the degree to which a code
module contains unstructured constructs. If the essential complexity
is excessively high, it leads to impenetrable code, i.e., a code that is
at higher risk due to difficulty in testing. Furthermore, the higher
value will lead to increased cost due to the need to refactor, or worse,
reengineer the code. The threshold value is 4. When the essential
complexity is more than 4 then the likelihood of the code being
unmaintainable is much higher. Please note that a high essential
complexity indicates increased maintenance costs with decreased
code quality. Some organizations have used the essential density
metric (EDM) defined as:

Essential Complexity
EDM =
Cyclomatic Complexity

Software-Testing_Final.indb 142 31-01-2018 [Link]

 White-Box (or Structural) Testing Techniques • 143

c. Integration complexity: It is a measure of the interaction between

the modules of code within a program. Say, S0 and S1 are two
derivatives of this complexity.
where S0  — Provides an overall measure of size and complexity
of a program’s design. It will not reflect the inter-
nal calculations of each module. It is the sum of all
the integration complexity in a program (∑iv(g)).
and S1  =  (S0 – Number of methods + 1).
This is primarily used to determine the number of tests for the
“some test” that is designed to ensure that the application would exe-
cute without issues in module interaction.
d. Cyclomatic density (CD): It is a measure of the amount of logic in
the code. It is expressed as follows:

CD = Decisions Made
Lines of Executable Code

By eliminating the size factor, this metric reduces complexity

strictly to modules that have unusually dense decision logic. Please
note that the higher the CD value, the denser the logic.
The CD metric should be in the range of 0.14 to 0.42 for the code to
be simple and comprehensible.
e. Pathological complexity: It represents an extremely unstructured
code which shows a poor design and hence a suggestion for code’s
reengineering. A value greater than one indicates poor coding
practices like branching into a loop or into a decision. Please note
that these conditions are not easy to replicate with modern post 3GL
languages.
f. Branch coverage: It is a measure of how many branches or decisions
in a module have been executed during testing.
If the branch coverage is <95% for new code or 75% for code under
maintenance then the test scripts require review and enhancement.
g. Basis path coverage: A measure of how many of the basis (cyclomatic,
V(G)) paths in a module have been executed. Path coverage is the
most fundamental of McCabe design. It indicates how much logic
in the code is covered or not covered. This technique requires more
thorough testing than branch coverage.

Software-Testing_Final.indb 143 31-01-2018 [Link]

 144 • Software Testing

If the path coverage is < 90% for new code or 70% for code under
maintenance then the test scripts require review and enhancement.
h. Boolean coverage: A technique used to establish that each condition
within a decision is shown by execution to independently and correctly
affect the outcome of the decision.
The major application of this technique is in safety critical sys-
tems and projects.
i. Combining McCabe metrics: Cyclomatic complexity is the basic
indicator for determining the complexity of logic in a unit of code. It
can be combined with other metrics.

1. Code review candidate

If V(G) > 10 and essential complexity/essential density exceeds 4, then the
unit needs a review.

2. Code refactoring
If V(G) > 10 and the condition
V(G) – EV(g) ≤ V(g) is true
Then, the code is a candidate for refactoring.

3. Inadequate comment content

If the graph between V(G) against comment % (in terms of LOC) does not
show a linear increase then the comment content need to be reviewed.

4. Test coverage
If the graph between V(G) against path coverage does not show a linear
increase then the test scripts need to be reviewed.

II. OO Metrics

a. Average V(G) for a class: If average V(G) > 10 then this metric
indicates a high level of logic in the methods of the class which in turn
indicates a possible dilution of the original object model. If the average
is high, then the class should be reviewed for possible refactoring.
b. Average essential complexity for a class: If the average is greater
than one then it may indicate a dilution of the original object model.

Software-Testing_Final.indb 144 31-01-2018 [Link]

 White-Box (or Structural) Testing Techniques • 145

If the average is high, then the class should be reviewed for possible
refactoring.
c. Number of parents: If the number of parents for a class is greater
than one then it indicates a potentially overly complex inheritance
tree.
d. Response for class (RFC): RFC is the count of all methods within
a class plus the number of methods accessible to an object of this
class due to implementation. Please note that the larger the number
of methods that can be invoked in response to a message, the greater
the difficulty in comprehension and testing of the class. Also, note
that low values indicate greater specialization. If the RFC is high then
making changes to this class will be increasingly difficult due to the
extended impact to other classes (or methods).
e. Weighted methods for class (WMC): WMC is the count of
methods implemented in a class. It is a strong recommendation that
WMC does not exceed the value of 14. This metric is used to show
the effort required to rewrite or modify the class. The aim is to keep
this metric low.
f. Coupling between objects (CBO): It indicates the number of non-
inherited classes this class depends on. It shows the degree to which
this class can be reused.
For dynamic link libraries (DLLs) this measure is high as the soft-
ware is deployed as a complete entity.
For executables (.exe), it is low as here reuse is to be encouraged.
Please remember this point:

Strong coupling increases the difficulty in comprehending

and testing a class. The objective is to keep it less than 6.

g. Class hierarchy level: It shows the degree of inheritance used. If it

is greater than 6, the increased depth increases the testing effort. If it
is less than 2 then the value shows a poor exploitation of OO. So, one
should aim for 2 and 3 levels only in our OO-code.
h. Number of methods (n): If number of methods (n) > 40 then it
shows that the class has too much functionality. So, it can be split into
several smaller classes. Please note that ideally one should aim for no
more than 20 methods in a class.

Software-Testing_Final.indb 145 31-01-2018 [Link]

 146 • Software Testing

i. Lack of cohesion between methods (LOCM): It is a metric used

to measure the dissimilarity of methods on a class by an instance
variable or attribute.

What is to be done?
The percentages of methods in a class using an attribute are averaged
and subtracted from 100. This measure is expressed in percentage.
Two cases arise:
i. If % is low, it means simplicity and high reusability.
ii. If % is high, it means a class is a candidate for refactoring and
could be split into two or more subclasses with low cohesion.
j. Combined OO metrics: V(G) can also be used to evaluate OO
systems. It is used with OO metrics to find out the suitable candidates
for refactoring.
By refactoring, we mean making a small change to the code which
improves its design without changing its semantics.

Rules for refactoring:

i. If avg. V(G) > 10 (high) and the number of methods (n) is < 10
(low) then the class requires refactoring.
ii. If avg. V(G) is low and the lack of cohesion is high then the class is a
suitable candidate for refactoring into two or more classes.
iii. If avg. V(G) is high and CBO is high then the class is a candidate
for refactoring.
iv. If CBO is high and lack of cohesion is high then the class is a
candidate for refactoring.

III.  Grammar Metrics

a. Line count: It is a size indicator. It is used in—

i. Estimation techniques like COCOMO2.
Number of defects
ii. Measuring defects =
1000 LOC
b. Nesting levels: Nesting of IF statements, switch, and loop constructs
can indicate unnecessarily complex conditions which makes future

Software-Testing_Final.indb 146 31-01-2018 [Link]

 White-Box (or Structural) Testing Techniques • 147

modifications quite difficult. So, refactoring may be done. Typical

industry standards are 4, 2, and 2 for IF, switch, and loop constructs
respectively.
c. Counts of decision types: It is used to show single outcome (IF
and loop) and multiple outcome decision statements. When used
in conjunction with V(G), then its value can determine if a method/
procedure/control/section is over complex and, hence, a suitable
candidate for refactoring.
d. Maximum number of predicates: This measure shows overly
complex decision statements which are candidates for refactoring.
e. Comment lines: It indicates the level of comments in a unit of code.
It shows:
Comment Lines
Documentation Level (within the code) =
LOC
Comment Lines
=
V(G)
Historically, a ratio of 15–25% of comments is adequate to enable any
user to understand the code.

[Link]. Cyclomatic Complexity, Its Properties and Its Meaning

in Tabular Form

McCabe’s cyclomatic metric, V(G) of a graph G with n vertices and e edges

is given by the formula:
V(G) = e – n + 2
Given a program, we associate it with a directed graph that has unique
entry and exit nodes. Each node in the graph, G, corresponds to a block of
statements in the program where flow is sequential and the arcs correspond
to branches taken in the program. This graph is known as a flow graph.
The cyclomatic complexity, V(G) provides us two things:
1. To find the number of independent paths through the program.
2. To provide an upper bound for the number of test cases that must
be executed in order to test the program thoroughly. The complexity
measure is defined in terms of independent paths. It is defined as

Software-Testing_Final.indb 147 31-01-2018 [Link]

 148 • Software Testing

any path through the program that introduces at least one new set of
processing statements or a new condition. See the following steps:
Step 1. Construction of flow graph from the source code or flow charts.
Step 2. Identification of independent paths.
Step 3. Computation of cyclomatic complexity.
Step 4. Test cases are designed.
Using the flow graph, an independent path can be defined as a path in
the flow graph that has at least one edge that has not been traversed before
in other paths. A set of independent paths that cover all the edges is a basis
set. Once the basis set is formed, test cases should be written to execute all
the paths in the basis set.

Properties of Cyclomatic Complexity

The following are the properties of cyclomatic complexity represented as
V(G):
1. V(G) ≥ 1.
2. V(G) is the maximum number of independent paths in graph, G.
3. Inserting and deleting functional statements to G does not
affect V(G).
4. G has only one path if and only if V(G) = 1. Consider this
example Here, V(G) = 1 and that graph, G, has only one path.
5. Inserting a new row in G, increases V(G) by unity.

Meaning of V(G) in Tabular Form

For small programs cyclomatic complexity can be calculated manually, but
automated tools are essential as several thousand of lines of code are possible
in each program in a project. It will be very difficult to manually create flow
graphs for large programs. There are several tools that are available in the
market which can compute cyclomatic complexity. Note that calculating the
complexity of a module after it has been built and tested may be too late. It
may not be possible to redesign a complex module after it has been tested.
Thus, some basic complexity checks must be performed on the modules
before embarking upon the testing (or even coding) phase. Based on the
complexity number that emerges from using the tool, one can conclude what
actions need to be taken for complexity measure using the table given below:

Software-Testing_Final.indb 148 31-01-2018 [Link]

 White-Box (or Structural) Testing Techniques • 149

TABLE 4.1  What V(G) Means?

Complexity What it means?

1–10 Well-written code, testability is high, cost/effort to maintain is
low.
10–20 Moderately complex, testability is medium, cost/effort to
maintain is medium.
20–40 Very complex, testability is low, cost/effort to maintain is high.
> 40 Not testable, any amount of money/effort to maintain may not
be enough.

[Link]. Basis Path Testing with Solved Examples

Basis path testing helps a tester to compute logical complexity measure,
V(G), of the code. This value of V(G), defines the maximum number of test
cases to be designed by identifying basis set of execution paths to ensure that
all statements are executed at least once. See the following steps:
1. Construct the flow graph from the source code or flow charts.
2. Identify independent paths.
3. Calculate cyclomatic complexity, V(G).
4. Design the test cases.
A program is represented in the form of a flow graph. A flow graph
consists of nodes and edges. Using the flow graph, an independent path can
be defined as a path in the flow graph that has at least one edge that has not
been traversed before in other paths. A set of independent paths that cover
all edges is a basis set. Once that basis set is formed, test cases should be
written to execute all the paths in the basis set.

Flow Graph Testing

The control flow of a program can be represented using a graphical rep-
resentation known as a “flow graph.” The flow graph is a directed graph
in which nodes are either entire statements or fragments of a statement.
Edges represent the flow of control. If u and v are nodes in the program
graph, there is an edge from node u to node v if the statement (fragment)

Software-Testing_Final.indb 149 31-01-2018 [Link]

 150 • Software Testing

corresponding to node v can be executed immediately after the statement

(fragment) corresponding to node u,

i.e., .
We next show the basic notations that are used to draw a flow graph:

FIGURE 4.3  Notations for Flow Graph.

Software-Testing_Final.indb 150 31-01-2018 [Link]

 White-Box (or Structural) Testing Techniques • 151

SOLVED EXAMPLES
EXAMPLE 4.1. Consider the following code:
void foo (float y, float a *, int n)
{
float x = sin (y) ;
if (x > 0.01)
z = tan (x) ;
else
z = cos (x) ;
for (int i = 0 ; i < x ; + + i) {
a[i] = a[i] * z ;
Cout < < a [i] ;
}
Draw its flow graph, find its cyclomatic complexity, V(G), and the independ-
ent paths.
SOLUTION. First, we try to number the nodes, as follows:
1.  void foo (float y, float a *, int n)
{
float x = sin (y) ;
if (x > 0.01)

2.  z = tan (x) ;

else

3.  z = cos (x) ;

4.  for (int i = 0; i < n; ++ i)

5.  {
6.  a[i] = a[i] * z ;
cout < < a [i] ‘
}

7. 
cout < < i ;
}

Software-Testing_Final.indb 151 31-01-2018 [Link]

 152 • Software Testing

So, its flow graph is shown in Figure 4.4. Next, we try to find V(G) by
three methods:

FIGURE 4.4  Flow Graph for Example 4.1.

a. V(G) = e – n + 2 (e – edges, n – nodes)

=8–7+2=3
b. V(G) = P + 1 (P- predicate nodes without degree = 2)
= 2 + 1 = 3. (Nodes 1 and 5 are predicate nodes)
c. V(G) = Number of enclosed regions + 1
=2+1=3
∴ V(G) = 3 and is same by all the three methods.
By V(G) = 3 we mean that it is a well written code, its testability is high, and
cost/effort to maintain is low.
Also, it means that there are 3 paths in this program which are indepen-
dent paths and they form a basis-set. These paths are given below:
Path 1: 1 – 2 – 4 – 5 – 7
Path 2: 1 – 3 – 4 – 5 – 7
Path 3: 1 – 3 – 4 – 5 – 6 – 7
Another basis-set can also be formed:
Path 1: 1 – 2 – 4 – 5 – 7
Path 2: 1 – 3 – 4 – 5 – 7
Path 3: 1 – 2 – 4 – 5 – 6 – 7

Software-Testing_Final.indb 152 31-01-2018 [Link]

 White-Box (or Structural) Testing Techniques • 153

This means that we must execute these paths at least once in order to
test the program thoroughly. So, test cases can be designed.
EXAMPLE 4.2. Consider the following program that inputs the marks of five
subjects of 40 students and outputs average marks and the pass/fail message.

h # include <stdio.h>
a. (1)   main ( ) {
(2)   int num_student, marks, subject, total;
b. h (3)   float average ;
(4)   num_student = 1;
h
e. (5)   while (num_student < = 40) {
f. h (6)   total = 0 ;

h
(7)   subject = 1;
(8)   while (subject < = 5) }
(9)   Scanf (“Enter marks: % d”, & marks);
(10) total = total + marks ;
(11) subject ++;
(12) }
g. (13) average = total/5 ;
h (14) if (average > = 50)
h. h (15) printf (“Pass... Average marks = % f”,
average);
(16) else
i. h (17) print (“FAIL ... Average marks are % f”,

average);
j. h (18) num_student ++;
(19) }
h
c. (20) printf (“end of program”);
d. h (21) }

Draw its flow graph and compute its V(G). Also identify the independent
paths.
SOLUTION. The process of constructing the flow graph starts with dividing
the program into parts where flow of control has a single entry and exit point.
In this program, line numbers 2 to 4 are grouped as one node (marked as
“a”) only. This is because it consists of declaration and initialization of varia-

Software-Testing_Final.indb 153 31-01-2018 [Link]

 154 • Software Testing

bles. The second part comprises of a while loop-outer one, from lines 5 to 19
and the third part is a single printf statement at line number 20.
Note that the second part is again divided into four parts—statements
of lines 6 and 7, lines 8 to 12, line 13, and lines 14–17, i.e., if-then-else
structure using the flow graph
notation, we get this flow graph
in Figure 4.5.
Here, “∗” indicates that the node
is a predicate node, i.e., it has an
outdegree of 2.
The statements corre-
sponding to various nodes are
given below:

Statement
Nodes Numbers
a 2–4
b 5
e 6–7
f 8
z 9–12
g 13–14
h 15
i 17
j 18
c 19 FIGURE 4.5  Flow Graph for Example 4.2.
d 20

Next, we compute its cyclomatic complexity, V(G):

a. V(G) = e – n + 2
= 14 – 11 + 2 = 3 + 2 = 5
b. V(G) = Number of predicate nodes (P) + 1
=4+1=5
[ Nodes b, f, g, and c are predicate nodes with two outgoing edges]

Software-Testing_Final.indb 154 31-01-2018 [Link]

 White-Box (or Structural) Testing Techniques • 155

c. V(G) = Number of enclosed regions + 1

=4+1=5
[ R1 – R4 are 4 enclosed regions and 1 corresponds to one outer region]
∴ V(G) = 5 by all three methods. Next, we identify a basis-set with five paths:
Path 1: a – b – d – e
Path 2: a – b – d – f – n – b – d – e
Path 3: a – b – c – g – j – k – m – n – b – d – e
Path 4: a – b – c – g – j – l – m – n – b – d – e
Path 5: a – b – c – g – h – i – n – b – d – e
Each of these paths consist of at least one new edge. Please remember
that this basis set of paths is NOT unique. Finally, test cases can be designed
for the independent path execution that have been identified above. This is
to ensure that all statements are executed at least once.

EXAMPLE 4.3. (Quadratic Equation Problem). This program reads a,

b, and c as the three coefficients of a quadratic equation ax2 + bx + c = 0. It
determines the nature of the roots of this equation. Draw its flow graph and
calculate its cyclomatic complexity.
SOLUTION. First, we write its procedure:
proc roots
1 int a, b, c;
2 D = b * b – 4 * a * c;
3 if (D < 0)
4 real = –b/2 * a ;// imaginary roots
D = – D;
num = pow ((double) D, (double) 0.5);
image = num/(2 * a);
5 else if (D = = 0)
6 root 1 = –b/(2 * a)
root 2 = root 1;
7 else if (D > 0)
8 root 1 = (–b + sqrt (d)/2 * a)
root 2 = (–b – sqrt(d)/2 * a)
9 end

Software-Testing_Final.indb 155 31-01-2018 [Link]

 156 • Software Testing

FIGURE 4.6  Flow Graph of Quadratic Equation Problem.

Now, we draw its flow graph as shown in Figure 4.6.

∴ V(G) = P + 1
= 3 + 1 = 4 [ Node 3, 5, and 7 are predicate nodes]
V(G) = Number of regions + 1
=3+1=4
Also, V(G) = e – n + 2 = 11 – 9 + 2 = 4
∴ We have 4 independent paths and they are:
Path 1: 1 – 2 – 3 – 4 – 9
Path 2: 1 – 2 – 3 – 5 – 6 – 9
Path 3: 1 – 2 – 3 – 5 – 7 – 8 – 9
Path 4: 1 – 2 – 3 – 5 – 7 – 1 – 2 ......
So, the test cases for each of the path are as follows:
Path 1: test case 1
a, b, c: valid input
expected results: D < 0, imaginary roots
Path 2: test case 2
a, b, c: valid input
expected results: D = 0, equal roots

Software-Testing_Final.indb 156 31-01-2018 [Link]

 White-Box (or Structural) Testing Techniques • 157

Path 3: test case 3

a, b, c: valid input
expected results: D > 0, root 1 and root 2 are real
Path 4: test case 4
a, b, c: valid input
expected results: D is not > 0, read a, b, c again

EXAMPLE 4.4. (Triangle Problem): This program reads a, b, c as the three

sides of a triangle and determines whether they form an isosceles, equilat-
eral, or scalene triangle. Draw its flow graph and calculate its V(G).
SOLUTION. We draw its flow chart first.

FIGURE 4.7  Flow Chart for Example 4.4.

Software-Testing_Final.indb 157 31-01-2018 [Link]

 158 • Software Testing

FIGURE 4.8  Flow Graph for the Triangle Problem.

So, we draw its flow graph shown in Figure 4.8.

∴ Its cyclomatic complexity, V(G) by all three methods is
a. V(G) = e – n + 2 = 10 – 8 + 2 = 4
b. V(G) = P + 1 = 3 + 1 = 4 [Nodes 2, 4, 6 are predicate nodes]
c. V(G) = Number of enclosed regions + 1 = 3 + 1 = 4.
∴ 4-test cases need to be designed. They are given below:
Path 1: test case 1
a, b, c: valid input
Expected 
results: if a = b or b = c or a = c
then message ‘isosceles triangle’ is
displayed.
Path 2: test case 2
a, b, c: valid input

Expected results: if a ≠ b ≠ c then message
‘scalene triangle’ is displayed.
Path 3: test case 3
a, b, c: valid input

Expected results: if a = b = c then message
‘equilateral triangle’ is displayed.

Software-Testing_Final.indb 158 31-01-2018 [Link]

 White-Box (or Structural) Testing Techniques • 159

Path 4: test case 4


a, b, c: valid float inputs and if a > 0 and b > 0
and c > 0 Expected results: proper inputs—a, b and c
and proper results.

EXAMPLE 4.5. Consider the following code snippet to search a number

using binary search:

void search (int key, int n, int a[])

{
int mid;
1. int bottom = 0;
2. int top = n – 1;
3. while (bottom <= top)
{
4. mid = (top + bottom)/2;
5. if (a[mid]==key)
{
6. printf (“\n element is found”);
7. return;
}
else
{
8. if (a[mid] < key)
9. bottom = mid + 1;
else
10. top = mid – 1;
}
}
11. }

a. Draw its flow graph

b. Find V(G) by all 3 methods
c. Derive test cases

Software-Testing_Final.indb 159 31-01-2018 [Link]

 160 • Software Testing

SOLUTION. Flow graph is shown in Figure 4.9.

FIGURE 4.9

∴ V(G) = [Total number of enclosed regions (R1, R2, R3)

+ 1 outer region where this graph is enclosed]
=3+1=4
or V(G) = e – n + 2 = 13 – 11 + 2 = 2 + 2 = 4
or V(G) = P + 1 = 3 + 1 = 4 [ Nodes 3, 5, and 8 are predicate
nodes with outdegree as 2]
Basis set of paths is
Path 1: 1, 2, 3, 4, 5, 6, 7, 11
Path 2: 1, 2, 3, 11
Path 3: 1, 2, 3, 4, 5, 8, 9, 3, …
Path 4: 1, 2, 3, 4, 5, 8, 10, 3, …
V(G) = 4
⇒  Paths are also 4 as given above.
⇒ These 4 paths must be executed at least once in order to test this pro-
gram thoroughly.
∴  Test cases are:

Software-Testing_Final.indb 160 31-01-2018 [Link]

Software-Testing_Final.indb 161
Test Test case Test case Test status
case id description Step Expected results Actual studies (P/F)
1. Checking bottom Set bottom = 0 Initially bottom <= Design
and top values of top = n – 1 top is true.
our list check if bottom true. Lists length
<= top by while gets reduced after
loop every iteration If
bottom >= top is
reached then return
to main

2. Checking if middle set mid = (top If a[mid] = key Design

element of array is + bottom)/2 then value then print
equal to key value compare if a[mid] message “even
= key found” and return
to main
3. If a[mid] < key set bottom = mid search right sublist Design
value + 1 and then goto for key element
while
4. If a[mid] > key set top = mid + search left sublist Design
value 1 and go back to for key element
while-loop

31-01-2018 [Link]
 162 • Software Testing

EXAMPLE 4.6. Consider the following

recursive code to find GCD of two num-
bers.
GCD (x, y)
{
0 if x = y
1 ret x;
2 else if x > y
3 GCD (x – y, y);
4 else
5 GCD (x, y – x)
FIGURE 4.10
6 }
Draw its flowgraph, find V(G), and basis
path set. Derive test cases also.
SOLUTION. The flowgraph for the GCD program is shown in Figure 4.10.
∴ V(G) = Enclosed regions + 1 = 2 + 1 = 3
V(G) = e – n + 2 = 7 – 6 + 2 = 1 + 2 = 3
= P + 1 = 2 + 1 = 3 ( Nodes 1 and 3 are predicate nodes)
Three independent paths are
1–3–4–6
1–3–5–6
1–2–6
Three test cases are:

Test Test case Actual Test case

case id description Step Expected results result status
1. x=y Read 2 nos. such GCD = x
that x = y
2. x>y Read 2 nos. such Call recursive
that x > y routine
GCD (x – y, y)
3. x<y Read 2 nos. such Call recursive
that x < y routine
GCD (x, y – x)

Software-Testing_Final.indb 162 31-01-2018 [Link]

 White-Box (or Structural) Testing Techniques • 163

1. In test case id–2, we call GCD function recursively with x = x – y and y
NOTES as it is.
2. In test case id–3, we call GCD function recursively with y = y – x and x
as it is.

EXAMPLE 4.7. What is a cyclomatic complexity? Find the following related

to given:
if(c1){
f1();
}else{
f2();
}
if(c2){
f3();
}else{
f4();
}
i. What is a cyclomatic number?
ii. How many test cases are required to achieve complete branch
average?
iii. How many test cases are required for complete path coverage?

SOLUTION.
„„ Cyclomatic complexity is a software metric that provides a quantitative
measure of the logical complexity of a program.
„„ Cyclomatic complexity has a foundation in graph theory and is computed
in one of the three ways:
i. The number of regions corresponds to the cyclomatic complexity.
ii. Cyclomatic complexity: E – N + 2 (E is the number of edges, and
N is number of nodes).
iii. Cyclomatic complexity: P + 1 (P is the number of predicate nodes).
Referring to the flow graph, the cyclomatic number is:
1. The flow graph has three regions.
2. Complexity = 8 edges – 7 nodes + 2 = 3
3. Complexity = 2 predicate nodes + 1 = 3 (Predicate nodes = C1, C2)

Software-Testing_Final.indb 163 31-01-2018 [Link]

 164 • Software Testing

FIGURE 4.11

Two test cases are required for complete branch coverage and four test cases
are required for complete path coverage.

Assumptions:
c1; if(i%2==0)
f1: EVEN()
f2: ODD()
c2: if(j > 0)
f3: POSITIVE()
f4: NEGATIVE()
i.e.
if(i%2==0){
EVEN();
}else{
ODD();
}
if(j < 0){
POSITIVE();
}else{
NEGATIVE();
}
Test cases that satisfy the branch coverage criteria, i.e., <c2, f1, c2, f3> and
<c1, f2, c2, f4>.

Sr. No. Test case: Value of i, j Decision or Branch Value of predicate

c1 if (i % 2 == 0) True
1. 10, 10
c2 if (j > 0) True
c1 if (i % 2 == 0) False
2. 5, –8
c2 if (j < 0) False

Software-Testing_Final.indb 164 31-01-2018 [Link]

 White-Box (or Structural) Testing Techniques • 165

Test cases that satisfies the path coverage criteria, i.e.,

<c1, f1, c2, f3>
<c1, f2, c2, f4>
<c1, f1, c2, f4>
<c1, f2, c2, f3>

Sr. Test case: Value

No. of i, j Decision or Branch Value of predicate
1. 10, 10 c1 True
c2 True
2. 7, –8 c1 False
c2 False
3. 10, –8 c1 True
c2 False
4. 7, 20 c1 False
c2 True

[Link].  DD Path Testing with Solved Examples

From a given flow graph, we can easily draw another graph that is known as
a decision-to-decision (DD) path graph. Our main concentration now is on
the decision nodes only. The nodes of the flow graph are combined into a
single node if they are in sequence.
EXAMPLE 4.8. Consider the following pseudo code:
0 Premium (age, sex, married)
1 {
2 premium = 500;
3 
if ((age < 25) and (sex==male) and (not
married))
4 premium = premium + 1500;
5 else {
6 if (married or (sex==female))
7 premium = premium–200;
8 if ((age > 45) and (age < 65))
9 premium = premium – 100;
10 }
11 }

Software-Testing_Final.indb 165 31-01-2018 [Link]

 166 • Software Testing

a. Design its program graph b. Design its DD path graph

c. List all DD-paths

SOLUTION.

a. Its program graph is as follows: b. Its DD-path graph is as follows:

Nodes Lines
A 0, 1, 2
B, C, D 3, 4, 5
E 6
F 7
G, H, I 8, 9, 10
J 11

FIGURE 4.12 FIGURE 4.13

c. The DD-paths are as follows:

Path–1: A–B–C–J
Path–2: A–B–D–E–G–I–J
Path–3: A–B–D–E–F–G–H–I–J
Path–4: A–B–D–E–G–H–I–J
EXAMPLE 4.9. Consider the following code snippet:
1 begin
2 float x, y, z = 0.0;
3 int count;
4 input (x, y, count);
5 do {

Software-Testing_Final.indb 166 31-01-2018 [Link]

 White-Box (or Structural) Testing Techniques • 167

6 if (x ≤ 0) {
7 if(y ≥ 0){
8 z = y*z + 1;
9 }
10 }
11 else {
12 z = 1/x;
13 }
14 y = x * y + z
15 count = count – 1
16 while (count > 0)
17 output (z);
18 end
Draw its data-flow graph. Find out whether paths (1, 2, 5, 6) and (6, 2, 5, 6)
are def-clear or not. Find all def-use pairs?
SOLUTION. We draw its data flow graph (DD-path graph) first.

where Nodes Lines

1 1, 2, 3, 4
2 5, 6
3 7
4 8, 9, 10
5 11, 12, 13
6 14, 15, 16
7 17, 18

FIGURE 4.14

Now, path (1, 2, 5, 6) is definition clear (def-clear) with respect to defi-

nition of x at node-1, i.e., d1(x).
So, d1(x) is live at node 6, i.e., x is being used at node-6 again. But path
(1, 2, 5, 6) is not def-clear with respect to d1(z) because of d5(z).

Software-Testing_Final.indb 167 31-01-2018 [Link]

 168 • Software Testing

Now, consider path (6, 2, 5, 6). It is def-clear with respect to d6 (count).

Variables y and z are used at node 4. Please note that we can easily check for
definitions d1(y), d6(y), d1(z), and d5(z) that they are live at node 4.

What about def-use pairs?

We have 2 types of def-use pairs:
1. That correspond to a definition and its c-use, (dcu). (“c” stands for
computational.)
2. That correspond to a definition and its p-use, (dpu). (“p” stands for
predicate condition.)
Let us then compute dcu (x, 1), i.e., the set of all c-uses associated with
the definition of x at node-1. Also note that there is a c-use of x at node-5 and
a def-clear path (1, 2, 5) from node-1 to node-5. So, node-5 is included in
dcu (x, 1). Similarly, we can include node-6 also in this set. So, we get:
dcu (x, 1) = {5, 6}

What is dpu (x, 1)?

As shown in program, there is a p-use of x at node-2 with outgoing edges
(2, 3) and (2, 5). The existence of def-clear paths from node-1 to each of these
two edges is easily seen in DD-path graph. There is no other p-use of x. So,
dpu (x, 1) = {(2, 3), (2, 5)}

We can now compute for all other variables as shown below:

Variable Defined at node (n) dcu (v, n) dpu (v, n)

x 1 {5, 6} {(2, 3), (2, 5)}
y 1 {4, 6} {(3, 4), (3, 6)}
y 6 {4, 6} {(3, 4), (3, 6)}
z 1 {4, 6, 7} {}
z 4 {4, 6, 7} {}
z 5 {4, 6, 7} {}
count 1 {6} {(6, 2), (6, 7)}
count 6 {6} {(6, 2), (6, 7)}

The spots that are not def-clear are trouble spots.

Software-Testing_Final.indb 168 31-01-2018 [Link]

 White-Box (or Structural) Testing Techniques • 169

[Link]. Graph Matrices Technique with Solved Examples

In graph matrix based testing, we convert our flow graph into a square matrix
with one row and one column for every node in the graph. The objective
is to trace all links of the graph at least once. The aim is again to find cyclo-
matic complexity, V(G) and, hence, the independent paths. If the size of graph
increases, it becomes difficult to do path tracing manually.

For example, Consider the Its graph-matrix is:

following flow graph: 1 2 3 4
1 a c

3 b

4 d

FIGURE 4.15  Flow Graph Example. FIGURE 4.16  Graph Matrix.

Now, consider another flow graph: ∴ Its graph matrix is:

1 2 3
1 a d

2 b+e

3 c

FIGURE 4.17  Flow Graph Example. FIGURE 4.18  Graph Matrix.

Note that if there are several links between two nodes then “+” sign denotes a
parallel link.
This is, however, not very useful. So, we assign a weight to each entry
of the graph matrix. We use “1” to denote that the edge is present and “0”
to show its absence. Such a matrix is known as a connection matrix. For the
Figure above, we will have the following connection matrix:

Software-Testing_Final.indb 169 31-01-2018 [Link]

 170 • Software Testing

1 2 3 4 1 2 3 4

1 1 1 1 1 1 2–1=1

2 2

3 1 1–1=0
3 1
4 1 1–1=0
4 1
1 + 1 = 2 = V(G)

FIGURE 4.19   Connection Matrix. FIGURE 4.20

Now, we want to compute its V(G). For this, we draw the matrix again,
i.e., we sum each row of the above matrix. Then, we subtract 1 from each
row. We, then, add this result or column of result and add 1 to it. This gives
us V(G). For the above matrix, V(G) = 2.
EXAMPLE 4.10. Consider the following flow graph:

FIGURE 4.21

Draw its graph and connection matrix and find V(G).

SOLUTION.
First, we draw its graph matrix: Now, its connection matrix is:

FIGURE 4.22 FIGURE 4.23

Software-Testing_Final.indb 170 31-01-2018 [Link]

 White-Box (or Structural) Testing Techniques • 171

Similarly, we can find two link or three link path matrices, i.e., A2, A3, ...
A . These operations are easy to program and can be used as a testing tool.
n–1

[Link].  Data Flow Testing with Examples

Data flow testing uses the sequence of variable access to select paths from
a control graph. The main focus of this structural or white-box testing tech-
nique is on the usage of variables. With a program data definition faults are
nearly as frequent (22% of all faults) as control flow faults (24 percent). Some
common data related faults include:
„„ Using an undefined or uninitialized variable.
„„ Unreachable code segments.
„„ Assigning a value to a variable more than once without an intermediate
reference.
„„ Deallocating or re-initializing a variable before its last use.
It is another form of white-box testing. Here the focal point is on the
usage of variables. Because data variables and data structures are an import-
ant part of any software, so we must check for variables— which have not
been defined but used or variables which are defined but not used or a vari-
able that is defined twice. These may be the main culprits or sources of
errors in our program. We need to identify them and remove the problems
associated with them.
Some terminology used during data flow testing is given below:
1. Defining node: A defining node, n, of the variable, v, if the value of the
variable, v, is defined at this node, n.
For example, a=b+c
↑
This is defining node for “a.”
2. Usage node: A usage node, n, of the variable, v, if the value of the
variable, v, is used at this node, n.
For example, a=b+c
↑
This is the usage node for “b” and “c.”
3. Definition use path (du-path): The path from a node, m, to node, n,
where the variable, v, is initially defined to the point where it is used in
the program is called as du-path.

Software-Testing_Final.indb 171 31-01-2018 [Link]

 172 • Software Testing

4. Definition clear path (dc-path): The path from node, m, to node, n,

such that there is no other node in this path that redefines the variable, v.
Please note that our objective is to find all du-paths and then iden-
tify those du-paths which are not dc-paths. We generate test cases for
du-paths that are not dc-paths.
Steps followed for data flow testing are given below:
Step 1:  Draw the program flow graph.
Step 2: Prepare a table for define/use status of all variables used in your
program.
Step 3:  Find all du-paths.
Step 4:  Identify du-paths that are not dc-paths.
This is data flow testing.

EXAMPLE 4.11. For the following C-like program, give the

1. flow graph.
2. def/use graph.
scanf(x, y);
if (y < 0)
pow = 0 – y;
else
pow = y;
z = 1.0;
while (pow!=0)
{
z = z * x;
pow = pow-1;
}
if (y < 0)
z = 1.0/z;
printf(z);
SOLUTION.
Let us rewrite the program again for easy drawing of its flow graph.
1 scanf(x, y); if (y < 0)
2 pow = 0 – y;
3 else pow = y;
4 z = 1.0;
5 while (pow! = 0)

Software-Testing_Final.indb 172 31-01-2018 [Link]

 White-Box (or Structural) Testing Techniques • 173

6 {z = z*x; pow = pow – 1;}

7 if (y < 0)
8 z = 1.0/z;
9 printf (z);

i. The flow graph of a given program is ii. The def/use graph for this
as follows: program is as follows:

FIGURE 4.24 FIGURE 4.25

Now, let us find its dcu and dpu. We draw a table again as follows:

(Node, var) dcu dpu

(1, x) {6} φ
(1, y) {2, 3} {(1, 2), (1, 3), (7, 8), (7, 9)}
(2, pow) {6} {(5, 6), (5, 7)}
(3, pow) {6} {(5, 6), (5, 7)}
(4, z) {6, 8, 9} φ
(6, z) {6, 8, 9} φ
(6, pow) {6} {(5, 6), (5, 7)}
(8, z) {9} φ

Software-Testing_Final.indb 173 31-01-2018 [Link]

 174 • Software Testing

EXAMPLE 4.12.  (The Commission Problem). Consider again the

­commission problem. The problem domain has already been discussed in
Chapter 2. Write its program, draw its flow graph and DD-path graph. Find
its V(G) and, hence, independent paths. Also, perform its data flow testing.

SOLUTION. We develop its program first:

1 Program commission (Input, Output)
2 Dim locks, stocks, barrels As Integer
3 
Dim lockprice, stockprice, barrel price
As real
4 
Dim totallocks, totalstocks, total barrels
As integer
5 
Dim locksales, stocksales, barrelsales As
real
6 Dim sales, commission As real
7 lockprice = 45.0
8 Stockprice = 30.0
9 barrelprice = 25.0
10 totallocks = 0
11 totalstocks = 0
12 total barrels = 0
13 Input (locks)
14 while NOT clocks = –1)
15 Input (stock, barrels)
16 totallocks = totallocks + locks
17 totalstocks = totalstocks + stocks
18 totalbarrels = totalbarrels + barrels
19 Input (locks)
20 Endwhile
21 Output (“Locks sold:”, total locks)
22 
Output (“Stocks sold:”, total stocks)
23 
Output (“Berrels sold:”, total
barrels)
24 locksales = lockprice * totallocks
25 stocksales = stockprice * totalstocks
26 
barrelsales = barrelprice *
totalbarrels
27 
sales = locksales + stocksales +
barrelsales
28 
Output (“Total sales:”, sales)
29 If (sales > 1800.0)

Software-Testing_Final.indb 174 31-01-2018 [Link]

 White-Box (or Structural) Testing Techniques • 175

30 Then
31 commission = 0.10 * 1000.0
32 commission = commission + 0.15 *
800
33 commission = commission + 0.20 *
(sales – 1800)
34 Else if (sales > 100.0)
35 then
36 commission = 0.10 * 1000.0
37 commission = commission + 0.15 *
(sales – 1000.0)
38 else commission = 0.10 * sales
39 Endif
40 Endif
41 Output (“Commission is $”,
commission)
42 End commission
Now, we draw its flow graph first:

FIGURE 4.26  Flow Graph of Commission Problem.

Software-Testing_Final.indb 175 31-01-2018 [Link]

 176 • Software Testing

Next, we draw its DD path graph:

Note that more compression exists in this
DD-path graph because of the increased compu-
tation in the commission problem.
The table below details the statement frag-
ments associated with DD-paths.

DD-path Nodes
A 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13
B 14
C 15, 16, 17, 18, 19
D 20, 21, 22, 23, 24, 25, 26, 27, 28
E 29
F 30, 31, 32, 33
G 34
H 35, 36, 37
I 38
J 39
K 40
FIGURE 4.27  DD Path Graph. L 41, 42

∴ We now compute its V(G):

a. V(G) = Number of enclosed regions + 1 = 3 + 1 = 4
b. V(G) = e – n + 2 = 14 – 12 + 2 = 2 + 2 = 4
c. V(G) = P + 1 = 3 + 1 = 4 [ Nodes B, E, and G are predicate nodes]
∴ There must exist four independent paths. They are given below.
Path 1: A – B – C – B – ...
Path 2: A – B – D – E – F – K – L
Path 3: A – B – D – E – G – H – J – K – L
Path 4: A – B – D – E – G – I – J – K – L
Next, we prepare the define/use nodes for variables in the commission
problem.

Software-Testing_Final.indb 176 31-01-2018 [Link]

 White-Box (or Structural) Testing Techniques • 177

Variables Defined at node Used at node

lockprice 7 24
stockprice 8 25
barrelprice 9 26
total locks 10, 16 16, 21, 24
total stocks 11, 17 17, 22, 25
total barrels 12, 18 18, 23, 26
locks 13, 19 14, 16
stocks 15 17
barrels 15 18
lock sales 24 27
stock sales 25 27
barrel sales 26 27
sales 27 28, 29, 33, 34, 37, 38
commission 31, 32, 33, 36, 37, 38 32, 33, 37, 41

Next, to find du-paths, we prepare the table:

Variable Path (beginning, end) nodes Definition clear

lockprice 7, 24 yes
stockprice 8, 25 yes
barrelprice 9, 26 yes
total stocks 11, 17 yes
total stocks 11, 22 no
total stocks 17, 25 no
total stocks 17, 17 yes
total stocks 17, 22 no
total stocks 17, 25 no
locks 13, 14 yes
locks 19,14 yes
locks 13, 16 yes
locks 19, 16 yes
(Continued)

Software-Testing_Final.indb 177 31-01-2018 [Link]

 178 • Software Testing

Variable Path (beginning, end) nodes Definition clear

sales 27, 28 yes
sales 27, 29 yes
sales 27, 33 yes
sales 27, 34 yes
sales 27, 37 yes
sales 27, 38 yes
commission 31, 32 yes
commission 31, 33 no
commission 31, 37 not possible
commission 31, 41 no
commission 32, 32 yes
commission 32, 33 yes
commission 32, 37 not possible
commission 32, 41 no
commission 33, 32 not possible
commission 33, 33 yes
commission 33, 37 not possible
commission 33, 41 yes
commission 36, 32 not possible
commission 36, 33 not possible
commission 36, 37 yes
commission 36, 41 no
commission 37, 32 not possible
commission 37, 33 not possible
commission 37, 37 yes
commission 37, 41 yes
commission 38, 32 not possible
commission 38, 33 not possible
commission 38, 37 not possible
commission 38, 41 yes

Software-Testing_Final.indb 178 31-01-2018 [Link]

 White-Box (or Structural) Testing Techniques • 179

The initial value definition for the variable, “titalstocks” occurs at node-
11 and it is first used at node-17. Thus, the path (11, 17) which consists of the
node sequence <11, 12, 13, 14, 15, 16, 17>, is definition clear.
The path (11, 22), which consists of the node sequence <11, 12, 13, 14,
15, 16, 17, 18, 19, 20> * & <21, 22> is not definition clear because the values
of total stocks are defined at node-11 and at node-17. The asterisk, *, is used
to denote zero or more repetitions.
Thus, out of 43 du-paths, 8-paths, namely, (11, 22), (17, 25), (17, 22),
(17, 25), (31, 33), (31, 41), (32, 41), and (36, 41), are not definition clear.
These 8-paths are the main culprits and thus the cause of the error.

Problem for Practice

1. Consider the problem of finding out the roots of a given quadratic
equation, ax2 + bx + c = 0.
Write its program in C and perform the following:
a. Draw the flow graph.
b. Draw the DD path graph.
c. Find V(G) by the three methods.
d. Perform its data flow testing and find all du- and dc-paths.

How to Develop Data Flow Test Cases?

The steps that are usually followed to develop data flow test cases are as fol-
lows:
1. Layout a control graph for the program.
2. For each variable, note the defined (d), used (u), and killed (k) actions
by node.
3. Trace all possible du- and dk-paths for each variable.
4. Merge the paths.
5. Test/check each path for testability.
6. Select test data for feasible paths.

Software-Testing_Final.indb 179 31-01-2018 [Link]

 180 • Software Testing

We shall now apply these steps to the following code:

1. void MyClass :: xyz (int x, int y, int z)
{
if (x>=10) {
2. x = x + 2;
y = y – 4;
if (x > z) {
3. y = x – z;
else
4. y = 7;
}
else {
5. y = z + x;
}
6. while (x > y) {
7. y = y + 1;
z = z – y;
}
8. if (x > z)
9. cout << x<<z;
10. cout << x<<y;
}
Step 1: Its flow graph is:

FIGURE 4.28  Flow Graph for Example.

Software-Testing_Final.indb 180 31-01-2018 [Link]

 White-Box (or Structural) Testing Techniques • 181

Step 2: N
 ext, we tabulate data actions. The D and U actions in xyz-class can
be read from the code.

Block Source code x y z

1. MyClass:: xyz
(int x; int y; int z) { D D D
if (x >= 10) { U
2. x = x + 2; UD
y = y – 4; UD
if (x > z) { U U
3. y = x – z; U D U
else
4. y = 7; D
}
else
5. y = z + x; U D U
}
6. while (x > y) { U U
7. y = y + 1; UD
z = z – y; U UD
}
8. if (x > z) U U
9. cout << x, z; U U
10. cout << x, y; } U U

Step 3: Trace data flow for X

There will be a set of paths for each d. Mark the d nodes for the variable on
the graph and trace the paths. The baseline trace method can be adapted for
this tracing. Tabulate the paths as you trace them. So, define/use paths for
“x” are shown in Figure 4.29.
Note that the variable-x has d actions at nodes-1 and -2.
∴ Paths from node-1:
1–5–6–8–9–10 DU–
1–5–6–8–10 DU–
1–2 DUD

Software-Testing_Final.indb 181 31-01-2018 [Link]

 182 • Software Testing

= Define
= Use

FIGURE 4.29  Trace Data Flow Graph for “X.”

And paths from node-2:

2–3–6–8–9–10 DU–
2–4–6–8–9–10 DU–
2–3–6–8–10 DU–
Similarly, trace data flow for y is as follows: Define/Use paths for y are
shown in Figure 4.30. Variable “y” has d actions are nodes 1, 2, 3, 4, 5, 7.
Paths from node-1:

where
= Define
= Use

FIGURE 4.30  Trace Data Flow for “y.”

Software-Testing_Final.indb 182 31-01-2018 [Link]

 White-Box (or Structural) Testing Techniques • 183

1–5 DUD
1–2 DUD
Paths from node-2:
2–3 DD
2–4 DD
Paths from node-3:
3–6–7 DUD
3–6–8–10 DU–
Paths from node-4:
4–6–7 DUD
4–6–8–10 DU–
Paths from node-5:
5–6–7 DUD
5–6–8–10 DU–
Paths from node-7:
5–6–7 DUD
5–6–8–10 DU–
Similarly, the trace data flow for “Z” is as
follows: Define/Use Paths for Z are shown
in Figure 4.31. The variable Z has d actions
at nodes-1 and-7.
∴ Paths from node-1 are:
1–5–6–7 DUD
1–5–6–8–9 DU– where
1–5–6–8 DU– = Define
1–2–3–6–7 DUD = Use
1–2–4–6–7 DUD
1–2–3–6–8–9 DU–
1–2–3–6–8 DU–
Paths from node-7:
7–6–8–9 DU– FIGURE 4.31  Trace Data Flow
7–6–7 DU– for “Z.”
7–6–8 DU–
Step 4: Merge the paths
a. Drop sub-paths: Many of the tabulated paths are sub-paths. For
example,
{1 2 3 6 8} is a sub-path of {1 2 3 6 8 9}.
So, we drop the sub-path.

Software-Testing_Final.indb 183 31-01-2018 [Link]

 184 • Software Testing

b. Connect linkable paths: Paths that end and start on the same node
can be linked. For example,
{1 5} {5 6 8 10}
becomes {1 5 6 8 10}
We cannot merge paths with mutually exclusive decisions. For example,
{1 5 6 8 9} and {1 2 3 6 7}
cannot be merged because they represent the predicate branches from
node-1.
So, merging all the traced paths provides the following set of paths:

Path Nodes visited

1 1–5–6–8–10
2 1–5–6–8–9–10
3 1–5–6 (7 6)* 8 10
4 1–5–6 (7 6)* 8 9 10
5 1–2–3–6–8–10
6 1–2–3–6–8–9–10
7 1–2–3–6 (7 6)* 8–10
8 1–2–3–6 (7 6)* 8–9–10
9 1–2–4–6–8–10
10 1–2–4–6–8–9–10
11 1–2–4–6 (7 6)* 8–10
12 1–2–4–6 (7 6)* 8–9–10

The (7 6)* means that path 7–6 (the loop) can be iterated. We require that a
loop is iterated at least twice.
Step 5: Check testability
We need to check path feasibility.
Try path 1: {1–5–6–8–10}

Condition Comment
1.  x ≤ 10 Force branch to node-5
2. y′ = z + x Calculation on node-5
3.  x ≤ z + x Skip node-7, proceed to node-8
4.  x ≤ z Skip node-9, proceed to node-10

Software-Testing_Final.indb 184 31-01-2018 [Link]

 White-Box (or Structural) Testing Techniques • 185

The values that satisfy all these constraints can be found by trial and error,
graphing, or solving the resulting system of linear inequalities.
The set x = 0, y = 0, z = 0 works, so this path is feasible.
Try path 5: {1–2–3–6–8–10}

Condition Comment
1.  x > 10 Force branch to node-2
2. x′ = x + 2 Calculation on node-2
3. y′ = y – 4 Calculation on node-2
4. x′ > z Force branch to node-3
5. x′ ≤ y′ Skip node-7, proceed to node-8
6. x′ ≤ z Skip node-9, proceed to node-10

By inspection, conditions 4 and 6 cannot both be true, so path 5 is dropped.

Step 6: Select test data
Path feasibility analysis also provides domain information for each path. This
can be used in conjunction with other domain constraints (from require-
ments, for example) to develop test data for each path.

Test Data
Input test data Output
Path Nodes visited x y z B9 B10
1. 1–5–6–8–10 0 0 0 – 0
9 0 9 – 18
2. 1–5–6–8–9–10 9 0 8 17 25
9 0 0 9 18
3. 1–5–6–(7 6)* 8–10 –1 0 –1 – –2
4. 1–5–6–(7–6)* 8–9–10 0 0 1 –1 –1
9 0 –1 –1 18
7. 1–2–3–6–(7–6)* 8–10 10 0 62 – 24
8. 1–2–3–4–(7–6)*8–9–10 10 0 12 –26 24
10 0 61 23 24
10. 1–2–3–6–8–9–10 10 0 0 12 24
12. 1–2–3–6–(7–6)*–8–9–10 10 0 1 1 24

Software-Testing_Final.indb 185 31-01-2018 [Link]

 186 • Software Testing

4.3. MUTATION TESTING VERSUS ERROR SEEDING —

DIFFERENCES IN TABULAR FORM
In the error-seeding technique, a predefined number of artificially gener-
ated errors is “sown” in the program code. After that, test runs are used to
detect errors and to examine the ratio between actual and artificial errors
based on the total number of detected errors. The artificially generated
errors are not known to the testers. Error seeding and mutation testing are
both error-oriented techniques and are generally, applicable to all levels of
testing. We shall now tabulate the differences between these two techniques
of white-box testing. They are as follows:

Error seeding Mutation testing

1. No mutants are present 1. Mutants are developed for
here. testing.
2. Here source code is tested 2. Here mutants are combined,
within itself. compared for testing to find
errors introduced.
3. Errors are introduced 3. Special techniques are used to
directly. introduce errors.
4. Test cases which detect 4. Here, test cases which kill
errors are used for testing. mutants are used for testing.
5. It is a less efficient error test- 5. It is more efficient than error
ing technique. ­seeding.
6. It requires less time. 6. It is more time consuming.
7. It is economical to p ­ erform. 7. It is expensive to perform.
8. It is a better method for big- 8. It is a better method for small
ger problems. size programs.

This mutation testing scheme was proposed by DeMillo in 1978. In this

testing technique, we mutate (change) certain statements in the source code
and check if the test code is able to find the errors. It is a technique that is
used to assess the quality of the test cases, i.e., whether they can reveal cer-
tain types of faults.

Software-Testing_Final.indb 186 31-01-2018 [Link]

 White-Box (or Structural) Testing Techniques • 187

For example, SDLC is viewed as a “root-finding” procedure. That is, a

designer submits an initial guess which is then iteratively tested for validity
until a correct solution is obtained. This is the principle of meta-­induction.
These mutants are run with an input data from a given test set. If a test
set can distinguish a mutant from the original program, i.e., it produces a
different execution result, the mutant is said to be killed. Otherwise, the
mutant is called as a live mutant. Please note that a mutant remains live
because it is equivalent to the original program, i.e., it is functionally identi-
cal to the original program or the test data is inadequate to kill the mutant.
If a test data is inadequate, it can be improved by adding test cases to kill the
live mutant. A test set which can kill all non equivalent mutants is said to be
adequate (mutation score). Now, the adequacy of a test set is measured as
follows:
No. of Killed Mutants
Adequacy of Test Set =
No. of Non equivalent Mutants

Advantages of Mutation Testing

1. It can show the ambiguities in code.
2. It leads to a more reliable product.
3. A comprehensive testing can be done.

Disadvantages of Mutation Testing

1. It is difficult to identify and kill equivalent mutants.
2. Stubborn mutants are difficult to kill.
3. It is a time consuming technique, so automated tools are required.
4. Each mutation will have the same size as that of the original program.
So, a large number of mutant programs may need to be tested against
the candidate test suite.

Software-Testing_Final.indb 187 31-01-2018 [Link]

 188 • Software Testing

4.4. COMPARISON OF BLACK-BOX AND WHITE-BOX

TESTING IN TABULAR FORM

Functional or black-box Structural or white-box

testing testing or glass-box testing
1. This method focus on functional 1. This method focuses on procedural
requirements of the software, i.e., details, i.e., internal logic of a pro-
it enables the software engineer gram.
to derive sets of input conditions
that will fully exercise all functional
requirements for a program.
2. It is NOT an alternative approach 2. It concentrates on internal logic,
to white-box technique rather is a mainly.
complementary approach that is
likely to uncover a different class of
errors.
3. Black-box testing is applied during 3. Whereas, white-box testing is per-
later stages of testing. formed early in the testing process.
4. It attempts to find errors in the fol- 4. Whereas, white-box testing attempts
lowing categories: errors in following cases:
a. Incorrect or missing functions a. Internal logic of your program
b. Interface errors b. Status of program
c. Errors in data structures or
external database access
d. Performance errors
e. Initialization and termination
errors
5. It disregards control structure of 5. It uses control structure of the pro-
procedural design (i.e., what is the cedural design to derive test cases.
control structure of our program, we
do not consider here).

Software-Testing_Final.indb 188 31-01-2018 [Link]

 White-Box (or Structural) Testing Techniques • 189

Functional or black-box Structural or white-box

testing testing or glass-box testing
  6. Black-box testing broadens our   6. White-box testing, as described
focus on the information domain by Hetzel is “testing in small,” i.e.,
and might be called as “testing testing small program components
in the large,” i.e., testing bigger (e.g., modules or small group of
monolithic programs. modules).

  7. Using black-box testing   7. Using white-box testing, the

­techniques, we derive a set of test ­software engineer can desire test
cases that satisfy following criteria: cases that:
a. Test cases that reduce (by a a. guarantee that all independent
count that is greater than 1) the paths within a module have
number of additional test cases been exercised at least once.
that must be designed to achieve b. exercise all logical decisions on
reasonable testing. their true and false sides.
b. Test cases that tell us something c. execute all loops at their
about the presence or absence boun­ daries and within their
of classes of errors rather than operational bounds.
an error associated only with the
d. exercise internal data structures
specific tests at hand.
to ensure their validity.
  8. It includes the tests that are con-   8. A close examination of procedural
ducted at the software interface. detail is done.
  9.  Are used to uncover errors.   9. Logical paths through the software
are tested by providing test cases,
that exercise specific sets of condi-
tions or loops.
10. To demonstrate that software 10. A limited set of logical paths be
­functions are operational, i.e., examined.
input is properly accepted and
­output is correctly produced.
Also, the integrity of external
information (e.g., a data base) is
maintained.

Software-Testing_Final.indb 189 31-01-2018 [Link]

 190 • Software Testing

4.5.  PRACTICAL CHALLENGES IN WHITE-BOX TESTING

The major challenges that we face during white-box testing are as follows:
1. Difficult for Software Developer to Pin-Point Defects From His Own
Creations: As discussed earlier, no one would like to point out errors
from their own creations. So, does a developer. So, usually we select a
different test team.
2. Even a Completely Tested Code May Not Satisfy Real Customer
Requirements: Developers do not have a full appreciation and favor
to external customer’s requirements or the domain knowledge. This
means that even after thorough verification and validation, common user
scenarios may get left out. So, we must address such scenarios.

4.6. COMPARISON ON VARIOUS WHITE-BOX TESTING

TECHNIQUES

Functional testing techniques always result in a set of test cases and s­ tructural
metrics are always expressed in terms of something countable like the num-
ber of program paths, the number of decision-to-decision paths (DD-paths),
and so on.

FIGURE 4.32  Shows Trends of Test Coverage Item(s).

Software-Testing_Final.indb 190 31-01-2018 [Link]

 White-Box (or Structural) Testing Techniques • 191

FIGURE 4.33  Shows Trend of Test Method Effort.

Figures 4.32 and 4.33 show the trends for the number of test cover-
age items and the effort to identify them as functions of structural testing
methods, respectively. These graphs illustrate the importance of choosing an
appropriate structural coverage metric.

4.7.  ADVANTAGES OF WHITE-BOX TESTING

1. White-box testing helps us to identify memory leaks. When we allocate
memory using malloc( ) in C, we should explicitly release that memory
also. If this is not done then over time there would be no memory
available for allocating memory on requests. This can be done using
debuggers that can also tally allocated and freed memory.
2. Performance analysis: Code coverage tests can identify the areas of a
code that are executed most frequently. Extra efforts can then be made
to check these sections of code. To do further performance improvement
techniques like caching, coprocessing or even parallel processing can be
considered.
3. Coverage tests with instrumented code is one of the best means of
identifying any violations of such concurrency constraints through
critical sections.
4. White-box testing is useful in identifying bottlenecks in resource usage.
For example, if a particular resource like RAM or ROM or even network

Software-Testing_Final.indb 191 31-01-2018 [Link]

 192 • Software Testing

is perceived as a bottleneck then instrumented code can help identify

where the bottlenecks are and point towards possible solutions.
5. White-box testing can help identify security holes in dynamically
generated code. For example, in case of Java, some intermediate code
may also be generated. Testing this intermediate code requires code
knowledge. This is done by white-box testing only.

SUMMARY
1. White-box testing can cover the following issues:
a. Memory leaks
b. Uninitialized memory
c. Garbage collection issues (in JAVA)
2. We must know about white-box testing tools also. They are listed below:
a. Purify by Rational Software Corporation
b. Insure++ by ParaSoft Corporation
c. Quantify by Rational Software Corporation
d. Expeditor by OneRealm Inc.

MULTIPLE CHOICE QUESTIONS

1. A testing which checks the internal logic of the program is
a. Black-box testing. b. White-box testing.
c. Both (a) and (b) d. None of the above.
2. The cyclomatic complexity, V(G) was developed by:
a. Howard b. McCabe
c. Boehm d. None of the above.
3. A node with indegree = 0 and outdegree ≠ 0 is called
a. Source node. b. Destination node.
c. Transfer node. d. None of the above.

Software-Testing_Final.indb 192 31-01-2018 [Link]

 White-Box (or Structural) Testing Techniques • 193

4. A node with indegree ≠ 0 and outdegree = 0 is called

a. Source node.
b. Destination node.
c. Predicate node.
d. None of the above.
5. V(G) is given by which formula?
a. V(G) = e – n + 2
b. V(G) = e – 2n + P
c. V(G) = e – 2n
d. None of the above.
6. A predicate node is one which has
a. Two outgoing edges.
b. No outgoing edges.
c. Three or more outgoing edges.
d. None of the above.
7. An independent path is one
a. That is a complete path from source to destination node.
b. That is a path which introduces at least one new set of processing
statements.
c. T
 hat is a path which introduces at most one new set of processing
statements.
d. None of the above.
8. The size of the graph matrix is the
a. Number of edges in the flow graph.
b. Number of nodes in the flow graph.
c. Number of paths in the flow graph.
d. Number of independent paths.

Software-Testing_Final.indb 193 31-01-2018 [Link]

 194 • Software Testing

9. In data flow testing, the objective is to find

a. All dc-paths that are not du-paths.
b. All du-paths.
c. All du-paths that are not dc-paths.
d. All dc-paths.
10. Mutation testing is related to
a. Fault seeding.
b. Functional testing.
c. Fault checking.
d. None of the above.

ANSWERS

1. b. 2. b. 3. a. 4. b.
5. a. 6. a. 7. b. 8. b.
9. c. 10. a.

CONCEPTUAL SHORT QUESTIONS WITH ANSWERS

Q. 1. Write a short paragraph on test coverage analyzers?
Ans. Coverage analyzers are a class of test tools that offer automated sup-
port for this approach to testing management. With this tool, the tes-
ter runs a set of test cases on a program that has been instrumented
by the coverage analyzer. The analyzer then uses the information
produced by the instrumentation code to generate a coverage report.
For example, in case of DD-path coverage, it identifies and labels all
DD-paths in the original program. When the instrumented program
is executed with test cases, the analyzer tabulates the DD-paths tra-
versed by each test case. So, a tester can experiment with different
sets of test cases to determine the coverage of each set.
Q. 2. What is slice-based testing?
Ans. A program slice is a set of program statements that contribute to or
affect a value for a variable at some point in the program. This is an
informal definition.

Software-Testing_Final.indb 194 31-01-2018 [Link]

 White-Box (or Structural) Testing Techniques • 195

Let us see more formal definitions of a program slice.

“Given a program, P and a set of V of variables in P, a slice on the
variable set V at statement n, written as S(V, n), is the set of all state-
ments in P that contribute to the values of variables in V.”
Note: Listing elements in a slice S(V, n) will be cumbersome because
the elements are the program statement fragments.
We can further refine the definition of a program slice.
“Given a program P and a program graph G(P) in which statements
and statement fragments are numbered and a set V of variables in
P, the slice on the variable set V at statement fragment n, written as
S(V, n), is the set of node numbers of all statement fragments in P
prior to n that contribute to the values of variables in V at statement
fragment n.”
The basic idea is to separate a program into components (slices) that
have some useful/functional meaning.
Please note that we have used the words “prior to” (in the 2nd defi-
nition) in the dynamic sense. So, a slice captures the execution time
behavior of a program with respect to the variables in the slice. This
develops a lattice, or a directed, acyclic graph of slices in which
nodes are slices and edges correspond to the subset relationship.
We have also used a word, “contribute,” which means that data
­declaration statements have an effect on the value of a variable. We
simply exclude all non-executable statements.
We have five forms of usages:
1.  P-use — used in a predicate (decision)
2.  C-use — used in computation
3.  O-use — used for output
4.  L-use — used for location (pointers, subscripts)
5.  I-use — used for iterations (counters, loops)
We also identify two forms of definitions nodes:
1.  I-def: defined by input
2.  A-def: defined by assignment
Now assume that the slice S(v, n) is a slice on one variable, i.e., the
set V consists of a single variable, v. If the statement fragment “n”
is a defining node for v, then n is included in the slice. Also, if the

Software-Testing_Final.indb 195 31-01-2018 [Link]

 196 • Software Testing

statement fragment “n” is a usage node for v, then n is not included

in the slice. P-uses and C-uses of other variables, i.e., not the v in the
slice set V, are included to the extent that their execution affects the
value of variable, v.
Tip: If the value of v is the same whether a statement fragment is
included or excluded, then exclude the statement fragment.
Also, O-use, L-use, and I-use nodes are excluded from the slices
as L-use and I-use variables are typically invisible outside their
­modules.
Q. 3. What is slice splicing?
Ans. If we decide to develop a program in terms of compatible slices then
we could code a slice and immediately test it. We can then code and
test other slices and merge them into a fair solid program. This is
known as slice splicing.
Q. 4. What are the different sources of knowledge for white-box testing?
Ans. The following are the knowledge sources for white-box or structural
testing:
a. High-level design
b. Detailed design
c. Control flow graphs
Q. 5. What is stress testing?
Ans. Stress testing is a testing technique used to determine if the system
can function when subjected to large volumes of data. It includes
areas like
a. Input transactions
b. Internal tables
c. Disk space
d. Output
e. Communications
f. Computer capacity
g. Interaction with users
If the application functions properly with stressed data then it is
assumed that it will function properly with normal volumes of work.
Objectives of stress testing: To simulate a production environ-
ment for determining that
a. Normal or above-normal volumes of transactions can be pro-
cessed through the transaction within the available time frame.

Software-Testing_Final.indb 196 31-01-2018 [Link]

 White-Box (or Structural) Testing Techniques • 197

b. The system is able to process large volumes of data.

c. Enough system capacity is available.
d. Users can perform their assigned tasks.
Q. 6. How to use stress testing?
Ans. Online systems should be stress tested by allowing people to enter
transactions of a normal or above normal pace.
Batch systems can be stress tested with large input batches. Please
note that the error conditions should be included in tested transac-
tions. Transactions that are used in stress testing can be obtained
from test data generators, test transactions created by the test
group, or the transactions previously processed in the production
­environment.
In stress testing, the system should be run as it would in the pro-
duction environment. Operators should use standard documenta-
tion and the people entering transactions or working with the system
should be the clerical personnel that will use the system.
Examples of stress testing:
i. Enter transactions that determine that sufficient disk space has
been allocated to the application.
ii. Overloading the communication network with transactions.
iii. Testing system overflow conditions by entering more transac-
tions that can be accommodated by tables, queues, and internal
storage facilities, etc.
Q. 7. What is execution testing? Give some examples.
Ans. Execution testing is used to determine whether the system can meet
the specific performance criteria. It includes the following:
a. Verifying the optimum use of hardware and software.
b. Determining the response time to online user requests.
c. Determining transaction processing turnaround time.
Execution testing can be done in any phase of SDLC. It can evaluate
a single aspect of the system like a critical routine in the system. We
can use hardware and software monitors or create quick and dirty
programs to evaluate the approximate performance of a completed
system. This testing may be executed onsite or offsite for the perfor-
mance of the test. Please note that the earlier the technique is used,
the higher is the assurance that the completed application will meet
the performance criteria.

Software-Testing_Final.indb 197 31-01-2018 [Link]

 198 • Software Testing

Q. 8. What is recovery testing? Give some examples.

Ans. Recovery testing is used to ensure that operations can be continued
even after a disaster.
It not only verifies the recovery process but also the effectiveness of
the component parts of that process. Its objectives are:
a. Document recovery procedures
b. Preserve adequate backup data
c. Training recovery personnel
Examples of recovery testing:
i. Inducing failures into one of the application system programs
during processing.
ii. Recovery could be conducted from a known point of integrity to
ensure that the available backup data was adequate for the recov-
ery process.
Q. 9. What is operations testing? Give some examples.
Ans. Operations testing is designed to determine whether the system is
executable during normal system operations. It evaluates both the
process and the execution of the process. During different phases of
SDLC, it can be used as follows:
Phase 1: Requirements Phase
During this phase, operational requirements can be evaluated to
determine the reasonableness and completeness of these require-
ments.
Phase 2: Design Phase
During this phase, the operating procedures should be designed and
thus can be evaluated.
Examples of operations testing:
i. Verifying that the file labeling and protection procedures func-
tion properly.
ii. Determining that the OS supports features and performs the
predetermined tasks.
Q. 10. What is security testing?
Ans. A testing used to identify defects that are very difficult to identify. It
involves determining that adequate attention is devoted to identify-
ing security risks and determining that sufficient expertise exists to
perform adequate security testing.

Software-Testing_Final.indb 198 31-01-2018 [Link]

 White-Box (or Structural) Testing Techniques • 199

Examples of security testing:

i. Determining that the resources being protected are identified.
ii. Unauthorized access on online systems to ensure that the system
can identify such accesses.
Q. 11. Draw a flowchart to show overall mutation testing process.
Ans.

FIGURE 4.34

Q. 12. How is FTR different from the management review?

Ans. We tabulate the differences between the two.

Software-Testing_Final.indb 199 31-01-2018 [Link]

 200 • Software Testing

FTR Management review

1. It is done to examine the product. 1. It is done to evaluate a project plan.
2. Its purpose is to evaluate 2. Its purpose is to ensure adequacy
the ­software elements like and completeness of each plan-
­requirements and design. ning ­document for meeting project
requirements.
3. It is done to ensure conformity to 3. It is done to ensure that project
specifications. ­activities are progressing per the
­planning documents.
4. The results of FTRs are given in 4. The results of the management
technical review reports (TRR). reviews are summarized in a
­management review report (MRR).
5. It is done to ensure the 5. The purpose here is to ensure proper
­integrity of changes to the allocation of resources.
­software elements.

REVIEW QUESTIONS
1. White-box testing is complementary to black-box testing, not alternative.
Why? Give an example to prove this statement.
2. a. What is a flow graph and what it is used for?
b. Explain the type of testing done using flow graph?
3. Perform the following:
a. Write a program to determine whether a number is even or odd.
b. Draw the paths graph and flow graph for the above problem.
c. Determine the independent path for the above.
4. Why is exhaustive testing not possible?
5. a. Draw the flow graph of a binary search routine and find its independent
paths.

Software-Testing_Final.indb 200 31-01-2018 [Link]

 White-Box (or Structural) Testing Techniques • 201

b. How do you measure

1. Test effectiveness?
2. Test efficiency?
6. Write short paragraphs on:
a. Mutation testing.
b. Error-oriented testing.
7. Differentiate between structural and functional testing in detail.
8. Will exhaustive testing guarantee that the program is 100% correct?
9. a. What is cyclomatic complexity? How can we relate this to independent
paths?
b. Explain the usefulness of error guessing-testing technique.
10. Differentiate between functional and structural testing.
11. Discuss the limitations of structural testing. Why do we say that complete
testing is impossible?
12. Explain define/use testing. Consider any example and show du-paths.
Also identify those du-paths that are not dc-paths.
13. Find the cyclomatic complexity of the graph given below:

FIGURE 4.35

Find all independent paths.

Software-Testing_Final.indb 201 31-01-2018 [Link]

 202 • Software Testing

14. What is data flow testing? Explain du-paths. Identify du- and dc-paths of
any example of your choice. Show those du-paths that are not dc-paths.
15. Write a short paragraph on data flow testing.
16. Explain the usefulness of error guessing testing technique.
17. Discuss the pros and cons of structural testing.
18. a. What are the problems faced during path testing? How they can be
minimized?
b. Given the source code below:
void foo (int a, b, c, d, e) {
if (a = = 0) {
return;
}
int x = 0;
if ((a = = b) or (c = = d)) {
x = 1;
}
e = 1/x;
}

List the test cases for statement coverage, branch coverage, and condition
coverage.
19. Why is error seeding performed? How it is different from mutation
testing?
20. a. Describe all methods to calculate the cyclomatic complexity.
b. What is the use of graph matrices?
21. Write a program to calculate the average of 10 numbers. Using data flow
testing design all du- paths and dc-paths in this program.
22. Write a short paragraph on mutation testing.
23. Write a C/C++ program to multiply two matrices. Try to take care of as
many valid and invalid conditions are possible. Identify the test data.
Justify.

Software-Testing_Final.indb 202 31-01-2018 [Link]

 White-Box (or Structural) Testing Techniques • 203

24. Discuss the negative effects of the following constructs from the white-
box testing point of view:
a. GO TO statements
b. Global variables
25. Write a C/C++ program to count the number of characters, blanks, and
tabs in a line. Perform the following:
a. Draw its flow graph.
b. Draw its DD-paths graph.
c. Find its V(G).
d. Identify du-paths.
e. Identify dc-paths.
26. Write the independent paths in the following DD-path graph.
Also calculate mathematically. Also name the decision nodes shown in
Figure 4.36.
27. What are the properties of cyclomatic complexity?

FIGURE 4.36

Software-Testing_Final.indb 203 31-01-2018 [Link]

 204 • Software Testing

28. Explain in detail the process to ensure the correctness of data flow in a
given fragment of code.
main
{
int K = 35, Z;
Z = check (K);
printf (“\n%d”, Z);
}
check (m)
{
int m;
if (m > 40)
return (1);
else
return (0);
}

29. Write a C program for finding the maximum and minimum out of three
numbers and compute its cyclomatic complexity using all possible
methods.
30. Consider the following program segment:
void sort (int a[], int n)
{
int i, j;
for (i = 1; i < n; i++)
for (j = i + 1, j < n; j++)
if (a[i] > a[j])
{
temp = a[i];
a[i] = a[j];
a[j] = temp;
}
i. Draw the control flow graph for this program segment.
ii. Determine the cyclomatic complexity for this program (give all
­intermediate steps).
iii. How is cyclomatic complexity metric useful?

Software-Testing_Final.indb 204 31-01-2018 [Link]

 White-Box (or Structural) Testing Techniques • 205

31. Explain data flow testing. Consider an example and show all “du” paths.
Also identify those “du” paths that are not “dc” paths.
32. Consider a program to find the roots of a quadratic equation. Its input is
a triplet of three positive integers (say a, b, c) from the interval [1, 100].
The output may be one of the following words—real roots, equal roots,
imaginary roots. Find all du-paths and those that are dc-paths. Develop
data flow test cases.
33. If the pseudocode below were a programming language, list the test
cases (separately) required to achieve 100% statement coverage and
path coverage.
1. If x = 3 then
2. Display_message x;
3. If y = 2 then
4. Display_message y;
5. Else
6. Display_message z;
7. Else
8. Display_message z;
34. Consider a program to classify a triangle. Draw its flow graph and DD-
path graph.

Software-Testing_Final.indb 205 31-01-2018 [Link]

Software-Testing_Final.indb 206 31-01-2018 [Link]
 CHAPTER

5
Gray-Box Testing
Inside this Chapter:
5.0. Introduction to Gray-Box Testing
5.1. What Is Gray-Box Testing?
5.2. Various Other Definitions of Gray-Box Testing
5.3. Comparison of White-Box, Black-Box, and Gray-Box Testing
Approaches in Tabular Form

5.0.  INTRODUCTION TO GRAY-BOX TESTING

Code coverage testing involves “dynamic testing” methods of executing the
product with pre-written test cases and finding out how much of the code
has been covered. If a better coverage of the code is desired, several itera-
tions of testing may be required. For each iteration, one has to write a new
set of test cases for covering those portions of code that were not covered
by earlier test cases. To do such types of testing, not only does one need to
understand the code and logic but one also needs to understand how to write
effective test cases that can cover good portions of the code. Understand-
ing of code and logic means white-box or structural testing whereas writing
effective test cases means black-box testing. So, we need a combination of
white-box and black-box techniques for test effectiveness. This type of test-
ing is known as “gray-box testing.” We must then understand that:
WHITE + BLACK = GRAY

Software-Testing_Final.indb 207 31-01-2018 [Link]

 208 • Software Testing

5.1.  WHAT IS GRAY-BOX TESTING?

Black-box testing focuses on software’s external attributes and behavior.
Such testing looks at an application’s expected behavior from the user’s point
of view. White-box testing/glass-box testing, however, tests software with
knowledge of internal data structures, physical logic flow, and architecutre at
the source code level. White-box testing looks at testing from the developer’s
point of view. Both black-box and white-box testing are critically important
complements of a complete testing effort. Individually, they do not allow
for balanced testing. Black-box testing can be less effective at uncovering
certain error types such as data-flow errors or boundary condition errors at
the source level. White-box testing does not readily highlight macro-level
quality risks in operating environment, compatibility, time-related errors,
and usability.

5.2. VARIOUS OTHER DEFINITIONS OF

GRAY-BOX TESTING
Gray-box testing incorporates the elements of both black-box and white-box
testing. It considers the outcome on the user end, system-specific t­ echnical
knowledge, and the operating environment. It evaluates the application
design in the context of the inter-operability of system components. The
gray-box testing approach is integral to the effective testing of web appli-
cations that comprise numerous components, both software and hardware.
These components must be tested in the context of system design to ­evaluate
their functionality and compatibility. Listed below are some more definitions
of gray-box testing:
“Gray-box testing consists of methods and tools derived from the knowledge
of the application internals and the environment with which it interacts,
that can be applied in black-box testing to enhance testing productivity, bug
­finding and bug analyzing efficiency.”
—Nguyen H.Q.
OR
“Gray-box testing is using inferred or incomplete structural or design
­information to expand or focus black-box testing.”
—Dick Bender
OR

Software-Testing_Final.indb 208 31-01-2018 [Link]

 Gray-Box Testing • 209

“Gray-box testing is designing of the test cases based on the knowledge of

algorithms, interval states, architectures or other high level descriptions of
program behavior.”
—Dong Hoffman
OR
“Gray-box testing involves inputs and outputs, but test design is educated by
information about the code or the program operation of a kind that would
normally be out of scope of view of the tester.”
—Cem Kaner
Gray-box testing is well suited for web application testing because it factors
in high level design environment and the inter-operability conditions. It will
serve problems that are not as easily considered by a black-box or white-box
analysis, especially problems of end-to-end information flow and distributed
hardware/software system configuration and compatibility. Context-specific
errors that are germane to web systems are commonly uncovered in this
process.

5.3. COMPARISON OF WHITE-BOX, BLACK-BOX, AND

GRAY-BOX TESTING APPROACHES IN TABULAR FORM
Before tabulating the differences between black-box, gray-box, and white-
box testing techniques, we must first understand that when we say test
­granularity, we mean the level of details. And when we say the highest, it
means that all internals are known.

FIGURE 5.1  Comparison of Black-Box, Gray-Box, and White-Box Techniques.

Software-Testing_Final.indb 209 31-01-2018 [Link]

 210 • Software Testing

Next, we tabulate the points of differences between them.

Black-box testing Gray-box testing White-box testing

  1. Low granularity Medium granularity High granularity
  2.  Internals NOT known Internals partially known Internals fully known
  3. Internals not required Internals relevant to testing Internal code of the application and
to be known are known database known
  4. Also known as Also known as translucent- Also known as
• Opaque-box testing box testing • Glass-box testing
• Closed-box testing • Clear-box testing
• Input-output testing • Design-based testing
• Data-driven testing • Logic-based testing
• Behavioral • Structural testing
• Functional testing • Code-based testing.
  5. It is done by end- It is done by end-users (user Normally done by testers and
users (user acceptance acceptance testing). Also developers.
testing). Also done by done by testers, developers.
tester, developers.
  6. Testing method where Here, internals partly Internals fully known
• System is viewed as a known (gray), but not fully
black-box known (white). Test design
• Internal behavior is based on the knowledge
of the program is of algorithm, interval states,
ignored architecture, or other high
• Testing is based upon level descriptions of the
external specifications program behavior.
  7. It is likely to be least It is somewhere in between. Potentially most exhaustive of the
exhaustive of the three. three.
   8. Requirements based. Better variety/depth in test Ability to exercise code with
Test cases based cases on account of high level relevant variety of data.
on the functional knowledge of internals.
specifications, as
internals not known.
  9. Being specification It would also not suffer from Because test cases are written based
based if would not the deficiency as described on the code, specifications missed in
suffer from the for white-box testing. coding would not be revealed.
deficiency as described
for white-box testing.
10. It is suited for It is suited for functional/ It is suited for all.
functional/ business business domain testing bit
domain testing. in depth.
(continued)

Software-Testing_Final.indb 210 31-01-2018 [Link]

 Gray-Box Testing • 211

Black-box testing Gray-box testing White-box testing

11. Not suited to algorithm Not suited to algorithm Appropriate for algorithm testing.
testing. testing.
12. It is concerned with Here in, we have a better It facilitates structural testing. It
validating outputs variety of inputs and the enables logic coverage, coverage of
for given inputs, the ability to extract test results statements, decisions, conditions,
application being from database for comparison path, and control flow within the
treated as a black-box. with expected results. code.
13. It can test only by trial It can test data domains, It can determine and therefore
and error data domains, internal boundaries, and test better: data domains, internal
internal boundaries, and overflow, if known. boundaries, and overflow.
overflow.

SUMMARY

1. As testers, we get ideas for test cases from a wide range of knowl-
edge areas. This is partially because testing is much more effective
when we know what types of bugs we are looking for. As testers of
complex systems, we should strive to attain a broad balance in our
knowledge, learning enough about many aspects of the software and
systems being tested to create a battery of tests that can challenge
the software as deeply as it will be challenged in the rough and tum-
ble day-to-day use.
2. Every tester in a test team need not be a gray-box tester. More is the
mix of different types of testers in a team, better is the success.

MULTIPLE CHOICE QUESTIONS

1. When both black-box and white-box testing strategies are required

to test a software then we can apply
a. Gray-box testing.
b. Mutation testing.
c. Regression testing.
d. None of the above.

Software-Testing_Final.indb 211 31-01-2018 [Link]

 212 • Software Testing

2. Which of the following is true?

a. There is no such testing named gray-box testing.
b. Gray-box testing is well suited for web applications.
c. Gray-box is same as white-box only.
d. None of the above.
3. Gray-box testing is also known as
a. Opaque-box testing. b. Clear-box testing.
c. Translucent testing. d. None of the above.
4. Gray-box testing involves
a. Low granularity. b. Medium granularity.
c. High granularity. d. None of the above.
5. Gray-box testing is done by
a. End users. b. Testers and developers.
c. Both (a) and (b) d. None of the above.

ANSWERS
1.  a.    2.  b.    3.  c.    4.  b.    5.  c.

CONCEPTUAL SHORT QUESTIONS WITH ANSWERS

Q. 1. G
 ray-box testing is based on requirement-based test case g­ eneration.
Explain.
Ans. Gray-box testing uses assertion methods to preset all the conditions
required, prior to a program being tested. Formal testing is one of
the commonly used techniques for ensuring that a core program is
correct to a very large degree. If the requirement specification lan-
guage is being used to specify the requirement, it would be easy to
execute the requirement stated and verify its correctness. Gray-box
testing will use the predicate and verifications defined in require-
ment specification language as inputs to the requirements based test
case generation phase.

Software-Testing_Final.indb 212 31-01-2018 [Link]

 Gray-Box Testing • 213

Q. 2. W hy is gray-box testing especially useful for web and Internet

­applications?
Ans. It is useful because the Internet is built around loosely integrated
components that connected via relatively well-defined interfaces.
It factors in high-level design, environment and inter-operability
conditions. It reveals problems of end-to-end information flow
and distributed hardware/software system configuration and com-
patibility. Please note that context specific errors are germane to
web systems and it is our gray-box testing only that uncovers these
errors.
Q. 3. Can we have only gray-box testers?
Ans. No, because a high level of success lies in the fact that we can have
a mix of different types of testers with different skill sets—like data-
base expert, security expert, test automation expert, etc.
Q. 4. What is the gray-box assumption for object-oriented software?
Ans. Typically, a reference state model for SUT is assumed. The t­esting
problem is to identify failure to conform to the reference model.
The tester must determine which state is actually obtained by
applying a distinguishing sequence and observing the resulting
­output. This increases the number of tests by a large amount. With
object-­oriented software, we assume that the internal state can be
­determined by
a. A method of activation state where activation is equated with state.
b. State reporting capability in the class-under-test (CUT).
c. Built-in test reporting in the CUT.
This is the Gray-Box assumption.

REVIEW QUESTIONS

1. Define “gray-box testing.”

2. Give various other definitions of gray-box testing.
3. Compare white-box, black-box and gray-box testing approaches?
4. How is gray-box testing related to white- and black-box testing?

Software-Testing_Final.indb 213 31-01-2018 [Link]

 214 • Software Testing

5. Gray-box testing is well suited for web applications testing. Why?

6. Assume that you have to build the real-time large database
­connectivity system software. Then what kinds of testing do you
think are suitable? Give a brief outline and justification for any four
kinds of testing.

Software-Testing_Final.indb 214 31-01-2018 [Link]

 CHAPTER

6
Reducing the Number of
Test Cases
Inside this Chapter:
6.0. Prioritization Guidelines
6.1. Priority Category Scheme
6.2. Risk Analysis
6.3. Regression Testing—Overview
6.4. Prioritization of Test Cases for Regression Testing
6.5. Regression Testing Technique—A Case Study
6.6. Slice Based Testing

6.0.  PRIORITIZATION GUIDELINES

The goal of prioritization of test cases is to reduce the set of test cases based
on some rational, non-arbitrary, criteria, while aiming to select the most
appropriate tests. If we prioritize the test cases, we run the risk that some of
the application features will not be tested at all.
The prioritization schemes basically address these key concepts:
a. What features must be tested?
b. What are the consequences if some features are not tested?
The main guide for selecting test cases is to assess the risk first. At the end of
the test case prioritization exercise, the tester and the project authority must
feel confident with the tests that have been selected for execution. If some-
one is distressed about the omitted tests, then re-evaluate the list and apply
another prioritization scheme to analyze the application from another point
of view. Ideally, the project authority and possibly other project members,
must buy-in—and sign off—to the prioritized set of tests.

Software-Testing_Final.indb 215 31-01-2018 [Link]

 216 • Software Testing

There are four schemes that are used for prioritizing the existing set test
cases. These reduction schemes are as follows:
1. Priority category scheme
2. Risk analysis
3. Interviewing to find out problematic areas
4. Combination schemes
All of these reduction methods are independent. No one method is bet-
ter than the other. One method may be used in conjunction with another
one. It raises confidence when different prioritization schemes yield similar
conclusions.
We will discuss these techniques now.

6.1.  PRIORITY CATEGORY SCHEME

The simplest scheme for categorizing tests is to assign a priority code directly
to each test description. One possible scheme is a three-level priority catego-
rization scheme. It is given below:
Priority 1: This test must be executed.
Priority 2: If time permits, execute this test.
Priority 3: If this test is not executed, the team won’t be upset.
So, assigning a priority code is as simple as writing a number adjacent
to each test description. Once priority codes have been assigned, the tester
estimates the amount of time required to execute the tests selected in each
category. If the time estimate falls within the allotted schedule, then the
partitioning exercise is completed and you have identified the tests to use.
Otherwise, further partitioning is required.
The second scheme can be a new five-level scale to further classify the
tests. This scheme is given below:
Priority 1a: This test must pass or the delivery will be out of date.
Priority 2a: Before final delivery, this test must be executed.
Priority 3a: If time permits, execute this test.
Priority 4a: This test can wait after the delivery date.
Priority 5a: We will probably never execute this test.
We try to divide the tests. For example, say tests from priority 2 are now
divided between priorities 3a, 4a, and 5a, we can downgrade or upgrade any
test in the similar fashion.

Software-Testing_Final.indb 216 31-01-2018 [Link]

 Reducing the Number of Test Cases • 217

6.2.  RISK ANALYSIS

All software projects benefit from risk analysis. Even non-critical software,
using risk analysis at the beginning of a project highlights the potential prob-
lem areas. This helps developers and managers to mitigate the risks. The
tester uses the results of risk analysis to select the most crucial tests.

How Risk Analysis Is Done?

Risk analysis is a well-defined process that prioritizes modules for testing.
A risk contains three components:
1. The risk, ri, associated with a project (i ← 1 to n).
2. The probability of occurrence of a risk, (li).
3. The impact of the risk, (xi).
Risk analysis consists of first listing the potential problems and then
assigning a probability and severity value for each identified problem. By
ranking the results, the tester can identify the potential problems most in
need of immediate attention and select test cases to address those needs.
During risk analysis we draw a table as shown below:

Risk
Problem Probability of Impact of exposure
ID Potential problem (ri) occurrence (li) risk (xi) = li * xi

A Loss of power 1 10 10
B Corrupt file header 2 1 2
C Unauthorized access 6 8 48
D Databases not 3 5 15
synchronized
E Unclear user 9 1 9
documentation
F Lost sales 1 8 8
G Slow throughput 5 3 15
: : : : :
: : : : :
: : : : :
FIGURE 6.1  Risk Analysis Table (RAT).

Software-Testing_Final.indb 217 31-01-2018 [Link]

 218 • Software Testing

Here, in Figure 6.1,

Problem ID. It is a unique identifier associated with a risk.
Potential problem. It is a brief description of the problem.
Probability of occurrence, (li). It is a probability value on a scale of 1 (low)
to 10 (high).
Severity of impact, (xi). It is a severity value on a scale of 1 (low) to 10 (high).
Risk exposure. It is defined as the product of li and xi.
In this table, the values of li and xi range from 1 to 10. Multiplying the
probability and severity values yields the risk exposure. “The higher the
risk exposure product, the more important it is to test for that condition.”
­Applying this rule to Figure 6.1, we will get the following rankings of the
potential risk problems based on the product of risk-exposure. The order of
­preference is:
C—D—G—A—E—F—B
Although problems D and G have the same risk exposure, they differ by
their probability and severity values.
For some organizations, this method may produce enough information.
Others may wish to do risk-matrix analysis also. We will now discuss risk
matrix analysis.

What Is a Risk Matrix?

A risk matrix allows the tester to evaluate and rank potential problems by
giving more weight to the probability or severity value as necessary. The soft-
ware tester uses the risk matrix to assign thresholds that classify the potential
problems into priority categories.
There are four main methods of forming risk matrices:
Method I: Typically, the risk matrix contains four quadrants, as shown in
Figure 6.2.
Each quadrant in Figure 6.2 represents a priority class defined as
follows:
Priority 1: High severity and high probability
Priority 2: High severity and low probability
Priority 3: Low severity and high probability
Priority 4: Low severity and low probability

Software-Testing_Final.indb 218 31-01-2018 [Link]

 Reducing the Number of Test Cases • 219

FIGURE 6.2  Method I. FIGURE 6.3  Method II.

We can see from the graph of Figure 6.2 that a risk with high severity is
deemed more important than a problem with high probability. Thus, all risks
mapped in the upper-left quadrant fall into priority 2.
For example, the risk-e which has a high probability of occurrence but a
low severity of impact is put under priority 3.
Method II: For an entirely different application, we may swap the defini-
tions of priorities 2 and 3, as shown in Figure 6.3.
An organization favoring Figure 6.3 seeks to minimize the total number
of defects by focusing on problems with a high probability of occurrence.
Dividing a risk matrix into quadrants is most common, testers can deter-
mine the thresholds using different types of boundaries based on application
specific needs.
Method III: Diagonal band prioritiza-
tion scheme.
If severity and probability tend to
be equal weight, i.e., if li = xi, then diag-
onal band prioritization scheme may
be more appropriate. This is shown in
Figure 6.4.
This threshold pattern is a com-
promise for those who have difficulty
in selecting between priority-2 and
­priority-3 in the quadrant scheme. FIGURE 6.4  Method III.

Software-Testing_Final.indb 219 31-01-2018 [Link]

 220 • Software Testing

Method IV: The problems with

high severity must be given the top
priority, irrespective of the value of
probability. This problem is solved
with method-IV and is shown in
Figure 6.5. The remainder of the
risk matrix is partitioned into sev-
eral lower priorities, either as
quadrants (Method-I and -II) or as
diagonal bands (Method-III).

FIGURE 6.5  Method IV.

6.3.  REGRESSION TESTING—OVERVIEW

Regression testing is done to ensure that enhancements or defect fixes made
to the software works properly and does not affect the existing functionality.
It is usually done during maintenance phase.
As a software system ages, the cost of maintaining the software dom-
inates the overall cost of developing the software. Regression testing is a
testing process that is used to determine if a modified program still meets its
specifications or if new errors have been introduced. Improvements in the
regression testing process would help reduce the cost of software.

6.3.1.  Differences between Regression and Normal Testing

Let us now compare normal and regression testing in the below Table.

Normal testing Regression testing

1. It is usually done during fourth phase 1. It is done during the maintenance
of SDLC. phase.
2. It is basically software’s verification 2. It is also called program
and validation. ­revalidation.
3. New test suites are used to test our 3. Both old and new test cases can be
code. used for testing.
4. It is done on the original software. 4. It is done on modified software.
5. It is cheaper. 5. It is a costlier test plan.

Software-Testing_Final.indb 220 31-01-2018 [Link]

 Reducing the Number of Test Cases • 221

6.3.2. Types of Regression Testing

Four types of regression testing techniques are discussed one-by-one. They are
1. Corrective regression testing
2. Progressive regression testing
3. Retest-all regression testing
4. Selective regression testing

[Link].  Corrective regression testing

Corrective regression testing applies when specifications are unmodified
and test cases can be reused.

[Link].  Progressive regression testing

Progressive regression testing applies when specifications are modified and
new test cases must be designed.

[Link].  The retest-all strategy

The retest-all strategy reuses all tests, but this strategy may waste time and
resources due to the execution of unnecessary tests. When the change to a
system is minor, this strategy would be wasteful.

[Link].  The selective strategy

The selective strategy uses a subset of the existing test cases to reduce the
retesting cost. In this strategy, a test unit must be rerun if and only if any of the
program entities, e.g., functions, variables etc., it covers have been changed.
The challenge is to identify the dependencies between a test case and the
program entities it covers. Rothermel and Harrold specified a typical selective
regression testing process. In general, the process includes the following steps:
Step 1. Identify affected software components after program, P, has been
modified to P′.
Step 2. Select a subset of test cases, T′, from an existing test suite, T, that
covers the software components that are affected by the modifica-
tion.
Step 3. Test modified program P′ with T′ to establish the correctness of P′
with respect to T′.

Software-Testing_Final.indb 221 31-01-2018 [Link]

 222 • Software Testing

Step 4.  Examine test results to identify failures.

Step 5. Identify and correct the fault(s) that caused a failure.
Step 6.  Update the test suite and test history for P′.
From these steps, we can observe the following characteristics of selective
regression testing:
1. Identifying the program components that must be retested and find-
ing those existing tests that must be rerun are essential.
2. When selected test cases satisfy retest criterion, new test cases are
not needed.
3. Once regression testing is done, it is necessary to update and store
the test information for reuse at a later time.
Regression testing is one kind of testing that is applied at all three ­levels
of testing. White suggested applying specification based (system) testing
before structure based (unit) testing to get more test case reuse because
­system testing cases could be reused in unit testing but the converse is
not true. McCarthy suggested applying regression unit testing first to find
faults early.

[Link]. Regression testing at unit level

Unit testing is the process of testing each software module to ensure that
its performance meets its specifications. Yau and Kishimoto developed a
method based on the input partition strategy. McCarthy provided a way to
automate unit regression testing.
Gupta, Harrold, and Soffa proposed an approach to data flow based
regression testing.

[Link]. Regression testing at integration level

Integration testing is the testing applied when individual modules are
­combined to form larger working units until the entire program is cre-
ated. Integration testing detects failures that were not discovered during
unit ­testing. Integration testing is important because approximately 40% of
­software errors can be traced to module integration problems discovered
during integration testing.
Leung and White introduced the firewall concept to regression testing at
the integration level. A firewall is used to separate the set of modules affected

Software-Testing_Final.indb 222 31-01-2018 [Link]

 Reducing the Number of Test Cases • 223

by program changes from the rest of the code. The modules enclosed in the
firewall could be those that interact with the modified modules or those that
are direct ancestors or direct descendants of the modified modules.
The firewall concept is simple and easy to use, especially when the
change to a program is small. By retesting only the modules and interfaces
inside the firewall, the cost of regression integration testing can be reduced.

[Link]. Regression testing at system level

System testing is testing of the entire software system against the system
specifications. It must verify that all system elements have been properly
integrated and perform allocated functions. It can be performed without the
knowledge of the software implementation at all.
Test Tube is a system developed at AT&T Bell laboratories to perform
system level regression testing. Test Tube is an example of a selective retest-
ing technique. Test Tube partitions a software system into basic code enti-
ties, then monitors the execution of a test case, analyzes its relationship with
the system under test, and determines which subset of the code entities the
test covers. There are plans to extend Test Tube to non-deterministic sys-
tems such as real-time telecommunications software.

[Link]. Regression testing of global variables

A global variable is an output parameter in a module where the variable is
defined and an input parameter for a module that uses the variable. The
global variables can be retested at the unit level by running the test that exer-
cise the changed code and the instruction referencing the global variable.
A global variable can be retested at the integration level. If any of its defining
modules have been changed then all of its using modules must be retested. The
regression testing of global variables is very time consuming and, therefore, costly.

[Link].  Comparison of various regression testing techniques

There are four criteria that form a framework for evaluation of different
selective regression testing techniques. They are discussed below:
1. Inclusiveness: It measures the extent to which a technique chooses
tests that will expose faults caused by program changes.
2. Precision: It measures the ability of a technique to avoid choosing
tests that will not reveal the faults caused by the changes.

Software-Testing_Final.indb 223 31-01-2018 [Link]

 224 • Software Testing

3. Efficiency: It measures the computational cost of a technique.

4. Generality: It measures the ability of a technique to handle different
language constructs and testing applications.

[Link]. Regression testing in object-oriented software

Object-oriented concepts such as inheritance and polymorphism present
unique problems in maintenance and, thus, regression testing of object-­
oriented programs. Several regression testing techniques have been extended
to retesting of object-oriented software.
Rothermal and Harrold extended the concept of selective regression
testing to object-oriented environments. Their algorithm constructs pro-
gram or class dependence graphs and uses them to determine which test
cases to select. To test a graph, driver programs are developed to invoke the
methods in a class in different sequences. The class dependence graph links
all these driver programs together by selecting one driver as the root of the
graph and adding edges from it to the public methods in the class. Now the
methods can be invoked in different sequences.
Abdullah and White extended the firewall concept to retesting object-
oriented software. A firewall is an imaginary boundary that encloses the
entities that must be retested in a modified program. Unlike the firewall in
a procedure-based program, the firewall in an object-oriented program is
constructed in terms of classes and objects. When a change is made to a class
or an object, the other classes or objects that interact with this changed class
or object must be enclosed in the firewall. Because the relations between
classes and objects are different, there should be different ways to construct
the firewalls for classes and objects.

6.4. PRIORITIZATION OF TEST CASES FOR

REGRESSION TESTING
Prioritization of tests requires a suitable cost criterion. Please understand
that tests with lower costs are placed at the top while those with higher costs
at the bottom.

What Cost Criterion to Use?

We could use multiple criteria to prioritize tests. Also, note that the tests
being prioritized are the ones selected using some test selection technique.
Thus, each test is expected to traverse at some modified portion of P′.

Software-Testing_Final.indb 224 31-01-2018 [Link]

 Reducing the Number of Test Cases • 225

Prioritization of regression tests offers a tester an opportunity to decide

how many and which tests to run under time constraints. When all tests
­cannot be run, one needs to find some criteria to decide when to stop t­ esting.
This decision could depend on a variety of factors such as:
a. Time constraints
b. Test criticality
c. Customer requirements

6.5.  REGRESSION TESTING TECHNIQUE—A CASE STUDY

Regression testing is used to confirm that fixed bugs have been fixed and
that new bugs have not been introduced. How many cycles of regression
testing are required will depend upon the project size. Cycles of regression
testing may be performed once per milestone or once per build. Regression
tests can be automated.

The Regression–Test Process

The regression-test process is shown in Figure 6.6.

FIGURE 6.6

This process assumes that P′ (modified is program) available for regres-

sion testing. There is a long series of tasks that lead to P′ from P.
Test minimization ignores redundant tests. For example, if both t1
and t2, test function, f in P, then one might decide to reject t2 in favor of t1.
The purpose of minimization is to reduce the number of tests to execute for
regression testing.
Test prioritization means prioritizing tests based on some criteria.
A  set of prioritized tests becomes useful when only a subset of tests can

Software-Testing_Final.indb 225 31-01-2018 [Link]

 226 • Software Testing

be executed due to resource constraints. Test selection can be achieved by

selecting a few tests from a prioritized list. The possible sequence to execute
these tasks is:

Test setup means the process by which AUT (application under test) is
placed in its intended or simulated environment and is ready to receive data
and output the required information. Test setup becomes more challenging
when we test embedded software like in ATMs, printers, mobiles, etc.
The sequence in which tests are input to an application is an important
issue. Test sequencing is very important for applications that have an inter-
nal state and runs continuously. For example, an online-banking software.
We then execute the test cases. Each test needs verification. This can
also be done automatically with the help of CASE tools. These tools com-
pare the expected and observed outputs. Some of the tools are:
a. Test Tube (by AT&T Bell Labs.) in 1994: This tool can do
­selective retesting of functions. It supports C.
b. Echelon (by Microsoft) in 2002: No selective retesting but does
test prioritization. It uses basic blocks to test. It supports C and
binary languages.
c. ATACLx Suds (by Telcordia Technologies) in 1992: It does
selective retesting. It allows test prioritization and minimization. It
does control/data flow average. It also supports C.

6.6.  SLICE-BASED TESTING

Static slicing may lead to an unduly large program slice. So, Korel and Laski
proposed a method for obtaining dynamic slices from program executions.
They used a method to extract executable and smaller slices and to allow

Software-Testing_Final.indb 226 31-01-2018 [Link]

 Reducing the Number of Test Cases • 227

more precise handling of arrays and other structures. So, we discuss dynamic
slicing.
Let “P” be the program under test and “t” be a test case against which P
has been executed. Let “l” be a location in P where variable v is used. Now,
the dynamic slice of P with respect to “t” and “v” is the set of statements in P
that lie in trace (t) and did effect the value of “v” at “l.” So, the dynamic slice
is empty if location “l” was not traversed during this execution. Please note
that the notion of a dynamic slice grew out of that of a static slice based on
program “P” and not on its execution.
Let us solve an example now.
EXAMPLE 6.1. Consider the following program:
1. main ( ) {
2. int p, q, r, z;
3. z = 0;
4. read (p, q, r);
5. if (p < q)
6. z = 1; //modified z
7. if (r < 1)
8. x = 2
9. output (z);
10. end
11. }
Test case (t1): <p = 1, q = 3, r = 2>. What will be
the dynamic slice of P with respect to variable “z” at line
9? What will be its static slice? What can you infer? If
t2: <p = 1, q = 0, r = 0> then what will be dynamic and FIGURE 6.7
static slices?
SOLUTION. Let us draw its flow graph first shown in Figure 6.7.
\ Dynamic slice (P) with respect to variable z at line 9 is td = <4, 5, 7, 8>
Static slice, ts = <3, 4, 5, 6, 7, 8>

Dynamic slice for any variable is generally smaller than the corresponding
NOTE
static slice.

Now, t2: <p = 1, q = 0, r = 0>

\ Its dynamic slice is statements (3, 4, 5) while the static slice does not
change.

Software-Testing_Final.indb 227 31-01-2018 [Link]

 228 • Software Testing

Dynamic slice contains all statements in trace (t) that had an effect on
NOTE
program output.

Inferences mode:
1. A dynamic slice can be constructed based on any program variable
that is used at some location in P, the program that is being modified.
2. Some programs may have several locations and variables of ­interest at
which to compute the dynamic slice, then we need to compute slices
of all such variables at their corresponding locations and then take
union of all slices to create a combined dynamic slice. This approach
is useful for regression testing of relatively small ­components.
3. If a program is large then a tester needs to find out the critical loca-
tions that contain one or more variables of interest. Then, we can
build dynamic slices on these variables.

SUMMARY
Regression testing is used to confirm that fixed bugs have, in fact, been fixed
and that new bugs have not been introduced in the process and that features
that were proven correctly functional are intact. Depending on the size of a
project, cycles of regression testing may be performed once per milestone
or once per build. Some bug regression testing may also be performed dur-
ing each acceptance test cycle, focusing on only the most important bugs.
Regression tests can be automated.

MULTIPLE CHOICE QUESTIONS

1. The main guide for selecting test cases is

a. To assess risks.
b. To assess quality.
c. Both (a) and (b)
d. None of the above.

Software-Testing_Final.indb 228 31-01-2018 [Link]

 Reducing the Number of Test Cases • 229

2. Which of the following is not a risk reduction scheme?

a. Priority category scheme
b. Risk analysis
c. Interviewing
d. None of the above.
3. The potential problems are identified and their probabilities and
impacts are given weights in which technique.
a. Priority categorization scheme
b. Risk analysis
c. Interviews
d. None of the above.
4. Risk exposure is given by which formula.
a. It is the product of probability of occurrence of risk and its impact
b. It is the sum of probability of its occurrence and its impact
c. It is the standard deviation of the sum of its probability and its
impact
d. None of the above.
5. A risk matrix
a. Allows the testers to evaluate and rank potential problems by
­giving weights to proabability or severity value as necessary.
b. Allows testers to assign thresholds that classify the potential
­problems into priority categories.
c. Both (a) and (b).
d. None of the above.
6. In diagonal band prioritization scheme
a. Severity equals probability.
b. Severity is never equal to probability.
c. Either (a) or (b).
d. Both (a) and (b).

Software-Testing_Final.indb 229 31-01-2018 [Link]

 230 • Software Testing

7. Some managers found out that

a. Probability of risk to occur is very important.
b. Problems with high severity must be given top priority.
c. Severity of risk is not at all important.
d. None of the above.
8. Which one of the following is NOT a regression testing strategy?
a. Correction regression testing
b. Retest-all strategy
c. Combinational explosion
d. None of the above.
9. Which of the following testing strategy is applicable at all three
­levels of testing?
a. White-box testing
b. Mutation testing
c. Regression testing
d. None of the above.
10. What is used to separate a set of modules affected by program
changes from the rest of the code?
a. Firewall
b. NIC
c. Emulator
d. None of the above.

ANSWERS

1.  a.   2.  d. 3.  b. 4.  a.

5.  c.   6.  a. 7.  b. 8.  c.
9.  c. 10.  a.

Software-Testing_Final.indb 230 31-01-2018 [Link]

 Reducing the Number of Test Cases • 231

CONCEPTUAL SHORT QUESTIONS WITH ANSWERS

Q. 1. Explain the risk reduction method?
Ans. The formula for quantifying risk also explains how to control or
­minimize risk. The formula is as follows:
Loss Due to Risk = Frequency of Occurrence ×
Loss Per Occurrence
For example, say ten users a day terminated their business website.
The average customer places $80 orders and thus,
Loss due to unfriendly website is = 10 * $80 = $800
Therefore, once the variables and the loss expectation formula have
been defined, controls can then be identified to minimize that risk.
Q. 2. Can you list the most common risk factors for various project types
given below:
a. MIT
b. Commercial Software Sectors
Ans. a.  For MIS Projects
Risk Factors % of Projects at Risk
1.  Creeping user requirements 80%
2.  Excessive schedule pressure 65%
3.  Low quality 60%
4.  Cost overruns 55%
5.  Inadequate configuration control 50%
b.  For Commercial Software Sectors
Risk Factors % of Projects at Risk
1.  Inadequate user documentation 70%
2.  Low-user satisfaction 55%
3.  Excessive time to market 50%
4.  Harmful competitive actions 45%
5.  Litigation expense 30%
Q. 3. What is SEI-SRE service?
Ans. The Software Engineering Institute (SEI) defines a Software Risk
Evaluation (SRE) service.
SRE is a diagnostic and decision-making tool that enables the iden-
tification, analysis, tracking, mitigation, and communications of risks
in software-intensive programs. It is used to identify and categorize

Software-Testing_Final.indb 231 31-01-2018 [Link]

 232 • Software Testing

specific program risks arising from product, process, management,

resources, and constraints.
An SRE can be used for:
a. Preparing for a critical milestone.
b. Recovery from crisis.
An SRE is conducted by a trained SEI-program team. By imple-
menting SRE, the management improves its ability to ensure success
through the creation of a proactive risk management methodology.
Q. 4. Why is smoke testing done?
Ans. Smoke testing is an integration testing approach that constructs and
tests software on a daily basis. It consists of:
a. Finding out the basic functionality that a product must satisfy.
b. Designing test cases to ensure that the basic functionality work
and combining these test cases into a smoke test suite.
c. To ensure that everytime a product is built, this suite is run first.
d. If this suite fails then escalate to the developers to find out the
changes or to roll back the changes to a state where the smoke
test suite succeeds.
Q. 5. How can regression testing be done at the module level and at the
product level?
Ans. a. At a module level, it may involve retesting module-­execution-
paths (MEPs) traversing the modification.
b. At a product level, this activity may involve retesting functions
that execute the modified area.
The effectiveness of these strategies is highly dependent on the
­utilization of test matrices, which enable identification of coverage
provided by particular test cases.

Software-Testing_Final.indb 232 31-01-2018 [Link]

 Reducing the Number of Test Cases • 233

REVIEW QUESTIONS

1. What do you mean by regression testing? Discuss the different types

of regression testing.
2. Define and discuss the following:
a. Regression testing and how regression test selection is done.
b. Selective retest and coverage techniques.
c. Minimization and safe techniques.
3. Write a short paragraph on regression testing.
4. What is regression testing? Discuss the regression test selection
problem.
5. What is the importance of regression testing?
6. Explain how risks are prioritized.
7. How can we reduce the number of test cases?
8. See the following priority scheme for test cases:
Priority 0: Test cases that check basic functionality are run for
accepting the build for further testing, and are run when the prod-
uct undergoes a major change. These test cases deliver a very high
project value to development teams and customers.
Priority 1: Uses the basic and normal setup and these test cases
deliver high project value to both development teams and c­ ustomers.
Priority 2: These test cases deliver moderate project value. They
are executed as part of the testing cycle and selected for regression
testing on a need basis.
Using this scheme of prioritization, prioritize the following test cases
as P0 or P1 or P2:
a. A test case for DBMS software that tests all options of a select
query.
b. A test case for a file system that checks for deallocation of free
space.

Software-Testing_Final.indb 233 31-01-2018 [Link]

 234 • Software Testing

c. A test case that checks the functionality of a router/bridge.

d. A test case that tests the 0S boot strap process with normal
­parameters.
e. A test case that checks a login form of a website.
9. Explain in brief the reduction schemes on prioritizing the test cases
to reduce required testing effort? As a tester, how do you evaluate
and rank potential problems? Suggest some guidelines of your own
approach to reduce the number of test cases.
10. Explain how risk matrix can be used to prioritize the test cases.
Explain giving an example. Why do we need to prioritize the test
cases?
11. a. What are prioritization categories and guidelines for test case
reduction?
b. What is the need for regression testing? How it is done?
12. Suppose your company is about to roll out an e-commerce applica-
tion. It is not possible to test the application on all types of browsers
on all platforms and operating systems. What steps would you take
in the testing environment to reduce the testing process and all pos-
sible risks?
13. List and explain prioritization guidelines.
14. What is regression testing and the explain different types of regres-
sion testing? Compare various regression testing techniques.
15. a. What is the role of risk matrix for the reduction of test cases?
b. How is risk analysis used in testing? Explain the role of the risk
analysis table in testing.

Software-Testing_Final.indb 234 31-01-2018 [Link]

 CHAPTER

7
Levels of Testing
Inside this Chapter:
7.0. Introduction
7.1. Unit, Integration, System, and Acceptance Testing Relationship
7.2. Integration Testing

7.0. INTRODUCTION
When we talk of levels of testing, we are actually talking of three levels of testing:
1. Unit testing
2. Integration testing
3. System testing
The three levels of testing are shown in Figure 7.1.

FIGURE 7.1  Levels of Testing.

Generally, system testing is functional rather than structural testing.

We will now study these testing techniques one-by-one.

Software-Testing_Final.indb 235 31-01-2018 [Link]

 236 • Software Testing

7.1. UNIT, INTEGRATION, SYSTEM, AND ACCEPTANCE

TESTING RELATIONSHIP
We can categorize testing as follows:

FIGURE 7.2

We will explain unit or

­ odule testing next. Consider the
m
­following ­diagram of unit testing as
shown in Figure 7.3.
Unit (or module) testing “is
the process of taking a module (an
atomic unit) and running it in iso-
lation from the rest of the software FIGURE 7.3  Unit Testing.
product by using prepared test
cases and comparing the actual
results with the results predicted by the specification and design module.” It
is a white-box testing technique.
Importance of unit testing:
1. Because modules are being tested individually, testing becomes
easier.
2. It is more exhaustive.
3. Interface errors are eliminated.

Software-Testing_Final.indb 236 31-01-2018 [Link]

 Levels of Testing • 237

TEST is one of the CASE tools for unit testing (Parasoft) that automati-
cally tests classes written in [Link] framework. The tester need not write
a single test or a stub. There are tools which help to organize and execute
test suites at command line, API, or protocol level. Some examples of such
tools are:

S. No. Kind of tool Software description Platforms

1. Deja Gnu testing Framework designed UNIX machines
framework for for regression testing
interactive or batch- and embedded system
oriented applications. testing.
2. E-SIM-Embedded E-SIM is a native Win 32, Solaris 5,
software simulation ­simulator for and LINUX
and testing ­embedded software.
environment.
3. SMARTS- It is the software Sun OS, Solaris,
Automated Test maintenance and MS WINDOWS
Management Tool. regression test system. 95/98/NT/ 2000
SMARTs automates
and simplifies testing
process.

7.2.  INTEGRATION TESTING

A system is composed of multiple components or modules that comprise
hardware and software. Integration is defined as the set of interactions
among components. Testing the interaction between the modules and inter-
action with other systems externally is called integration testing.
It is both a type of testing and a phase of testing. The architecture and
design can give the details of interactions within systems; however, testing
the interactions between one system and another system requires a detailed
understanding of how they work together. This knowledge of integration
depends on many modules and systems. These diverse modules could have
different ways of working when integrated with other systems. This intro-
duces complexity in procedures and in what needs to be done recognizing
this complexity, a phase in testing is dedicated to test their interactions,
resulting in the evolution of a process. This ensuing phase is called the
­integration testing phase.

Software-Testing_Final.indb 237 31-01-2018 [Link]

 238 • Software Testing

7.2.1.  Classification of Integration Testing

As shown in Figure 7.4 integration testing is classified as follows:

- - -

FIGURE 7.4

We will discuss each of these techniques in the following sections.

7.2.2.  Decomposition-Based Integration

When we talk of decomposition-based integration testing techniques, we
usually talk of the functional decomposition of the system to be tested which
is represented as a tree or in textual form. It is further classified as top-down,
bottom-up, sandwich, and big bang integration strategies. All these integra-
tion orders presume that the units have been separately tested, thus, the goal
of decomposition-based integration is to test the interfaces among separately
tested units.

[Link]. Types of decomposition-based techniques top-down

integration approach

It begins with the main program, i.e., the root of the tree. Any lower-level
unit that is called by the main program appears as a “stub.” A stub is a piece
of throw-away code that emulates a called unit. Generally, testers have to
develop the stubs and some imagination is required. So, we draw.
Where “M” is the main program and “S”
represents a stub from the f­ igure, we find out
that:
Number of Stubs Required = ­
(Number of Nodes – 1)
Once all of the stubs for the main
­ rogram have been provided, we test the main
p
program as if it were a ­
­ standalone  unit. FIGURE 7.5  Stubs.

Software-Testing_Final.indb 238 31-01-2018 [Link]

 Levels of Testing • 239

We could apply any of the appropriate functional and s­ tructural ­techniques

and look for faults. When we are convinced that the main p ­ rogram logic
is correct, we gradually replace the stubs with the actual code. Top‑down
integration follows a breadth-first traversal (bfs) of the functional
­
­decomposition tree.

[Link]. Bottom-up integration approach

It is a mirror image to the top-down order with the difference that stubs are
replaced by driver modules that emulate units at the next level up in the
tree. In bottom-up integration, we start with the leaves of the decomposition
tree and test them with specially coded drivers. Less throw-away code exists
in drivers than there is in stubs. Recall that there is one stub for each child
node in the decomposition tree.
Most systems have a fairly high fan-out
near the leaves, so in the ­bottom-up integra-
tion order, we will not have as many drivers.
This is ­partially offset by the fact that the
driver modules will be more complicated.
See Figure 7.6 FIGURE 7.6  Drivers.
where “D” represents a driver module
and “M” represents other m ­ odules.
Number of drivers required = (No. of nodes – No. of leaf nodes)
For Figure 7.6, drivers required = 5 – 4 = 1,
i.e., 1 driver module (D) is required.

[Link]. Sandwich integration approach

It is a combination of top-down and bottom-up integration. There will be
less stub and driver development effort. But the problem is in the difficulty
of fault isolation that is a consequence of big bang integration.
Because this technique is the combination of the top-down and
­bottom-up integration approaches, it is also called bidirectional ­integration.
It is performed initially with the use of stubs and drivers. Drivers are used to
provide upstream connectivity while stubs provide downstream connectivity.
A driver is a function which redirects the request to some other component
and stubs simulate the behavior of a missing component. After the function-
ality of these integrated components are tested, the drivers and stubs are
discarded. This technique then focuses on those components which need
focus and are new. This approach is called as sandwich integration.

Software-Testing_Final.indb 239 31-01-2018 [Link]

 240 • Software Testing

In the product development phase when a transition happens from

t­ wo-tier architecture to three-tier architecture, the middle tier gets ­created as
a set of new components from the code taken from bottom-level a­ pplications
and top-level services.

[Link]. Big-bang integration
Instead of integrating component by component and testing, this approach
waits until all the components arrive and one round of integration testing is
done. This is known as big-bang integration. It reduces testing effort and
removes duplication in testing for the multi-step component integrations.
Big-bang integration is ideal for a product where the interfaces are stable
with fewer number of defects.

[Link].  Pros and cons of decomposition-based techniques

The decomposition-based approaches are clear (except big-bang integra-
tion). They are built with tested components. Whenever a failure is observed,
the most recently added unit is suspected. Integration testing progress is
easily tracked against the decomposition tree. The top-down and bottom-up
terms suggest breadth-first traversals of the decomposition tree but this is
not mandatory.
One common problem to functional decomposition is that they are
­artificial and serve the needs of project management more than the need of
software developers. This holds true also for decomposition-based testing.
The whole mechanism is that units are integrated with respect to structure.
This presumes that correct behavior follows from individually correct units
and correct interfaces.
The development effort for stubs and drivers is another drawback to
these approaches and this is compounded by the retesting effort. We try to
compute the number of integration test sessions for a given decomposition
tree. A test session is defined as one set of tests for a specific configuration
actual code and stubs. Mathematically,
Sessions = Nodes – Leaves + Edges
For example, if a system has 42 integration testing sessions then it means
42 separate sets of integration test cases, which is too high.

Software-Testing_Final.indb 240 31-01-2018 [Link]

 Levels of Testing • 241

[Link]. Guidelines to choose integration method and conclusions

S. No. Factors Suggested method

1. Clear requirements and design. Top-down approach.
2. Dynamically changing requirements, Bottom-up approach.
design, and architecture.
3. Changing architecture and stable Sandwich (or bi-directional)
design. approach.
4. Limited changes to existing Big-bang method.
architecture with less impact.
5. Combination of above. Select any one after proper analysis.

7.2.3.  Call Graph-Based Integration

One of the drawbacks of decomposition-based integration is that the basis
is the functional decomposition tree. But if we use the call graph-based
­technique instead, we can remove this problem. Also, we will move in the
direction of structural testing. Because call graph is a directed graph thus,
we can use it as a program graph also. This leads us to two new approaches
to integration testing which are discussed next.

[Link].  Pairwise integration

The main idea behind pairwise integration is to eliminate the stub/driver
development effort. Instead of developing stubs and drivers, why not use the
actual code? At first, this sounds like big-bang integration but we restrict a
session to only a pair of units in the call graph. The end result is that we have
one integration test ses-
sion for each edge in the
call graph. This is not
much of a reduction in
sessions from either top-
down or bottom-up but
it is a drastic reduction
in stub/driver develop-
ment. Four pairwise
integration sessions are
shown in Figure 7.7. FIGURE 7.7  Pairwise Integration.

Software-Testing_Final.indb 241 31-01-2018 [Link]

 242 • Software Testing

[Link]. Neighborhood Integration
The neighborhood of a node in a graph is the set of nodes that are one edge
away from the given node. In a directed graph, this includes all of the imme-
diate predecessor nodes and all of the immediate successor nodes. Please
note that these correspond to the set of stubs and drivers of the node.
For example, for node-16, neighborhood nodes are 9, 10, and 12 nodes
as successors and node-1 as predecessor node.
We can always compute the number of neighbors for a given call graph.
Each interior node will have one neighborhood plus one extra in case leaf
nodes are connected directly to the root node.

NOTE An interior node has a non-zero in-degree and a non-zero out-degree.

So, we have the following formulas:

1. Interior nodes = Nodes – (Source
Nodes + Sink Nodes)
2. Neighborhoods = Interior Nodes +
Source Nodes
Substituting 1st  equation in 2nd
equation above, we get:
Neighborhoods = Nodes – Sink Nodes
Neighborhood integration yields a
drastic reduction in the number of inte-
gration test sessions. It avoids stub and
driver development. The end result is Neighbors of
that the neighborhoods are essentially the Node-16
sandwiches that we slipped past in the FIGURE 7.8  Neighborhood
previous section. Integration.

[Link].  Pros and cons

The call graph-based integration techniques move away from a purely
­structural basis toward a behavioral basis. These techniques also elimi-
nate the stub/driver development effort. This technique matches well with
the developments characterized by builds. For example, sequences of the
­neighborhood can be used to define builds.

Software-Testing_Final.indb 242 31-01-2018 [Link]

 Levels of Testing • 243

The biggest drawback to call graph-based integration testing is the

fault isolation problem, especially for large neighborhoods. Another prob-
lem occurs when a fault is found in a node (unit) that appears in several
neighborhoods. Obviously, we resolve the fault but this means changing the
unit’s code in some way, which in turn means that all the previously tested
­neighborhoods that contain the changed node need to be retested.

7.2.4. Path-Based Integration with its Pros and Cons

We already know that the combination of structural and functional testing is
highly desirable at the unit level and it would be nice to have a similar capa-
bility for integration and system testing. Our goal for integration testing is:
“Instead of testing interfaces among separately developed and tested units,
we focus on interactions among these units.” Here, cofunctioning might be a
good term. Interfaces are structural whereas interaction is behavioral.
We now discuss some basic terminologies that are used in this technique.
1. Statement fragment. It is a complete statement. These are the
nodes in the program graph.
2. Source node. A source node in a program is a statement fragment
at which program execution begins or resumes. The first executable
statement in a unit is clearly a source node. Source nodes also occur
immediately after nodes that transfer control to other units.
3. Sink node. It is a statement fragment in a unit at which program
execution terminates. The final executable statement in a program
is clearly a sink node, so are the statements that transfer control to
other units.
4. Module execution path (MEP). It is a sequence of statements
that begins with a source node and ends with a sink node with no
intervening sink nodes.
5. Message. A message is a programming language mechanism by
which one unit transfers control to another unit. Depending on the
programming language, messages can be interpreted as subroutine
invocations, procedure calls, and function references. We follow the
convention that the unit that receives a message always eventually
returns control to the message source. Messages can pass data to
other units.

Software-Testing_Final.indb 243 31-01-2018 [Link]

 244 • Software Testing

6. Module-to-module path (MM-path). An MM-path is an inter-

leaved sequence of module execution paths (MEPs) and messages.
7. Module-to-module path graph (MM-path graph). Given a set
of units, their MM-path graph is the directed graph in which nodes
are module execution paths and edges correspond to messages and
returns from one unit to another. The effect of these definitions is
that program graphs now have multiple source and sink nodes. This
would increase the complexity of unit testing but the integration
testing presumes unit testing is complete. Also, now our goal is to
have an integration testing analog of DD-paths, as done earlier.
The basic idea of an MM-path is that we can now describe sequences
of module execution paths that include transfers of control among separate
units. These transfers are by messages, therefore, MM-paths always repre-
sent feasible execution paths and these paths cross unit boundaries. We can
find MM-paths in an extended program graph in which nodes are module
execution paths and edges are messages.
Consider a hypothetical example as shown in Figure 7.9.
Herein, module-A calls module-B, which in turn calls module-C. Note
from Figure 7.9 that MM-path begins with and ends in the main program
only. This is true for traditional (or procedural) software.

FIGURE 7.9  MM-Path Across Three Units (A, B, and C).

In module-A, nodes 1 and 5 are source nodes and nodes 4 and 6 are sink
nodes. Similarly, in module-B, nodes 1 and 3 are source nodes and nodes 2

Software-Testing_Final.indb 244 31-01-2018 [Link]

 Levels of Testing • 245

and 4 are sink nodes. Module-C has a single source node 1 and a single sink
node, 5. This can be shown as follows:

Module Source-node Sink-node

A 1, 5 4, 6
B 1, 3 2, 4
C 1 5
So, the seven module execution paths are as follows:
MEP (A, 1) = <1, 2, 3, 6>
MEP (A, 2) = <1, 2, 4>
MEP (A, 3) = <5, 6>
MEP (B, 1) = <1, 2>
MEP (B, 2) = <3, 4>
MEP (C, 1) = <1, 2, 4, 5>
MEP (C, 2) = <1, 3, 4, 5>
These are the module execu-
tion paths. We can now define an
integration testing analog of the
FIGURE 7.10  MM-Path Graph Derived
DD-path graph that serves unit
from Figure 7.9.
testing so effectively.
Now, its MM-path graph is shown in Figure 7.10.
Herein, the solid arrows indicate messages and the dotted arrows
­represent returns.
Also, note that a program path is a sequence of DD-paths and an
MM-path is a sequence of module execution paths.

What Are the Endpoints on MM-Paths?

Two criteria that are observed and are behavioral put endpoints on MM-paths:
1. Message quiescence
2. Data quiescence
Message quiescence occurs when a unit that sends no messages is
reached. For example, module-C in Figure 7.9.
Data quiescence occurs when a sequence of processing culminates
in the creation of stored data that is not immediately used. This happens
in a data flow diagram. Points of quiescence are natural endpoints for an
MM-path.

Software-Testing_Final.indb 245 31-01-2018 [Link]

 246 • Software Testing

Pros and Cons

1. MM-paths are a hybrid of functional (black-box) and structural
(white-box) testing. They are functional because they represent
actions with inputs and outputs. As such, all the functional testing
techniques are potentially applicable. The structural side comes
from how they are identified, particularly the MM-path graph.
The net result is that the cross-check of the functional and struc-
tural approaches is consolidated into the constructs for path-based
integration testing. We therefore avoid the pitfall of structural test-
ing and, at the same time, integration testing gains a fairly seamless
junction with system testing. Path-based integration testing works
equally well for software developed in the traditional waterfall
­process or with one of the composition-based alternative life-cycle
models. Later, we will also show that the concepts are equally appli-
cable to object-oriented software testing.
2. The most important advantage of path-based integration testing is
that it is closely coupled with the actual system behavior, instead
of the structural motivations of decomposition and call graph-based
integration.
3. The disadvantage of this technique is that more effort is needed to
identify the MM-paths. This effort is probably offset by the elimina-
tion of stub and driver development.

7.2.5. System Testing
System testing focuses on a complete, integrated system to evaluate compli-
ance with specified requirements. Tests are made on characteristics that are
only present when the entire system is run.

[Link]. What is system testing?

The testing that is conducted on the complete integrated products and
­solutions to evaluate system compliance with specified requirements on
functional and non functional aspects is called system testing. It is done after
unit, component, and integration testing phases.

Software-Testing_Final.indb 246 31-01-2018 [Link]

 Levels of Testing • 247

As we already know, a system is defined as a set of hardware, soft-

ware, and other parts that together provide product features and solutions.
In order to test the entire system, it is necessary to understand the product’s
behavior as a whole. System testing brings out issues that are fundamental to
the design, architecture, and code of the whole product.
System-level tests consist of batteries of tests that are designed to fully
exercise a program as a whole and check that all elements of the integrated
system function properly. System-level test suites also validate the useful-
ness of a program and compare end results against requirements.
System testing is the only testing phase that tests both functional and
non functional aspects of the product.
On the functional side, system testing focuses on real-life customer
usage of the product and solutions. It simulates customer deployments. For
a general-purpose product, system testing also means testing it for different
business verticals and applicable domains such as insurance, banking, asset
management, and so on.
On the non-functional side, it brings into consideration different testing
types which are also called quality factors.

[Link]. Why is system testing done?

System testing is done to:
1. Provide independent perspective in testing as the team becomes
more quality centric.
2. Bring in customer perspective in testing.
3. Provide a “fresh pair of eyes” to discover defects not found earlier
by testing.
4. Test product behavior in a holistic, complete, and realistic
­environment.
5. Test both functional and non functional aspects of the product.
6. Build confidence in the product.
7. Analyze and reduce the risk of releasing the product.
8. Ensure all requirements are met and ready the product for accept-
ance testing.
Explanation: An independent test team normally does system
­testing. This independent test team is different from the team that does

Software-Testing_Final.indb 247 31-01-2018 [Link]

 248 • Software Testing

the ­component and integration testing. The system test team generally
reports to a manager other than the product-manager to avoid conflicts
and to p­ rovide freedom to individuals during system testing. Testing the
product with an independent perspective and combining that with the
perspective of the customer makes system testing unique, different, and
effective.
The behavior of the complete product is verified during system testing.
Tests that refer to multiple modules, programs, and functionality are included
in system testing. This task is critical as it is wrong to believe that individually
tested components will work together when they are put together.
System testing is the last chance for the test team to find any leftover
defects before the product is handed over to the customer.
System testing strives to always achieve a balance between the objective
of finding defects and the objective of building confidence in the product
prior to release.
The analysis of defects and their classification into various categories
(called as impact analysis) also gives an idea about the kind of defects that
will be found by the customer after release. If the risk of the customers
getting exposed to the defects is high, then the defects are fixed before the
release or else the product is released as such. This information helps in
planning some activities such as providing workarounds, documentation on
alternative approaches, and so on. Hence, system testing helps in reducing
the risk of releasing a product.
System testing is highly complementary to other phases of testing. The
component and integration test phases are conducted taking inputs from
functional specification and design. The main focus during these testing
phases are technology and product implementation. On the other hand, cus-
tomer scenarios and usage patterns serve as the basis for system testing.

[Link]. Functional versus non functional system testing (in tabular form)

We are now in a position to state an equation:
System testing = Functional testing + Non functional testing

We first tabulate the differences between functional and non functional

testing in a tabular form.

Software-Testing_Final.indb 248 31-01-2018 [Link]

 Levels of Testing • 249

Functional testing Non functional testing

1. It involves the product’s 1. It involves the product’s quality
­functionality. ­factors.
2. Failures, here, occur due to code. 2. Failures occur due to either archi-
tecture, design, or due to code.
3. It is done during unit, component, 3. It is done in our system testing
integration, and system testing phase.
phase.
4. To do this type of testing only 4. To do this type of testing, we need
domain of the product is required. domain, design, architecture, and
product’s knowledge.
5. Configuration remains same for a 5. Test configuration is different for
test suite. each test suite.

Thus, functional testing helps in verifying what the system is supposed

to do. It aids in testing the product’s features or functionality. It has only two
results—requirements met or not met. It should have very clear expected
results documented in terms of the behavior of the product. It has simple
methods and steps to execute the test cases. Functional testing results nor-
mally depend on the product and not on the environment. It uses a pre-
determined set of resources. It requires in-depth customer, product, and
domain knowledge to develop different test cases and find critical defects. It
is performed in all phases of testing, i.e., unit, component, integration, and
system testing.
Non functional testing is performed to verify the quality factors such
as reliability, scalability, etc. These quality factors are also called non
­functional requirements. It requires the expected results to be documented
in ­qualitative and quantifiable terms. It requires a large amount of resources
and the results are different for different configurations and resources. It
is a very complex method as a large amount of data needs to be collected
and analyzed. The focus point is to qualify the product. It is not a defect
finding exercise. Test cases for non functional testing includes a clear pass/
fail ­criteria.
However, test results are concluded both on pass/fail definitions and on
the experiences encountered in running the tests. Non functional test results
are also determined by the amount of effort involved in executing them and
any problems faced during execution. For example, if a performance test
met the pass/fail criteria after 10 iterations, then the experience is bad and

Software-Testing_Final.indb 249 31-01-2018 [Link]

 250 • Software Testing

the test result cannot be taken as pass. Either the product or the non func-
tional testing process needs to be fixed here.
Non functional testing requires understanding the product behavior,
design, architecture, and also knowing what the competition provides. It also
requires analytical and statistical skills as the large amount of data generated
requires careful analysis. Failures in non functional testing affect the design
and architecture much more than the product code. Because non functional
testing is not repetitive in nature and requires a stable product, it is per-
formed in the system testing phase.
The differences listed in the table above are just the guidelines and not
the dogmatic rules.
Because both functional and non functional aspects are being tested in
the system testing phase so the question that arises is—what is the ratio of
the test-cases or effort required for the mix of these two types of testing? The
answer is here: Because functional testing is a focus area starting from the
unit testing phase while non functional aspects get tested only in the system
testing phase, it is a good idea that a majority of system testing effort be
focused on the non functional aspects. A 70%–30% ratio between non func-
tional and functional testing can be considered good and 50%–50% ratio is
a good starting point. However, this is only a guideline and the right ratio
depends more on the context, type of release, requirements, and products.

[Link]. Functional system testing techniques

As functional testing is performed at various testing phases, there are two
problems that arise. They are:
1. Duplication: It refers to the same tests being performed multiple
times.
2. Gray area: It refers to certain tests being missed out in all the
phases.
A small percentage of duplication across phases cannot be avoided as
different teams are involved performing cross-reviews (i.e., involving teams
from earlier phases of testing) and looking at the test cases of the previous
phase before writing system test cases can help in minimizing the duplica-
tion. However, a small percentage of duplication is advisable as then differ-
ent test teams will test the features with different perspectives thus yielding
new defects.
Gray-areas in testing happens due to a lack of product knowledge, lack
of knowledge of customer usage, and lack of coordination across test teams.
These areas (missing of tests) arise when a test team assumes that a p
­ articular

Software-Testing_Final.indb 250 31-01-2018 [Link]

 Levels of Testing • 251

test may be performed in the next phase. So, the guideline is—“A test case
moved from a later phase to an earlier phase is a better option than delaying
a test case from an earlier phase to a later phase, as the purpose of testing is
to find defects as early as possible.” This has to be done after completing all
tests meant for the current phase, without diluting the tests of the current
phase.
We are now in a position to discuss various functional system testing
techniques in detail. They are discussed one by one.

[Link].1.  Design/Architecture verification

We can compare functional testing with integration testing. They are given
in table below:

S. No. Integration testing System testing

1. The test cases are created by The test cases are created first
looking at interfaces. and verified with design and
architecture.
2. The integration test cases focus The functional system test
on interactions between mod- focuses on the behavior of the
ules or components. complete product.

In this method of system testing, the test cases are developed and
checked against the design and architecture to see whether they are actual
product-level test cases. This technique helps in validating the product
­features that are written based on customer scenarios and verifying them
using product implementation.
If there is a test case that is a customer scenario but failed validation
using this technique, then it is moved to the component or integration test-
ing phase. Because functional testing is performed at various test phases, it is
important to reject the test cases and move them to an earlier phase to catch
defects early and avoid any major surprise at later phases.
We now list certain guidelines that are used to reject test cases for ­system
functional testing. They are:
1. Is this test case focusing on code logic, data structures, and unit of
the product?
If yes, then it belongs to unit testing.
2. Is this specified in the functional specification of any component?
If yes, then it belongs to component testing.

Software-Testing_Final.indb 251 31-01-2018 [Link]

 252 • Software Testing

3. Is this specified in design and architecture specification for integra-

tion testing?
If yes, then it belongs to integration testing.
4. Is it focusing on product implementation but not visible to customers?
If yes, then it is focusing on implementation to be covered in unit/
component/integration testing.
5. Is it the right mix of customer usage and product implementation?
If yes, then it belongs to system testing.

[Link].2.  Business vertical testing (BVT)

Using and testing the product for different business verticals such as bank-
ing, insurance, asset management, etc. and verifying the business operations
and usage is called as business vertical testing. In this type of testing, the
procedure in the product is altered to suit the process in the business. For
example, in personal loan processing, the loan is approved first by the senior
officer and then sent to a clerk. User objects such as a clerk and officer are
created by the product and associated with the operations. This is one way of
customizing the product to suit the business. Some operations that can only
be done by some user objects is called a role-based operation. BVT involves
three aspects. They are:
1. Customization
2. Terminology
3. Syndication
Customization: It is important that the product understands the business
processes and includes customization as a feature so that different business
verticals can use the product. With the help of this feature, a general work-
flow of a system is altered to suit specific business verticals.
Terminology: To explain this concept, we consider a common example of
e-mail. When an e-mail is sent in a loan processing system than it is called
a loan application. An e-mail sent in the insurance context may be called a
claim and so on. The users would be familiar with this terminology rather
than the generic terminology of “e-mail.” So, the user interface should reflect
these terminologies rather than use generic terminology e-mails, which may
dilute the purpose and may not be understood clearly by the users. An e-mail
sent to a blood bank cannot take the same priority as an internal e-mail sent
to an employee by another employee. These differentiations need to be
made. Some e-mails need to be tracked. For example, an e-mail to a blood

Software-Testing_Final.indb 252 31-01-2018 [Link]

 Levels of Testing • 253

bank service needs a prompt reply. Some mail can be given automated mail
replies also. Hence, the terminology feature of the product should call the
e-mail appropriately as a claim or a transaction and also associate the profile
and properties in a way a particular business vertical works.
Syndication: Not all the work needed for business verticals is done by prod-
uct development organizations only. Even the solution integrators, service
providers pay a license fee to a product organization and sell the products
and solutions using their name and image. In this case, the product name,