Generating Test Cases -

Testing Techniques
Atul Gupta
 • the most commonly used method
for Software Validation
• executing the code under test
(CUT) with a set of test cases and
observing the behavior
• different from static approaches
Software • employed at different levels
• Unit
Testing • Integration
• System
• Acceptance Testing
• Testing strategies
• Specification-based (black-box)
• Code based (white-box)
• Specialized testing (testing NFRs)
2
Source for Test Generation?

S P

T
 Black-box testing White-box testing
Basic • Tests that validate
business
• Tests that validate
internal program logic
Testing requirements (what
the system is
(control flow , data
structures, data flow)
Strategies supposed to do)
• Test cases are • Test cases are derived
derived from the by examination of the
requirements internal structure of
specification of the the CUT.
CUT. No knowledge
of internal program
structure is used.
• Also known as -- • Also known as --
functional, data- structural or logic-
driven, or driven testing
Input/Output
testing
Generating Test Cases

Black-box Testing Methods

S P S P

T
T

Method A Method B
Generating Test Cases

White-box Testing Methods

S P S P

T
T

Method A Method B
 • Black box testing (Specification
Based)
• Equivalence Class Partitioning
Basic Testing • Boundary Value Analysis
Strategies • Cause-effect graphing
• Combinatorial Testing
• Model based Testing
• White box testing (Code Based)
• Control Flow Based
• Data Flow based
• Mutation Testing
Consider the following program
1. program triangle (int a, int b, int c) ;
2. IsATriangle : boolean;
3. BEGIN
4. readln (a,b,c);
5. IF (a < b + c) AND (b < a + c) AND (c < a + b)
6. THEN IsATriangle :=TRUE
7. ELSE IsATriangle := FALSE ;
8. IF IsATriangle
9. THEN IF (a = b) AND (b = c)
10. THEN Writeln ('Triangle is Equilateral') ;
11. ELSE IF (a <> b) AND (a <> c) AND (b <> c)
12. THEN Writeln ('Triangle is Scalene') ;
13. ELSE Writeln ('Triangle is Isosceles') ;
14. ELSE WRITELN('Not a Triangle') ;
15. END.
 • Partition the program input
domain into equivalence classes
Equivalence (according to the specifications)
• The rationale is that test of a
Class representative value of each
Partitioning class is equivalent to a test of
any other value of the same
class.
• identify valid as well as invalid
equivalence classes
• One test case from each
equivalence class
Equivalence Classes
x2

x1
a b c d
Equivalence Classes Testing
x2

x1
a b c d
Weak Normal Equivalence Class Testing
Equivalence Classes Testing
x2

x1
a b c d
Strong Normal Equivalence Class Testing
Equivalence Classes Testing
x2

x1
a b c d
Weak Robust Equivalence Class Testing
Equivalence Classes Testing
x2

x1
a b c d
Strong Robust Equivalence Class Testing
 input condition 0 <= x <= max

valid equivalence invalid equivalence

Equivalence class : 0 <= x classes : x < 0, x >
<= max max
Classes:

An Example

3 test cases
Identifying Equivalence Classes
Input Condition Valid Eq. Classes Invalid Eq. Classes

Range of values one valid two invalid

(eg. 1 - 200) (value within range)

A number N one valid two invalid values

(less and more than N)

Set of input values (Nominal) one valid Eq. class total one
each handled for each value (e.g. any value not
differently by the (total n) in valid input set )
program (e.g. A, B, C, … (n))

A Condition one one

(e.g. Id name must begin (e.g. it is a letter) (e.g. it is not a letter)
with a letter )
 • Assign a unique number to
each equivalence class
• Until all valid equivalence
Identifying classes have been covered
Test Cases by test cases, write a new
test case covering as many
for of the uncovered valid
Equivalence equivalence classes as
possible.
Classes • Each invalid equivalence
class cover by a separate
test case.
Boundary Value Analysis
• Generally combined with Equivalence Class
Partitioning
• Design test cases that exercise values that lie at
the boundaries of an input equivalence class.
• Also identify output equivalence classes, and
write test cases to generate o/p at the boundaries
of the output equivalence classes.

Example: input condition 0 <= x <= max

Test for values : 0,1, x, max-1, max ( valid inputs)
: -1, max+1 (invalid inputs)
Input Domain of a Function of two Variables
x2

x1
a b
BVA Test cases
x2

x1
a b
BVA Test cases
x2

x1
a b
Robustness Testing
BVA Test cases
x2

x1
a b
Worse-Case Testing
BVA Test cases
x2

x1
a b
Robust Worse-Case Testing
Cause Effect Graphing Technique
A technique that aids in selecting test cases for
combinations of input conditions in a systematic way
Steps:
1. Identify the causes (input conditions) and effects
(output conditions) of the program under test.
2. For each effect, identify the causes that can
produce that effect. Draw a Cause-Effect Graph.
3. Generate one test case for each combination of
input conditions that make some effect to be
true.
 Example
Consider a program with the following:
input conditions Output conditions
C1: command is credit E1: print invalid command
C2: command is debit E2: print invalid A/C
C3: A/C is valid E3: print debit amount not valid
C4: Transaction amount E4: debit A/C
not valid E5: credit A/C
 Example: Cause-Effect Graph
input conditions Output conditions
C1: command is credit E1: print invalid command
C2: command is debit E2: print invalid A/C
C3: A/C is valid E3: print debit amount not valid
C4: Transaction amount E4: debit A/C not and E1
not valid E5: credit A/C
C1
not and E2
C2
and and E3

C3
and E5
and
C4
and and E4
Example
The Decision table showing the combinations of input
conditions that make an effect true. (summarized from Cause
Effect Graph)
Write test cases to exercise each Rule in decision Table.

Example: C1 0 1 - - 1
C2 0 - 1 1 -
C3 - 0 1 1 1
C4 - - 0 0 0

E1 1
E2 1
E3 1
E4 1
E5 1
White Box Testing

White-box Test case design techniques

Control-Flow Testing Data-Flow Testing

Statement coverage All p-use
Decision coverage
All c-use
Condition coverage
Decision-Condition All d-use
coverage All uses
MC/DC
Multi-condition coverage Mutation Testing
Basis Path Testing
Loop testing
White Box Testing
The program structure used in structural testing is the CFG i.e.
control flow graph
1
1 read x,y;
2 z:=1; 2
3 while not y=0 do
4 if x mod 2 =1 3 end
5 z:=z*x 4
6 y:=y/2
5
7 x:=x*x
end Nodes - represent statements
67 Edges - represent the flow
 • Statement Coverage (SC)
• write enough test cases to
White Box execute every
statement at least once
Test-Case
Design TER (Test Effectiveness Ratio)

TER (SC) = statement coverage

overage achieved

# 𝑠𝑡𝑎𝑡𝑒𝑚𝑒𝑛𝑡 𝑒𝑥𝑒𝑟𝑐𝑖𝑠𝑒𝑑
=
𝑡𝑜𝑡𝑎𝑙 𝑛𝑜 𝑜𝑓 𝑠𝑡𝑎𝑡𝑒𝑚𝑒𝑛𝑡
 void function eval (int A, int B, int X )
{
if ( A > 1) and ( B = 0 ) then
X = X / A;
Example if ( A = 2 ) or ( X > 1) then
X = X + 1;
}

Statement Coverage test cases:

1) A = 2, B = 0, X = ? ( X can be
assigned any value)
 Branch Vs. Statement

• To have statement
coverage is sufficient one
Branch coverage
testcase (1).
• For branch coverage we
need two testcases (2) and (2)
(3).
(3)
• Suppose that instead of
S1
putting the edge (3) in * Statement
we put (3) in ** (error!);
branch coverage reveals * Coverage
the failure, statement S2 (1)
coverage no! error

**
Branch Coverage implementation
To determine whether or not each branch is traversed, we can place a
‘counter’ on each branch...
read x, y

‘Probe’ added Count[1] := 1

‘Program instrumented’

z := 1

where not … Count[3] := 1 exit

N.B. At the beginning Count is an

Count[2] := 1
array where each element is set to 0.

09/11/2023 35
Code instrumentation to get
Coverage Data
• Now we are ready to test the program instrumented for a set of testcases.
• To reach complete coverage we need a set of testcases such as
this condition is verified:

Count1 or Count2 or … or Countn = (1, 1, 1, …, 1)

Countj = vector ‘Count’ after execution of testcase j

or = bit-or
Example:
(1, 0, 1) or (1, 0, 0) = (1, 0, 1)

09/11/2023 36
 1
1 read x,y; Count[1]=1
Example 2 z:=1;
CFG
2
3 while not y=0 do
4 if x mod 2 =1 Count[3]=1
Count[2]=1 3 exit
5 z:=z*x
Count[4]=1 4
6 y:=y/2
5 Count[5]=1
7 x:=x*x
end
67

testcases counter values

x y count[1] count[2] count[3] count[4] count[5]

10 0 1 0 1 0 0
20 1 1 1 1 1 0

1 1 1 1 0
- The branch ‘5’ is not traversed.
- We have to find a new testcase that will force traversal of this
branch (ex. x=5, y=2)
 • Decision coverage (Branch coverage)
• write test cases to exercise the true
White Box and false outcomes of every
decision
Test-Case TER = branches exercised / total
branches
Design • Condition coverage (Predicate
coverage)
• write test cases such that each
condition in a decision takes on all
possible outcomes at least once
• may not always satisfy decision
coverage
Example
void function eval (int A, int B, int X )
{
if ( A > 1) and ( B = 0 ) then
X = X / A;
if ( A = 2 ) or ( X > 1) then
X = X + 1;
}

Decision Coverage (DC) test cases:

1) A = 3, B = 0, X = 1 (acd)
2) A = 2, B = 1, X = ? (abe)
Example
• Condition Coverage (CC)
test cases must cover
conditions
A>1, A<=1, B=0, B !=0
A=2, A !=2, X >1, X<=1

• Test cases:
1) A = 1, B = 0, X = 3 (abe)
2) A = 2, B = 1, X = 1 (abe)

• CC does not satisfy decision

coverage (DC) here
 • Decision Condition coverage
• write test cases such that each
White Box condition in a decision takes on
all possible outcomes at least
Test-Case once and each decision takes on
all possible outcomes at least
Design once
• Multiple Condition coverage (Full
Predicate)
• write test cases to exercise all
possible combinations of True
and False outcomes of conditions
within a decision
Example
• Decision Condition (DC)
coverage test cases must
cover conditions
A>1, A<=1, B=0, B !=0
A=2, A !=2, X >1, X<=1
and
also ( A > 1 and B = 0) T, F
( A = 2 or X > 1) T, F
• Test cases:
1) A = 2, B = 0, X = 4 (ace)
2) A = 1, B = 1, X = 1 (abd)
Example
• Multiple Condition coverage
must cover conditions
1) A >1, B =0 5) A=2, X>1
2) A >1, B !=0 6) A=2, X <=1
3) A<=1, B=0 7) A!=2, X > 1
4) A <=1, B!=0 8) A !=2, X<=1

• Test cases:
1) A = 2, B = 0, X = 4 (covers 1,5)
2) A = 2, B = 1, X = 1 (covers 2,6)
3) A = 1, B = 0, X = 2 (covers 3,7)
4) A = 1, B = 1, X = 1 (covers 4,8)
Basis Path Testing
1. Draw the control flow graph (CFG) of the
program using the program design
specification or code

2. Compute the Cyclomatic Complexity (CC)

V(G) of the CFG using any of the formulas:
V(G) = #Edges - #Nodes + 2
or V(G) = #regions in flow graph
or V(G) = #predicates + 1
or V(G) = # linearly independent paths
 V(G) = 6 regions
CC: An Example
V(G) = #Edges - #Nodes
+2
= 17 - 13 + 2 = 6
V(G) = 5 predicate-nodes
+1=6
6 linearly
independent paths
 3. Determine a basis set of linearly
independent paths.
Basis Path 4. Prepare test cases that will force
execution of each path in the
Testing Basis set.
(contd)
The value of Cyclomatic
complexity provides an upper
bound on the number of tests
that must be designed to
guarantee coverage of all
program statements.
 • Aims to expose bugs in loops
• Fundamental Loop Test criteria
1) bypass the loop altogether
Loop Testing 2) one pass through the loop
3) two passes through the loop
before exiting
4) a typical number of passes
through the loop, unless
covered by some other test
5) max number of passes (loop
exit condition in far-loop)
 • Select test paths of a
program based on the
Definition-Use (DU) chain
Data Flow of variables in the program.
• All definitions
Testing • All-c-uses
• All-p-uses
• all-uses
• all-du-paths
• Write test cases to cover
every DU path in at least
once.
Dataflow-based testing
•A def-use graph is constructed from the
control flow graph
•A statement in the control flow graph (in
which each statement is a node) can be of
these types
• def: represents definition of a var (i.e. when var
is on the lhs)
• c-use: computational use of a var
• p-use: var used in a predicate for control transfer

50
 • A def-use graph is constructed by
associating vars with nodes and edges
in the control flow graph
Data flow • For a node I, def(i) is the set of vars
for which there is a global def in I
based… • For a node I, C-use(i) is the set of
vars for which there is a global c-use
in I
• For an edge, p-use(I,j) is set of vars
for which there is a p-use for the
edge (I,j)
• Def clear path from I to j wrt x: if no def
of x in the nodes in the path
51
Testing
Dataflow based criteria
• all-defs: for every node I, and every x in def(i) there
is a def-clear path
• For def of every var, one of its uses (p-use or c-use) must
be tested
• all-p-uses: all p-uses of all the definitions should be
tested
• All p-uses of all the defs must be tested
• Some-c-uses, all-c-uses, some-p-uses are some
other criteria

Testing 52
An Example
Relationship between different criteria

Testing 55
Black-Box • Black box testing can detect errors
such as
☺ incorrect functions, missing
functions
vs.  Cannot detect design errors,
coding errors, unreachable code,
hidden functions

White-Box • White box testing can detect errors

such as
☺ logic errors, design errors
Testing  Cannot detect whether the
program is performing its expected
functions, missing functionality.

Both methods of testing are required.

 • Exhaustive Black-box
testing is generally not
Is Complete possible because the input
Testing domain for a program may
be infinite or incredibly large.
Possible ?
• Exhaustive White-box
testing is generally not
possible because a
program usually has a very
large number of paths.
 both criteria are NOT good !

When to Stop • Stop when the

Testing ? scheduled time for
testing expires
• Stop when all the test
cases execute without
detecting errors
 1. Make use of Coverage
based Testing
Better Test
Completion Example: Test cases derived
Criteria from
1) satisfying multi-
condition coverage and
2) boundary-value
analysis and
3) cause-effect graphing
 State the completion criteria
in terms of number of errors
to be found. This requires:
Better Test
Completion • An estimate of total number of
Criteria errors in the program
• An estimate of the % of errors
that can be found through testing
• An estimates of what fraction of
errors originate in particular
design processes, and during what
phases of testing they get
detected.
 Better Test Completion Criteria
 Plot the number of errors found per unit time during a test phase.
 The rate of error detection falls below a specified threshold

40- 40-

# Errors found
# Errors found

30- 30-
20- 20-
10- 10-
1 2 3 4 5 6 1 2 3 4 5 6
Week Week
Testing Principles (1)
--- Glen Myers

• A good test case is one likely to show an error.

• Description of expected output or result is an
essential and crucial part of the test-case
definition.
• A programmer should avoid attempting to test
his/her own program.
• Testing is more effective and successful if
performed by an Independent Test Team.
Testing Principles (2)
--- Glen Myers

• Avoid on-the-fly testing. Document all test

cases.
• Test valid as well as invalid cases.
• Thoroughly inspect all test results.
• More detected errors implies even more errors
present in the code.
Testing Principles (3)
--- Glen Myers

• Decide in advance when to stop testing

• Do not plan testing effort under the tacit
assumption that no errors will be found.
• Testing is an extremely creative and
intellectually challenging task.
What is a good test?
 • Testing is the most popular
way of validating the software
• Exhaustive testing is generally
not possible nor feasible
• Testing Strategies
• White-box
Summary • Black-box
• Both strategies are required
for a comprehensive testing of
software
• Testing can show the presence
of bugs, but not their absence
• Testing is very creative and
intellectually challenging task
References

• Beizer, Boris (1990). Software Testing Techniques (Second Edition ed.)

• Myers, Glenford J. (1979). The Art of Software Testing. John Wiley
and Sons
• Wikipedia [Link]