Page 1
How to measure test effectiveness
using DDP (Defect Detection Percentage)
Prepared by
Grove Consultants
Dorothy Graham
Mark Fewster
Lloyd Roden
Clive Bates
Grove House
40 Ryles Park Road
Macclesfield
SK11 8AH UK
Llwyncynhwyra
Cwmdu
Llandeilo
SA19 7EW UK
95 Stonebridge
Orton Malborne
Peterborough
PE2 5NT UK
2 Nursery Close
Hilperton Village
Wiltshire
BA14 7RP UK
Tel +44 1625 616279
[email protected]Tel +44 1558 685180
[email protected]Tel +44 1733 239183
[email protected]Tel +44 1225 761670
[email protected]www.grove.co.uk
Grove Consultants
Contents
Introduction: what is DDP
Examples of calculating DDP
Case studies
Advice about DDP & Conclusion
These slides are based on the DDP Advanced Workshop
at the EuroSTAR Conference in December 2003.
Grove Consultants, 2003
DDP031227
www.grove.co.uk
011 44 8702 406172
Page 2
What is the value of testing?
objectives
- to find defects
- to give confidence
useful,
- to assess risk
easy
to do
measures
- defect-based measures
found (Defect Detection Percentage)
fixed (Defect Fix Percentage)
not covered here
but can be useful
- confidence-based measures
(plus other metrics)
- risk-based measures
When NOT to use DDP
when you dont have many defects
- in test or in production (i.e. very high quality software)
your defect tracking is immature, purely subjective,
untrustworthy, or non-existent
the software products you produce
- are never used by anyone (no live running)
- it doesnt matter how many defects are in them
it is impossible to get data on defects found in live
running
- (difficult is OK!)
youre not interested in improving
Grove Consultants, 2003
DDP031227
www.grove.co.uk
011 44 8702 406172
Page 3
How effective are we at finding faults?
faults (defects) found in testing
or
faults (defects) found after testing
Defects found
in testing
Start
Defects found
afterwards
Release
Not found
- yet
Benchmark point
Defect Detection Percentage (DDP)
Defects found by this testing
Total defects including those found afterwards
"this" testing could be
- a test stage, e.g. component, integration, acceptance,
regression, etc.
- all testing for a function or subsystem
- all testing for a system
Grove Consultants, 2003
DDP031227
www.grove.co.uk
011 44 8702 406172
Page 4
Effectiveness at finding defects
Defects
Found
100%
90%
80%
70%
60%
DDP
50%
40%
30%
20%
10%
0%
50
45
40
35
30
25
20
15
10
5
0
Release
Time
Defects found
Defects found
Total defects
50
10 after testing: 38
12
19
24
27
35
37
0
50
62
69
74
77
85
87
in testing: 42
found: 88
Grove Consultants, 2003
DDP031227
DDP =
50
= 57
81 %
72
68
65
59
62
69
74
77
85
87
88
www.grove.co.uk
011 44 8702 406172
Page 5
Contents
Introduction: what is DDP
Examples of calculating DDP
Case studies
Advice about DDP & Conclusion
DDP example
Testing
Live
Running
DDP
after
Live
150
50
75%
DDP =
Grove Consultants, 2003
DDP031227
150
150 + 50
150
200
75%
www.grove.co.uk
011 44 8702 406172
DDP Exercise 1
DDP Exercise 1
The following data has been recorded for a project.
Calculate the DDP of each testing stage based on all the defect information.
Fault Information
Testing stage
Number of
faults
DDP
Official testing module
and integration
299
= ----------- = -------- = ___%
Tool testing &
development
40
= ----------- = -------- = ___%
Release testing
19
= ----------- = -------- = ___%
User Acceptance test
10
= ----------- = -------- = ___%
Pilot
= ----------- = -------- = ___%
Live Running (after one
month)
20
Hint: you dont really need a calculator just round the numbers to the nearest 10 and you will be
close enough!
Defects found in this stage of testing
DDP = -----------------------------------------------------------------------Defects found in this and all subsequent stages of testing
Grove Consultants
EuroSTAR Dec 2003
Page 1 of 2
DDP Exercise 1
DDP Exercise 1 Solution: DDP Calculation
Testing stage
No. faults
1) Official testing module
and integration
299
2) Tool testing &
development
40
3) Release testing
19
DDP
75%
300 / 400
40%
40 / 100
33%
20 / 60
4) User Acceptance test
10
25%
10 / 40
5) Pilot
33% (or 31%)
10 / 30 ( 9 / 29)
6) Live Running (after one
month)
20
n/a
How did we get these figures?
Remember DDP = Defects found in testing / all subsequent defects
Stage 1 Official testing
Test stage 1 found approximately 300 defects this is the numerator (top)
The sum of all the subsequent stages is = 40 + 20 (rounded up) + 10 + 10 (rounded up) + 20 = 100
So the denominator (bottom of the equation) is 300 + 100 = 400
DDP for Stage 1 is therefore 300/400 or 75%
Stage 2 Tool testing
Test stage 2 found 40 defects this is the numerator (top)
The sum of all the subsequent stages is = 20 (rounded up) + 10 + 10 (rounded up) + 20 = 60
So the denominator (bottom of the equation) is 40 + 60 = 100
DDP for Stage 2 is therefore 40/100 = 40%
Stage 3 Release testing
Test stage 3 found 19 defects (round up to 20) this is the numerator (top)
The sum of all the subsequent stages is = 10 + 10 (rounded up) + 20 = 40
So the denominator (bottom of the equation) is 20 + 40 = 60
DDP for Stage 3 is therefore 20/60 = 33%
Stage 4 User Acceptance test
Test stage 4 found 10 defects this is the numerator (top)
The sum of all the subsequent stages is = 10 (rounded up) + 20 = 30 (29 to be exact)
So the denominator (bottom of the equation) is 10 + 30 = 40
DDP for Stage 4 is therefore 10/40 = 25%
Stage 5 Pilot
Test stage 5 found 9 defects (round up to 10) this is the numerator (top)
The sum of all the subsequent stages is = 20 (the only remaining stage is live running)
So the denominator (bottom of the equation) is 10 + 20 = 30
DDP for Stage 4 is therefore 10/30 = 33% (31% if you calculate 9/29)
There is no DDP for live running, since the live running total goes into the calculation of all the
previous DDPs.
Grove Consultants
EuroSTAR Dec 2003
Page 2 of 2
Page 6
DDP is not percent of total defects
Stage of testing
"Official" (Mod & Int)
defects
Percentage
299
75%
"Tool" & development
40
10%
Release testing
19
5%
User Acceptance test
10
2.5%
2.5%
20
5%
Pilot
Live use (1 mo.)
100%
Source: Client
this is not DDP (its %)
DDP compares testing processes
Stage of testing
"Official" (Mod & Int)
defects
Percentage
299
75%
"Tool" & development
40
40%
Release testing
19
33%
User Acceptance test
10
25%
31%
Pilot
Live use (1 mo.)
20
live use determines DDP
Grove Consultants, 2003
DDP031227
www.grove.co.uk
011 44 8702 406172
Page 7
DDP example: ST DDP after UAT
ST
UAT
100
50
Live
UAT
DDP
after
Live
All test
DDP
after
Live
67%
100
DDP =
ST DDP ST DDP
after
after
UAT
Live
100 + 50
100
150
67%
DDP example: ST DDP after Live
ST
UAT
Live
100
50
100
DDP =
Grove Consultants, 2003
DDP031227
100
100 + 50 + 100
ST DDP ST DDP
after
after
UAT
Live
67%
UAT
DDP
after
Live
All test
DDP
after
Live
40%
100
250
40%
www.grove.co.uk
011 44 8702 406172
Page 8
DDP example: UAT DDP after Live
ST
UAT
Live
100
50
100
50
DDP =
50 + 100
ST DDP ST DDP
after
after
UAT
Live
67%
40%
50
150
UAT
DDP
after
Live
All test
DDP
after
Live
33%
33%
DDP example: all test DDP after Live
ST
UAT
Live
100
50
100
DDP =
Grove Consultants, 2003
DDP031227
100 + 50
100 + 50 + 100
ST DDP ST DDP
after
after
UAT
Live
67%
40%
150
250
UAT
DDP
after
Live
All test
DDP
after
Live
33%
60%
60%
www.grove.co.uk
011 44 8702 406172
DDP Exercise 2
DDP Exercise 2
The following data has been recorded for a project.
Calculate the DDPs in the columns on the right. The first one has been done as an example.
(A calculator may be useful for some of these your mobile phone has one!)
Fault Information
Release
System Test
User Acceptance Test
Live running
(1 month)
Release 1
100
50
100
Release 2
150
50
10
Release 3
200
50
50
Release 4
50
25
125
ST DDP
after UAT
ST DDP
after LR
67%
40%
UAT DDP
after LR
33%
All test DDP
after LR
60%
How did we get these figures?
Release 1
ST DDP after UAT: 100 / (100 + 50) = 100 / 150 = 67%
ST DDP after LR: 100 / (100 + 50 + 100) = 100 / 250 = 40%
UAT DDP after LR: 50 / (50 + 100) = 50 / 150 = 33%
(Remember not to include the ST defects here)
All test DDP after LR: (100 + 50) / (100 + 50 + 100) = 150 / 250 = 60%
Grove Consultants
EuroSTAR Dec 2003
Page 1 of 2
DDP Exercise 2
DDP Exercise 2 Solution: DDP Calculations
Release
System Test
User Acceptance Test
Live running
(1 month)
ST DDP
after UAT
ST DDP
after LR
UAT DDP
after LR
All test DDP
after LR
Release 1
100
50
100
67%
40%
33%
60%
Release 2
150
50
10
75%
71%
83%
95%
Release 3
200
50
50
80%
67%
50%
83%
Release 4
50
25
125
67%
25%
17%
38%
How did we get these figures?
Release 2
ST DDP after UAT: 150 / (150 + 50) = 150 / 200 = 75%
ST DDP after LR: 150 / (150 + 50 + 10) = 150 / 210 = 71%
UAT DDP after LR: 50 / (50 + 10) = 50 / 60 = 83%
All test DDP after LR: (150 + 50) / (150 + 50 + 10) = 200 / 210 = 95%
Release 3
ST DDP after UAT: 200 / 200 + 50) = 200 / 250 = 80%
ST DDP after LR: 200 / (200 + 50 + 50) = 200 / 300 = 67%
UAT DDP after LR: 50 / (50 + 50) = 50 / 100 = 50%
All test DDP after LR: (200 + 50) / (200 + 50 + 50) = 250 / 300 = 83%
Release 4
ST DDP after UAT: 50 / (50 + 25) = 50 / 75 = 67%
ST DDP after LR: 50 / (50 + 25 + 125) = 50 / 200 = 25%
UAT DDP after LR: 25 / (25 + 125) = 25 / 150 = 17%
All test DDP after LR: (50 + 25) / (50 + 25 + 125) = 75 / 200 = 38%
Grove Consultants
EuroSTAR Dec 2003
Page 2 of 2
Page 9
Contents
Introduction: what is DDP
Examples of calculating DDP
Case studies
Advice about DDP & Conclusion
Case studies from clients
1 mo
10 mo
Year 1
70%
50% est
Year 2
92%
Finance (insurance)
Operating system
System Test Group DDP = 38%
(before performance testing)
Priority 1 & 2 only: DDP = 31%
23% to 87% by application
Defects: 1 / 4 160 / 40
Scientific software
(chemical analysis)
Not useful for low numbers of defects
Grove Consultants, 2003
DDP031227
www.grove.co.uk
011 44 8702 406172
Page 10
Information Technology
RESULTS SO FAR
E ffe c tiv e n e s s o f T e s tin g
100
90
80
S y s te m T e s t
UAT
70
60
CDR for Sirius
Skynet
Alibi/Claims
CARRS Release 4
ACRaSh v1.2
IMR Pre-Screening
IBIS Euro Phase 2
IBIS Release 3
CARRS Release 3
40
BWS Phase 1b
50
Source: Dave Norman, EuroSTAR02 DDP Advanced Workshop, with permission
Information Technology
MESSAGES
Conclusions
UAT more variable than ST mainly personnel
Target zone for ST : 75 - 90%
Factors behind the figures
size, complexity, tester experience, time, documentation
whether UAT started before ST was finished
20
15
10
o r h e re ?
14/12/01
13/12/01
12/12/01
11/12/01
cost of finding
10/12/01
09/12/01
cost, severity of those you missed
s to p p e d h e re ?
25
08/12/01
Figures dont tell you
30
07/12/01
where on the S-curve when stopped
Page 20
Grove Consultants, 2003
DDP031227
www.grove.co.uk
011 44 8702 406172
Page 11
DDP Summary for AP Europe
Project
or App.
Months
DDP
DDP Status
Before New Testing Process
S4
50%
ESTIMATED
After New Testing Process
R1
3
81%
LBS
4
91%
CP
7
100%
DS
3
95%
APC
4
93%
ELCS
4
95%
SMS
3
96%
C
4
96%
E7 (US)
5
83%
E7 (Eur)
1
97%
FINAL
FINAL
FINAL
FINAL
FINAL
FINAL
FINAL
FINAL
FINAL
NEW
Comments
Major re-engineering
Reporting System
Eur impl. of US system
Enhancement Release
Global Enhancements
Global Enhancements
Source: Stuart Compton, Air Products plc
Rolling DDP
Software Testing Defect Detection Percentage Measure
(rolling quarterly produced values looking back four quarters)
Period under review
# Projects
Analysed
Historical Estimate
Rolling 1 Qtrs DDP to Q1 Y1
Rolling 2 Qtrs DDP to Q2 Y1
Rolling 3 Qtrs DDP to Q3 Y1
Rolling 4 Qtrs DDP to Q4 Y1
Rolling 4 Qtrs DDP to Q1 Y2
Rolling 4 Qtrs DDP to Q2 Y2
Rolling 4 Qtrs DDP to Q3 Y2
Rolling 4 Qtrs DDP to Q4Y2
Rolling 4 Qtrs DDP to Q1 Y3
Rolling 4 Qtrs DDP to Q2 Y3
Rolling 4 Qtrs DDP to Q3 Y3
Rolling 4 Qtrs DDP to Q4 Y3
Target
n/a
n/a
n/a
n/a
n/a
80
80
80
80
90
90
90
90
Defects in
Testing
Total
Defects
Prod'n
Bugs
1111
1400
289
1171
1466
295
1211
1508
297
1492
1807
315
2034
2129
95
1974
2063
89
2086
2204
118
1976
2087
111
DDP
50
79
80
80
83
96
96
95
95
?
?
?
?
Source: Stuart Compton, Air Products plc
Grove Consultants, 2003
DDP031227
www.grove.co.uk
011 44 8702 406172
Page 12
Contents
Introduction: what is DDP
Examples of calculating DDP
Case studies
Advice about DDP & Conclusion
DDP benefits
DDP can highlight
- test process improvements
- the effect of severe deadline pressure
- overlapping test phases
can raise the profile of testing
can help predict future defect levels
is applicable over different projects
- reflects testing process in general
on-going monitoring
Grove Consultants, 2003
DDP031227
www.grove.co.uk
011 44 8702 406172
Page 13
Advice about DDP
What is not important
- projects of difference sizes, scope, duration, technology, life
cycle approaches
- how you collect and analyse defect data
- really good accuracy or detailed classification (e.g.
duplicates, where inserted)
- how well the project actually went
What is important
- consistency of defect data collection and analysis
either use severity or not (same approach)
May matter / may not
- different people, different test approaches, complexity
DDP limitations
not useful unless you have a reasonable
number of defects and projects, over time
never use it to measure individuals!
- only test efforts
dont get sucked into too much detail (too
soon)
- dont do hard things do easy things
it reflects what is happening
- not [just] how good a job you are doing of testing
not related to efficiency / cost
Grove Consultants, 2003
DDP031227
www.grove.co.uk
011 44 8702 406172
Page 14
Problems and solutions - technical
time frame for defects found in live
- this is arbitrary / whatever makes sense for you
many people use 1 month, some use 3 or 6 months
different test stages
- you can measure any stage as long as you have defects that
came afterwards
but dont measure individual people!!
incremental / RAD
- you have choices accumulate, or measure until next release
different defect tracking systems
- ok to combine for different stages or if consistently recorded
Problems and solutions - perceptions
communication
- DDP is a communication tool, giving information about the
testing that has been done
dealing with misconceptions
- very important that those who see DDP understand what it is
and what it means (and doesnt mean)
interpreting results
- the measure will raise questions to be answered
can be used to predict defects not found yet
- if DDP is stable, using current test defect numbers
- needs caution
Grove Consultants, 2003
DDP031227
www.grove.co.uk
011 44 8702 406172
Page 15
Problems and solutions getting started
starting to measure
- start by calculating DDP of a release that is now live
getting data from live running
- if you dont normally have live defect data,
ask for it / search it out
what DDP to measure first
- most people start with System Test
- consider looking at highest severity only to start
ease of data collection & calculation
- ideally should be automatic and painless
- can use test management tool or defect tracking tool
Problems and solutions data accuracy
most common stumbling block
- what about duplicates?
- what about enhancement requests?
- what if some arent really defects?
the same answer always applies
- it doesnt matter how you do it
as long as you do it the same way each time!
most useful aspect of DDP
- trends, changes over time
Grove Consultants, 2003
DDP031227
www.grove.co.uk
011 44 8702 406172
Page 16
The technical persons trap
were testers we can see all the problems!
- you will think of lots of problems with this metric
- yes, DDP (as any measure) can be mis-used
but that doesnt mean it cant be useful
take the high level view
- DDP, warts and all, computed simply and
consistently, can help you monitor your testing
processes
What does it mean?
DDP is very high (> 95%)
- testing is very good?
- system not been used much yet?
- next stage of testing was very poor?
e.g. ST looks good but UAT was poor, ST after UAT is
high but live running will find many defects!
DDP is low (< 60%)
- testing is poor?
- requirements were very poor, affecting tests?
- poor quality software (too many to find in the time)?
- deadline pressure testing was squeezed?
Grove Consultants, 2003
DDP031227
www.grove.co.uk
011 44 8702 406172
Page 17
Options for measuring DDP
What defects?
- simplest: all test defects / all defects so far
- by severity level
How "deep" to go?
- deeper levels give more detailed information
- deeper levels more complex to measure
Start simple
- simple information is much better than none
- learn from what information you have
Conclusion
DDP is a useful measure
- easy to calculate
- based on defect data you probably already have
- can tell you how effective your testing efforts are
and how other things affect it
DDP requires counts of defects
- but does not need great accuracy
Grove Consultants, 2003
DDP031227
www.grove.co.uk
011 44 8702 406172
