Introduction to Software

Testing
Content

◼ What is software testing

◼ Why do we need software testing
◼ How do we do software testing

2
What Is Software Testing
◼ Software testing is one of the methods to
assure software quality.
◼ Software quality assurance involves
validation and verification of software.
Validation: requirements specification
consistent
Do we build the right software?
Verification: specification implementation
consistent
Do we build the software right?
3
Software Verification

◼ Software can be verified statically or

dynamically.
◼ Static verification techniques do not execute
software and contain review, inspection,
walkthrough, and analysis.
◼ Dynamic verification techniques execute
software and contain various testing
techniques.

4
Software Testing

◼ Software testing cannot prove the

correctness of software.
◼ Software testing can only find as many errors
in software as possible.

5
Error, Fault, Failure, and Incident

◼ People make errors.

◼ A fault in the documents or programs is the
result of an error.
◼ A failure occurs when a fault executes.
◼ When a failure occurs, it may or may not be
readily apparent to the user. An incident
alerts the user to the occurrence of a failure.

6
Test Cases
◼ Software testing is the act of executing
software with a suite of test cases so that it
can find faults in the program.
◼ A test case contains a list of test inputs and a
list of corresponding expected outputs.
◼ Each test case is associated with a specific
program behavior.

7
The Testing Life Cycle
error

Fault
Specification fault
Resolution
error

Fault
Design fault
Isolation
error

Fault
Coding fault Static verification
Classification
can be used to
incident
detect faults in
specification and
Testing
design early.

8
Why Do We Need Software Testing

◼ Software prevails in our living environment.

Quality of software significantly influences our
quality of life.
◼ Software faults in critical software systems
may cause dramatic damages on our lives
and finance.

9
Example: THERAC-25 Radiation Therapy

◼ In 1986 two cancer patients at the East

Texas cancer Center received fatal radiation
overdoses from the computer-controlled
radiation therapy machine.
◼ A software bug on mishandled race condition.

10
Example: Shooting down Airbus 320

◼ In 1988 USA troops mistook an Airbus 320 as

an F-14 and shot down the Airbus 320.
◼ 290 people dead.
◼ A software bug in the tracking software.

11
Example: London Ambulance Service
◼ London Ambulance Service (LAS) Computer
Aided Dispatch automates many of the
human-intensive processes of manual
dispatch systems associated with ambulance
services in the UK.
◼ Failure of the LAS on 26 and 27 November
1992.
◼ At 23:00 on November 28 the LAS eventually
instigated a backup procedure, after the
death of at least 20 patients.
12
Example: Mars Climate Orbiter
◼ In 1999, the Mars Climate Orbiter is
supposed to relay signals from the Mars
Polar Lander once it reached the surface of
the planet.
◼ The Mars Climate Orbiter smashed into the
planet instead of reaching a safe orbit.
◼ A software bug that fails to convert English
measures to metric values.
◼ This incident costs $165 million.

13
Current Status

◼ Carefully made programs have 5 faults per

1000 lines of code (LOC).
◼ Windows XP has 45M LOC, so it may have
225000 faults.

14
Comparison with Automobile Industry

◼ If the automobile industry had developed like

the software industry, we would all be driving
$25 cars.
◼ If car were like software, they would crash
twice a day for no reason, and when you
called for service, they would tell you to
reinstall the engine.

15
Economic Impact

◼ In 2002 a study commissioned by the

National Institute of Standards and
Technology found that software errors cost
the U.S. economy about $59.5 billion
annually, or about 0.6 percent of the gross
domestic product.

16
How Do We Do Software Testing

correct
faults of omission faults of comission
S I

Specification Implementation
(expected) (observed)

17
Specified, Implemented, and Tested
Program Behaviors

Specification Implementation
(expected) S I (observed)
2
5 6
1
4 3
7
T
Test Cases
(verified)

18
Test Cases

◼ Test case generation

◼ Test case execution

19
Test Case Execution

test input

test case software

expected output output

correct equal? incident

20
Test Case Generation

◼ There are two approaches to determining test

cases: functional testing (or black box testing)
and structural testing (or white box testing).
◼ Functional testing: the software is viewed as a
black box and test cases are determined from
the functions described in the specification.
◼ Structural testing: the software is viewed as a
white box and test cases are determined from
the structure designed in the implementation.

21
Functional Testing
Specification Implementation
S I
A
T
Test Cases

Specification Implementation
S I
B
T
Test Cases

22
Structural Testing
Specification Implementation
S I
A
T
Test Cases

Specification Implementation
S I
B
T
Test Cases

23
Functional Testing v.s. Structural Testing

◼ It is hard for functional testing to identify

behaviors that are not specified (faults of
comission).
◼ It is hard for structural testing to identify
behaviors that are not implemented (faults of
omission).
◼ Neither approach is sufficient, both
approaches are needed.

24
Tracking of Incidents

◼ An incident tracking system keeps track of

the incidents that should be fixed so that all
incidents are properly resolved.

25
Tracking of Incidents

◼ Anyone who needs to know about an incident

should learn of it soon after it’s reported.
◼ No incident will go unfixed merely because
someone forgot about it.
◼ No incident will go unfixed on the whim of a
single programmer.
◼ A minimum of incidents will go unfixed merely
because of poor communication.

26
Regression Testing

◼ Regression testing reuses the test cases to

test the fixed (or a new version of ) software
to make sure that the fixing does not
introduce new faults into the software.

27
Levels of Testing (V-Model)
verify
Acceptance
Requirements
Testing

verify System
Specification
Testing

verify Integration
Design
Testing

verify
Unit
Coding
Testing

28
Costs of Software Quality (Control Costs)

◼ Prevention costs include investments in

quality infrastructure and quality activities that
are not directed to a specific project or
system, being general to the organization.
◼ Appraisal costs include the costs of activities
performed for a specific project or system for
the purpose of detecting software errors.

29
Costs of Software Quality (Failure of
Control Costs)
◼ Internal failure costs include costs of
correcting errors that have been detected by
design reviews, software tests and
acceptance tests and completed before the
software is installed at customer sites.
◼ External failure costs include all costs of
correcting failures detected by customers or
the maintenance team after the software
system has been installed.

30
Software Quality Levels v.s. Costs

Quality costs Total cost of software quality

Minimal
total Total control cost
cost
Total failure of
of
control cost
software
quality low Quality level high
Optimal software quality level

31
Test Coverage Criteria

◼ Software quality costs affects software quality

levels.
◼ We can determine when to stop the software
testing based on available software quality
resources.
◼ The test coverage criteria are used to
determine the termination of software testing.

32