Software Testing and Quality Assurance

Theory and Practice

Chapter 1
Basic Concepts and Preliminaries

1
 What is Software?
 What is Software?
– More than computer programs
– Computer programs, procedures, and possibly
associated documentation and data pertaining to
the operation of a computer system.

 Two major types of Software:

• Generic –Stand alone, sold on open market
• Customized –For specific customer

2
 What is Software Quality?
• Software Quality (as per ISO/ IEC 9126):
The totality of functionality and features of a
software product that contribute to its ability to
satisfy stated or implied needs.

• Software Quality (as IEEE Std 610):

The degree to which a component, system or process
meets specified requirements and/or user/customer
needs and expectations.

3
 What is Software Quality?

 According to ISO 9126, software quality consists of:

– Functionality
– Reliability
– Usability
– Efficiency
– Maintainability
– Portability

4
 Introduction
 What’s the main challenges of software development
now-a-days?
– High Cost
– Difficult to deliver on Time
– Low Quality

5
 Introduction

 What is Software Quality Assurance?

 What is Software Testing?

 What are the differences between them?

6
 Introduction
 Software Quality Assurance (SQA):
 Defined as a planned and systematic approach to the evaluation
of the quality of and adherence to software product standards,
processes, and procedures.
 An umbrella activity that is applied throughout the software
process.
 Consists of a means of monitoring the software engineering
processes and methods used to ensure quality.
 An effective approach to produce high quality software.

7
 Introduction
 Software Testing:
 Software Testing is the process of executing a system or
component under specified conditions with the intent of
finding defects/bugs and to verify that it satisfies specified
requirements.
 Main goal ==> To detect bugs
 Have different levels
 Static testing vs. Dynamic testing
 Manual testing vs. Automated testing

8
 QA vs. Testing

Software Quality Assurance Software Testing

• Process-oriented activity • Product-oriented activity
• Oriented to bug prevention • Oriented to bug detection

9
 Introduction

 What is the COST of a bug?

10
 The Quality Revolution
• Started in Japan by Deming, Juran, and Ishikawa during 1940s
• In 1950s, Deming introduced statistical quality control to Japanese
engineers
• Statistical quality control (SQC) is a discipline based on
measurement and statistics
– SQC methods use seven basic quality management tool
• Pareto analysis, Trend Chart, Flow chart, Histogram, Scatter diagram,
Control chart, Cause and effect diagram
• “Lean principle” was developed by Taiichi Ohno of Toyota
“A systematic approach to identifying and eliminating waste through continuous
improvement, flowing the product at the pull of the customer in pursuit of
perfection.”

11
 The Quality Revolution

Figure 1.1: The Shewhart cycle

• Deming introduced Shewhart’s PDCA cycle to Japanese researchers

• It illustrate the activity sequence:
– Setting goals
– Assigning them to measurable milestones
– Assessing the progress against the milestones
– Take action to improve the process in the next cycle

12
 The Quality Revolution
• In 1954, Juran spurred the move from SQC to TQC (Total Quality
Control)
• Key Elements of TQC:
– Quality comes first, not short-term profits
– The customer comes first, not the producer
– Decisions are based on facts and data
– Management is participatory and respectful of all employees
– Management is driven by cross-functional committees
• An innovative methodology developed by Ishikawa called cause-
and-effect diagram

Figure 1.2: Ishikawa diagram

13
 The Quality Revolution
• National Broadcasting Corporation (NBC) of United States broadcast a
documentary

“If Japan Can ... Why Can’t We?” on June 24th, 1980

• Leaders in United States started emphasizing on quality

• In 1987 Malcolm Baldrige National Quality Award was introduced in U.S.A

Similar to the Deming prize in Japan

• In Baldrige National Award the quality is viewed as:

Something defined by the customer

• In Deming prize, the quality is viewed as:

Something defined by the producer by conformance to specifications

14
 Software Quality
• Five Views of Software Quality:
– Transcendental view
– User’s view
– Manufacturing view
– Product view
– Value-based view
• Software Quality in terms of quality factors and criteria
– A quality factor represents behavioral characteristic of a system
• Examples: correctness, reliability, efficiency, and testability
– A quality criterion is an attribute of a quality factor that is related to software
development
• Example: modularity is an attribute of software architecture
• Quality Models
– Examples: ISO 9126, CMM, TPI, and TMM

15
 Role of Testing
• Software quality assessment divide into two categories:

– Static analysis
• It examines the code and reasons over all behaviors that might arise during
run time
– Examples: Code review, inspection, and algorithm analysis

– Dynamic analysis
• Actual program execution to expose possible program failure
• One observe some representative program behavior, and reach conclusion
about the quality of the system

• Static and Dynamic Analysis are complementary in nature

• Focus is to combines the strengths of both approaches

16
 Verification and Validation
• Verification
– Evaluation of software system that help in determining whether the product of
a given development phase satisfy the requirements established before the start
of that phase
• Building the product correctly

• Validation
– Evaluation of software system that help in determining whether the product
meets its intended use
• Building the correct product

17
 Failure, Error, Fault and Defect
• Failure
– A failure is said to occur whenever the external behavior of a system does not
conform to that prescribed in the system specification
• Error
– An error is a state of the system.
– An error state could lead to a failure in the absence of any corrective action by
the system
• Fault
– A fault is the adjudged cause of an error
• Defect
– It is synonymous of fault
– It a.k.a. bug

18
 The Notion of Software Reliability
• It is defined as the probability of failure-free operation of a software
system for a specified time in a specified environment

• It can be estimated via random testing

• Test data must be drawn from the input distribution to closely

resemble the future usage of the system

• Future usage pattern of a system is described in a form called

operational profile

19
 The Objectives of Testing
• It does work

• It does not work

• Reduce the risk of failures

• Reduce the cost of testing

20
 What is a Test Case?
• Test Case is a simple pair of
<input, expected outcome>
• State-less systems: A compiler is a stateless system
– Test cases are very simple
• Outcome depends solely on the current input
• State-oriented: ATM is a state oriented system
– Test cases are not that simple. A test case may consist of a sequences of
<input, expected outcome>
• The outcome depends both on the current state of the system and the
current input
• ATM example:
 < check balance, $500.00 >,
< withdraw, “amount?” >,
< $200.00, “$200.00” >,
< check balance, $300.00 >

21
 Expected Outcome
• An outcome of program execution may include
– Value produced by the program
– State Change
– A sequence of values which must be interpreted together for the outcome to be
valid

• A test oracle is a mechanism that verifies the correctness of program

outputs
– Generate expected results for the test inputs
– Compare the expected results with the actual results of execution of the IUT

22
 The Concept of Complete Testing
• Complete or exhaustive testing means
“There are no undisclosed faults at the end of test phase”

• Complete testing is near impossible for most of the system

– The domain of possible inputs of a program is too large
• Valid inputs
• Invalid inputs
– The design issues may be too complex to completely test
– It may not be possible to create all possible execution environments of the
system

23
 The Central Issue in Testing

Figure 1.5: A subset of the input domain exercising a subset of the program behavior

• Divide the input domain D into D1 and D2

• Select a subset D1 of D to test program P
• It is possible that D1 exercise only a part P1 of P

24
 Testing Activities

Figure 1.6: Different activities in process testing

• Identify the objective to be tested

• Select inputs
• Compute the expected outcome
• Set up the execution environment of the program
• Execute the program
• Analyze the test results
25
 Testing Level
• Unit testing
– Individual program units, such as
procedure, methods in isolation
• Integration testing
– Modules are assembled to construct
larger subsystem and tested
• System testing
– Includes wide spectrum of testing
such as functionality, and load
• Acceptance testing
– Customer’s expectations from the
system
– Two types of acceptance testing
• UAT Figure 1.7: Development and testing phases
• BAT in the V model
– UAT: System satisfies the contractual
acceptance criteria
– BAT: System will eventually pass the
user acceptance test

26
 Testing Level

Figure 1.8: Regression testing at different software testing levels

• New test cases are not designed

• Test are selected, prioritized and executed
• To ensure that nothing is broken in the new version of the software

27
 Source of Information for Test Selection
• Requirement and Functional Specifications
• Source Code
• Input and output Domain
• Operational Profile
• Fault Model
– Error Guessing
– Fault Seeding
– Mutation Analysis

28
 White-box and Black-box Testing
• White-box testing a.k.a. structural • Black-box testing a.k.a. functional
testing testing
• Examines source code with focus on: • Examines the program that is
– Control flow accessible from outside
– Data flow • Applies the input to a program and
• Control flow refers to flow of control observe the externally visible outcome
from one instruction to another • It is applied to both an entire program
• Data flow refers to propagation of as well as to individual program units
values from one variable or constant • It is performed at the external
to another variable interface level of a system
• It is applied to individual units of a • It is conducted by a separate software
program quality assurance group
• Software developers perform
structural testing on the individual
program units they write

29
 Test Planning and Design
• The purpose is to get ready and organized for test execution
• A test plan provides a:
– Framework
• A set of ideas, facts or circumstances within which the tests will be
conducted
– Scope
• The domain or extent of the test activities
– Details of resource needed
– Effort required
– Schedule of activities
– Budget
• Test objectives are identified from different sources
• Each test case is designed as a combination of modular test
components called test steps
• Test steps are combined together to create more complex tests
30
 Monitoring and Measuring Test Execution
• Metrics for monitoring test execution

• Metrics for monitoring defects

• Test case effectiveness metrics

– Measure the “defect revealing ability” of the test suite
– Use the metric to improve the test design process

• Test-effort effectiveness metrics

– Number of defects found by the customers that were not found by the test
engineers

31
 Test Tools and Automation
• Increased productivity of the testers • The test cases to be automated are
well defined
• Better coverage of regression testing
• Test tools and an infrastructure are in
place
• Reduced durations of the testing
phases
• The test automation professionals
have prior successful experience in
• Reduced cost of software maintenance automation

• Increased effectiveness of test cases • Adequate budget have been allocation

for the procurement of software tools

32
 Test Team Organization and Management

Figure 16.1: Structure of test groups

• Hiring and retaining test engineers is a challenging task

• Interview is the primary mechanism for evaluating applicants
• Interviewing is a skills that improves with practice
• To retain test engineers management must recognize the importance of testing
efforts at par with development effort

33