REGRESSION TESTING

The process of re-testing of software after modifications is called regression

testing. Regression testing is the process of re-testing the modified parts of the
software and ensuring that no new errors have been introduced into previously tested
source code due to these modifications. Therefore, regression testing tests both the
modified source code and other parts of the source code that may be affected by the
change. It serves several purposes like –

1. Increases confidence in the correctness of the modified program.

2. Locates errors in the modified program.
3. Preserves the quality and reliability of the software.
4. Ensures the software’s continued operation.

We typically think of regression testing as a software maintenance activity;

however, we also perform regression testing during the latter stage of software
development. This latter stage starts after we have developed test plans and test
suites and used them initially to test the software. During this stage of development,
we fine-tune the source code and correct errors in it, hence these activities resemble
maintenance activities.

REGRESSION TESTING PROCESS

Regression testing is a very costly process and consumes a significant amount
of resources. Whenever a failure is experienced, it is reported to the software team.
The team may like to debug the source code to know the reason(s) for this failure.
After identification of the reason(s), the source code is modified and we generally
do not expect the same failure again. In order to ensure this correctness, we re-test
the source code with a focus on modified portion(s) of the source code and also on
affected portion(s) of the source code due to modifications. We need test cases that
target the modified and affected portions of the source code. We may write new test
cases, which may be a ‘time and effort consuming’ activity. We neither have enough
time nor reasonable resources to write new test cases for every failure. Another
option is to use the existing test cases which were designed for development testing
and some of them might have been used during development testing. The existing
test suite may be useful and may reduce the cost of regression testing. As we all
know, the size of the existing test suite may be very large and it may not be possible
to execute all tests.
Various steps related to regression testing process are shown below-
REGRESSION TESTING PROCESS
 RISK ANALYSIS
RISK
Risk is a problem that may cause some loss or threaten the success of the
project, but, which has not happened yet. Risk is defined as the “probability of
occurrence of an undesirable event and the impact of occurrence of that event.”
To understand whether an event is really risky needs an understanding of the
potential consequences of the occurrences / non occurrences of that event. Risks may
delay and over-budget a project. Risky projects may also not meet specified quality
levels. Hence, there are two things associated with risk as given below:

(i) Probability of occurrence of a problem (i.e. an event)

(ii) Impact of that problem

Risk analysis is a process of identifying the potential problems and then

assigning a ‘probability of occurrence of the problem’ value and ‘impact of that
problem’ value for each identified problem. Both of these values are assigned
on a scale of 1 (low) to 10 (high). A factor ‘risk exposure’ is calculated for every
problem which is the product of ‘probability of occurrence of the problem’
value and ‘impact of that problem’ value. The risks may be ranked on the basis
of its risk exposure. A risk analysis table may be prepared as given below. These
values may be calculated on the basis of historical data, past experience, intuition
and criticality of the problem. We should not confuse with the mathematical scale
of probability values which is from 0 to 1. Here, the scale of 1 to 10 is used for
assigning values to both the components of the risk exposure.

Risk Analysis Table Format

 Risk Analysis Table of ‘University Registration System’

RISK MATRIX
Risk matrix is used to capture identified problems, estimate their probability
of occurrence with impact and rank the risks based on this information. We may use
the risk matrix to assign thresholds that group the potential problems into priority
categories. The risk matrix is shown in figure 1 given below with four quadrants.
Each quadrant represents a priority category.
 1. Threshold by quadrant

The priority category in defined as follows:

Priority category 1 (PC-1) = High probability value and high impact value
Priority category 2 (PC-2) = High probability value and low impact value
Priority category 3 (PC-3) = Low probability value and high impact value
Priority category 4 (PC-4) = Low probability value and low impact value

In this case, a risk with high probability value is given more importance than a
problem with high impact value.

We may change this and may decide to give more importance to high impact value
over the high probability value and is shown in Figure 2. Hence, PC-2 and PC-3 will
swap, but PC-1 and PC-4 will remain the same.
 2. Alternative threshold by quadrant

There may be situations where we do not want to give importance to any value and
assign equal importance. In this case, the diagonal band prioritization scheme may
be more suitable as shown in figure 3 given below. This scheme is more appropriate
in situations where we have difficulty in assigning importance to either ‘probability
of occurrence of the problem’ value or ‘impact of that problem’ value.

We may also feel that high impact value must be given highest priority irrespective
of the ‘probability of occurrence’ value. A high impact problem should be addressed
first, irrespective of its probability of occurrence value. This prioritization scheme is
given in Figure 4. Here, the highest priority (PC-1) is assigned to high impact value
and for the other four quadrants; any prioritization scheme may be selected.
We may also assign high priority to high ‘probability of occurrence’ values
irrespective of the impact value as shown in Figure 5. This scheme may not be
popular in practice. Generally, we are afraid of the impact of the problem. If the
impact value is low, we are not much concerned. In the risk analysis table (see Table
), ambiguous documentations (S. No. 7) have high ‘probability of occurrence of
problem’ value (8), but impact value is very low (1). Hence, these faults are not
considered risky faults as compared to ‘unauthorized access’ (S. No. 5) where
‘probability of occurrence’ value is very low (1) and impact value is very high (10).

3. Threshold by diagonal quadrant

4. Threshold based on high ‘Impact of Problem’ value
 5. Threshold based on high ‘probability of occurrence of problem’ value

After the risks are ranked, the high priority risks are identified. These risks are
required to be managed first and then other priority risks in descending order. These
risks should be discussed in a team and proper action should be recommended to
manage these risks. A risk matrix has become a powerful tool for designing
prioritization schemes. Estimating the probability of occurrence of a problem may
be difficult in practice. Fortunately, all that matters when using a risk matrix is the
relative order of the probability estimates (which risks are more likely to occur) on
the scale of 1 to 10. The impact of the problem may be critical, serious, moderate,
minor or negligible. These two values are essential for risk exposure which is
used to prioritize the risks.
 DIFFERENCE BETWEEN DEVELOPMENT TESTING AND
REGRESSION TESTING

S.NO. DEVELOPMENT REGRESSION TESTING

TESTING

1 We write test cases. We may use already available

test cases.
2 We want to test all portions We want to test only
of the source code. modified portion of the
source code and the portion
affected by the modifications.
3 We do development testing We may have to do
just once in the lifetime of the regression testing many times
software. in the lifetime of the
software.
4 We do development testing to We do regression testing to
obtain confidence about the obtain confidence about the
correctness of the software. correctness of the modified
portion of the software.
5 Performed under the pressure Performed in crisis situations,
of release date. under greater time
constraints.
6 Separate allocation of budget Practically no time and
and time. generally no separate budget
allocation.
7 Focus is on the whole Focus is only on the modified
software with the objective portion and other affected
finding faults. portions with the objective of
ensuring the correctness of
the modifications.
8 Time and effort consuming Not much time and effort is
activity (40% to 70%). consumed as compared to
development testing.
 PRIORITIZATION GUIDELINES
Test case prioritization refers to the process of ranking test cases based on
their importance. This ensures that the most crucial aspects of a software system
are tested early, leading to efficient testing, particularly in regression testing. To
prioritize the test cases we may follow various guidelines that helps us to
prioritize test cases. Prioritization guidelines play very crucial role in
prioritization of test cases. Few guidelines are as follows-

1. We may indicate the order with which a test case may be addressed. This
process is known as prioritization of test cases. A test case with the highest rank
has the highest priority and the test case with the second highest rank has the
second highest priority and as so on.

2. Prioritization should not discard any test case. The efficiency of the regression
testing is dependent upon the criteria of prioritization.

3. Prioritization should be based on varieties of test case prioritization. There are

two varieties of test case prioritization i.e. general test case prioritization and
version specific test case prioritization.
In general test case prioritization, for a given program with its test suite, we
prioritize the test cases that will be useful over a succession of subsequent
modified versions of the original program without any knowledge of
modification(s). In the version specific test case prioritization, we prioritize the
test cases, when the original program is changed to the modified program, with
the knowledge of the changes that have been made in the original program.

4. Prioritization guidelines should address two fundamental issues like:

1. What functions must be tested?
o Identify the key functionalities of the software that need to be tested,
especially those that are critical for the software’s operation.
2. What are the consequences of not testing certain functions?
o Assess the risks and potential impacts of not testing specific
functions (e.g., missed bugs, system failure, and customer
dissatisfaction).
5. Risk-Based Prioritization should be follow during prioritization because test
cases that cover risky functions are given higher priority to minimize the chances
of critical failures. Risk analysis plays a crucial role in prioritizing test cases,
based on factors like:
 o Complexity of the functionality.
o Criticality of the function (how important it is to the system).
o Impact of failure (ranging from no impact to severe outcomes like
data loss or human harm).

DIFFERENCE BETWEEN GENERAL TEST CASE PRIORITIZATION

AND VERSION-SPECIFIC TEST CASE PRIORITIZATION
1. General Test Case Prioritization:
o Involves prioritizing test cases based on their usefulness over
multiple subsequent modified versions of the software.
o Performed without knowing what modifications will be made to the
software.
2. Version-Specific Test Case Prioritization:
o Prioritizes test cases after knowing the specific changes made in the
new version of the program.
o Focuses on testing areas of the software that have been directly
impacted by the changes.
 PRIORITY CATEGORY SCHEME
A priority code is assigned to each test case to indicate its level of
importance. This helps to categorize the test cases and ensures that the most
critical tests are executed first.
Priority Codes Based on Importance:
• Priority Code 1: Essential test case
o These tests are crucial and must be executed to ensure the core
functionality of the system works as expected.
• Priority Code 2: Important test case
o These are important for maintaining the overall quality but might not
be immediately critical.
• Priority Code 3: Execute if time permits
o These test cases are nice to have but are only executed if there is
sufficient time after testing the critical and important cases.
• Priority Code 4: Not important test case
o Test cases that are not critical to the functionality or user experience.
They are executed last, if at all.
• Priority Code 5: Redundant test case
o These tests may overlap with other test cases or cover scenarios
already tested and can often be skipped.
Alternative Priority Codes (Customer Focused):
• Priority Code 1: Important for the customer
o Focuses on test cases that ensure customer needs are met.
• Priority Code 2: Required to increase customer satisfaction
o Test cases that improve user experience or reliability, even if not
immediately critical.
• Priority Code 3: Helps to increase market share
o Test cases that may not impact functionality but could help in
improving product competitiveness.
 We may design any priority category scheme, but a scheme based on
technical considerations always improves the quality of the product and should
always be encouraged.

DESIGNING A PRIORITY CATEGORY SCHEME

A priority scheme is a structured method to categorize test cases based on
a set of guidelines, which ensures a systematic approach to addressing different
levels of importance. We may design any priority category scheme, but a scheme
based on technical considerations always improves the quality of the product and
should always be encouraged. There are various factors that affects the
prioritization scheme, few of them are as follows -

• Technical Considerations:
o Focus on the quality of the product.
o Test cases addressing high-risk areas or critical functionalities
should receive highest priority.
• Customer Requirements or Market Conditions:
o In some cases, priority may be assigned based on customer feedback
or competitive needs, focusing on customer satisfaction and product
performance in the market.
 REDUCING THE NUMBER OF TEST CASES

Test case reduction is an essential activity and we may select those test cases that
execute the modification(s) and the portion of the program that is affected by the
modification(s). We may minimize the test suite or prioritize the test suite in order
to execute the selected number of test cases. There are two ways to minimize the
number of test cases -

1 Minimization of Test Cases

We select all those test cases that traverse the modified portion of the program
and the portion that is affected by the modification(s). If we find the selected
number very large, we may still reduce this using any test case minimization
technique. These test case minimization techniques attempt to find redundant test
cases. A redundant test case is one which achieves an objective which has already
been achieved by another test case. The objective may be source code coverage,
requirement coverage, variables coverage, branch coverage, specific lines of
source code coverage, etc.

2 Prioritization of Test Cases

We may indicate the order with which a test case may be addressed. This process
is known as prioritization of test cases. A test case with the highest rank has the
highest priority and the test case with the second highest rank has the second
highest priority and as so on. Prioritization does not discard any test case. The
efficiency of the regression testing is dependent upon the criteria of prioritization.
There are two varieties of test case prioritization i.e. general test case
prioritization and version specific test case prioritization.

In general test case prioritization, for a given program with its test suite, we
prioritize the test cases that will be useful over a succession of subsequent
modified versions of the original program without any knowledge of
modification(s).
In the version specific test case prioritization, we prioritize the test cases, when
the original program is changed to the modified program, with the knowledge of
the changes that have been made in the original program. Prioritization guidelines
should address two fundamental issues like:

(i) What functions of the software must be tested?

(ii) What are the consequences if some functions are not tested?
All prioritization guidelines should be designed on the basis of risk analysis. The
risk analysis may be based on complexity, criticality, impact of failure, etc. The
most important is the ‘impact of failure’ which may range from ‘no impact’ to
‘loss of human life’ and must be studied very carefully.

The simplest priority category scheme is to assign a priority code to every test
case. The priority code may be based on the assumption that “test case of priority
code 1 is more important than test case of priority code 2.” We may have priority
codes as follows:

Priority code 1 : Essential test case

Priority code 2 : Important test case
Priority code 3 : Execute, if time permits
Priority code 4 : Not important test case
Priority code 5 : Redundant test case

There may be other ways for assigning priorities based on customer requirements
or market conditions like:
Priority code 1 : Important for the customer
Priority code 2 : Required to increase customer satisfaction
Priority code 3 : Help to increase market share of the product

We may design any priority category scheme, but a scheme based on technical
considerations always improves the quality of the product and should always be
encouraged.
 REGRESSION TEST CASES SELECTION

Test suite design is an expensive process and its size can grow quite large.
Most of the times, running an entire test suite is not possible as it requires a
significant amount of time to run all test cases. Many techniques are available
for the selection of test cases for the purpose of regression testing. Few
techniques are as follows -

1 Select All Test Cases

This is the simplest technique where we do not want to take any risk. We want
to run all test cases for any change in the program. This is the safest technique,
without any risk. A program may fail many times and every time we will
execute the entire test suite. This technique is practical only when the size of
the test suite is small. For any reasonable or large sized test suite, it becomes
impractical to execute all test cases.

2 Select Test Cases Randomly

We may select test cases randomly to reduce the size of the test suite. We
decide how many test cases are required to be selected depending upon time
and available resources. When we decide the number, the same number of test
cases is selected randomly. If the number is large, we may get a good number
of test cases for execution and testing may be of some use. But, if the number
is small, testing may not be useful at all. In this technique, our assumption
is that all test cases are equally good in their fault detection ability.
However, in most of the situations, this assumption may not be true. We want
to re - test the source code for the purpose of checking the correctness of the
modified portion of the program. Many randomly selected test cases may
not have any relationship with the modified portion of the program.
However, random selection may be better than no regression testing at
all.

3 Select Modification Traversing Test Cases

We select only those test cases that execute the modified portion of the
program and the portion which is affected by the modification(s). Other test
cases of the test suite are discarded. Actually, we want to select all those test
cases that reveal faults in the modified program. These test cases are known
as fault revealing test cases. There is no effective technique by which we can
find fault revealing test cases for the modified program.

We may effectively implement any test case selection technique with the help
of a testing tool. The modified source code and source code affected by
modification(s) may have to be identified systematically and this selected area
of the source code becomes the concern of test case selection. As the size of
the source code increases, the complexity also increases, and need for an
efficient technique also increases accordingly.