A Case Study on Regression Test Suite Maintenance in System

Evolution

Mats Skoglund Per Runeson

Dept. of Computer and Systems Sciences Dept. of Communication Systems
Stockholm University/Royal Institute of Technology Lund University
Forum 100, 164 40 Kista, Sweden Box 118, 221 00 Lund, Sweden
[email protected] [email protected]

Abstract ferent change strategies. For example, the change may

start from the higher-level parts (the GUI) or start
When a system is maintained, its automated test with the lower levels first. Different strategies may lead
suites must also be maintained to keep the tests up to to different number of changes required to update the
date. Even though practice indicates that test suite main- system. In a study performed by Rajlich and Gosavi
tenance can be very costly we have seen few studies [10] three different change strategies were compared.
considering the actual efforts for maintenance of test- In their study one of the change strategies studied re-
ware. We conducted a case study on an evolving system quired less number of changes compared to the other
with three updated versions, changed with three differ- two and thus was the most cost effective strategy for
ent change strategies. Test suites for automated unit and making changes to the system. However, in that study
functional tests were used for regression testing the ex- the tests were not updated and hence one may argue
tended applications. With one change strategy more that the maintenance cycle is not completely finished.
changes were made in the tests code than in the sys- To address some of the issues regarding test suite
tem that was tested, and with another strategy no changes maintenance we launched a case study with a two-
were needed for the unit tests to work. folded goal. First, it is an exploratory study to inves-
tigate issues regarding test suite maintenance. Second,
it is a follow-up study to Rajlich and Gosavi’s study,
1. Introduction where the impacts of the change strategies are investi-
gated when the maintenance cycle is taken further, i.e.
Most systems developed today are developed in an when the regression tests are updated and executed.
evolutionary fashion [13, 9]. Testing and test automa- Section 2 of this paper presents related work and
tion is an area that specifically benefits from the evo- the case study is described in Section 3. General ob-
lutionary approach. By enabling reuse of investments servations are reported in Section 4 and details of the
made in test programs and environments, higher costs case study operations are presented in Section 5. Sec-
for developing test automation support can be justi- tion 6 discusses the results and Section 7 gives direc-
fied and hence higher quality software can be achieved. tions for future work.
Testing of existing functionality from an earlier re-
lease of a program is referred to as regression test-
ing. When using automated tests, regression tests may 2. Related Work
be conducted very efficiently according to Fewster and
Graham [5, page 9]. However, they also conclude that Fewster and Graham [5, chap. 7] give an overview of
maintenance costs are more significant for automated issues related to how to build maintainable test. They
testing than for manual [5, page 191]. Despite this, we provide good advice, probably based on practical ex-
have seen very few studies considering the actual ef- perience, but little empirical justification is provided.
forts for maintenance of testware in an evolutionary Binder proposes a systematic method for regression
environment. test suite maintenance [2, page 769]. This approach is
One issue which may impact for the test suite evolu- however neither further analyzed with respect to effi-
tion is that a software system may be changed with dif- ciency nor with respect to success at all.

Proceedings of the 20th IEEE International Conference on Software Maintenance (ICSM’04)

1063-6773/04 $20.00 © 2004 IEEE
 Regression test suite reduction is another area that of the system. The same ownership feature was imple-
intends to select a subset of regression test cases, with- mented three times, using three different change strate-
out considering the automation, see e.g. [6, 12]. gies, resulting in three different new versions of the
The “test-first” approach to software development system. These three change strategies were: Change
has appeared within eXtreme Programming (XP) [1]. propagation, Refactoring and Role splitting. With the
The principle is that test cases for automated unit test Change propagation strategy the programmer changes
are written first, then the code for implementing the a class and then visits all classes interacting with the
function is written. In combination with refactoring, changed class to investigate whether they also need
i.e. continuous change of the code and the testware, to be changed. The Refactoring strategy groups code
the test suites and the system are maintained at the affected by a change into fewer classes. Role splitting
same time. strategy is based on the assumption that a class or a
method may be used in different roles and thus an up-
3. The Case Study date should only affect one of the roles. By splitting a
method into two similar versions only one of them may
The case study was conducted on the “Drawlet” need to be changed while the other one can remain the
framework, a system that has been proposed as a test- same, leading to fewer changes. Details regarding the
bed for software evolution case studies [4]. The Drawlet change strategies and the changes that were made in
framework is a Java version of the original drawing the three different versions can be found in [10, 8].
program “HotDraw”, developed by Beck and Cunning- For this case study the original system was viewed as
ham1 . It is an extensible framework that may be used the baseline and the other three versions as enhanced
to add drawing functionality to a host application. This versions ready for regression testing. This case study
package includes the complete source code of about focuses on the regression part of the testing, thus new
17800 LOC for the system as well as test code (ap- tests that should be developed to test the new func-
proximately 4800 LOC) for unit testing with the JU- tionality is outside the scope of this paper.
nit testing framework [7]. It also includes host appli-
cations with graphical user interfaces as examples on 3.2. Testing
how to use the framework, see Figure 1.
In the baseline package there are 19 source files
giving 385 unit tests with 1369 assertions testing the
classes in the system2 .
No functional tests were included in the baseline
package and to our knowledge no such tests for the sys-
tem are available elsewhere. Thus, specification-based
functional tests were derived from the specification of
the Drawlet system found in [8]. The procedure for cre-
ating the functional tests is detailed in an extended ver-
sion of this paper [14].
A total number of 141 functional test cases were
created for the functions of the baseline and realised
Figure 1. The Drawlet host application used and automated using a record/playback testing tool
for Java applications named Abbot3 .

3.3. Case Study Operation and Analysis

3.1. Evolution Procedure

The functionality that was added in the study by The case study operation is divided into four phases:
Rajlich and Gosavi was intended to give users owner- environment setup, build configuration, execute unit test
ship to the figures they create, thus hindering other and execute functional tests, see Figure 2.
users from manipulating them [10]. The new feature
was implemented by modification of about 100 lines 2 In JUnit terminology an assertion is a test that is checked and
of code but had big impact on the overall concepts passed, an error is the result of an unanticipated problem dur-
ing the execution of a test and a failure is a test that was exe-
cuted but did not pass.
1 http://www.rolemodelsoftware.com/drawlets 3 http://abbot.sourceforge.net

Proceedings of the 20th IEEE International Conference on Software Maintenance (ICSM’04)

1063-6773/04 $20.00 © 2004 IEEE
 Change prop. (91) Role splitting (87)
environment
Compile Execute Compile Execute
build
setup configuration
unsuccessful change Test LOC. 82 42 0 0
unit

successful
tests
Table 1. Modifications to make the tests compile
execute unsuccessful
change and execute for Change propagation and Role
unit
unit tests
tests
splitting. Number of LOC modified to implement
successful the system changes is displayed in parentheses

execute unsuccessful change

functional functional
tests tests ment increases the threats of researcher bias. The re-
successful

searcher has in this case worked with the systems dur-

ing 6 months part time to evolve it, however, focused
Done on understanding which changes should be done in the
Figure 2. Case Study Process interfaces towards the test programs.
We consider the results in general being indepen-
dent of the researchers. Still it depends on the certain
The procedure was repeated with both the Change case observed, but this is inherent in the case study
propagation and the Role splitting systems, of which methodology.
the Change propagation version was randomly chosen
to be tested first. Our analysis focused on the changes
that were made in order to build the new configura-
4. General Observations
tions and to execute the tests on the updated versions
Change propagation. The baseline tests could not be
of the system. The task of adapting the tests to the up-
successfully built with the change propagation version
dated systems was recorded by logging the steps of the
of the system. The Java compiler reported 246 errors in
work during its progress. This log was then analysed to-
the test code and thus the tests needed to be adapted.
gether with the different versions of the system and the
When these compilation errors were corrected the
test cases.
unit tests were executed using the JUnit testrunner.
The first test run resulted in 10 errors and 110 fail-
3.4. Threats to Validity ures. Thus further changing of the test code was re-
quired. Numerical data regarding the changes is dis-
We used a model by Padgett, summarized by Rob- played in Table 1.
son [11], to review and control the validity in the cur- In [10] it is reported that 91 lines of code (0.5%) were
rent work. The model analyses three kinds of threats modified in the baseline to create the Change propa-
to validity: reactivity, researcher bias and respondent gation version of the system. We modified 124 lines
bias, of which the latter type is the only one applica- (2.5%) in the test code to adapt the tests to these
ble to the current study. changes in the baseline. Thus we modified approxi-
For researcher bias threats, triangulation, peer de- mately 1.4 test code lines per modified system code
briefing, member checking, negative case analysis and line.
audit trail are actions taken to reduce threats.
The audit trail strategy is adopted during the case Refactoring. The baseline tests could not be built with
study, i.e. full record of activities during the study is the Refactoring version of the system due to e.g.
kept, using an engineering logbook to enable tracing of changed method interfaces. It was however possible
the events during the case study. This is also a source to build and execute the system with the tests we
for the negative case analysis, which means that alter- had adapted to the Change propagation version. The
native explanations are sought for during peer debrief- Refactoring system was created by modifications of 95
ing, where the second author has acted as a quality source lines, i.e. approximately the same number of
assurance person for the case study. The triangula- lines changed as in the Change propagation version.
tion strategy is applied by having several versions of Role splitting. The baseline tests and the Role split-
the same system and apply the case study to those in ting version of the system could directly be success-
a randomized order. The main strategy for reducing fully built and executed without making any changes
the threats of researcher bias is by using the Drawlets to either the test code or the system, see Table 1. It was
system, which is developed by others, and successfully also possible to build and execute this version together
used in an earlier case study [10]. The time of involve- with the tests we had adapted to the Change propaga-

Proceedings of the 20th IEEE International Conference on Software Maintenance (ICSM’04)

1063-6773/04 $20.00 © 2004 IEEE
 tion version without errors or failures. This was pos- Another change required to make the tests run was
sible since the strategy used for changing the system that some compound statements with methods calls
was to create new methods instead of changing the ex- had to be split up due to how the new ownership con-
isting ones when implementing the new ownership role. cept was implemented.
Hence, there were methods that could be called directly More details regarding the case study operation can
from the baseline test code as well as new methods that be found in [14].
could be called by the adapted test suite code. In [10]
it is reported that 87 lines of code were changed to cre-
6. Discussion
ate this version. Thus the Role splitting strategy re-
quired the least number of lines to be changed both in The protocol changes, i.e. method signature
the system and in the tests. changes, caused compilation of tests to fail since meth-
Functional Tests. The first event when each of the up- ods that were called in the original test code did not ex-
dated versions begin to execute is that a dialog box was ist any longer. However, in one version, changed with
displayed where a session owner id and name should be the Role splitting strategy, methods were added in-
entered followed by OK. Since the program stops the stead of changed and thus the existing methods’
execution and waits for the dialog box to be closed this protocols remained the same and the tests could com-
also stops the execution of the tests. Thus we had to ex- pile without errors. This led to that the Role splitting
tend the functional test configuration to complete the strategy was in this case study superior the oth-
dialog box to make the tests to run. Since the tests were ers in terms of effort required to make the regression
executed one after another, without closing the appli- tests run. However, nothing can be said of the qual-
cation in between, the dialog box only needed to be ity of the (adapted) baseline tests when running them
handled once at the beginning of the execution of the on the new versions since no analysis of what the base-
functional tests. All functional tests could be executed line tests actually test in the new versions were per-
without further changes to the tests. formed in this case study.
Changing the test code to call the new methods
5. The Case Study Operation in Details was not made by search and replace of method names
since there were method calls that should not be
First, the baseline system was installed and executed changed. This is an implication of the feature in object-
to verify that it was runnable and usable. Then the unit orientation that several methods may have the same
and functional tests were verified to work with the base- name in different classes, or even within the same class.
line version. The next step was to install and regres- This hinders the use of a naive search and replace.
sion test the Change propagation version. The extension of the number of parameters to meth-
ods was fixed by creating a dummy session owner that
Compiling. The only type of compilation error re- was assigned a value of our choice. However, we have
ported from the compiler was a cannot resolve sym- not analysed whether the test code requires more so-
bol for methods that had change names and been ex- phisticated changes than just passing the extra param-
tended with an extra parameter. All 246 compilation eter in order to actually perform the same tests on the
errors were fixed by fixing the code the same way, i.e. new implementation as it did in the baseline. Not sur-
the calls were changed to the new names and newly cre- prisingly, a conclusion is that a good knowledge of the
ated dummy variables passed as extra arguments. changed implementation is required during test main-
Executing the Tests. One common error found 35 times tenance in order to actually make the test cases per-
was a Java NullPointerExceptions. This was a result form the same tests on the new implementation.
of the baseline test code had not assigned any owner The internal implementation changes inside meth-
to the figures involved in the tests. ods led to problems when executing the unit test cases.
In some circumstances the session owner must be set The adaptation of the tests to set the session owner in
in the canvas in the GUI in order for the updated sys- the canvas led to the tester must supervise the exe-
tem to work properly. However, the variable represent- cution of the tests since the session owner dialog box
ing the session owner in the canvas class was declared must be completed 10 times for the tests to complete.
as protected and thus could not be set directly from The only change to our functional tests was the ad-
the test code. The only public method we could use dition of response to the dialog box appearing when
to set the session owner object was a method that dis- the application starts. Thus the implementation of the
played the login dialog box, thus halting further exe- ownership features had only minor affects on our func-
cution of the tests. tional test cases. However, should we had chosen to in-

Proceedings of the 20th IEEE International Conference on Software Maintenance (ICSM’04)

1063-6773/04 $20.00 © 2004 IEEE
 clude the tool buttons and other graphical elements in vestigate the quality of the resulting test suites. Inter-
the assertions, then all of our test cases would have esting issues would be to analyse whether the adapted
failed since there were two new buttons added to the tests actually tests the same features in the updated
top toolbar in the new versions of the system. Thus, we system as they did in the baseline.
agree with among others Fewster and Graham [5] and
Burk and Coyner [3] that a tester should be very care- Acknowledgments
ful when the GUI is used in automated tests.
If failures are found in an industrial setting the as- The authors would like to thank Václav Rajlich and
sumption may be that the failures are due to problems Prashant Gosavi for giving us access to the three up-
in the system under test. This assumption can turn out dated Drawlet versions we used in this case study.
to be costly if it shows after a long time that the fail-
ures were instead due to problems with the tests. In References
our case study we made the assumption that all er-
rors and failures were due to problems with the test [1] K. Beck. Embracing change with extreme programming.
code. There were however some faults in the new im- IEEE Computer, 32(10):70 – 77, 1999.
plementations, and perhaps it took us longer to cor- [2] R. V. Binder. Testing Object-Oriented Systems – Mod-
rect these due to our faulty assumption. Thus, during els, Patterns, and Tools. Addison-Wesley, 2000.
regression testing and test maintenance the best as- [3] E. M. Burke and B. M. Coyner. Java Extreme Program-
ming Cookbook. O’Reilly, 2003.
sumption should rather be that errors and failures are
[4] S. Demeyer, T. Mens, and M. Wermelinger. Towards a
due to the system under test, the test code, or both.
software evolution benchmark. In Proc. Int. Workshop
Further, the faults in the changed system caused some on Principles of Software Evolution, 2001.
of the functional tests to fail. This hindered further ex- [5] M. Fewster and D. Graham. Software Test Automation.
ecution of the tests since the tests were dependent on Addison-Wesley, 1999.
each other. This could be solved by not creating se- [6] P. G. Frankl, G. Rothermel, K. Sayre, and F. I. Voko-
quences of tests as we did but instead explicitly set up los. An emprical comparison of two safe regression test
the prerequisites for each test case independently and selection techniques. In Proceedings of IEEE/ACM In-
run them independently. However, in this case we only ternational Conference on Empirical Software Enginer-
needed to add one prerequisite for the functional test to ing, pages 195 – 204, 2003.
make them all run. If all tests would have been set up [7] E. Gamma and K. Beck. The JUnit test framework.
independently, the completion of the dialog box would http://www.junit.org/index.htm.
probably have been required to be added for each of the [8] P. Gosavi. Case study of change propagation in object
independent test cases and this would have required a oriented software. Master’s thesis, Wayne State Univer-
larger effort. sity, 2003.
[9] V. Rajlich and K. Bennett. A staged model for the soft-
ware life cycle. IEEE Computer, 33(7):66 – 71, 2000.
7. Future Work [10] V. Rajlich and P. Gosavi. A case study of unanticipated
incremental change. In Proceedings of the International
The Role splitting strategy seems to have positive ef- Conference on Software Maintenance (ICSM’02), pages
fects on the efforts required to make changes to systems 442–451, 2002.
and regression test suites. It would be interesting to in- [11] C. Robson. Real World Research: A Resource for So-
vestigate the effects of using the Role splitting strategy cial Scientists and Practitioner-Researchers. Blackwell
for several subsequent releases of a system with respect Publisher, 2nd edition, 1999.
to system degeneration. [12] G. Rothermel and M. J. Harrold. Analyzing regression
We believe that some of the problems found would test selection techniques. IEEE Transactions on Soft-
ware Engineering, 22(8):529 – 551, 1996.
have been discovered earlier if a test-first approach had
[13] P. Runeson, C. Andersson, and M. Höst. Test processes
been used, e.g. the compound statement split-up and
in software product evolution - a qualitative survey on
the requirement to complete a dialog box during the the state of practice. Journal of Software Maintenance
execution of the tests. A direction for future work is and Evolution, 15(1):41–59, 2003.
to investigate whether these kinds of problems are re- [14] M. Skoglund and P. Runeson. A case study on testware
duced when a test-first approach is used. maintenance and change strategies in system evolution.
This case study was aimed at investigating the ef- Technical Report 04-007, Department of Computer and
fort required to adapt baseline tests to make them com- Systems Sciencies (DSV), 2004.
pile and execute together with new versions of a sys-
tem. An interesting follow-up study would be to in-

Proceedings of the 20th IEEE International Conference on Software Maintenance (ICSM’04)

1063-6773/04 $20.00 © 2004 IEEE