2018 IEEE International Symposium on Software Reliability Engineering Workshops

Software Reliability Assessment: Modeling and

Algorithms
Vidhyashree Nagaraju
Department of Electrical and Computer Engineering
University of Massachusetts Dartmouth
North Dartmouth, MA, USA
email: vnagaraju@[Link]

Abstract—Non-homogeneous Poisson process (NHPP) software model complexity as well as apply stable and efficient algo-
reliability growth models (SRGM) enable quantitative assessment rithms to estimate the parameters of these models.
of the software testing process. Software reliability models rang- Some of the preliminary contributions of this thesis includes
ing from simple to complex have been proposed to characterize framework for heterogeneous single changepoint software re-
failure data that results from a variety of testing factors as well
as non-uniform expenditure of testing effort. In order to predict liability growth models and an improved algorithm for SRGM
the reliability of software accurately, it is important to apply incorporating testing-effort.
models that both characterize the observed failure data well and
make accurate predictions of the future. Efficient and robust A. A Heterogeneous Single Changepoint Software Reliability
algorithms to quickly estimate the model parameters despite Growth Model
inaccuracy in the initial estimates are also highly desirable.
Ultimately, emphasis should be placed on predictive accuracy Most software reliability growth models characterize the
over complexity to best serve users of the research. software testing process as a function of time. However, many
This work presents the results of the preliminary contributions factors influence failure during testing such as the test strategy
of the proposal including: (i) a heterogeneous single changepoint and environment. These factors manifest in the fault detection
framework considering different models before and after the process as changepoints with visibly distinct curves before
changepoint and (ii) comparison of testing effort models with
a simple model as well as a testing effort model fit with an ECM and after a changepoint. Several researchers have proposed
algorithm to emphasize the importance of model predictive accu- non-homogeneous Poisson process software reliability models
racy over increased model complexity. The preliminary findings with one or more changepoints to better characterize data sets.
will be used to serve as the basis of the overall contributions of However, a striking limitation of previous research is that only
the dissertation. homogeneous combinations of failure distributions before and
Index Terms—Non-homogeneous Poisson process, software
reliability growth models, changepoint, testing effort, optimal after changepoints are considered, which is often not the case
release, expectation conditional maximization algorithm, least when real data sets are plotted.
squares estimation Work accomplished to date: Preliminary work on the disser-
tation presents a heterogeneous single changepoint framework,
I. I NTRODUCTION considering different models before and after the changepoint
Critical systems have become increasingly software inten- and applies algorithms to maximize the likelihood of these
sive, necessitating reliable software to ensure proper operation. models. An exponential and S-shaped distributions are con-
Non-homogeneous Poisson process (NHPP) software reliabil- sidered before and after the changepoint. Heterogeneous mod-
ity growth models (SRGM) are commonly used to characterize els are compared with existing homogeneous models using
the fault detection process as a function of testing time, which goodness-of-fit measures such as Akaike information criterion
enables quantitative assessment of software reliability and (AIC). The proposed expectation conditional maximization
assists in making decisions about the software testing process (ECM) algorithm identifies the maximum likelihood estimates
such as mean time to failure, reliability, and optimal release of the model parameters. Online changepoint analysis is also
time. While traditional models characterize the failure data as described. Experimental results suggest that heterogeneous
a function of testing time, the fault detection process can be changepoint models characterize six out of 10 failure data sets
affected by various factors such as non-uniform testing-effort. better, indicating that the heterogeneous modeling framework
Incorporating such factors into software reliability models complements existing homogeneous models well.
tends to increase model complexity and may not characterize
the failure data any better than simpler models. Moreover, B. An Improved Algorithm for Non-homogeneous Poisson
parameter estimation becomes more challenging when models Process Software Reliability Growth Models incorporating
are complex and it is therefore critical to avoid unnecessary Testing-Effort
Many early software reliability models assumed that testing-
This work was partially supported by (i) the Naval Air Warfare Center effort was constant throughout software testing. To remove
(NAVAIR) under contract N00421-16-T-0373 and (ii) the National Science
Foundation under Grant Number #1526128. this assumption, researchers proposed models incorporating

978-1-5386-9443-5/18/$31.00 ©2018 IEEE 166

DOI 10.1109/ISSREW.2018.000-4
testing-effort, yet this significantly increases model complexity alone may not provide an accurate measure of testing-effort.
to the degree that most previous studies utilized a two-step As a result, SRGMs which implicitly relate fault detection
procedure involving least squares estimation (LSE) and algo- to testing-effort through the elapsed testing time may not
rithms such as Newton’s method to estimate the parameters of adequately characterize the fault detection process (Musa
a testing-effort model. This approach may limit the quality of and Okumoto [16], Trachtenberg [17], Everett [18], Tian et
the model fit achieved. Moreover, the trend over the past 30 al. [19]). To overcome this possible limitation, testing-effort
years has been to propose progressively more complex models, models (Brooks and Motley [20], Ohba [21], Tohma et al.
sacrificing practical considerations such as predictive accuracy. [22], Yamada et al. [23]) were proposed, which model fault
Work accomplished to date: A two-step procedure based on detection in terms of the effort dedicated to the tasks associated
the expectation conditional maximization algorithm referred to with testing.
as the ECM/ECM approach to obtain the parameter estimates The optimal release problem was first formulated by Goel
of a software reliability growth model incorporating testing- and Okumoto [24] in the context of software reliability,
effort is proposed. The results of the proposed approach are considering reliability and cost criteria. Koch and Kubat [25]
compared to past methods as well as a simpler model that incorporated a penalty cost for the delay of software re-
does not consider testing-effort to assess if the additional lease. Yamada and Osaki [26] modeled cost and reliability as
complexity introduced by testing-effort functions compromises multiple objectives. Pham [27] explicitly addressed imperfect
predictive accuracy. Results indicate that the ECM/ECM ap- debugging [28], random life cycle duration, and penalty cost.
proach achieves a better goodness of fit with respect to four The models by Pham and Zhang [29], [30] modeled fault
measures, including three predictive measures. In some cases, removal times, warranty cost, and risk of software failures.
the simpler model omitting testing-effort outperforms meth- Huang and Lyu [31], [32] considered testing efficiency as
ods considering testing-effort. These results suggest that the characterized by the generalized logistic testing effort function.
proposed ECM/ECM approach can achieve better parameter Lin et al. [10] derived optimal release policies considering cost
estimates than the previously proposed LSE/MLE approach and reliability criteria for multiple changepoint models, while
and that algorithms to improve model fit and predictive ac- Inoue et al. [33] derived optimal release time expressions for
curacy may better serve users of software reliability growth single changepoint models incorporating testing effort. Yang et
models than models of increased complexity. al. [34] proposed a method to manage the risk associated with
uncertainty in the expected cost. Xie [35] presented a optimal
C. Related Work software release policy under parametric uncertainty, enabling
The changepoint problem was first investigated in 1954 by decisions according to a user’s risk tolerance.
Page [1], while Zhao [2], [3] was the first to study changepoint
problems in software reliability in the context of the Jelinski- II. P ROPOSED S COPE OF D ISSERTATION
Moranda (JM) failure rate model [4]. The model was then The proposed scope of the dissertation and future directions
extended to consider both interfailure and grouped data [5] for includes a framework to model various factors that may
the Goel-Okumoto (GO) model, while Zou [6] studied change- impact the testing process and propose algorithms to improve
point detection for the GO NHPP SRGM with failure time the performance and predictive accuracy of models. Specific
data. Shyur [7] considered an imperfect debugging SRGM contributions include:
with a single changepoint, whereas Wang et al. [8] presented a 1) A comprehensive analysis of single changepoint model
GO changepoint SRGM with imperfect debugging as well as with homogeneous and heterogeneous failure distribu-
methods to perform nonparametric estimation for failure time tions before and after the changepoint.
and grouped data. Other studies considered a single change- 2) An online approach to optimal release that selects an
point model with logistic test-effort function [9], Weibull- appropriate model based on multiple goodness-of-fit
type test effort function [10], and environmental factors [11]. measures in order to inform the decision process.
There have also been studies to estimate the optimal release 3) Multi-stage numerical algorithms that are both robust
times using changepoint models [12]. Most of these studies to error in the initial parameter estimates and computa-
considers a single changepoint model with simple exponential tionally efficient in order to estimate the parameters of
type failure distribution before and after the changepoint. models regardless of their complexity.
Many software reliability growth models characterize the
cumulative number of failures detected as a function of A. Proposed Approach
testing time [13] assuming constant testing-effort throughout Specific approaches and methods to achieve these contribu-
[14]. However, the effort expended during testing may be tions include:
consumed non-uniformly by activities such as generating, 1) Comprehensive analysis of single changepoint software
scheduling and executing test cases, reviewing their results, reliability growth model: Models are generally compared
and notifying developers of the required actions. A testing- based on information theoretic or predictive goodness-of-fit
effort function characterizes effort in terms of various factors measures. Most, but not all, studies [7], [10], [36] use measures
including the project, process, and the personnel [15]. Due to that contain the word predictive in their name even though they
the dependence of testing-effort on several factors, testing time do not employ standard methods from statistics, which fit the

167
model to a subset of the data and use the remaining data to of single changepoint models and submit an article to a
assess the predictive capability of the model. To overcome relevant journal. By this time, I would have also completed my
limitations of the past research, we will perform a systematic comprehensive exam which is comprised of a written exam
comparison between models with a single changepoint and followed by a proposal defense. The online optimal release
with no changepoint as well as between homogeneous and framework is an ongoing work for the past two years and
heterogeneous single changepoint models with respect to the the proposed approach as well as results will be validated by
Akaike information criterion and predictive sum of squares the end of Spring 2019. Finally, the robust numerical algo-
error. A detailed mathematical description of the changepoint rithm framework may take longer since the validation of the
models considering combinations of the Goel-Okumoto and framework requires exhaustive study on various combination
Delayed s-shaped models will be provided. The online change- of models. However, I plan to complete this by the end of Fall
point analysis will be illustrated for homogeneous combina- 2019 during when I plan to submit my dissertation. Numerical
tions of the changepoint models. The predictive capability algorithm work may be extended beyond Fall 2019 depending
of the models with and without changepoint will also be on the computational complexity.
compared to provide an objective perspective on the accuracy
of the release decisions enabled by these models.
2) Online optimal release time for non-homogeneous Pois-
son process software reliability growth model: With the avail-
ability of many software reliability models, it is difficult to se-
lect one best model to characterize the failure data and estima-
tion of optimal release time is critical considering factors such
as time and budget. Most of the previous studies estimate the
optimal release time without necessarily considering the model
that best characterizes the data. Moreover, traditional methods
derive equations to determine if the optimal release time has
passed, but do not suggest how to implement this in a process Fig. 1. PhD timeline
during the testing process. Therefore, we propose to develop
a framework to assess whether or not the software should be
C. Possible threats to validity
released. To accomplish this, we will use a combination of
information theoretic and predictive goodness of fit measures Some of the challenges that may be encountered during
to make a release decision by computing the potential release validation of the proposed research are
times of the competing models and comparing their relative 1) Delay in obtaining software failure data from organi-
goodness of fit. zations to validate the proposed models. However, the
3) Robust multi-stage numerical algorithms: Previous approach can also be validated on publicly available and
software reliability models apply methods such as least simulated datasets.
squares [37] or maximum likelihood estimation [38] with 2) Computational complexity of the robust numerical algo-
algorithms such as Newton’s method [39], metaheuristics [40], rithms on complex models that consider many aspects
machine learning [41], expectation maximization (EM) [42], of software testing such as changepoints, testing-effort,
and expectation conditional maximization (ECM) [43]. These environmental factors, imperfect debugging etc.
alternative algorithms possess advantages and disadvantages. 3) Compatibility in translating the current research into
Some are fast but unstable and may fail to converge, while a Software Failure and Reliability Assessment Tool
others are more stable but slower. Instability is undesirable (SFRAT) programmed using R statistical language.
because failure to converge produces incorrect estimates a
tool user may inadvertently rely on to make decisions, while ACKNOWLEDGMENT
inefficient algorithms slow the user’s work. Multi-stage models This work is advised by Dr. Lance Fiondella, an assistant
will possess greater complexity than traditional models and professor in the Department of Electrical and Computer En-
therefore require an appropriate combination of stability and gineering at University of Massachusetts Dartmouth.
speed. Combinations of two or more existing algorithms to This work was partially supported by (i) the Naval Air
achieve a suitable balance between speed and stability will Warfare Center (NAVAIR) under contract N00421-16-T-0373
enable implementation of more complex multi-stage models and (ii) the National Science Foundation under Grant Number
in a computer-aided tool. #1526128.

B. Timeline and Milestones R EFERENCES

Figure 1 shows the timeline and planned work towards [1] E. Page, “Continuous inspection schemes,” Biometrika, vol. 41, no. 1/2,
pp. 100–115, 1954.
completion of the PhD dissertation. [2] M. Zhao, “Change-point problems in software and hardware reliability,”
By the end of Fall 2018, I intend to complete the framework Communications in Statistics-Theory and Methods, vol. 22, no. 3, pp.
for analysis of homogeneous and heterogeneous combination 757–768, 1993.

168
 [3] ——, “Statistical reliability change-point estimation models,” in Hand- [28] P. Kapur and R. Garg, “Optimal sofware release policies for software re-
book of Reliability Engineering. Springer, 2003, pp. 157–163. liability growth models under imperfect debugging,” RAIRO-Operations
[4] M. Lyu, Ed., Handbook of Software Reliability Engineering. New York, Research, vol. 24, no. 3, pp. 295–305, 1990.
NY: McGraw-Hill, 1996. [29] H. Pham and X. Zhang, “A software cost model with warranty and
[5] Y. Chang, “Estimation of parameters for nonhomogeneous Poisson risk costs,” IEEE Transactions on Computers, vol. 48, no. 1, pp. 71–75,
process: Software reliability with change-point model,” Communications 1999.
in Statistics-Simulation and Computation, vol. 30, no. 3, pp. 623–635, [30] H. Pham, “Software reliability and cost models: Perspectives, compari-
2001. son, and practice,” European Journal of Operational Research, vol. 149,
[6] F. Zou, “A change-point perspective on the software failure process,” no. 3, pp. 475–489, 2003.
Software Testing, Verification and Reliability, vol. 13, no. 2, pp. 85–93, [31] C.-Y. Huang, “Cost-reliability-optimal release policy for software relia-
2003. bility models incorporating improvements in testing efficiency,” Journal
[7] H. Shyur, “A stochastic software reliability model with imperfect- of Systems and Software, vol. 77, no. 2, pp. 139–155, 2005.
debugging and change-point,” Journal of Systems and Software, vol. 66, [32] C.-Y. Huang and M. R. Lyu, “Optimal release time for software systems
no. 2, pp. 135–141, 2003. considering cost, testing-effort, and test efficiency,” IEEE transactions
[8] Z. Wang and J. Wang, “Parameter estimation of some NHPP software on Reliability, vol. 54, no. 4, pp. 583–591, 2005.
reliability models with change-point,” Communications in Statistics, [33] S. Inoue, Y. Nakagawa, and S. Yamada, “Optimal software shipping time
Simulation and Computation, vol. 34, no. 1, pp. 121–134, 2005. estimation based on a change-point hazard rate model,” International
[9] C. Huang, “Performance analysis of software reliability growth models Journal of Reliability, Quality and Safety Engineering, vol. 21, no. 2,
with testing-effort and change-point,” Journal of Systems and Software, 2014.
vol. 76, no. 2, pp. 181–194, 2005. [34] B. Yang, H. Hu, and L. Jia, “A study of uncertainty in software cost
[10] C. Lin, C. Huang, and J. Chang, “Integrating generalized Weibull- and its impact on optimal software release time,” IEEE Transactions on
type testing-effort function and multiple change-points into software Software Engineering, vol. 34, no. 6, pp. 813–825, 2008.
reliability growth models,” in IEEE Asia-Pacific Software Engineering [35] M. Xie, X. Li, and S. Ng, “Risk-based software release policy under
Conference, 2005. parameter uncertainty,” Proceedings of the Institution of Mechanical
[11] J. Zhao, H. Liu, G. Cui, and X. Yang, “Software reliability growth model Engineers, Part O: Journal of Risk and Reliability, vol. 225, no. 1,
with change-point and environmental function,” Journal of Systems and pp. 42–49, 2011.
Software, vol. 79, no. 11, pp. 1578–1587, 2006. [36] S. Inoue and S. Yamada, “Software reliability growth modeling with
change-point and its goodness-of-fit comparisons,” in Software Engi-
[12] S. Inoue and S. Yamada, “Software reliability assessment with multiple
neering, Business Continuity, and Education. Springer, 2011, pp. 354–
changes of testing-environment,” IEICE Transaction on Fundamentals
361.
of Electronics, Communications and Computer Sciences, vol. 98, no. 10,
[37] D. Kleinbaum, L. Kupper, A. Nizam, and K. Muller, Applied regression
pp. 2031–2041, 2015.
analysis and other multivariable methods, 4th Edition. Australia:
[13] J. D. Musa, “A theory of software reliability and its application,” IEEE
Thomson Learning Inc. Brooks/Cole, 2008.
transactions on software engineering, no. 3, pp. 312–327, 1975.
[38] R. Hogg, J. McKean, and A. Craig, Introduction to mathematical
[14] S. S. G. P. N. Marinos and K. S. Trivedi, “Important milestones in statistics. (6th ed). Upper Saddle River, New Jersey: Prentice Hall,
software reliability modeling,” Proceedings of Software Engineering 2005.
and Knowledge Engineering (SEKE’96), Lake Tahoe, NV, pp. 345–352, [39] R. Burden and J. Faires, Numerical Analysis, 8th ed. Belmont, CA:
1996. Brooks/Cole, 2005.
[15] B. W. Boehm et al., Software engineering economics. Prentice-hall [40] T. Minohara and Y. Tohma, “Parameter estimation of hyper-geometric
Englewood Cliffs (NJ), 1981, vol. 197. distribution software reliability growth model by genetic algorithms,”
[16] J. Musa and K. Okumoto, “A logarithmic Poisson execution time model in IEEE International Symposium on Software Reliability Engineering,
for software reliability measurement,” in International Conference on 1995, pp. 324–329.
Software Engineering, 1984, pp. 230–238. [41] Y.-S. Su and C.-Y. Huang, “Neural-network-based approaches for
[17] M. Trachtenberg, “A general theory of software-reliability modeling,” software reliability estimation using dynamic weighted combinational
IEEE Transactions on Reliability, vol. 39, no. 1, pp. 92–96, 1990. models,” Journal of Systems and Software, vol. 80, no. 4, pp. 606–615,
[18] W. Everett, “An extended execution time software reliability model,” 2007.
in IEEE International Symposium on Software Reliability Engineering, [42] A. Dempster, N. Laird, and D. Rubin, “Maximum likelihood from
1992, pp. 4–13. incomplete data via the EM algorithm,” Journal of the Royal Statistical
[19] J. Tian, P. Lu, and J. Palma, “Test-execution-based reliability measure- Society: Series B, vol. 39, no. 1, pp. 1–38, 1977.
ment and modeling for large commercial software,” IEEE Transactions [43] X.-L. Meng and D. Rubin, “Maximum likelihood estimation via the
on Software Engineering, vol. 21, no. 5, pp. 405–414, 1995. ECM algorithm: A general framework,” Biometrika, vol. 80, no. 2, pp.
[20] W. Brooks and R. Motley, “Analysis of discrete software reliability 267–278, 1993.
models (radc-tr-80-84),” IBM Federal Systems Div, MD, USA, Tech.
Rep., 1980.
[21] M. Ohba, “Software reliability analysis models,” IBM Journal of Re-
search and Development, vol. 28, no. 4, pp. 428–443, 1984.
[22] Y. Tohma, R. Jacoby, Y. Murata, and M. Yamamoto, “Hyper-geometric
distribution model to estimate the number of residual software faults,”
in IEEE International Computer Software and Applications Conference,
1989, pp. 610–617.
[23] S. Yamada, H. Ohtera, and H. Narihisa, “Software reliability growth
models with testing-effort,” IEEE Transactions on Reliability, vol. R-
35, no. 1, pp. 19–23, apr 1986.
[24] K. Okumoto and A. Goel, “Optimum release time for software systems
based on reliability and cost criteria,” Journal of Systems and Software,
vol. 1, no. 4, pp. 315–318, 1980.
[25] H. S. Koch and P. Kubat, “Optimal release time of computer software,”
IEEE Transactions on Software Engineering, no. 3, pp. 323–327, 1983.
[26] S. Yamada and S. Osaki, “Optimal software release policies with
simultaneous cost and reliability requirements,” European Journal of
Operational Research, vol. 31, no. 1, pp. 46–51, 1987.
[27] H. Pham, “A software cost model with imperfect debugging, random
life cycle and penalty cost,” International Journal of Systems Science,
vol. 27, no. 5, pp. 455–463, 1996.

169