Uncertainty Propagation through Software Dependability Models

Digest of Papers. Twenty-Eighth Annual International Symposium on Fault-Tolerant Computing (Cat. No.98CB36224)

We address problems in modelling the reliability of multiple-version software, and present models intended to improve the understanding of the various ways failure dependence between versions can arise. The previous models, by Eckhardt and Lee and by Littlewood and Miller, described what behaviour could be expected "on average" from a randomly chosen pair of "independently generated" versions. Instead, we address the problem of predicting the reliability of a specific pair of versions. The concept of "variation of difficulty" between situations to which software may be subject is central to the previous models cited. We show that it has even more far-reaching implications than previously found. In particular, we consider the practical implications of two phenomena: varying probabilities of failure over input sub-domains or operating regimes; and positive correlation between successive executions of control software. Our analysis provides some practical advice for regulators, and useful insight into non-intuitive aspects of the failure process of diverse software.

Software reliability analysis is performed at various stages during the process of engineering software as an attempt to evaluate if the software reliability requirements have been (or might be) met. In this report, I present a summary of some fundamental black-box and white-box software reliability models. I also present some general shortcomings of these models and suggest avenues for further research.

Software Engineering, IEEE …, 1978

This paper examines the most widely used reliability modeLs. The models discussed fall into two categories, the data domain and the time domain. Besides tracing the historical development of the various models their advantages and disadvantages are analyzed. This includes models based on discrete as weil as continuous probability distributions. How well a given model performs its purpose in a specific economic environment will determine the usefulness of the model. Each of the models is examined with actual data as to the applicability of the error fmding process.

IEEE Transactions on Software Engineering, 1990

In spite of much research effort, there is no universally applicable software reliability growth model which can be trusted to give accurate predictions of reliability in all circumstances. Worse, we are not even in a position to be able to decide a priori which of the many models is most suitable in a particular context. Our own recent work has tried to resolve this problem by developing techniques whereby, for each program, the accuracy of various models can be analyzed. A user is thus enabled to select that model which is giving the most accurate reliability predictions for the particular program under examination. One of these ways of analyzing predictive accuracy, which we call the u-plot, in fact allows a user to estimate the relationship between the predicted reliability and the true reliability. In this paper we show how this can be used to improve reliability predictions in a very general way by a process of recalibration. Simulation results show that the technique gives improved reliability predictions in a large proportion of cases. However, a user does not need to trust the efficacy of recalibration, since the new reliability estimates produced by the technique are truly predictive and so their accuracy in a particular application can be judged using the earlier methods. The generality of this approach would therefore suggest that it be applied as a matter of course whenever a software reliability model is used. Indeed, although this work arose from the need to address the poor performance of soffware reliability models, it is likely to have applicability in other areas such as reliability growth modeling for hardware.

1994

We consider the dependability of programs of an iterative nature. The dependability of software structures is usually analysed using models that are strongly limited in their realism by the assumptions made to obtain mathematically tractable models and by the lack of experimental data. Among the assumptions made, the independence between the outcomes of successive executions, which is often false, may lead to significant deviations of the result obtained from the real behaviour of the program under analysis. Experiments and theoretical justifications show the existence of contiguous failure regions in the program input space and that, for many applications, the inputs often follow a trajectory of contiguous points in the input space. In this work we present a model in which dependencies among input values of successive iterations are taken into account in studying the dependability of iterative software. We consider also the possibility that repeated, non fatal failures may together cause mission failure. We evaluate the effects of these different hypotheses on 1) the probability of completing a fixed-duration mission, and 2) a performability measure.

Software Engineering, IEEE Transactions on, 1985

A number of analytical models have been proposed during the past 15 years for assessing the reliability of a software system. In this paper we present an overview of the key modeling approaches, provide a critical analysis of the underlying assumptions, and assess the limitations and applicability of these models during the software development cycle. We also propose a step-by-step procedure for fitting a model and illustrate it via an analysis of failure data from a mediumsized real-time command and control software system.

2002

Software reliability models (SRMs) are classified into time domain models and counting process models. The time domain model is the stochastic model based on the sequence of inter-failure times. Jelinski and Moranda model [4] and Schick and Wolverton model [13] are the most classical models belonging to this class. On the other hand, the counting process models have gained popularity for describing the stochastic behavior of the number of software failures observed in the testing phase. The most well-known and tractable models are non-homogeneous Poisson process (NHPP) models. Goel and Okumoto [3], Yamada, Ohba and Osaki [15], Musa and Okumoto [9] develop representative NHPP models. These SRMs are based on the different debugging scenarios, and can catch qualitatively typical (but not general) reliability growth phenomena observed in the testing phase of software products. It should be noted, however, that the SRMs based on past observations may not be always useful in the software testing process. Because one cannot catch the global trend of the software failure occurrence phenomena in the initial testing phase. In other words, aunified modeling framework comprising some typical reliability growth patterns should be developed for robust software reliability assessment. Langberg and Singpurwalla [7] show that several SRMs can be comprehensively viewed by adopting aBayesian point of view. Miller [8] extends the Langberg and Singpurwalla's idea and considers an exponential order statistics model. Raftery [12], Kuo and Yang [5] investigate the modeling framework based on the generalized order statistics (GOS), and discuss several parameter estimation methods from the standpoint of both Bayesian and non-Bayesian statistics. In the GOS modeling framework, the SRMs can be characterized by only the fault detection time distribution. This article proposes phase-type SRMs based on the GOS of software failure data. To unify some existing SRMs, the phase-type distribution [10], which represents the software fault detection time distribution, is used to represent the GOS of software failure data. Also, we provide aunified estimation method for model parameters in the phase-type SRMs. The usual estimation method, such as the maximum likelihood estimation (MLE) based on the Newton's method, does not function well in many cases, since the phase-type SRMs often have many model parameters. To overcome this problem, we develop the EM (expectation-maximization) algorithms [1, 2, 11] to compute the maximum likelihood estimates of model parameters.

Handbook of Reliability Engineering

... Siddhartha R. Dalal ... t)/[1 − F(t)]. These models are Markovian but not strongly Markovian, except when F is exponential; minor variations of this case were studied by Jelinski and Moranda [15], Shooman [16], Schneidewind [17], Musa [18], Moranda [19], and Goel and Okomoto ...

IEEE Int'l Symp. Software Reliability Eng, 1991

2009

This paper develops Classical and Bayesian methods for quantifying the uncertainty in reliability for a system of mixed series and parallel components for which both go/no-go and variables data are available. Classical methods focus on uncertainty due to sampling error. Bayesian methods can explore both sampling error and other knowledge-based uncertainties. To date, the reliability community has focused on qualitative statements about uncertainty because there was no consensus on how to quantify them. This paper provides a proof of concept that workable, meaningful quantification methods can be constructed. In addition, the application of the methods demonstrated that the results from the two fundamentally different approaches can be quite comparable. In both approaches, results are sensitive to the details of how one handles components for which no failures have been seen in relatively few tests.