Software Testing Based on Formal Specification

2009

Abstract Formal methods and testing are two important approaches that assist in the development of high-quality software. While traditionally these approaches have been seen as rivals, in recent years a new consensus has developed in which they are seen as complementary. This article reviews the state of the art regarding ways in which the presence of a formal specification can be used to assist testing.

TAPSOFT '95: Theory and Practice of Software Development, 1995

The paper presents a theory of program testing based on formal specifications. The formal semantics of the specifications is the basis for a notion of an exhaustive test set. Under some minimal hypotheses on the program under test, the success of this test set is equivalent to the satisfaction of the specification. The selection of a finite subset of the exhaustive test set can be seen as the introduction of more hypotheses on the program, called selection hypotheses. Several examples of commonly used selection hypotheses are presented. Another problem is the observability of the results of a program with respect to its specification: contrary to some common belief, the use of a formal specification is not always sufficient to decide whether a test execution is a success. As soon as the specification deals with more abstract entities than the program, program results may appear in a form which is not obviously equivalent to the specified results. A solution to this problem is proposed in the case of algebraic specifications.

1999

A recent method combines model checkers with specification-based mutation analysis to generate test cases from formal software specifications. However high-level software specifications usually must be reduced to make analysis with a model checker feasible. We propose a new reduction, parts of which can be applied mechanically, to soundly reduce some large, even infinite, state machines to manageable pieces. Our work differs from other work in that we use the reduction for generating test sets, as opposed to the typical goal of analyzing for properties. Consequently, we have different criteria, and we prove a different soundness rule. Informally, the rule is that counterexamples from the model checker are test cases for the original specification. The reduction changes both the state machine and temporal logic constraints in the model checking specification to avoid generating unsound test cases. We give an example of the reduction and test generation. 1 * Partially supported by the National Science Foundation under grant number CCR-99-01030 feasible. Our approach is to define a reduction from a given specification to a smaller one that is more likely to be tractable for a model checker. We tailor the reduction for test generation, as opposed to the usual goal of analysis. A broad span of research from early work on algebraic specifications [13] to more recent work such as [21] addresses the problem of relating tests to formal specifications. In particular, counterexamples from model checkers are potentially useful test cases. In addition to our use of the Symbolic Model Verifier (SMV) model checker [19] to generate mutation adequate tests [2], Callahan, Schneider, and Easterbrook use the Simple PROMELA Interpreter (SPIN) model checker [16] to generate tests that cover each block in a certain partitioning of the input domain [8]. Gargantini and Heitmeyer use both SPIN and SMV to generate branch-adequate tests from Software Cost Reduction (SCR) requirements specifications [14]. The model checking approach to formal methods has received considerable attention in the literature, and readily available tools such as SMV and SPIN are capable of handling the state spaces associated with realistic problems [11]. Although model checking began as a method for verifying hardware designs, there is growing evidence that model checking can be applied with considerable automation to specifications for relatively large software systems, such as the Traffic Alert &: Collision Avoidance System (TCAS) II [9]. The increasing usefulness of model checkers for software systems makes them attractive targets for use in aspects of software development other than pure analysis, which is their primary role today. Model checking has been successfully applied to a wide variety of practical problems, including hardware design, protocol analysis, operating systems, reactive system analysis, fault tolerance, and security. The chief advantage of

Software Testing, Verification and Reliability, 2016

In the context of testing from algebraic specifications, a first natural testing hypothesis is to suppose that programs and test cases are denoted respectively, by Σ-algebras and ground formulas. Hence, the test case interpretation is defined with respect to the formula satisfaction. So, the set of test cases is a subset of semantic consequences restricted to ground formulas with often further observational conditions on sorts to denote the ones equipped with an equality in the targeted programming language. This subset should ensure program correctness with respect to their specifications, i.e. for every uncorrect program P , there exists a test case that P interprets as "false" by a computation. We then say that this set of test cases is exhaustive. The main interest of such an exhaustive test set is that by submitting any of its test cases, correct programs cannot be rejected as incorrect or dually incorrect programs cannot be accepted as correct. Hence, it is appropriate to start the process of selecting test sets with reasonable sizes. However, depending on the nature of specifications and programs, this subset is not necessarily exhaustive. In this paper, we study conditions to ensure this exhaustiveness property for several algebraic formalisms (equational, conditional positive, quantifier-free and with quantifiers).

Lecture Notes in Computer Science, 1985

A B~C T We present a method and a tool for generating test sets from algebraic data type specifications. We give formal definitions of the basic concepts required in our approach of functional testing. Then we discuss the problem of testing algebraic data types implementations. This allows the introduction of additional hypotheses and thus the description of an effective method for generating test sets. The method can be improved by using PROLOG. Indeed, it turns out that PROLOG is a very well suited tool for generating test sets in this context. Applicability of the method is discussed and a complete example is given.

2012 International Conference on Systems and Informatics (ICSAI2012), 2012

A program required to be tested in practice often has no available source code for some reason and how to adequately test such a program is still an open problem. In this paper, we describe a formal speci cation-based testing approach to tackle this challenge. The principal idea is rst to formalize the informal requirements into formal operation speci cations that take the interface scenarios of the program into account, and then utilize the speci cations for test case generation and test result analysis. An example and case study of applying the approach to an IC card system is presented to illustrate its usage and analyze its performance.

In the context of testing from algebraic specifications, a first natural testing hypothesis is to suppose that programs and test cases are denoted respectively, by Σ-algebras and ground formulas. Hence, the test case interpretation is defined with respect to the formula satisfaction. So, the set of test cases is a subset of semantic consequences restricted to ground formulas with often further observational conditions on sorts to denote the ones equipped with an equality in the targeted programming language. This subset should ensure program correctness with respect to their specifications, i.e. for every uncorrect program P , there exists a test case that P interprets as "false" by a computation. We then say that this set of test cases is exhaustive. The main interest of such an exhaustive test set is that by submitting any of its test cases, correct programs cannot be rejected as incorrect or dually incorrect programs cannot be accepted as correct. Hence, it is appropria...

At the end of a software development process, there is a need for software verification. The aim of the software verification is to assess the software product to determine conformance to its specification. There are a few methods which can be used for doing this. However, the most popular method is software testing. For software testing to be done effectively, there is a need to select proper test cases such that all aspects of the software can be tested. In this paper we describe a technique for generating test cases automatically from the formal specification of the software. We also describe a testing system based on this technique.

2003

Our work focuses on the use of formal techniques in order to increase the quality of HCI software and of all the processes resulting from the development, verification, design and validation activities. This paper shows how the B formal technique can be used for user tasks modelling and validation. A trace based semantics is used to describe either the HCI or the user tasks. Each task is modelled by a sequence of fired events. Each event is defined in the abstract specification and design of the HCI system.

… and Formal Methods, 2007. SEFM 2007, 2007