Relative Correctness: A Bridge Between Proving and Testing

2016

Relative correctness is the property of a program to be morecorrect than another with respect to a specification. Whereas traditionally we distinguish between two categories of candidate programs, namely correct programs and incorrect programs, relative correctness arranges candidate programs on a partial ordering structure, whose maximal elements are the correct programs. Also, whereas traditionally we deploy proof methods on correct programs to prove their correctness and we deploy testing methods on incorrect programs to detect and remove their faults, relative correctness enables us to bridge this gap by showing that we can deploy static analytical methods to an incorrect program to prove that while it may be incorrect, it is still more-correct than another. We are evolving a technique, called debugging without testing, in which we can remove a fault from a program and prove that the new program is more-correct than the original, all without any testing (and its associated uncer...

2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, 2015

In the process of trying to define what is a software fault, we have found that to formally define software faults we need to introduce the concept of relative correctness, i.e. the property of a program to be more-correct than another with respect to a given specification. A feature of a program is a fault (for a given specification) only because there exists an alternative to it that would make the program more-correct with respect to the specification. In this paper, we explore applications of the concept of relative correctness in program testing, program repair, and program design. Specifically, we argue that in many situations of software testing, fault removal and program repair, testing for relative correctness rather than absolute correctness leads to clearer conclusions and better outcomes. Also, we find that designing programs by stepwise correctness-enhancing transformations rather than by stepwise correctness-preserving refinements leads to simpler programs and is more tolerant of designer mistakes.

IEEE Transactions on Software Engineering, 37 (1): 109-125, 2011

We present an integrated method for program proving, testing, and debugging. Using the concept of metamorphic relations, we select necessary properties for target programs. For programs where global symbolic evaluation can be conducted and the constraint expressions involved can be solved, we can either prove that these necessary conditions for program correctness are satisfied, or identify all inputs that violate the conditions. For other programs, our method can be converted into a symbolic testing approach. Our method extrapolates from the correctness of a program for tested inputs to the correctness of the program for related untested inputs. The method supports automatic debugging through the identification of constraint expressions that reveal failures.

Electronic Proceedings in Theoretical Computer Science, 2016

Relative correctness is the property of a program to be more-correct than another with respect to a given specification. Whereas the traditional definition of (absolute) correctness divides candidate program into two classes (correct, and incorrect), relative correctness arranges candidate programs on the richer structure of a partial ordering. In other venues we discuss the impact of relative correctness on program derivation, and on program verification. In this paper, we discuss the impact of relative correctness on program testing; specifically, we argue that when we remove a fault from a program, we ought to test the new program for relative correctness over the old program, rather than for absolute correctness. We present analytical arguments to support our position, as well as an empirical argument in the form of a small program whose faults are removed in a stepwise manner as its relative correctness rises with each fault removal until we obtain a correct program.

2002

Automated verification tools vary widely in the types of properties they are able to analyze, the complexity of their algorithms, and the amount of necessary user involvement. In this paper we propose a framework for step-wise automatic verification and describe a lightweight scalable program analysis tool that combines abstraction and model checking. The tool guarantees that its True and False answers are sound with respect to the original system.

2016 IEEE International Conference on Software Testing, Verification and Validation (ICST), 2016

It is so inconceivable to debug a program without testing it that these two words are used nearly interchangeably. Yet we argue that using the concept of relative correctness we can indeed remove a fault from a program and prove that the fault has been removed, by proving that the new program is more correct than the original. This is a departure from the traditional roles of proving and testing methods, whereby static proof methods are applied to a correct program to prove its correctness, and dynamic testing methods are applied to an incorrect program to expose its faults.

2005

ACM Sigsoft Software Engineering Notes, 2000

We study how program analysis can be used to:

International Conference on Software Engineering, 1976

This paper explores a technique for proving the correctness and termination of programs simultaneously. This approach, the intermittentassertion method, involves documenting the program with assertions that must be true at some time when control passes through the corresponding point, but that need not be true every time. The method, introduced by BurstaH, promises to provide a valuable complement to the more conventional methods. The intermittent-assertion method is presented with a number of examples of correctness and termination proofs. Some of these proofs are markedly simpler than their conventional counterparts. On the other hand, it is shown that a proof of correctness or termination by any of the conventional techniques can be rephrased directly as a proof using intermittent assertions. Finally, it is shown how the intermittent-assertion method can be applied to prove the validity of program transformations and the correctness of continuously operating programs.

2000

Much research in computer science, ever since its inception, has been devoted the problem:���How can we be sure that a computer program is correct?��� The general problem is extremely difficult, and the enormous variety of computer software in use demands a corresponding variety of approaches: eg structured design methods [YC86], automated testing [Ber91] and model checking [GL94].