Property-based testing

The Robot Operating System (ROS) is an open source framework for the development of robotic software, in which a typical system consists of multiple processes communicating under a publishersubscriber architecture. A great deal of development time goes into orchestration and making sure that the communication interfaces comply with the expected contracts (e.g. receiving a message leads to the publication of another message). Orchestration mistakes are only detected during runtime, stressing the importance of component and integration testing in the verification process. Property-based Testing is fitting in this context, since it is based on the specification of contracts and treats tested components as black boxes, but there is no support for it in ROS. In this paper, we present a first approach towards automatic generation of test scripts for property-based testing of various configurations of a ROS system. CCS CONCEPTS • Software and its engineering → Software testing and debugging; Publish-subscribe / event-based architectures; • Computer systems organization → Robotics;

In this article, we report from a 22-months long action research study in which we evaluate the usefulness of a set of software development tools in an industrial setting, a small software company. We focus on how developers in the industry use and adopt these tools, what expectations they have on them, how the tools can be improved, and how the adoption process itself can be improved. We describe these change processes from a methodological perspective, how we monitored the processes, how we reviewed the outcomes, and the strategies that we applied. We show how the processes evolved, intermediate results, and the steps that were taken along the way based on the outcomes. We believe that the described study may inspire other tool-developers and/or researchers to organize similar studies to further our understanding of the complex processes involved in the adoption of software development tools in industry.

During the testing of context-sensitive middleware-based software, the middleware checks the current situation to invoke the appropriate functions of the applications. Since the middleware remains active and the situation may continue to evolve, however, the conclusion of some test cases may not easily be identified. Moreover, failures appearing in one situation may be superseded by subsequent correct outcomes and, therefore, be hidden. We alleviate the above problems by making use of a special kind of situation, which we call checkpoints, such that the middleware will not activate the functions under test. We recommend testers to generate test cases that start at a checkpoint and end at another. Testers may identify relations that associate different execution sequences of a test case. They then check the results of each test case to detect any contravention of such relations. We illustrate our technique with an example that shows how hidden failures can be detected. We also report the experimentation carried out on an RFID-based location-sensing application on top of a context-sensitive middleware.

Status of this Memo This document specifies an Internet standards track protocol for the Internet community, and requests discussion and suggestions for improvements. Please refer to the current edition of the "Internet Official Protocol Standards" (STD 1) for the standardization state and status of this protocol. Distribution of this memo is unlimited.

The Robot Operating System (ROS) is an open source framework for the development of robotic software, in which a typical system consists of multiple processes communicating under a publishersubscriber architecture. A great deal of development time goes into orchestration and making sure that the communication interfaces comply with the expected contracts (e.g. receiving a message leads to the publication of another message). Orchestration mistakes are only detected during runtime, stressing the importance of component and integration testing in the verification process. Property-based Testing is fitting in this context, since it is based on the specification of contracts and treats tested components as black boxes, but there is no support for it in ROS. In this paper, we present a first approach towards automatic generation of test scripts for property-based testing of various configurations of a ROS system. CCS CONCEPTS • Software and its engineering → Software testing and debugging; Publish-subscribe / event-based architectures; • Computer systems organization → Robotics;

The Robot Operating System (ROS) is an open source framework for the development of robotic software, in which a typical system consists of multiple processes communicating under a publishersubscriber architecture. A great deal of development time goes into orchestration and making sure that the communication interfaces comply with the expected contracts (e.g. receiving a message leads to the publication of another message). Orchestration mistakes are only detected during runtime, stressing the importance of component and integration testing in the verification process. Property-based Testing is fitting in this context, since it is based on the specification of contracts and treats tested components as black boxes, but there is no support for it in ROS. In this paper, we present a first approach towards automatic generation of test scripts for property-based testing of various configurations of a ROS system. CCS CONCEPTS • Software and its engineering → Software testing and debugging; Publish-subscribe / event-based architectures; • Computer systems organization → Robotics;

Software-defined networks are being widely adopted and used in large and complex networks supporting critical operations. Their increasing importance highlights the need for effective validation of SDN topologies and routing policies both prior and during operation. The policies that configure an SDN deployment come from several, possibly conflicting sources. This may lead to undesired effects such as node isolation, network partitions, performance drops and routing loops. Such effects can be formulated as automatically testable reusable conditions using property-based testing (PBT). This approach allows to automatically determine and formulate as a counterexample the minimum set of conflicting rules. The approach is especially useful when policies are configured in an incremental manner. PBT techniques are particularly good at automatic counterexample shrinking and have the potential of being extremely effective in this area.

The Robot Operating System (ROS) is an open source framework for the development of robotic software, in which a typical system consists of multiple processes communicating under a publishersubscriber architecture. A great deal of development time goes into orchestration and making sure that the communication interfaces comply with the expected contracts (e.g. receiving a message leads to the publication of another message). Orchestration mistakes are only detected during runtime, stressing the importance of component and integration testing in the verification process. Property-based Testing is fitting in this context, since it is based on the specification of contracts and treats tested components as black boxes, but there is no support for it in ROS. In this paper, we present a first approach towards automatic generation of test scripts for property-based testing of various configurations of a ROS system. CCS CONCEPTS • Software and its engineering → Software testing and debugging; Publish-subscribe / event-based architectures; • Computer systems organization → Robotics;

Software stakeholders who do not have a technical profile (i.e. users, clients) but do want to take part in the development and/or quality assurance process of software, have an unmet need for communication on what is being tested during the development life-cycle. The transformation of test properties and models into semi-natural language representations is one way of responding to such need. Our research has demonstrated that these transformations are challenging but feasible, and they have been implemented into a prototype tool called readSpec. The readSpec tool transforms universally-quantified test properties and stateful test models - the two kinds of test artifacts used in property-based testing - into plain text interpretations. The tool has been successfully evaluated on the PBT artifacts produced and used within the FP7 PROWESS project by industrial partners.

We present a case study in which a novel testing tool, Quviq QuickCheck, is used to test an industrial implementation of the Megaco protocol. We considered positive and negative testing and we used our developed specification to test an old version in order to estimate how useful QuickCheck could potentially be when used early in development. The results of the case study indicate that, by using Quviq QuickCheck, we would have been able to detect faults early in the development. We detected faults that had not been detected by other testing techniques. We found unclarities in the specifications and potential faults when the software is used in a different setting. The results are considered promising enough to Ericsson that they are investing in an even larger case study, this time from the beginning of the development of a new product.

Context-sensitive middleware-based software is an emerging kind of ubiquitous computing application. The components of such software communicate proactively among themselves according to the situational attributes of their environments, known as the "contexts" The actual process of accessing and updating the inter-component contexts lies with the middleware. The latter invokes the relevant local and remote operations whenever any context inscribed in the situation-aware interface is satisfied. Since the applications operate in a highly dynamic environment, the testing of context-sensitive software is complex and challenging.

Metamorphic testing is a property-based testing strategy. It recommends that, even if a test case does not reveal any failure, follow-up test cases should be further constructed from the original to check whether the software satisfies some necessary conditions of the problem to be implemented. This paper proposes to use isotropic properties of contexts as metamorphic relations for testing context-sensitive software. For instance, distinct points on the same isotropic curve of contexts would entail comparable responses by the components. This notion of testing context relations is novel, robust, and intuitive to users.