Property-based testing for the robot operating system
Andre Rocha Santos2018, Proceedings of the 9th ACM SIGSOFT International Workshop on Automating TEST Case Design, Selection, and Evaluation
Sign up for access to the world's latest research
checkGet notified about relevant papers
checkSave papers to use in your research
checkJoin the discussion with peers
checkTrack your impact
Abstract
The Robot Operating System (ROS) is an open source framework for the development of robotic software, in which a typical system consists of multiple processes communicating under a publishersubscriber architecture. A great deal of development time goes into orchestration and making sure that the communication interfaces comply with the expected contracts (e.g. receiving a message leads to the publication of another message). Orchestration mistakes are only detected during runtime, stressing the importance of component and integration testing in the verification process. Property-based Testing is fitting in this context, since it is based on the specification of contracts and treats tested components as black boxes, but there is no support for it in ROS. In this paper, we present a first approach towards automatic generation of test scripts for property-based testing of various configurations of a ROS system. CCS CONCEPTS • Software and its engineering → Software testing and debugging; Publish-subscribe / event-based architectures; • Computer systems organization → Robotics;
Related papers
Integration testing for robotic systems
Software Quality Journal
Conventional approaches for the validation of robotic systems include simulations and functional software testing. Although these approaches are interesting, they do not offer coverage information to guide the testing activity. In this sense, the introduction of coverage testing to the validation of mobile robotic systems seems to promise. This paper proposes a integration testing approach for robotic systems in a simulation environment. The approach is composed of test scenarios, constructed based on functional testing, and structural testing towards revealing defects and providing a better systematization of the testing activity in this application domain. The proposed approach focuses on systems developed in ROS, the robot operating system, in which communication can be established through a publish/subscribe interaction schema. The paper provides an example of the way the integration testing approach can be applied to robotic systems. An experimental study conducted with developers of mobile robotic systems evaluated the gains and proved the applicability of the approach to the industry. The results confirmed its advantages for the integration testing of mobile robotic systems. It can check functional behavior (test scenarios), or reach a high structural coverage (coverage criteria), and reveal common defects in mobile robotic systems.
A Generic Testing Framework for Test Driven Development of Robotic Systems
Lecture Notes in Computer Science, 2015
This paper proposes a generic framework for test driven development of robotic systems. The framework provides functionalities for developing and running unit tests in a language and middleware independent manner. Tests are developed as independent plug-ins to be loaded and executed by an automated tool. Moreover, a fixture manager prepares the setup (e.g., running robot drivers or simulator) and actively monitors that all the required resources are available before and during the execution of the tests. These functionalities effectively accelerate the development process and cover different levels of robotic system testing. The paper describes the framework and provides realistic examples to show how it has been used to support software development on our robotic platform.
Integration testing criteria for mobile robotic systems
Proceedings of the 27th International Conference on Software Engineering and Knowledge Engineering, 2015
Testing activity applied to mobile robotic systems is a challenge because new features, such as non-determinism of inputs, communication among components and time constraints must be considered. Simulation has been used to support the development and validation of these systems. Coverage testing criteria can contribute to this scenario adding mechanisms for measuring quality during the development of systems. This paper presents a test model and a set of coverage criteria to test the interaction among the components of mobile robotic systems. The model and criteria focus on robotic systems developed in ROS, a Robotic Operational System in which communication is established through publish/subscribe interaction schema. The testing criteria were evaluated using a robotic application. The results confirm that the use of coverage testing criteria has advantages for integration testing of mobile robotic systems.
ROSRV: Runtime Verification for Robots
Runtime Verification, 2014
We present ROSRV, a runtime verification framework for robotic applications on top of the Robot Operating System (ROS [8]), a widely used open-source framework for robot software development. ROSRV aims to address the safety and security issues of robots by providing a transparent monitoring infrastructure that intercepts and monitors the commands and messages passing through the system. Safety and security properties can be defined in a formal specification language, and are ensured by automatically generated monitors. ROSRV integrates seamlessly with ROS-no change in ROS nor the application code is needed. ROSRV has been applied and evaluated on a commercial robot.
Automatic property checking of robotic applications
2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2017
Robot software controllers are often concurrent and time critical, and requires modern engineering approaches for validation and verification. With this motivation, we have developed a tool and techniques for graphical modelling with support for automatic generation of underlying mathematical definitions for model checking. It is possible to check automatically both general properties, like absence of deadlock, and specific application properties. We cater both for timed and untimed modelling and verification. Our approach has been tried in examples used in a variety of robotic applications.
Model-Based Regression Testing of Autonomous Robots
SDL 2017: Model-Driven Engineering for Future Internet, 2017
Testing is a common technique to assess quality of systems. Regression testing comes into view, when changes are introduced to the system under test and re-running all tests is not practical. Numerous techniques have been introduced to select tests only relevant to a given set of changes. These are typically based on source code, however, modelbased development projects use models as primary artifacts described in various domain-specific languages. Thus, regression test selection should be performed directly on these models. We present a method and a case study on how model-based regression testing can be achieved in the context of autonomous robots. The method uses information from several domain-specific languages for modeling the robot's context and configuration. Our approach is implemented in a prototype tool, and its scalability is evaluated on models from the case study.
Roborebeca: A Newframework to Design Verified Ros-Based Robotic Programs
2019
Robotic technology helps humans in different areas such as manufacturing, health care and educa-tion. Due to the ubiquitous revolution, today’s focus is on mobile robots and their applications in a ...
Formal Verification of ROS-Based Robotic Applications Using Timed-Automata
2017 IEEE/ACM 5th International FME Workshop on Formal Methods in Software Engineering (FormaliSE)
Robotic technologies are continuously transforming the domestic and the industrial environments. Recently the Robotic Operating System (ROS), an open source middleware framework, has been widely adopted both by industry and academia, becoming the de facto standard for developing robot applications. Guaranteeing the correct behaviour of robotic systems is, however, challenging due to their potential for parameterization and heterogeneity. Although different approaches exist, focusing on concrete domain spaces for specific scenarios, no general approach to reason about ROS systems has yet arisen. This paper proposes an approach to model and verify ROS systems using real time properties, focusing on one of the main features of ROS, the communication between nodes. It takes low-level parameters into account, such as queue sizes and timeouts, and uses timed automata as the modelling language. The physical robot Kobuki is used as a complex case study, over which properties are automatically verified using the UPPAAL model checker, enabling the identification of problematic parameter combinations.
Reverse Engineering of Middleware for Verification of Robot Control Architectures
Lecture Notes in Computer Science, 2014
We consider the problem of automating the verification of distributed control software relying on publish-subscribe middleware. In this scenario, the main challenge is that software correctness depends intrinsically on correct usage of middleware components, but structured models of such components might not be available for analysis, e.g., because they are too large and complex to be described precisely in a cost-effective way. To overcome this problem, we propose to identify abstract models of middleware as finite-state automata, and then to perform verification on the combined middleware and control software models. Both steps are carried out in a computer-assisted way using state-of-the-art techniques in automata-based identification and verification. Our main contribution is to show that the combination of identification and verification is feasible and useful when considering typical issues that arise in the implementation of distributed control software. * The final publication is available at Springer via http://dx.
ROSMOD: a toolsuite for modeling, generating, deploying, and managing distributed real-time component-based software using ROS
2015 International Symposium on Rapid System Prototyping (RSP), 2015
This paper presents the Robot Operating System Model-driven development tool suite, (ROSMOD) an integrated development environment for rapid prototyping component-based software for the Robot Operating System (ROS) middleware. ROSMOD is well suited for the design, development and deployment of large-scale distributed applications on embedded devices. We present the various features of ROSMOD including the modeling language, the graphical user interface, code generators, and deployment infrastructure. We demonstrate the utility of this tool with a real-world case study: an Autonomous Ground Support Equipment (AGSE) robot that was designed and prototyped using ROSMOD for the NASA Student Launch competition, 2014-2015.
Related papers
A Study on Challenges of Testing Robotic Systems
2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST), 2020
Robotic systems are increasingly a part of everyday life. Characteristics of robotic systems such as interaction with the physical world, and integration of hardware and software components, differentiate robotic systems from conventional software systems. Although numerous studies have investigated the challenges of software testing in practice, no such study has focused on testing of robotic systems. In this paper, we conduct a qualitative study to better understand the testing practices used by the robotics community, and identify the challenges faced by practitioners when testing their systems. We identify a total of 12 testing practices and 9 testing challenges from our participants' responses. We group these challenges into 3 major themes: Real-world complexities, Community and standards, and Component integration. We believe that further research on addressing challenges described with these three major themes can result in higher adoption of robotics testing practices, more testing automation, and higher-quality robotic systems. Index Terms-robotics testing; testing challenges; qualitative study;
Robotic system testing with AMSA framework
2018
Nowadays robotic systems are a combination of complex software and hardware components providing sophisticated functionalities. Robotic control systems evolving in an uncertain environment are generally developed case by case for specific deployment platforms. For complex and realistic systems, simulation plays a central role during design by providing testing facilities. In this paper, we propose to model robotic architectures using the modeldriven framework AMSA. In order to facilitate simulation and testing, we propose to incorporate different behaviors in the model through parameterizations. For testing purpose, we define scenarios as a sequence of parameter modifications. From architectural and scenario models, code is generated for the robotic middleware ROS. During simulation, scenarios are used to evaluate different controller behaviors for different contexts. The approach has been experimented for the MDE challenge: a rover must follow another leader rover. The approach all...
From Offline Towards Real-Time Verification for Robot Systems
IEEE Transactions on Industrial Informatics, 2018
Iterative Model Driven Integration Checks of Component Based Robotic Systems
A robot's software ecosystem often comprises a set of heterogeneous software components, acquiring, exchanging, fusioning, and deriving data to trigger a desired behaviour, state, or action of the robot. Due to the nature of component based development [2], and component interaction respectively, an essential, and often crucial part of robotic system development is frequent integration testing. While unit tests already provide an established way of checking modelled constraints on single components, tests including complete component based (sub-)systems have not been widely studied in robotics so far. We have identified the following problem statements with respect to component based system testing so far.
RoboChart: modelling and verification of the functional behaviour of robotic applications
Software & Systems Modeling
Robots are becoming ubiquitous: from vacuum cleaners to driverless cars, there is a wide variety of applications, many with potential safety hazards. The work presented in this paper proposes a set of constructs suitable for both modelling robotic applications and supporting verification via model checking and theorem proving. Our goal is to support roboticists in writing models and applying modern verification techniques using a language familiar to them. To that end, we present RoboChart, a domain-specific modelling language based on UML, but with a restricted set of constructs to enable a simplified semantics and automated reasoning. We present the RoboChart metamodel, its well-formedness rules, and its process-algebraic semantics. We discuss verification based on these foundations using an implementation of RoboChart and its semantics as a set of Eclipse plug-ins called RoboTool. Keywords State machines • Formal semantics • Process algebra • CSP • Model checking • Timed properties • Domain-specific language for robotics Communicated by Dr. Jeff Gray.
ROBOTIC PROCESS AUTOMATION (RPA) TOWARDS AUTOMATION TESTING
International Journal of Software Engineering & Applications (IJSEA), 2024
In the rapidly evolving realm of software development, Robotic Process Automation (RPA) has emerged as a pivotal technology that holds the potential to redefine the landscape of automation testing. This scholarly article delves into the integration of RPA within the realm of automation testing, elucidating its capacity to augment efficiency and precision. The primary aim of this investigation is to evaluate the efficacy of RPA tools in automating repetitive and time-consuming testing procedures that have traditionally been conducted by human testers. Through a comparative analysis of diverse RPA tools and a series of empirical case studies, we substantiate how RPA can substantially diminish manual exertion while concurrently expanding coverage and dependability in test scenarios. Our findings reveal that RPA, when effectively implemented, not only expedites the testing cycle but also contributes to the production of more robust and dependable software products. The article concludes by deliberating on the broader implications of these findings for the field of software testing, with an emphasis on the necessity for organizations to adapt to RPA-driven methodologies. This scholarly investigation lays the groundwork for future research endeavors aimed at optimizing RPA strategies and their integration with other emerging technologies in the realm of automation testing.
Towards an Actor-based Approach to Design Verified ROS-based Robotic Programs using Rebeca
Procedia Computer Science, 2019
Robotic technology helps humans in different areas such as manufacturing, health care and education. Due to the ubiquitous revolution, today's focus is on mobile robots and their applications in a variety of cyber-physical systems. ROS is a wll-known and powerful middleware that facilitates software development for mobile robots. However, this middleware does not support assuring properties such as timeliness and safety of ROS-based software. In this paper we present an integration of Timed Rebeca modeling language with ROS to synthesize verified robotic software. First, a conceptual model of robotic programs is developed using Timed Rebeca. After verifying a set of user-defined correctness properties on this model, it is translated to a ROS program automatically. Experiments on some small-scale case studies illustrates the applicability of the proposed integration method.
Towards Property Oriented Testing
Electronic Notes in Theoretical …, 2007
Conformance testing is a kind of functional testing where a formally verified specification is considered and test cases are generated so that conclusions can be established regarding the possibility of acceptance/rejection of conforming/nonconforming implementations. If the focus is on a complete specification, test suites may be impractical and even infinite with unclear relations between test cases and the specification. Property oriented testing focuses on particular, eventually critical, properties of interest. The specification of one or more properties drives the test process that checks whether they are satisfied by an implementation. Properties are often stated as test purposes, targeting testing at a particular functionality. This paper presents an overview of approaches to property oriented testing for reactive systems, focusing on labelled and symbolic transition systems.
Verified ROS-Based Deployment of Platform-Independent Control Systems
Lecture Notes in Computer Science, 2015
The paper considers the problem of model-based deployment of platform-independent control code on a specific platform. The approach is based on automatic generation of platform-specific glue code from an architectural model of the system. We present a tool, ROSGen, that generates the glue code based on a declarative specification of platform interfaces. Our implementation targets the popular Robot Operating System (ROS) platform. We demonstrate that the code generation process is amenable to formal verification. The code generator is implemented in Coq and relies on the infrastructure provided by the CompCert and VST tool. We prove that the generated code always correctly connects the controller function to sensors and actuators in the robot. We use ROSGen to implement a cruise control system on the LandShark robot. Disciplines
A framework for automated testing of automation systems
2010
Increasing complexity of software components in automation systems require systematic and frequent testing approaches. Test-First Development (TFD) -an established approach in business IT software developmentpromises to support test automation in automation systems development. Nevertheless, linking test case generation, execution, and reporting requires a sound framework to support testing processes more efficiently. In this paper we present a framework for automating test processes based on UML models and TFD. Applying this framework in prototype applications in industry environment identified the framework as promising candidate to improve automation systems development and product quality.
Loading Preview
Sorry, preview is currently unavailable. You can download the paper by clicking the button above.