A Study on Challenges of Testing Robotic Systems
Edgar Velazquez2020, 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST)
Sign up for access to the world's latest research
checkGet notified about relevant papers
checkSave papers to use in your research
checkJoin the discussion with peers
checkTrack your impact
Abstract
Robotic systems are increasingly a part of everyday life. Characteristics of robotic systems such as interaction with the physical world, and integration of hardware and software components, differentiate robotic systems from conventional software systems. Although numerous studies have investigated the challenges of software testing in practice, no such study has focused on testing of robotic systems. In this paper, we conduct a qualitative study to better understand the testing practices used by the robotics community, and identify the challenges faced by practitioners when testing their systems. We identify a total of 12 testing practices and 9 testing challenges from our participants' responses. We group these challenges into 3 major themes: Real-world complexities, Community and standards, and Component integration. We believe that further research on addressing challenges described with these three major themes can result in higher adoption of robotics testing practices, more testing automation, and higher-quality robotic systems. Index Terms-robotics testing; testing challenges; qualitative study;
Related papers
Integration testing for robotic systems
Software Quality Journal
Conventional approaches for the validation of robotic systems include simulations and functional software testing. Although these approaches are interesting, they do not offer coverage information to guide the testing activity. In this sense, the introduction of coverage testing to the validation of mobile robotic systems seems to promise. This paper proposes a integration testing approach for robotic systems in a simulation environment. The approach is composed of test scenarios, constructed based on functional testing, and structural testing towards revealing defects and providing a better systematization of the testing activity in this application domain. The proposed approach focuses on systems developed in ROS, the robot operating system, in which communication can be established through a publish/subscribe interaction schema. The paper provides an example of the way the integration testing approach can be applied to robotic systems. An experimental study conducted with developers of mobile robotic systems evaluated the gains and proved the applicability of the approach to the industry. The results confirmed its advantages for the integration testing of mobile robotic systems. It can check functional behavior (test scenarios), or reach a high structural coverage (coverage criteria), and reveal common defects in mobile robotic systems.
A Generic Testing Framework for Test Driven Development of Robotic Systems
Lecture Notes in Computer Science, 2015
This paper proposes a generic framework for test driven development of robotic systems. The framework provides functionalities for developing and running unit tests in a language and middleware independent manner. Tests are developed as independent plug-ins to be loaded and executed by an automated tool. Moreover, a fixture manager prepares the setup (e.g., running robot drivers or simulator) and actively monitors that all the required resources are available before and during the execution of the tests. These functionalities effectively accelerate the development process and cover different levels of robotic system testing. The paper describes the framework and provides realistic examples to show how it has been used to support software development on our robotic platform.
Robotics software engineering: a perspective from the service robotics domain
Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2020
Robots that support humans by performing useful tasks (a.k.a., service robots) are booming worldwide. In contrast to industrial robots, the development of service robots comes with severe software engineering challenges, since they require high levels of robustness and autonomy to operate in highly heterogeneous environments. As a domain with critical safety implications, service robotics faces a need for sound software development practices. In this paper, we present the first large-scale empirical study to assess the state of the art and practice of robotics software engineering. We conducted 18 semi-structured interviews with industrial practitioners working in 15 companies from 9 different countries and a survey with 156 respondents from 26 countries from the robotics domain. Our results provide a comprehensive picture of (i) the practices applied by robotics industrial and academic practitioners, including processes, paradigms, languages, tools, frameworks, and reuse practices, (ii) the distinguishing characteristics of robotics software engineering, and (iii) recurrent challenges usually faced, together with adopted solutions. The paper concludes by discussing observations, derived hypotheses, and proposed actions for researchers and practitioners. CCS CONCEPTS • Computer systems organization → Robotics; • Software and its engineering;
Integration testing criteria for mobile robotic systems
Proceedings of the 27th International Conference on Software Engineering and Knowledge Engineering, 2015
Testing activity applied to mobile robotic systems is a challenge because new features, such as non-determinism of inputs, communication among components and time constraints must be considered. Simulation has been used to support the development and validation of these systems. Coverage testing criteria can contribute to this scenario adding mechanisms for measuring quality during the development of systems. This paper presents a test model and a set of coverage criteria to test the interaction among the components of mobile robotic systems. The model and criteria focus on robotic systems developed in ROS, a Robotic Operational System in which communication is established through publish/subscribe interaction schema. The testing criteria were evaluated using a robotic application. The results confirm that the use of coverage testing criteria has advantages for integration testing of mobile robotic systems.
ROBOTIC PROCESS AUTOMATION (RPA) TOWARDS AUTOMATION TESTING
International Journal of Software Engineering & Applications (IJSEA), 2024
In the rapidly evolving realm of software development, Robotic Process Automation (RPA) has emerged as a pivotal technology that holds the potential to redefine the landscape of automation testing. This scholarly article delves into the integration of RPA within the realm of automation testing, elucidating its capacity to augment efficiency and precision. The primary aim of this investigation is to evaluate the efficacy of RPA tools in automating repetitive and time-consuming testing procedures that have traditionally been conducted by human testers. Through a comparative analysis of diverse RPA tools and a series of empirical case studies, we substantiate how RPA can substantially diminish manual exertion while concurrently expanding coverage and dependability in test scenarios. Our findings reveal that RPA, when effectively implemented, not only expedites the testing cycle but also contributes to the production of more robust and dependable software products. The article concludes by deliberating on the broader implications of these findings for the field of software testing, with an emphasis on the necessity for organizations to adapt to RPA-driven methodologies. This scholarly investigation lays the groundwork for future research endeavors aimed at optimizing RPA strategies and their integration with other emerging technologies in the realm of automation testing.
Improving Software Robot Maintenance in Large-Scale Environments–is Center of Excellence a Solution?
IEEE Access
Robotic Process Automation (RPA) performs high-volume tasks such as checking invoices. However, the governance and maintenance of large-scale software robot environments can be challenging when robot servers perform automatized tasks simultaneously for customer organizations with complex programming rules, dedicated parameters, and dependencies on timetables. A multivocal literature review (MLR) was conducted to explore whether there are 1) mechanisms to improve software robot maintenance in large-scale robot environments, 2) or software robot maintenance practices for scalable RPA in organizations providing shared services, 3) or governance models for optimizing the performance of software robot maintenance, and 4) is the Center of Excellence (CoE) one of the success factors concerning large-scale robot environments. By doing this, 5) we found eleven functional requirements for the monitoring tool to support maintenance in a large-scale environment. In addition, we adapted them to the RPA monitoring tool abilities for the Finnish Government Shared Services Centre for Finance and HR (Palkeet). As a result, the eleven functional requirements and the monitoring tool abilities are adaptable for other large-scale environments to improve software robot maintenance. However, commercial monitoring tools for RPA maintenance do not fulfil functional requirements, and organizations in large-scale environments must develop their monitoring tools. Based on MLR, either in-house or outsourced CoE seems to be one of the success factors in RPA maintenance in large-scale environments. INDEX TERMS Center of excellence, CoE, governance, maintenance, robotic process automation, RPA. I. INTRODUCTION 18 Organizations are increasingly interested in improving pro-19 cesses by utilizing robotic process automation (RPA) to 20 maintain competitiveness in a rapidly changing technology 21 environment. RPA refers to the programming of activities and 22 transactions that automatically repeats the performance of a 23 similar task on data sets or/and systems according to specific 24 rules, replacing human work. RPA is defined as a ''precon-25 figured software instance that uses business rules and pre-26 defined activity choreography to complete the autonomous 27 execution'', where the execution is ''a combination of pro-28 cesses, activities, transactions, and tasks in one or more 29 unrelated software systems to deliver a result or service with 30 The associate editor coordinating the review of this manuscript and approving it for publication was Hongli Dong. a human exception management'' [1]. The automated process 31 can reduce errors and costs from manual work and perform 32 repetitive routine tasks in agile, influencing the efficiency and 33 quality of the operations. The RPA benefits are apparent; for 34 example, software robots can work 24/7, increase productiv-35 ity and daily throughput, improve accuracy, increase client 36 satisfaction, and decrease operating risks [2, pp. 11 and 12]. 37 Software robots of the RPA solutions ''mimic'' human 38 interactions with systems to perform specific tasks in 39 processes [1], [3], [4], [5], [6], [7], [8], [9], [10]. Usually, soft-40 ware robots perform routine tasks that are rule-based, repet-41 itive, well-structured, and high-volume [2, pp. 32], [5], [7], 42 [10]. Furthermore, RPA is non-invasive: it does not change 43 the involved software systems because the robot utilizes the 44 software systems like a user, but automation is sitting ''on top 45 of them'' [7], [8], [10].
Design and Development of Hardware Platform for Testing Robotics and Automation Applications
The 21 st century is the century of robotics and automation in various fields of engineering to reduce human efforts. Robotics and automation has been proved since long time a helpful process of minimizing human labor and increasing throughput of production. Being a most preferred technology in various ways, it is necessary to design, develop and test the applications of robotics and automation. The design and development is done at various levels with respect to software and also testing of such applications is done on non-hardware platforms. It will be more advantageous to have a hardware platform to test such applications in order to understand its functionality in a better way than non-hardware platform. The paper titled design and development of hardware platform for testing robotics and automation applications is one such non-software platform.
Learning Software Engineering Basics Through Robotics
2016
Software engineering is a scientific discipline that deals with all phases of (requirements analysis, design and modeling, implementation, evaluation and validation, as well as maintenance). Learning various concepts of software engineering is not a trivial task, and requires more than a one semester course. LEGO MINDSTORMS NXT kit is a popular and affordable education robotic platform, used as an educational tool in various areas. This paper describes how the LEGO MINDSTORMS robots can be used for teaching the basics of software engineering. We have also evaluated the outcomes of the course in order to reveal the effects of using this off-the-shelf robot kit.
Mastering Software Robot Development Projects: Understanding the Association between System Attributes Design Practices
Proceedings of the Annual Hawaii International Conference on System Sciences, 2019
Software robots tend to increasingly take over organizational processes. However, little is known about principles of developing as opposed to using robotic systems, such as RPA robots and chatbots. Therefore, based on a comparative case study, this paper elaborates how different types of robots, due to distinguishing system attributes, relate to different design practices that arise from varying challenges of transforming existing routines into robots.
A Requirements Definition Framework for the Robotic Systems Domain - An Exploratory Study
Anais do WER18 - Workshop em Engenharia de Requisitos, 2018
A robotic system is a physical device composed by special-purpose hardware and software that must be integrated to enable a robotic system to function as expected. The first step towards the modeling robotic systems is to determine its requirements, which is a description of how the system should behave together with constraints on its operation. Much research effort in the last two decades has been invested to support requirements engineering (RE) for general systems. However, to our knowledge, there is no framework to properly support RE for robotic systems. Hence, we conducted an exploratory study aiming to answer the question "What technical features affect the RE for the robotic systems domain?" Based on what we learned in the study, we point out some challenges and issues in RE for the robotic systems domain. Additionally, we provide a framework to identify and manage several technical features which affect RE for this domain.
Related topics
Related papers
Robotic Standard Development Life Cycle in Action
Journal of Intelligent & Robotic Systems
Robotics is a fast-growing field which requires the efficient development of adapted standards. Hence, in this paper, we propose a development methodology to support the robot standardization effort led by international, technical, and professional associations such as the Institute of Electrical and Electronics Engineers (IEEE). Our proposed standard development life cycle is a middle-out, iterative, collaborative, and incremental approach we have successfully applied to the development of the new IEEE Ontological Standard for Ethically Driven Robotics and Automation Systems (IEEE P7007 Standard).
A systematic review of applying modern software engineering techniques to developing robotic systems
Ingenieria e Investigación
Los robots se han convertido en colaboradores habituales de nuestra vida diaria. Los sistemas robóticos son cada vez más complejos y, como consecuencia, crece la necesidad de aplicar nuevas técnicas ingenieriles a su proceso de desarrollo. Los enfoques tradicionales que se utilizan en el proceso de desarrollo de estos sistemas de software están alcanzando sus límites; las metodologías utilizadas actualmente y las herramientas de soporte no alcanzan para atender las necesidades de estos procesos complejos. Para fomentar la reutilización y el mantenimiento de código es esencial separar el conocimiento estable del dominio de robótica en las tecnologías de implementación, que varían rápidamente. Este artículo presenta una revisión sistemática de la utilización actual de técnicas modernas de ingeniería de software en el desarrollo de sistemas robóticos y su nivel de automatización. El objetivo del estudio es el de resumir la evidencia existente respecto a la aplicación de dichas tecnologías en el campo de los sistemas robóticos para identificar carencias en la investigación actual con el fin de sugerir áreas en futuras propuestas y proporcionar las bases para posicionar adecuadamente nuevas actividades de investigación.
Property-based testing for the robot operating system
Proceedings of the 9th ACM SIGSOFT International Workshop on Automating TEST Case Design, Selection, and Evaluation, 2018
The Robot Operating System (ROS) is an open source framework for the development of robotic software, in which a typical system consists of multiple processes communicating under a publishersubscriber architecture. A great deal of development time goes into orchestration and making sure that the communication interfaces comply with the expected contracts (e.g. receiving a message leads to the publication of another message). Orchestration mistakes are only detected during runtime, stressing the importance of component and integration testing in the verification process. Property-based Testing is fitting in this context, since it is based on the specification of contracts and treats tested components as black boxes, but there is no support for it in ROS. In this paper, we present a first approach towards automatic generation of test scripts for property-based testing of various configurations of a ROS system. CCS CONCEPTS • Software and its engineering → Software testing and debugging; Publish-subscribe / event-based architectures; • Computer systems organization → Robotics;
The robotics experience
IEEE Robotics & Automation Magazine, 2009
Usability Squared: Principles for doing good systems research in robotics
ArXiv, 2019
Despite recent major advances in robotics research, massive injections of capital into robotics startups, and significant market appetite for robotic solutions, large-scale real-world deployments of robotic systems remain relatively scarce outside of heavy industry and (recently) warehouse logistics. In this paper, we posit that this scarcity comes from the difficulty of building even merely functional, first-pass robotic applications without a dizzying breadth and depth of expertise, in contrast to the relative ease with which non-experts in cloud computing can build complex distributed applications that function reasonably well. We trace this difficulty in application building to the paucity of good systems research in robotics, and lay out a path toward enabling application building by centering usability in systems research in two different ways: privileging the usability of the abstractions defined in systems research, and ensuring that the research itself is usable by applicat...
Robotic system testing with AMSA framework
2018
Nowadays robotic systems are a combination of complex software and hardware components providing sophisticated functionalities. Robotic control systems evolving in an uncertain environment are generally developed case by case for specific deployment platforms. For complex and realistic systems, simulation plays a central role during design by providing testing facilities. In this paper, we propose to model robotic architectures using the modeldriven framework AMSA. In order to facilitate simulation and testing, we propose to incorporate different behaviors in the model through parameterizations. For testing purpose, we define scenarios as a sequence of parameter modifications. From architectural and scenario models, code is generated for the robotic middleware ROS. During simulation, scenarios are used to evaluate different controller behaviors for different contexts. The approach has been experimented for the MDE challenge: a rover must follow another leader rover. The approach all...
Using competitions to advance the development of standard test methods for response robots
Proceedings of the Workshop on Performance Metrics for Intelligent Systems, 2012
Competitions are an effective aid to the development and dissemination of standard test methods, especially in rapidly developing, fields with a wide variety of requirements and capabilities such as Urban Search and Rescue robotics. By exposing the development process to highly developmental systems that push the boundaries of current capabilities, it is possible to gain an insight into how the test methods will respond to the robots of the future. The competition setting also allows for the rapid iterative refinement of the test methods and apparatuses in response to new developments. For the research community, introducing the concepts behind the test methods at the research and development stage can also help to guide their work towards the operationally relevant requirements embodied by the test methods and apparatuses. This also aids in the dissemination of the test methods themselves as teams fabricate them in their own laboratories and re-use them in work outside the competition. In this paper, we discuss how international competitions, and in particular the RoboCupRescue Robot League competition, have played a crucial role in the development of standard test metrics for response robots as part of the ASTM International Committee of Homeland Security Applications; Operational Equipment; Robots (E54.08.01). We will also discuss how the competition has helped to drive a vibrant robot developer community towards solutions that are relevant to first responders.
An Architecture-oriented Approach to System Integration in Collaborative Robotics Research Projects — An Experience Report
2015
Effective system integration requires strict adherence to strong software engineering standards, a practice not much favoured in many collaborative research projects. We argue that component-based software engineering (CBSE) provides a way to overcome this problem because it provides flexibility for developers while requiring the adoption of only a modest number of software engineering practices. This focus on integration complements software re-use, the more usual motivation for adopting CBSE. We illustrate our argument by showing how a large-scale system architecture for an application in the domain of robot-enhanced therapy for children with autism spectrum disorder (ASD) has been implemented. We highlight the manner in which the integration process is facilitated by the architecture implementation of a set of placeholder components that comprise stubs for all functional primitives, as well as the complete implementation of all inter-component communications. We focus on the component-port-connector meta-model and show that the YARP robot platform is a well-matched middleware framework for the implementation of this model. To facilitate the validation of port-connector communication, we configure the initial placeholder implementation of the system architecture as a discrete event simulation and control the invocation of each component's stub primitives probabilistically. This allows the system integrator to adjust the rate of inter-component communication while respecting its asynchronous and concurrent character. Also, individual ports and connectors can be periodically selected as the simulator cycles through each primitive in each subsystem component. This ability to control the rate of connector communication considerably eases the task of validating component-port-connector behaviour in a large system. Ultimately, over and above its well-accepted benefits for software re-use in robotics, CBSE strikes a good balance between software engineering best practice and the socio-technical problem of managing effective integration in collaborative robotics research projects.
Lessons from Applying Modern Software Methods and Technologies to Robotics
2000
Autonomous mobile robots of today are software- intensive systems. The rapidly growing code base of robotics projects therefore requires advanced software development methods and technologies, to ensure their scalability, reliability and maintainability. In this paper, we analyze modern concepts and technologies that we applied to our software development process and show how they contribute to enhancing these design dimensions. I.
The Flourishing Landscape of Robot Standardization [Industrial Activities]
IEEE Robotics & Automation Magazine
Loading Preview
Sorry, preview is currently unavailable. You can download the paper by clicking the button above.