Software Test Automation in Practice: Empirical Observations

Software Quality Journal, 2018

An important challenge in software testing has been to improve the degree of attainable automation, in developing advanced techniques for generating the test inputs as well as in finding innovative support procedures to automate the testing process. Test automation is a very active field of research, covering different aspects such as tools, application domains, the lifecycle development process, project dimensions, and empirical work reporting successful results or failures (Graham and Fewster 2012). Test automation is also highly relevant in practice: the market for automated test support tools is expanding, opening relevant business opportunities for new innovative testing platforms. The ultimate challenge is the development of a powerful integrated test environment that goes beyond automated test execution and by itself can automatically take care Software Qual J

The objective of this study is to find out current position and betterment needs in the automated software testing. For this, an industrial survey of Pakistan IT market was conducted to identify the benefits, challenges and automated software testing tools that are being utilized. Through the results of the study, it is concluded that in local industry of Pakistan "Automated Software Testing (AST) "has been introduced for the last few years and some gaps have been identified between literature and practices. There is need of further research in automated software testing as well as the challenges should be solved to make AST process more effective.

International Journal of System Assurance Engineering and Management, 2011

The study explores the current state of test automation in software testing organizations by focusing on the views and observations of managers, testers and developers in each organization. The case study was conducted in selected organizational units that develop and test technical software for industrial automation or telecommunication domains. The data was collected with 41 theme-based interviews in each unit. The interview data was analyzed qualitatively by using the grounded theory research method. It was found that although test automation was viewed as beneficial, it was not utilized widely in the companies. The main benefits of test automation were quality improvement, the possibility to execute more tests in less time and fluent reuse of testware. The major disadvantages were the costs associated with developing test automation especially in dynamic customized environments. Such issues as properties of tested products, attitudes of employees, resource limitations, and customers influenced the level of test automation in the case organizations.

2012 7th International Workshop on Automation of Software Test (AST), 2012

The increased importance of Test Automation in software engineering is very evident considering the number of companies investing in automated testing tools nowadays, with the main aim of preventing defects during the development process. Test Automation is considered an essential activity for agile methodologies being the key to speed up the quality assurance process. This paper presents empirical observations and the challenges of a test team new to agile practices and Test Automation using open source testing tools integrated in software projects that use the Scrum methodology. The results obtained showed some important issues to be discussed and the Test Automation practices collected based on the experiences and lessons learned.

Anais do XIV Simpósio Brasileiro de Qualidade de Software (SBQS 2015)

Testing is an essential activity to ensure quality of software systems, but it is expensive and time consuming. Thus, testing automation would be an alternative to improve test productivity and save costs. However, many organizations refuse to use test automation or had failed on implement it because they do not know how to deal with the implementation of a test automation strategy fitted to their goals and expectations. Most of them underestimate or have no knowledge about test automation factor of success. In addition, although there are many works and maturity models focused on improving the testing process, few ones focus on test automation issues. The main contribution of this paper is to propose a hierarchical model called Test Automation’s Pyramid of Needs (TAPN). TAPN is inspired in the Maslow’s Pyramid of Needs administration theory and it comprises five levels that influence on the success of test automation initiatives in software organizations. TAPN intends to help organ...

TJPRC, 2014

The growing importance of Software in the present scenario can be attributed to the fact that software is utilized in several different ways for different issues to provide different types of solutions. The subject of Software Engineering is so vast and varied that it encompasses several areas such as manufacturing of Refrigerators, Air Conditioning Systems, Automobiles, Space Engineering, Wireless and Mobile Communications. Therefore, in most cases, software usually need to adhere to a specifications so that they can perform to produce only those results that are expected and enunciated. In general, keeping in view of the present scenario, a software professional goes through a certain process to establish that the software conforms to a given specification. This process, verification and validation (V & V), ensures that the software conforms to its specification and that the customers ultimately receive what they ordered. Software testing is one of the techniques to use during V & V. To be able to use resources in a better way, computers should be able to help out in the “art of software testing” to a higher extent, than is currently the case today. The main issue is to not retrench human resources from the process of software testing itself altogether but to consider the fact that software engineering is an art and science of identifying suitable solutions to critical problems by not just involving computers alone to solve the testing issues but software engineers should themselves participate and come out with better and suitable solutions to those problems where computers are clearly not that good to provide appropriate solutions. This research work presents a systematic and methodical approach aimed at examining, classifying and improving the concept of automated software testing and is built upon the assumption that software testing could be automated to a higher extent. Throughout this thesis an emphasis has been put on “real life” applications and the testing of these applications. One of the contributions in this thesis is the research aimed at uncovering different issues with respect to automated software testing. The research is performed through a series of case studies and experiments which ultimately also leads to another contribution—a model for expressing, clarifying and classifying software testing and the automated aspects thereof. An additional contribution in this thesis is the development of framework which in turns acts as a broad substratum for a framework for object message pattern analysis of intermediate code representations. This is achieved keeping in view of the procedures relating to code optimization and effective code generation at all levels. 260 Chittineni Aruna & R. Siva Ram Prasad Impact Factor (JCC): 6.8785 Index Copernicus Value (ICV): 3.0 The results that are expected in this thesis indicate how software testing can be improved, extended and better classified with respect to automation aspects. The main contribution lays in the investigations and the improvements related issues with regard to automated software testing. A comprehensive study has been made by the researchers to tackle the very crust of the problem of software testing methodologies. It has to be noted that testing software in an automated way has been a goal for researchers and industry during the last few decades. Nevertheless, the success rate has not been readily available. Some tools that are partly automated have evolved, while at the same time the methodologies for testing software has improved, thus leading to an overall improvement. Even today, much of the software testing procedures are performed manually which is not reliable, prone to errors, inefficient and also a costly affair. Even today several questions still remain unanswered. For instance, we have the question of when and how to stop the process of testing software when is it not economically feasible to continue to test software? It is very much evident that investing so much of time and money for testing an application which will be used only for a few times, in a non-critical environment, is probably a waste of resources.[1] At the same time, when software engineers develop software that will be placed in a critical domain and extensively used, an answer to that question needs to be found. Secondly, there is the problem of resources. It is to be noted that small- and medium-sized companies are today, as always, challenged by their resources, or to be more precise, the lack thereof. Deadlines must be kept at all costs even when, in some cases, the cost turns out to be the actual reliability of their products. In addition to the growing complexity of software, the size of the problem becomes more and more complex and huge. Keeping in view of the question of complexity and to be more precise based on the fact that software has grown in complexity and size which have become very much part of the problem of software testing and the type of research work that needs to be done to improvise on these issues is to be given a methodical and systematic approach.

Welcome to this special section on the Automation of Software Test. It is inspired by the 2015 IEEE/ACM International Workshop on Automation of Software Test (AST 2015) held in Firenze, Italy, on 23 and 24 May 2015, in conjunction with the ICSE 2015 conference. The authors of selected papers presented at AST 2015 were invited to submit extended and revised versions of their corresponding workshop papers. At the same time, an open call-for-papers was announced publicly and distributed in the software engineering community. Eighteen papers were submitted. Each paper was reviewed by active researchers and experts in the related subject areas. The review process has followed the journal's review protocol and applied the journal's quality standards. Eight papers were finally accepted for inclusion.

Testing Automation has been growing in software engineering. Many organizations are investing in automated testing in order to prevent defects and increase testing effectiveness during software development. In agile methodologies, this task is considered an important activity, considered the key of the agile testing. This paper presents three testing automation strategies applied to three different software projects adopting Scrum agile methodology. The results indicated positive agile practices to be considered when adopting testing automation strategy, such as: team collaboration, task distribution, testing tools, knowledge managements. The challenges, results, and lessons learned from this experience are also discussed.

Informatica Economica, 2014

Testing is an art and science that should ultimately lead to lower cost businesses through increasing control and reducing risk. Testing specialists should thoroughly understand the system or application from both the technical and the business perspective, and then design, build and implement the minimum-cost, maximum-coverage validation framework. Test Automation is an important ingredient for testing large scale applications. In this paper we discuss several test automation frameworks, their advantages and disadvantages. We also propose a custom automation framework model that is suited for applications with very complex business requirements and numerous interfaces.

International Journal of Computer Science Engineering (IJCSE), 2019

The growing demand for delivering quality software faster "Quality at Speed" requires faster and successful execution of software testing to ensure its standard. Utilizing appropriate testing method(s) and right test automation tools/framework are two defining factors for a successful and effective software testing project. Using one testing method will not be sufficient to test software and ensure its standard, a combination of some appropriate testing techniques is often required. Likewise, is no one tool that can satisfy all automated testing needs which makes finding the right tool combination difficult. Knowing the various testing methods and tools/frameworks is the first step towards achieving a successful and efficient software testing. This article presents a comprehensive study of test automation tools and frameworks. Firstly, automated testing and their categories were explained, followed by an explanation of the various test automation frameworks. Finally, a brief explanation and comparison of some of the most commonly used automation tools was presented.

Journal of Software: Evolution and Process, 2020

Context: Software testing is an important and costly software engineering activity in the industry. Despite the efforts of the software testing research community in the last several decades, various studies show that still many practitioners in the industry report challenges in their software testing tasks. Objective: To shed light on industry's challenges in software testing, we characterize and synthesize the challenges reported by practitioners. Such concrete challenges can then be used for a variety of purposes, e.g., research collaborations between industry and academia. Method: Our empirical research method is opinion survey. By designing an online survey, we solicited practitioners' opinions about their challenges in different testing activities. Our dataset includes data from 72 practitioners from eight different countries. Results: Our results show test management and test automation are considered the most challenging among all testing activities by practitioners. Our results also include a set of 104 concrete challenges in software testing that may need further investigations by the research community. Conclusion: We conclude that the focal points of industrial work and academic research in software testing differ. Furthermore, the paper at hand provides valuable insights concerning practitioners' "pain" points and, thus, provides researchers with a source of important research topics of high practical relevance.

2010

The main goal of the seminar "Practical Software Testing: Tool Automation and Human Factors" was to bring together academics working on algorithms, methods, and techniques for practical software testing, with practitioners, interested in developing more soundly-based and well-understood testing processes and practices. The seminar's purpose was to make researchers aware of industry's problems, and practitioners aware of research approaches. The seminar focused in particular on testing automation and human factors: Tool automation. Automation of testing is a crucial concern in industry. It is only with automation that testing becomes practical and scalable to the size of a typical system with which the industry has to deal. Test automation or tool support spans the spectrum from test planning, generation, minimization, execution, oracle checking, to management. Test automation can exploit not only knowledge from the code under test but also from available models or specifications. Human factors. Human factors play important roles in software testing. Given the code under test, tools can try to automate the generation of test inputs as much as possible, but test oracles still need to come from testers, who specify them in the form of specifications, properties, or test assertions, or directly inspect the actual test outputs for correctness. In addition, tools are not always perfect to deal with software complexity; testers need to cooperate with tools to effectively carry out testing tasks, by giving guidance to tools and interpreting results produced by tools. Thus testers need to be well trained.

International Journal of Advanced Research in Science, Communication and Technology (IJARSCT), 2023

As the domain of software development continues to progress in terms of intricacy and magnitude, the integration of automated testing has become a vital element in ensuring the excellence and dependability of software. The implementation of automated testing frameworks assumes a pivotal role in streamlining the testing procedure; nevertheless, the selection of the appropriate framework category presents challenges and necessitates making prudent compromises. This academic paper presents a comprehensive exploration of automated testing frameworks, exhaustively investigating their diverse categories, attributes, and the associated compromises that emerge from their adoption. The paper initiates with a comprehensive examination of the literature, encompassing a concise overview of the current research and industry patterns regarding test automation frameworks. Following this, it presents an intricate evaluation of various types of frameworks, such as keyword-driven, data-driven, and behavior-driven frameworks, clarifying their individual principles and functionalities. One of the main topics explored in this paper concerns the factors that influence the choice of a particular type of test automation framework. By presenting a framework for decision-making, we offer guidance to professionals and organizations in making informed decisions that align with their project requirements, team expertise, and testing goals. Furthermore, this paper critically evaluates the trade-offs and challenges associated with each type of framework, addressing concerns related to different aspects of the framework. To assist professionals and decision-makers, we conclude by providing best practices and recommendations for effectively implementing and managing test automation frameworks. Moreover, we investigate the perspective of test automation frameworks, accentuating emerging patterns and contemplating the consequences of developing technologies like artificial intelligence and machine learning. In summary, this paper presents a comprehensive guide to the different types and trade-offs of test automation frameworks, equipping software professionals with the knowledge and insights necessary to make well-informed decisions and enhance the efficiency and effectiveness of their testing processes.

Thesis Research, 2023

The goal of this study was to examine the dynamics and applicability of manual testing in the fast-changing software development industry. The purpose was to understand the nuances, advantages, and potential limitations of manual testing in today's digital age dominated by automation. Utilising a mix of qualitative and quantitative research methodologies, we gathered insights from 8 experienced manual testers, including a test manager. The major findings reveal that, despite the surge in automated testing tools and technologies, there remains a substantial value in manual testing, particularly in areas that require a human touch, intuitive understanding, and subjective judgement. This is especially evident in the realms of user experience, real-world scenario testing, and situations where adaptability is key. While automation is undeniably pivotal, manual testing retains an indispensable role, emphasizing the need for a balanced approach to ensure software quality. Future research recommendations have also been identified to guide the industry towards best practices and further explorations.

INTERNATIONAL JOURNAL OF INNOVATIVE RESEARCH AND CREATIVE TECHNOLOGY, 2020

The software development lifecycle places significant emphasis on testing, which is required for identifying issues and ensuring high software quality. This article examines the two primary approaches to software testing, manual and automated, and highlights the critical role of automation in enhancing efficiency and effectiveness. It aims to present time-and costeffective strategies for software testing, detailing how automated testing simplifies quality evaluation and accelerates processes compared to manual testing. The research outlines the unique advantages and disadvantages of both approaches, emphasizing the importance of automation tools in streamlining testing procedures and customizing them to specific scenarios. Additionally, the paper discusses the broader implications of automation testing in modern development environments, particularly in agile and DevOps settings, where rapid feedback and continuous integration are essential. Key benefits of automated testing such as improved test coverage, consistency, and resource optimization are explored, alongside the challenges of implementing a successful automation strategy, including framework selection, test case definition, and script maintenance. Finally, this paper provides valuable insights into the methodologies and best practices that enhance software quality and expedite product delivery through Peoplesoft Test Framework automation.