Evaluating a Test Automation Decision Support Tool
Kesina Baral2019
Sign up for access to the world's latest research
checkGet notified about relevant papers
checkSave papers to use in your research
checkJoin the discussion with peers
checkTrack your impact
Abstract
Goldrich and Flamm developed the MITRE Automated Test Decision Framework (ATDF) to help MITRE government sponsors (and, via sharing on GitHub, development organizations in general) move from manually tested legacy software towards automated test, continuous integration, continuous deployment, and, ultimately, DevOps. Often such legacy systems comprise multiple components with manual test procedures. The objective of the empirical study described in this paper is to determine whether ATDF usefully ranks components with respect to Return on Investment (ROI) when introducing automated tests. ROI is simply the ratio of profit to cost. When adding automated tests, what will be the profit that these tests will carry? What is the cost or level of effort to engineer a sufficient set of automated tests? Our evaluation approach models ROI using static defect counts identified by SonarLint and estimated cost to complete testing. We found positive Pearson correlations between normalized ATDF ra...
Related papers
Quantifying Success: Measuring ROI in Test Automation
CARI Journals, 2023
The purpose of this paper to provide strategic decision to implement test automation with measurable ROI is a favoured approach for organizations seeking to improve the quality of their software, expedite delivery, and mitigate testing expenses. Methodology: The present paper, titled "Quantifying Success: Measuring ROI in Test Automation," delves deeply into the fundamental aspects of ROI evaluation within the realm of test automation. It outlines the methodology behind ROI calculation, explores the cost factors and advantages associated with automation, and sheds light on the optimal strategies for achieving a favourable ROI. Findings: By presenting real-world case studies and examples, this paper effectively demonstrates the practical application of ROI measurement in test automation. The formula for Return on Investment (ROI) typically falls under the category of being "basic". While costs are relatively simple to measure, assessing the worth of potential advantages proves to be considerably more challenging. Unique Contribution to Theory, Policy, and Practice: Approaches to calculating ROI often oversimplify various aspects of test automation, subsequently resulting in inaccurate outcomes. These inaccuracies, in turn, foster impractical expectations among management in relation to test automation, ultimately culminating in failure at individual, team, and occasionally project levels, thereby offering valuable insights to organizations seeking to optimize the efficacy of their testing endeavours.
A Framework For Evaluating Performance Of Software Testing Tools
2020
Software plays a pivotal role in this technology era. Due to its increased applicable domains, quality of the software being developed is to be monitored and controlled. Software organization follows many testing methodologies to perform quality management. Testing methodologies can be either manual or automated. Automated testing tools got massive acceptability among software professionals due to its enhanced features and functionalities than that of manual testing. There are hundreds of test automation tools available, among which some perform exceptionally well. Due to the availability of large set of tools, it is a herculean task for the project manager to choose the appropriate automation tool, which is suitable for their project domain. In this paper, we derive a software testing tool selection model which evaluates the performance aspects of a script-based testing tool. Experimental evaluation proves that, it can be used to compare and evaluate various performance characteris...
Impediments for software test automation: A systematic literature review
Automated software testing is a critical enabler for modern software development, where rapid feedback on the product quality is expected. To make the testing work well, it is of high importance that impediments related to test automation are prevented and removed quickly. An enabling factor for all types of improvement is to understand the nature of what is to be improved. We have performed a systematic literature review of reported impediments related to software test automation to contribute to this understanding. In this paper, we present the results from the systematic literature review: The list of identified publications, a categorization of identified impediments, and a qualitative discussion of the impediments proposing a socio-technical system model of the use and implementation of test automation. KEYWORDS impediments, software testing, systematic literature review, test automation, test execution 1 INTRODUCTION Software testing is by far the most commonly used method for quality assurance and quality control in a software development organization, and a very important part of the development process [IP1]. Testing makes it possible to get information about the difference between the actual and the required behavior of the software product [1] when delivered. The importance and complexity of software testing is reflected by the costs involved: 30% to 80% of the development costs are reported to be related to testing [2,3][IP2][4], and studies on release time indicate that most of the release time is consumed by testing [5]. The cost and time involved in testing can be managed through test automation, where the execution of a test is performed by software instead of a human. Test automation can be used to improve the process effectiveness, for example, by reducing the risk for human errors [IP3] and make the tests more repeatable [6]. Test automation can also be used to improve the process efficiency, for example, by enabling the use of the staff for other tasks [IP4] and perform more testing in less time [6][IP5]. Test automation is a core component in agile development [IP6], where it is used both for rapid feedback and to enable testing by everyone that deliver code. Methods such as continuous integration [7], test-driven development, and automated acceptance testing [8] makes test automation an everyday activity in an agile organization [9]. The high use of test automation also makes the automated tests mission critical: If the test automation stops working or slows down, the development will also stop or slow down.
Estimating Return on Investment for GUI Test Automation Frameworks
2019
Automated graphical user interface (GUI) tests can reduce manual testing activities and increase test frequency. This motivates the conversion of manual test cases into automated GUI tests. However, it is not clear whether such automation is cost-effective given that GUI automation scripts add to the code base and demand maintenance as a system evolves. In this paper, we introduce a method for estimating maintenance cost and Return on Investment (ROI) for Automated GUI Testing (AGT). The method utilizes the existing source code change history and has the potential to be used for the evaluation of other testing or quality assurance automation technologies. We evaluate the method for a real-world, industrial software system and compare two fundamentally different AGT frameworks, namely Selenium and EyeAutomate, to estimate and compare their ROI. We also report on their defect-finding capabilities and usability. The quantitative data is complemented by interviews with employees at the ...
A New Approach for Evaluating Software Testing Automation Tools
International Journal of Scientific Research in Computer Science, Engineering and Information Technology, 2019
The environments where the build to focus the warehouse to maintain all the business aspects are satisfied and the quality are obtained on the functionalities are acting up to the users desire. The objective of this paper is to initiate a mechanism to evaluate automation tools effectively, at the moment. There are various systems built and even if they are platform independent and also if existing with additional features they are having many demerits that don’t satisfy the end user. A significant contribution of this paper is development of this metric suite that facilitates assessment and selection of a desired testing tool for automated testing and in relation to the environment for the software is developed. In software company it is a high value for project management team selecting the tools that may satisfy the requirement satisfactions highlighted by the end user to be satisfied to maintain the business mix.
TAPN: Test Automation’s Pyramid of Needs
Anais do XIV Simpósio Brasileiro de Qualidade de Software (SBQS 2015)
Testing is an essential activity to ensure quality of software systems, but it is expensive and time consuming. Thus, testing automation would be an alternative to improve test productivity and save costs. However, many organizations refuse to use test automation or had failed on implement it because they do not know how to deal with the implementation of a test automation strategy fitted to their goals and expectations. Most of them underestimate or have no knowledge about test automation factor of success. In addition, although there are many works and maturity models focused on improving the testing process, few ones focus on test automation issues. The main contribution of this paper is to propose a hierarchical model called Test Automation’s Pyramid of Needs (TAPN). TAPN is inspired in the Maslow’s Pyramid of Needs administration theory and it comprises five levels that influence on the success of test automation initiatives in software organizations. TAPN intends to help organ...
Manual to Automated Testing: An Effort-Based Approach for Determining the Priority of Software Test Automation
2015
Test automation allows performing difficult and time consuming manual software testing tasks efficiently, quickly and repeatedly. However, development and maintenance of automated tests is expensive, so it needs a proper prioritization what to automate first. This paper describes a simple yet efficient approach for such prioritization of test cases based on the effort needed for both manual execution and software test automation. The suggested approach is very flexible because it allows working with a variety of assessment methods, and adding or removing new candidates at any time. The theoretical ideas presented in this article have been successfully applied in real world situations in several software companies by the authors and their colleagues including testing of real estate websites, cryptographic and authentication solutions, OSGi-based middleware framework that has been applied in various systems for smart homes, connected cars, production plants, sensors, home appliances, ...
Trade-off between automated and manual testing: A production possibility curve cost model
2016
Testing is always important for Software Quality Assurance (SQA) activities and key cost multiplier in software development. The decision to automate or not to automate a test case is critical. In this paper we discuss the possibility of test automation and in relation to the trade-off between manual and automated test cases. We purpose a Production cost frontier based technique to distinguish the point of automation and manual test within the cost constraints. Our objective is to identify the facts that up to what extant a testing process can be automated. In this paper a cost model is proposed for deciding the proportion of automated and manual testing. The objective is to find best possible combination of these two and production possibility in one type by eliminating the other type of testing
Towards a Test Automation Improvement Model (TAIM)
2014 IEEE Seventh International Conference on Software Testing, Verification and Validation Workshops, 2014
In agile software development, industries are becoming more dependent on automated test suites. Thus, the test code quality is an important factor for the overall system quality and maintainability. We propose a Test Automation Improvement Model (TAIM) defining ten key areas and one general area. Each area should be based on measurements, to fill the gap of existing assessments models. The main contribution of this paper is to provide the outline of TAIM and present our intermediate results and some initial metrics to support our model. Our initial target has been the key area targeting implementation and structure of test code. We have used common static measurements to compare the test code and the source code of a unit test automation suite being part of a large complex telecom subsystem. Our intermediate results show that it is possible to outline such an improvement model and our metrics approach seems promising. However, to get a generic useful model to aid test automation evolution and provide for comparable measurements, many problems still remain to be solved. TAIM can as such be viewed as a framework to guide the research on metrics for test automation artifacts.
An Empirical Evaluation of System and Regression Testing Abstract
2008
We had the opportunity to conduct an empirical study in the context of the testing environment for a large commercial product. The particular goal of the organization for which this study was done, was to gain a strong understanding of how particular aspects of their testing practice impact on the quality of the released products. In this paper we present some of the results of that research as it relates to the verification of intuitive claims of those in this industrial environment, and documented claims from other research about the relationships between several parameters. The parameters of interest to the organization were: breadth of system and regression testing of software components defined by code coverage, number of defects discovered by an in-house test team prior to the release of those software components, and number of defects discovered by the customer in the field subsequent to the release of those software components. 1
Related papers
Empirical Observations on Software Testing Automation
2009 International Conference on Software Testing Verification and Validation, 2009
This study explores the factors that affect the use of software testing automation in different types of organizations. A case study was conducted in five organizations that develop and test technical software for the automation or telecommunication domains. The data was collected from interviews with managers, testers, and developers in each organization. The grounded theory was used as the data analysis method. According to the results, the factors which emphasize the wide use of testing automation include that the tested products are generic and independent of thirdparty systems. Low human involvement in testing, steady underlying technology, and reusability support the use of testing automation.
Technical debt in test automation
Proceedings - IEEE 5th International Conference on Software Testing, Verification and Validation, ICST 2012, 2012
Automated test execution is one of the more popular and available strategies to minimize the cost for software testing, and is also becoming one of the central concepts in modern software development as methods such as test-driven development gain popularity. Published studies on test automation indicate that the maintenance and development of test automation tools commonly encounter problems due to unforeseen issues. To further investigate this, we performed a case study on a telecommunication subsystem to seek factors that contribute to inefficiencies in use, maintenance, and development of the automated testing performed within the scope of responsibility of a software design team. A qualitative evaluation of the findings indicates that the main areas of improvement in this case are in the fields of interaction design and general software design principles, as applied to test execution system development.
Trade-off between automated and manual software testing
International Journal of System Assurance Engineering and Management, 2011
The study explores the current state of test automation in software testing organizations by focusing on the views and observations of managers, testers and developers in each organization. The case study was conducted in selected organizational units that develop and test technical software for industrial automation or telecommunication domains. The data was collected with 41 theme-based interviews in each unit. The interview data was analyzed qualitatively by using the grounded theory research method. It was found that although test automation was viewed as beneficial, it was not utilized widely in the companies. The main benefits of test automation were quality improvement, the possibility to execute more tests in less time and fluent reuse of testware. The major disadvantages were the costs associated with developing test automation especially in dynamic customized environments. Such issues as properties of tested products, attitudes of employees, resource limitations, and customers influenced the level of test automation in the case organizations.
Metrics for Quantification of the Software Testing Tools Effectiveness
American Journal of Software Engineering and Applications, 2015
An automated testing tool helps the testers to quantify the quality of software by testing the software automatically. To quantify the quality of software there is always a requirement of good testing tools, which satisfy the testing requirement of the project. Although there is a wide range of testing tools available in the market and they vary in approach, quality, usability and other characteristics. Selecting the appropriate testing tool for software there is a requirement of a methodology to prioritize them on the basis of some characteristics. We propose a set of metrics for measuring the characteristics of the automated testing tools for examination and selection of automated testing tools. A new extended model which is proposed provides the metrics to calculate the effectiveness of functional testing tools on the basis of operability. The industry will be benefited as they can use these metrics to evaluate functional tools and they can further make selection of tool for their software required to be tested and hence reduce the testing effort, saving time and gaining maximum monetary benefit.
Cost effective software test metrics
2008
This paper discusses software test metrics and their ability to show objective evidence necessary to make process improvements in a development organization. When used properly, test metrics assist in the improvement of the software development process by providing pragmatic, objective evidence of process change initiatives. This paper also describes several test metrics that can be implemented, a method for creating a practical approach to tracking & interpreting the metrics, and illustrates one organization's use of test metrics to prove the effectiveness of process changes. Also, this paper provides the Balanced Productivity Metrics (BPM) strategy and approach in order to design and produce useful project metrics from basic test planning and defect data. Software test metrics is a useful for test managers, which aids in precise estimation of project effort, addresses the interests of metric group, software managers of the software organization who are interested in estimating software test effort and improve both development and testing processes.
Test Automation Process Improvement in a DevOps Team: Experience Report
13th IEEE International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2020, 2020
How to successfully conduct test automation process improvement (TAPI) for continuous development is the challenging issue faced by many software organizations. In this paper, we present an experience report on TAPI in one DevOps team in F-Secure (a Finnish software company). The team builds Windows application software and exists in F-Secure's TAPI culture. The team self-reports high satisfaction and maturity in test automation for continuous development. To study their TAPI, we reviewed a collection of experience notes, team reflection reports and telemetry result reports. The several meetings were hold to discuss the details. As the results, first, we reported that, based on the understanding of the team, test automation maturity for continuous development is defined as a set of indicators, e.g., the increasing speed to release, improving productivity of the team, high test efficiency. Second, we demonstrate that a set of critical success factors have a major impact on successfully carrying out its TAPI, e.g., incremental approach, the whole team effort, test tool choice and architecture, telemetry. Third, we discuss that TAPI practices in the observed team have similarities and differences with the ones described in prior literature. The team believes that this view of the understanding of test automation maturity and the success of TAPI in practices should be studied in order to build a comprehensive test automation improvement model that aids the guidelines of TAPI for the software industry.
Software Test Automation in Practice: Empirical Observations
Advances in Software Engineering, 2010
The objective of this industry study is to shed light on the current situation and improvement needs in software test automation. To this end, 55 industry specialists from 31 organizational units were interviewed. In parallel with the survey, a qualitative study was conducted in 12 selected software development organizations. The results indicated that the software testing processes usually follow systematic methods to a large degree, and have only little immediate or critical requirements for resources. Based on the results, the testing processes have approximately three fourths of the resources they need, and have access to a limited, but usually sufficient, group of testing tools. As
Institutionen för datavetenskap Systematic Review on Automated Testing Types, Effort and ROI
Software organizations always want to build software by minimizing their resources to reduce overall cost and by maintaining high quality to produce reliable software. Software testing helps us to achieve these goals in this regard. Software testing can be manual or automated. Manual testing is a very expensive activity. It takes much time to write test cases and run them one by one. It can be error-prone due to much involvement of human throughout the process.
Measuring the Efficiency of Introducing a Systematic Tool Based Testing Process
Abstract This paper , is based , upon , a ,software Process Improvement Experiment (PIE) under the auspices of the European Systems and Software Initiative (ESSI) being conducted at , Israel Aircraft Industries. The objective of the PIE is the application of advanced requirements management and software testing methods
An empirical evaluation of system and regression testing
2002
We had the opportunity to conduct an empirical study in the context of the testing environment for a large commercial product. The particular goal of the organization for which this study was done, was to gain a strong understanding of how particular aspects of their testing practice impact on the quality of the released products. In this paper we present some of the results of that research as it relates to the verification of intuitive claims of those in this industrial environment, and documented claims from other research about the relationships between several parameters. The parameters of interest to the organization were: breadth of system and regression testing of software components defined by code coverage, number of defects discovered by an in-house test team prior to the release of those software components, and number of defects discovered by the customer in the field subsequent to the release of those software components.
Loading Preview
Sorry, preview is currently unavailable. You can download the paper by clicking the button above.