QA metrics

Contents
Intro:..................................................................................................................................................2
Foundation: Issue reports..................................................................................................................3
Defect metrics:......................................................................................................................................4
Metric: Bugs per hour........................................................................................................................4
Metric: Regressions per hour.............................................................................................................5
Metric: Incorrect defect reports.........................................................................................................6
Metric: Escaped defects.....................................................................................................................7
Metric: Defect Age............................................................................................................................8
Metric: Time to detect.......................................................................................................................9
Cost metrics.........................................................................................................................................10
Metric: Test Cost.............................................................................................................................10
Metric: Cost per bug fix / report......................................................................................................11
Metric: Cost of not testing...............................................................................................................12
Metric: Customer satisfaction..........................................................................................................13
Test management metrics...................................................................................................................14
Metric: Test Coverage.....................................................................................................................14
Metric: Test Case Effectiveness.......................................................................................................15
Metric: Test Execution Status and Test completion status...............................................................16
Metric: Test velocity and estimates.................................................................................................17
Metric: Test case reliability/specificity............................................................................................18
 Metric: Lead time............................................................................................................................19
Developer metrics...............................................................................................................................20
Metric: Defects per requirement (Defect Density)...........................................................................20
Metric: Defect Distribution over Time............................................................................................21
Metric: Defects per software change/build/version..........................................................................22
Metric: Fix failure rate.....................................................................................................................23

Intro:

Effective software testing is crucial for delivering high-quality products. To ensure that software
testing efforts are efficient and effective, it is important to measure and analyse various metrics
related to the testing process. This document provides an overview of the different metrics that can
be used to evaluate and improve software testing, including defect metrics, cost metrics, test
management metrics, and developer metrics. By tracking these metrics, QA teams can gain valuable
insights into the testing process, identify areas for improvement, and ultimately deliver better
outcomes.

Foundation: Issue reports

Description: Issues are the foundation of QA, and most metrics use them in some capacity. If your
defect reports do not have the right information none of the other reports will matter.
A few points to consider when setting up your reporting process.

 Standards for naming your reports. Summaries should be concise and provide as much
information as possible.
 Use a common bug description template. All projects need to use the same template.
Includes the common requirements of repro steps, actual and expected results,
reproduction rate.
 Rules for attachments, what formats and resolutions are acceptable? How do issues need to
be marked?
 Priority and Severity in separate fields. This is something easy to overlook when starting.
These are not necessarily the same even, so they often go hand in hand.
 Priority or Urgency is how quickly something needs to be addressed. This can range from
dropping everything and addressing it immediately for blocking issues to never as the impact
is too insignificant. The important thing to remember is that it does not indicate the impact
of the issue or hard it is to fix. A common example of an issue that does not match Priority
with impact is typos. Spelling the name of the game in the main menu wrong has little to no
impact on the game, however, it would be very high on the priority list.

One of the main issues is that it can be hard to determine the priority of an issue. That’s why
it is advisable to define data that can be used to at least in part determine how to judge the
issue. For example, visibility can be one such data point. Is an issue seen by more than 90%
of players? That’s a good point supporting it being a higher priority.

 Severity or Impact: The other side of the medal is severity or impact. How bad the issue is
when it happens. This again can range widely from basically no negative impact to crashes
and corruption of game data. In the above example of the title typo. The issue would be low
severity, the game is perfectly playable with it.

Severity is often used on its own to determine priority. And in a lot of cases, the two do
overlap. The game crashing every time you use an item is both a high priority and high
severity.
The aim should be to address as many issues as possible. But the realities of game
development mean resources need to be allocated and using both makes it a lot easier to
decide what issues to address first.

 Time spent to find and report the issue. Depending on your configuration this could also be
recorded in the task for the tests that found the defect.
 Component/Label for the game area. A separate field that tracks the area or requirement
the issue is present in.
 Versions, fields for the version the issue have been found in, the version the issue has been
reported by the dev to be fixed in and the version QA used to confirm the fix.

Those are just some of the basic fields that can and should be used. More can be added depending
on the requirements of the projects and teams.
Defect metrics:
Metric: Bugs per hour

Description: The number of issues reported on average per hour and Analyst.

A very simple metric to judge the performance of Analysts and the overall defect density of a
project.

The main advantage of this is the ease of use, time testing and the number of defects that should
always be known. The exact detail might vary depending on the solutions used, for example without
time tracking the best data available might just be the number of days someone was assigned to a
project. Better data could account for overhead like meetings or training.

But regardless of the quality of the data, some number can always be calculated.

The availability and ease of use mean this is a great metric for quick comparisons between projects
or groups of Analysts over a longer timeframe.

The main downside is its lack of granularity. It does not consider the types of tests performed and
the expected failure rate. We would expect more issues to be found in the early stages of the
development cycle compared to the final sign-off. The same goes for Analysts and defect severity. It
assumes that time and effort are always equal whiles in reality things are more complicated. An
Analyst who spent 2h finding and reporting a major issue compared to someone who found 4 minor
collision issues in the same time, is far more complicated to judge.

How to implement: Record time spent on a project and the number of issues reported.

Uses: Good for Quick and simple comparisons, not suitable for more advanced insights.

Metric: Regressions per hour

Description: The number of regressions performed per hour and Analyst.

This is the counterpart to bugs per hour and most points that apply to the former apply to this.

It is a good way of quickly judging the progress of a project. The number of regressions is more
predictable than the number of issues. You can add fixes to a change list outside of a defect report,
but in general, the number will be closely aligned with the number of reported defects.

The higher predictability means this is more reliable when estimating the outstanding work on a
project. In particular, if it is a large project with a large enough number of tasks so that the duration
evens for each regression evens out.

It has however the same downfall as the bugs per hour metric when it comes to judging the
performance of an Analyst or a group of people. There are not enough details to compare two sets
of people fairly.

How to implement: Record time spent on the project and the number of issues regressed

Uses: Good for basic ballpark estimates on how long a large number of regressions will take. Limited
use as a metric for team performance in the absence of better options.

Metric: Incorrect defect reports

Description: There are multiple reasons an issue might be created that should not have been
reported.

Unless the project is very small it is unlikely every person on the project knows every issue reported
by heart. Most issues can be described in different ways and human error does exist. Duplicate
issues will always be reported to some degree.

While the reason for each type of incorrect report varies, they all should be tracked. This is in most
cases done through the defect status or resolution.

A defect database is only as good as the reports it contains and having a large number of incorrect
issues diminishes its usefulness for everyone.

A non-exhaustive list of common types and their causes can also be caused by simple human error or
inattention.

Duplicate issues, insufficient naming standards or issue

Cannot reproduce, insufficient repro step description and internal confirmation.

Not enough information, lack of training on reporting issues, insufficient reporting standards

As designed, lack of design documents and internal communication.

How to implement: Different resolutions or statuses for each type

Uses: Provide insight into where mistakes are coming from and provide the data required to address
them

Metric: Escaped defects

Description: The fundamental reason for having QA is to find and report issues. If this is not achieved
to an acceptable level all other metrics lose their meaning.

Escaped defects track how often an issue is not found during the QA process. There can be several
reasons for this, from human error to insufficient test coverage or incomplete documentation.
Missing issues result from a failure in the software development and testing process.

That being that not every issue is created equally. A rare graphical glitch that affects 1% of players is
not the same as an issue that breaks save games for 90% of players.

While QA should always aim to find all issues the reality of QA is that exhaustive testing is not
possible there are always going some issues that slip the net.

Therefore, part of this metric also must be the impact and financial loss caused by missing the
defect.

The number and severity and where possible potential cost of escaped defects needs to be tracked.

How to implement: We can simplify the process by focusing on issues that end up in the defect
tracker. If an issue is too minor to be properly reported it can be discarded for this metric.

All defects to consider would then be in the database. Which allows us to assign an origin designator
to them. In most cases, this is already done by default through the reporter. Issues found by QA are
reported by QA people and so on.

We can add a field or another attribute that indicates the issue has been added based on live reports
and is considered a defect. Now we have a list of all escaped defects. We can then use the other
metrics to gauge impact and cost.

Uses: Quality control for QA, confirm we are doing our jobs correctly and if not provide data on
where the gaps are.

Metric: Defect Age

Description: The age of reported issues for a project, also includes calculated values like the average
and mean age.
Leaving issues in a project unactioned for too long creates issues for all departments. Confidence
goes down over time as additional changes to the codebase might have already addressed the issue
or removed the feature entirely. The severity or priority of an issue might also change, old issues
might block new content.

Regular sweeps of the database are required to confirm which issues are still active, and while these
usually find a good percentage fixed, they do create an additional cost.

This can be mitigated by tracking the age of issues in a project and ideally agreeing on limits. This can
be a simple reminder email to the assignee once an issue reaches a certain age. Or different
approaches for priority levels and game areas.

How to implement: The data is included by default as each issue reported has a created/reported
date. Further work is required to define how to use this. Is only the maximum value tracked, and is
the average or mean age of issues tracked? What limits and triggers are selected?

Uses: There are multiple uses for this, on the QA side defect age can indicate where to focus
additional efforts, particularly if combined with defect clustering or requirements. This allows us to
pinpoint areas of a product that have been tested in the past but whose issues are not yet
addressed. Ideally, all changes would be documented and subsequently confirmed by QA, but the
realities of development mean that sometimes issues will be addressed without it being
documented.

This data makes it easier to direct sweeps of the database for issues that got addressed but are not
documented as fixed.

The related second use of this reporting the development efforts to Production/Dev. This metric
helps to highlight areas where there is a discrepancy between the expected state of development
and the actual stage.

Metric: Time to detect

Description: The time between a defect being introduced and it’s reported.

The goal is to reduce TTD as much as possible to catch issues earlier in the development process and
address them before they become more difficult or expensive to fix.
How to implement: This is not easy to track as data from multiple disciplines is likely required.

The first area of concern is to get the period for each issue. The report date is trivial to track the
change that caused it and the date of its implementation is more complex. The development would
need to find the cause of the issue and confirm which change caused the problem, having DevQA
can help in this regard. A simpler less reliable way is to compare it to the last known passed test
case.

There also needs to be an agreement on what counts as an issue for this metric. For example, how
are gameplay changes considered? Cases where something is not functionally defective but is
changed for design reasons.

Once both dates have been established the next problem is how to extract the data and combine it
into a useful graph. Does the test management solution have a way of tracking this or is there a need
to process the data in external tools like Excel?

Uses: This is important because it helps to identify potential bottlenecks and inefficiencies in the
testing process. By measuring the time to detect, teams can identify areas where improvements can
be made, such as implementing automated testing, improving the effectiveness of manual testing, or
reviewing new testing techniques or tools.

Cost metrics
Metric: Test Cost

Description: The overall cost of QA

Test cost in QA refers to the financial cost associated with testing a product. This includes mainly the
cost of the Analysts doing their work, but this can include the cost of hiring QA testers, purchasing
necessary equipment and tools, and any other expenses related to the testing process.

Test cost can be an important factor to consider when budgeting and can be influenced by various
factors, including the complexity of the product, the scope of the testing required, and the resources
needed to complete the testing process. Test cost can also be a useful metric for tracking the
efficiency and effectiveness of a QA team, as it can help to identify areas where costs can be reduced
or optimized.

How to implement: Time tracking is required for this. At the bare minimum the number of people
assigned to a project and the days they worked. Where more data is available, like individual task
times, this should be used. Similarly, invoices for external vendors and general costs like office space
and training should be accounted for.

Uses:

Knowing the test cost for QA can help to accurately budget for testing activities, ensuring that
sufficient resources are allocated for this important phase of the development process.

Resource allocation: The cost for QA can be used to determine the most cost-effective way to
allocate resources for testing, such as by identifying which testing activities require the most
resources and prioritizing them accordingly.

By comparing the test cost for QA to the potential returns from a successful product launch or
another business outcome, a company can determine the potential return on investment for their
testing efforts. This is currently done with the PDD process.

Continuous improvement: By tracking the test cost for QA over time, we can identify trends and
areas for improvement, such as by identifying and addressing inefficiencies in the testing process or
by identifying opportunities to reduce the overall cost of testing.

Metric: Cost per bug fix / report

Description: How much does it cost in total to fix an issue

This metric is one step further down the line from the before mentioned Test Cost. While test cost
looks at the whole project or company this one looks at individual issues.
The QA metric cost per bug fix is a measure of the resources (typically time and money) required to
fix a bug in a software application. The cost can be calculated on an individual basis if there is strong
enough data or the total cost of bug fixing can be divided among all issues.

One use of cost per bug fix is to help prioritize which bugs to fix first. For example, if the cost of fixing
a high-priority bug is significantly higher than the cost of fixing a lower-priority bug, it may make
more sense to fix the lower-priority bug first.

Without the data on how long it took to address the issues, this can still be used with just the QA
input to calculate the cost per report.

It is also important to account for overhead like project lead time and training time.

How to implement:

Identify the resources required to fix a bug. This could include the time required to identify and fix
the bug, as well as any costs associated with the fix (such as the cost of additional hardware or
software

Track the total cost of fixing each bug. This can be done by keeping a record of the time and
resources required to fix each bug for example in JIRA.

Uses:

Identifying trends: By tracking the cost per bug fix over time, you can identify trends in the efficiency
of your QA process. For example, if the cost per bug fix is steadily increasing, it may indicate that
your QA process is becoming less efficient.

Setting benchmarks: By comparing the cost per bug fix to industry benchmarks or your historical
data, you can determine whether your QA process is performing at an acceptable level.

Identifying areas for improvement: By analysing the cost per bug fix for different areas of your QA
process, you can identify areas where improvements can be made. For example, if the cost per bug
fix is significantly higher for manual testing compared to automated testing, it may be worth
investing in more automation to reduce costs.

Metric: Cost of not testing

Description: The cost of not conducting tests in finical and other terms.

The cost of not testing refers to the cost of fixing any feature that went into production without
testing, failed and needed fixing.
The cost can be financial as it is more expensive to address an issue in production but there are
other areas to consider such as increased customer complaints, lost revenue or reputational
damage.

Calculating the cost of not testing might seem counterintuitive but is a great way to establish the
necessity of QA functions or find areas whose impact on the project needs to be evaluated.

Monitoring this QA metric is especially important if you must justify increased budgetary or hiring
requests to stakeholders.

How to implement:

This is not a metric that should be tracked by default. It relays heavily on the experience of the QA
team and requires substantial manual work to calculate. Reports on this should be created on a
case-by-case basis to provide additional support to ongoing discussions regarding the QA needs of a
project.

Uses:

There are two main use cases for this. Highlighting the risk of not spending enough resources on QA,
in particular during the early stages.

The second is the opposite, highlighting areas where too much time might be spent. If the cost of
testing something is £10000, and the worst outcome is £5000 the testing strategy needs to be
evaluated.

Metric: Customer satisfaction

Description: How satisfied the end user is with the product they received.
Measuring customer satisfaction is crucial for any business, as it can provide valuable insights into
the success of a product or service.
However, simply focusing on metrics such as bug fixes and performance improvements may not be
enough to fully gauge customer satisfaction. A technically sound product may still fail to meet the
needs and expectations of its target audience. For example, a product that meets all technical
requirements but lacks the features that users desire may not be successful.

In contrast, a product that is engaging and enjoyable to use may succeed even if there are some
residual technical issues. This is especially true in the world of gaming, where user engagement and
enjoyment are often more important than technical perfection. A game with some remaining bugs
may still be highly rated if it is fun and immersive.

In other words, a red sign-off indicating unresolved technical issues may be less important if the final
product has a high approval rating from users. This highlights the importance of considering
customer satisfaction as a key metric for product success.

It's important to think about whether it will be a long-term title. If a product is intended to have a
long lifespan, it's critical to continue listening to user feedback and making updates and
improvements over time.

How to implement:

One way to gather additional data points is by implementing beta programs or Early Access
initiatives, which enable users to test the product and provide feedback before the official release.
This can provide valuable insights into user experience and highlight potential areas for
improvement.

In addition, conducting in-house usability and acceptance testing can greatly improve the quality of
data collected. These tests allow product teams to identify and address issues in a controlled
environment before they affect the end user.

Uses:
Tracking user satisfaction is best suited for a mortem process. Once all other metrics are ready for
review, this helps to put them into perspective. Did we release a good product because of our
testing or despite it?

Test management metrics

Metric: Test Coverage
Description: Percentage of tests run for a given area/requirement.

This is another fundamental metric that is usually tracked in some way even if not called test
coverage. An excel test case that counts the number of executed tests is one of the easiest examples
of this.

This can be tracked on multiple levels. From just tracking the executed test cases down to tracking
every step inside a test case. The deeper you go the more challenging it becomes to track this
without getting bogged down in admin tasks. This should also include tracking of platforms and
versions.

A good test management solution should take care of most of this and allow for precise tracking of
when and where each test has been run and by whom. And as a result, tell you exactly what parts of
an application have been sufficiently covered.

It’s easy to calculate this with two quick formulas:

 Test Execution = Number of tests already run/Total tests to be run) x 100

 Requirements Coverage = Number of requirements covered by existing tests/Total number

of requirements) x 100

How to implement:

Basic versions can be done manually or by using simple excel files. But ideally, this is tracked by a
dedicated test management solution that automates most of the process.

Uses Estimates on remaining time for test plans. Overall performance indicator and more
importantly key metric in assessing how completely an area or product has been tested.

Metric: Test Case Effectiveness

Description: Number of issues found per execution of a test case/test plan.

At its core, this metric will tell you how many issues on average you can expect if you run a particular
test case or plan.
With some exceptions, you would expect this to improve over time. Reflecting more experience in
your team and a better understanding of the previous metric, defect density. Allowing for more
targeted test cases.

How to implement: Can be done by tracking the test case in a separate field of the defect report.
Preferably done by using a test management solution that allows an automatic link between test
execution and issues.

Uses: This is the tricky part of this metric. How to read the numbers. The number of issues to expect
is not constant.

Some test cases are run not with the expectation of finding issues but to confirm no new issues have
been introduced. Others are run to find as many issues as possible. And there are plenty more
reasons for having a particular test case.

The varieties of requirements for these mean numbers can be hard to standardize across the board.
A rate that is perfectly fine for a test case might be unacceptable for another.

To add to this raw numbers are usually not as descriptive as numbers accounting for severity and
priority. For example, in two test cases, has produced 10 issues and another 3 issues. If you just look
at the raw numbers, the one with 10 is finding more issues and might be considered better. But if
you look at the issues in more detail things can change. In this example out of the 10 issues, one was
a 1 high priority and severity issue 2 were medium and 7 were waived and are not going to be fixed.
The 3 issues from the second test case were all high-priority and severity issues. Making it a lot
harder to say which one is the more efficient.

Metric: Test Execution Status and Test completion status

Description: The status of all tests currently being executed or expected to be executed and the
percentage of test cases for each resolution.

It is important to have access to up-to-date information on the number of tests that have passed,
failed, is blocked, incomplete, or not executed. This data can be presented as numbers and/or
percentages and is crucial for daily or weekly reporting. This metric provides a quick overview of a
team's progress and how it tracks against expectations.

Test execution tracks how close to completing a test cycle is.

Test completion tracks the outcome of each test case. In short, the former tells you when testing
while complete while the latter tells you how bad the results are.

This is usually the percentage of test cases for each outcome, for example, the percentage of passed,
failed, and blocked tests.

How to implement: The Implementation for this strongly depends on how tasks are tracked. A lot of
test management solutions of inbuild functionality for this. At its core what you need to be able to
track is the state of each test case. Once that data is available tools like Excel can be used to improve
presentation.

It can be used for long-term tracking but is generally more suited during the test execution phase to
notice and address immediate concerns.

Uses: Being able to determine the status of a project or test cycle at a glance is an essential ability
capability when trying to manage a project.

Metric: Test velocity and estimates

Description: The measurement of how quickly test cases are executed and how they compare to the
initial estimates.

How to implement:
The first step is to make sure all test cases have an associated estimate, most test management
solutions will have functionality for this. It is also beneficial if the tasks are created so they all take
roughly the same time to execute. Similar execution times make basic estimates easier.
The second part is that time needs to be tracked, again most software can do this and generally the
more involved part is to get all parties to log accurate time against the testing tasks.

Uses:
When creating test cases, an estimation of the time required to execute them is made. This
estimation can vary based on the complexity of the test case and other factors. Measuring the actual
time taken to execute the tests allows for better planning and adjustment of testing activities. For
instance, if the tests are taking longer than expected, adjustments can be made to staffing and
resources to stay on schedule.

By monitoring and analysing the difference between the actual and estimated time for test
execution, forecasting for future test cycles can be improved. This enables the QA team to make
more accurate estimates and allocate resources more effectively. By continually improving the
estimation process, more accurate forecasts can be created, helping the team to work more
efficiently and effectively.

Metric: Test case reliability/specificity

Description: How reliable are the results of a test set?

This metric measures how reliably an issue adheres to expectations.

This is not a measurement of does a test case or set finds issues but if those are what you expect
from an ideal test case for this feature.

The number of failed test conditions and issues raised should correlate. If a test set checks for 5
conditions but 30 issues are logged it indicates other areas have been picked up by it as well.
If the feature under test is error-free the test case should not report issues anymore.

The issues reported are issues and not incorrect reports, for example, duplicates or issues arising
from unclear documentation.

Depending on the implantation other data points can be included in this.

The main question this metric is trying to answer is, did we find these issues because the test cases
were well designed and executed or just because we had eyes on the product?

How to implement: This is one of the harder-to-implement metrics. It requires a high amount of
derived data and manual assessment. Information such as defect numbers needs to be manually
reviewed to confirm their specificity.

With a solid test management structure this can be mitigated to some extent but ultimately to get
the most precise data this will have to be collated manually.

Uses: Test case reliability is used to direct test planning. Test cases that are known to be too
imprecise can be redesigned or where the specificity is not required replaced by more open-ended
testing mythologies.

Metric: Lead time

Description: The timespan between a change being added to a build and QA either starting or
completing related test activities. Mostly due to test management activities and general overhead.

It is important to clearly define what is included in Lead Time, such as the time from change
submission to testing initiation.

How to implement: This is both easy and hard. The theory is very simple to take the time stamp of
submission to QA and compare it to the timestamp of the QA task running the required checks.

The hard part is to do this in an automated way. As doing it manually for each change would create
too much overhead.
Uses: This is a useful indicator of how flexible and agile the test process is. A long delay between
submission and execution can indicate problems with the test management process such as
bottlenecks and ineffective processes.

Developer metrics
Metric: Defects per requirement (Defect Density)

Description: The number of issues found per requirement, gameplay area or any other way a
product is divided. How areas are divided can be project specific. It can tie into the development
process, in particular in agile environments. It can be based on aspects of the game, like levels or
items, or any other metric that works for the project. The important part is that all stakeholders
agree on the division. A metric where QA reports issues per level is useless if all other departments
would rather have the density based on discipline.

Not all areas of a product will be at the same level of development. Some areas will be further
ahead, some areas will have higher priority. And even in cases where all parts are supposed to be at
the same stage, conventional wisdom holds that about 80% of issues come from about 20% of
features. It is therefore important to know what this 20% for each project is. If you know where
issues are clustered, you can plan appropriate test cycles and assign resources efficiently. Ideally, all
areas would get the same attention however the reality is that limitations in time and money mean
tough decisions need to be made on where to spend resources.

How to implement: It is possible to do this manually. You can use tags/components/labels/etc to

mark issues. And then count the number of issues per category. A better way is to use a system that
allows you to associate issues with the areas directly. For example, in Xray, the development task for
making a character could be covered with the tests for it and the issues found.

Uses: If you know where issues are clustered you can plan appropriate test cycles and assign
resources efficiently. Ideally, all areas would get the same attention however the reality is that
limitations in time and money mean tough decisions need to be made on where to spend resources.

Metric: Defect Distribution over Time

Description: Number of issues reported by the time

Defect Distribution over Time is a metric that shows the number of defects found at different points
in time, usually over the course of the development or testing process. This metric can be useful for
identifying trends or patterns in the number of defects found, and for identifying areas of the
software where defects are more likely to occur.

How to implement:

Uses:
Identifying areas of the game where defects are more common by correlating the number of issues
with the integration dates for functionality so that additional testing or debugging can be focused on
those areas.

Identifying problems with the development process that may be contributing to the number of
defects found, so that those problems can be addressed.

Providing visibility into the number of defects found over time, so that stakeholders can track the
progress of testing and debugging efforts.

Metric: Defects per software change/build/version

Description: The number of defects found in a specific version or build of a product

This measure is often used to assess the quality of the software development process and to identify
areas where improvements can be made. This measure is useful for identifying trends in defects and
for comparing the quality of different software versions or releases. It can also be used to assess the
effectiveness of QA processes and to identify areas where more resources or attention may be
needed to ensure the quality of the software.

It could also be used to track the efficiency of the testing process, as a high number of defects per
software change may indicate that the testing process is not adequately covering all relevant areas
of the game or that defects are not being detected and addressed promptly.

How to implement:
This can be tracked by adding the version of the issue found into each defect report.

Uses:

To measure the quality of the codebase: By tracking the number of defects per software change, a
team can gauge the overall quality of the codebase. A high number of defects per change indicates
that the codebase may have issues that need to be addressed, while a low number of defects per
change indicates that the codebase is of high quality.

To improve the efficiency of the testing process: By tracking the number of defects per software
change, a team can determine how much time and effort is required to test a given change. This can
help the team optimize the testing process and allocate resources more efficiently.

To identify and address problems with the development process: By tracking the number of defects
per software change, a team can identify problems with the development process that may be
contributing to defects. This can help the team address those problems and improve the overall
quality of the codebase.

Metric: Fix failure rate

Description: Percentage of claimed fixed that are rejected as not fixed.

The fix failure rate is a metric used to measure the effectiveness of bug fixes. It is defined as the
number of bugs that were fixed but subsequently found to be still occurring or recurring in
subsequent releases, divided by the total number of bugs that were fixed.

How to implement:

Before gathering data, QA and Development should agree on which issues are included and how not
fixed is defined and how to handle partial fixes. For example, a performance fix that improves the
average FPS from 35 to 59 but fails to reach the required 60 FPS. How is this counted as it improved
the problem but not reached the required level?

Uses: Fix failure rate is an important metric for evaluating the effectiveness of the bug-fixing process
and identifying areas for improvement in the development and testing processes. A high fix failure
rate can indicate a lack of thoroughness in testing or inadequacies in the bug-fixing process on the
developer side, while a low fix failure rate indicates a high level of quality in the development
process.

Version: 1
Author: Simon Sandrock
Disclosure: ChatGPT was used in the creation of this document