It’s that time of year again, with the publication of the 2024-25 World Quality Report!

I’ve reviewed this annual epic from Capgemini for the last four years and will do so again here, albeit for the last time (see my reviews of the 2018/19, 2020/21, 2022/23 and 2023/24 reports in previous blog posts).

This 16th edition is titled “New futures in focus”, only slightly different from last year’s “The future up close”.

I’ve taken a similar approach to reviewing this year’s effort, comparing and contrasting it with the previous reports where appropriate. This review is again lengthy, but you’ll still save plenty of time by reading my summary compared to devouring the 106 pages of the report itself!

TL;DR

It was a bad omen when the first thing I noticed on opening the PDF report in Chrome was a mistake in the report’s title meta-data – “Brochure Potrait” appeared in the document header and in the name of the browser tab. Moving right along…

The 16th edition of the “World Quality Report” is the fifth consecutive report I’ve reviewed in detail on this blog. It’s a hefty report, yet consistently fails to produce the genuine insights I’d expect following such a mammoth undertaking (including about 900 hours of interviews with senior representatives at large organizations). In some ways, I’d argue it’s a report “by CTOs, for CTOs”.

The previous year’s report made bold claims about AI and these continue in the current report. While the authors conclude that uptake of Gen AI is huge and revolutionary, I’m not convinced that the data supports this view and neither does my experience in the industry.

“Quality Engineering” is not clearly differentiated from testing again and many other terms are used without clear definition (e.g. Green IT) so there’s significant potential for confusion, both in interpreting the results and for respondents answering the questions.

I still think it would be valuable to include some questions around human testing to help us to understand what’s going in these large organizations, an observation I’ve made in all of my previous reviews of these reports.

Once again, the focus areas of the report changed almost completely, from eight last year to six (largely different) areas this year, making cross-report comparisons difficult or impossible. The sample set of organizations appears to be the same as last year (maybe in the interests of year-on-year comparison), so I really don’t understand why the report doesn’t stick to a standard set of focus areas year on year.

There continue to be many problems with this report. The lack of responses from smaller organizations mean that the results remain heavily skewed to very large corporate environments. There are numerous errors that really should have been picked up in proofing and the report is poorly copyedited, resulting in a report that feels like a bunch of disparate sections stuck together with no consistent voice. It’s good to see that the data visualizations this year are much simpler and more consistent than last year, though, making the results much easier to interpret.

While these reports blow with the wind of trends, I’d recommend that you keep an eye on trends in testing but don’t get prematurely attached to them. Focusing on building excellent foundations in the craft of testing will likely be time better spent, enabling you to navigate the winds of change throughout your career. Big name industry reports like this one carry substantial weight – regardless of their content – so stay mindful of the hype and adopt a critical thinking mindset when it comes to the conclusions made in reports like this.

About the survey (p96-101)

This report maintains its trend of becoming longer every year, running to 106 pages – up from 96 pages last year and “just” 80 pages the year before that.

I find it useful to look at the “About the survey” section of the report first to understand where the data came from to build it and support its recommendations and conclusions.

Notably new in the study design this year is that “…advanced AI-driven tools were integrated into the research process to enhance data quality”, so this needs to be kept in mind when looking at the findings of the report, I think.

The survey size was 1775, up slightly from the previous report (with 1750). Given the very similar survey size, it’s surprising the resulting report is longer.

The organizations taking part were again all of over 1000 employees, with the largest number (34% of responses) coming from organizations of over 10,000 employees. The response breakdown by organizational size was essentially the same as that of the previous four reports, strongly suggesting that it’s the same organizations contributing every time. I’d love to see the responses from those in smaller organizations where the technological and organizational context is likely to be very different.

While responses came from 33 countries (up from 32 in the previous report), they were heavily skewed to North America and Western Europe, with the US alone contributing 16% and then France with 8%. Industry sector spread was similar to past reports, with “Hi-Tech” (19%), “Financial Services” (15%) and “Public Sector/Government” (11%) topping the list (and all exactly the same percentages as the previous report, reinforcing my hypothesis that the same organizations are reporting each time).

The types of people who provided survey responses this year was also very similar to previous reports, with CIOs at the top (24% again), followed by QA Testing Managers and IT Directors. These three roles comprised over half (59%) of all responses – again exactly the same breakdown as the previous report.

Introduction (p4-5)

The introduction sets the scene for a big focus on AI (or “Gen AI”) in the report, so no surprises here. This finding troubles me, given the large organization focus of this report’s underlying data:

The other striking result is that there is more acknowledgment of the importance of quality (or rather the risk and impact of insufficient quality) – but organizations still need to work to give true strategic attention to the topic of quality.

Executive summary (p6-7)

The high-level summary of the report also makes a big deal about AI, but still manages to make some noteworthy observations.

On the model for where QE (“Quality Engineering”, a term that’s used throughout, never defined and sometimes used interchangeably with testing) fits into teams:

First, the integration of quality engineers within agile teams has become a standard practice, with 40% of organizations now embedding these experts directly into their agile processes. Second, we are witnessing a rise in organizations that not only integrate quality engineers into agile teams but also maintain dedicated Quality Engineering roles operating independently to ensure comprehensive coverage and oversight.

There’s a refreshing honesty around the actual use cases for AI:

The debate on which Quality Engineering & Testing activities will benefit most from Gen AI remains unresolved. This year’s survey highlights a growing focus on leveraging Gen AI for test reporting and data generation over test case creation.

While I noted in my review of last year’s report that the expectations of “QE experts” were getting very high (specifically including skills in coding, BDD and TDD alongside their idea of the typical QE skillset), this year they ramp it up even more:

Over the past decade, the shift to Agile methodologies, cloud computing, and smarter technologies has transformed quality engineers into SDETs (Software Development Engineers in Test) and, further, into full-stack test engineers. The skill set requirements for quality engineers have now expanded even further, encompassing data proficiency, AI expertise, Gen AI capabilities, and product engineering skills. However, this evolution does not diminish the fundamental need for risk-based test strategies, human collaboration, and deep business expertise—elements that remain crucial for ensuring comprehensive and effective Quality Engineering.

This unicorn “full-stack test engineer” nonsense really needs to stop. In my recent experience, it’s hard enough to find good testers without all these additional expectations on individuals. Many of the skills they’re throwing onto the “full-stack” are genuine specialties in their own right and we should treat them as such.

Key recommendations (p8-9)

This year’s recommendations are across six areas (down from eight last year and it’s not obvious which ones are the same and which are different, though “Intelligent Products Validation” appears similar to the previous “Intelligent product testing” and “Quality Engineering in Sustainability” probably relates to the previous “Quality & Sustainability”), as follows:

• Quality Engineering in Agile
• Quality Engineering Automation
• Quality Engineering and AI
• Intelligent Products Validation
• Quality Engineering in Sustainability
• Data Quality

Changing the areas and categories every year is very unhelpful and makes cross-report comparison too difficult. If I was being cynical, I would argue that this could be a deliberate move to make it hard or impossible to see how well their predictions pan out over time. There are three to five recommendations made in each area, so let’s dig into the details, area by area.

Current trends in Quality Engineering & Testing (p10-61)

Half of the report is focused on current trends, broken down into the six areas detailed in the previous section. As usual, the most interesting content is to be found in this part of the report. I’ve broken down my analysis into the same sections as the report. Sitting comfortably?

Quality Engineering in Agile

The introductory spiel for this section has an unusual tone and I particularly dislike the “Great reset” language (triggering inevitable World Economic Forum flashbacks to the atrocities of the pandemic response for me). This “reset” mantra appears again later in this section of the report.

The first set of remarkable data is around the “top 5 skills for your Quality Engineering associates” with an astonishing 70% saying “Quality Engineering skills”!:

There is a data error in this first chart: note that “AI/ML and Gen AI skills” are repeated, with one showing 66% and the other 57%. Putting two and two together from the commentary on this data, it seems that the 57% stat is for “coding skills”.

The next set of data relates to organizational structure:

I find the options on offer here quite difficult to differentiate and I can imagine respondents having difficulty in choosing the right options for their organization’s context. On this, the authors say:

A notable change is the decrease in the use of traditional Testing Centers of Excellence (TCoE), with only 27% of respondents reporting their continued use. This marks a substantial drop from the 70% who relied on TCoEs last year. While the results from last year’s survey seem high, likely due to conflicting interpretations of “TCoE”, the decrease this year is clear and consistent with other survey responses. Concurrently, 40% of respondents now have quality engineers embedded within Agile teams, highlighting a trend towards embedding Quality Engineering into Agile workflows.

The 70% claiming to use TCoEs in the last report always looked wrong (as I pointed out in my review last time) so it’s no surprise that a very different response was found this time. This also shows some confirmation bias on the part of the authors and they are acknowledging that their poor (or, more accurately, complete lack of) definition of the terms they use is likely impacting on the validity of the results.

The supporting commentary for figures 3 and 6 appear to be the wrong way around so it makes for a confusing read! Moving on to looking at challenges for QE adoption:

As development skills have become less critical and the focus has intensified on Gen AI and core Quality Engineering competencies, it appears that the broader value of Quality Engineering is not being fully recognized. The core problem may not lie in the alignment with development teams, but rather in demonstrating tangible value. Despite an increase in the use of advanced technologies like Gen AI and expanded automation coverage, the perceived value of Quality Engineering remains underwhelming.

This is quite a remarkable observation and is a sad indictment of trend following, if that is indeed what these large organizations are trying to do. The WQR has been championing the move to QE – and away from what I’d call testing – for many years, but this change of focus appears to be failing to achieve good outcomes for anyone.

Turning to their recommendations in this area (remember, it’s “Quality Engineering in Agile”), these two stood out to me:

Integrate quality engineers directly into product teams to ensure their work
is closely connected with product development and outcomes.

Maintain the independence of testing. As systems continue to increase in
complexity with multiple technologies and hosting locations, the benefit of
an independent testing team will pay dividends.

These recommendations seem to contradict each other, unless “quality engineers” are not being thought of as providing testing services. This murky distinction between QE and testing in this report (and commonly in the broader industry) is leading to a lot of confusion, pointless rebranding and problems for those of us focused on testing as a craft in its own right.

Quality Engineering Automation

This section of the report leads with AI, of course, asking about the extent to which organizations are using it to enhance the “maturity of test automation”:

My read of this data is that, at best, 29% of the respondents are really doing anything meaningful with Gen AI today. The authors choose to put a more positive spin on uptake:

In 2023, 69% of organizations were experimenting with innovative automation solutions like Low Code/No Code or AI-based automation frameworks. Fast forward to 2024, and the landscape has shifted by leaps and bounds. New futures are in focus – as 29% of organizations have fully integrated Gen AI into their test automation processes, while 42% are actively exploring its potential.

Further to this, when asked about the benefits of Gen AI in enhancing test automation, there were no surprises – “Faster automation” came out top (72%) and “Reduce testing effort/resources” was close behind at 62%.

The following claim doesn’t appear to be supported by any of the data in this report:

The survey results reveal that the global average level of test automation has now gradually increased to 44%

It’s hard to know what this really means – I can imagine different organizations measuring their “level of test automation” quite differently (especially as this is a difficult thing to measure meaningfully), so this average level is probably not indicative of anything – 44% might be great, might be terrible or might mean nothing at all.

The data around business outcomes achieved through test automation caught my eye:

So “Over half of the respondents highlighted that automation reduces manual effort” while the most popular response suggests that there is “improved testing coverage” which increases “confidence in IT”. I find this interesting as those business stakeholders gaining confidence from seeing increases in reported test coverage likely have no idea what’s being measured or the quality of the automation that’s been built.

Turning inevitably to “cost benefits of automation”:

It’s pleasing to see that a large proportion of respondents don’t use cost benefits as the primary driver for test automation, but the authors still claim “One of the key benefits of automation is reducing operating costs”. There appears to be another error here, I assume the the last bar should be labelled “Decreased operational costs due to additional tooling” (since increasing these costs wouldn’t generally be seen as beneficial).

Looking at the “talent requirements” around automation next:

I think it’s a big ask that “31% of respondents identified the need for full-stack engineers – quality professionals with additional expertise across the technology stack, including infrastructure, cloud, performance, resiliency, and reliability.” It’s good to see such a low percentage of respondents believing “developers can do all forms of testing so separate testing is not required”, but I struggle to believe that “Manual testing is still prevalent due to specific application architectures” in only 10% of organizations (and I don’t see why the latter part of that response was included or relevant). If we are to believe the previous claim that the “global average level of test automation” is only 44%, what testing activities are covering the rest if only 10% is “manual testing”?

The recommendations are not very exciting, though the emphasis on increasing the use of AI stuck out to me:

Harness the potential of Gen AI to enhance and accelerate test automation. Gen AI goes far beyond the generation of automated test scripts and helps with the realization of self-adaptive test automation system, driving efficiency and effectiveness.

I didn’t see any data in this section of the report to support the efficiency or effectiveness claims made in this recommendation.

Quality Engineering and AI

Given the authors’ overwhelming focus on AI, I was surprised to see this being one of the shorter sections of the report. The opening gambit makes a bold claim:

The results from this year’s survey indicate what we believe is the new future – Gen AI-augmented Quality Engineering. We found that 68% of respondents have moved beyond the experimentation phase and have adopted Gen AI platforms to improve their overall IT efficiency and accelerate their speed to market.

The 68% stat is a little misleading, as it’s based on these results:

The authors have combined the first two responses to produce their 68% stat, but half of these respondents are not actively using Gen AI solutions yet. The next set of data is quite interesting, looking at the testing-related use cases for Gen AI:

The authors note that their client experiences don’t match this data, so they continue to recommend focusing on the bottom two use cases as “there are greater gains to be made in those areas”. I’m encouraged by this statement, though:

Gen AI isn’t about replacing the human touch or magically improving testing quality on its own. Instead, it’s a game-changer for boosting the productivity of quality engineers.

This section closes with a big call to action:

Although the sheer volume of data may feel overwhelming, one thing is clear: Gen AI will revolutionize Quality Engineering. Whether you jump in headfirst or just dip your toes in, you need to start your adoption now!

The jury is still out on the use of Gen AI for most of the organizations contributing to this report, so this call to action seems too strong – and makes me wonder whether the authors are hallucinating just like the Gen AIs they’re discussing.

Intelligent Products Validation

The first two data sets in this section of the report are closely related, essentially asking about the importance of different types of testing for “intelligent products” (a term that isn’t clearly defined in the report):

These results seem contradictory to me. For example, looking at “Security”, 60% rate the security “test phase” as being very important (and the highest of all the phases), yet only 23% said that security was the most important aspect of validating an intelligent product (a worrying stat in itself!). These contradictory results don’t seem to worry the authors, though, who conclude:

When it comes to testing intelligent products, the emphasis on different test phases directly mirrors what respondents consider crucial for validation

I didn’t spot much else of interest in this section, with the recommendations unsurprisingly focusing on increasing the use of AI.

Quality Engineering in Sustainability

Turning to sustainability and “Green IT” (another term used in the report that is not clearly defined), the opening data relates to the prioritization of sustainability:

The authors conclude that “… a whopping total of 98% of organizations acknowledge that sustainability is extremely crucial to them!” while, in reality, the first three options are worded in such a way that I’d fully expect every organization to pick one of them (especially given the senior folks being interviewed). There’s another data error in this chart too, with the response percentages adding up to 102%.

The next set of data looks at focus areas to validate to drive Green IT:

I don’t understand how some of these choices are related to sustainability or Green IT, e.g. “the ability of IT systems, devices or software to work together for seamless functionality” doesn’t seem to me to have anything to do with efficient use of resources, sustainability, etc.

I find the data in the next chart impossible to believe:

The claim that almost half (43%) of the surveyed organizations are monitoring the environmental impact of every type of testing beggars belief. I have no idea how you would even go about doing that in any meaningful way. This data seems to feed into the recommendation to “Practice sustainable testing”, whatever that means.

Data Quality

There are few revelations in the section of the report relating the quality of test data. When it comes to provisioning test data, it appears more organizations are turning to AI (unsurprisingly):

There is a data/proofing error in the commentary on this data, saying “More organizations are turning to AI-generated test data (49%)”.

Looking at the issue of bias in test data next:

Again, AI is being seen as silver bullet here but kudos to the authors for acknowledging the potential issues with this (especially given the prevalence of using AI to also generate the data!):

Almost a third of organizations rely on AI to check data quality and remove biases (34%), but this approach often lacks transparency and context, which can unintentionally reinforce existing biases.

In most ways, this section of the report feels like the same old, same old with a sprinkling of AI while not suggesting any genuine improvements in the area of test data management, an observation also made by the authors in conclusion:

For over 15 years, we have been asking questions about data and its importance in the World Quality Report. Each year, organizations talk about focusing more on this, yet the same perceptions persist about the quality and importance of data. Despite the critical role data plays in AI and organizational success, many organizations still do not give it the focus it deserves.

Sector analysis (p62-95)

The sector analysis has generally not been as interesting as the trends section in previous reports and the pattern continues this year. The authors identify the same eight sectors as the previous year’s report (albeit presented in a different order for some reason), viz.

• Automotive
• Manufacturing
• Consumer products, retail and distribution
• Healthcare and life sciences
• Public sector
• Financial services
• Telco, media and technology
• Energy, utilities, natural resources and chemicals

A few things caught my attention in the sector analysis section:

• In the Retail sector, 31% “Believe that an environmental impact is monitored for each testing activity.”
• In the Public sector, 34% “Prefer to use developers to perform automated testing rather than dedicated SDETs” and this is the first time I’ve seen the conflation of AI and “shift-left”, I think: “The AI revolution, particularly the shift-left approach, is driving the need for faster, cheaper, and more predictable solutions.”
• In the Financial Services sector, “Approximately 50% of financial institutions are now looking to reduce their dependency on IT services from India and the US, instead opting to relocate resources to Latin America (LATAM)…. For example, Mexico has become the fourth largest IT market, with a growing pool of IT professionals and STEM graduates.” This is an interesting development and makes sense from a timezone perspective for US organizations, so it’ll be fascinating to see how the established outsourcing players respond to this geographical shift. A remarkable 75% “Responded that Quality Engineering skills are considered the most critical for Quality Engineering associates in the financial sector”!
• In the Telco sector, “… the need for critical testing is still new, and while organizations are slowly understanding the implications of overlooking it, the impact will be increasingly apparent in the future.”

(There are more basic proofing errors in this section of the report with repeated text in both the Retail (in the “Shifts in consumer buying channels” section) and Public (in the “Managing complexity in Cloud-based environments”) sectors.)

Geography-specific reports

The main World Quality Report was again supplemented by a number of short reports for specific locales. I only reviewed the Australia/New Zealand report and spotted a couple of interesting observations.

Firstly:

In ANZ, there’s been a shift from a scaled Agile approach to a more balanced methodology. Organizations are reassessing the value of dedicated testing teams versus integrating testing within engineering squads. While early testing and shift-left practices are gaining momentum, the specialized expertise in functional assurance, integration, and continuous testing is being reaffirmed.

I’m not sure how to interpret this, whether they’re saying there’s a move towards dedicated testing teams or away from them and towards embedded testers in teams. This is another example of poor wording in the report that should have been caught by decent copy editing.

The comment around automation is pretty stunning:

Automation … continues to be both an opportunity and a challenge. While some organizations are early adopters, many still struggle with the complexities of automating business-as-usual (BAU) projects.

Calling anyone undertaking automation today an “early adopter” seems absurd to me when we’ve had automation in various forms available to us in testing for decades. This very report in previous years has indicated the growing adoption and importance of automation.

Continuing the trend of “AI everywhere” in this report, it says this about using AI specifically for “hyper-automating test design”:

Gen AI is rapidly becoming a hot topic in ANZ, with organizations exploring its potential to address efficiency pain points within the testing cycle. While there is growing interest in leveraging Gen AI for hyper-automating test design, there is also a cautious approach towards trusting its outcomes. Some organizations have paused their experimentation to centralize control and ensure reliability.

I read this as saying “we tried to automate test case creation using AI and it didn’t work very well”!

I missed the announcement of the publication of the 2023-24 World Quality Report so I’m catching up here to maintain my record of reviewing this annual epic from Capgemini (I reviewed the 2018/19, 2020/21 and 2022/23 reports in previous blog posts).

I’ve taken a similar approach to reviewing this year’s effort, comparing and contrasting it with the previous reports where appropriate. This review is a lengthy post, but I’m still doing you a favour even if you read this in its entirety compared to the 96 pages of the actual report!

TL;DR

The 15th edition of the “World Quality Report” is the fourth I’ve reviewed in detail and this is the thickest one yet (in every sense?).

As expected, AI is front and centre this year which is somewhat ironic given AI & ML barely got a mention in last year’s report. The hype is real and some of the claims made by respondents around their use and expected benefits of AI strike me as being very optimistic indeed. Their faith in the accuracy of their training data is particularly concerning.

Realizing value from automation clearly remains a huge challenge and I found the data around automation in this year’s report more depressing than usual. I got the feeling that AI is seen as the silver bullet in solving automation woes and I think a lot of organizations are about to be very disappointed.

It would have been nice to see some questions around human testing to help us to understand what’s going in these large organizations but, alas, there was nothing in the report to enlighten us in this area. The prevalence of Testing Centres of Excellence (CoEs) is another hot topic, despite previous reports suggesting that such CoEs were becoming less common as the movement towards agility marched on.

There continue to be many problems with this report – from the sources of its data, to the presentation of findings, and through to the conclusions drawn from the data. The lack of responses from smaller organizations mean that the results remain heavily skewed to very large corporate environments, which perhaps goes some way to explaining why my lived reality working with organizations to improve their testing and quality practices is quite different to that described in this report.

“Quality Engineering” is not clearly delineated from testing, with the concepts often being used interchangeably – this is confusing at best and potentially misleading.

The focus areas of the report changed almost completely, from “six pillars of QE” last year to eight (largely different) areas this year, making comparisons from one report to the next difficult or impossible. The sample set of organizations is the same as last year in the interests of year-on-year comparison, so why change the focus areas making the value of any such comparisons highly questionable? Is this a deliberate ploy or just poor study design?

Unlike the content of these reports, my advice remains steadfast – don’t believe the hype, do your own critical thinking and don’t take the conclusions from such surveys and reports at face value. While I think it’s worth keeping an interested eye on trends in our industry, don’t get too attached to them – the important ones will surface and then you can consider them more deeply. Instead, focus on building excellent foundations in the craft of testing that will serve you well no matter what the technology du jour happens to be.

The survey (pages 87-91)

This year’s report runs to 96 pages, the longest since I’ve been reviewing them (up from a mere 80 pages last year). I again looked at the “About the study” section of the report first as it’s important to get a picture of where the data came from to build the report and support its recommendations and conclusions.

The survey size was again 1750, the same number as for the 2022/23 report.

The organizations taking part were again all of over 1000 employees, with the largest number (35% of responses) coming from organizations of over 10,000 employees. The response breakdown by organizational size was the same as that of the previous three reports, with the same organizations contributing every time. While this makes cross-report comparisons perhaps more valid, the lack of input from smaller organizations unfortunately continues and inevitably means that the report is heavily biased & unrepresentative of the testing industry as a whole.

While responses came from 32 countries (as per the 2022/23 report), they were heavily skewed to North America and Western Europe, with the US alone contributing 16% and then France with 9%. Industry sector spread was similar to past reports, with “Hi-Tech” (19%), “Financial Services” (15%) and “Public Sector/Government” (11%) topping the list.

The types of people who provided survey responses this year was also very similar to previous reports, with CIOs at the top (24% again), followed by QA Testing Managers and IT Directors. These three roles comprised over half (59%) of all responses.

Introduction (pages 4-5)

The introduction is the usual jargon-laden opening to the report, saying little of any value. But, there’s no surprise when it comes to the focus of this year’s epic:

…the emergence of a true game changer in the field of software and quality engineering: Generative AI adoption to augment our engineering skills, accelerated like never before. The lack of focus on quality seen in the last few years is becoming more visible now and has brought back the emphasis on the Hybrid Testing Center of Excellence (TCoE) model, indicating somewhat of a reversal trend.

Do the survey’s findings reflect the game-changing nature of generative AI around quality engineering? What’s with the “lack of focus on quality seen in the last few years” when previous reports have been glowing about QE and its importance in the last couple of years? And what exactly is a “Hybrid Testing Center of Excellence”? Let’s delve in to find out.

Executive Summary (pages 6-7)

While the Executive Summary is – as you’d expect – a fairly high level summary of the report’s findings, a couple of points are worth highlighting from it. Firstly:

…almost all organizations have transitioned from conventional testing to agile quality management. Evidently, they understand the necessity of adapting to the fast-paced digital world. An agile quality culture is permeating organizations, albeit often at an individual level rather than at a holistic program level. Many organizations are
adopting a hybrid mode of Agile. In fact, 70% of organizations still see value in having a traditional Testing Center of Excellence (TCoE), indicating somewhat of a reversal trend.

I’m intrigued by what the authors mean by “conventional testing” and “agile quality management”, as well as the fact that the majority of organizations still adopt a “traditional” TCoE. Secondly:

What is clear is the extended knowledge and skills that are required from the QE experts who operate in agile teams. Coding skills in particular (C#, Java, SQL, Python), and business-driven development (BDD) and test-driven development (TDD) competencies, are in demand.

The idea that “QE experts” need coding skills and competency in BDD and TDD strikes me as unrealistic. I’m not sure whether the authors are referring to expert testers with some development skills, expert developers with some testing skills or some superhuman combination of tester, developer and BA (remembering that, of course, BDD is neither a development or testing skill, per se).

With all the “game-changing” talk around AI, there’s a nod to reality:

A significant percentage (31%) remains skeptical about the value of AI in QA, emphasizing the importance of an incremental approach.

Key recommendations (pages 8-9)

The “six pillars of QE” from the last report (viz. “Agile quality orchestration”, “Quality automation”, “Quality infrastructure testing and provisioning”, “Test data provisioning and data validation”, “The right quality indicators” and “Increasing skill levels”) no longer warrant a mention, with this year’s recommendations being in these eight areas instead:

• Business assurance
• Agile quality management
• QE lifecycle automation
• AI (the future of QE)
• Quality ecosystem
• Digital core reliability
• Intelligent product testing
• Quality & sustainability

The recommendations are generally of the cookie cutter variety and could have been made regardless of the survey results in many cases. A couple of them stood out, though. Firstly, under “Digital core reliability”:

Use newer approaches like test isolation, contract testing etc., to drive more segmentation and higher automated test execution.

The idea that test isolation and contract testing are leading edge approaches in the automation space is indicative of how far behind many large organizations must be. Secondly, under “Intelligent product testing”:

Invest in AI solutions for test prioritization and test case selection to drive maximum value from intelligent testing.

I was wondering what “intelligent product testing” was referring to, so maybe the authors are suggesting a delegation of the intelligence aspect of testing to AI? I’m aware of a number of tools that claim to prioritize tests and make selections from a library of such test cases. But I’m also aware that good test prioritization relies on a lot of contextual inputs, so I’m dubious about any AI’s ability to do this “to drive maximum value” from (intelligent) testing.”

Current trends in Quality Engineering & Testing (p11-59)

Exactly half of the report is focused on current trends, broken down into the eight areas detailed in the previous section. Some of the most revealing content is to be found in this part of the report. I’ve broken down my analysis into the same sections as the report. Strap yourself in.

Business assurance

This area was not part of the previous year’s report. There’s something of a definition of what “business assurance” is, even if I don’t feel much wiser based on it:

Business assurance encompasses a systematic approach to determining business risks and focusing on what really matters the most. It consists of a comprehensive plan, methodologies, and approaches to ensure that business operation processes and outcomes are aligned with business standards and objectives.

One point of focus in this section is value stream mapping (which featured in the previous report) but the authors’ conclusions about its widespread adoption appear at odds with the data:

Many organizations are running pilots apparently, but their statement that “Many businesses are now shifting from mere output to a results-driven mindset with value stream mapping (VSM)” seems to go too far.

Moving on to how QE is involved in “Business Assurance testing” (which is not defined and I’m unclear as to what it actually is):

This data was from a small sample (314 of the 1750 respondents) and their conclusion that “both business stakeholders and testers are working together during the UAT process to drive value” (from the second bar) is hardly a revelation.

I remain unclear why this area is included in this report as it doesn’t seem related to quality or testing in a strong way. The first recommendation in this section of the report is “Leverage the testing function to deliver business outcomes” – isn’t this what good testing always helps us to do?

Agile quality management

This area wasn’t in the previous year’s report (the closest subject matter was under “Quality Orchestration in Agile Enterprises”) and the authors remind us that “last year, we noticed a paradigm shift in AQM [Agile Quality Management] through the emphasis placed on embracing agile principles rather than merely implementing agile methodologies”. The opening paragraph of this section sets the scene, sigh (emphasis is mine):

The concept of agile organizations has been a part of boardroom conversation over the past decade. Businesses continue to pursue the goal of remaining relevant in volatile, uncertain, complex, and ambivalent environments. Over time, this concept has evolved into much more than a mere development methodology. It has become a way of thinking and a mindset that emphasizes continuous improvement, adaptability, and customer-centricity. This evolution is clearly represented in the trends shaping agile quality management (AQM) practices today.

I’m going to assume “ambivalent” is a typo (or ChatGPT error). Much more worrisome to me is the idea that agile has evolved from being a development methodology to “a way of thinking and a mindset that emphasizes continuous improvement, adaptability, and customer-centricity” – this is exactly the opposite of what I’ve seen! The great promise shown by the early agilists has been hijacked, certified, confused, process-ized, packaged and watered down, moving away from being a way of thinking to a commodity process/methodology. Either the authors’ knowledge of the history of the agile movement is lacking or they’re confused (or both). Having said that, if they genuinely have evidence that there’s a move away from “agile as methodology” (aka “agile by numbers”) to “agile as mindset”, then I think that would be a positive thing – but I failed to spot any such evidence from the data they share.

The first data looks at how QE is organized to support agile projects, with a whopping 70% still relying on a “Traditional Testing Center of Excellence” and essentially all the rest using “product aligned QE” (which I think basically means having QE folks embedded with the agile teams).

Turning to skills:

Worryingly, there is no mention here of actual testing skills, but the authors remark:

Organizations are now prioritizing development skills over traditional testing skills as the most critical skills for quality engineers. Development-focused skills like C#/Java/SQL/Python and CI/CD are all ranked in the top 5, while traditional testing skills like automation and performance tooling ranked at the bottom of the results.

Looking at this skills data, they also say:

We feel this is in alignment with the industry’s continued focus on speed to market; as quality engineers continue to adopt more of a developer mindset, their ability to introduce automation earlier in the lifecycle increases as is their ability to troubleshoot and even remediate defects on a limited basis.

I find the lack of any commentary around human testing skills (outside of coding, automation, DevOps, etc.) deeply concerning. These skills are not made redundant by agile teams/”methodologies” and the lack of humans experiencing software before it’s released is not a trend that’s improving quality, in my opinion.

Turning to the challenges faced by quality engineers in Agile projects:

This is a truly awful presentation of the data, the lack of contrast in the colours used on the various bars makes it almost unreadable. The authors highlight one data point, the 77% of UK organizations for which “lack of knowledge of Agile techniques” was the most common challenge, saying:

This could indicate the speed at which these organizations moved into a product-aligned QE model while still trying to utilize their traditional testers.

They slip in and out of the use of terminology such as “Agile techniques” throughout the report (which is unhelpful) and they appear to be claiming that “traditional testers” moving into an Agile “product-aligned QE model” have a lack of knowledge – maybe they’re referring to good human testers being asked to become (QE) developers, in which case this lack of knowledge is to be expected.

The following text directly contradicts the previous data on the very high prevalence of “traditional” testing CoEs:

Many quality engineering organizations predicted the evolution and took a more proactive stance by shifting from a traditional ‘Testing Center of Excellence’ approach to a product-based quality engineering setup; however, most organizations are still in the process of truly integrating their quality engineering teams into an agile-centric model. As shown in the diagram… only 4% of respondents report that more than 50% of their quality engineering teams are operating in a more agile-centric pod-based model.

In closing out this section, the authors say (emphasis is mine):

…quality engineering teams are asked to take on a more development-focused persona and build utilities for the rest of the organization to leverage rather than focusing solely on safeguarding quality. The evolution of agile practices, the integration of AI and ML, and the synergy between DevOps and agile are transforming quality engineering in infinitely futuristic ways. The question is, when will organizations embrace these changes en-masse, adopt proactive strategies, and make them a norm? Next year’s report will probably unveil the answer.

I find it highly unlikely that the answer to this question will be revealed in the next report, as such questions have rarely – if ever – been answered in the past. Given how difficult it’s been for large organizations to move towards agile ways of working (despite 20+ years of trying), I’d suggest that en masse movement towards QE is unlikely to eventuate before the next big thing distracts these trend followers from realizing whatever benefits the QE model was designed to bring.

QE lifecycle automation

This section of the report is on all things automation, now curiously being referred to as “QE lifecycle automation”. It kicks off with the benefits the respondents are looking for from automation:

I’ll make no comment on these expectations apart from mentioning that my own experience of implementing many automation initiatives hasn’t generally seen “reduced testing effort” or “improved test efficiency” (though I’m not sure what they mean by that here).

Moving into where organizations intend to focus their automation efforts in the coming year:

The authors say:

We were expecting AI to steal the limelight this year since it is the buzzword in the boardroom these days. We think that AI is the new tide that needs to be ridden with caution, which means quality engineering and testing (QE&T) teams need to understand how AI-based tools work for them, how they can do their jobs better, and bring better outcomes for their customers.

More than 50% of the respondents were eager to see testing AI with automation, which is well ahead of the other top focus areas like security and mobile.

Interestingly, we found that functional (18% of respondents) and requirements (20% of respondents) automation were of the least focus for organizations, presumably because of the challenges in automating and regular updating involved in these areas. Perhaps, this is an area where we can expect to see AI tools becoming a key part of automation toolkits.

Surprise surprise, it’s all about AI! It’s time to ride the tide of AI, folks. What does “Testing AI” mean? Is it testing systems that have some aspect of AI within them or using AI to assist with testing? Whatever it is, it’s the top priority apparently.

I also don’t understand what “Functional” means and how other automation focus areas are related to it or not. For example, if I implement a UI-driven test on a mobile device using automation, does that come under “Functional” or “Mobile” (or both). It’s hard for me to fathom how respondents answer such questions without understanding these distinctions.

The last part of the quote above is illustrative of the state of our industry – building robust and maintainable automation is just too hard, so we’ll deprioritize that and get distracted by this shiny AI thing instead.

The data around integration of automated tests into pipelines makes for shocking reading:

The authors note:

Overall, the survey revealed that only 3% of respondents’ organizations have more than half of their suites integrated which may be due to diminishing returns or lack of QE access to orchestration instances. This comes as a surprise since most of the organizations now have some automated test suites integrated into their pipelines for smoke and regression testing.

I agree that this low percentage of organizations actually getting their automated test suites into pipelines is surprising! Maybe they shouldn’t be so worried about trying to make use of AI, but rather to actually make use of what they already have in a more meaningful and efficient way. The response to the next question “What are the most impactful challenges preventing your Quality Engineering organization from integrating into the DevOps/DevSecOps pipeline?” revealed that a whopping 58% (the top answer) said “Quality Engineering team doesn’t 58% have access to enterprise CI instance”, I rest my case. With no sense of irony about the report’s findings more generally, the authors say:

What was worth noting was that the more senior respondents (those furthest from the actual automation) reported higher levels of integration than those solely responsible for the work

The much vaunted dominance of low/no code & AI automation solutions is blown away by this data, with not much going on outside of pilots:

The data in this section of the report, as per similar sections in previous reports, only goes to show how difficult it is for large organizations to successfully realize benefits from automation. Maybe “AI” will solve this problem, but I very much doubt it as the problems are generally not around the technology/tools/toys.

AI (the future of QE)

With AI being such a strong focus of this year’s report, this section is where we get into the more detailed data. Firstly, it’s no surprise that organizations see AI as the next big thing in achieving “higher productivity”:

Looking at this data, it’s all about speed with any consideration around improving quality coming well down the list (“lesser defects” – which should be “fewer defects” of course – coming last on this list). Expressing their surprise at this lack of focus on using AI to reduce defects, the authors say (emphasis is mine):

With Agile and DevOps practices being adopted across organizations, there is more continuous testing with multiple philosophies like “fail fast” and “perpetual beta” increasing the tolerance for defects being found, as long as they can be fixed quickly and efficiently.

I find the response around training data in this question quite disturbing:

The authors are quite bullish on this, saying:

…the trust in training data for AI solutions is very high, which in turn reflects the robust infrastructure and processes that organizations have developed over the years to collect continuous telemetry from all parts of the quality engineering process.

I’m less than convinced this is the reason for such confidence and it feels to me like organizations want to feel confident but have little evidence to back up that confidence. I really hope I’m wrong about that, since AI solutions are clearly being seen as useful ways to drive significant decisions, in particular around testing.

The next question is where the reality of AI usage is revealed:

The use cases seem very poorly thought out, though. For example, does performance testing belong in “Performance testing or Test prioritization” or “Performance testing/engineering”? And why would performance testing be lumped together with test prioritization when, to me at least, they’re such different use cases.

I’ll wrap up my observations on this AI section with this data:

I’m suspicious that all of these statements get almost the same level of agreement.

Quality ecosystem

This section (which is similar in content to what was known as “Quality infrastructure testing and provisioning” in the previous report) kicks off by looking at “cloud testing”:

…82% of the survey respondents highlighted cloud testing as mandatory for applications on cloud. This highlights a positive and decisive shift in the testing strategy that organizations are taking on cloud and infrastructure testing. It also demonstrates how important it is to test cloud-related features for functional and non-functional aspects of applications. This change in thinking is a result of organizations realizing that movement to cloud alone does not make the system available and reliable.

The authors consider this data to be very positive, but my initial thought was why this number wouldn’t be 100%?! If your app is in/on the cloud, where else would you test it?

The rest of this section was more cloud stuff, SRE, chaos engineering, etc. and I didn’t find anything noteworthy here.

Digital core reliability

In this section of the report (which had no obvious corresponding section in last year’s report), the focus is on foundational digital (“core”) technology and how QE manages it. In terms of “ensuring quality”:

About half of the respondents say that a dedicated QA team is responsible for “ensuring the quality” but there’s also a high percentage still using CoEs or having testers embedded into product/feature teams according to this data.

Turning to automation specifically for digital core testing:

The authors go on to discuss an “enigma” around the low level of automation revealed by this data (emphasis is mine):

This clearly is due to the same top challenges around testing digital core solutions – the complexity of the environment owing to the mix of tools, dependencies related to environment and data availability. When it is hard to test, it is even harder to automate.

There’s also another contradiction we need to address – while 31% of organizations feel the pressure to keep up with the pace of development teams developing digital core solutions, 69% of organizations do not feel the pressure. With <40% of automation coverage and digital core solutions becoming more SaaS and less customized, it is a bit of an enigma to unravel. Why don’t organizations feel the pressure to keep up? Is that because they have large QA teams rushing to complete all testing manually? Or are teams stressed by the number and frequency of code drops coming in for testing?

It’s good to see the authors acknowledging that the data is contradictory in this area. There are other possible causes of this “enigma”. Maybe the respondents didn’t answer honestly (or, more generously, misunderstood what they were being asked in some of the survey questions). Maybe they don’t feel the pressure to keep up because they’ve lowered their bar in terms of what they test (per the previous commentary on the “increasing tolerance for defects”).

The following is one of the more sensible suggestions in the entire report:

When it comes to test automation, replicating what functional testers do manually step by step through a tool might not be the best possible approach. Trying to break the end-to-end tests into more manageable pieces with a level of contract testing could be a more sustainable approach for building automated tests.

Wrapping up this section, the focus turns to skills required by testers when testing “digital core”:

The top answer was “Quality Assurance”, is that a testing skillset? It seems awfully broad in answer to the question, in any case. The report’s tendency for confirmation bias rears its ugly head again, with the authors making their own judgements that are unsupported by the data (emphasis is mine):

When we looked at the data, quality assurance skills were rated as the most important skillset over domain or platform skills required for testing digital core solutions. This datapoint feels like bit of an outlier when compared to two other datapoints – 35% of organizations still utilize business users in validating digital core solutions and 33% of organizations pointed to gaps in domain expertise as a challenge to overcome when testing digital core solutions. The truth is while solid testing and quality assurance skills are sought after due to the nature of digital core solutions, domain expertise remains invaluable.

Intelligent product testing

This section (again not obviously close to the content of a section from last year’s report) looks to answer this bizarre question:

What really goes into creating the perfect intelligent product testing system ecosystem?

The first data in this section refers to different test activities and how important they are considered to be:

The presentation of this data doesn’t make sense to me, since the respondents were asked to rate the importance of various activities while the results are expressed as percentages. Most of the test activities get very similar ratings anyway, with the authors noting that 36% seems low for “E2E Testing”:

..we were surprised to learn that only 36% thought end-to-end (E2E) testing was necessary. It has been established that the value of connected products is distributed throughout the entire chain, and its resilience depends on the weakest link in the chain, and yet, the overall value of E2E testing has not been fully recognized. One reason we suspect could be that it requires significant investment in test benches, hardware, and cloud infrastructure for E2E testing.

Note how the analysis of the data changes again here – they refer to the necessity of a test activity (E2E testing in this case), not a rating of its importance. I had to chuckle at their conclusion here, which essentially says “E2E testing is too hard so we don’t do it”.

I see no compelling evidence in the report’s data to support the opening claim in the following:

One of the significant trends observed in the WQR 2023 survey is the expectation for improving the test definition. The increasing complexity of products, along with hyper-personalization to create a unique user experience, necessitates passing millions of test cases to achieve the perfect user experience. Realistically speaking, it’s not possible to determine all possible combinations.

The next question has the same rating vs. percentage data issue as the one discussed above, but somewhat supports their claim I suppose:

I again don’t see the evidence to support the following statement (nor do I agree with it, based on my experience):

For product testers, the latest findings suggest that automation toolchain skills and programming languages are now considered mainstream and no longer considered core competencies.

Quality & sustainability

It’s good to see some focus on sustainability in the IT industry, especially when “reports show that data centers and cloud contribute more to greenhouse emissions than the aviation sector”.

This section of the report looks specifically at the role that “quality engineering play in reducing the environmental impact of IT programs and systems”, which seems a little odd to me. I don’t really understand how anyone could answer the following question (and what it even means to do so):

Although the data in this next chart doesn’t excite me, I had to include it here as it takes the gong (amongst a lot of competition) for the worst way of presenting data in the report:

The final chart I’ll share from this section of the report again suffers from a data issue, in that the question it relates to (which itself seems poorly posed and too broad) is one of frequency while the data is shown as a percentage (of what?):

Perhaps I’m reading this wrong, but it seems that almost none of the organizations surveyed are really doing very much to test for sustainability (which is exactly what I’d expect).

Sector analysis (p60-85)

As usual, I didn’t find the sector analysis to be as interesting as the trends section. The authors identify the same eight sectors as the previous year’s report, viz.

• Automotive
• Consumer products, retail and distribution
• Energy, utilities, natural resources and chemicals
• Financial services
• Healthcare and life sciences
• Manufacturing
• Public sector
• Technology, media and telecoms

Last year’s report provided the same four metrics for each sector but this year a different selection of metrics was presented by way of summary for each sector. A selection of some of the more surprising metrics follows.

• In the Consumer sector, 50% of organizations say only 1%-25% of their organizations have adopted Quality Engineering.
• In the Consumer sector, 76% of organizations are using traditional testing centres of excellence to support agile projects
• In the Financial Services sector, 75% of respondents said they are using or plan to use advanced automation solutions like No Code/ Low Code or AI based automation frameworks.
• In the Financial Services sector, 82% of organizations support agile projects through a traditional Testing Center of Excellence.
• In Healthcare, 81% of organizations have confidence in the accuracy of data used to train AI platforms.
• In the Public sector, 60% of organizations said the top-most issue in agile adoption was lack of time to test.

Geography-specific reports

The main World Quality Report was supplemented by a number of short reports for specific locales. I only reviewed the Australia/New Zealand one and didn’t find much of interest there, with the focus of course being on AI, leading the authors to suggest:

We think that prompt engineering skills are the need of the hour from a technical perspective.