The State of Test Automation: Insights From The Field

In the past 6 weeks, I have engaged with hundreds of QA professionals at various industry events. These conversations have shed light on several common themes that are impacting the world of software testing. 

One of the aspects I genuinely cherish in my role as a VP of Product at Element34 is the opportunity to step out of the office and engage directly with QA professionals in face-to-face conversations. There's no substitute for these real-life dialogues when it comes to truly understanding what's happening in the field - they provide invaluable insights into the challenges, needs, and the dynamic landscape of software testing. 

Here are some of the key points that I’ve heard (and confirmed) from these conversations:

1. The Mystery Behind Test Execution Environments

One surprising revelation was that many QA professionals are unsure about where their tests are actually being executed. Most people said, they never really thought of it or minded to ask.  I think this lack of awareness can pose significant challenges as it hampers their ability to validate the reliability of their test results. If I don’t know where and how my tests are being executed, do I really understand the results I’m seeing? 

It's essential for QA teams to have full understanding and visibility (and ideally also some level of control) over their test execution environments to ensure they can validate the accuracy of their testing outcomes.

2. Selenium Still Dominates, By Far!

The majority of QA professionals I spoke to confirmed what the industry has been hinting at: Selenium continues to be the primary tool for end to end test automation for web applications. 

It’s always hard to find trustworthy sources of data for this stuff, so we decided to run a survey during the TestFlix conference and found that almost 59% of respondents use Selenium, which is miles apart from any other framework - 8% use Playwright, while Cypress and Appium came at 9.5% each. 14% of respondents selected “other” and I regret not having a follow up question asking “which other?”. I wouldn’t say these results are scientific since the sample was relatively small, but they do confirm my suspicions that Selenium is still top of mind, although Playwright seems to be picking up pace and might emerge as a formidable competitor in the short-term.

3. The Dilemma of Data Privacy in Testing

Many QA professionals confirmed to be using real or realistic data in their tests.   What's alarming is the lack of understanding or concern about the implications of using sensitive data in testing environments. This becomes particularly concerning when sensitive data is used in conjunction with SaaS test execution solutions, as it exposes organizations to the risk of data breaches and compliance issues. Many of these conversations happened outside of Europe, where data privacy laws may not be  that strict. Still, the lack of  awareness of the legal implications (along with the financial and reputational implications that might result) is concerning. Test data privacy and compliance need to be top priorities for QA teams, but there’s clearly a need for a lot more education of these topics in the industry to ensure that everyone at least knows what they’re doing and what they are risking.

4. The Ambiguity of AI in Test Automation

While the market is flooded with solutions touting AI capabilities for test automation, QA professionals remain uncertain about how to leverage AI in their processes. Questions about the actual impact, potential gaps, and the overall effectiveness of AI solutions persist. It's essential to navigate through the marketing hype and evaluate AI tools seriously, considering their practical applications in test automation. 
‍

In conclusion, our interactions with QA professionals have highlighted several crucial areas where improvements are needed. Test automation experts must strive to gain control over their test execution environments, understand the tools at their disposal, and ensure that their testing practices align with data privacy and compliance regulations. Moreover, they should approach AI in test automation with a critical eye, moving beyond the buzzwords to evaluate the genuine benefits AI can bring to their testing processes. By addressing these, QA professionals can elevate the quality and security of their testing efforts, ultimately delivering more reliable software products. 

At Element34, we are committed to helping QA professionals overcome many of these challenges with our unique, secure, and compliant test automation solution that runs behind your firewall. To learn more about how Element34 can empower your test automation journey, visit our website or get in touch with our team today.