Introduction to AI in test case generation
Test cases are a set of instructions and conditions used to validate that software functions as intended. Test cases are an essential component of software development, ensuring that the software not only meets quality standards but also functions as intended.
AI-powered test case generator tools present themselves as a weapon that can help development teams slay the twin-dragons of feature production and code-maintenance, in an ever-changing technological landscape. But are these tools a perfectly crafted katana that is ready to slice through any scenario or just a blunt hammer that slams bangs against everything without any specificity?
In the following article, we will explore both sides of the technology and whether or not your organization is in a position to embrace it.
“We are about to turn over most test design and validation to Artificial Intelligence (AI). Instead of humans manually creating all the test automation, machines will write and execute test code, and continually improve as they learn from human input.” - Jason Arbon, Test.ai
The foreboding quotation above comes from a Medium article written by Jason Arbon in March 2016. Arbon, the author of “How Google Tests Software”, uses this article as a launchpad to make an initial attack on the traditional way of writing tests. Here he focuses on the unbelievable capabilities of AI and the future that he had envisioned for it. Long before people had Googled the letters, ‘GPT’.
It has not yet been two years since OpenAI shared the idea of generative AI with the world, and yet we see this tool (and its many rivals) being integrated into nearly every aspect of our daily lives. Their product, ChatGPT, caused a massive disruption in the market and had many CEOs and board members all say the same thing, “I’ve just heard about AI. How can we use AI?”.
To answer this question, let’s look at some of the advantages of using AI in test case generation.
Advantages of AI-driven test case generation
- Reduction in manual effort and time
- AI is often touted as the evolution of test automation, with more and more processes being automated with AI. Many companies are seeing a reduction in time-to-production with the use of AI tooling.
- Increased test coverage and depth
- As discussed multiple times in this document, (Tricentis) Tosca enabled Tesla to increase the testing coverage and depth by successfully generating comprehensive test scenarios that covered various edge cases and potential user interactions - something that is critical for autonomous driving features.
- Adaptability to changing requirements and environments
- The adaptability to ensure that a company’s testing process remains effective and relevant, even as its platform rapidly evolves to meet market demands, is critical in today's era. Tools like Applitools AI-driven visual testing allow teams to dynamically adjust test cases based on changes in today's dynamic world.
- Improved consistency and reliability in test cases
- AI tools can generate standardized test cases that reduce human error and variability, leading to more reliable test results.
- Ability to uncover hidden defects through sophisticated algorithms
- The absence-of-errors fallacy states that the lack of errors in software does not guarantee the absence of defects. To identify hidden defects in their platform, LinkedIn used Test.ai to leverage the tool’s sophisticated algorithms to identify subtle bugs and performance issues that traditional testing methods had missed.
With these advantages in mind, it's clear that AI-driven test case generation can significantly enhance the software testing process. But what tools can organizations use to secure these benefits?
Popular AI-driven test case generation tools
While numerous AI-driven testing tools have emerged, here are a few of the most popular and widely used tools in the industry today:
- Test.ai (Est. 2015) - Test.ai creates AI-powered scripts, which can be used across all digital surfaces for testing, helping companies move from manual to automated testing.
- Testim (Est. 2015) - Testim's AI analyzes all the DOM changes to make sure your test is stable and reliable.
- Mabl (Est. 2017) - Mabl’s native Auto-Healing feature autonomously detects visual changes in your application and updates your tests to ensure stable end-to-end testing.
- Functionize (Est. 2015) - Functionize uses machine learning to automate web testing with a unique, adaptive approach that's driven by state-of-the-art algorithms.
- Applitools (Est. 2013) - Applitools uses Visual AI to automatically validate the entire application pages, enabling teams to test all app elements in one go.
- Tricentis Tosca (Est. 2007) - Tosca’s AI-driven, model-based approach allows you to achieve higher automation rates while maintaining resilient test automation suites.
- Sauce Labs (Est. 2008) - Sauce Labs provides the world’s largest cloud-based platform for the automated testing of web and mobile applications.
Each of these tools offers fairly unique features and capabilities, allowing organizations to choose the best fit for their specific needs and testing requirements.
After the initial excitement has died down and they have answered the question of “How”, those same CEOs and board members often identify the costs and their own skills shortage and then ask, “Should we use AI?”
Pitfalls of AI-driven test case generation
Knowing which tactics to avoid is key to progress on the battlefield, so here we look at some points to make a case for not choosing an AI-driven test case generation tool, without having a qualified QA team to support it:
- Quality of Generated Test Cases
- The value of tests is not in the quantity but rather in the quality. In 2021, Target saw some bugs slipping through to production, affecting the user experience and causing minor disruptions due to the implementation of Functionalize. This is because the AI tool generated test cases that were often simplistic or irrelevant.
- AI uses the most common scenarios and does not “think like a tester”
- While AI may generate test cases for logging in, checking balances, and transferring funds, for a banking application, it may overlook scenarios whereby a user might perform multiple simultaneous login attempts from different devices, potentially missing a security vulnerability.
- AI is not imaginative
- An AI-tester at Rockstar North may focus on features of gameplay, such as: Running, driving, and shooting. But it may miss a test case whereby a human tester has the character on a track and waits for a train to strike the character.
- Potential dependence on initial data quality and training
- An AI-driven test case generation tool may create test cases for facial recognition software based on data that only has caucasian characters. Thus, errors may occur when non-caucasian characters attempt to access the features.
- Lack of human-like creativity and empathy in understanding nuanced user scenarios
- An AI-driven test case generation tool, introduced for customer service chatbots, might focus on functional aspects like response times and accuracy. However, this tool may fail to create test cases that assess the chatbot’s ability to handle emotionally charged interactions or to understand and respond to user frustration or confusion, which human testers would naturally consider.
- Security and privacy risks
- Some AI models require access to sensitive customer data to generate realistic test scenarios for an application. If the security measures are weak then it becomes a target for hacking. Thus risking data breaches and compromising customer information. One recent example is from Rabbit. tech’s R1 (June 2024).
Knowledge is power. While understanding the risks and rewards is crucial, it's even more beneficial for companies to identify where AI-driven test case generation can be most impactful.
Who is AI-driven test case generation for?
Below is a list of scenarios where AI-driven test case generation could be beneficial, along with examples of successful implementations in these contexts:
- Non-QA team members: Ideal for coders and stakeholders, AI-driven tools can bridge the gap, helping teams shift left in their testing without waiting for the QA member to manually create the test cases.
- Shopify (2019) - Shopify implemented AI-driven test case generation to empower coders to execute tests on their code before handing the feature over to the QA team. This allowed the teams to speed up the delivery process by ensuring a higher-quality product was delivered to the QA team.
- Automation Engineer, with little time to create manual testing documentation: Long before the green field of perfectly managed sprint tasks and constant delivery velocity, automation engineers often find themselves pressed for time. This is where AI-driven test case generation can relieve some of the pressure by quickly generating comprehensive lists of test cases.some text.
- Netflix (2017) - Automation engineers at Netflix leverage the power of AI tools to generate test cases for repetitive and complex tasks. This ensures that the engineers can focus on scripting and not manual test case generation.
- QA teams looking to augment their testing capabilities: Even high-performing teams can benefit from this type of automation when trying to expand the scope of their testing, uncover hidden bugs, or simply ensure that they deliver a higher-quality product.some text.
- IBM (2021) - IBM Rational Test Workbench is the tool that the QA team uses to scale their capabilities by automatically creating a comprehensive suite of test cases. This allows the QA team at IBM to focus on exploratory testing and test automation.
- Organizations aiming for continuous integration and continuous delivery (CI/CD) in a rapidly changing environment: AI-driven test case generation can enhance CI/CD pipelines by accelerating speed to production and ensuring robust testing coverage for large or small teams. some text.
- Amazon (2018) - Amazon leveraged AI-driven test case generation within their CI/CD pipelines for AWS, resulting in an ability to swiftly validate code in pull requests, and ensure rapid development while maintaining overall code quality.
- Projects with high complexity and large-scale testing needs: From ABN Amro to Zalando, having systems that are rigorously tested to ensure compliance, security, and functionality across numerous applications and platforms is paramount to maintaining business continuity. AI-driven test case generation can create scenarios that will not only ensure the functionality of a single application but also consider user journeys and regulatory requirements.some text.
- HSBC Bank (2017) - Tricentis Tosca has a massive reach and is seen here, where HSBC Banks used Tricentis Tosca for AI-driven test case generation across their banking platforms. This tool automates the creation and execution of test cases, under the supervision of the dedicated QA team, enabling HSBC to manage large-scale testing needs effectively.
Conclusion
The battle to be ahead of the competition is won by having the newest features and services, as well as ensuring that all of those are verified and validated. For the time being, a platoon of human QA team at the helm still proves to be the most effective way to ensure that quality products are delivered to customers. As the battlefield changes, their implements need to change with that. The metaphorical choice between wielding a finely crafted sword or a blunt hammer underscores the importance of strategic adoption and integration of AI in testing practices. Choosing the right tool for the organization and the scenario.
Still unsure about whether or not AI testing tools are the right fit for you?
Let spriteCloud light the way! With a wealth of experience in quality assurance and testing, our expert team can guide you through the process, tailor solutions to your specific needs, and ensure your projects achieve the highest standards of quality.
Begin your journey towards enhanced software testing processes.
Visit our Contact page or send an email to projects@spritecloud.com to learn more about our services.