Launchable by CloudBees—our new AI-augmented QA capability—can be an effective tool to help wring out the bottlenecks caused by traditional testing efforts.
DevSecOps teams face a constant tug-of-war. They’re under pressure to release features at high velocity. However, conducting thorough quality assurance (QA) takes time.
Many organizations are opting for AI-augmented quality assurance tools to help diminish some of this whiplash. Consider all of the work involved, from code commit to the time bits land in production or a ticket lands in the ticketing system. Most of this work is manual. Utilizing AI-augmented QA can eliminate some of the effort associated with manual testing, allowing developers to ship releases faster—without compromising quality.
The adoption of AI-powered tools is just beginning in the dev world. Coding tools such as GitHub Co-Pilot have already been widely adopted. However, according to ICONIQ’s 2024 State of AI Report, a significant opportunity still exists for specialized tools that can uplevel engineering work across DevSecOps, the quality assurance process, and more.
The Co-heads of AI at CloudBees, Harpreet Singh (VP, Product) and Kohsuke Kawaguchi (VP, Engineering and Jenkins co-creator), discussed this topic during their webinar session at DevOps World, “Using AI-augmented QA to improve delivery velocity.” As co-founders of Launchable by CloudBees, the pair shared their expertise in using AI to help improve the dev-test iteration loop.
Let’s dive deeper into the benefits of using AI-augmented testing tools in QA efforts, and discuss a few common scenarios where AI can reduce headaches, improve software quality, and accelerate releases.
How Is AI Used In QA?
AI can be used to improve the efficiency of test execution, failure triaging, and overall test management in the software development QA process. In short, AI tools can help turn a firehose of test failures into a more digestible drip.
As Harpreet put it, “Tests are a lot like taxes. You absolutely have to do them. Everybody decides that they're going to run their entire test suites. You've been told over a period of time to write more and more tests, more often. All that is good. But what that does is, it slows you down!”
Traditionally, developers would run every test, all the time, because there wasn’t any technology to support a different approach. Finding the right test to run for changes coming in was like finding a needle in a haystack. AI happens to be particularly suited for finding that needle.
More tests don’t necessarily equate to better outcomes. The real challenge comes in identifying the tests that matter most. AI-augmented QA offers smarter, more efficient ways to run tests and manage failures. The end result is dev teams finding calm amidst the chaos and shipping with confidence.
What Are AI-Augmented Software Testing Tools?
AI-augmented software tools are technologies that leverage artificial intelligence and machine learning to optimize and enhance traditional testing processes. By utilizing AI-augmented software tools to intelligently select and prioritize tests, DevSecOps teams can reduce manual triaging, keep test suits healthy, and save time for more strategic tasks. These tools also bring the potential to continuously optimize and automate testing over time.
Launchable by CloudBees, for example, uses AI to help remove friction from the dev-test loop by enabling DevSecOps teams to shorten test runs, improve test triaging, and access comprehensive test health insights and test failure analysis.
What Are the Benefits of AI-Augmented Testing?
DevSecOps teams can reap many benefits from implementing AI-augmented QA in their testing process, including:
Faster Feedback Loops: AI-augmented QA helps DevOps teams run only the most relevant tests. This gives developers quicker insights into whether their changes are working as expected, making it especially important for teams that follow continuous integration and continuous delivery (CI/CD) practices.
Increased Productivity: When AI tools can help automate test selection and failure triage, developers can spend less time managing tests and more time writing code. Not only will developers be more productive; they’ll usually be happier because they’re spending time on more high-priority work.
Improved Test Reliability: By identifying and addressing flaky tests, AI software testing enhances the reliability of the test suite. Developers can be more confident in the feedback they receive from their tests.
Quicker releases. Better time management. Greater reliability. That’s what makes AI software testing so effective.
4 Examples of the Power of AI-Augmented QA
So, what does this look like in practice? Now that we’ve discussed the benefits of using AI-powered testing tools to augment your QA efforts, let’s break down a few scenarios where it comes in handy:
1. Triaging test failures and faster root cause analysis using GenAI
Let’s imagine you’re a developer working on a code change. It’s likely to be the first time you’ve interacted with the test results.
Perhaps your test data shows 20 failures. Many developers will simply look at the first few failures they see, make code changes based on those, and resubmit the test. If they don't guess correctly, they’ll be forced to run the test again (and again) until they triage the failure.
Now add Predictive Test Selection to the mix. An AI-augmented testing tool could elect the most relevant tests to run based on code changes, reducing the need to run the entire test suite every time a change is made. It would leverage data to predict which tests are most likely to fail, reducing the noise within your test suite.
2. Conducting a QA “morning routine”
In every organization, someone has to be responsible (officially or unofficially) for the overall quality of the main branch. Sometimes, this is a senior developer or a dedicated QA team.
This individual often has a “morning routine”; the QA equivalent of brushing one’s teeth. They’ll show up in the morning, scan everything that occurred in the main branch, identify the changes, and decide which issues need to be addressed. The morning routine is about finding the key issues that underlie the failures that occurred overnight and prioritizing them.
Let’s introduce AI into the equation. An AI tool could conduct this scan on its own, and deliver an email to the team that summarizes what happened in the last 24 hours. This is especially useful for teams working on large projects with multiple contributors, as it ensures that everyone starts the day with the same set of information.
Instead of spending the first part of their day in the main branch, the senior developer or QA analyst could get started on more strategic work.
3. Keeping test suites healthy
There’s also the longer-term task of maintaining the overall health of a test suite. Test suites can become "noisy" over time as more tests are added, often leading to unreliable or "flaky" tests. Flaky tests can be frustrating because they provide inconsistent feedback, making it harder to trust the test results.
“Feedback time is king,” Kohsuke said. “Most people know how long the whole test suite is taking, but a surprising number of people don’t know the top three longest-running tests within that test suite. Just cutting a few tests out of the stream, you can substantially reduce the amount of feedback time.”
Bringing AI tools into the testing process allows developers to quantify which tests are more flaky than others, replacing tribal knowledge with data-driven insights.
4. Using machine learning to reduce test execution time
What if you could run your failing tests first? Tests run in random order are not optimized for low time to first failure.
"Imagine AI could make some reasonable guess at which tests are more likely to fail,” Kohsuke added. “Even if crudely meaningful, then what we can do is, for example, run these more sensitive tests or more-likely-to-fail tests first, and that'll get your feedback more quickly.”
By running tests that are likely to fail first, DevOps teams can reduce the time to first failure by a significant margin. They can use AI to reduce Test Feedback Delay (TFD) even more by running a subset on a more frequent interval
Launchable by CloudBees: The AI-Augmented QA Approach
AI-augmented QA is reshaping how organizations approach testing and quality assurance.
By leveraging AI and machine learning, tools such as Launchable by CloudBees can help DevOps teams alleviate bottlenecks, maintain high-quality software, and expedite releases.
Launchable by CloudBees offers an AI co-pilot for test suite optimization. With predictive test solutions, intelligent test failure diagnosis, and thorough test suite insights, Launchable helps DevOps teams achieve shorter test runs, smarter test triaging, and comprehensive failure analysis.
“It’s a Swiss Army knife approach that allows you to look at your test suites in a variety of different ways,” Harpreet said. “It’s dynamic and tailored to the changes that are coming in.”
Book a demo to see AI-augmented QA in action.