Industry: Software
Geography: Global
Summary: A leading provider of DevOps test data management solutions was on a mission to accelerate its product release cycles while maintaining the highest quality standards.
Challenge: The company’s platform integrates with numerous third-party tools, including cloud services, databases, and enterprise applications. While this flexibility benefits customers, it also creates an extensive testing burden. Ensuring compatibility with every update required running hundreds of test cases, leading to test suite execution times that spanned several days.
Solution: The engineering team implemented CloudBees Smart Tests, predictive test selection, to reduce the time spent on running unnecessary tests, or finding critical problems too late in the development cycle and improve the efficiency of test runs overall.
Results:
Regression testing time cut by 80%
Cut pre-commit testing time by 66% from an average of 6 hours to 2 hours
Thousands of test execution hours saved over the course of a year
Developers enjoyed much shorter feedback loops and can commit changes to a release much sooner
Reduced cloud costs due to shorter test run times
Product: CloudBees Smart Tests
Test data management leader cut product release testing time and cloud costs with CloudBees AI-powered Smart Tests.
Background
A leading provider of DevOps test data management solutions was on a mission to accelerate its product release cycles while maintaining the highest quality standards. The company’s platform helps businesses securely and efficiently deploy test data for development, testing, and analytics environments. By automating data security and streamlining test data delivery, the company empowers developers to speed up application development without compromising compliance.
The engineering team, led by a Principal DevOps Engineer, was focused on eliminating roadblocks in the software development and release process. Their primary challenge was addressing inefficiencies in test suite execution, particularly around lengthy regression testing and slow feedback loops.
Objectives
To better serve customers, compete in the market, and keep its DevOps Data Platform as consistently up-to-date as possible, the company set the goal of increasing its product release frequency. Specifically, the development team wanted to:
Move to a monthly release cadence
Reconfigure their release process to enable faster go-to-market
Maintain equal or greater product quality and data integrity
However, a significant hurdle stood in the way: a bloated and time-consuming release testing process.
The Challenge
As a widely adopted solution, the company’s platform integrates with numerous third-party tools, including cloud services, databases, and enterprise applications. While this flexibility benefits customers, it also creates an extensive testing burden. Ensuring compatibility with every update required running hundreds of test cases, leading to test suite execution times that spanned several days.
The prolonged testing process had multiple drawbacks:
Slowed development cycles – Engineers had to wait for test results before proceeding with new code commits.
Escalating cloud costs – Running full test suites consumed significant computing resources.
Developer inefficiency – Engineers had to manually sift through test failures, often spending hours identifying whether an issue was new or recurring.
“Dealing with hundreds of test cases is a huge pain point for developers,” said the Principal DevOps Engineer. “Every week, they’re hit with a flood of issues and have to go through each one, asking, ‘Is this a new problem?’ or ‘Is this something we’ve seen before?’ Then comes the painstaking task of figuring out what’s actually happening.”
The team needed a way to streamline their release testing without sacrificing quality—and ideally, while reducing cloud spend.
Solution
To address this, the engineering team implemented CloudBees Smart Tests, predictive test selection and AI triage capabilities, to reduce the time spent on finding and triaging critical bugs and improve the efficiency of test runs.
The CloudBees Smart Tests capability intelligently identifies tests that are likely to fail based on code changes and other historical signals, resulting in test failure diagnostics.
For this DevOps team, CloudBees introduced an AI-driven test suite subset and binning recommendations based on business impact. This system takes in all the tests in a test suite, narrows them down to subsets, and allows the team to execute these subsets in parallel, much earlier in the development cycle. This approach allowed the team to focus on the most relevant tests early on, ensuring faster feedback loops and optimized resource usage.
Results
The unique capabilities that helped this DevOps team achieve these outcomes are distributed along the QA and testing phases:
| QA Phase | Before CloudBees Smart Tests | With CloudBees Smart Tests |
|---|---|---|
| Pre-Commit Testing |
|
|
| Nightly Regression Testing |
|
|
The integration of these tools has allowed the team to significantly reduce the duration of test cycles, making the process more manageable for developers. While the predictive test selection has already shown benefits, the principal engineer anticipates that the AI-driven test failure triage capabilities will have a more substantial impact on improving the speed and accuracy of identifying and resolving issues.
"It was a game-changer for developers to know that within a couple of hours, they’d have meaningful test results rather than waiting for huge, time-consuming suites to process… it is much more consumable for developers, providing confidence the changes were tested well.” - Principal DevOps Engineer
Looking at the data in greater detail, the new testing process can be summarized as follows:
As demonstrated in the confidence model, the team now achieves 90% confidence in detecting failures with significantly fewer tests. The new approach reduced pre-commit test time from 5-6 hours to just 60-120 minutes, enabling developers to iterate faster while maintaining software integrity.
Outcomes
The CloudBees AI-Powered Smart Tests solution has helped the company tackle its testing issues head-on. The vast improvements seen in testing time-savings mean that the company has been able to achieve its goal of accelerating delivery to monthly product releases while preserving its testing confidence and high-quality standards.
CloudBees Smart Tests Key Outcomes
| Streamline cross-functional collaboration around QA | Reduce cloud costs | Increase innovation capacity to stay competitive |
|---|---|---|
| Cloud regression testing times | 3-5 Cloud instances saved per test hour | x2 annual release velocity |
On top of all that, the company has found the cost of operating CloudBees Smart Tests to be considerably less than the cost of running three to five virtual machines in the cloud for the same amount of time. Consequently, using CloudBees Smart Tests to run only the most necessary tests has resulted in an annual reduction of +3 cloud infrastructure VM costs for the organization.
Next steps
Having seen strong results with the platform so far, the team is now looking to expand their use of CloudBees AI-powered Smart Tests to leverage the intelligent test failure diagnosticsIntelligent Test Failure Diagnostics capability. This solution will enable the DevOps team to:
Automate and accelerate root cause analysis, reducing the manual time spent triaging and analyzing known failures
Intelligently collate and summarize error data from multiple logs across several pieces of the test environment
Easily pinpoint new and recurring test issues, and stay on top of issues with features such as Dynamic Issue Updating