Industry: Information Technology & Services
Geography: Latin America
Summary: After many years of manual continuous integration, TODO1 successfully implemented a CloudBees solution to automate processes and establish a robust DevOps culture.
Challenge: Automate manual processes to reduce errors and speed deployments
Solution: Adopted CloudBees and implemented DevOps methodologies and governance to support the CI/CD implementations of its banking customers
Results:
Automated manual processes
Increased productivity
Eliminated manual errors
Created reusable components and pipelines
Established strong governance practices
Set the stage for business growth
Product: CloudBees CD
Founded in 2000 and based in Miami, TODO1 Services provides innovative digital solutions to banks and financial institutions in Latin America. With a software-as-a-service (SaaS) delivery model, the company offers online and mobile banking solutions to both consumer and commercial markets as well as multichannel collections, digital banking and security solutions. TODO1’s SaaS applications handle approximately nine billion transactions a year from more than 15 million users.
The company’s platform consists of configurable components. According to Juan Felipe Cardona, Director of TODO1’s Solution Architecture Team, "We configure the components of our platform differently depending on each client’s unique requirements.”
With a flexible operational and support model that TODO1 can adjust to meet customer needs, the development teams deploy a specific number of releases every month for each client based on contractual obligations. "For one of our big clients, we’re required to roll-out at least 10 deployments each month. Before CloudBees, our deployments were very manual…very painful,” said Cardona.
“With CloudBees, we can deploy releases 80% faster, and we can invest that time in other projects."
Juan Felipe Cardona
Director, Solution Architecture Team
TODO1
Challenge
TODO1 has been doing continuous integration since 2009. They used a combination of tools to support their efforts, including Apache Maven, Nexus Repository for artifacts, SonarQube for code quality, Bitbucket for continuous integration and Jira for QA. But by 2017, the mix of tools had run its course at TODO1. The development team had maxed out the number of artifacts it could deploy in Maven and the operations team was struggling to roll-out multiple releases each night for several different clients.
"The developers were doing all of this work manually,” Cardona explained. "They were tired from working all night and they were making mistakes. We needed to eliminate the human errors and make the deployment process more efficient.” At the same time, the QA team was manually certifying releases by following instructions on a Word document to locate installers and artifacts and then run detailed validations. "Once the QA team finished a certification, they used Jira to let the operations team know the release was ready for production,” Cardona said. "It was all very painful.”
Solution
It Started with a Webinar
When asked why TODO1 chose CloudBees, Cardona explained that he attended a webinar to learn about CloudBees’ CI/CD solution. "We saw the tool. We downloaded the community edition. We tried it and we realized that it was what we needed to solve our problems,” Cardona said. The development team tested some automated deployments and quickly realized they needed access to more technical notes than they could get with the community edition. After conducting a formal proof-of-concept, they also realized they needed two CloudBees instances for high availability. With that due diligence, the TODO1 development team purchased CloudBees CD licenses in 2018. "We didn’t need to look at other vendors. Our decision to go with CloudBees was very straightforward,” Cardona said.
Incremental Steps to Demonstrate Success
Although the development team thought CloudBees could provide the automation they needed, there wasn’t a formal DevOps team or processes to govern their efforts. So the development team started small with a single client deployment in their QA environment. According to Cardona, "We started with a test, and it was very successful. We deployed the release about 80% faster than with our manual deployment process.”
Once the development team showcased the successful QA test to stakeholders, they pushed the release into production. But of course, there were a few challenges. Since TODO1’s production network is physically separated from its QA and integration environments, the development team had to coordinate closely with the operations, networking, infrastructure and security teams to get access to the different environments. This led to the creation of the company’s DevOps governance framework. "We created internal processes so that only a few people from the operations team could actually deploy releases into production. No other humans could touch those environments. CloudBees made that segregation of duties much easier,” Cardona said.
The development team slowly gained buy-in from different stakeholder teams. As Cardona explained, “We went through a lot of reviews with our security team. And because of our banking clients, we’re ISO 27000, and we have yearly audits. We went through step-by-step checks with our compliance team. It wasn’t easy, but we were crossing small bridges every day until we got to the point that the compliance team was comfortable with our automated processes.”
A Formal DevOps Team
According to Cardona, "Because we increased the speed of deployments and showed good results, the operations team started asking us to help them automate other layers of infrastructure, like the user interface, middle network, database and integration middleware.” With a successful track record and a demand for more process automation, TODO1 created an official DevOps team in 2019. Their team of three helped put additional automation processes in place, and they expanded the DevOps efforts to encompass more client projects. "We’ve created standards, documentation and technical training for different tools. We’ve also produced pipeline templates for similar products and components that developers can reuse between projects,” Cardona said.
To ensure compliance and strong governance, the DevOps team created four pillars, each with specific items and processes:
Configuration management. This pillar details versioning standards for infrastructure as code, databases, the publication of artifacts and homogenous deployments between environments.
Continuous integrations. This outlines requirements for rapid compilations, minimum standards for daily integrations, technical debt and thresholds to break a pipeline when quality isn’t what it should be.
Continuous deployment. This is all about deployment processes for the development, certification and production environments to ensure clients aren’t impacted during deployments. It also explains policies for automatic roll-backs and database deployments.
Continuous testing. This describes all of the different tests—for units, acceptance, integrations, performance and security as well as a continuous hacking strategy.
The team’s release management policies encompass all four pillars as well as the different pipelines for development, certification and production to ensure the company achieves the minimum number of deployments per month per client according to each customer’s contract.
DevOps Put to the Test
When TODO1 landed a big new client not long after the DevOps team was established, the new automated processes and governance policies were put to the test. The client had a very strong internal DevOps team, and TODO1 was contractually obligated to follow certain principles and rules for deployments. "It was quite a challenge because of all of the requirements we had to fulfill, and many different teams in our company (networking, operations, security) were involved to ensure we met the contract parameters. Because of CloudBees, we were able to meet the requirements 100%. And, we keep working to improve all of our processes,” said Cardona.
Internal Adoption
Cardona readily admitted that the shift to DevOps wasn’t always easy, but the different IT teams are all on-board now—according to the feedback he hears about CloudBees. "The day-to-day processes have become really great for everyone. If we didn’t have CloudBees, our teams definitely would be negatively affected,” he said.
Even More Automation
When asked what’s next for TODO1’s DevOps team, Cardona said he wanted to automate development and deployment processes for every client and set the stage for the company to add more clients. The team also plans to create a library of reusable pipelines, tackle a backlog of testing projects and embark on a roadmap of product enhancements for clients. "We need to study our roadmap and get support from CloudBees to figure out how to deal with the projects,” Cardona said.
Results
Cultural improvements
"The transition to CloudBees was a bit of a challenge for our developers. Now, however, the day-to-day processes are really great for them. We hear very positive feedback,” said Cardona.
Ease of use
"The CloudBees tool hasn’t been a challenge for us, because it has a lot of plugins for integrating with almost every tool,” he said.
Increased efficiency
"In just nine months, we saw positive results. Our developers were able to do more deployments per night than with our previous manual processes,” he said.
Meeting client expectations
"With CloudBees, we can meet our client expectations—100%,” he said.
Positive feedback
"Our security, infrastructure, operations and QA teams have all been very positive about CloudBees,” he said.
Integration middleware
"The operations team saw how automation improved our deployment process. They started asking us to automate other components, like database, integration middleware and user interface updates,” he said.
Productivity boost
"With CloudBees, we can deploy releases 80% faster, and we can invest that time in other projects,” he said.
Regulatory compliance
"We had to assure our compliance team that with the automations, we were still meeting our regulatory requirements. We showed them the checks step-by-step. It wasn’t easy, but we were crossing small bridges every day until we got to the point that the compliance team was comfortable with our automated processes,” he said.
Troubleshooting
"We do have some complicated client cases, but with the help of CloudBees, it’s been easy to solve the challenges,” he said.
Why CloudBees
"There are some other providers on the market that could help us, but we think CloudBees has the most complete solution,” he said.