By Stefan Simenon, Head of IT Tooling and Software Development and Wiebe de Roos, CI/CD Consultant, ABN AMRO
In many ways, it is more accurate to describe today’s most successful banks as IT organizations with banking licenses than as financial institutions with IT departments. The competitive pressure from smaller, specialized FinTech companies and larger, multinational banks forces all players in the industry to continuously improve or risk being left behind.
That was the scenario that our bank, ABN AMRO, faced a few years ago. The third-largest bank in the Netherlands, ABN AMRO has more than 22,000 employees and 5,000 working in IT. Though we now have more than 300 agile development teams, a little more than three years ago we used predominantly a waterfall approach. Innovation and speed to market were hampered by numerous factors including long lead times – both for starting projects and for moving them into production; late stage code merges leading to software quality issues that were detected too late in the process; manual handovers and approvals; infrequent production releases and poor coordination between Dev and Ops. We once estimated that it would take six months to get a simple “Hello World” application deployed to production using this process.
The move to agile began to address some of the challenges that were slowing us down, but it was not enough. To really speed software delivery, we needed continuous integration (CI), continuous delivery (CD) and – our ultimate goal – continuous deployment. The initiative we kicked off to realize this CI/CD and DevOps vision consisted of two main parts: “Pave the Way” and “Make it Happen.” Paving the way was about the technical prerequisites – tools, infrastructure, pipelines and so on – needed for automation. As an early first step, we ran proof-of-concept projects with a few teams eager to try something new using new pipelines that we set up based on Jenkins, Maven, Nexus and Bitbucket.
"The move to agile began to address some of the challenges that were slowing us down, but it was not enough. To really speed software delivery, we needed continuous integration (CI), continuous delivery (CD) and – our ultimate goal – continuous deployment."
The success of these early projects generated excitement, and we began taking the next step: Making it Happen. As more and more teams began adopting continuous integration and continuous delivery practices, our focus was on the changing mindsets, behaviors and processes so that we could effectively employ the tools and technical infrastructure that we had put in place. During this phase we concentrated on getting teams to follow five key principles:
Automate all repetitive tasks
Integrate quickly and often (at a minimum once per day)
Hold everyone equally responsible for the success of their project
Keep changes small (this may require breaking monolithic applications into smaller pieces, to make it possible to deliver new business functionality in two-week sprints and push it rapidly to production)
Get feedback as early as possible, and give the feedback continuously to the whole team
Throughout our DevOps transformation, the CloudBees Jenkins Platform has been central to our implementation of continuous integration and continuous delivery practices, particularly on the CI side. Our CloudBees environment includes 10 controllers (managed with CloudBees Jenkins Operations Center), more than 80 shared executors and 250 plugins. It supports 1,500 developers on 300 teams and runs 16,000 jobs. All of this is hosted on our internal infrastructure; more than 70 virtual machines in our data center are allocated to Jenkins.
To keep pace with the growth we are seeing – we continue to onboard teams every week – we decided recently to transition to CloudBees Jenkins Enterprise hosted on AWS. This move offers several advantages, both for IT teams and for agile development teams.
First, it will allow decentralized maintenance because each team can have their own controller. Teams will be able to install their own plugins without having to wait for the IT team to do it for them, and removing this bottleneck will help them stay on their sprint schedule. Providing a controller for each team fosters increased team autonomy and reduces configuration conflicts because each team is responsible for its own controller and can experiment with them as they see fit, without the risk of affecting another team’s work.
"All of these improvements, which continue to aggregate and build as we onboard more teams and refine our continuous integration and continuous delivery practices, have put ABN AMRO in a stronger position relative to our FinTech and banking competitors by enabling our teams to rapidly deliver the high-quality, secure software and frequent updates that our industry demands."
Another key advantage of CloudBees Jenkins Enterprise is automated scaling. In the past, when our Jenkins servers became overloaded we needed to manually add virtual machines, a process that sometimes could take weeks. With CloudBees Jenkins Enterprise running on AWS with Docker containers, we can scale automatically, and in a matter of a few minutes, to increase build capacity, while still maintaining multiple connections to tools running in our on-premise data center.
Of course, as a bank security is a top priority for us. We identified potential security issues up front and made a plan to resolve or mitigate them so that we can work in a public cloud in a very secure way that has been endorsed by ABN AMRO’s cloud approval board. Moving forward, as Kubernetes comes to AWS, we plan to take advantage of Jenkins on Kubernetes as well and complete our full roll-out by the start of 2019.
When embarking on a DevOps transformation, it’s important to secure the commitment and involvement of senior management and set expectations appropriately enterprise-wide. Meaningful change does not happen overnight, and in fact it can take large organizations three to eight years to implement continuous integration and continuous delivery effectively.
Having put the time and effort in at ABN AMRO, we are now seeing significant benefits that span many teams across our organization, including shorter lead times, an increased number of successful builds and improved code quality and security. Our Internet banking team, for example, went from four releases per year to 18. Another team doubled their velocity after a single sprint. Yet another team reduced their build times from five hours to five minutes. And we have already seen some teams achieve continuous deployment, in which they use a fully automated process from code commit to production deployment with no human intervention.
All of these improvements, which continue to aggregate and build as we onboard more teams and refine our continuous integration and continuous delivery practices, have put ABN AMRO in a stronger position relative to our FinTech and banking competitors by enabling our teams to rapidly deliver the high-quality, secure software and frequent updates that our industry demands.
What we know is that as we embrace continuous integration and continuous delivery and ultimately continuous deployment, we’re never done. At ABN AMRO, we’ll continue to seek ways to improve our approach to support our strategic goals, stay ahead of the competition and serve our customers in the best ways possible.
Stefan Simenon, Head of IT Tooling and Software Development
Wiebe de Roos, Continuous Integration/Continuous Delivery Consultant