BT’s Journey to a Global, Multi-Vendor DevOps Ecosystem

By Rajesh Premchandran, IT Director, Head of Software Engineering Excellence, BT and Madhu Madhanan, Global Head DevSecOps Transformation, Wipro

BT is the oldest telecommunications company in the world, dating back to the mid-1800s. As the company has grown, evolved into new markets, acquired businesses and kept pace with IT trends, modernizing our technology estate has also become important. As a result, BT embarked on a journey to radically simplify product offerings, create more lean business models and build a scalable technology platform to facilitate growth. A key focus of our transformation within engineering teams was DevOps transformation and our goal was to scale DevOps across BT’s global enterprise.

Because we already had a rich network of SI partners and software vendors, we knew we wanted more than a tool management approach to DevOps. Instead, we wanted to create a collaborative ecosystem to empower our engineers, developers and partners to drive change. That meant we approached DevOps as a platform with standards and governance that also allowed flexibility so our teams could continually experiment and learn. We also integrated actionable insights into the tools and solutions to support data-driven decision-making.

That gets us to our two P’s and two M’s: platform, partners, metrics and mindset. This is our North Star, if you will – the four dimensions on which we based our transformational DevOps program.

We worked closely with Wipro to design and implement DevOps at BT, and we paid close attention to best practices, industry standard cycle times, deployment frequencies, mean times to resolve issues and change failure rates. We benchmarked all of those points against industry-leading models.

In addition to our SI partners and software vendors, we had many other stakeholders, such as business units, lines of businesses and centers of excellence, for aspects like testing, infrastructure, as well as a large engineering community. We brought those synergies together to create an executable DevOps roadmap to which everyone could agree.

We conducted stakeholder workshops in eight cities to understand our current baseline of assets and best practices. We discovered that we already had many reusable solutions – we called them islands of excellence – that could be scaled across the organization. During the workshops, we also identified planned solutions and we conducted a hackathon to generate new ideas to solve specific problems. For every capability we wanted to build, augment, consolidate or centralize, we created a business case, gathered feedback and prioritized according to outcomes. This effort led to what we called our transformational backlog. The workshops helped us gain buy-in and shared ownership as we moved forward on our DevOps journey. But, more importantly, we broke down silos so teams felt comfortable working together.

The idea of cross pollination and crowdsourcing may sound counter-intuitive when you think about development teams and partners who generally work independently. We created a centralized composable DevOps platform that is a mechanism for what we called systemic enforcement. All of the tools and capabilities are available in a self-service marketplace so the different groups can continue to work independently. However, when a team runs into a problem or needs guidance, they can reach out to the central DevOps platform team to ask if other groups have solved the same challenge.

This cross pollination of ideas and solutions encourages collaboration. And because solutions from different teams are part of the centralized marketplace, people can see metrics showing how many times their solution has been downloaded by other parts of the organization – which reinforces ownership, pride and collaboration.

And here’s where systemic enforcement comes into play. Governance of enterprise standards has become automatic. The self-service marketplace enforces standards with templates and reusable solutions.

We wanted a single source of truth where all data from the underlying tools and solutions is readily available to developers and management. This includes metrics such as lead times and reliability that help people make informed decisions as they build pipelines and develop solutions.

We’re now a year into our DevOps journey, and we’ve got more than 200 pipelines on our platform. We’ve created a DevOps Center of Excellence that’s had a positive impact on our development teams. Although we’re continuing to improve our DevOps platform as we go forward, we believe we’ve successfully created a great developer experience.

This blog is based on excerpts from Rajesh and Madhu’s DevOps World presentation titled: “Federated DevSecOps: Leveraging MANY Global SIs to Accelerate BT’s DevOps Transformation."