Internal Revenue Service Modernizes with Agency-wide DevOps Initiative

Industry:  Federal Government 

Geography:  United States 

Summary: To modernize software development programs and increase productivity, the U.S. Internal Revenue Service adopted DevOps practices using CloudBees CI. 

Challenge: Eliminate inefficiencies related to the agency’s legacy manual and siloed development practices 

Solution: Adopted CloudBees CI to support CI and CD process automation, accelerate application development and lay the foundation for continued modernization and innovation 

Results: 

• Achieved significant time savings from CI/CD process automation - approx. 21,000 engineering hours per year 

• Improved, built-in security avoids production delays 

• Increased visibility into quality checks throughout development lifecycle 

• Winner of 2020 CloudBees Innovation Award for Diversity in DevOps 

Product: CloudBees CI 

Part of the U.S. Department of the Treasury, the Internal Revenue Service (IRS) is the revenue service of the United States federal government. The IRS is responsible for collecting taxes and administering the Internal Revenue Code, the main body of federal tax law of the United States. The duties of the IRS include providing tax assistance to taxpayers and pursuing and resolving instances of erroneous or fraudulent tax filings. 

With more than 74,000 employees and a budget of approximately $11 billion, the IRS easily matches the size and scope of most major corporations. And like most large modern organizations, software lies at the heart of its operations. 

"The IRS is a very big and very mature organization,” says Amin Qazi, technical advisor at the IRS. Qazi and his team advise about 80 organizations across the agency and oversee a large share of the application development and integration services for the IRS. "We try to make sure that everyone is going in the same direction,” he says. 

For the last five years, Qazi and his colleague Rupesh Kumar, solution director of Agile/DevOps at Citizant, a prime contractor to the IRS, have been helping the IRS modernize its software engineering programs by introducing DevOps practices and tools to development teams across the agency. 

When the two experts joined the IRS in 2015, the agency’s software development operations were battling inefficiencies. Developing and deploying applications was difficult in part because of the fragmented, siloed way in which engineering teams worked. "Since the IRS is a very big organization, you had to know the right people to get applications deployed and there was a lot of orchestration work that you needed to do,” says Qazi. "It was hard to keep track of who was doing what, when and where.” 

That’s when Qazi and Kumar started looking at continuous integration and continuous delivery (CI/CD) tools to streamline development lifecycles. "We realized that we needed to get into DevOps and CI/CD as quickly as possible,” Qazi says. 

The group’s initial attempt at rolling out homegrown CI/CD tools encountered resistance from dev teams who were worried that complying with a new engineering framework might be a distraction from their "day job” of writing code. Switching teams to Jenkins, however, helped change their minds. But the open source solution had difficulty accommodating the huge collection of existing tools used by dev teams across the agency. That motivated the group to find a more scalable and versatile CI/CD platform that was built to handle the demands of an organization the size of the IRS – and CloudBees CI fit the bill. 

"One of the things we liked about CloudBees CI was its scalability and fault tolerance,” Qazi says. Also attractive was CloudBees CI’s rich ecosystem of verified plugins that helped the group take the CI/CD solution "from adoption to scaling it out.” To prepare for rolling out CloudBees CI, the group leveraged the agency’s "joint innovation lab” – similar to a development sandbox – to explore and test out capabilities such as CloudBees Role-based Access Control and elastic search. 

"The early results helped us get buy-in from the stakeholders and showed them how much more CloudBees CI can do beyond what open source can provide,” said Qazi. "That’s when the decision was made to create an enterprise standard within the IRS around CloudBees CI.” 

The group started deploying the new platform with a set of pilot projects using a CloudBees-orchestrated pipeline. "That’s when we started getting traction and other teams got interested,” Kumar says. "We did a lot of road shows showcasing our successes with these five projects. Then we started onboarding more and more projects.” 

A lot of developers were sold on DevOps when they found out that the efficiency and automation improvements would save hours of deployment time and allow them to spend more weekends at home with their family instead of at work. "Those were the things that connected with the developers,” Qazi says. "They started understanding the value of it, and they started working with us.” 

CloudBees CI has since been adopted by the vast majority of IRS development teams, who have learned to onboard and manage pipelines on their own, rarely needing the central team’s help. "That’s what we wanted them to do,” Qazi says. "Rather than engaging us and having our team become a bottleneck, they went ahead and did it themselves. And once they get used to it, there is no going back.” 

Productivity improvements from the move to DevOps became evident right away. As teams adopted automated testing processes, for example, they saved hours of testing per application. Embedding security testing early in the pipeline avoided last-minute production delays. And plugins, such as CloudBees Label Throttling, helped developers avoid machine overloading and increase the speed of builds. 

Today, the DevOps team regularly meets with CloudBees to explore opportunities for introducing more efficiencies and innovations. They also meet with other federal agencies biweekly to share what they’ve been doing with CloudBees. Among other modernization efforts, the group is looking to introduce new release orchestration tools, move some projects to the cloud, and adopt application containerization and infrastructure-as-code technologies for new projects. 

All of these initiatives will be orchestrated through CloudBees CI, Qazi and Kumar say. "DevOps is our major modernization effort,” says Qazi. "All the other efforts are part of it and are being rolled into that effort to make sure that we are all aligned.” 

Qazi and Kumar are proud of all that the team has accomplished together and for the development teams throughout the IRS. "Always look at things from the perspective of the customer, because at the end of the day, they are the ones who will be using your services,” says Qazi. "Don’t be afraid of failing because if you fail, you’ve learned a lesson. But be sure to fail smart so that you don’t waste too much time or effort. Just feed it back and then do better the next time.” 

Significant time savings

Productivity improvements from DevOps-driven automation and process streamlining with CloudBees CI is saving an estimated 21,000 hours per year at IRS for all the projects onboarded to date. 

Increased testing productivity

The IRS development team saw significant productivity gains from automated testing. "Previously using manual testing methods, it took about 20 minutes to complete a single test, adding up to almost a week of testing for any given application,” Qazi says. "CloudBees CI accelerated the process by enabling automated connections to the agency’s testing tool, reducing testing times from 20 minutes per individual test case to about 20 minutes for the entire suite of tests,” he said. 

Faster builds

CloudBees Label Throttling plugin helped teams avoid machine overloading, increasing the speed of builds. 

Built-in security

Security testing is now built into the pipeline, helping teams avoid last-minute delays from a negative security test finding late in the cycle. "There is no longer a question about our code not being security compliant the day before going into production,” says Qazi. "As a big organization, we really see the impact of this tenfold.” 

Quality assurance

Using CloudBees CI dashboards, IRS teams can see the progress of code through the entire lifecycle and ensure that all necessary security and quality scans have been completed. "That is where the win comes in,” says Qazi. "This is where you can really see the benefits of the CloudBees CI platform.”