Let's Get Back to Coding! 5 Approaches to Solving Pipeline Bottlenecks

Not to brag, but the dev teams I manage have great velocity. We can crank out quality code to the calming hum of a white noise machine. But like all stories, it didn't start out this way. Not even close. It takes time, communication, collaboration, failures, successes, and a lot of meetings talking about productivity (which were awkward at times, but they allowed us to pinpoint frustrations… Plus, Ronnie would make his award winning cookies, so win-win). I’m sure you can find an inspirational meme out there that involves a cat looking out a window in quiet contemplation, but I'm going to jump right in and share with you what worked for my teams.

First, some facts to set the scene: on an average day, the typical developer can write about 100 lines of code. And that's what the more productive developers achieve. Some developers may produce as few as five or ten lines of code each day. That's not an especially high number, given that the typical phone app contains something like 50,000 lines of code. It takes many, many developer-days to write an application—let alone maintain it on an ongoing basis—if programmers can churn out only a few hundred lines of code per week.

That's why development teams need to constantly look for ways to accelerate code production. They need to identify bottlenecks that slow the development of software, then find tools and processes that can mitigate the challenges they face in continuously writing and deploying applications. (This is what those cookie filled meetings were for.)

Now, we all know that to achieve this goal, continuous integration (CI) pipelines are a great place to start. Although there are many reasons why code production may end up taking more time than it should, problems in initiating, configuring, and executing CI pipelines are some of the most common. The more time developers have to spend on tasks like setting up and managing CI pipelines or troubleshooting failed builds, the less time they have to do their main job: writing innovative code. Plus, developers who are able to spend more time coding and less time fighting with CI tools tend to be happier developers. This is the biggest takeaway, because this is what makes developers happy. They want to write code that matters and that contributes to the end result, regardless of whether the code supports a better user experience (who hasn’t experienced the onset rage of an app that doesn’t deliver what you need NOW?), or added security (so anyone can feel safe and secure using your software, including grandma and grandpa), or a sweet program loved by many adoring fans.

Most programmers choose their careers because they like coding, not because they like manually setting up software or sorting through CI logs to figure out why a build is taking longer than it should. 

OK, enough of the backstory—why are you here? (And thank you for continuing to read past my strange personality!). I want to share my story on tackling that piece of software delivery life cycle “machinery”—the pipelines. 

We all know a well-structured CI pipeline ensures that developers can write, integrate, and build new application code quickly and efficiently. However, setting up an efficient CI pipeline can be a lot of work. That's especially true in cases where development teams need to initiate multiple pipelines, each of which requires a somewhat different configuration. If developers have to set up each CI pipeline from scratch and tweak it manually, they may end up spending weeks initiating pipelines before they write their first line of code. Where’s the “do not repeat yourself” (DRY) experience here? It's frustrating to reinvent the wheel over and over again, especially if it keeps you away from your goals.

The need to ensure that pipeline configurations conform with organizational standards or regulatory requirements only makes the issue more complicated. It can be challenging for developers to know which standards their pipelines need to meet, let alone implement those standards in an efficient way. 

So here it is, finally, a list of features that we adopted to tackle our pipeline challenges. I’ll add the solution we choose at the end of the blog. The challenges we saw were pretty universal, and the solution we chose solved these for us and provided measurable success that made everyone happy—including our CEO, who actually smiled. It was awkward, but there was a smile. My hope is you identify with some of these challenges and find a solution that works for you. 

Pipeline Templates and Pipeline Template Catalogs are the key here in not repeating yourself with pipeline creation. These features let CI admins and development teams define standard pipeline configurations in the form of templates. After creating a template like this, admins or developers can share it across the organization by configuring a Pipeline Template Catalog. The catalog organizes the pipeline templates into groups depending on your teams. The back end has a different set of templates to choose from versus the front end. All are maintained from one location, live, and available for teams to grab and deploy automatically (via the beautiful act of self-service).

The result is not only much less time and effort on the part of developers to initiate pipelines, but also a lot less guesswork in determining how to configure pipelines to meet domain-specific requirements. 

While templates are a great way to streamline the initiation of pipelines across multiple projects and development teams, the pipeline configuration that works for one team may not suffice for another. Meaning that it needs to be flexible to meet the needs of today as well as tomorrow. 

For example, I had a team that loved this one tool integration, but it wasn’t built into our CI configuration. We needed a way to allow them the flexibility to add to their configuration yet still conform to our tested and approved CI config setup. What to do? I certainly wasn't going to stand in the way of my developers, who wanted this tool to be more productive.

To tackle this, we first give every developer full admin control over the CI pipeline. That solved the challenge of allowing developers to customize their configurations. The problem we found with this approach was that it's a recipe for chaos and nonconformance. If individual developers can deploy any plugin or configuration change they want without oversight or controls in place, teams are likely to end up with pipelines that violate organizational or regulatory standards, and/or that are unstable. We needed to work together on this.

A better solution for us was to leverage Configuration as Code (CasC) which provides the foundation for a manageable, Git-based workflow that lets developers request CI configuration changes, then validates them using automatic tests before the changes are applied. The process works like this:

• When a developer wants to change the configuration of a managed controller (in my example, a Jenkins® instance)  the developer issues a pull request on Git

• The pull request triggers a new managed controller instance, where the configuration changes will be tested automatically

• If the tests pass, the pull request will be merged into the main branch of the targeted managed controller so that the changes take effect

This approach solves two key challenges. First, it provides a self-service, automated process through which developers can request CI configuration changes. There is no need to navigate institutional bureaucracy or track down admins to request a change.

Second, the Git-based approach ensures that configuration changes are properly tested and validated before they are applied. In this way, development teams avoid the risk of introducing changes that cause pipeline instability or nonconformance.

Developers can have their sweet setup and stability too.

Even when pipeline configurations are thoroughly tested and validated, there's always a chance that they won't run as expected. And when something goes wrong—for example, when a build fails to complete or takes much longer than expected—the responsibility for figuring out why falls to developers.

In traditional CI environments, troubleshooting pipeline execution issues is not very efficient. It typically requires developers to check build logs or CI console interfaces in order to track down the source of a build problem. That translates to context-switching, which is a big distraction from coding. 

For our solution, we utilized an SCM Reporting feature that automatically pushes information about build issues into GitHub or BitBucket. And we utilized a Slack plugin that sent targeted messages directly to the individual developers who committed the code associated with a CI job. Therefore only those developers who need to know about build issues are notified, without distracting other developers from their coding work. Bonus—the Slack plugin made it possible to push build information to Slack as well.

The result? My developers can track build statuses and identify issues without having to switch out of the coding environment that they rely on for doing their main jobs. And we reduced the notification noise that devs saw, which surprisingly eats up a lot of time!

CI pipelines don't exist in a vacuum. In many cases, one pipeline can't run until another pipeline completes. The challenge that pipeline dependencies present in a traditional CI environment is that developers must manually keep track of the status of multiple pipelines and trigger new jobs by hand based on that status. Monitoring pipelines manually is not only tedious, but it also creates a huge distraction for developers who would rather be coding.

For our solution, we looked at a tool that removed this manual aspect from our CI process, regardless of what team they were on. We used the Cross Team Collaboration feature, which made it possible to coordinate pipelines by generating notification events in one pipeline, then sharing the notifications with other pipelines.

For example, we had a pipeline that generated a JAR file. The JAR file is used by a different pipeline to complete a separate task. With Cross Team Collaboration, we configured an event notification in the first pipeline using code, then configured a trigger for the second pipeline based on the event notification from pipeline one. The result is a fully automated solution for connecting the two pipelines. Instead of relying on the team managing the first pipeline to notify the second pipeline's team when the JAR file is ready, each pipeline can be kept in sync and executed automatically. I don't know if you’ve ever experienced this, but names were often updated, and this would ultimately update the path. We had so many “Path Not Found” errors because of this simple change—it was frustrating. Event-based triggers where pipelines can listen for each other was a perfect solution for us. 

In a perfect world, each CI instance in your organization would be active only when necessary. It would shut down or hibernate when there are no builds to run, in order to make its infrastructure resources available to other pipelines. 

In our real world, staying on top of resource consumption was difficult. Developers routinely leave CI instances active even when they shouldn't be, and underlying infrastructure platforms (like Kubernetes) offer no automated way of knowing when an instance should be shut down. As a result, fewer resources were available to jobs that actually need them (AKA - lots of grumbling about lack of capacity). Plus, we were left paying for infrastructure we didn't need. That is a “fun” conversation to have with your VP, who signs the checks, and then to turn around and bring that fun conversation back to your team. This shouldn't happen, not in today's age of automated CI. 

Our solution was to use a hibernating feature to automatically shut down or hibernate CI instances after a predefined period of inactivity. To set up the feature, we enabled it in the Helm chart we used to configure the Kubernetes cluster that hosted our managed controllers. We configured automatic hibernation, and tracked the status of hibernated managed controllers, through the interface. By shutting down inactive managed controllers automatically, developers made the resources that were tied up by those controllers available for other controllers that were actually active. The result was faster builds, thanks to more memory and CPU for each build.

Bonus—when inactive managed controllers shut down or hibernate, Kubernetes can automatically scale down its total node count. That reduces infrastructure costs for the business. And lessens the number of uncomfortable conversations with the person who pays the bills.

CI software should help developers work faster and more efficiently. But when developers have to spend lots of time manually setting up, modifying, and troubleshooting CI pipelines, these tools can turn into more trouble than they're worth. They become a source of distraction and frustration for developers who would rather be writing code.

For our solution to the five challenges we identified, we leveraged CloudBees CI features like Pipeline Templates, Pipeline Template Catalogs, Configuration as Code, Contextual Feedback, Cross Team Collaboration, and Hibernating Managed Controllers. 

Our experience with this entire solution was that it turned our CI software from a liability into an engine of developer productivity. The ultimate result is more value delivered to customers in less time, not to mention greater developer satisfaction, since efficient CI means that developers can focus on what they love and are good at—writing code. 

In the meantime, download the 5 Approaches to Solving Pipeline Bottlenecks eBook to learn more.