How to Identify Major Blockers in a CI/CD Pipeline

Continuous integration, development, and delivery pipelines help speed software development. An excellent example of this is Autodesk's increased productivity since adopting CI/CD. When properly implemented, pipelines reduce your time to market and the reliability of new features and releases. 

But do you know what's happening inside your pipelines? Can you identify and remove the blockers that keep it from being as efficient and effective as possible? It's not easy to know if blockers exist and where they are. 

So, it's difficult to know how -- and where -- to focus your efforts. Throughout this article, I’m going to help you identify some of the major blockers you may encounter and suggest a range of ways to remove them by considering a series of questions. 

What Is Your Company's CI/CD Culture?

Before diving into the technical aspects, let's talk about your CI/CD culture. Is it a positive one, where everyone is fully engaged? Do they appreciate its significance to the company's performance and success? Or are they only marginally aware that CI/CD exists, what it does, why it's there, and whether it's working? 

As with any subject, the more engaged people are around a topic, the more supportive they're likely to be. When they're engaged and supportive, they can become ever more invested in that subject’s success. Its success becomes their success. 

Furthermore, when this kind of engagement exists, you're less likely to encounter roadblocks when you need further resources to improve it. For these reasons, I strongly encourage you to consider your company's CI/CD culture. 

Here are three suggestions how: 

1. Run group education sessions where you highlight the existing benefits.

2. Bring in external experts to discuss real-world success use cases.

3. Hold regular webinars or group chats that show what's going on and how it's helping.

These are just a handful of ways. I'm confident you can think of others. 

What Are Your Pipeline's Metrics?

Now let’s consider more quantitative methods; after all, if you don't know where you are now, you can't know where and how you need to improve to get to where you’re going. 

1 - How Quickly Can Developers Provision New Environments?

Regardless of whether developers are new to your company, new to your department, or new to a given codebase, you should ask yourself: 

• Can they provision new development environments quickly?

• How fast can they go from zero to ready to work?

• Is it easy to throw away an existing environment and start over?

If developers can't provision their development environments rapidly, they're not as productive as they could be. 

Is the process of provisioning a new development environment left up to each developer? Do they need to manually step through a checklist, run scripts, and identify dependencies? Or can an environment be automatically provisioned with an industry-standard provisioning tool such as Ansible, Terraform, or Docker? 

2 - How Quickly Do Tests Run?

How long does your entire test suite take to run? Does it complete quickly, or does it drag on for hours -- even days? 

There is no average runtime, as no two applications are ever the same. And I'm not suggesting that there should be, or even is, a standard minimum or maximum execution time. What I am saying, however, is that they can't take a long time to run. The longer it takes, the more likely developers will avoid running them. 

Given that: 

• Where can you improve its performance?

• Are all the tests necessary?

• Have tests been written that don't measure anything meaningful?

• Could you write the tests more efficiently?

• Should you invest resources in improving the development team’s testing knowledge?

Review the current test suites, whether unit, acceptance, integration, system, etc. and see how you can improve them. And consider investing further resources in ongoing testing education. 

3 - How Quickly Can You Deploy New Features?

A streamlined CI/CD pipeline makes you more competitive because you can release fixes and features faster.

What do your deployments look like? Have they grown organically into a monolithic or convoluted mess? Could a new set of eyes streamline or simplify them? Is it time to bring on people with specialized experience? 

Does legacy baggage exist, causing the process to take too long, perhaps because of older software or outdated features? 

4 - Do You Have Blockers or Bottlenecks?

We've already talked about long-running unit tests or tests that just aren't run. But they're not the only things that can bog down your pipelines. 

Here are a few more blockers and bottlenecks: 

• Do you have too many dependencies between jobs? Dependencies are a form of coupling, and tightly coupled jobs are fragile. They're easy to break and difficult to put back together.

• How many stages and jobs do you have? Simpler is always better.

• Are your code repositories too big? Large repositories take longer to update and check out and can be signs of a monolithic architecture. Break up your code to streamline your pipelines.

What Is Your Review Process?

This examination of your CI/CD pipelines isn't a one-time thing. Schedule regular reviews after each major code release, alongside periodic intervals. 

How Transparent Is Your Pipeline?

It's one thing to have a pipeline in place, but how transparent is it? For example: 

• Do you have the build, integration, and deployment metrics located and published in an easily accessible location, such as on a monitor in the development office?

• Is it easy to see when a build has failed and why?

• Is this information accessible to everyone?

My intent here isn't to infer that someone within your organization might be dishonest or have questionable integrity. I say it because when the information is readily accessible, it brings several significant benefits. 

These include: 

• The ability to hold people accountable.

• Group awareness of declining -- and improving -- quality standards.

• Positive affirmation of successful outcomes, such as new features and bug fixes.

As a result, I suggest that, if you don't do it already, you should begin ensuring that you're collecting as much information on your pipelines as possible. Then make that information as readily accessible, in as usable a manner, as possible. 

Don't just show the raw numbers, though. They're great. But they don't always give proper context. Show trends over time as well. Make it easy to know whether the pipeline is improving or declining. That way, it's easy(er) to know what action to take and when. 

How Long Does It Take from Story to Feature Release?

Now that we've considered a series of areas in isolation let's step back and take more of a bird's-eye view of the pipeline. How long does it take you to go from feature story to feature release? 

What's that process like? Is it a streamlined, well-documented, well-defined, and smooth process? Or is it more haphazard and hodgepodge? By viewing it this way, you can get a deeper, more thorough perspective on how it all fits together and works. 

After you've reviewed the situation, consider these questions: 

• Do you have a standard template for writing your stories?

• Do you have a standardized review and vetting process for them?

• How do you publish them?

• How long is it from when they're written to when the feature has to be implemented (or abandoned)?

• Is the way that you're writing your cards excluding some members of your organization -- namely your developers? Said another way, do the cards contain unnecessary jargon, acronyms, or other insider terms?

• Are they overly complicated, taking too long to create, or attempting to do too much?

• What's the reporting process from end to end? Is it effective and transparent?

This is just a sample of things you can look at. For some background on how much of a difference effective  CI/CD can make on your release process, read about Tietoevry's experience here. 

Conclusion

While I've not always been as tangible, technical, or explicit as I could have been, These questions are a good starting point to help improve a CI/CD pipeline's efficacy. Suppose you jump in without considering the process holistically or appreciating the key areas that need addressing. In that case, you run the risk of spending a lot of time and effort for minimal gain. 

I hope that if you're keen to improve the efficiency and effectiveness of your CI/CD pipeline, this post will give you a good start. Please take the advice as a jumping-off point, designed to stimulate your thoughts and efforts, one that you can build on over time. 

It's a profound and complex area that can't be solved or improved quickly or easily. But, when done continuously, improving your CI/CD pipeline can return significant rewards in time, cost, and more. 

Read More

• An excellent example of how continuous integration, development, and delivery pipelines help speed software development is Autodesk's increased productivity since adopting CI/CD.

• Read some lessons from Intel’s Software Engineering Transformation.

• Learn more about how to achieve DevOps success with a holistic approach. 

For some background on how much of a difference effective  CI/CD can make on your release process, read about Tietoevry's experience here.