8 CI/CD Best Practices for Your DevOps Journey

According to a recent IDC study , the worldwide DevOps software market reached $2.9 billion in 2017 and is forecasted to reach $6.6 billion in 2022. With over 50% of organizations embracing DevOps last year, continuous integration (CI) and continuous delivery (CD) have become an integral part of the software development process.

Your business is likely embarking on a DevOps transformation of its own. In this blog post, we will discuss CI/CD best practices that can help accelerate adoption no matter where you are in your DevOps journey.

But first, let’s take a step back and set the stage for why we practice DevOps in the first place.

Continuous integration and continuous delivery refers to the process of developing and delivering software in short cycles of build-configure-deploy-test-release underpinned by automation. In short, we do CI/CD to deliver higher-quality software frequently and predictably.

CI has become practically ubiquitous to software development. Therefore it is not surprise that as the bread and butter of development, CI is a key practice in DevOps organizations.

CD is not as widely adopted as CI, but is key to shifting left. CD starts with development, building, unit testing, static code analysis (SCA) and static analysis security testing (SAST) on CI. Eventually, the pipeline extends the automation to functional, integration, performance and security testing, along with configuration management and deployment. These are critical areas for automation -- and in a DevOps context they are essential.

I would even make the jump and say that if you aren’t doing CD, you aren’t doing DevOps well. Why? Because traditional IT operations don’t get involved until AFTER deployment into production - when real users begin hitting the application. How can you expect IT operations to shift left and embrace a culture of DevOps if there is a giant, unautomated chasm in your software delivery pipeline? Asking them to shift left in this context is paramount to asking them to jump of a cliff. If you are doing CI alone, you are likely sole inhabitants of ‘Developer Island’. It might be sunny there, but you are decidedly alone and separate from the rest of the business.

I know what you’re thinking. This all makes great sense, so why isn’t the CD pipeline automation the norm? The answer is simple… this stuff can be hard and transformation doesn’t happen overnight. It usually begins on the left with CI and as the DevOps initiative matures, CD is the next logical step. It’s certainly not serial out of necessity, but this is the adoption pattern we often see.

In fact, I have news. Not only does your DevOps transformation not happen overnight, it’s never over. This is all about continuous improvement, which means nothing sits still. There isn’t a DevOps train that terminates at the end of the line. The minute you’ve controllered these best practices, new ones will emerge because every day we as a community get better at delivering software.

Which brings me to the topic of shift left and continuous everything! When you have both CI and CD pipeline automation in place, you are in a fantastic position from a tools perspective to shift left. This is a perspective shift from problem detection to prevention which means testing earlier, and earlier - effectively shortening test cycle times while preserving code quality. The result? Better software, delivered more quickly that shortens the feedback loop between end users and developers.

But shifting left isn’t just about tools. It’s also about people and practices - the proverbial DevOps trinity . It means everybody has a stake in the reliable, repeatable delivery of high value software. It means that the documentation team joins the engineering team when stories are being picked for a release. It means that product marketing comes to that same meeting with research on the market opportunity for such a feature. It means that product managers sit on analyst inquiry calls with the analyst relations team. It means that passionate customers beta test features and give meaningful feedback that can be immediately acted upon. It means that before a feature is even released, sales is trained, primed and ready to do pipeline generation. It means DevOps is at the very heart of the business and every function from marketing, to sales to support and services is totally informed and invested from ideation onward. No more chucking features over the wall. No more islands. No more excuses. That’s continuous everything .

So you’ve got a CI/CD strategy. You’re shifting left. Now what? A critical - and often overlooked step in DevOps transformation is measurement. It’s time to take credit for how increasingly amazing you are doing! (or understand where you are falling short, DevOps is about honestly accepting and addressing failure or in efficiencies...together). You need visibility in order to know which lever to pull next in your path to continuous improvement. You’re delivering code faster than ever! But how’s the quality? Is it failing tests? Is it unstable in production? How much rework is required? You won’t know if you don’t track it. Implementing and measuring the success of the best practices in DevOps is crucial if you wish to make your automation cycle faster over time. If, like many, you jumped headlong into your DevOps journey without putting monitoring in place - it’s not too late. Consider keeping track of the deployment frequency, change lead time, change success/failure rate, mean time to recovery (MTTR), and the rate of security tests passes in your organization to stay on top of your DevOps transformation.

It’s not possible to overemphasize the need for security in a world where breaches and vulnerabilities continue to cause massive reputation and financial losses to businesses of all sizes and capabilities. Since the CI/CD system offers access to your codebase and credentials to deploy in various environments, it is often the prime target. Recall the infamous Uber breach where hackers gained access to AWS credentials that were embedded in the GitHub software repository - stealing the personal information of 57 million users and 600,000 drivers . Uber then paid to cover it up, but that’s a different story. It’s certainly not uncommon for credentials to be stored in private repositories for automation purposes. Therefore, you should consider isolating your CI/CD systems, placing them in secure internal networks. VPNs, strong two-factor authentication and identity and access management systems will help you enforce ‘the principle of least privilege’ and restrict exposure to threats. For example, containerize your agents and place them on secure networks. In addition, you should also ensure that security is baked into the development process from start to finish. This is commonly known as DevSecOps .

For effective implementation of DevOps, a microservices architecture is the best way forward. However, re-architecting your existing applications can be a daunting task, and you may consider an incremental approach where you maintain your mission-critical systems and weave the new architecture around it. This will help you gradually replace the old system with the new architecture.

Tools like Jira and Bugzilla can help you achieve better visibility into the progress of your software and collaborate easily with distributed teams. You will also need a version control system like Git, which creates a ‘single source of truth’ for your team, allows tracking of changes in the code-base, and is a life-saver whenever a rollback is required. By enabling the teams to collaborate and integrate changes into the shared repository, GitOps can significantly improve your MTTR.

The goal of reducing (or even gasp… eliminating) branching is to spend more time on development and less time on version control. However, to make the most of GitOps, developers are expected to commit directly to the main branch or merge changes from their local branches in at least once a day. This will force developers to deal with smaller, bite-size bits of integration pain instead of the massive integration pain (and associated rework) that happens when attempting to merge many branches into the trunk just before release.

Eliminate any practices where source code is built multiple times. Even if the software has to be built, packaged, or bundled, you should execute that step only once and promote your binaries. Most successful CI implementations include the build process as the first step in the CI/CD cycle to package the software in a clean environment. This eliminates oversights and reduces the chance for errors to be introduced and/or missed anytime later. Further, the resulting artifact should be versioned and uploaded to Git each time so that whenever it is pulled the build does not change.

While an incremental approach to automation sounds good, organizations transitioning from manual to automated processes often find it difficult to decide which processes to automate first. For example, it is beneficial to automate the process for compiling the code first. As developers need to commit code on a daily basis, it makes sense to do automated smoke tests. Unit tests are usually automated first to reduce the workload on developers.

Consequently, you can automate functional testing , followed by UI testing. Functional tests do not usually require frequent updates in the automation script, unlike UI tests which have more frequent changes. The main idea is to think about all possible dependencies and evaluate their impact to prioritize automation sensibly.

Frequent releases are only possible if the software is in a release-ready state and you have tested it in a production-like environment. That’s why the best practice is to add a deployment stage which closely resembles the production environment before the release. Some release best practices include:

• Canary deployment . Releasing to a subset of users, testing with that base and rolling it out to the wider population if successful (or rolling it back for iteration if it’s not).

• Blue green deployment . You begin with two identical production environments. One is live in production. The other is idle. When a new release is rolled out the changes are pushed to the idle environment. Then they switch - the environment containing the new release becomes the live environment. If something goes wrong, you can immediately roll back to the other environment (the one that does not contain the new release). If all is well - the environments are brought to parity once more.

• A/B Testing. A/B testing is similar in flavor to - but not to be confused with- blue green deployments. A/B testing is a way of testing features within the application for things like usability. The better performing variant of the feature wins. This is not a release methodology.

You should consider running tests in containers as this approach allows the quality assurance team to reduce the number of environment variables and changes present between the development and production environments. The primary advantage of using such ephemeral testing environments is that they add agility to your CI/CD cycle. The QA team does not have to pull a build from a CI server and install it in a separate testing environment; instead, it can run tests against a container image. It is much easier to spin up containers (they have no separate installation or configuration requirement) and also destroy them when not needed.

The primary goal of DevOps or CI/CD best practices is to automate the process of building, testing and releasing software. This means you will need access to DevOps tools that help you simplify automation as well as get better visibility into the progress of your software. Further, there should be a mechanism to track DevOps performance metrics throughout the software delivery lifecycle and raise alerts for quick recovery when something goes wrong in release or deployment.

Deciding which DevOps tools to implement and where to invest first can cause “analysis paralysis” - even in organizations that have been working through their DevOps transformation for some time. You can avoid getting stuck in a similar position and jumpstart your DevOps transformation with the CloudBees Starter Kit . Learn more about this starter kit here .

• Learn more about DevSecOps in CI/CD .