One of my favorite moments at Jenkins World last year came when tech guru Jez Humble , during his keynote address , led the crowd in a quick show of hands about who’s truly doing continuous integration (CI). Since the seats were filled with some of the most dedicated DevOps pros you’ll find anywhere, one would assume that everybody would be totally on the ball with continuous integration.
Humble asked a few leading questions:
Who isn’t feeding builds into their organization’s main pipeline regularly?
Who can’t fix a software issue in under 10 minutes? Those who can’t or don’t, put your hands down.
A number of hands dropped after each question. Finally, a fraction of the original number of hands were still in the air. These were all Jenkins users!
Humble’s message: A large percentage of people think they’re doing continuous integration, but they really aren’t.
“Continuous integration is really hard,” Humble told the crowd. “And a lot of people redefine it to mean whatever they’re already doing.”
Are you really doing continuous integration? It’s an important question. Continuous delivery (CD) and DevOps, after all, are disrupting the market and providing businesses with a huge competitive advantage. Continuous delivery is built on the tried-and-true practices of continuous integration. Organizations that seek to recognize the benefits of continuous delivery have often failed to fully understand the concepts of continuous integration.
As Humble related at Jenkins World during his keynote address, the organizations that are doing continuous integration correctly are all following a few basic rules. For example, developers’ working copies are synchronized with a shared mainline at least daily, preferably several times a day. Each integration gets verified by an automated build to detect errors as quickly as possible. Organizations that aren’t following these steps aren’t really doing continuous integration properly.
Continuous integration is a development team practice that generates real benefits for an entire organization. Engineers in charge of implementing continuous integration practices want to achieve those benefits and follow the modern practices their peers are following. They’re just running into problems trying to make it happen.
They hear about how other organizations are implementing continuous integration, and then make decisions about whether they can do the same inside their four walls. But every organization is different. The DevOps team may have a vision for what continuous integration looks like in their organization, but it might not fit squarely inside commonly accepted definitions of CI.
Organizations that don’t follow the core principles of continuous integration very likely will run into problems delivering crisp, functioning builds on a regular basis. Over time, the initiative will lose momentum and team members will become disenchanted. People who are resistant to change (which is most of us) will revert back to their old practices, if they don’t see evidence of the benefits of change, even in small increments. In this case, you have multiple problems. Your builds are still riddled with errors. Your team has lost faith in the implementation. You’ve lost critical time, and now you need to restart the project.
Organizations that don't implement continuous integration correctly often face cultural issues. Engineers are great at solving technical problems, but CI requires a shift in culture. Culture is hard to change. You can bring in a continuous integration tool and check most of the boxes that apply to what CI is supposed to represent, but succeeding with CI requires a change in how you work and how you work together. If the culture of the team doesn’t change, they’re going to have a hard time implementing continuous integration.
Bottom line: Doing continuous integration is like putting on a raincoat over your best suit. You have committed to aggressive delivery schedules and promised stellar versions of the company’s new app because you’re confident that the coat provided by your continuous integration principles will protect your project. If the raincoat isn’t waterproof, your suit will get ruined. Same situation with CI. Leave your project open to the elements, and it could end up underwater.
If it sounds like properly aligned CI processes are designed to eliminate errors, that’s not true. Continuous integration itself is a process designed to embrace failure. We want to create a system where developers can fail often – and fail fast, so they can find and fix errors early and quickly. Continuous integration is like a guardrail on a windy road – giving developers the confidence to drive fast, because they’re protected if their builds veer too close to the edge.
The core principles and practices of continuous integration date back at least 15 years, when Martin Fowler introduced the term, and they apply today just as they applied then. Here are seven practices organizations need to follow to truly do CI correctly:
Commit to the mainline: This is table stakes for continuous integration. A developer can set up an automated build and have the build run on every commit. But if the culture is to not commit frequently, it won’t matter. If a developer waits three weeks to commit or branches off for three weeks, he has delayed the integration and broken the principles. If a build breaks, the team has to sort through three weeks of work to figure out where it broke.
Maintain a single-source repository: In complex applications, developers often branch and maintain changes off of a trunk (branch) or main. The branching creates complexity and prevents everyone working with a single source of truth. Teams need to commit/merge to trunk or main at least once per day, or even better for every change.
Automate the build: This is a practice most organizations tend to do well. However, what some who claim to practice CI are simply doing is scheduled builds (i.e. nightly builds), or continuous builds but they are not actually testing or validating each build. Without validation of the build, you are not doing continuous integration.
Make builds self-testing: The first step of the validation process is to know that a build with problems actually failed. The next step is to determine if the product of the build is operational and that the build performs as we expect it to. This testing should be included as part of the build process. This consists of fast functional and non-functional testing.
Build quickly: If it takes too long to build an app, developers will be reluctant to commit changes regularly or there will be larger changesets. In either case, no one will spot a failure quickly. By building quickly and integrating quickly, you can isolate changes quickly. If it takes hours to run, you might have 20 to 30 more changes during that time and it will be difficult to quickly spot problems.
Test in a clone: Validation processes verify that software performs as expected in its intended environment. If you test in a different kind of environment, it may give you false results.
Fix broken builds immediately: It’s critical for development teams to find problems fast and fix them immediately, so they don’t move downstream. Years ago, Toyota instituted a “stop-the-line” approach where workers could pull a rope and stop the manufacturing process if they spotted a problem. CI sets up a process where builds are validated and committed continuously, so if something goes wrong, it’ll be easy to fix.
Despite all the challenges organizations face implementing true continuous integration, it’s important to note how far the software development community has come in following modern processes that create true value for their operations. Many are working hard to make changes and improve their DevOps practices. The biggest obstacles to pristine CI are our cultural, emotional and technical attachments to legacy technologies. These are the enemy of change. Even when you get beyond the “culture of can’t,” carrying the baggage of legacy practices can provide a formidable obstacle.
The other challenge organizations face is overplaying their hands. Everybody wants to transform their operations using methodologies like CI, but few want to talk honestly about how they still need to improve their processes.
More than a decade after continuous integration first became an industry term, we’re still trying hard to do it right.
Brian Dawson is a DevOps evangelist at CloudBees.
Follow him on Twitter .