GitOps: Dev, With a Dash of Ops!

GitOps is a paradigm or a set of practices that empowers developers to perform tasks which typically fall under the purview of IT operations. GitOps requires us to describe and observe systems with declarative specifications that eventually form the basis of continuous everything. (As a refresher, continuous everything includes but is not limited to continuous integration (CI), testing, delivery, deployment, analytics, governance with many more to come).

GitOps combines Git with Kubernetes' convergence properties and serves as an operating model for developing and delivering Kubernetes-based infrastructure and applications. GitOps empowers developers to take on operations in a way that says “You own it, you ship it!” and graduates from being a slogan everyone chants at tech conferences to a reality that can be executed day in and day out.

In this blog, we will explore the what, the why and the how of GitOps in more detail, so that teams have a better appreciation of the wins that result from this investment. To conclude, we will discuss Jenkins X , an open source software project that incorporates a slick implementation of GitOps to help develop and deliver cloud native applications on Kubernetes.

GitOps upholds the principle that Git is the one and only source of truth. GitOps requires the desired state of the system to be stored in version control such that anyone can view the entire audit trail of changes. All changes to the desired state are fully traceable commits associated with committer information, commit IDs and time stamps. This means that both the application and the infrastructure are now versioned artifacts and can be audited using the gold standards of software development and delivery.

GitOps is based off a Git-based source code management system and hence GitHub, GitLab and Bitbucket are natural choices. Note that Bitbucket has floated a GitOps-y approach by the name of BDDA , pronounced like 'Buddha', for "Build-Diff, Deploy-Apply".

GitOps requires us to describe the desired state of the whole system using a declarative specification for each environment. This becomes the system of record. You can describe your environments such as test, staging, and production in a code repo, along with the application version that resides in that environment.

For example, we can describe everything in the Kubernetes model as a declaration. The Kubernetes API server accepts a declaration as an input, and then continually tries to drive (or converge) the cluster to the desired state described in the declaration.

Observability (o11y) is a measurable property of a physical system that makes it capable of being observed. GitOps advocates systems to be designed in a way such that they are liable of being observed, explored and understood. Git commits cause verifiable updates to the Kubernetes cluster. Properties of the cluster should be observable in the cluster so that we can detect if the desired and observed states are:

• the same (as in, whether the desired and observed states eventually converged) or

• different (as in, the desired and observed states diverged).

After all, seeing is believing! Along with observability, controllability represents a major concept of modern control system theory. In order to be able to do whatever we want with a dynamic system under controlled input, the system must be controllable. Observability goes a long way in enabling us to make sound decisions on how to control the system better.

GitOps is a set of practices as discussed above and is tool-agnostic. Like we saw, it has principles rooted in versioning in source control, declaring specifications meant for the desired state and observability. However, in this blog, we will hone in on Kubernetes and how its convergence properties make for an apt implementation of GitOps.

With containers now omnipresent, Docker is the standard currency of exchange in a continuous everything platform. After some initial tussle, Kubernetes has emerged as the de-facto leader in cluster management and container orchestration. This provided some reprieve to a heavily fragmented industry that witnessed an alarming number of competing tools going through revolving doors.

However, while setting up and managing Kubernetes clusters can be fun for folks who like to tinker with infrastructure, some application developers and testers do not want to get bogged down with logistical and administrative fire drills. Even folks who feel comfortable managing Kubernetes on their own admit that it inflates their total cost of ownership (TCO).

Major cloud providers cashed in with managed Kubernetes solutions like Google Kubernetes Engine , Amazon Elastic Container Service for Kubernetes , Azure Kubernetes Service , Oracle Cloud Infrastructure Container Engine for Kubernetes and others, thus alleviating some of the pain points around managing Kubernetes. Our industry boasts a diverse array of technology stacks, tools, platforms and cloud strategies and hence we needed an operating model for Kubernetes that scales and doesn’t suffer from vendor lock-in. To my joy, GitOps landed in that sweet spot.

Segregation of duties (SoD) is a controversial topic and is often viewed as an impediment to continuous everything. SoD warns us against giving total control to one individual or a group of people to ensure clear separation of concerns. This prevents any one individual or group from becoming too powerful.

Traditional teams misinterpret this notion of balancing power by manually handing off versioned artifacts from one silo to the next and requesting sign-offs via a ticketing system. Manual handoffs and sign-offs interrupt the continuous paradigm and often leads to two or more disjoint pipelines that have to be manually orchestrated. Additionally, the malpractice of throwing things over the wall and having them thrown right back at us, only to throw them over again, reflects poor culture and often leads to a blame game.

With GitOps, all changes are made using pull requests that have time stamps and participants associated. By definition, a pull request informs team members that I have pushed changes to a repository that need to advance through the guided workflow. Hence there is full clarity on who changed what and when and who the change was approved by. Additionally, during audits and production issues, GitOps makes it painless to trace issues back to the root cause since all changes are recorded as commits in version control.

Kubectl exposes the machinery of the Kubernetes object model, which is quite complex. Instead of using kubectl to update the cluster directly, teams should interact with the system at a higher level of abstraction. Update scripts, that typically tend to group kubectl commands, are not always deterministic or idempotent. With this scripting approach, CI does not converge to a declarative model of the cluster. Instead, it throws an error and the pipeline aborts.

Also, to author effective scripts, teams need to understand intricate system semantics that requires expertise that most do not possess and others do not want to invest into. So, the question is: how can we steer clear from this anti-pattern of scripting a set of Kubernetes updates and executing that script as a pipeline step to push changes to the cluster?

With a GitOps flavored pipeline, teams can update their Kubernetes cluster via Git. For the umpteenth time, let’s try to (re)hash the not-so-fine line between continuous integration and continuous delivery (CD) and this time, in the brave new world of GitOps! Without splitting hairs, CI means merging updates to the controller branch and CD means the Kubernetes cluster should update itself based on those CI updates. Straightforward enough?

The next section continues to peel the onion and expose the inner mechanics of GitOps. It should resonate with teams who develop and deliver cloud native applications on Kubernetes.

With GitOps, troubleshooting becomes easier and faster, since we fix a production issue via a pull request rather than tinkering directly with a running system. If we limit cluster access to a handful of privileged admins, we can apply the same Git workflow to both operations and development. Changes to the application and cluster can then be contained to the following activities:

• Updates to container images

• Updates to the declarative specification, or in other words, the desired state

• Errors in the environment, like container malfunctions

Operating via pull requests reduces the number of variables which transform deployments into crime scenes! Fortunately (or, unfortunately for people who revel in heroics and theatrics), code deployers are no longer awarded “war hero” status for manually moving bits from point A to point B.

If a group of configuration updates is made by a human, the observed state in the cluster can drift from the desired state declared in Git, thus causing divergence. When the desired and observed states are different, Kubernetes provides a convergence mechanism to drive the observed state towards what the teams declared as the desired state in version control. The Kubernetes orchestrator will apply changes to the cluster until its state has converged to the declared configuration.

After a configurable interval, an alert could be generated if the states are still divergent. As an implementation detail, a tool that can send alerts is kubediff . Convergence is eventual and could be indicated by zero alerts. As a refresher, idempotence is the property of certain operations in mathematics and computer science whereby they can be applied multiple times without changing the result beyond the initial application. Convergence should be idempotent, as in, multiple applications of convergence should have the same outcome. To sum it up, think of convergence as a reconciliation loop that eventually brings the cluster to its desired state.

This model works for any Kubernetes resource and is extensible using Kubernetes Custom Resource Definitions (CRDs). In the next piece, let’s get into further implementation details with Jenkins X.

While different vendors are bringing GitOps solutions to the table, Jenkins X is an OSS project that has taken continuous everything for cloud native applications on Kubernetes by storm. Jenkins X leverages the best OSS projects in the Kubernetes ecosystem to implement GitOps. Jenkins X has a command line tool called jx that’s a one-stop shop for:

• Rapid creation of clusters on managed Kubernetes solutions from major cloud providers

• Quickstarts in major programming languages to fire up greenfield applications

• Importing existing brownfield applications that preserve your current investments

• Automated setup of pipelines shipping containerized applications

• Automated (or, optionally gated) promotions to production to experiment with new ideas

• Automated rollbacks in case those experiments don’t go well

Long story short, developing and delivering cloud native applications all the way to production through automated pipelines has become boring! Jenkins X leverages GitOps generously to setup and manage environments and application versions in Git repos, and not surprisingly, trends as one of the hottest OSS projects in the world of continuous everything.

GitOps is the recipe for success when it comes to continuous everything for cloud native applications on Kubernetes. Configuration as code and infrastructure as code are evergreen concepts that have been around for a while and GitOps, which surfaced much later, aligns well with those novel concepts. So, Git it right and you can recover from a total wipe out with little or no stress. Burnouts might finally become a thing of the past and we will be left with nothing short of zillions of happy and productive teams!

• See how GitOps can be used with CloudBees CodeShip and Terraform

• Read more about opinionated Jenkins X and Kubernetes