Cloudy with a chance of Kubernetes
With containers now omnipresent, Docker is the standard currency of exchange in a continuous everything platform. After some initial tussle, Kubernetes has emerged as the de-facto leader in cluster management and container orchestration. This provided some reprieve to a heavily fragmented industry that witnessed an alarming number of competing tools going through revolving doors.
However, while setting up and managing Kubernetes clusters can be fun for folks who like to tinker with infrastructure, some application developers and testers do not want to get bogged down with logistical and administrative fire drills. Even folks who feel comfortable managing Kubernetes on their own admit that it inflates their total cost of ownership (TCO).
Major cloud providers cashed in with managed Kubernetes solutions like Google Kubernetes Engine, Amazon Elastic Container Service for Kubernetes, Azure Kubernetes Service, Oracle Cloud Infrastructure Container Engine for Kubernetes and others, thus alleviating some of the pain points around managing Kubernetes. Our industry boasts a diverse array of technology stacks, tools, platforms and cloud strategies and hence we needed an operating model for Kubernetes that scales and doesn’t suffer from vendor lock-in. To my joy, GitOps landed in that sweet spot.
SoD != manual handoffs and sign-offs
Segregation of duties (SoD) is a controversial topic and is often viewed as an impediment to continuous everything. SoD warns us against giving total control to one individual or a group of people to ensure clear separation of concerns. This prevents any one individual or group from becoming too powerful.
Traditional teams misinterpret this notion of balancing power by manually handing off versioned artifacts from one silo to the next and requesting sign-offs via a ticketing system. Manual handoffs and sign-offs interrupt the continuous paradigm and often leads to two or more disjoint pipelines that have to be manually orchestrated. Additionally, the malpractice of throwing things over the wall and having them thrown right back at us, only to throw them over again, reflects poor culture and often leads to a blame game.
With GitOps, all changes are made using pull requests that have time stamps and participants associated. By definition, a pull request informs team members that I have pushed changes to a repository that need to advance through the guided workflow. Hence there is full clarity on who changed what and when and who the change was approved by. Additionally, during audits and production issues, GitOps makes it painless to trace issues back to the root cause since all changes are recorded as commits in version control.
CI/CD on Kubernetes need not be daunting
Kubectl exposes the machinery of the Kubernetes object model, which is quite complex. Instead of using kubectl to update the cluster directly, teams should interact with the system at a higher level of abstraction. Update scripts, that typically tend to group kubectl commands, are not always deterministic or idempotent. With this scripting approach, CI does not converge to a declarative model of the cluster. Instead, it throws an error and the pipeline aborts.
Also, to author effective scripts, teams need to understand intricate system semantics that requires expertise that most do not possess and others do not want to invest into. So, the question is: how can we steer clear from this anti-pattern of scripting a set of Kubernetes updates and executing that script as a pipeline step to push changes to the cluster?
With a GitOps flavored pipeline, teams can update their Kubernetes cluster via Git. For the umpteenth time, let’s try to (re)hash the not-so-fine line between continuous integration and continuous delivery (CD) and this time, in the brave new world of GitOps! Without splitting hairs, CI means merging updates to the master branch and CD means the Kubernetes cluster should update itself based on those CI updates. Straightforward enough?
The next section continues to peel the onion and expose the inner mechanics of GitOps. It should resonate with teams who develop and deliver cloud native applications on Kubernetes.
How GitOps works
Changes are made via pull requests
With GitOps, troubleshooting becomes easier and faster, since we fix a production issue via a pull request rather than tinkering directly with a running system. If we limit cluster access to a handful of privileged admins, we can apply the same Git workflow to both operations and development. Changes to the application and cluster can then be contained to the following activities:
Updates to container images
Updates to the declarative specification, or in other words, the desired state
Errors in the environment, like container malfunctions
Operating via pull requests reduces the number of variables which transform deployments into crime scenes! Fortunately (or, unfortunately for people who revel in heroics and theatrics), code deployers are no longer awarded “war hero” status for manually moving bits from point A to point B.
Convergence ensures eventual consistency
If a group of configuration updates is made by a human, the observed state in the cluster can drift from the desired state declared in Git, thus causing divergence. When the desired and observed states are different, Kubernetes provides a convergence mechanism to drive the observed state towards what the teams declared as the desired state in version control. The Kubernetes orchestrator will apply changes to the cluster until its state has converged to the declared configuration.
After a configurable interval, an alert could be generated if the states are still divergent. As an implementation detail, a tool that can send alerts is kubediff. Convergence is eventual and could be indicated by zero alerts. As a refresher, idempotence is the property of certain operations in mathematics and computer science whereby they can be applied multiple times without changing the result beyond the initial application. Convergence should be idempotent, as in, multiple applications of convergence should have the same outcome. To sum it up, think of convergence as a reconciliation loop that eventually brings the cluster to its desired state.
This model works for any Kubernetes resource and is extensible using Kubernetes Custom Resource Definitions (CRDs). In the next piece, let’s get into further implementation details with Jenkins X.
Continuous everything on Kubernetes with Jenkins X
While different vendors are bringing GitOps solutions to the table, Jenkins X is an OSS project that has taken continuous everything for cloud native applications on Kubernetes by storm. Jenkins X leverages the best OSS projects in the Kubernetes ecosystem to implement GitOps. Jenkins X has a command line tool called jx that’s a one-stop shop for:
Rapid creation of clusters on managed Kubernetes solutions from major cloud providers
Quickstarts in major programming languages to fire up greenfield applications
Importing existing brownfield applications that preserve your current investments
Automated setup of pipelines shipping containerized applications
Automated (or, optionally gated) promotions to production to experiment with new ideas
Automated rollbacks in case those experiments don’t go well
Long story short, developing and delivering cloud native applications all the way to production through automated pipelines has become boring! Jenkins X leverages GitOps generously to setup and manage environments and application versions in Git repos, and not surprisingly, trends as one of the hottest OSS projects in the world of continuous everything.