Under the Hood of Jenkins X - A CI/CD Solution for Cloud Applications on Kubernetes

Motivation

First, we articulated the business value that Jenkins X in automating Continuous Everything for Kubernetes . We encourage you to read it first.

If you recall, Jenkins X as an automated CI/CD solution for cloud applications on Kubernetes. Jenkins X automates the installation, configuration, and upgrades of tools, so that users can learn at their own pace. This vastly simplifies the user experience, since users don’t have to stomach all the various pieces on Day One, and can learn at their own pace.

With the built-in automation, teams can focus on releasing quality and secure software frequently and predictably to customers, without worrying about the shipping logistics. At CloudBees, we think it will be easier for our users if we tease things apart even before you start wondering what’s under the hood. This blog provides a quick overview on some of those “undercover agents,” including Jenkins itself, so that you can seamlessly continue on your journey in the continuous paradigm!

Jenkins X introduces the jx command line - a single command to rule them all.

You can either:

• Install the “jx” binary on your local machine, or

• Use the Google Cloud shell . The shell comes with most of the tools that you will need like Git, gcloud, kubectl, etc.

Some examples of using jx are:

• “jx create cluster gke “ creates a new Kubernetes cluster on Google Cloud

• “jx create quickstart” creates a new app and imports the generated code into Git and Jenkins for CI/CD

• “jx open jenkins” opens Jenkins in a browser

• “jx sync” synchronises your local files to a devpod

• “jx cloudbees” opens the CloudBees CD for Kubernetes app in a browser

• etc.

You can type “jx help” to find the various operations you can perform in Jenkins X.

Find details here.

Helm charts:

• Manage Kubernetes applications in a predictable and repeatable fashion, which means, they:

  • Define the app

  • Install the app

  • Upgrade the app

• Are versioned in source control, and hence can be audited

• Are easy to create and publish, and hence reduce communication overhead

• Make rollbacks to the last known good version stress-free (and drama-free)

In our modest example, Jenkins X creates the following Helm chart asset(s) in Git:

• https://github.com/junibutler/gojenx/tree/controller/charts/gojenx

Find details here.

Let’s understand the basics of a chart repository first.

A chart repository is:

• An HTTP server that houses an index.yaml file and optionally some packaged charts

• A location where packaged charts can be stored and shared

ChartMuseum:

• Is an open-source Helm chart repository written in Go

• Is Compatible with cloud storage backends like

  • Google Cloud Storage

  • Amazon S3

  • Microsoft Azure Blob Storage

  • Alibaba Cloud OSS Storage

  • Openstack Object Storage

• Generates the repository index (index.yaml) based on packages found in storage

  • You are no longer required to maintain your own version of index.yaml

  • If you store your own version of index.yaml, it will be completely ignored

• Provides a CLI for

  • Installation

  • Configuration

  • etc.

• Provides an API for:

  • Uploading a new chart version

  • Deleting a chart version

  • Uploading a new chart package to storage

  • Listing all the charts

  • etc.

Find details here.

Skaffold is a command line tool that:

• Facilitates CD for Kubernetes applications

• You can iterate on your application source code locally then deploy to local or remote Kubernetes clusters

• Handles the workflow for building, pushing and deploying your application, and leverages that workflow in an automated pipeline to promote applications from test to staging to production.

Jenkins X creates the following build and deploy workflow yaml using Skaffold:

• https://github.com/junibutler/gojenx/blob/controller/skaffold.yaml

Find details here.

Jenkins is an open source automation tool that:

• Is a self-contained Java-based program that can be used as a simple CI server or turned into a continuous delivery hub

• Has a plugin ecosystem to support building, testing, and deploying any project

• Requires Jenkinsfile that supports pipeline-as-code syntax/DSL in two formats:

  • Declarative syntax (recommended)

  • Scripted syntax

Jenkins X creates the following pipeline to promote through various stages:

• https://github.com/junibutler/gojenx/blob/controller/Jenkinsfile

Find details here.

Docker:

• Standardizes and automates the way we build, manage, and secure applications using containers

• Creates true independence between applications and infrastructure

• Creates a model for better collaboration and innovation between developers, QA, release, operations and security professionals

Jenkins X creates the following image for the containerized application:

• https://github.com/junibutler/gojenx/blob/controller/Dockerfile

Find details here.

Kubernetes runs production workloads at Google, which translates to running billions of containers per week

Among many other things, Kubernetes:

• Automates deployment, scaling, and management of containerized apps

• Groups containers that make up an application into logical units for easy management and discovery

• Gives you freedom to take advantage of on-premise, hybrid, or public cloud infrastructure, letting you effortlessly migrate workloads

• Restarts containers that fail

• Replaces and reschedules containers when nodes die

• Kills containers that don’t respond to health checks

• Scales your application up and down:

  • Automatically based on CPU usage or

  • Manually with a command or a UI

Find details here.

Ksync:

• Syncs files between our local file system and a Kubernetes cluster.

• Transparently updates containers running on the cluster from your local checkout.

• Enables developers to use their favorite IDEs to work from inside a cluster instead of from outside it. There is no reason to wait minutes to test code changes when you can see the results in seconds.

“jx sync” within Jenkins X allows us to sync files between our local lie system to the devpods on the Kubernetes cluster. This prevents typical problems like “works on my machine!” from slowing down business, even when everyone knows there are problems in production.

Find details here.

Prow is the system that handles GitHub events and commands for Kubernetes. It hails from the Kubernetes ecosystem and was created at Google. It is an event based solution that provides a loosely coupled architecture ideal for the cloud native world. It leverages ChatOps to interact with a build system. Prow manages its job configuration in Git. So, in the event of a wipeout, pipelines can be restored.

Prow includes a scalable and highly available webhook event handler that writes ProwJob CRDs to Kubernetes based on git events, so that other microservices can perform operations like running CI/CD pipelines. These git events can be triggered by new PRs and issues, comments, merges and pushes.

Jenkins X includes Prow out of the box when creating clusters on GKE (Google Kubernetes Engine) with Terraform using “jx create terraform” or when using the feature flag on “jx create cluster gke --prow” or “jx install --prow” commands. Prow, along with Knative build, makes serverless Jenkins possible that addresses problems originating from static Jenkins controllers.

Find details here.

Knative Build is a cloud native solution that extends Kubernetes with CRDs and provides ways to build applications from source. Teams can chain steps that run sequentially in the same Kubernetes pod using different containers where they can share state between each step.

Build templates are Kubernetes CRDs and are automatically upgraded with each Jenkins X release. The Prow configuration that is generated by Jenkins X when creating or importing applications refers to a build template. Knative build, along with Prow, makes serverless Jenkins possible that addresses problems originating from static Jenkins controllers.

Teams can leverage build templates to reference the type of Kubernetes pod that is created to run builds. These templates allow teams to specify which docker image to execute your build in, what environment variables exist at build time, what service accounts and secrets are in play and what volumes should be mounted.

For those of us who have tinkered with Kubernetes, we know it is difficult to setup and manage. Jenkins X alleviates that pain and leverages the managed Kubernetes solutions offered by major cloud providers, like:

• Google Cloud (GKE)

• Amazon Web Services (EKS)

• Microsoft Azure (AKS)

• Oracle Cloud (OKE)

• etc.

Let’s take the example of Google cloud. The Google cloud sdk provides the command-line interface for Google Cloud Platform products and services, like:

• Gcloud, which manages authentications, local configurations, developer workflow, and interactions

• Kubectl, which orchestrates the deployment and management of Kubernetes container clusters on gcloud

• etc.

Jenkins X automates CI/CD for cloud applications on Kubernetes. So that engineers can focus on writing and testing products that wow the world, Jenkins X abstracts away a lot of the shipping logistics that would otherwise consume your time and energy.

Jenkins X gets you started at the speed of light, however, the onus lies on you to gradually and eventually catch up to these underlying details. While we are sure you are well on your way to design and implement resilient pipelines, we wanted to make sure you have these details at your fingertips.

Pipeline is no longer a pipe dream - so, happy coding!